A sample suffers range restriction (RR) when its variance is reduced comparing with its population variance and, in turn, it fails representing such population. If the RR occurs over the latent factor, not directly over the observed variable, the researcher deals with an indirect RR, common when using convenience samples. This work explores how this problem affects different outputs of the factor analysis: multivariate normality (MVN), estimation process, goodness-of-fit, recovery of factor loadings, and reliability. In doing so, a Monte Carlo study was conducted. Data were generated following the linear selective sampling model, simulating tests varying their sample size ( = 200 and 500 cases), test size ( = 6, 12, 18, and 24 items), loading size ( = .50, .70, and .90), and restriction size (from = 1, .90, .80, and so on till .10 selection ratio). Our results systematically suggest that an interaction between decreasing the loading size and increasing the restriction size affects the MVN assessment, obstructs the estimation process, and leads to an underestimation of the factor loadings and reliability. However, most of the MVN tests and most of the fit indices employed were nonsensitive to the RR problem. We provide some recommendations to applied researchers.
AlpuO.YuksekD. (2016). Comparison of some multivariate normality tests: A simulation study. International Journal of Advanced and Applied Sciences, 3(12), 73–85. https://doi.org/g8cw
2.
ArnettJ. J. (2008). The neglected 95%: Why American psychology needs to become less American. The American Psychologist, 63(7), 602–614. https://doi.org/c54kcm
BlancaM. J.AlarcónR.ArnauJ.BonoR.BendayanR. (2018). Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?Behavior Research Methods, 50(3), 937–962. https://doi.org/gdqrwq
5.
BrownT. A. (2015). Confirmatory factor analysis for applied research. Guilford.
6.
ChouC. P.BentlerP. M.SatorraA. (1991). Scaled test statistics and robust standard errors for non-normal data in covariance structure analysis: A Monte Carlo study. British Journal of Mathematical and Statistical Psychology, 44(2), 347–357. https://doi.org/dx97ns
7.
CohenJ. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. https://doi.org/cxqf2s
8.
CronbachL. J. (1951). Coefficient α and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/cc5
9.
DahlkeJ. A.WiernikB. M. (2020). Not restricted to selection research: Accounting for indirect range restriction in organizational research. Organizational Research Methods, 23(4), 717–749. https://doi.org/gf9crt
10.
DillonW. R.KumarA.MulaniN. (1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101(1), 126–135. https://doi.org/bsgpv8
11.
EbnerB.HenzeN. (2020). Tests for multivariate normality —A critical review with emphasis on weighted L2-statistics. TEST, 29, 845–892. https://doi.org/g8cx
12.
EtikanI.MusaS. A.AlkassimR. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/dwgz
13.
FifeD. A. (2013). The Achilles heel of psychology: How convenience sampling affects parameter estimates [Doctoral dissertation, University of Oklahoma]. https://hdl.handle.net/11244/319258
14.
FifeD. A.MendozaJ. L.TerryR. (2012). The assessment of reliability under range restriction: A comparison of α, ω, and test–retest reliability for dichotomous data. Educational and Psychological Measurement, 72(5), 862–888. https://doi.org/gf4t9c
15.
ForeroC. G.Maydeu-OlivaresA.Gallardo-PujolD. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/fqxz8j
16.
Gonzalez-EstradaE.Villasenor-AlvaJ. A. (2017). Goft: Tests of fit for some probability distributions [R package version 1.3.4]. https://CRAN.R-project.org/package=goft
17.
HanelP. H. P.VioneK. C. (2016). Do student samples provide an accurate estimate of the general public?PLOS ONE, 11(12), Article e0168354. https://doi.org/f9kc4p
18.
HenzeN.WagnerT. (1997). A new approach to the BHEP tests for multivariate normality. Journal of Multivariate Analysis, 62(1), 1–23. https://doi.org/cvtz72
19.
HenzeN.ZirklerB. (1990). A class of invariant consistent tests for multivariate normality. Communications in Statistics—Theory and Methods, 19(10), 3595–3617. https://doi.org/bwng9v
20.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/dbt
21.
HunterJ. E.SchmidtF. L.LeH. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594–612. https://doi.org/bt4t68
22.
IBM Corp. (2017). IBM SPSS Statistics for Windows. Version 25.
23.
JacksonP. H.AgunwambaC. C. (1977). Coefficient α and the internal structure of tests. Psychometrika, 42(4), 567–578. https://doi.org/d98rr5
24.
JohnsG. (1991). Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Processes, 49(1), 80–104. https://doi.org/cst6hg
25.
KirkegaardE. O.NyborgH. (2021). Intelligence and general psychopathology in the Vietnam experience study: A closer look. Mankind Quarterly, 61(4), 792–819. https://doi.org/hckx
26.
KlineR. B. (2015). Principles and practice of structural equation modeling. Guilford.
27.
KolenikovS.BollenK. A. (2012). Testing negative error variances: Is a Heywood case a symptom of misspecification?Sociological Methods & Research, 41(1), 124–167.
LeiM.LomaxR. G. (1991). The effect of varying degrees of nonnormality. Structural Equation Modeling, Structural Equation Modeling: A Multidisciplinary Journal, 12(1), 1–27. https://doi.org/bnf8ks
30.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
31.
MacCallumR. C.WidamanK. F.PreacherK. J.HongS. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637. https://doi.org/b3v4p6
32.
MardiaK. V. (1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519–530. https://doi.org/fdt2bp
33.
Maydeu-OlivaresA. (2017). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 383–394. https://doi.org/gfppfh
34.
McCredieM. N.MoreyL. C. (2018). Who are the Turkers? A characterization of MTurk workers using the Personality Assessment Inventory. Assessment, 26(5), 759–766. https://doi.org/gg9rmf
35.
McDonaldR. P. (1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23, 1–21. https://doi.org/fqg277
36.
McNeishD. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/gcsk2k
37.
MecklinC. J.MundfromD. J. (2005). A Monte Carlo comparison of the Type I and Type II error rates of tests of multivariate normality. Journal of Statistical Computation and Simulation, 75(2), 93–107. https://doi.org/c4x9gm
38.
MendozaJ. L.MumfordM. (1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12(3), 282–293. https://doi.org/fh756j
39.
MookD. G. (1983). In defense of external invalidity. The American Psychologist, 38(4), 379–387. https://doi.org/b5pbjk
40.
MurrayA. L.JohnsonW.McGueM.IaconoW. G. (2014). How are conscientiousness and cognitive ability related to one another? A re-examination of the intelligence compensation hypothesis. Personality and Individual Differences, 70, 17–22. https://doi.org/f6gvt6
41.
NielsenM.HaunD.KärtnerJ.LegareC. H. (2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. https://doi.org/gf8x27
42.
OltmannsJ. R.WidigerT. A. (2020). The five-factor personality inventory for ICD-11: A facet-level assessment of the ICD-11 trait model. Psychological Assessment, 32(1), 60. https://doi.org/gg638m
43.
OndéD.AlvaradoJ. M. (2020). Reconsidering the conditions for conducting confirmatory factor analysis. The Spanish Journal of Psychology, 23(E55), 1–15. https://doi.org/g8cz
44.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 688–701. https://doi.org/68c
45.
OppongF. B.AgbedraS. Y. (2016). Assessing univariate and multivariate normality, a guide for non-statisticians. Mathematical Theory and Modeling, 6, 26–33.
46.
PearsonK. (1903). I. Mathematical contributions to the theory of evolution—XI. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 200(321–330), 1–66. https://doi.org/dgb3gs
47.
PetersonR. A.MerunkaD. R. (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67(5), 1035–1041. https://doi.org/gf8wmt
48.
PfaffelA.KollmayerM.SchoberB.SpielC. (2016). A missing data approach to correct for direct and indirect range restrictions with a dichotomous criterion: A simulation study. PLOS ONE, 11(3), Article e0152330. https://doi.org/f8whvv
49.
R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
50.
ReeM. J.CarrettaT. R. (2020). A systematic review of corrections for range restriction: Five things consultants should know (Technical Report). Air Force Research Laboratory.
51.
RevelleW. (2020). Psych: Procedures for psychological, psychometric, and personality research [R package version 2.0.12]. Northwestern University. https://CRAN.R-project.org/package=psych
52.
RomeuJ. L.OzturkA. (1993). A comparative study of goodness-of-fit tests for multivariate normality. Journal of Multivariate Analysis, 46(2), 309–334. https://doi.org/fb9dq2
53.
RosseelY. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. http://www.jstatsoft.org/v48/i02/
54.
RoystonJ. P. (1982). An extension of Shapiro and Wilk’s W test for normality to large samples. Journal of the Royal Statistical Society: Series C (Applied Statistics), 31(2), 115–124. https://doi.org/dh6fpx
55.
RoystonJ. P. (1983). Some techniques for assessing multivariate normality based on the Shapiro-Wilk W. Journal of the Royal Statistical Society: Series C (Applied Statistics), 32(2), 121–133. https://doi.org/bfm784
56.
SackettP. R.LaczoR. M.ArveyR. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807–825. https://doi.org/c2g7gw
57.
SackettP. R.YangH. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85(1), 112–118. https://doi.org/c6npmd
58.
SatorraA.BentlerP. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In von EyeA.CloggC. C. (Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). SAGE.
59.
SatterthwaiteF. E. (1941). Synthesis of variance. Psychometrika, 6, 309–316. https://doi.org/dgq9g8
60.
SchweizerK.RenX.ZellerF. (2019). On modeling the ceiling effect observed in cognitive data in the framework of confirmatory factor analysis. Psychological Test and Assessment Modeling, 61(3), 333–353.
61.
SearsD. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515–530. https://doi.org/fgzfzc
62.
SkrondalA. (2000). Design and analysis of Monte Carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167. https://doi.org/fmvfpd
63.
SorrelM. A.GarcıaL. F.AlujaA.RollandJ.-P.RossierJ.RoskamI.AbadF. J. (2021). Cross-cultural measurement invariance in the personality inventory for DSM-5. Psychiatry Research, 304, 114134. https://doi.org/g8cv
64.
ThalmayerA. G.ToscanelliC.ArnettJ. J. (2021). The neglected 95% revisited: Is American psychology becoming less American?The American Psychologist, 76, 116–129. https://doi.org/gg4vbg
65.
ThorndikeR. L. (1949). Personnel selection; test and measurement techniques. Wiley.
66.
Trizano-HermosillaI.AlvaradoJ. M. (2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7(769), 567–578. https://doi.org/gfjz27
Villasenor-AlvaJ. A.González EstradaE. (2009). A generalization of Shapiro–Wilk’s test for multivariate normality. Communications in Statistics—Theory and Methods, 38(11), 1870–1883. https://doi.org/btnf9h
70.
WorthingtonR. L.WhittakerT. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/fp9htj
71.
XiménezC. (2006). A Monte Carlo study of recovery of weak factor loadings in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 13(4), 587–614. https://doi.org/d5mzgn
