Range Restriction Affects Factor Analysis: Normality,Estimation,Fit,Loadings,and Reliability

Abstract

A sample suffers range restriction (RR) when its variance is reduced comparing with its population variance and, in turn, it fails representing such population. If the RR occurs over the latent factor, not directly over the observed variable, the researcher deals with an indirect RR, common when using convenience samples. This work explores how this problem affects different outputs of the factor analysis: multivariate normality (MVN), estimation process, goodness-of-fit, recovery of factor loadings, and reliability. In doing so, a Monte Carlo study was conducted. Data were generated following the linear selective sampling model, simulating tests varying their sample size ( $N$ = 200 and 500 cases), test size ( $J$ = 6, 12, 18, and 24 items), loading size ( $L$ = .50, .70, and .90), and restriction size (from $R$ = 1, .90, .80, and so on till .10 selection ratio). Our results systematically suggest that an interaction between decreasing the loading size and increasing the restriction size affects the MVN assessment, obstructs the estimation process, and leads to an underestimation of the factor loadings and reliability. However, most of the MVN tests and most of the fit indices employed were nonsensitive to the RR problem. We provide some recommendations to applied researchers.

Keywords

range restriction convenience sampling factor analysis validation simulation

Get full access to this article

View all access options for this article.

References

Alpu

Yuksek

(2016). Comparison of some multivariate normality tests: A simulation study. International Journal of Advanced and Applied Sciences, 3(12), 73–85. https://doi.org/g8cw

Arnett

J. J.

(2008). The neglected 95%: Why American psychology needs to become less American. The American Psychologist, 63(7), 602–614. https://doi.org/c54kcm

Asparouhov

Muthén

(2010). Simple second order chi-square correction. Mplus technical appendix. https://www.statmodel.com/download/WLSMV_new_chi21.pdf

Blanca

M. J.

Alarcón

Arnau

Bono

Bendayan

(2018). Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit? Behavior Research Methods, 50(3), 937–962. https://doi.org/gdqrwq

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. Guilford.

Chou

C. P.

Bentler

P. M.

Satorra

(1991). Scaled test statistics and robust standard errors for non-normal data in covariance structure analysis: A Monte Carlo study. British Journal of Mathematical and Statistical Psychology, 44(2), 347–357. https://doi.org/dx97ns

Cohen

(1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. https://doi.org/cxqf2s

Cronbach

L. J.

(1951). Coefficient α and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/cc5

Dahlke

J. A.

Wiernik

B. M.

(2020). Not restricted to selection research: Accounting for indirect range restriction in organizational research. Organizational Research Methods, 23(4), 717–749. https://doi.org/gf9crt

10.

Dillon

W. R.

Kumar

Mulani

(1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101(1), 126–135. https://doi.org/bsgpv8

11.

Ebner

Henze

(2020). Tests for multivariate normality —A critical review with emphasis on weighted L2-statistics. TEST, 29, 845–892. https://doi.org/g8cx

12.

Etikan

Musa

S. A.

Alkassim

R. S.

(2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/dwgz

13.

Fife

D. A.

(2013). The Achilles heel of psychology: How convenience sampling affects parameter estimates [Doctoral dissertation, University of Oklahoma]. https://hdl.handle.net/11244/319258

14.

Fife

D. A.

Mendoza

J. L.

Terry

(2012). The assessment of reliability under range restriction: A comparison of α, ω, and test–retest reliability for dichotomous data. Educational and Psychological Measurement, 72(5), 862–888. https://doi.org/gf4t9c

15.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/fqxz8j

16.

Gonzalez-Estrada

Villasenor-Alva

J. A.

(2017). Goft: Tests of fit for some probability distributions [R package version 1.3.4]. https://CRAN.R-project.org/package=goft

17.

Hanel

P. H. P.

Vione

K. C.

(2016). Do student samples provide an accurate estimate of the general public? PLOS ONE, 11(12), Article e0168354. https://doi.org/f9kc4p

18.

Henze

Wagner

(1997). A new approach to the BHEP tests for multivariate normality. Journal of Multivariate Analysis, 62(1), 1–23. https://doi.org/cvtz72

19.

Henze

Zirkler

(1990). A class of invariant consistent tests for multivariate normality. Communications in Statistics—Theory and Methods, 19(10), 3595–3617. https://doi.org/bwng9v

20.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/dbt

21.

Hunter

J. E.

Schmidt

F. L.

(2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594–612. https://doi.org/bt4t68

22.

IBM Corp. (2017). IBM SPSS Statistics for Windows. Version 25.

23.

Jackson

P. H.

Agunwamba

C. C.

(1977). Coefficient α and the internal structure of tests. Psychometrika, 42(4), 567–578. https://doi.org/d98rr5

24.

Johns

(1991). Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Processes, 49(1), 80–104. https://doi.org/cst6hg

25.

Kirkegaard

E. O.

Nyborg

(2021). Intelligence and general psychopathology in the Vietnam experience study: A closer look. Mankind Quarterly, 61(4), 792–819. https://doi.org/hckx

26.

Kline

R. B.

(2015). Principles and practice of structural equation modeling. Guilford.

27.

Kolenikov

Bollen

K. A.

(2012). Testing negative error variances: Is a Heywood case a symptom of misspecification? Sociological Methods & Research, 41(1), 124–167.

28.

Korkmaz

Goksuluk

Zararsiz

(2014). MVN: An R package for assessing multivariate normality. The R Journal, 6(2), 151–162. https://journal.r-project.org/archive/2014-2/korkmaz-goksuluk-zararsiz.pdf

29.

Lei

Lomax

R. G.

(1991). The effect of varying degrees of nonnormality. Structural Equation Modeling, Structural Equation Modeling: A Multidisciplinary Journal, 12(1), 1–27. https://doi.org/bnf8ks

30.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Addison-Wesley.

31.

MacCallum

R. C.

Widaman

K. F.

Preacher

K. J.

Hong

(2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637. https://doi.org/b3v4p6

32.

Mardia

K. V.

(1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519–530. https://doi.org/fdt2bp

33.

Maydeu-Olivares

(2017). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 383–394. https://doi.org/gfppfh

34.

McCredie

M. N.

Morey

L. C.

(2018). Who are the Turkers? A characterization of MTurk workers using the Personality Assessment Inventory. Assessment, 26(5), 759–766. https://doi.org/gg9rmf

35.

McDonald

R. P.

(1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23, 1–21. https://doi.org/fqg277

36.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/gcsk2k

37.

Mecklin

C. J.

Mundfrom

D. J.

(2005). A Monte Carlo comparison of the Type I and Type II error rates of tests of multivariate normality. Journal of Statistical Computation and Simulation, 75(2), 93–107. https://doi.org/c4x9gm

38.

Mendoza

J. L.

Mumford

(1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12(3), 282–293. https://doi.org/fh756j

39.

Mook

D. G.

(1983). In defense of external invalidity. The American Psychologist, 38(4), 379–387. https://doi.org/b5pbjk

40.

Murray

A. L.

Johnson

McGue

Iacono

W. G.

(2014). How are conscientiousness and cognitive ability related to one another? A re-examination of the intelligence compensation hypothesis. Personality and Individual Differences, 70, 17–22. https://doi.org/f6gvt6

41.

Nielsen

Haun

Kärtner

Legare

C. H.

(2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. https://doi.org/gf8x27

42.

Oltmanns

J. R.

Widiger

T. A.

(2020). The five-factor personality inventory for ICD-11: A facet-level assessment of the ICD-11 trait model. Psychological Assessment, 32(1), 60. https://doi.org/gg638m

43.

Ondé

Alvarado

J. M.

(2020). Reconsidering the conditions for conducting confirmatory factor analysis. The Spanish Journal of Psychology, 23(E55), 1–15. https://doi.org/g8cz

44.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 688–701. https://doi.org/68c

45.

Oppong

F. B.

Agbedra

S. Y.

(2016). Assessing univariate and multivariate normality, a guide for non-statisticians. Mathematical Theory and Modeling, 6, 26–33.

46.

Pearson

(1903). I. Mathematical contributions to the theory of evolution—XI. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 200(321–330), 1–66. https://doi.org/dgb3gs

47.

Peterson

R. A.

Merunka

D. R.

(2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67(5), 1035–1041. https://doi.org/gf8wmt

48.

Pfaffel

Kollmayer

Schober

Spiel

(2016). A missing data approach to correct for direct and indirect range restrictions with a dichotomous criterion: A simulation study. PLOS ONE, 11(3), Article e0152330. https://doi.org/f8whvv

49.

R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

50.

Ree

M. J.

Carretta

T. R.

(2020). A systematic review of corrections for range restriction: Five things consultants should know (Technical Report). Air Force Research Laboratory.

51.

Revelle

(2020). Psych: Procedures for psychological, psychometric, and personality research [R package version 2.0.12]. Northwestern University. https://CRAN.R-project.org/package=psych

52.

Romeu

J. L.

Ozturk

(1993). A comparative study of goodness-of-fit tests for multivariate normality. Journal of Multivariate Analysis, 46(2), 309–334. https://doi.org/fb9dq2

53.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. http://www.jstatsoft.org/v48/i02/

54.

Royston

J. P.

(1982). An extension of Shapiro and Wilk’s W test for normality to large samples. Journal of the Royal Statistical Society: Series C (Applied Statistics), 31(2), 115–124. https://doi.org/dh6fpx

55.

Royston

J. P.

(1983). Some techniques for assessing multivariate normality based on the Shapiro-Wilk W. Journal of the Royal Statistical Society: Series C (Applied Statistics), 32(2), 121–133. https://doi.org/bfm784

56.

Sackett

P. R.

Laczo

R. M.

Arvey

R. D.

(2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807–825. https://doi.org/c2g7gw

57.

Sackett

P. R.

Yang

(2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85(1), 112–118. https://doi.org/c6npmd

58.

Satorra

Bentler

P. M.

(1994). Corrections to test statistics and standard errors in covariance structure analysis. In von Eye

Clogg

C. C.

(Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). SAGE.

59.

Satterthwaite

F. E.

(1941). Synthesis of variance. Psychometrika, 6, 309–316. https://doi.org/dgq9g8

60.

Schweizer

Ren

Zeller

(2019). On modeling the ceiling effect observed in cognitive data in the framework of confirmatory factor analysis. Psychological Test and Assessment Modeling, 61(3), 333–353.

61.

Sears

D. O.

(1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515–530. https://doi.org/fgzfzc

62.

Skrondal

(2000). Design and analysis of Monte Carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167. https://doi.org/fmvfpd

63.

Sorrel

M. A.

Garcıa

L. F.

Aluja

Rolland

J.-P.

Rossier

Roskam

Abad

F. J.

(2021). Cross-cultural measurement invariance in the personality inventory for DSM-5. Psychiatry Research, 304, 114134. https://doi.org/g8cv

64.

Thalmayer

A. G.

Toscanelli

Arnett

J. J.

(2021). The neglected 95% revisited: Is American psychology becoming less American? The American Psychologist, 76, 116–129. https://doi.org/gg4vbg

65.

Thorndike

R. L.

(1949). Personnel selection; test and measurement techniques. Wiley.

66.

Trizano-Hermosilla

Alvarado

J. M.

(2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7(769), 567–578. https://doi.org/gfjz27

67.

Tucker

L. R.

MacCallum

R. C.

(1997). Exploratory factor analysis. http://inis.jinr.ru/sl/M_Mathematics/MV_Probability/MVas_Applied%20statistics/Tucker%20L.R.,%20MacCallum%20R.C.%20Exploratory%20factor%20analysis%20(1997)(459s).pdf

68.

Venables

W. N.

Ripley

B. D.

(2002). Modern Applied Statistics with S (4th ed.). Springer. http://www.stats.ox.ac.uk/pub/MASS4

69.

Villasenor-Alva

J. A.

González Estrada

(2009). A generalization of Shapiro–Wilk’s test for multivariate normality. Communications in Statistics—Theory and Methods, 38(11), 1870–1883. https://doi.org/btnf9h

70.

Worthington

R. L.

Whittaker

T. A.

(2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/fp9htj

71.

Ximénez

(2006). A Monte Carlo study of recovery of weak factor loadings in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 13(4), 587–614. https://doi.org/d5mzgn