This article proposes a comprehensive approach for assessing the quality and appropriateness of exploratory factor analysis solutions intended for item calibration and individual scoring. Three groups of properties are assessed: (a) strength and replicability of the factorial solution, (b) determinacy and accuracy of the individual score estimates, and (c) closeness to unidimensionality in the case of multidimensional solutions. Within each group, indices are considered for two types of factor-analytic models: the linear model for continuous responses and the categorical-variable-methodology model that treats the item scores as ordered-categorical. All the indices proposed have been implemented in a noncommercial and widely known program for exploratory factor analysis. The usefulness of the proposal is illustrated with a real data example in the personality domain.
BeauducelA. (2011). Indeterminacy of factor scores in slightly misspecified confirmatory factor models. Journal of Modern Applied Statistical Methods, 10, 583-598.
3.
BeauducelA.HilgerN. (2017). The determinacy of the regression factor score predictor based on continuous parameter estimates from categorical variables. Communications in Statistics—Theory and Methods, 46, 3417-3425.
4.
BeauducelA.HilgerN. (2017). On the bias of factor score determinacy coefficients based on different estimation methods of the exploratory factor model. Communications in Statistics—Simulation and Computation, 46, 6144-6154.
5.
BockR. D.MislevyR. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
6.
BrownA.CroudaceT. (2015). Scoring and estimating score precision using multidimensional IRT. In ReiseS. P.RevickiD. A. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 307-333). New York, NY: Routledge.
7.
BussA. H.PerryM. (1992). The aggression questionnaire. Journal of Personality and Social Psychology, 63, 452-459.
8.
CattellR. B. (1986). The psychometric properties of tests: Consistency, validity and efficiency. In CattellR. B.JohnsonR. C. (Eds.), Functional psychological testing (pp. 54-78). New York, NY: Brunner/Mazel.
9.
CliffN. (1977). A theory of consistency of ordering generalizable to tailored testing. Psychometrika, 42, 375-399.
10.
CulpepperS. A. (2013). The reliability and precision of total scores and IRT estimates as a function of polytomous IRT parameters and latent trait distribution. Applied Psychological Measurement, 37, 201-225.
11.
FerrandoP. J. (2009). Difficulty, discrimination, and information indices in the linear factor analysis model for continuous item responses. Applied Psychological Measurement, 33, 9-24.
12.
FerrandoP. J.Lorenzo-SevaU. (2013). Unrestricted item factor analysis and some relations with item response theory (Technical report). Department of Psychology, Universitat Rovira i Virgili, Tarragona. Retrieved from http://psico.fcep.urv.es/utilitats/factor
13.
FerrandoP. J.Lorenzo-SevaU. (2016). A note on improving EAP trait estimation in oblique factor-analytic and item response theory models. Psicologica, 37, 235-247.
14.
FerrandoP. J.Lorenzo-SevaU. (2017). Program FACTOR at 10: Origins, development and future directions. Psicothema, 29, 236-240.
15.
GorsuchR. L. (1983). Factor analysis. Hillsdale, NJ: LEA.
16.
GreenB. F.BockR. D.HumphreysL. G.LinnR. L.ReckaseM. D. (1984). Technical guidelines for assessing computerized adaptative tests. Journal of Educational Measurement, 21, 347-360.
17.
GriceJ. W. (2001). Computing and evaluating factor scores. Psychological Methods, 6, 430-450.
18.
GuttmanL. (1955). The determinacy of factor score matrices with implications for five other basic problems of common-factor theory. British Journal of Statistical Psychology, 8, 65-81.
19.
HancockG. R. (2001). Effect size, power, and sample size determination for structured means modeling and MIMIC approaches to between-groups hypothesis testing of means on a single latent construct. Psychometrika, 66, 373-388.
20.
HancockG. R.MuellerR. O. (2000). Rethinking construct reliability within latent variable systems. In CudekR.duToitS. H. C.SorbomD. F. (Eds.), Structural equation modeling: Present and future (pp. 195-216). Lincolnwood, IL: Scientific Software.
21.
HarmanH. H. (1962). Modern factor analysis (2nd ed.). Chicago, IL: University of Chicago Press.
22.
KrijnenW. P.WansbeekT.ten BergeJ. M. F. (1996). Best linear predictors for factor scores. Communications in Statistics—Theory and Methods, 25, 3013-3015.
23.
Lorenzo-SevaU. (1999). Promin: A method for oblique factor rotation. Multivariate Behavioral Research, 34, 347-356.
24.
MaraunM. D. (1996). Metaphor taken as math: Indeterminacy in the factor analysis model. Multivariate Behavioral Research, 31, 517-538.
25.
MarshH. W.MorinA. J.ParkerP. D.KaurG. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85-110.
26.
McDonaldR. P. (1982). Linear versus models in item response theory. Applied Psychological Measurement, 6, 379-396.
27.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: LEA.
28.
MellenberghG. J. (1996). Measurement precision in test score and item response models. Psychological Methods, 1, 293-299.
29.
MulaikS. A. (2010). Foundations of factor analysis. Boca Raton, FL: CRC Press.
30.
OlssonU.DrasgowF.DoransN. J. (1982). The polyserial correlation coefficient. Psychometrika, 47, 337-347.
31.
RajuN. S.PriceL. R.OshimaT. C.NeringM. L. (2007). Standardized conditional SEM: A case for conditional reliability. Applied Psychological Measurement, 31, 169-180.
32.
RaykovT. (2002). Analytic estimation of standard error and confidence interval for scale reliability. Multivariate Behavioral Research, 37, 89-103.
33.
ReiseS. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696.
34.
ReiseS. P.BonifayW. E.HavilandM. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129-140.
35.
ReiseS. R.CookK. F.MooreT. M. (2015). Evaluating the impact of multidimensionality on unidimensional item response theory model parameters. In ReiseS. P.RevickiD. A. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 13-40). New York, NY: Routledge.
36.
RhemtullaM.Brosseau-LiardP. E.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373.
37.
RodriguezA.ReiseS. P.HavilandM. G. (2016a). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137-150.
38.
RodriguezA.ReiseS. P.HavilandM. G. (2016b). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98, 223-237.
39.
SamejimaF. (1977). Weakly parallel tests in latent trait theory with some criticism of classical test theory. Psychometrika, 42, 193-198.
40.
StuckyB. D.ThissenD.Orlando EdelenM. (2013). Using logistic approximations of marginal trace lines to develop short assessments. Applied Psychological Measurement, 37, 41-57.
41.
ten BergeJ. M.KiersH. A. (1991). A numerical approach to the approximate and the exact minimum rank of a covariance matrix. Psychometrika, 56, 309-315.
42.
ThurstoneL. L. (1935). The vectors of mind. Chicago, IL: University of Chicago Press.
43.
Vigil-ColetA.Lorenzo-SevaU.Morales-VivesF. (2015). The effects of ageing on self-reported aggression measures are partly explained by response bias. Psicothema, 27, 209-215.
44.
YuanK. H.ChanW.MarcoulidesG. A.BentlerP. M. (2016). Assessing structural equation models by equivalence testing with adjusted fit indexes. Structural Equation Modeling, 23, 319-330.
45.
YuleG. U. (1907). On the theory of correlation for any number of variables, treated by a new system of notation. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 79, 182-193.
