We conducted an extensive Monte Carlo study of factor-rotation local solutions (LS) in multidimensional, two-parameter logistic (M2PL) item response models. In this study, we simulated more than 19,200 data sets that were drawn from 96 model conditions and performed more than 7.6 million rotations to examine the influence of (a) slope parameter sizes, (b) number of indicators per factor (trait), (c) probabilities of cross-loadings, (d) factor correlation sizes, (e) model approximation error, and (f) sample sizes on the local solution rates of the oblimin and (oblique) geomin rotation algorithms. To accommodate these design variables, we extended the standard M2PL model to include correlated major factors and uncorrelated minor factors (to represent model error). Our results showed that both rotation methods converged to LS under some conditions with geomin producing the highest local solution rates across many models. Our results also showed that, for identical item response patterns, rotation LS can produce different latent trait estimates with different levels of measurement precision (as indexed by the conditional standard error of measurement). Follow-up analyses revealed that when rotation algorithms converged to multiple solutions, quantitative indices of structural fit, such as numerical measures of simple structure, will often misidentify the rotation that is closest in mean-squared error to the factor pattern (or item-slope pattern) of the data-generating model.
BernaardsC. A.JennrichR. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65(5), 676–696. https://doi.org/10.1177/0013164404272507
BockR. D.MislevyR. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405
6.
BoltD. M.LallV. F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 27(6), 395–414. https://doi.org/10.1177/0146621603258350
BoxG. E. P. (1979). Robustness in the strategy of scientific model building. In LaunerR. L.WilkinsonG. N. (Eds.), Robustness in statistics (pp. 201–236). Academic Press. https://doi.org/10.1016/B978-0-12-438150-6.50018-2
9.
BriggsN. E.MacCallumR. C. (2003). Recovery of weak common factors by maximum likelihood and ordinary least squares estimation. Multivariate Behavioral Research, 38(1), 25–56. https://doi.org/10.1207/S15327906MBR3801_2
10.
BrowneM. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36(1), 111–150. https://doi.org/10.1207/S15327906MBR3601_05
CaiL. (2010). High-dimensional exploratory item factor analysis by a metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75(1), 33–57. https://doi.org/10.1007/s11336-009-9136-x
13.
CaiL.ChungS. W.LeeT. (2021). Incremental model fit assessment in the case of categorical data: Tucker–Lewis index for item response theory modeling. Prevention Science, 24, 455–466. https://doi.org/10.1007/s11121-021-01253-4
14.
CaiL.HansenM. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66(2), 245–276. https://doi.org/10.1111/j.2044-8317.2012.02050.x
CarrollJ. B. (1953). An analytical solution for approximating simple structure in factor analysis. Psychometrika, 18(1), 23–38. https://doi.org/10.1007/BF02289025
17.
CattellR. B.GorsuchR. L. (1963). The uniqueness and significance of simple structure demonstrated by contrasting organic “natural structure” and “random structure” data. Psychometrika, 28, 55–67. https://doi.org/10.1007/BF02289548
18.
CattellR. B.MuerleJ. L. (1960). The “maxplane” program for factor rotation to oblique simple structure. Educational and Psychological Measurement, 20(3), 569–590. https://doi.org/10.1177/001316446002000314
19.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(1), 1–29. https://doi.org/10.18637/jss.v048.i06
ChungS.CaiL. (2021). Cross-classified random effects modeling for moderated item calibration. Journal of Educational and Behavioral Statistics, 46(6), 651–681. https://doi.org/10.3102/1076998620983908
22.
CudeckR.BrowneM. W. (1992). Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value. Psychometrika, 57, 357–369. https://doi.org/10.1007/BF02295424
23.
DaveyT.NeringM. L.ThompsonT. (1997). Realistic simulation of item response data. ACT Research Report Series 97-4 [Technical Report]. American College Testing Program.
24.
De AyalaR. J. (2013). The theory and practice of item response theory. Guilford Publications.
25.
De WinterJ. C.DodouD. (2016). Common factor analysis versus principal component analysis: A comparison of loadings by means of simulations. Communications in Statistics-Simulation and Computation, 45(1), 299–321. https://doi.org/10.1080/03610918.2013.862274
26.
FalkC. F.CaiL. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328–347. https://doi.org/10.1037/met0000059
HendricksonA. E.WhiteP. O. (1964). Promax: A quick method for rotation to oblique simple structure. British Journal of Statistical Psychology, 17(1), 65–70. https://doi.org/10.1111/j.2044-8317.1964.tb00244.x
29.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
30.
KaiserH. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3), 187–200. https://doi.org/10.1007/BF02289233
31.
KimK. H. (2008). Meta-analyses of the relationship of creative achievement to both IQ and divergent thinking test scores. The Journal of Creative Behavior, 42(2), 106–130. https://doi.org/10.1016/j.cpr.2019.03.001
32.
KirbyJ. N.DayJ.SagarV. (2019). The “flow” of compassion: A meta-analysis of the fears of compassion scales and psychological functioning. Clinical Psychology Review, 70, 26–39. https://doi.org/10.1016/j.cpr.2019.03.001
33.
KrachtJ. D. (2022). Make some noise: Methods for generating data from imperfect factor models [Unpublished doctoral dissertation]. University of Minnesota.
34.
KrachtJ. D.WallerN. G. (2022). Assessing dimensionality in non-positive definite tetrachoric correlation matrices: Does matrix smoothing help?Multivariate Behavioral Research, 57(2–3), 385–407. https://doi.org/10.1080/00273171.2020.1859350
35.
LaiK.GreenS. B. (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree. Multivariate Behavioral Research, 51(2–3), 220–239. https://doi.org/10.1080/00273171.2015.1134306
36.
LiuT.WangC.XuG. (2022). Estimating three-and four-parameter mirt models with importance-weighted sampling enhanced variational auto-encoder. Frontiers in Psychology, 13, Article 935419. https://doi.org/10.3389/fpsyg.2022.935419
37.
MacCallumR. C.TuckerL. R. (1991). Representing sources of error in the common-factor model: Implications for theory and practice. Psychological Bulletin, 109(3), 502–511. https://doi.org/10.1037/0033-2909.109.3.502
38.
MarshH. W.HauK.-T.GraysonD. (2005). Goodness of fit in structural equation models. In Maydeu-OlivaresA.McArdleJ. J. (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald (pp. 275–340). Lawrence Erlbaum Associates.
39.
Maydeu-OlivaresA. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research and Perspectives, 11(3), 71–101. https://doi.org/10.1080/15366367.2013.831680
40.
Maydeu-OlivaresA.JoeH. (2008). An overview of limited information goodness-of-fit testing in multidimensional contingency tables. In ShigemasuK.OkadaA.ImaizumiT.HoshinoT. (Eds.), New trends in psychometrics (pp. 253–262). Universal Academy Press.
41.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075
42.
McDonaldR. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
43.
MeyerG. J.FinnS. E.EydeL. D.KayG. G.MorelandK. L.DiesR. R.. . .ReedG. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56(2), 128–165. https://doi.org/10.1037/0003-066X.56.2.128
44.
MonroeS.CaiL. (2015). Evaluating structural equation models for categorical outcomes: A new test statistic and a practical challenge of interpretation. Multivariate Behavioral Research, 50(6), 569–583. https://doi.org/10.1080/00273171.2015.1032398
45.
MulaikS. A. (2010). Foundations of factor analysis (2nd ed.). Chapman and Hall (CRC Press).
46.
NguyenH. V.WallerN. G. (2023). Local minima and factor rotations in exploratory factor analysis. Psychological Methods, 28(5), 1122–1141. https://doi.org/10.1037/met0000467
47.
R Core Team. (2022). R: A language and environment for statistical computing [Computer software manual]. https://www.R-project.org/
RipleyB.VenablesB.BatesD. M.HornikK.GebhardtA.FirthD.RipleyM. B. (2023). Package ’MASS’ [Computer software manual]. http://www.stats.ox.ac.uk/pub/MASS4/
50.
RozeboomW. W. (1992). The glory of suboptimal factor rotation: Why local minima in analytic optimization of simple structure are more blessing than curse. Multivariate Behavioral Research, 27, 585–599. https://doi.org/10.1207/s15327906mbr2704_5
51.
SavaleiV. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932. https://doi.org/10.1177/0013164412452564
52.
SavaleiV. (2021). Improving fit indices in structural equation modeling with categorical data. Multivariate Behavioral Research, 56(3), 390–407. https://doi.org/10.1080/00273171.2020.1717922
53.
SteigerJ. H. (1994). Factor analysis in the 1980’s and the 1990’s: Some old debates and some new developments. In BorgI.MohlerP. P. (Eds.), Trends and perspectives in empirical social research (pp. 201–223). De Gruyter. https://doi.org/10.1515/9783110887617.201
54.
SteigerJ. H.LindJ. C. (1980, May). Statistically based tests for the number of common factors [Paper presentation]. The Annual Meeting of the Psychometric Society, Iowa City, IA, United States.
55.
TakaneY.De LeeuwJ. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52(3), 393–408. https://doi.org/10.1007/BF02294363
56.
ThurstoneL. L. (1947). Multiple-factor analysis: A development and expansion of the vectors of mind. University of Chicago Press.
57.
TuckerL. R.KoopmanR. F.LinnR. L. (1969). Evaluation of factor analytic research procedures by means of simulated correlation matrices. Psychometrika, 34(4), 421–459. https://doi.org/10.1007/BF02290601
58.
VelicerW. F.FavaJ. L. (1998). Affects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231–251. https://doi.org/10.1037/1082-989X.3.2.231
WarmT. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427–450. https://doi.org/10.1007/BF02294627
62.
WoodR.WilsonD.GibbonsR.SchillingS.MurakiE.BockR. (2003). TESTFACT 4 for windows: Test scoring, item statistics, and full-information item factor analysis [Computer software manual]. Scientific Software International.
63.
WuH.BrowneM. W. (2015). Quantifying adventitious error in a covariance structure as a random effect. Psychometrika, 80, 571–600. https://doi.org/10.1007/s11336-015-9451-3
64.
XiménezC.Maydeu-OlivaresA.ShiD.RevueltaJ. (2022). Assessing cutoff values of SEM fit indices: Advantages of the unbiased srmr index and its cutoff criterion based on communality. Structural Equation Modeling: A Multidisciplinary Journal, 29(3), 368–380. https://doi.org/10.1080/10705511.2021.1992596
65.
YangJ. S.CaiL. (2014). Dealing with measurement error in estimating a cross-level interaction: Nonlinear multilevel latent variable modeling approach with a metropolis-hastings robbins-monro algorithm. KAERA Research Forum, 1(1), 55–71.
66.
YatesA. (1987). Multivariate exploratory data analysis: A perspective on exploratory factor analysis. State University of New York Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
