This study investigates the performance of robust maximum likelihood (ML) estimators when fitting and evaluating small sample latent growth models with non-normal missing data. Results showed that the robust ML methods could be used to account for non-normality even when the sample size is very small (e.g., N < 100). Among the robust ML estimators, “MLR” was the optimal choice, as it was found to be robust to both non-normality and missing data while also yielding more accurate standard error estimates and growth parameter coverage. However, the choice “MLMV” produced the most accurate p values for the χ2 test statistic under conditions studied. Regarding the goodness of fit indices, as sample size decreased, all three fit indices studied (i.e., comparative fit index, root mean square error of approximation, and standardized root mean square residual) exhibited worse fit. When the sample size was very small (e.g., N < 60), the fit indices would imply that a proposed model fit poorly, when this might not be actually the case in the population.
AsparouhovT.MuthénB. (2005). Multivariate statistical modeling with survey data. In Proceedings of the Federal Committee on Statistical Methodology (FCSM) Research Conference.
2.
AsparouhovT.MuthénB. (2010). Simple second order chi-square correction scaled chi-square statistics (Technical appendix). Los Angeles, CA.
3.
BandalosD. L.LeiteW. (2013). The role of simulation in structural equation modeling. Structural equation modeling: A second course (2nd ed., pp. 625–666). Information Age.
4.
BauerD. J.CurranP. J. (2003). Distributional assumptions of growth mixture models: Implications for overextraction of latent trajectory classes. Psychological Methods, 8(3), 338–363.
5.
BentlerP. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.
6.
BollenK. A. (1989). Structural equations with latent variables. Wiley.
7.
BollenK. A.CurranP. J. (2006). Latent curve models: A structural equation perspective (Vol. 467). Wiley-Interscience.
8.
BoomsmaA. (1982). The robustness of LISREL against small sample sizes in factor analysis models. In JöreskogK. G.WoldH. (Eds.), Systems under indirect observation: Causality, structure, prediction, (Part 1, pp. 149–173). Amsterdam: North-Holland.
9.
BradleyJ. V. (1978). Robustness?British Journal of Mathematical and Statistical Psychology, 31(2), 144–152.
10.
Brosseau-LiardP. E.SavaleiV. (2014). Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49(5), 460–470.
11.
Brosseau-LiardP. E.SavaleiV.LiL. (2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47(6), 904–930.
12.
ChouC. P.BentlerP. M.PentzM. A. (1998). Comparisons of two statistical approaches to study growth curves: The multilevel model and the latent curve analysis. Structural Equation Modeling: A Multidisciplinary Journal, 5(3), 247–266.
13.
CurranP. J.WestS. G.FinchJ. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16.
14.
EndersC. K. (2010). Applied missing data analysis. Guilford Press.
15.
FergusonC. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532–538.
16.
FerrerE.BalluerkaN.WidamanK. F. (2008). Factorial invariance and the specification of second-order latent growth models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4(1), 22.
17.
FinneyS.DiStefanoC. (2013). Nonnormal and categorical data in structural equation modeling. In HancockG. R.MuellerR. O. (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Information Age.
18.
FoldnesN.OlssonU. H. (2016). A simple simulation technique for nonnormal data with prespecified skewness, kurtosis, and covariance matrix. Multivariate Behavioral Research, 51(2-3), 207–219.
19.
GaoC.ShiD.Maydeu-OlivaresA. (2020). Estimating the maximum likelihood root mean square error of approximation (RMSEA) with non-normal data: A Monte-Carlo study. Structural Equation Modeling: A Multidisciplinary Journal, 27(2), 192–201.
HattonD. D.SiderisJ.SkinnerM.MankowskiJ.BaileyD. B.JrRobertsJ.MirrettP. (2006). Autistic behavior in children with fragile X syndrome: Prevalence, stability, and the impact of FMRP. American Journal of Medical Genetics Part A, 140(17), 1804–1813.
22.
HedekerD.GibbonsR. D. (2006). Longitudinal data analysis. Wiley.
23.
HooglandJ. J.BoomsmaA. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26(3), 329–367.
24.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
25.
JöreskogK. G.SörbomD. (1988). LISREL 7: A guide to the program and applications (2nd ed.). International Education Services.
26.
Maydeu-OlivaresA. (2017). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 383–394.
27.
McNeishD. (2017). Brief research report: Growth models with small samples and missing data. The Journal of Experimental Education, 86(4), 1–12.
28.
McNeishD.HarringJ. R. (2017a). Correcting model fit criteria for small sample latent growth models with incomplete data. Educational and Psychological Measurement, 77(6), 990–1018.
29.
McNeishD.MattaT. (2018). Differentiating between mixed-effects and latent-curve approaches to growth modeling. Behavior Research Methods, 50(4), 1398–1414.
30.
McNeishD.StapletonL. M. (2016a). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51(4), 495–518.
31.
McNeishD. M.HarringJ. R. (2017b). Clustered data with small sample sizes: Comparing the performance of model-based and design-based approaches. Communications in Statistics-Simulation and Computation, 46(2), 855–869.
32.
McNeishD. M.StapletonL. M. (2016b). The effect of small sample size on two-level model estimates: A review and illustration. Educational Psychology Review, 28(2), 295–314.
33.
MeredithW.TisakJ. (1984, June). “Tuckerizing” curves. Paper presented at the meeting of the psychometric society, Santa Barbara, CA.
MicceriT. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105(1), 156–166.
36.
MoshagenM. (2012). The model size effect in SEM: Inflated goodness-of-fit statistics are due to the size of the covariance matrix. Structural Equation Modeling: A Multidisciplinary Journal, 19(1), 86–98.
MuthénL. K.MuthénB. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599–620.
39.
NunnallyJ. C.BernsteinI. H. (1967). Psychometric theory (Vol. 226). McGraw-Hill.
40.
Olvera AstiviaO. L.ZumboB. D. (2015). A cautionary note on the use of the Vale and Maurelli method to generate multivariate, nonnormal data for simulation purposes. Educational and Psychological Measurement, 75(4), 541–567.
41.
PavlovG.ShiD.Maydeu-OlivaresA. (2020). Chi-square difference tests for comparing nested models: An evaluation with non-normal data. Structural Equation Modeling: A Multidisciplinary Journal, 27(6), 908–917.
42.
PornprasertmanitS.MillerP.SchoemannA. M. (2012). R packagesimsem: SIMulated structural equation modeling. Available from the Comprehensive R Archive Network: http://cranr-project.org.
43.
R Development Core Team. (2015). R: A Language and environment for statistical computing. Author.
44.
RossC. A. (1997). Dissociative identity disorder: Diagnosis, clinical features, and treatment of multiple personality. John Wiley & Sons.
45.
RubinD. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592.
46.
SatorraA.BentlerP. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In Von EyeA.CloggC. C. (Eds.), Latent variable analysis. Applications for developmental research (pp. 399–419). SAGE.
47.
SavaleiV. (2010). Expected versus observed information in SEM with incomplete normal and nonnormal data. Psychological Methods, 15, 352–367.
48.
SavaleiV. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 149–160.
49.
SavaleiV. (2018). On the computation of the RMSEA and CFI from the mean-and-variance corrected test statistic with nonnormal data in SEM. Multivariate Behavioral Research, 53(3), 419–429.
50.
ShiD.LeeT.FairchildA. J.Maydeu-OlivaresA. (2019). Fitting ordinal factor analysis models with missing data: A comparison between pairwise deletion and multiple imputation. Educational and Psychological Measurement, 80(1), 41–66.
51.
ShiD.LeeT.Maydeu-OlivaresA. (2019). Understanding the model size effect on SEM fit indices. Educational and Psychological Measurement, 79(2), 310–334.
52.
ShiD.LeeT.TerryR. A. (2018). Revisiting the model size effect in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 25(1), 21–40.
53.
SteigerJ.LindJ. C. (1980, May). Statistically based tests for the number of common factors. Paper presented at the annual meeting of the annual spring meeting of the psychometric society, Iowa City.
WuW.WestS. G.TaylorA. B. (2009). Evaluating model fit for growth curve models: Integration of fit indices from SEM and MLM frameworks. Psychological Methods, 14(3), 183.
56.
ZhengX. (2017). Latent growth curve analysis with item response data: Model specification, estimation, and panel attrition [Unpublished doctoral dissertation]. University of Maryland.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
