Multilevel structural equation modeling (MSEM) allows researchers to model latent factor structures at multiple levels simultaneously by decomposing within- and between-group variation. Yet the extent to which the sampling ratio (i.e., proportion of cases sampled from each group) influences the results of MSEM models remains unknown. This article explores how variation in the sampling ratio in MSEM affects the measurement of Level 2 (L2) latent constructs. Specifically, we investigated whether the sampling ratio is related to bias and variability in aggregated L2 construct measurement and estimation in the context of doubly latent MSEM models utilizing a two-step Monte Carlo simulation study. Findings suggest that while lower sampling ratios were related to increased bias, standard errors, and root mean square error, the overall size of these errors was negligible, making the doubly latent model an appealing choice for researchers. An applied example using empirical survey data is further provided to illustrate the application and interpretation of the model. We conclude by considering the implications of various sampling ratios on the design of MSEM studies, with a particular focus on educational research.
AlbersC.LakensD. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental and Social Psychology, 74, 187-195. https://doi.org/10.1016/j.jesp.2017.09.004
AsparouhovT.MuthénB. O. (2007b). Computationally efficient estimation of multilevel high-dimensional latent variable models. Paper presented at the Joint Statistical Meeting, Salt Lake City, UT. https://www.statmodel.com/papers.shtml
4.
BellB. A.FerronJ. M.KromreyJ. D. (2008). Cluster size in multilevel models: The impact of sparse data structures on point and interval estimates in two-level models. Proceedings of the Joint Statistical Meetings, Survey Research Section (pp. 1122-1129). American Statistical Association.
5.
BellB. A.FerronJ. M.KromreyJ. D. (2009). The effect of sparse data structures and model misspecification on point and interval estimates in multilevel models. Annual meeting of the American Educational Research Association, San Diego, CA.
BlieseP. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In KleinK. J.KozlowskiS. W. (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). Jossey-Bass.
8.
BlieseP. D.ChanD.PloyhartR. E. (2007). Multilevel methods: Future directions in measurement, longitudinal analyses, and nonnormal outcomes. Organizational Research Methods, 10(4), 551-563. https://doi.org/10.1177/1094428107301102
9.
BloomH. S.Richburg-HayesL.BlackA. R. (2007). Using covariates to improve precision: Empirical guidance for studies that randomize schools to measure the impacts of educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30-59. https://doi.org/10.3102/0162373707299550
10.
BollenK. A. (1989). Structural equations with latent variables. Wiley.
11.
ClarkeP.WheatonB. (2007). Addressing data sparseness in contextual population research using cluster analysis to create synthetic neighborhoods. Sociological Methods & Research, 35(3), 311-351. https://doi.org/10.1177/0049124106292362
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Academic Press.
13.
CronbachL. J. (1976). Research on classrooms and schools: Formulation of questions, design, and analysis. Stanford University Evaluation Consortium.
14.
CroonM. A.van VeldhovenM. J. P. M. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12(1), 45-57. https://doi.org/10.1037/1082-989X.12.1.45
15.
DepaoliS.CliftonJ. P. (2015). A Bayesian approach to multilevel structural equation modeling with continuous and dichotomous outcomes. Structural Equation Modeling, 22(3), 327-351. https://doi.org/10.1080/10705511.2014.937849
16.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72-91. https://doi.org/10.1037/a0032138
17.
GelmanA. (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis, 1(3), 515-534. https://doi.org/10.1214/06-BA117A
18.
GelmanA.CarlinJ. B.SternH. S.RubinD. B. (2004). Bayesian data analysis (2nd ed.). Chapman & Hall.
19.
HallquistM. N.WileyJ. F. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling, 25(4), 621-638. https://doi.org/10.1080/10705511.2017.1402334
20.
HedgesL.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60-87. https://doi.org/10.3102/0162373707299706
21.
HooglandJ. J.BoomsmaA. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Research Methods, 26(3), 329-367. https://doi.org/10.1177/0049124198026003003
22.
HoxJ. J.MaasC. J. M. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8(2), 157-174. https://doi.org/10.1207/S15328007SEM0802_1
23.
HsuH.LinJ.KwokO.AcostaS.WillsonV. (2016). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo study. Educational and Psychological Measurement, 77(1), 5-31. https://doi.org/10.1177/0013164416642823
JiaY.KonoldT. (2019). Moving to the next level: Doubly latent multilevel mediation models with a school climate illustration. Journal of Experimental Education, 89(2), 422-440. https://doi.org/10.1080/00220973.2019.1675136
26.
KimE. S.DedrickR. F.ChunhuaC.FerronJ. (2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881-898. https://doi.org/10.1080/00273171.2016.1228042
27.
KimE. S.KowkO.YoonMyeongsun. (2012). Testing factorial invariance in multilevel data: A Monte Carlo study. Structural Equation Modeling, 19(2), 250-267. https://doi.org/10.1080/10705511.2012.659623
28.
KlineR. B. (2011). Principles and practice of structural equation modeling (3rd ed.). Guilford Press.
29.
KonoldT. R. (2018). A multilevel MTMM approach to estimating the influences of contextual factors on trait and informant-based method effects in assessment of school climate. Journal of Psychoeducational Assessment, 36(5), 464-476. https://doi.org/10.1177/0734282916683286
30.
LarsonK. E.PasE. T.BottianiJ. H.KushJ. M.BradshawC. P. (2020). A multidimensional and multilevel examination of student engagement and secondary schools teachers’ use of classroom management practices. Journal of Positive Behavior Interventions. Advance online publication. https://doi.org/10.1177/1098300720929352
31.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
32.
LüdtkeO.MarshH. W.RobitzschA.Trautwein. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy-bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444-467. https://doi.org/10.1037/a0024376
33.
LüdtkeO.MarshH. W.RobitzschA.TrautweinU.AsparouhovT.MuthénB. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203-229. https://doi.org/10.1037/a0012869
MarshH. W.LüdtkeO.NagengastB.TrautweinU.MorinA. J. S.AbduljabbarA. S.KöllerO. (2012) Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106-124. https://doi.org/10.1080/00461520.2012.670488
36.
MarshH. W.LüdtkeO.RobitzschA.TrautweinU.AsparouhovT.MuthénB.NagengastB. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764-802. https://doi.org/10.1080/00273170903333665
MehtaP. D.NealeM. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10(3), 259-284. https://doi.org/10.1037/1082-989X.10.3.259
39.
MorinA. J. S.MarshH. W.NagengastB.ScalasL. F. (2014). Doubly latent multilevel analyses of classroom climate: An illustration. Journal of Experimental Education, 82(2), 143-167. https://doi.org/10.1080/00220973.2013.769412
40.
MurrayD. M.ShortB. (1995). Intra-class correlation among measures related to alcohol use by young adults: Estimates, correlates, and applications in intervention studies. Journal of Studies on Alcohol, 56(6), 681-692. https://doi.org/10.15288/jsa.1995.56.681
41.
MuthénB. O. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115-132. https://doi.org/10.1007/BF02294210
42.
MuthénB. O. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54(4), 557-585. https://doi.org/10.1007/BF02296397
43.
MuthénB. O. (1990). Mean and covariance structure analysis of hierarchical data. Paper presented at the Psychometric Society meeting in Princeton, NJ, June1990. UCLA Statistics Series 62. http://www.statmodel.com/bmuthen/articles/Article_032.pdf
MuthénB.AsparouhovT. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In HoxJ.RobertsJ. K. (Eds.), Handbook of advanced multilevel analysis (pp. 15-40). Taylor & Francis.
47.
MuthénL. K.MuthénB. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599-620. https://doi.org/10.1207/S15328007SEM0904_8
NagengastB.MarshH. W. (2011). The negative effect of school-average ability on science self-concept in the UK, the UK countries and around the world: The Big-Fish-Little-Pond-Effect for PISA 2006. Educational Psychology, 31(5), 629-656. https://doi.org/10.1080/01443410.2011.586416
50.
PreacherK. J.ZyphurM. J.ZhangZ. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209-233. https://doi.org/10.1037/a0020141
51.
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org
52.
Rabe-HeskethS.SkrondalA. (2008). Multilevel and longitudinal modeling using Stata (2nd ed.). StataCorp LP.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
55.
RaudenbushS. W.BrykA. S.CheongY. F.CongdonR. T.du ToitM. (2019). HLM 8: Hierarchical linear and nonlinear modeling. Scientific Software International, Inc.
56.
RaudenbushS. W.LiuX. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199-213. https://doi.org/10.1037/1082-989X.5.2.199
57.
RosseelY. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02
58.
ShinY.RaudenbushS. W. (2010). A latent cluster-mean approach to the contextual effects model with missing data. Journal of Educational and Behavioral Statistics, 35(1), 26–53. https://doi.org/10.3102/1076998609345252
59.
SnijdersT. A. B.BoskerR. J. (1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18(3), 237-259. https://doi.org/10.2307/1165134
60.
SnijdersT. A. B.BoskerR. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Sage.
61.
SpybrookJ.BloomH.CogdonR.HillC.MartinezA.RaudenbushS. (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software. http://hlmsoft.net/od/od-manual-20111016-v300.pdf
62.
StapletonL. M.YangJ. S.HancockG. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481-520. https://doi.org/10.3102/1076998616646200
SunS.KonoldT. R.FanX. (2011). Effects of latent variable nonnormality and model misspecification on testing structural equation modeling interactions. Journal of Experimental Education, 79(3), 231-256. https://doi.org/10.1080/00220973.2010.481683
65.
TourangeauK.NordC.LêT.SorongonA. G.NajarianM. (2009). Early childhood longitudinal study, kindergarten class of 1998-99 (ECLS-K): Combined user’s manual for the ECLS-K eighth-grade and K-8 full sample data files and electronic codebooks. National Center for Education Statistics.
66.
WangM.-T.DegolJ. L. (2016). School climate: A review of the construct, measurement, and impact on student outcomes. Educational Psychology Review, 28(2), 315-352. https://doi.org/10.1007/s10648-015-9319-1
67.
YuanK. H.BentlerP. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. In SobelM. E.BeckerM. P. (Eds.), Sociological methodology (pp. 165-200). Basil Blackwell.
68.
YuanK.-H.ChanW. (2008). Structural equation modeling with near singular covariance matrices. Computational Statistics and Data Analysis, 52, 4842-4858. https://doi.org/10.1016/j.csda.2008.03.030
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
