Multilevel modeling (MLM) is a statistical technique for analyzing clustered data. Despite its long history, the technique and accompanying computer programs are rapidly evolving. Given the complexity of multilevel models, it is crucial for researchers to provide complete and transparent descriptions of the data, statistical analyses, and results. Ten years have passed since the guidelines for reporting multilevel studies were initially published. This study reviewed new advancements in MLM and revisited the reporting practice in MLM in the past decade. A total of 301 articles from 19 journals representing different subdisciplines in education and psychology were included in the systematic review. The results showed improvement in some areas of the reporting practices, such as the number of models tested, centering of predictors, missing data treatment, software, and estimates of variance components. However, poor practices persist in terms of model specification, description of a missing mechanism, power analysis, assumption checking, model comparisons, and effect sizes. Updates on the guidelines for reporting multilevel studies and recommendations for future methodological research in MLM are presented.
AgrestiA.CaffoB.Ohman-StricklandP. (2004). Examples in which misspecification of a random effects distribution reduces efficiency, and possible remedies. Computational Statistics & Data Analysis, 47(3), 639–653. https://doi.org/10.1016/j.csda.2003.12.009
AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191
4.
ArendM. G.SchäferT. (2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24(1), 1–19. https://doi.org/10.1037/met0000195
5.
BaayenR. H.DavidsonD. J.BatesD. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/j.jml.2007.12.005
BaldwinS. A.FellinghamG. W. (2013). Bayesian methods for the analysis of small sample multilevel data with a complex variance structure. Psychological Methods, 18(2), 151–164. https://doi.org/10.1037/a0030642
8.
BarrD. J. (2013). Random effects structure for testing interactions in linear mixed-effects models. Frontiers in Psychology, 4, Article 328. https://doi.org/10.3389/fpsyg.2013.00328
9.
BarrD. J.LevyR.ScheepersC.TilyH. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001
10.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
11.
BatesM. D.CastellanoK. E.Rabe-HeskethS.SkrondalA. (2014). Handling correlations between covariates and random slopes in multilevel models. Journal of Educational and Behavioral Statistics, 39(6), 524–549. https://doi.org/10.3102/1076998614559420
12.
BauerD. J.CaiL. (2009). Consequences of unmodeled nonlinear effects in multilevel models. Journal of Educational and Behavioral Statistics, 34(1), 97–114. https://doi.org/10.3102/1076998607310504
13.
BauerD. J.SterbaS. K. (2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16(4), 373–390. https://doi.org/10.1037/a0025813
14.
BellB. A.MorganG. B.SchoenebergerJ. A.KromreyJ. D.FerronJ. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10(1), 1–11. https://doi.org/10.1027/1614-2241/a000062
15.
BerkhofJ.SnijdersT. A. (2001). Variance component testing in multilevel models. Journal of Educational and Behavioral Statistics, 26(2), 133–152. https://doi.org/10.3102/10769986026002133
16.
BrauerM.CurtinJ. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411. https://doi.org/10.1037/met0000159
17.
BreslowN. E.ClaytonD. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88(421), 9–25. https://doi.org/10.1080/01621459.1993.10594284
18.
BreslowN. E.LinX. (1995). Bias correction in generalized linear mixed models with a single component of dispersion. Biometrika, 82(1), 81–91. https://doi.org/10.1093/biomet/82.1.81
19.
BrowneW. J. (2019). MCMC estimation in MLwiN Version 3.03. Centre for Multilevel Modelling, University of Bristol.
20.
BrowneW. J.DraperD. (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1(3), 473–514. https://doi.org/10.1214/06-BA117
21.
BrykA. S.RaudenbushS. W. (1992). Hierarchical linear models: Applications and data analysis methods. Sage.
22.
BürknerP. C. (2017). Advanced Bayesian multilevel modeling with the R package brms. The R Journal, 10/1(July), 395–411. https://doi.org/10.32614/RJ-2018-017
23.
CapanuM.GὄnenM.BeggC. B. (2013). An assessment of estimation methods for generalized linear mixed models with binary outcomes. Statistics in Medicine, 32(26), 4550–4566. https://doi.org/10.1002/sim.5866
24.
CarrollR.LawsonA. B.FaesC.KirbyR. S.AregayM.WatjouK. (2015). Comparing INLA and OpenBUGS for hierarchical Poisson modeling in disease mapping. Spatial and Spatio-Temporal Epidemiology, 14–15(July–October), 45–54. https://doi.org/10.1016/j.sste.2015.08.001
25.
ChoS. J.Rabe-HeskethS. (2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55(1), 12–25. https://doi.org/10.1016/j.csda.2010.04.015
26.
ChungH.BeretvasS. N. (2012). The impact of ignoring multiple membership data structures in multilevel models. British Journal of Mathematical and Statistical Psychology, 65(2), 185–200. https://doi.org/10.1111/j.2044-8317.2011.02023.x
27.
CrainiceanuC. M.RuppertD. (2004). Likelihood ratio tests in linear mixed models with one variance component. Journal of the Royal Statistical Society: Series B: Statistical Methodology, 66(1), 165–185. https://doi.org/10.1111/j.1467-9868.2004.00438.x
28.
DedrickR. F.FerronJ. M.HessM. R.HogartyK. Y.KromreyJ. D.LangT. R.NilesJ. D.LeeR. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. https://doi.org/10.3102/0034654308325581
29.
DepaoliS.Van de SchootR. (2017). Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist. Psychological Methods, 22(2), 240–261. https://doi.org/10.1037/met0000065
30.
DongN.KelceyB.SpybrookJ.MaynardR. A. (2017). PowerUp!-Moderator: A tool for calculating statistical power and minimum detectable effect size of the moderator effects in cluster randomized trials (Version 1.08) [Computer software]. Casual Evaluation. http://www.causalevaluation.org/
31.
DongN.MaynardR. A. (2013). PowerUp! A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24–67. https://doi.org/10.1080/19345747.2012.673143
FangX.LiJ.WongW. K.FuB. (2016). Detecting the violation of variance homogeneity in mixed models. Statistical Methods in Medical Research, 25(6), 2506–2520. https://doi.org/10.1177/0962280214526194
34.
FerronJ.DaileyR.YiQ. (2002). Effects of misspecifying the first-level error structure in two-level models of change. Multivariate Behavioral Research, 37(3), 379–403. https://doi.org/10.1207/S15327906MBR3703_4
35.
FinchH. (2017). Multilevel modeling in the presence of outliers: A comparison of robust estimation methods. Psicologica: International Journal of Methodology and Experimental Psychology, 38(1), 57–92.
36.
FournierD. A.SkaugH. J.AnchetaJ.IanelliJ.MagnussonA.MaunderM. N.NielsenA.SibertJ. (2012). AD Model Builder: Using automatic differentiation for statistical inference of highly parameterized complex nonlinear models. Optimization Methods & Software, 27(2), 233–249. https://doi.org/10.1080/10556788.2011.597854
37.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
38.
GelmanA. (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis, 1(3), 515–534. https://doi.org/10.1214/06-BA117A
39.
GoldsteinH. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73(1), 43–56. https://doi.org/10.1093/biomet/73.1.43
40.
GreenP.MacLeodC. J. (2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7(4), 493–498. https://doi.org/10.1111/2041-210X.12504
41.
HadfieldJ. D. (2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software, 33(2), 1–22. https://doi.org/10.18637/jss.v033.i02
42.
HedekerD. (2008). Multilevel models for ordinal and nominal variables. In de LeeuwJ.MeijerE. (Eds.), Handbook of multilevel analysis (pp. 237–274). Springer. https://doi.org/10.1007/978-0-387-73186-5_6
43.
HedgesL. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. https://doi.org/10.3102/1076998606298043
44.
HedgesL. V. (2011). Effect sizes in three-level cluster randomized experiments. Journal of Educational and Behavioral Statistics, 36(3), 346–380. https://doi.org/10.3102/1076998610376617
45.
HeisigJ. P.SchaefferM. (2019). Why you should always include a random slope for the lower-level variable involved in a cross-level interaction. European Sociological Review, 35(2), 258–279. https://doi.org/10.1093/esr/jcy053
46.
HalekohU.HøjsgaardS. (2014). A Kenward-Roger approximation and parametric bootstrap methods for tests in linear mixed models: The R package pbkrtest. Journal of Statistical Software, 59(9), 1–30. https://doi.org/10.18637/jss.v059.i09
47.
HoxJ. J. (1998). Multilevel modeling: When and why. In BalderjahnI.MatharR.SchaderM. (Eds.), Classification, data analysis, and data highways (pp. 147–154). Springer. https://doi.org/10.1007/978-3-642-72087-1_17
48.
HoxJ. J.MoerbeekM.Van de SchootR. (2010). Multilevel analysis: Techniques and applications (2nd ed.). Routledge.
49.
JoeH. (2008). Accuracy of Laplace approximation for discrete response mixed models. Computational Statistics & Data Analysis, 52(12), 5066–5074. https://doi.org/10.1016/j.csda.2008.05.002
50.
KassR. E.NatarajanR. (2006). A default conjugate prior for variance components in generalized linear mixed models (comment on article by Browne and Draper). Bayesian Analysis, 1(3), 535–542. https://doi.org/10.1214/06-BA117B
51.
KelceyB.DongN.SpybrookJ.ShenZ. (2017). Experimental power for indirect effects in group-randomized studies with group-level mediators. Multivariate Behavioral Research, 52(6), 699–719. https://doi.org/10.1080/00273171.2017.1356212
KenwardM. G.RogerJ. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3), 983–997. https://doi.org/10.2307/2533558
54.
KenwardM. G.RogerJ. H. (2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53(7), 2583–2595. https://doi.org/10.1016/j.csda.2008.12.013
55.
KeselmanH.AlginaJ.KowalchunkR.WolfingerR. (1998). A comparison of two approaches for selecting covariance structures in the analysis of repeated measurements. Communications in Statistics: Simulation and Computation, 27(3), 591–604. https://doi.org/10.1080/03610919808813497
56.
KimE. S.DedrickR. F.CaoC.FerronJ. M. (2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881–898. https://doi.org/10.1080/00273171.2016.1228042
KlokeJ. D.McKeanJ. W.RashidM. M. (2009). Rank-based estimation and associated inferences for linear models with cluster correlated errors. Journal of the American Statistical Association, 104(485), 384–390. https://doi.org/10.1198/jasa.2009.0116
60.
KreftI. G. G. (1996). Are multilevel techniques necessary? An overview, including simulation studies. California State University, Los Angeles.
61.
KuznetsovaA.BrockhoffP. B.ChristensenR. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13
62.
KwokO.WestS.GreenS. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42(3), 557–592. https://doi.org/10.1080/00273170701540537
63.
LaiM. H. C. (2018, December3). bootmlm: Bootstrap resampling for multilevel models (R Package Version 0.0.1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.1879127
64.
LaiM. H. C. (2020a). Bootstrap confidence intervals for multilevel standardized effect size. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1746902
65.
LaiM. H. C. (2020b). Composite reliability of multilevel data: It’s about observed scores and construct meanings. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000287
66.
LaiM. H. C.KwokO. M. (2014). Standardized mean differences in two-level cross-classified random effects models. Journal of Educational and Behavioral Statistics, 39(4), 282–302. https://doi.org/10.3102/1076998614532950
67.
LaiM. H. C.KwokO. M. (2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229
68.
LaiM. H. C.KwokO. M. (2016). Estimating standardized effect sizes for two-and three-level partially nested data. Multivariate Behavioral Research, 51(6), 740–756. https://doi.org/10.1080/00273171.2016.1231606
69.
LandauS.StahlD. (2013). Sample size and power calculations for medical studies by simulation when closed form expressions are not available. Statistical Methods in Medical Research, 22(3), 324–345. https://doi.org/10.1177/0962280212439578
70.
LaneS. P.HennesE. P. (2018). Power struggles: Estimating sample size for multilevel relationships research. Journal of Social and Personal Relationships, 35(1), 7–31. https://doi.org/10.1177/0265407517710342
71.
LangeK. L.LittleR. J.TaylorJ. M. (1989). Robust statistical modeling using the t distribution. Journal of the American Statistical Association, 84(408), 881–896. https://doi.org/10.2307/2290063
72.
LeBeauB. (2016). Impact of serial correlation misspecification with the linear mixed model. Journal of Modern Applied Statistical Methods, 15(1), 389–416. https://doi.org/10.22237/jmasm/1462076400
73.
LeckieG.BrowneW.GoldsteinH.MerloJ.AustinP. (2019, April). Variance partitioning in multilevel models for count data [Paper presentation]. 12th International Multilevel Conference, Utrecht, Netherlands. https://arxiv.org/ftp/arxiv/papers/1911/1911.06888.pdf
74.
LeeW.GrimmK. J. (2018). Generalized linear mixed-effects modeling programs in R for binary outcomes. Structural Equation Modeling: A Multidisciplinary Journal, 25(5), 824–828. https://doi.org/10.1080/10705511.2018.1500141
75.
LindstromM. J.BatesD. M. (1988). Newton-Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83(404), 1014–1022. https://doi.org/10.1080/01621459.1988.10478693
76.
LiuH.ZhangZ.GrimmK. J. (2016). Comparison of inverse Wishart and separation-strategy priors for Bayesian estimation of covariance parameter matrix in growth curve analysis. Structural Equation Modeling: A Multidisciplinary Journal, 23(3), 354–367. https://doi.org/10.1080/10705511.2015.1057285
77.
LiuI.AgrestiA. (2005). The analysis of ordered categorical data: An overview and a survey of recent developments. Test, 14(1), 1–73. https://doi.org/10.1007/BF02595397
LuN.TangW.HeH.YuQ.Crits-ChristophP.ZhangH.TuX. (2009). On the impact of parametric assumptions and robust alternatives for longitudinal data analysis. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 51(4), 627–643. https://doi.org/10.1002/bimj.200800186
80.
LukeS. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y
81.
LunnD.SpiegelhalterD.ThomasA.BestN. (2009a). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28(25), 3049–3067. https://doi.org/10.1002/sim.3680
82.
LunnD.SpiegelhalterD.ThomasA.BestN. (2009b). Rejoinder to commentaries on “The BUGS project: Evolution, critique and future directions.”Statistics in Medicine, 28(25), 3081–3082. https://doi.org/10.1002/sim.3691
83.
LuoW.CappaertK. J.NingL. (2015). Modelling partially cross-classified multilevel data. British Journal of Mathematical and Statistical Psychology, 68(2), 342–362. https://doi.org/10.1111/bmsp.12050
84.
LuoW.KwokO. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44(2), 182–212. https://doi.org/10.1080/00273170902794214
85.
LuoW.KwokO. (2012). The consequences of ignoring individuals’ mobility in multilevel growth models: A Monte Carlo study. Journal of Educational and Behavioral Statistics, 37(1), 31–56. https://doi.org/10.3102/1076998610394366
86.
MaasC. J.HoxJ. J. (2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46(3), 427–440. https://doi.org/10.1016/j.csda.2003.08.006
87.
MarshH. W.LüdtkeO.NagengastB.TrautweinU.MorinA. J.AbduljabbarA. S.KöllerO. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488
MathieuJ. E.AguinisH.CulpepperS. A.ChenG. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951–966. https://doi.org/10.1037/a0028380
90.
MatuschekH.KlieglR.VasishthS.BaayenH.BatesD. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94(June), 305–315. https://doi.org/10.1016/j.jml.2017.01.001
McCullochC. E.NeuhausJ. M. (2011). Misspecifying the shape of a random effects distribution: Why getting it wrong may not matter. Statistical Science, 26(3), 388–402. https://doi.org/10.1214/11-STS361
93.
McDonaldR. P.HoM. H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. https://doi.org/10.1037/1082-989X.7.1.64
McNeishD. (2017). Small sample methods for multilevel modeling: A colloquial elucidation of REML and the Kenward-Roger correction. Multivariate Behavioral Research, 52(5), 661–670. https://doi.org/10.1080/00273171.2017.1344538
McNeishD.WentzelK. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52(2), 200–215. https://doi.org/10.1080/00273171.2016.1262236
99.
MoehringK.SchmidtA. (2013). “MLT”: Module providing different tools for multilevel modeling [Computer software]. Boston College Department of Economics.
100.
MoerbeekM. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5
101.
MoineddinR.MathesonF. I.GlazierR. H. (2007). A simulation study of sample size for multilevel logistic regression models. BMC Medical Research Methodology, 7, Article 34. https://doi.org/10.1186/1471-2288-7-34
102.
MurphyD.PituchK. (2009). The performance of multilevel growth curve models under an autoregressive moving average process. Journal of Experimental Education, 77(3), 255–284. https://doi.org/10.3200/JEXE.77.3.255-284
NakagawaS.SchielzethH. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012.00261.x
105.
NatarajanR.KassR. E. (2000). Reference Bayesian methods for generalized linear mixed models. Journal of American Statistician Association, 95(449), 227–237. https://doi.org/10.1080/01621459.2000.10473916
106.
PinheiroJ.BatesD. (2006). Mixed-effects models in S and S-PLUS. Springer Science + Business Media.
107.
PinheiroJ. C.ChaoE. C. (2006). Efficient Laplacian and adaptive Gaussian quadrature algorithms for multilevel generalized linear mixed models. Journal of Computational and Graphical Statistics, 15(1), 58–81. https://doi.org/10.1198/106186006X96962
108.
RasbashJ.GoldsteinH. (1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19(4), 337–350. https://doi.org/10.2307/1165397
109.
RasbashJ.SteeleF.BrowneW. J.GoldsteinH. (2019). A user’s guide to MLwiN, v3.03. Centre for Multilevel Modelling, University of Bristol.
110.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. Sage.
111.
RaudenbushS. W.BrykA. S.CheongY. F.CongdonR. (2019). HLM 8 for Windows [Computer software]. Scientific Software International.
112.
RaudenbushS. W.SpybrookJ.CongdonR.LiuX. F.MartinezA.BloomH. (2011). Optimal design software for multi-level and longitudinal research (Version 3.01) [Computer software]. William T. Grant Foundation. http://www.wtgrantfoundation.org
113.
RightsJ. D.SterbaS. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309–338. https://doi.org/10.1037/met0000184
114.
RodriguezG.GoldmanN. (1995). An assessment of estimation procedures for multilevel models with binary responses. Journal of the Royal Statistical Society, Series A: Statistics in Society, 158(1), 73–89. https://doi.org/10.2307/2983404
115.
RodriguezG.GoldmanN. (2001). Improved estimation procedures for multilevel models with binary response: A case study. Journal of the Royal Statistical Society, Series A: Statistics in Society, 164(2), 339–355. https://doi.org/10.1111/1467-985X.00206
116.
RoosM.HeldL. (2011). Sensitivity analysis in Bayesian generalized linear mixed models for binary data. Bayesian Analysis, 6(2), 259–278. https://doi.org/10.1214/11-BA609
117.
RueH.MartinoS.ChopinN. (2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(2), 319–392. https://doi.org/10.1111/j.1467-9868.2008.00700.x
118.
SAS Institute. (2017). SAS/STAT® 14.3 user’s guide.
119.
SatterthwaiteF. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. https://doi.org/10.2307/3002019
120.
ScheiplF.GrevenS.KüchenhoffH. (2008). Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models. Computational Statistics & Data Analysis, 52(7), 3283–3299. https://doi.org/10.1016/j.csda.2007.10.022
121.
ScherbaumC.PesnerE. (2018). Power analysis for multilevel research. In HumphreyS. E.LeBretonJ. M. (Eds.), The handbook of multilevel theory, measurement, and analysis (pp. 498–534). American Psychological Association. https://doi.org/10.1037/0000115-015
122.
SchielzethH.DingemanseN. J.NakagawaS.WestneatD. F.AllegueH.TeplitskyC.RéaleD.DochtermannN. A.GaramszegiL. Z.Araya-AjoyY. G. (2020). Robustness of linear mixed-effects models to violations of distributional assumptions. Methods in Ecology and Evolution, 11(9), 1141–1152. https://doi.org/10.1111/2041-210X.13434
123.
SchreiberJ. B.GriffinB. W. (2004). Review of multilevel modeling and multilevel studies in The Journal of Educational Research (1992–2002). Journal of Educational Research, 98(1), 24–34. https://doi.org/10.3200/JOER.98.1.24-34
124.
SecoG. V.GarcíaM. A.GarcíaM. P. F.RojasP. E. L. (2013). Multilevel bootstrap analysis with assumptions violated. Psicothema, 25(4), 520–528. https://doi.org/10.7334/psicothema2013.58
125.
SelfS. G.LiangK. Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398), 605–610. https://doi.org/10.1080/01621459.1987.10478472
126.
SingerJ. M.RochaF. M.NobreJ. S. (2017). Graphical tools for detecting departures from linear mixed model assumptions and some remedial measures. International Statistical Review, 85(2), 290–324. https://doi.org/10.1111/insr.12178
StaggsV. S. (2017). Comparison of naïve, Kenward–Roger, and parametric bootstrap interval approaches to small-sample inference in linear mixed models. Communications in Statistics: Simulation and Computation, 46(3), 1933–1943. https://doi.org/10.1080/03610918.2015.1019002
132.
StapletonL. M.YangJ. S.HancockG. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200
133.
StataCorp. (2019). Stata statistical software: Release 16.
134.
StramD. O.LeeJ. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50(4), 1171–1177. https://doi.org/10.2307/2533455
135.
Van der LeedenR.MeijerE.BusingF. M. (2008). Resampling multilevel models. In de LeeuwJ.MeijerE. (Eds.), Handbook of multilevel analysis (pp. 401–433). Springer. https://doi.org/10.1007/978-0-387-73186-5_11
XuR. (2003). Measuring explained variation in linear mixed effects models. Statistics in Medicine, 22(22), 3527–3541. https://doi.org/10.1002/sim.1572
138.
ZegerS. L.KarimM. R. (1991). Generalized linear-models with random effects: A Gibbs sampling approach. Journal of American Statistician Association, 86(413), 79–86. https://doi.org/10.1080/01621459.1991.10475006
139.
Zondervan-ZwijnenburgM.PeetersM.DepaoliS.Van de SchootR. (2017). Where do priors come from? Applying guidelines to construct informative priors in small sample research. Research in Human Development, 14(4), 305–320. https://doi.org/10.1080/15427609.2017.1370966
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
