Reporting Practice in Multilevel Modeling: A Revisit After 10 Years

Abstract

Multilevel modeling (MLM) is a statistical technique for analyzing clustered data. Despite its long history, the technique and accompanying computer programs are rapidly evolving. Given the complexity of multilevel models, it is crucial for researchers to provide complete and transparent descriptions of the data, statistical analyses, and results. Ten years have passed since the guidelines for reporting multilevel studies were initially published. This study reviewed new advancements in MLM and revisited the reporting practice in MLM in the past decade. A total of 301 articles from 19 journals representing different subdisciplines in education and psychology were included in the systematic review. The results showed improvement in some areas of the reporting practices, such as the number of models tested, centering of predictors, missing data treatment, software, and estimates of variance components. However, poor practices persist in terms of model specification, description of a missing mechanism, power analysis, assumption checking, model comparisons, and effect sizes. Updates on the guidelines for reporting multilevel studies and recommendations for future methodological research in MLM are presented.

Keywords

methodological development multilevel modeling reporting practices systematic review

Get full access to this article

View all access options for this article.

References

Agresti

Caffo

Ohman-Strickland

(2004). Examples in which misspecification of a random effects distribution reduces efficiency, and possible remedies. Computational Statistics & Data Analysis, 47(3), 639–653. https://doi.org/10.1016/j.csda.2003.12.009

Agresti

Natarajan

(2001). Modeling clustered ordered categorical data: A survey. International Statistical Review, 69(3), 345–371. https://doi.org/10.1111/j.1751-5823.2001.tb00463.x

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

Arend

M. G.

Schäfer

(2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24(1), 1–19. https://doi.org/10.1037/met0000195

Baayen

R. H.

Davidson

D. J.

Bates

D. M.

(2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/j.jml.2007.12.005

Baldwin

S. A.

Bauer

D. J.

Stice

Rohde

(2011). Evaluating models for partially clustered designs. Psychological Methods, 16(2), 149–165. https://doi.org/10.1037/a0023464

Baldwin

S. A.

Fellingham

G. W.

(2013). Bayesian methods for the analysis of small sample multilevel data with a complex variance structure. Psychological Methods, 18(2), 151–164. https://doi.org/10.1037/a0030642

Barr

D. J.

(2013). Random effects structure for testing interactions in linear mixed-effects models. Frontiers in Psychology, 4, Article 328. https://doi.org/10.3389/fpsyg.2013.00328

Barr

D. J.

Levy

Scheepers

Tily

H. J.

(2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

10.

Bates

Mächler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

11.

Bates

M. D.

Castellano

K. E.

Rabe-Hesketh

Skrondal

(2014). Handling correlations between covariates and random slopes in multilevel models. Journal of Educational and Behavioral Statistics, 39(6), 524–549. https://doi.org/10.3102/1076998614559420

12.

Bauer

D. J.

Cai

(2009). Consequences of unmodeled nonlinear effects in multilevel models. Journal of Educational and Behavioral Statistics, 34(1), 97–114. https://doi.org/10.3102/1076998607310504

13.

Bauer

D. J.

Sterba

S. K.

(2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16(4), 373–390. https://doi.org/10.1037/a0025813

14.

Bell

B. A.

Morgan

G. B.

Schoeneberger

J. A.

Kromrey

J. D.

Ferron

J. M.

(2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10(1), 1–11. https://doi.org/10.1027/1614-2241/a000062

15.

Berkhof

Snijders

T. A.

(2001). Variance component testing in multilevel models. Journal of Educational and Behavioral Statistics, 26(2), 133–152. https://doi.org/10.3102/10769986026002133

16.

Brauer

Curtin

J. J.

(2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411. https://doi.org/10.1037/met0000159

17.

Breslow

N. E.

Clayton

D. G.

(1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88(421), 9–25. https://doi.org/10.1080/01621459.1993.10594284

18.

Breslow

N. E.

Lin

(1995). Bias correction in generalized linear mixed models with a single component of dispersion. Biometrika, 82(1), 81–91. https://doi.org/10.1093/biomet/82.1.81

19.

Browne

W. J.

(2019). MCMC estimation in MLwiN Version 3.03. Centre for Multilevel Modelling, University of Bristol.

20.

Browne

W. J.

Draper

(2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1(3), 473–514. https://doi.org/10.1214/06-BA117

21.

Bryk

A. S.

Raudenbush

S. W.

(1992). Hierarchical linear models: Applications and data analysis methods. Sage.

22.

Bürkner

P. C.

(2017). Advanced Bayesian multilevel modeling with the R package brms. The R Journal, 10/1(July), 395–411. https://doi.org/10.32614/RJ-2018-017

23.

Capanu

Gὄnen

Begg

C. B.

(2013). An assessment of estimation methods for generalized linear mixed models with binary outcomes. Statistics in Medicine, 32(26), 4550–4566. https://doi.org/10.1002/sim.5866

24.

Carroll

Lawson

A. B.

Faes

Kirby

R. S.

Aregay

Watjou

(2015). Comparing INLA and OpenBUGS for hierarchical Poisson modeling in disease mapping. Spatial and Spatio-Temporal Epidemiology, 14–15(July–October), 45–54. https://doi.org/10.1016/j.sste.2015.08.001

25.

Cho

S. J.

Rabe-Hesketh

(2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55(1), 12–25. https://doi.org/10.1016/j.csda.2010.04.015

26.

Chung

Beretvas

S. N.

(2012). The impact of ignoring multiple membership data structures in multilevel models. British Journal of Mathematical and Statistical Psychology, 65(2), 185–200. https://doi.org/10.1111/j.2044-8317.2011.02023.x

27.

Crainiceanu

C. M.

Ruppert

(2004). Likelihood ratio tests in linear mixed models with one variance component. Journal of the Royal Statistical Society: Series B: Statistical Methodology, 66(1), 165–185. https://doi.org/10.1111/j.1467-9868.2004.00438.x

28.

Dedrick

R. F.

Ferron

J. M.

Hess

M. R.

Hogarty

K. Y.

Kromrey

J. D.

Lang

T. R.

Niles

J. D.

Lee

R. S.

(2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. https://doi.org/10.3102/0034654308325581

29.

Depaoli

Van de Schoot

(2017). Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist. Psychological Methods, 22(2), 240–261. https://doi.org/10.1037/met0000065

30.

Dong

Kelcey

Spybrook

Maynard

R. A.

(2017). PowerUp!-Moderator: A tool for calculating statistical power and minimum detectable effect size of the moderator effects in cluster randomized trials (Version 1.08) [Computer software]. Casual Evaluation. http://www.causalevaluation.org/

31.

Dong

Maynard

R. A.

(2013). PowerUp! A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24–67. https://doi.org/10.1080/19345747.2012.673143

32.

Donohue

M. C.

Gamst

A. C.

Edland

S. D.

Donohue

(2013). Package “longpower.” Biometrics, 53, 937–947.

33.

Fang

Wong

W. K.

(2016). Detecting the violation of variance homogeneity in mixed models. Statistical Methods in Medical Research, 25(6), 2506–2520. https://doi.org/10.1177/0962280214526194

34.

Ferron

Dailey

(2002). Effects of misspecifying the first-level error structure in two-level models of change. Multivariate Behavioral Research, 37(3), 379–403. https://doi.org/10.1207/S15327906MBR3703_4

35.

Finch

(2017). Multilevel modeling in the presence of outliers: A comparison of robust estimation methods. Psicologica: International Journal of Methodology and Experimental Psychology, 38(1), 57–92.

36.

Fournier

D. A.

Skaug

H. J.

Ancheta

Ianelli

Magnusson

Maunder

M. N.

Nielsen

Sibert

(2012). AD Model Builder: Using automatic differentiation for statistical inference of highly parameterized complex nonlinear models. Optimization Methods & Software, 27(2), 233–249. https://doi.org/10.1080/10556788.2011.597854

37.

Geldhof

G. J.

Preacher

K. J.

Zyphur

M. J.

(2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138

38.

Gelman

(2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis, 1(3), 515–534. https://doi.org/10.1214/06-BA117A

39.

Goldstein

(1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73(1), 43–56. https://doi.org/10.1093/biomet/73.1.43

40.

Green

MacLeod

C. J.

(2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7(4), 493–498. https://doi.org/10.1111/2041-210X.12504

41.

Hadfield

J. D.

(2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software, 33(2), 1–22. https://doi.org/10.18637/jss.v033.i02

42.

Hedeker

(2008). Multilevel models for ordinal and nominal variables. In de Leeuw

Meijer

(Eds.), Handbook of multilevel analysis (pp. 237–274). Springer. https://doi.org/10.1007/978-0-387-73186-5_6

43.

Hedges

L. V.

(2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. https://doi.org/10.3102/1076998606298043

44.

Hedges

L. V.

(2011). Effect sizes in three-level cluster randomized experiments. Journal of Educational and Behavioral Statistics, 36(3), 346–380. https://doi.org/10.3102/1076998610376617

45.

Heisig

J. P.

Schaeffer

(2019). Why you should always include a random slope for the lower-level variable involved in a cross-level interaction. European Sociological Review, 35(2), 258–279. https://doi.org/10.1093/esr/jcy053

46.

Halekoh

Højsgaard

(2014). A Kenward-Roger approximation and parametric bootstrap methods for tests in linear mixed models: The R package pbkrtest. Journal of Statistical Software, 59(9), 1–30. https://doi.org/10.18637/jss.v059.i09

47.

Hox

J. J.

(1998). Multilevel modeling: When and why. In Balderjahn

Mathar

Schader

(Eds.), Classification, data analysis, and data highways (pp. 147–154). Springer. https://doi.org/10.1007/978-3-642-72087-1_17

48.

Hox

J. J.

Moerbeek

Van de Schoot

(2010). Multilevel analysis: Techniques and applications (2nd ed.). Routledge.

49.

Joe

(2008). Accuracy of Laplace approximation for discrete response mixed models. Computational Statistics & Data Analysis, 52(12), 5066–5074. https://doi.org/10.1016/j.csda.2008.05.002

50.

Kass

R. E.

Natarajan

(2006). A default conjugate prior for variance components in generalized linear mixed models (comment on article by Browne and Draper). Bayesian Analysis, 1(3), 535–542. https://doi.org/10.1214/06-BA117B

51.

Kelcey

Dong

Spybrook

Shen

(2017). Experimental power for indirect effects in group-randomized studies with group-level mediators. Multivariate Behavioral Research, 52(6), 699–719. https://doi.org/10.1080/00273171.2017.1356212

52.

Kelley

Preacher

K. J.

(2012). On effect size. Psychological Methods, 17(2), 137–152. https://doi.org/10.1037/a0028086

53.

Kenward

M. G.

Roger

J. H.

(1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3), 983–997. https://doi.org/10.2307/2533558

54.

Kenward

M. G.

Roger

J. H.

(2009). An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis, 53(7), 2583–2595. https://doi.org/10.1016/j.csda.2008.12.013

55.

Keselman

Algina

Kowalchunk

Wolfinger

(1998). A comparison of two approaches for selecting covariance structures in the analysis of repeated measurements. Communications in Statistics: Simulation and Computation, 27(3), 591–604. https://doi.org/10.1080/03610919808813497

56.

Kim

E. S.

Dedrick

R. F.

Cao

Ferron

J. M.

(2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881–898. https://doi.org/10.1080/00273171.2016.1228042

57.

Kim

J. S.

Frees

E. W.

(2007). Multilevel modeling with correlated effects. Psychometrika, 72(4), 505–533. https://doi.org/10.1007/s11336-007-9008-1

58.

Kloke

McKean

J. W.

(2014). Nonparametric statistical methods using R. CRC Press. https://doi.org/10.1201/b17501

59.

Kloke

J. D.

McKean

J. W.

Rashid

M. M.

(2009). Rank-based estimation and associated inferences for linear models with cluster correlated errors. Journal of the American Statistical Association, 104(485), 384–390. https://doi.org/10.1198/jasa.2009.0116

60.

Kreft

I. G. G.

(1996). Are multilevel techniques necessary? An overview, including simulation studies. California State University, Los Angeles.

61.

Kuznetsova

Brockhoff

P. B.

Christensen

R. H. B.

(2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13

62.

Kwok

West

Green

(2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42(3), 557–592. https://doi.org/10.1080/00273170701540537

63.

Lai

M. H. C.

(2018, December 3). bootmlm: Bootstrap resampling for multilevel models (R Package Version 0.0.1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.1879127

64.

Lai

M. H. C.

(2020a). Bootstrap confidence intervals for multilevel standardized effect size. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1746902

65.

Lai

M. H. C.

(2020b). Composite reliability of multilevel data: It’s about observed scores and construct meanings. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000287

66.

Lai

M. H. C.

Kwok

O. M.

(2014). Standardized mean differences in two-level cross-classified random effects models. Journal of Educational and Behavioral Statistics, 39(4), 282–302. https://doi.org/10.3102/1076998614532950

67.

Lai

M. H. C.

Kwok

O. M.

(2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229

68.

Lai

M. H. C.

Kwok

O. M.

(2016). Estimating standardized effect sizes for two-and three-level partially nested data. Multivariate Behavioral Research, 51(6), 740–756. https://doi.org/10.1080/00273171.2016.1231606

69.

Landau

Stahl

(2013). Sample size and power calculations for medical studies by simulation when closed form expressions are not available. Statistical Methods in Medical Research, 22(3), 324–345. https://doi.org/10.1177/0962280212439578

70.

Lane

S. P.

Hennes

E. P.

(2018). Power struggles: Estimating sample size for multilevel relationships research. Journal of Social and Personal Relationships, 35(1), 7–31. https://doi.org/10.1177/0265407517710342

71.

Lange

K. L.

Little

R. J.

Taylor

J. M.

(1989). Robust statistical modeling using the t distribution. Journal of the American Statistical Association, 84(408), 881–896. https://doi.org/10.2307/2290063

72.

LeBeau

(2016). Impact of serial correlation misspecification with the linear mixed model. Journal of Modern Applied Statistical Methods, 15(1), 389–416. https://doi.org/10.22237/jmasm/1462076400

73.

Leckie

Browne

Goldstein

Merlo

Austin

(2019, April). Variance partitioning in multilevel models for count data [Paper presentation]. 12th International Multilevel Conference, Utrecht, Netherlands. https://arxiv.org/ftp/arxiv/papers/1911/1911.06888.pdf

74.

Lee

Grimm

K. J.

(2018). Generalized linear mixed-effects modeling programs in R for binary outcomes. Structural Equation Modeling: A Multidisciplinary Journal, 25(5), 824–828. https://doi.org/10.1080/10705511.2018.1500141

75.

Lindstrom

M. J.

Bates

D. M.

(1988). Newton-Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83(404), 1014–1022. https://doi.org/10.1080/01621459.1988.10478693

76.

Liu

Zhang

Grimm

K. J.

(2016). Comparison of inverse Wishart and separation-strategy priors for Bayesian estimation of covariance parameter matrix in growth curve analysis. Structural Equation Modeling: A Multidisciplinary Journal, 23(3), 354–367. https://doi.org/10.1080/10705511.2015.1057285

77.

Liu

Agresti

(2005). The analysis of ordered categorical data: An overview and a survey of recent developments. Test, 14(1), 1–73. https://doi.org/10.1007/BF02595397

78.

Loy

(2016). Package HLMdiag (Version 0.4.0) [Computer software]. http://cran.r-project.org/web/packages/HLMdiag/HLMdiag.pdf

79.

Tang

Crits-Christoph

Zhang

(2009). On the impact of parametric assumptions and robust alternatives for longitudinal data analysis. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 51(4), 627–643. https://doi.org/10.1002/bimj.200800186

80.

Luke

S. G.

(2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y

81.

Lunn

Spiegelhalter

Thomas

Best

(2009a). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28(25), 3049–3067. https://doi.org/10.1002/sim.3680

82.

Lunn

Spiegelhalter

Thomas

Best

(2009b). Rejoinder to commentaries on “The BUGS project: Evolution, critique and future directions.” Statistics in Medicine, 28(25), 3081–3082. https://doi.org/10.1002/sim.3691

83.

Luo

Cappaert

K. J.

Ning

(2015). Modelling partially cross-classified multilevel data. British Journal of Mathematical and Statistical Psychology, 68(2), 342–362. https://doi.org/10.1111/bmsp.12050

84.

Luo

Kwok

(2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44(2), 182–212. https://doi.org/10.1080/00273170902794214

85.

Luo

Kwok

(2012). The consequences of ignoring individuals’ mobility in multilevel growth models: A Monte Carlo study. Journal of Educational and Behavioral Statistics, 37(1), 31–56. https://doi.org/10.3102/1076998610394366

86.

Maas

C. J.

Hox

J. J.

(2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46(3), 427–440. https://doi.org/10.1016/j.csda.2003.08.006

87.

Marsh

H. W.

Lüdtke

Nagengast

Trautwein

Morin

A. J.

Abduljabbar

A. S.

Köller

(2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488

88.

Martino

Akerkar

Rue

(2011). Approximate Bayesian inference for survival models. Scandinavian Journal of Statistics, 38(3), 514–528. https://doi.org/10.1111/j.1467-9469.2010.00715.x

89.

Mathieu

J. E.

Aguinis

Culpepper

S. A.

Chen

(2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951–966. https://doi.org/10.1037/a0028380

90.

Matuschek

Kliegl

Vasishth

Baayen

Bates

(2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94(June), 305–315. https://doi.org/10.1016/j.jml.2017.01.001

91.

McCullagh

(1980). Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological), 42(2), 109–127. https://doi.org/10.1111/j.2517-6161.1980.tb01109.x

92.

McCulloch

C. E.

Neuhaus

J. M.

(2011). Misspecifying the shape of a random effects distribution: Why getting it wrong may not matter. Statistical Science, 26(3), 388–402. https://doi.org/10.1214/11-STS361

93.

McDonald

R. P.

M. H. R.

(2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. https://doi.org/10.1037/1082-989X.7.1.64

94.

McNeish

(2016). Estimation methods for mixed logistic models with few clusters. Multivariate Behavioral Research, 51(6), 790–804. https://doi.org/10.1080/00273171.2016.1236237

95.

McNeish

(2017). Small sample methods for multilevel modeling: A colloquial elucidation of REML and the Kenward-Roger correction. Multivariate Behavioral Research, 52(5), 661–670. https://doi.org/10.1080/00273171.2017.1344538

96.

McNeish

(2019). Poisson multilevel models with small samples. Multivariate Behavioral Research, 54(3), 444–455. https://doi.org/10.1080/00273171.2018.1545630

97.

McNeish

Stapleton

L. M.

(2016). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51(4), 495–518. https://doi.org/10.1080/00273171.2016.1167008

98.

McNeish

Wentzel

K. R.

(2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52(2), 200–215. https://doi.org/10.1080/00273171.2016.1262236

99.

Moehring

Schmidt

(2013). “MLT”: Module providing different tools for multilevel modeling [Computer software]. Boston College Department of Economics.

100.

Moerbeek

(2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5

101.

Moineddin

Matheson

F. I.

Glazier

R. H.

(2007). A simulation study of sample size for multilevel logistic regression models. BMC Medical Research Methodology, 7, Article 34. https://doi.org/10.1186/1471-2288-7-34

102.

Murphy

Pituch

(2009). The performance of multilevel growth curve models under an autoregressive moving average process. Journal of Experimental Education, 77(3), 255–284. https://doi.org/10.3200/JEXE.77.3.255-284

103.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus user’s guide (8th ed). Muthén & Muthén.

104.

Nakagawa

Schielzeth

(2013). A general and simple method for obtaining R² from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012.00261.x

105.

Natarajan

Kass

R. E.

(2000). Reference Bayesian methods for generalized linear mixed models. Journal of American Statistician Association, 95(449), 227–237. https://doi.org/10.1080/01621459.2000.10473916

106.

Pinheiro

Bates

(2006). Mixed-effects models in S and S-PLUS. Springer Science + Business Media.

107.

Pinheiro

J. C.

Chao

E. C.

(2006). Efficient Laplacian and adaptive Gaussian quadrature algorithms for multilevel generalized linear mixed models. Journal of Computational and Graphical Statistics, 15(1), 58–81. https://doi.org/10.1198/106186006X96962

108.

Rasbash

Goldstein

(1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19(4), 337–350. https://doi.org/10.2307/1165397

109.

Rasbash

Steele

Browne

W. J.

Goldstein

(2019). A user’s guide to MLwiN, v3.03. Centre for Multilevel Modelling, University of Bristol.

110.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Sage.

111.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

(2019). HLM 8 for Windows [Computer software]. Scientific Software International.

112.

Raudenbush

S. W.

Spybrook

Congdon

Liu

X. F.

Martinez

Bloom

(2011). Optimal design software for multi-level and longitudinal research (Version 3.01) [Computer software]. William T. Grant Foundation. http://www.wtgrantfoundation.org

113.

Rights

J. D.

Sterba

S. K.

(2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309–338. https://doi.org/10.1037/met0000184

114.

Rodriguez

Goldman

(1995). An assessment of estimation procedures for multilevel models with binary responses. Journal of the Royal Statistical Society, Series A: Statistics in Society, 158(1), 73–89. https://doi.org/10.2307/2983404

115.

Rodriguez

Goldman

(2001). Improved estimation procedures for multilevel models with binary response: A case study. Journal of the Royal Statistical Society, Series A: Statistics in Society, 164(2), 339–355. https://doi.org/10.1111/1467-985X.00206

116.

Roos

Held

(2011). Sensitivity analysis in Bayesian generalized linear mixed models for binary data. Bayesian Analysis, 6(2), 259–278. https://doi.org/10.1214/11-BA609

117.

Rue

Martino

Chopin

(2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(2), 319–392. https://doi.org/10.1111/j.1467-9868.2008.00700.x

118.

SAS Institute. (2017). SAS/STAT® 14.3 user’s guide.

119.

Satterthwaite

F. E.

(1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. https://doi.org/10.2307/3002019

120.

Scheipl

Greven

Küchenhoff

(2008). Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models. Computational Statistics & Data Analysis, 52(7), 3283–3299. https://doi.org/10.1016/j.csda.2007.10.022

121.

Scherbaum

Pesner

(2018). Power analysis for multilevel research. In Humphrey

S. E.

LeBreton

J. M.

(Eds.), The handbook of multilevel theory, measurement, and analysis (pp. 498–534). American Psychological Association. https://doi.org/10.1037/0000115-015

122.

Schielzeth

Dingemanse

N. J.

Nakagawa

Westneat

D. F.

Allegue

Teplitsky

Réale

Dochtermann

N. A.

Garamszegi

L. Z.

Araya-Ajoy

Y. G.

(2020). Robustness of linear mixed-effects models to violations of distributional assumptions. Methods in Ecology and Evolution, 11(9), 1141–1152. https://doi.org/10.1111/2041-210X.13434

123.

Schreiber

J. B.

Griffin

B. W.

(2004). Review of multilevel modeling and multilevel studies in The Journal of Educational Research (1992–2002). Journal of Educational Research, 98(1), 24–34. https://doi.org/10.3200/JOER.98.1.24-34

124.

Seco

G. V.

García

M. A.

García

M. P. F.

Rojas

P. E. L.

(2013). Multilevel bootstrap analysis with assumptions violated. Psicothema, 25(4), 520–528. https://doi.org/10.7334/psicothema2013.58

125.

Self

S. G.

Liang

K. Y.

(1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398), 605–610. https://doi.org/10.1080/01621459.1987.10478472

126.

Singer

J. M.

Rocha

F. M.

Nobre

J. S.

(2017). Graphical tools for detecting departures from linear mixed model assumptions and some remedial measures. International Statistical Review, 85(2), 290–324. https://doi.org/10.1111/insr.12178

127.

Snijders

T. A.

Bosker

R. J.

(1994). Modeled variance in two-level models. Sociological Methods & Research, 22(3), 342–363. https://doi.org/10.1177/0049124194022003004

128.

Snijders

T. A. B.

Bosker

R. J.

(2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage.

129.

Snijders

T. A.

Bosker

R. J.

Guldemond

(1996). PINT user’s manual (Version 1.6) [Computer software]. KIP. https://www.kip.com/software.php

130.

Spiegelhalter

Thomas

Best

Lunn

(2014). OpenBUGS user manual (Version 3.2.3) [Computer software manual]. http://www.openbugs.net/Manuals/Manual.html

131.

Staggs

V. S.

(2017). Comparison of naïve, Kenward–Roger, and parametric bootstrap interval approaches to small-sample inference in linear mixed models. Communications in Statistics: Simulation and Computation, 46(3), 1933–1943. https://doi.org/10.1080/03610918.2015.1019002

132.

Stapleton

L. M.

Yang

J. S.

Hancock

G. R.

(2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200

133.

StataCorp. (2019). Stata statistical software: Release 16.

134.

Stram

D. O.

Lee

J. W.

(1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50(4), 1171–1177. https://doi.org/10.2307/2533455

135.

Van der Leeden

Meijer

Busing

F. M.

(2008). Resampling multilevel models. In de Leeuw

Meijer

(Eds.), Handbook of multilevel analysis (pp. 401–433). Springer. https://doi.org/10.1007/978-0-387-73186-5_11

136.

Wiencierz

Greven

Küchenhoff

(2011). Restricted likelihood ratio testing in linear mixed models with general error covariance structure. Electronic Journal of Statistics, 5(2011), 1718–1734. https://projecteuclid.org/download/pdfview_1/euclid.ejs/1323785606

137.

(2003). Measuring explained variation in linear mixed effects models. Statistics in Medicine, 22(22), 3527–3541. https://doi.org/10.1002/sim.1572

138.

Zeger

S. L.

Karim

M. R.

(1991). Generalized linear-models with random effects: A Gibbs sampling approach. Journal of American Statistician Association, 86(413), 79–86. https://doi.org/10.1080/01621459.1991.10475006

139.

Zondervan-Zwijnenburg

Peeters

Depaoli

Van de Schoot

(2017). Where do priors come from? Applying guidelines to construct informative priors in small sample research. Research in Human Development, 14(4), 305–320. https://doi.org/10.1080/15427609.2017.1370966

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB

0.03 MB