Design Effect in Multilevel Settings: A Commentary on a Latent Variable Modeling Procedure for Its Evaluation

Abstract

A latent variable modeling-based procedure is discussed that permits to readily point and interval estimate the design effect index in multilevel settings using widely circulated software. The method provides useful information about the relationship of important parameter standard errors when accounting for clustering effects relative to conducting single-level analyses. The approach can also be employed as an addendum to point and interval estimation of the intraclass correlation coefficient in empirical research. The discussed procedure makes it easily possible to evaluate the design effect in two-level studies by utilizing the popular latent variable modeling methodology and is illustrated with an example.

Keywords

design effect interval estimation intraclass correlation coefficient latent variable modeling multilevel setting two-level study.

Get full access to this article

View all access options for this article.

References

Casella

Berger

J. O.

(2002). Statistical inference. Wadsworth.

Das Gupta

. (2008). Asymptotic theory of statistics and probability. Springer.

Efron

Tibshiriani

R. J.

(1993). An introduction to the bootstrap. Chapman & Hall. https://doi.org/10.1007/978-1-4899-4541-9

Goldstein

(2011). Multilevel statistical models. Arnold. https://doi.org/10.1002/9780470973394

Hox

J. J.

Moerbeek

van de Schoot

(2018). Multilevel analysis. Techniques and applications. Taylor & Francis. https://doi.org/10.4324/9781315650982

Kish

(1965). Sampling techniques. Wiley.

Lai

M. H. C.

Kwok

(2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. Journal of Experimental Education, 83(3), 423-438. https://doi.org/10.1080/00220973.2014.907229

Maas

C. J.

Hox

J. J.

(2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46(3), 427-440. https://doi.org/10.1016/j.csda.2003.08.006

Maas

Hox

(2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86-92. https://doi.org/10.1027/1614-2241.1.3.86

10.

Mortimore

Sammons

Stoll

Lewis

Ecob

(1988). School matters, the junior years. Open Books. https://doi.org/10.1525/9780520330375

11.

Muthén

B. O.

(2002). Beyond SEM: General latent variable modeling. Behaviormetrika, 29, 87-117. https://doi.org/10.2333/bhmk.29.81

12.

Muthén

B. O.

Satorra

(1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316. https://doi.org/10.2307/271070

13.

Muthén

L. K.

Muthén

B. O.

(2021). Mplus user’s guide. Muthén & Muthén.

14.

Peugh

J. L.

(2010). A practical guide to multilevel modeling. Journal of School Psychology, 48(1), 85-112. https://doi.org/10.1016/j.jsp.2009.09.002

15.

Rabe-Hesketh

Skrondal

(2012). Multilevel and longitudinal modeling with Stata. Stata Press.

16.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models. Applications and data analysis methods (2nd ed.). Sage.

17.

Raykov

(2011). Intra-class correlation coefficients in hierarchical designs: Evaluation using latent variable modeling. Structural Equation Modeling, 18, 73-90. https://doi.org/10.1080/10705511.2011.534319

18.

Raykov

Marcoulides

G. A.

(2006). A first course in structural equation modeling. Lawrence Erlbaum.

19.

Raykov

Marcoulides

G. A.

(2015). Intra-class correlation coefficients in hierarchical design studies with discrete response variables: A note on a direct interval estimation procedure. Educational and Psychological Measurement, 75(6), 1063-1070. https://doi.org/10.1177/0013164414564052

20.

Raykov

Marcoulides

G. A.

Akaeze

(2017). Comparing between and within group variances in a two-level study: A latent variable modeling approach to evaluating their relationship. Educational and Psychological Measurement, 77(2), 351-361. https://doi.org/10.1177/0013164416634166

21.

StataCorp. (2019). Stata Base Reference Manual: Release 16. StataCorp LLC.