How Many Cases per Cluster? Operationalizing the Number of Units per Cluster Relative to Minimum Detectable Effects in Two-Level Cluster Randomized Evaluations with Linear Outcomes

Abstract

In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical power. Typically, researchers state that “about 30 units per cluster” is the most that will yield benefit towards statistical precision. To avoid rules of thumb not grounded in statistical theory and practical considerations, and instead provide guidance for this question, the ratio of the minimum detectable effect size (MDES) to the larger MDES with one less unit per cluster is related to the key parameters of the cluster randomized design. Formulas for this subsequent difference effect size ratio (SDESR) at a given number of units are provided, as are formulas for finding the number of units for an assumed SDESR. In general, the point of diminishing returns occurs with smaller numbers of units for larger values of the intraclass correlation.

Keywords

statistical power sample size cluster randomized designs experiments

Get full access to this article

View all access options for this article.

References

Aguinis

Harden

E. E.

(2010). Sample size rules of thumb: Evaluating three common practices. In Statistical and Methodological Myths and Urban Legends (pp. 267–360). Routledge.

Bloom

H. S.

(1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556.

Bloom

H. S.

Bos

J. M.

Lee

S. W.

(1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23(4), 445–469.

Bloom

H. S.

Richburg-Hayes

Black

A. R.

(2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59.

Cohen

(1992). A power primer. Psychological Bulletin, 112(1), 155–159.

Demidenko

(2007). Sample size determination for logistic regression revisited. Statistics in Medicine, 26, 3385–3397.

Hedberg

E. C.

Hedges

L. V.

(2014). Reference values of within-district intraclass correlations of academic achievement by district characteristics: Results from a meta-analysis of district-specific values. Evaluation Review, 38(10), 546–582.

Hedberg

E. C.

(2017b). Introduction to power analysis: Two-group studies (Vol. 176). Sage Publications.

Hedberg

E. C.

(2017a). Intraclass correlations. In Allen

(Ed.), The sage encyclopedia of communication research methods (Vols. 1-4). Sage Publications, Inc.

10.

Hedges

L. V.

(1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 107–128.

11.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.

12.

Hedges

L. V.

Hedberg

E. C.

(2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(10), 445–489.

13.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177.

14.

Kish

(1965). Survey sampling. Wiley.

15.

Liu

X. S.

(2013). Statistical Power Analysis for the Social and Behavioral Sciences: Basic and Advanced Techniques. Routledge.

16.

Min

Park

C. G.

Scott

L. D.

(2016). Evaluating technical efficiency of nursing care using data envelopment analysis and multilevel modeling. Western Journal of Nursing Research, 38(11), 1489–1508.

17.

Murray

D. M.

(1998). Design and Analysis of Group-Randomized Trials (Vol. 29). Oxford University Press.

18.

Raudenbush

S. W.

(1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185.

19.

Raudenbush

S. W.

Martinez

Spybrook

(2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5–29.

20.

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ .

21.

Ryan

T. P.

(2013). Sample Size Determination and Power. John Wiley & Sons.

22.

White

M. D.

Mora

V. J.

Orosco

Hedberg

E. C.

(2021). Moving the needle: Can training alter officer perceptions and use of de-escalation? Policing: An International Journal, 44(43), 418–436.

23.

Wilson

S. J.

Price

C. S.

Kerns

S. E. U.

Dastrup

S. D.

Brown

S. R.

(2019). Title IV-E prevention services clearinghouse handbook of standards and procedures, version 1.0, OPRE Report # 2019–56. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.20 MB