Sage Journals: Discover world-class research

Abstract

It is a healthy exercise to debate the merits of using effect-size benchmarks to interpret research findings. However, these debates obscure a more central insight that emerges from empirical distributions of effect-size estimates in the literature: Efforts to improve education often fail to move the needle. I find that 36% of effect sizes from randomized control trials of education interventions with standardized achievement outcomes are less than 0.05 SD. Publication bias surely masks many more failed efforts from our view. Recognizing the frequency of these failures should be at the core of any approach to interpreting the policy relevance of effect sizes. We can aim high without dismissing as trivial those effects sizes that represent more incremental improvement.

Keywords

achievement educational policy effect size research utilization

Get full access to this article

View all access options for this article.

References

Agodini

Harris

(2010). An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness, 3(3), 199–253.

Andrews

Kasy

(2019). Identification of and correction for publication bias. American Economic Review, 109(8), 2766–2794.

Chan

A. W.

Hróbjartsson

Haahr

M. T.

Gøtzsche

P. C.

Altman

D. G.

(2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA, 291(20), 2457–2465.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

DellaVigna

Linos

(2022). RCTs to scale: Comprehensive evidence from two nudge units. Econometrica, 90(1), 81–116.

Dickerson

(2005). Publication bias: Recognizing the problem, understandings its origins and scope, and preventing harm. In Rothstein

H. R.

Sutton

A. J.

Borenstein

(Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 11–34). Wiley

Evans

D. K.

Yuan

(2022). How big are effect sizes in international education studies? Educational Evaluation and Policy Analysis, 44(3), 532–540.

Fryer

R. G.

Jr. (2017). The production of human capital in developed countries: Evidence from 196 randomized field experiments. In Banerjee

A. V.

Duflo

(Eds.), Handbook of economic field experiments (Vol. 2, pp. 95–322). North-Holland.

Glass

G. V.

McGaw

Smith

M. L.

(1981). Meta-analysis in social research. SAGE.

10.

Jackson

C. K.

Mackevicius

(2021). The distribution of school spending impacts (No. w28517). National Bureau of Economic Research.

11.

Kaplan

R. M.

Irvin

V. L.

(2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PloS One, 10(8), Article e0132382. https://doi.org/10.1371/journal.pone.0132382

12.

Kelley

Preacher

K. J.

(2012). On effect size. Psychological Methods, 17(2), 137.

13.

Kraft

M. A.

(2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253.

14.

Lortie-Forgues

Inglis

(2019). Rigorous large-scale educational RCTs are often uninformative: Should we be concerned? Educational Researcher, 48(3), 158–166.

15.

Morris

C. N.

(1983). Parametric empirical Bayes inference: Theory and applications. Journal of the American Statistical Association, 78(381), 47–55.

16.

Rothstein

H. R.

Sutton

A. J.

Borenstein

(2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Wiley.

17.

Simpson

(2019). Separating arguments from conclusions: The mistaken role of effect size in educational policy research. Educational Research and Evaluation, 25(1–2), 99–109.

18.

Simpson

(2021). Benchmarking a misnomer: A note on “Interpreting effect sizes in education interventions.” Educational Researcher. Advance online publication. https://doi.org/10.3102/0013189X20985448

19.

Tyack

D. B.

Cuban

(1995). Tinkering toward utopia. Harvard University Press.

20.

von Hippel

P. T.

Bellows

. (2018). How much does teacher quality vary across teacher preparation programs? Reanalyses from six states. Economics of Education Review, 64, 298–312.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB