There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.
AlasuutariP.BickmanL.BrannenJ. (Eds.). (2008). The SAGE handbook of social research methods. London: Sage.
2.
BaguleyT. (2009). Standardized or simple effect size: What should be reported?British Journal of Psychology, 100, 603–617.
3.
BloomH. S.HillC. J.BlackA. B.LipseyM. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1, 289–328.
4.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. Hoboken, NJ: John Wiley & Sons.
5.
BurkhardtH.SchoenfeldA. H. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3–14.
6.
CheungA.SlavinR. E. (2016). How methodological features of research studies affect effect sizes. Educational Researcher, 45(5), 283–292.
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
8.
DienesZ. (2011). Bayesian versus orthodox statistics: Which side are you on?Perspectives on Psychological Science, 6, 274–290.
9.
DienesZ.CoultonS.HeatherN. (2018). Using Bayes factors to evaluate evidence for no effect: Examples from the SIPS project. Addiction, 113, 240–246.
HattieJ. A. C. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. Oxford, UK: Routledge.
15.
HigginsJ. P.AltmanD. G.GotzscheP. C.JuniP.MoherD.OxmanA. D., . . . Cochrane Bias Methods Group. (2011). Cochrane Statistical Methods Group: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ: British Medical Journal, 343, d5928.
16.
HillC. J.BloomH. S.BlackA. R.LipseyM. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172–177.
17.
IoannidisJ. P. A. (2005). Why most published research findings are false. PLoS Medicine, 8, e124.
18.
JeffreysH. (1961). Theory of probability (3rd ed.). Oxford, UK: Oxford University Press
19.
JohnL. K.LoewensteinG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
20.
KarlssonP.BergmarkA. (2015). Compared with what? An analysis of control-group types in Cochrane and Campbell reviews of psychosocial treatment efficacy with substance use disorders. Addiction, 110, 420–428.
21.
LipseyM. W.WilsonD. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209.
22.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
23.
MakelM. C.PluckerJ. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304–316.
24.
MookD. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
35.
Society for Research on Educational Effectiveness. (n.d.). Registry of efficacy and effectiveness studies. Retrieved from https://www.sree.org/pages/registry.php
36.
WeisburdD.PetrosinoA.MasonG. (1993). Design sensitivity in criminal justice experiments. Crime and Justice, 17, 337–379.
37.
YoungN. S.IoannidisJ. P.Al-UbaydliO. (2008). Why current publication practices may distort science. PLoS Medicine, 5(10), e201.
38.
ZimmermanB. J.BanduraA.Martinez-PonsM. (1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29, 663–676.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
