Educators who are urged to use evidence-based practices to improve instruction often end up disappointed at the results, which fall short of those touted in the research and by the What Works Clearinghouse. Stanley Pogrow explains how common strategies researchers use to demonstrate evidence of success, such as statistical significance of or effect size, can make results seem more dramatic than they are. Such measures are more valuable to researchers than to educators who are looking for interventions that will make a difference in their classrooms or schools. Pogrow offers recommendations for both educators and the research community for improving the state of education research.
BairdM.D.PaneJ.F. (2019). Translating standardized effects of education programs into more interpretable metrics. Educational Researcher, 48 (4), 217–218.
2.
BerwickD.M. (2008). The science of improvement. Journal of the American Medical Association, 299 (10), 1182–1184.
3.
BoulayB.GoodsonB.OlsenR.McCormickR.FryeM.Gan.K.…SamaM. (2018). The Investing in Innovation Fund: Summary of 67 evaluations. Final Report. U.S. Department of Education, Institute of Education Sciences.
4.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum.
5.
Center for Research on Education Outcomes. (2013). National charter school study II. Stanford University.
6.
Center for Research on Education Outcomes. (2023). As a matter of fact: The national charter school study III. Stanford University.
7.
D’AgostinoJ.PogrowS. (2023, April13). Identifying highly effective interventions [Conference presentation]. American Educational Research Association Annual Meeting, Chicago, IL.
8.
GawandeA. (2007). Better: A surgeon’s notes on performance. Metropolitan Books.
9.
IoannidisJ.P. (2005). Why most published research findings are false. PLoS Medicine, 2 (8).
10.
James-BurdumyJ.S.MansfieldW.DekeJ.CareyN.Lugo-GillJ.HersheyA.DouglasA.…PendletonA. (2009, June8). Effectiveness of selected supplemental reading comprehension interventions: Impacts on a first cohort of fifth-grade students. U.S. Department of Education, Institute of Education Sciences.
11.
KozolJ. (2005). Confections of apartheid: A stick-and-carrot pedagogy for the children of our inner-city poor. Phi Delta Kappan, 87 (4), 265–275.
12.
KozolJ. (2006). Success for All: Trying to make an end run around inequality and segregation. Phi Delta Kappan, 87 (8), 624–626.
KristofN. (2023, Feb11). Two-thirds of kids struggle to read, and we know how to fix it. New York Times.
15.
LipseyM.W.PuzioK.YunC.HebertM.A.Steinka-FryK.ColeM.W.…BusickM.D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. Institute of Education Sciences, U.S. Department of Education.
16.
Lortie-ForgeusH.InglisM. (2019). Rigorous large-scale educational RCTs are often uninformative: Should we be concerned?Educational Researcher, 48 (3), 158–166.
17.
PogrowS. (2002). Success for All is a failure. Phi Delta Kappan, 83 (6), 463–468.
18.
PogrowS. (2019). How effect size (Practical Significance) misleads clinical practice: The case for switching to practical benefit to assess applied research findings. The American Statistician, 73 (sup1), 223–234.
19.
SteinerD. (2021). Make sense of the research: A primer for educational leaders. Phi Delta Kappan, 103 (3), 43–47.
20.
VenezkyR.L. (1998). An alternative perspective on Success for All. In K. Wong (Ed.), Advances in Educational Policy (Vol. 4, pp. 145–65). JAI Press.
21.
WassersteinR.L.SchirmA.L.LazarN.A. (2019). Moving to a world beyond “p< 0.05.”The American Statistician, 73 (sup1), 1–19.
22.
ZiliakS.T.McCloskeyD.N. (2004). Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33, 427–456.
