Most evaluators have embraced the goal of evidence-based practice (EBP). Yet, many have criticized EBP review systems that prioritize randomized control trials and use various criteria to limit the studies examined. They suggest this could produce policy recommendations based on small, unrepresentative segments of the literature and recommend a more traditional, inclusive approach. This article reports two empirical studies assessing this criticism, focusing on the What Works Clearinghouse (WWC). An examination of outcomes of 252 WWC reports on literacy interventions found that 6% or fewer of the available studies were selected for review. Half of all intervention reports were based on only one study of a program. Data from 131 studies of a reading curriculum were used to compare conclusions using WWC procedures and more inclusive procedures. Effect estimates from the inclusive approach were more precise and closer to those of other reviews. Implications are discussed.
AdamsG.EngelmannS. (1996). Research on Direct Instruction: 25 Years beyond DISTAR. Seattle, WA: Educational Achievement Systems.
2.
American Evaluation Association. (2003). American Evaluation Association response to U.S. Department of Education, notice of proposed priority, federal registry RIN 1890-ZA00, November 4, 2003. Retrieved fromhttp://www.eval.org/p/cm/ld/fid=95
3.
BiglanA.AryD.WagenaarA. C. (2000). The value of interrupted time-series experiments for community intervention research. Prevention Science, 1, 31–49.
4.
BiglanA.FlayB. R.KomroK. A.WagenaarA. C.KjellstrandJ. (2012). Adaptive time-series designs for evaluating complex multicomponent interventions in neighborhoods and communities. Eugene, OR: Oregon Research Institute.
5.
BormanG. D.HewesG. M.OvermanL. T.BrownS. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73, 125–230.
6.
CampbellD. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.
7.
CampbellD. T.StanleyJ. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
8.
ChenH. TDonaldsonS. I.MarkM. M. (2011). Validity frameworks for outcome evaluation. In ChenH. T.DonaldsonS. I.MarkM. M. (Eds.), Advancing validity in outcome evaluation: Theory and practice. New Directions for Evaluation (Vol. 130, pp. 5–16).
9.
ConfreyJ. (2006). Comparing and contrasting the National Research Council report on Evaluating curricular effectiveness with the What Works Clearinghouse approach. Educational Evaluation and Policy Analysis, 28, 195–213.
10.
CookT. D.CampbellD. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally.
11.
CoughlinC. (2014). Outcomes of Engelmann’s Direct Instruction: Research syntheses. In StockardJ. (Ed.), The science and success of Engelmann’s Direct Instruction (pp. 25–54). Eugene, OR: NIFDI Press.
12.
DonaldsonS. I.ChristieC. A. (2005). The 2004 Claremont debate: Lipsey vs. Scriven: Determining causality in program evaluation and applied research: Should experimental evidence be the gold standard?Journal of MultiDisciplinary Evaluation, 2, 60–77.
13.
DonaldsonS. I.ChristieC. A.MarkM. M. (2009). What counts as credible evidence in applied research and evaluation practice?Los Angeles, CA: Sage.
14.
DonaldsonS. I.PattonM. Q.FettermanD. M.ScrivenM. (2010). The 2009 Claremont debates: The promise and pitfalls of utilization-focused and empowerment evaluation. Journal of MultiDisciplinary Evaluation, 6, 15–57.
15.
DrakeR. E.LatimerE. A.LeffH. S.McHugoG. J.BurnsB. J. (2004). What is evidence?Child and Adolescent Psychiatric Clinics of North American, 13, 717–728.
16.
FisherR. A. (1925). Statistical methods for research workers. New York, NY: McGraw-Hill.
17.
FisherR. A. (1935). The design of experiments. London, England: Oliver & Boyd.
18.
GarganiJ.DonaldsonS. I. (2011). What works for whom, where, why, for what, and when? Using evaluation evidence to take action in local contexts. In ChenH. T.DonaldsonS. I.MarkM. M. (Eds.), Advancing validity in outcome evaluation: Theory and practice. New directions for evaluation (Vol. 130, pp. 17–30). San Francisco, CA: Wiley.
19.
GrangerR. C.MaynardR. (2015). Unlocking the potential of the “what works” approach to policymaking and practice: Improving impact evaluations. American Journal of Education, 36, 558–569.
HattieJ. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, England: Routledge.
22.
HempenstallK. (2014). What works? Evidence-based practice in education is complex. Australian Journal of Learning Difficulties, 19, 113–127.
23.
JacobsW. J.SiscoM.HillD.MalterF.FigueredoA. J. (2012). Evaluating theory-based evaluation: Information, norms and adherence. Evaluation and Program Planning, 35, 354–369.
24.
KazdinA. E. (2004). Evidence-based treatments: Challenges and priorities for practice and research. Child and Adolescent Psychiatric Clinics of North America, 13, 923–940.
25.
National Research Council. (2002). Scientific research in education. Committee on Scientific Principles for Education Research. In ShavelsonR. J.TowneL. (Eds.), Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
26.
OanceaA.PringR. (2008). The importance of being thorough: On systematic accumulations of “What Works” in education research. Journal of Philosophy of Education, 47, 15–39.
27.
Office of the Federal Register, National Archives and Records Administration. (2005). Scientifically based evaluation methods, Department of Education, notice of final priority, RIN 1890-ZA00. Federal Register, 70, 3586–3589.
28.
PopperK. R. (1962). Conjectures and refutations: The growth of scientific knowledge. New York, NY: Basic Books.
29.
SchmidtW. P. (2014). The elusive effect of water and sanitation on the global burden of disease. Tropical Medicine and International Health, 19, 522–527.
30.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
31.
ShoenfeldA. H. (2006). What doesn’t work: The challenges and failure of the What Works Clearinghouse to conduct meaningful reviews of studies of mathematics curricula. Educational Research, 35, 13–21.
32.
SloaneF. (2008). Through the looking glass: Experiments, quasi-experiments, and the medical model. Educational Researcher, 37, 41–46.
33.
SlocumT. A.DetrichR.SpencerT. D. (2012). Evaluating the validity of systematic reviews to identify empirically supported treatments. Education and Treatment of Children, 35, 201–233.
34.
SlocumT. A.SpencerT. D.DetrichR. (2012). Best available evidence: Three complementary approaches. Education and Treatment of Children, 35, 153–181.
35.
St. ClairT.CookT. D.HallbergK. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35, 311–327.
TallmadgeG. K. (1977). The Joint Dissemination Review Panel ideabook. Washington, DC: National Institute of Education and U.S. Office of Education.
38.
TallmadgeG. K. (1982). An empirical assessment of norm-referenced evaluation methodology. Journal of Educational Measurement, 19, 97–112.
39.
What Works Clearinghouse. (2008). WWC procedures and standards handbook (Version 1.0). Washington, DC: Institute of Education Sciences. Retrieved fromhttp://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19
40.
What Works Clearinghouse. (2014). WWC procedures and standards handbook (Version 3.0). Washington, DC: Institute of Education Sciences. Retrieved fromhttp://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19
41.
WoodT. W. (2014). Examining the inaccuracies and mystifying policies and standards of the What Works Clearinghouse: Findings from a FOIA request(NIFDI Technical Report 2014-5). Eugene, OR: National Institute for Direct Instruction.
42.
ZvochK. (2012). How does fidelity of implementation matter? Using multilevel models to detect relationships between participant outcomes and the delivery and receipt of treatment. American Journal of Evaluation, 33, 547–565.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
