Sage Journals: Discover world-class research

Abstract

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that “works.” When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why. Accumulating and learning from possible failure mechanisms is not standard practice in education research, and it is not common to design interventions with causes of failure in mind. This chapter develops Boruch and Ruby’s proposition that the education sciences would benefit from a systematic approach to the study of failure. We review and taxonomize recent reports of large-scale randomized controlled trials in K–12 schooling that yielded at least one null or negative major outcome, including the nature of the event and reasons (if provided) for why it occurred. Our purpose is to introduce a broad framework for thinking about educational interventions that do not produce expected effects and seed a cumulative knowledge base on when, how, and why interventions do not reach expectations. The reasons why an individual intervention fails to elicit an outcome are not straightforward, but themes emerge when researchers’ reports are synthesized.

Get full access to this article

View all access options for this article.

References

Abbiati

Argentin

Pennisi

(2014). Learning from implementation: The case of the evaluation of a professional development programme for mathematics teachers in Italy. In Kalliola

(Ed.), Evaluation as a tool for research, learning, and making things better (pp. 223–240). Cambridge Scholars Publishing.

Achieve. (2018). Proficient vs. prepared 2018: Disparities between state tests and the 2017 National Assessment of Educational Progress (NAEP). https://www.achieve.org/files/Proficient%20vs.%20Prepared%20May2018_1.pdf

Allensworth

Ponisciak

Mazzeo

(2009). The schools teachers leave: Teacher mobility in Chicago public schools. Consortium on Chicago School Research at the University of Chicago.

Alvarez & Marsal Holdings LLC. (2018, January 26). Final report: District of Columbia Public Schools Audit and Investigation (Contract Number # CW57247). https://osse.dc.gov/sites/default/files/dc/sites/osse/release_content/attachments/Report%20on%20DCPS%20Graduation%20and%20Attendance%20Outcomes%20-%20Alvarez%26Marsal.pdf

Arens

Stoker

Barker

Shebby

Wang

Cicchinelli

L. F.

Williams

J. M.

(2012). Effects of curriculum and teacher professional development on the language proficiency of elementary English language learner students in the central region (NCEE 2012-4013). Mid-continent Research for Education and Learning.

Augustine

Engberg

Grimm

Lee

Wang

Christianson

Joseph

A. A.

(2018). Can restorative practices improve school climate and curb suspensions? An evaluation of the impact of restorative practices in a mid-sized urban school district. RAND Corporation.

Augustine

C. H.

McCombs

J. S.

Pane

J. F.

Schwartz

H. L.

Schweig

McEachin

Siler-Evans

(2016). Learning from summer: Effects of voluntary summer learning programs on low-income urban youth. RAND Corporation.

Bakker

van Dijk

Wicherts

J. M.

(2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060

Bandeira de Mello

Rahman

Fox

M. A.

C. S.

(2019). Mapping state proficiency standards onto NAEP scales: Results from the 2017 NAEP reading and mathematics assessments (NCES 2019-040). Institute of Education Sciences, National Center for Education Statistics.

10.

Boruch

Allen-Platt

Gerstner

(2019). To randomize or not to randomize? That is the question. New Directions in Evaluation, 2019(163), 73–82. https://doi.org/10.1002/ev.20373

11.

Boruch

Merlino

Bowden

Baker

Chao

(2016). In search of terra firma: Administrative records on teachers’ positional instability across subjects, grades, and schools and the implications for deploying randomized controlled trials. https://repository.upenn.edu/gse_pubs/393/

12.

Boruch

Ruby

(2015). To flop is human: Inventing better scientific approaches to anticipating failure. In Emerging Trends in the Social and Behavioral Sciences. https://doi.org/10.1002/9781118900772.etrds0362

13.

Bos

Sanchez

Tseng

Rayyes

Ortiz

Sinicrope

(2012). Evaluation of quality teaching for English learners (QTEL) professional development (NCEE 2012-4005). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

14.

Boyd

Lankford

Loeb

Wyckoff

(2002). Initial matches, transfers, and quits: Career decisions and the disparities in average-age teacher qualifications across schools (Stanford CEPA Working Paper). https://cepa.stanford.edu/sites/default/files/Initial_Matches_Transfers_and_Quits.pdf

15.

Bryk

A. S.

Gomez

Grunow

LeMahieu

(2015). Learning to improve: How America’s schools can get better at getting better. Harvard Education Publishing.

16.

Bureau of Transportation Statistics. (2020). Glossary. https://www.transtats.bts.gov/glossary.asp

17.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Rand McNally.

18.

Cavalluzzo

Lowther

D. L.

Mokher

Fan

(2012). Effects of the Kentucky Virtual Schools’ hybrid program for algebra I on grade 9 student math achievement (NCEE 2012-4020). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

19.

Chao

Park

Boruch

(2016). Ambient positional instability among Illinois teachers, AY 2007–2012: A briefing (CRESP Working Paper/Briefing). https://repository.upenn.edu/gse_pubs/395

20.

Coalition for Evidence-Based Policy. (2013). Randomized controlled trials commissioned by the Institute of Education Sciences since 2002: How many found positive versus weak or no effects. http://coalition4evidence.org/wp-content/uploads/2013/06/IES-Commissioned-RCTs-positive-vs-weak-or-null-findings-7-2013.pdf

21.

Code of Federal Regulations. (2020). Recurrent training. https://www.ecfr.gov/cgi-bin/text-idx?SID=47e3db32559d6354948f7080e1960aa4&mc=true&node=pt14.3.121&rgn=div5#se14.3.121_1427

22.

Cordray

D. S.

Pion

G. M.

Brandt

Molefe

(2013). The impact of the Measures of Academic Progress (MAP) program on student reading achievement (NCEE 2013-4000). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

23.

Core Standards. (2019). About the standards: Development process. http://www.corestandards.org/about-the-standards/development-process/

24.

Crain-Dorough

Elder

A. C.

(2021). Absorptive capacity as a means of understanding and addressing the disconnects between research and practice. Review of Research in Education, 45(1), 67–100. https://doi.org/10.3102/0091732X21990614

25.

Cronbach

(1975). Between the two disciplines of scientific psychology. American Psychologist, 30(2), 116–127. https://doi.org/10.1037/h0076829

26.

Dahlin

K. B.

Chuang

Roulet

T. J.

(2018). Opportunity, motivation, and ability to learn from failures and errors: Review, synthesis, and ways to move forward. Academy of Management Annals, 12(1), 252–277. https://doi.org/10.5465/annals.2016.0049

27.

Dawson

Yeomans

Brown

E. R.

(2018). Methodological challenges in education RCTs: Reflections from England’s Education Endowment Foundation. Educational Research, 60(3), 292–310. https://doi.org/10.1080/00131881.2018.1500079

28.

Dawson

S. L.

(2018). Sharing successes and hiding failures: “Reporting bias” in learning and teaching research. Studies in Higher Education, 43(8), 1405–1416. https://doi.org/10.1080/03075079.2016.1258052

29.

Eddy

R. M.

Ruitman

H. T.

Hankel

Sloper

(2010). The effects of Pearson Prentice Hall literature (2010) on student performance: Efficacy study. Cobblestone Applied Research and Evaluation, Inc.

30.

Foorman

(2016). Introduction to the special issue: Challenges and solutions to implementing effective reading intervention in schools. New Directions for Child and Adolescent Development, 2016(154), 7–10. https://doi.org/10.1002/cad.20172

31.

Franco

Malhotra

Simonovits

(2014). Public bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

32.

Frisone

Hooks

Boruch

(2016). Ambient positional instability among core subject Arkansas public school teachers: Interim report. https://repository.upenn.edu/gse_pubs/394

33.

Gersten

(2016). Commentary: The tyranny of time and the reality principle. New Directions for Child and Adolescent Development, 2016(154), 113–116. https://doi.org/10.1002/cad.20171

34.

Ginsburg

Smith

(2016). Do randomized controlled trials meet the “gold standard”? American Enterprise Institute.

35.

Gonzalez

Macintyre

Beccar-Varela

(2018). Challenges in adolescent reading intervention: Evidence from a randomized control trial. Mathematica Policy Research.

36.

Greaves

Sianesi

Sibieta

Amin-Smith

Callanan

Hudson

(2016). Achieve together: Evaluation report and executive summary. Education Endowment Foundation.

37.

Hall

L. S.

Callahan

(2018). It’s one of the biggest failures yet in K–12 philanthropy. What are the lessons? https://www.insidephilanthropy.com/home/2018/7/54/another-lesson-in-k-12-philanthropy-the-gates-teacher-effectiveness-initiative

38.

Hanushek

E. A.

(1996). Outcomes, costs and incentives in schools. In Improving America’s schools: The role of incentives (pp. 29–52). National Research Council.

39.

Heller

J. I.

(2012). Effects of Making Sense of SCIENCE™ professional development on the achievement of middle school students, including English language learners (NCEE 2012-4002). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

40.

Herrington

C. D

Maynard

(Eds.). (2019). Randomized controlled trials meet the real world: The nature and consequence of null findings. Educational Researcher, 48(9), 577–579. https://doi.org/10.3102/0013189X19891441

41.

Hill

C. J.

(2019) Commentary on the null results special issue. Educational Researcher, 48(9), 608–610. https://doi.org/10.3102/0013189X19891432

42.

Hill

H. C.

Erickson

(2019). Using implementation fidelity to aid in interpreting program impacts: A brief review. Educational Researcher, 48(9), 590–598. https://doi.org/10.3102/0013189X19891436

43.

Husain

Wishart

Marshall

Frankenberg

Bussard

Chidley

Hudson

Votjkova

Morris

(2018). Family skills: Evaluation report and executive summary. Education Endowment Foundation.

44.

Jaciw

Toby

Lai

Lin

(2012). Measuring the average impact of an iPad algebra program. Empirical Education Inc.

45.

Jaciw

A. P.

Schellinger

A. M.

Lin

Zacamy

Toby

(2016). Effectiveness of Internet-Based Reading Apprenticeship Improving Science Education (iRAISE). Empirical Education Inc.

46.

Jacob

R. T.

Doolittle

Kemple

Somers

M. A.

(2019). A framework for learning from null results. Educational Researcher, 48(9), 580–589. https://doi.org/10.3102/0013189X19891955

47.

Jayanthi

Gersten

Taylor

M. J.

Smolkowski

Dimino

(2017). Impact of the Developing Mathematical Ideas professional development program on grade 4 students’ and teachers’ understanding of fractions (REL 2017-256). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast.

48.

Jerrim

Austerberry

Crisan

Ingold

Morgan

Pratt

Smith

Wiggins

(2015). Mathematics mastery: Secondary evaluation report. Education Endowment Foundation.

49.

Kim

J. S.

(2019). Making every student count: Learning from replication failure to improve intervention research. Educational Researcher, 48(9), 599–607. https://doi.org/10.3102/0013189X19891428

50.

Kuijpers

C. C. H.

Fronczek

van de Goot

F. R. W.

Niessen

H. W. M.

van Diest

P. J.

Jiwa

(2014). The value of autopsies in an era of high-tech medicine: Discrepant findings persist. Journal of Clinical Pathology, 67(6), 512–519. https://doi.org/10.1136/jclinpath-2013-202122

51.

Lakens

Scheel

A. M.

Isager

P. M.

(2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. https://doi.org/10.1177/2515245918770963

52.

Lankford

Loeb

Wyckoff

(2002). Teacher sorting and the plight of urban schools: A descriptive analysis. Educational Evaluation and Policy Analysis, 24(1), 37–62. https://doi.org/10.3102/01623737024001037

53.

Lavrakas

P. J.

Traugott

M. W.

Kennedy

Holbrook

A. L.

de Leeuw

E. D.

West

B. T.

(Eds.). (2019). Experimental methods in survey research: Techniques that combine random sampling with random assignment. John Wiley.

54.

Levin

Quinn

(2003). Missing opportunities: How we keep high-quality teachers out of urban classrooms. The New Teacher Project.

55.

Lindsay

Davis

Stephan

Proger

(2017). Impacts of Ramp-Up to Readiness after one year of implementation (REL 2017-241). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest.

56.

Louisiana Believes. (2019a). Instructional materials evaluation—Student standards review: enVision Math Common Core. https://www.louisianabelieves.com/docs/default-source/curricular-resources/pearson-envision-math-grades-k-6.pdf?sfvrsn=ac0a831f_10

57.

Louisiana Believes. (2019b). Guidance for instructional materials review. https://www.louisianabelieves.com/docs/default-source/curricular-resources/guidance-for-textbooks-and-instructional-materials-reviews.pdf?sfvrsn=2

58.

Maas

Lake

(2015, January). A blueprint for effective and adaptable school district procurement. Center for Reinventing Public Education. https://files.eric.ed.gov/fulltext/ED558568.pdf

59.

Martin

McBride

Brims

Doubell

Pote

Clarke

(2018, February 22). Evaluating early intervention programmes: Six common pitfalls, and how to avoid them. https://www.eif.org.uk/resource/evaluating-early-intervention-programmes-six-common-pitfalls-and-how-to-avoid-them

60.

Maynard

(2006). Presidential address: Evidence-based decision making: What will it take for the decision makers to care? Journal of Policy Analysis and Management, 25(2), 249–265. https://doi.org/10.1002/pam.20169

61.

McLaughlin

M. W.

Mitra

(2001). Theory-based change and change-based theory: Going deeper, going broader. Journal of Educational Change, 2(4), 301–323. https://doi.org/10.1023/A:1014616908334

62.

Ming

N. C.

Goldenberg

L. B.

(2021). Research worth using: (Re)framing research evidence quality for educational policymaking and practice. Review of Research in Education, 45(1), 129–169. https://doi.org/10.3102/0091732X21990620

63.

Modarres

(1993). What every engineer should know about reliability and risk analysis. Center for Reliability Engineering.

64.

Murphy

Weinhardt

Wyness

(2018). Who teaches the teachers? An RCT of peer-to-peer observation and feedback in 181 schools (CEP Discussion Paper No. 1565). Centre for Economic Performance.

65.

National Implementation Research Network. (2016). Active implementation practice and science. https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-Briefs-1-ActiveImplementationPracticeAndScience-10-05-2016.pdf

66.

Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(Eds.). (2015). Handbook of practical program evaluation (4th ed.). Jossey Bass/Wiley.

67.

Pane

J. F.

McCaffrey

D. F.

Slaughter

M. E.

Steele

J. L.

Ikemoto

G. S.

(2010). An experiment to evaluate the efficacy of cognitive tutor geometry. Journal of Research on Educational Effectiveness, 3(3), 254–281. https://doi.org/10.1080/19345741003681189

68.

Papay

J. P.

Bacher-Hicks

Page

L. C.

Marinell

W. H.

(2017). The challenge of teacher retention in urban schools: Evidence of variation from a cross-site analysis. Educational Researcher, 46(8), 434–448. https://doi.org/10.3102/0013189X17735812

69.

Papay

J. P.

Kraft

M. A.

(2016). The productivity costs of inefficient hiring practices: Evidence from late teacher hiring. Journal of Policy Analysis and Management, 35(4), 791–817.

70.

Petroski

(1992). To engineer is human. Vintage Books.

71.

Pigott

T. D.

Valentine

J. C.

Polanin

J. R.

Williams

R. T.

Canada

D. D.

(2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. http://dx.doi.org/10.3102/0013189X13507104

72.

Rayes

Lee

S. S.

Boruch

(2016). Ambient positional instability among teachers in Minnesota public schools: 2010–2011 to 2014–2015. http://repository.upenn.edu/gse_pubs/400

73.

Resendez

Azin

Strobel

(2009). A study on the effects of Pearson’s 2009 enVisionMATH program: Final summative report. PRES Associates.

74.

Rickles

Zeiser

West

(2018). Accounting for student attrition in power calculations: Benchmarks and guidance. Journal of Research on Educational Effectiveness, 11(4), 622–644. https://doi.org/10.1080/19345747.2018.1502384

75.

Rosenthal

(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

76.

Roy

Styles

Walker

Morrison

Nelson

Kettlewell

(2018). Best practice in grouping student intervention A: Best practice in setting. Evaluation report and Executive Summary. Education Endowment Foundation.

77.

Schweppenstedde

Reid

(2015, September 18). How failure can feed success: Using evidence on “what does not work” to improve services and external recognition [Blog post]. https://www.rand.org/blog/2015/09/how-failure-can-feed-success-using-evidence-on-what.html

78.

Shojania

K. G.

Burton

E. C.

(2008). The vanishing nonforensic autopsy. New England Journal of Medicine, 358, 873–875. https://doi.org/10.1056/NEJMp0707996

79.

Snook

Marshall

J. M.

Newman

R. M.

(2003, January). Physics of failure as an integrated part of design for reliability. Paper presented at the IEEE Proceedings Annual Reliability and Maintainability Symposium; Tampa, FL. https://doi.org/10.1109/RAMS.2003.1181901

80.

Song

Herman

(2010). Critical issues and common pitfalls in designing and conducting impact studies in education. Educational Evaluation and Policy Analysis, 32(3), 351–371. https://doi.org/10.3102/0162373710373389

81.

Spybrook

(2014). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. Journal of Experimental Education, 82(3), 334–357. https://doi.org/10.1080/00220973.2013.813364

82.

Spybrook

Raudenbush

S.W.

(2009). An examination of the precision and technical accuracy of the first wave of group randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318. https://doi.org/10.3102/0162373709339524

83.

Spybrook

Zhang

Kelcey

Dong

(2020). Learning from cluster randomized trials in education: An assessment of the capacity of studies to determine what works, for whom and under what conditions. Educational Evaluation and Policy Analysis, 42(3), 354–374. https://doi.org/10.3102/0162373720929018

84.

Stecher

B. M.

Holtzman

D. J.

Garet

M. S.

Hamilton

L. S.

Engberg

Steiner

E. D.

Robyn

Baird

M. D.

Gutierrez

I. A.

Peet

E. D.

Brodziak de los Reyes

Fronberg

Weinberger

Hunter

G. P.

Chambers

(2019). Intensive partnerships for effective teaching enhanced how teachers are evaluated but had little effect on student outcomes. RAND Corporation. https://www.rand.org/pubs/research_briefs/RB10009-1.html

85.

Strauss

(2018, June 29). Bill Gates spent hundreds of millions of dollars to improve teaching. New report says it was a bust. The Washington Post. https://beta.washingtonpost.com/news/answer-sheet/wp/2018/06/29/bill-gates-spent-hundreds-of-millions-of-dollars-to-improve-teaching-new-report-says-it-was-a-bust/

86.

Styles

Clarkson

Fowler

(2014). Chatterbooks: Evaluation report and executive summary. Education Endowment Foundation.

87.

Taylor

J. A.

West

(2020). Estimating teacher attrition for impact study design. Educational Researcher, 49(1), 68–70. https://doi.org/10.3102/0013189X19880550

88.

U.S. Chamber of Commerce. (2007). Leaders and laggards: A state-by-state report card on educational effectiveness. https://www.uschamberfoundation.org/leaders-and-laggards/app/docs/2007.pdf

89.

USDOE Office of Inspector General. (2018a, November 27). Calculating and reporting graduation rates in Utah (ED-OIG/A06R0004). https://www2.ed.gov/about/offices/list/oig/auditreports/fy2019/a06r0004.pdf

90.

USDOE Office of Inspector General. (2018b, January 11). Calculating and reporting graduation rates in California (ED-OIG/A02Q0005). https://www2.ed.gov/about/offices/list/oig/auditreports/fy2018/a02q0005.pdf

91.

Vaden-Kiernan

Borman

Caverly

Bell

Sullivan

Ruiz de Castilla

Fleming

D. R.

Henry

Long

Hughes Jones

(2018). Findings from a multiyear scale-up effectiveness trial of Open Court Reading. Journal of Research on Educational Effectiveness, 11(1), 109–132. https://doi.org/10.1080/19345747.2017.1342886

92.

Valentine

(2019). Expecting and learning from null results. Educational Researcher, 48(9), 611–613. https://doi.org/10.3102/0013189X19891440

93.

Weiss

(2002). What to do until the random assigner comes. In: Mosteller

Boruch

(Eds.), Evidence matters: Randomized trials in education research (pp. 198–233). Brookings Institution.

94.

Weiss

C. H.

(1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41–55. https://doi.org/10.1002/ev.1086

95.

Welsh

(2021). Assessing the quality of education research through its relevance to practice: An integrative review of research-practice partnerships. Review of Research in Education, 45(1), 170–194. https://doi.org/10.3102/0091732X20985082

96.

West

Ainscow

Wigelsworth

Troncoso

(2017). Challenge the gap: Evaluation report and executive summary. Education Endowment Foundation.

97.

What Works Clearinghouse. (2015). WWC Standards Brief: Attrition standard. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_brief_attrition_080715.pdf

98.

Wigelsworth

Squires

Birchinall

Kalambouka

Lendrum

Black

Troncoso

Santos

Ashworth

Britteon

(2018). Friends for life: Evaluation report and executive summary. Education Endowment Foundation.

99.

Wiggins

Parrao

C. G.

Austerberry

Ingold

(2017). Foreign language learning in primary school. Education Endowment Foundation.

100.

Wiggins

Sawtell

Jerrim

(2017). Learner response system evaluation. Education Endowment Foundation.

101.

Wilkins

Gersten

Decker

L. E.

Grunden

Brasiel

Brunnert

Jayanthi

(2012). Does a summer reading program based on Lexiles affect reading comprehension? (NCEE 2012-4006). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

102.

Williams

K. R.

Mattson

S. A.

(2006). Qualitative lessons from a community-based violence prevention project with null findings. New Directions for Evaluation, 2006(110), 5–17. https://doi.org/10.1002/ev.183

103.

Worth

Sizmur

Walker

Bradshaw

Styles

(2017). Teacher observation: Evaluation report and executive summary. Education Endowment Foundation.

104.

Frisone

Hooks

Boruch

(2016). Ambient positional instability in New Jersey public schools: 1996–1997 to 2011–2012. https://repository.upenn.edu/cgi/viewcontent.cgi?article=1401&context=gse_pubs

105.

Younie

(2006). Implementing government policy on ICT in education: Lessons learnt. Education Information Technology, 11, 385–400. https://doi.org/10.1007/s10639-006-9017-1

106.

Zinth

(2005). State textbook adoption. Education Commission of the States.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB

Toward a Science of Failure Analysis: A Narrative Review

Abstract

Get full access to this article

References

Supplementary Material