Randomized experiments are a strong design for establishing impact evidence because the random assignment mechanism theoretically allows confidence in attributing group differences to the intervention. Growth of randomized experiments within educational studies has been widely documented. However, randomized experiments within education have received criticism for implementation challenges and for ignoring context. Additionally, limited guidance exists for programs that are tasked with both implementation and evaluation within the same funding period. This study draws on a research team's experiences and examines opportunities and challenges in conducting a multisite randomized evaluation of an internship program for teacher candidates. We discuss how problems were collaboratively addressed and adjusted to align with local realities and demonstrate how the research team, in consultation with local stakeholders, addressed methodological and program implementation problems in the field. Recommendations for future research are provided.
BellS. H.PeckL. R. (2016). On the feasibility of extending social experiments to wider applications. Journal of MultiDisciplinary Evaluation, 12(27), 93–112. https://doi.org/10.56645/jmde.v12i27.452
3.
BellgA. J.BorrelliB.ResnickB.HechtJ.MinicucciD. S.OryM.OgedegbeG.OrwigD.ErnstD.CzajkowskiS. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23(5), 443–451. https://doi.org/10.1037/0278-6133.23.5.443
BloomH. S.BloomS. E. P. (1981). Household participation in the section 8 existing housing program. Evaluation Review, 5(3), 325–340. https://doi.org/10.1177/0193841X8100500
6.
BonellC.FletcherA.MortonM.LorencT.MooreL. (2012). Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine75(12), 2299–2306. https://doi.org/10.1016/j.socscimed.2012.08.032
7.
BridgesD.SmeyersP.SmithR. (Eds.). (2009). Evidence-based education policy: What evidence? What basics? What policy?. Oxford, UK: Wiley-Blackwell.
8.
CampbellM.FitzpatrickR.HainesA.KinmonthA. L.SandercockP.SpiegelhalterD.TyrerP. (2000). Framework for design and evaluation of complex interventions to improve health. British Medical Journal, 321(7262), 694–696. https://doi.org/10.1136/bmj.321.7262.694
9.
ChatterjiM. (2007). Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. American Journal of Evaluation, 28(3), 239–255. https://doi.org/10.1177/1098214007304884
10.
ConlonE. L.BraunE. J.BabbittE. M.CherneyL. R. (2020). Treatment fidelity procedures for an aphasia intervention within a randomized controlled trial: Design, feasibility, and results. American Journal of Speech-Language Pathology, 29 (1S), 412–424. https://doi.org/10.1044/2019_AJSLP-CAC48-18-0227
11.
ConnollyP.BiggartS.MillerL.O'HareL.ThurstonA. (2017). Using randomized controlled trials in education. Sage.
12.
ConnollyP.KeenanC.UrbanskaK. (2018). The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980-2016. Educational Research, 60(3), 276–291. https://doi.org/10.1080/00131881.2018.1493353
13.
CookT. D.ScrivenM.CorynC. L. S.EvergreenS. D. H. (2010). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 3(11), 105–117. https://doi.org/10.1177/1098214009354918
14.
CraigP.DieppeP.MacintyreS.MichieS.NazarethI.PetticrewM. (2008). Developing and evaluating complex interventions: The new medical research council guidance. BMJ, 337(7676), 979–983. https://doi.org/10.1136/bmj.a1655
15.
DattaJ.PetticrewM. (2013). Challenges to evaluating complex interventions: A content analysis of published papers. BMC Public Health, 13(1), 568–568. https://doi.org/10.1186/1471-2458-13-568
16.
DawsonA.YeomansE.BrownE. R. (2018). Methodological challenges in education RCTs: Reflections from England's education endowment foundation. Educational Research, 60(3), 292–310. https://doi.org/10.1080/00131881.2018.1500079
17.
DembyH.JennerL.GregoryA.JennerE. (2020). Structuring evaluation partnerships: Exploring contrasts in researcher-practitioner roles and responsibilities when implementing randomized experiments in real-world settings. American Journal of Evaluation, 41(4), 531–546. https://doi.org/10.1177/1098214020941089
18.
DixonJ.BiehalN.GreenJ.SinclairI.KayC.ParryE. (2014). Trials and tribulations: Challenges and prospects for randomised controlled trials of social work with children. The British Journal of Social Work, 44(6), 1563–1581. https://doi.org/10.1093/bjsw/bct035
19.
DonovanJ.MillsN.SmithM.BrindleL.JacobyA.PetersT.FrankelS.NealD.HamdyF. (2002). Quality improvement report: Improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. British Medical Journal, 325(7367), 766–769. https://doi.org/10.1136/bmj.325.7367.766
20.
EricksonF.GutierrezK. (2002). Comment: Culture, rigor, and science in educational research. Educational Researcher, 31(8), 21–24. https://doi.org/10.3102/0013189X031008021
21.
FeserD. (2023). Innovation intermediaries revised: A systematic literature review on innovation intermediaries’ role for knowledge sharing. Review of Managerial Science, 17(5), 1827–1862. https://doi.org/10.1007/s11846-022-00593-x
22.
FlayB. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15(5), 451–474. https://doi.org/10.1016/0091-7435(86)90024-1
23.
HammersleyM. (2007). Educational research and evidence-based practice. Sage.
24.
Institute of Education Sciences, U.S. Department of Education, & National Science Foundation. (2013). Common guidelines for education research and development. https://ies.ed.gov/pdf/CommonGuidelines.pdf
25.
JacksonM. (2007). Systems approaches to management. Kluwer Academic/Plenum Publishers.
26.
JacobR. T.DoolittleF.KempleJ.SomersM.-A. (2019). A framework for learning from null results. Educational Researcher, 48(9), 580–589. https://doi.org/10.3102/0013189X19891955
KaplanA.CromleyJ.PerezT.DaiT.MaraK.BalsaiM. (2020). The role of context in educational RCT findings: A call to redefine “evidence-based practice”. Educational Researcher, 49(4), 285–288. https://doi.org/10.3102/0013189X20921862
29.
KimJ. S. (2019). Making every study count: Learning from replication failure to improve intervention research. Educational Researcher, 48(9), 599–607. https://doi.org/10.3102/0013189X19891428
30.
LawtonJ.JenkinsN.DarbyshireJ. L.HolmanR. R.FarmerA. J.HallowellN. (2011). Challenges of maintaining research protocol fidelity in a clinical care setting: A qualitative study of the experiences and views of patients and staff participating in a randomized controlled trial. Trials, 12(108), 1–10. https://doi.org/10.1186/1745-6215-12-108
31.
MacintyreS.PetticrewM. (2000). Good intentions and received wisdom are not enough. Journal of Epidemiology and Community Health, 54(11), 802–803. https://doi.org/10.1136/jech.54.11.802
MichieS.YardleyL.WestR.PatrickK.GreavesF. (2017). Developing and evaluating digital interventions to promote behavior change in health and health care: Recommendations resulting from an international workshop. Journal of Medical Internet Research, 19(6), e232–e232. https://doi.org/10.2196/jmir.7126
35.
Miller-YoungJ.PothC. N. (2022). Complexifying’ our approach to evaluating educational development outcomes: Bridging theoretical innovations with frontline practice. International Journal for Academic Development, 27(4), 386–399. https://doi.org/10.1080/1360144X.2021.1887876
36.
MooreL.Moore GrahamF. (2011). Public health evaluation: Which designs work, for whom and under what circumstances?Journal of Epidemiology and Community Health65(7), 596–597. https://doi.org/10.1136/jech.2009.093211
37.
National Research Council (2002). Scientific research in education. National Academy Press.
38.
OdomS. L. (2021). Education of students with disabilities, science, and randomized controlled trials. Research and Practice for Persons with Severe Disabilities, 46, 132–145. https://doi.org/10.1177/15407969211032341
39.
O'DonohueW. T.MasudaA.LilienfeldS. (2022). Avoiding questionable research practices in applied psychology (1st ed.). Springer International Publishing. https://doi.org/10.1007/978-3-031-04968-2
40.
PhillipsD. C. (2006). A guide for the perplexed: Scientific educational research, methodolatry, and the gold versus platinum standards. Educational Research Review, 1, 15–26. https://doi.org/10.1016/j.edurev.2006.01.003
41.
RossiP. H.LipseyM. W.HenryG. T. (2018). Evaluation: A systematic approach. Sage.
42.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701. https://doi.org/10.1037/h0037350
43.
ScrivenM. (2008). A summative evaluation of RCT methodology and an alterative approach to causal research. Journal of Multiisciplinary Evaluation, 5(9), 11–24. https://doi.org/10.56645/jmde.v5i9.160
44.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houston Mifflin.
45.
SiddiquiN.GorardS.SeeB. H. (2018). The importance of process evaluation for randomised control trials in education. Educational Research, 60(3), 357–370. https://doi.org/10.1080/00131881.2018.1493349
46.
SimonN. S.JohnsonS. M. (2015). Teacher turnover in high-poverty schools: What we know and can do. Teachers College Record, 117(3), 1–36. https://doi.org/10.1177/0161468115117003
47.
SutcherL.Darling-HammondL.Carver-ThomasD. (2016). A coming crisis in teaching? Teacher supply, demand, and shortages in the U.S.https://doi.org/10.54300/247.242
TothA. A.BanksG. C.MellorD.O’BoyleE. H.DicksonA.Davis D. J.DeHavenA.BochantinJ.BornsJ. (2021). Study preregistration: An evaluation of a method for transparent reporting. Journal of Business and Psychology, 36(4), 553–571. https://doi.org/10.1007/s10869-020-09695-3
YeatonW. H.SechrestL. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 156–167. https://doi.org/10.1037/0022-006X.49.2.156
