Restricted accessResearch articleFirst published online 2025-9
Where Community-Engaged Research and Randomized Controlled Trials Align: Lessons from Implementation of an Experimental Evaluation in Partnership with a Community-Based Afterschool Program
There has been increased focus within the evaluation field on the value of community-engaged research (CER) methods; however, CER is not easily seen as compatible with experimental evaluation methods (or randomized controlled trial, RCT). For instance, in an experimental design, researchers leverage randomization to create a counterfactual; this can inhibit flexibility in implementation often connected to CER practice. This article describes the central tenets of both CER and randomized evaluations, highlighting tensions between principles of each. It describes a community-partnered RCT designed to evaluate the impact of an afterschool youth music program. We outline our process of designing and implementing the study in partnership with a community-based organization, including successes and challenges. We identify key lessons and make recommendations for evaluators who seek to engage in similar work.
AlemánX.DuryeaS.GuerraN. G.McEwanP. J.MuñozR.StampiniM.WilliamsonA. A. (2017). The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science, 18(7), 865–878. https://doi.org/10.1007/s11121-016-0727-3
2.
Allen-PlattC.GerstnerC. C.BoruchR.RubyA. (2021). Toward a science of failure analysis: A narrative review. Review of Research in Education, 45(1), 223–252. https://doi.org/10.3102/0091732X20985074
3.
AltmanD. G. (1995). Sustaining interventions in community systems: On the relationship between researchers and communities. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 14(6), 526–536. https://doi.org/10.1037//0278-6133.14.6.526
AmmermanA.Corbie-SmithG.St. GeorgeD. M. M.WashingtonC.WeathersB.Jackson-ChristianB. (2003). Research expectations among African American church leaders in the praise! project: A randomized trial guided by community-based participatory research. American Journal of Public Health, 93(10), 1720–1727. https://doi.org/10.2105/AJPH.93.10.1720
BellS. H.PeckL. R. (2016a). On the “how” of social experiments: Experimental designs for getting inside the black box. New Directions for Evaluation, 2016(152), 97–107. https://doi.org/10.1002/ev.20210
9.
BellS. H.PeckL. R. (2016b). On the feasibility of extending social experiments to wider applications. Journal of MultiDisciplinary Evaluation, 12(27), 93–112. https://doi.org/10.56645/jmde.v12i27.452
10.
BickmanL.ReichS. M. (2009). Randomized controlled trials: A gold standard with feet of clay? In DonaldsonS. I.ChristieC.MarkM. M. (Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 51–77). Sage Publications. https://doi.org/10.4135/9781412995634.d10
11.
BromleyE.MikesellL.JonesF.KhodyakovD. (2015). From subject to participant: Ethics and the evolving role of community in health research. American Journal of Public Health, 105(5), 900–908. https://doi.org/10.2105/AJPH.2014.302403
12.
BrownL. D.TandonR. (1983). Ideology and political economy in inquiry: Action research and participatory research. The Journal of Applied Behavioral Science, 19(3), 277–294. https://doi.org/10.1177/002188638301900306
ChungB.JonesL.DixonE. L.MirandaJ.WellsK. (2010). Using a community partnered participatory research approach to implement a randomized controlled trial: Planning the design of community partners in care. Journal of Health Care for the Poor and Underserved, 21(3), 780–795. https://doi.org/10.1353/hpu.0.0345
15.
CiprianoC.StramblerM. J.NaplesL. H.HaC.KirkM.WoodM., … DurlakJ. (2023). The state of evidence for social and emotional learning: A contemporary meta-analysis of universal school-based SEL interventions. Child Development, 94(5), 1181–1204. https://doi.org/10.1111/cdev.13968
16.
Clinical and Translational Science Awards Consortium. (2011). Principles of Community Engagement, Second Edition. (NIH Publication No. 11-7782). Washington, DC: U.S. Government Printing Office. https://stacks.cdc.gov/view/cdc/11699
17.
CollinsS. E., ClifasefiS. L., StantonJ., The LEAP Advisory Board, StraitsK. J. E., Gil-KashiwabaraE., Rodriguez EspinosaP., … .& WallersteinN. (2018). Community-based participatory research (CBPR): Towards equitable involvement of community in psychology research. American Psychologist, 73(7), 884–898. https://doi.org/10.1037/amp0000167
DembyH.JennerL.GregoryA.JennerE. (2020). Structuring evaluation partnerships: Exploring contrasts in researcher–practitioner roles and responsibilities when implementing randomized experiments in real-world settings. American Journal of Evaluation, 41(4), 531–546. https://doi.org/10.1177/1098214020941089
20.
DurlakJ. A.WeissbergR. P.DymnickiA. B.TaylorR. D.SchellingerK. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x
21.
EvansD. K. (2023). Towards improved and more transparent ethics in randomized controlled trials in development social science. Journal of Development Effectiveness, 16(3), 1–11. https://doi.org/10.1080/19439342.2023.2196978
22.
FinnJ. D.AchillesC. M. (1990). Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27(3), 557–577. https://doi.org/10.2307/1162936
23.
GelmanA. (2018). Benefits and limitations of randomized controlled trials: A commentary on Deaton and Cartwright. Social Science & Medicine, 210, 48–49. https://doi.org/10.1016/j.socscimed.2018.04.034
24.
GoodkindJ. R.AmerS.ChristianC.HessJ. M.BybeeD.IsaksonB. L., … ShantzekC. (2017). Challenges and innovations in a community-based participatory randomized controlled trial. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 44(1), 123–130. https://doi.org/10.1177/1090198116639243
25.
GoodmanL. A.EpsteinD.SullivanC. M. (2018). Beyond the RCT: Integrating rigor and relevance to evaluate the outcomes of domestic violence programs. American Journal of Evaluation, 39(1), 58–70. https://doi.org/10.1177/1098214017721008
26.
HamiltonG.FreedmanS.GennetianL.MichalopoulosC.WalterJ.Adams-CiardulloD., & Ahluwalia,S. (2001). How effective are different welfare-to-work approaches? Five-year adult and child impacts for eleven programs. U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families, and U.S. Department of Education; Washington, DC: 2001. https://aspe.hhs.gov/reports/how-effective-are-different-welfare-work-approaches-five-year-adult-child-impacts-eleven-programs-0
HaritonE.LocascioJ. J. (2018). Randomised controlled trials—the gold standard for effectiveness research. BJOG : An International Journal of Obstetrics and Gynaecology, 125(13), 1716. https://doi.org/10.1111/1471-0528.15199
29.
HeckmanJ. J. (1991). Randomization and Social Policy Evaluation Revisited(Working Paper No. 107). National Bureau of Economic Research. https://doi.org/10.3386/t0107
30.
HennessyS. L.SachsM. E.IlariB.HabibiA. (2019). Effects of music training on inhibitory control and associated neural networks in school-aged children: A longitudinal study. Frontiers in Neuroscience, 13, 1–16. https://doi.org/10.3389/fnins.2019.01080
31.
Hesse-BiberS. (2012). Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qualitative Inquiry, 18(10), 876–889. https://doi.org/10.1177/1077800412456964
32.
HolochwostS.PropperC.WolfD.WilloughbyM.FisherK.KolaczJ., … JaffeeS. (2017). Music education, academic achievement, and executive functions. Psychology of Aesthetics, Creativity, and the Arts, 11(2), 147–166. https://doi.org/10.1037/aca0000112
33.
IsraelB. A.SchulzA. J.ParkerE. A.BeckerA. B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19(1), 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173
34.
JaciwA. P.LinL.SchellingerA. (2018). Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement. Methodological Innovations, 11(2), 2059799118787751. https://doi.org/10.1177/2059799118787751
35.
JagoshJ.BushP. L.SalsbergJ.MacaulayA. C.GreenhalghT.WongG., … PluyeP. (2015). A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health, 15(1), 725. https://doi.org/10.1186/s12889-015-1949-1
JonesL.KoegelP.WellsK. B. (2008). Bringing experimental design to community-partnered participatory research. In MinklerM.WallersteinN. (Eds.), Community-Based participatory research for health: From process to outcomes (2nd ed, pp. 67–85). Jossey-Bass.
38.
KerstetterK. (2012). Insider, outsider, or somewhere between: The impact of researchers’ identities on the community-based research process. Journal of Rural Social Sciences, 27(2), 99–117. https://egrove.olemiss.edu/jrss/vol27/iss2/7
39.
Kraemer DiazA. E.Spears JohnsonC. R.& ArcuryT. A. (2013). Variation in the interpretation of scientific integrity in community-based participatory health research. Social Science & Medicine (1982), 97, 134–142. https://doi.org/10.1016/j.socscimed.2013.08.023
40.
LeykumL. K.PughJ. A.LanhamH. J.HarmonJ.McDanielR. R. (2009). Implementation research design: Integrating participatory action research into randomized controlled trials. Implementation Science, 4(1), 69. https://doi.org/10.1186/1748-5908-4-69
41.
MajnoM. (2012). From the model of El Sistema in Venezuela to current applications: Learning and integration through collective music education. Annals of the New York Academy of Sciences, 1252(1), 56–64. https://doi.org/10.1111/j.1749-6632.2012.06498
42.
McNeilC. B.CapageL. C.BahlA.BlancH. (1999). Importance of early intervention for disruptive behavior problems: Comparison of treatment and waitlist-control groups. Early Education and Development, 10(4), 445–454. https://doi.org/10.1207/s15566935eed1004_2
43.
MichenerL.CookJ.AhmedS. M.YonasM. A.Coyne-BeasleyT.Aguilar-GaxiolaS. (2012). Aligning the goals of community-engaged research: Why and how academic health centers can successfully engage with communities to improve health. Academic Medicine, 87(3), 285–291. https://doi.org/10.1097/ACM.0b013e3182441680
44.
MikesellL.BromleyE.KhodyakovD. (2013). Ethical community-engaged research: A literature review. American Journal of Public Health, 103(12), e7–e14. https://doi.org/10.2105/AJPH.2013.301605
45.
MinklerM. (2004). Ethical challenges for the “outside” researcher in community-based participatory research. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 31(6), 684–697. https://doi.org/10.1177/1090198104269566
46.
MinklerM. (2010). Linking science and policy through community-based participatory research to study and address health disparities. American Journal of Public Health, 100(Suppl 1), S81–S87. https://doi.org/10.2105/AJPH.2009.165720
MuhammadM.WallersteinN.SussmanA. L.AvilaM.BeloneL.DuranB. (2015). Reflections on researcher identity and power: The impact of positionality on community based participatory research (CBPR) processes and outcomes. Critical Sociology, 41(7–8), 1045–1063. https://doi.org/10.1177/0896920513516025
Palmer-WackerlyA. L.ReyesM. S.AliS. H.Gocchi CarrascoK.HabeckerP.HouskaK., … DombrowskiK. (2021). Examining and evaluating multilevel communication within a mixed-methods, community-based participatory research project in a rural, minority-majority U.S. Town. Journal of Applied Communication Research, 49(2), 129–147. https://doi.org/10.1080/00909882.2020.1851042
51.
PeckL. R. (2020). Leveraging experimental evaluations for understanding causal mechanisms. New Directions for Evaluation, 2020(167), 145–160. https://doi.org/10.1002/ev.20422
52.
PlegerL.SagerF. (2018). Betterment, undermining, support and distortion: A heuristic model for the analysis of pressure on evaluators. Evaluation and Program Planning, 69, 166–172. https://doi.org/10.1016/j.evalprogplan.2016.09.002
53.
QuerstretD.CropleyM.Fife-SchawC. (2018). The effects of an online mindfulness intervention on perceived stress, depression and anxiety in a non-clinical sample: A randomised waitlist control trial. Mindfulness, 9(6), 1825–1836. https://doi.org/10.1007/s12671-018-0925-0
54.
SancassianiF.PintusE.HolteA.PaulusP.MoroM. F.CossuG., … LindertJ. (2015). Enhancing the emotional and social skills of the youth to promote their wellbeing and positive development: A systematic review of universal school-based randomized controlled trials. Clinical Practice and Epidemiology in Mental Health : CP & EMH, 11(Suppl 1 M2), 21–40. https://doi.org/10.2174/1745017901511010021
55.
SchmidliF. H.PlegerL. E.HadornS. (2023). Don’t you forget about me: Independence of evaluations from the perspective of US evaluation clients—An exploratory study. Evaluation, 29(1), 110–132. https://doi.org/10.1177/13563890221138876
56.
ScrivenM. (2008). A summative evaluation of RCT methodology: An alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11–24. https://doi.org/10.56645/jmde.v5i9.160
57.
ShalowitzM. U.IsaccoA.BarquinN.Clark-KauffmanE.DelgerP.NelsonD., … WagenaarK. A. (2009). Community-based participatory research: A review of the literature with strategies for community engagement. Journal of Developmental & Behavioral Pediatrics, 30(4), 350–361. https://doi.org/10.1097/DBP.0b013e3181b0ef14
58.
SmallS. A. (1995). Action-oriented research: Models and methods. Journal of Marriage and the Family, 57(4), 941–953. https://doi.org/10.2307/353414
Sobol-GoldbergS.RabinowitzJ.GrossR. (2013). School-based obesity prevention programs: A meta-analysis of randomized controlled trials. Obesity, 21(12), 2422–2428. https://doi.org/10.1002/oby.20515
61.
SperlingJ.LeeV. K.SchmidL.GrayM.KimJ. Y. (2023). A randomized study of El Sistema-inspired youth music education programming: Evidence addressing executive function and social–emotional learning. Psychology of Aesthetics, Creativity, and the Arts. Advance online publication. https://doi.org/10.1037/aca0000596
TedrowV. A.ZelayaC. E.KennedyC. E.MorinS. F.Khumalo-SakutukwaG.SweatM. D.CelentanoD. D. (2012). No “magic bullet”: Exploring community mobilization strategies used in a multi-site community based randomized controlled trial: Project accept (HPTN 043). AIDS and Behavior, 16(5), 1217–1226. https://doi.org/10.1007/s10461-011-0009-9
65.
TrickettE. J. (2011). Community-based participatory research as worldview or instrumental strategy: Is it lost in translation(al) research?American Journal of Public Health, 101(8), 1353–1355. https://doi.org/10.2105/AJPH.2011.300124
66.
ViswanathanM.AmmermanA.EngE.GarlehnerG.LohrK. N.GriffithD., … WhitenerL. (2004). Community-based participatory research: assessing the evidence: summary. Agency for Healthcare Research and Quality (US). https://www.ncbi.nlm.nih.gov/books/NBK11852/
67.
WallersteinN. B.DuranB. (2006). Using community-based participatory research to address health disparities. Health Promotion Practice, 7(3), 312–323. https://doi.org/10.1177/1524839906289376
68.
WallersteinN.DuranB. (2010). Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(Suppl 1), S40–S46. https://doi.org/10.2105/AJPH.2009.184036
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
