Where Community-Engaged Research and Randomized Controlled Trials Align: Lessons from Implementation of an Experimental Evaluation in Partnership with a Community-Based Afterschool Program

Abstract

There has been increased focus within the evaluation field on the value of community-engaged research (CER) methods; however, CER is not easily seen as compatible with experimental evaluation methods (or randomized controlled trial, RCT). For instance, in an experimental design, researchers leverage randomization to create a counterfactual; this can inhibit flexibility in implementation often connected to CER practice. This article describes the central tenets of both CER and randomized evaluations, highlighting tensions between principles of each. It describes a community-partnered RCT designed to evaluate the impact of an afterschool youth music program. We outline our process of designing and implementing the study in partnership with a community-based organization, including successes and challenges. We identify key lessons and make recommendations for evaluators who seek to engage in similar work.

Keywords

community-based research community-based evaluation randomized controlled trial experimental evaluation university-community partnership

Get full access to this article

View all access options for this article.

References

Alemán

Duryea

Guerra

N. G.

McEwan

P. J.

Muñoz

Stampini

Williamson

A. A.

(2017). The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science, 18(7), 865–878. https://doi.org/10.1007/s11121-016-0727-3

Allen-Platt

Gerstner

C. C.

Boruch

Ruby

(2021). Toward a science of failure analysis: A narrative review. Review of Research in Education, 45(1), 223–252. https://doi.org/10.3102/0091732X20985074

Altman

D. G.

(1995). Sustaining interventions in community systems: On the relationship between researchers and communities. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 14(6), 526–536. https://doi.org/10.1037//0278-6133.14.6.526

American Evaluation Association. (n.d.). American Evaluation Association Guiding Principles. https://www.eval.org/Portals/0/AEA_289398-18_GuidingPrinciples_Brochure_2.pdf

American Evaluation Association. (2018). The 2018 AEA Evaluator Competencies . https://www.eval.org/Portals/0/Docs/AEA%20Evaluator%20Competencies.pdf

Ammerman

Corbie-Smith

St. George

D. M. M.

Washington

Weathers

Jackson-Christian

(2003). Research expectations among African American church leaders in the praise! project: A randomized trial guided by community-based participatory research. American Journal of Public Health, 93(10), 1720–1727. https://doi.org/10.2105/AJPH.93.10.1720

Baldassarri

Abascal

(2017). Field experiments across the social sciences. Annual Review of Sociology, 43(1), 41–73. https://doi.org/10.1146/annurev-soc-073014-112445

Bell

S. H.

Peck

L. R.

(2016a). On the “how” of social experiments: Experimental designs for getting inside the black box. New Directions for Evaluation, 2016(152), 97–107. https://doi.org/10.1002/ev.20210

Bell

S. H.

Peck

L. R.

(2016b). On the feasibility of extending social experiments to wider applications. Journal of MultiDisciplinary Evaluation, 12(27), 93–112. https://doi.org/10.56645/jmde.v12i27.452

10.

Bickman

Reich

S. M

. (2009). Randomized controlled trials: A gold standard with feet of clay? In Donaldson

S. I.

Christie

Mark

M. M.

(Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 51–77). Sage Publications. https://doi.org/10.4135/9781412995634.d10

11.

Bromley

Mikesell

Jones

Khodyakov

(2015). From subject to participant: Ethics and the evolving role of community in health research. American Journal of Public Health, 105(5), 900–908. https://doi.org/10.2105/AJPH.2014.302403

12.

Brown

L. D.

Tandon

(1983). Ideology and political economy in inquiry: Action research and participatory research. The Journal of Applied Behavioral Science, 19(3), 277–294. https://doi.org/10.1177/002188638301900306

13.

Cartwright

(2007). Are RCTs the gold standard? BioSocieties, 2(1), 11–20. https://doi.org/10.1017/S1745855207005029

14.

Chung

Jones

Dixon

E. L.

Miranda

Wells

(2010). Using a community partnered participatory research approach to implement a randomized controlled trial: Planning the design of community partners in care. Journal of Health Care for the Poor and Underserved, 21(3), 780–795. https://doi.org/10.1353/hpu.0.0345

15.

Cipriano

Strambler

M. J.

Naples

L. H.

Kirk

Wood

, … Durlak

(2023). The state of evidence for social and emotional learning: A contemporary meta-analysis of universal school-based SEL interventions. Child Development, 94(5), 1181–1204. https://doi.org/10.1111/cdev.13968

16.

Clinical and Translational Science Awards Consortium. (2011). Principles of Community Engagement, Second Edition. (NIH Publication No. 11-7782). Washington, DC: U.S. Government Printing Office. https://stacks.cdc.gov/view/cdc/11699

17.

Collins

S. E

., Clifasefi

S. L

., Stanton

., The LEAP Advisory Board, Straits

K. J. E

., Gil-Kashiwabara

., Rodriguez Espinosa

., … .& Wallerstein

. (2018). Community-based participatory research (CBPR): Towards equitable involvement of community in psychology research. American Psychologist, 73(7), 884–898. https://doi.org/10.1037/amp0000167

18.

Deaton

Cartwright

(2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005

19.

Demby

Jenner

Gregory

Jenner

(2020). Structuring evaluation partnerships: Exploring contrasts in researcher–practitioner roles and responsibilities when implementing randomized experiments in real-world settings. American Journal of Evaluation, 41(4), 531–546. https://doi.org/10.1177/1098214020941089

20.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x

21.

Evans

D. K.

(2023). Towards improved and more transparent ethics in randomized controlled trials in development social science. Journal of Development Effectiveness, 16(3), 1–11. https://doi.org/10.1080/19439342.2023.2196978

22.

Finn

J. D.

Achilles

C. M.

(1990). Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27(3), 557–577. https://doi.org/10.2307/1162936

23.

Gelman

(2018). Benefits and limitations of randomized controlled trials: A commentary on Deaton and Cartwright. Social Science & Medicine, 210, 48–49. https://doi.org/10.1016/j.socscimed.2018.04.034

24.

Goodkind

J. R.

Amer

Christian

Hess

J. M.

Bybee

Isakson

B. L.

, … Shantzek

(2017). Challenges and innovations in a community-based participatory randomized controlled trial. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 44(1), 123–130. https://doi.org/10.1177/1090198116639243

25.

Goodman

L. A.

Epstein

Sullivan

C. M.

(2018). Beyond the RCT: Integrating rigor and relevance to evaluate the outcomes of domestic violence programs. American Journal of Evaluation, 39(1), 58–70. https://doi.org/10.1177/1098214017721008

26.

Hamilton

Freedman

Gennetian

Michalopoulos

Walter

Adams-Ciardullo

., & Ahluwalia,

. (2001). How effective are different welfare-to-work approaches? Five-year adult and child impacts for eleven programs. U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families, and U.S. Department of Education; Washington, DC: 2001. https://aspe.hhs.gov/reports/how-effective-are-different-welfare-work-approaches-five-year-adult-child-impacts-eleven-programs-0

27.

Hansen

H. F.

(2005). Choosing evaluation models: A discussion on evaluation design. Evaluation, 11(4), 447–462. https://doi.org/10.1177/1356389005060265

28.

Hariton

Locascio

J. J.

(2018). Randomised controlled trials—the gold standard for effectiveness research. BJOG : An International Journal of Obstetrics and Gynaecology, 125(13), 1716. https://doi.org/10.1111/1471-0528.15199

29.

Heckman

J. J.

(1991). Randomization and Social Policy Evaluation Revisited (Working Paper No. 107). National Bureau of Economic Research. https://doi.org/10.3386/t0107

30.

Hennessy

S. L.

Sachs

M. E.

Ilari

Habibi

(2019). Effects of music training on inhibitory control and associated neural networks in school-aged children: A longitudinal study. Frontiers in Neuroscience, 13, 1–16. https://doi.org/10.3389/fnins.2019.01080

31.

Hesse-Biber

(2012). Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qualitative Inquiry, 18(10), 876–889. https://doi.org/10.1177/1077800412456964

32.

Holochwost

Propper

Wolf

Willoughby

Fisher

Kolacz

, … Jaffee

(2017). Music education, academic achievement, and executive functions. Psychology of Aesthetics, Creativity, and the Arts, 11(2), 147–166. https://doi.org/10.1037/aca0000112

33.

Israel

B. A.

Schulz

A. J.

Parker

E. A.

Becker

A. B.

(1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19(1), 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173

34.

Jaciw

A. P.

Lin

Schellinger

(2018). Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement. Methodological Innovations, 11(2), 2059799118787751. https://doi.org/10.1177/2059799118787751

35.

Jagosh

Bush

P. L.

Salsberg

Macaulay

A. C.

Greenhalgh

Wong

, … Pluye

(2015). A realist evaluation of community-based participatory research: Partnership synergy, trust building and related ripple effects. BMC Public Health, 15(1), 725. https://doi.org/10.1186/s12889-015-1949-1

36.

John Snow, Inc. (2012). Engaging your community: a Toolkit for partnership, collaboration, and action. https://publications.jsi.com/JSIInternet/Inc/Common/_download_pub.cfm?id=14333&lid=3

37.

Jones

Koegel

Wells

K. B.

(2008). Bringing experimental design to community-partnered participatory research. In Minkler

Wallerstein

(Eds.), Community-Based participatory research for health: From process to outcomes (2nd ed, pp. 67–85). Jossey-Bass.

38.

Kerstetter

(2012). Insider, outsider, or somewhere between: The impact of researchers’ identities on the community-based research process. Journal of Rural Social Sciences, 27(2), 99–117. https://egrove.olemiss.edu/jrss/vol27/iss2/7

39.

Kraemer Diaz

A. E.

Spears Johnson

C. R.

& Arcury

T. A.

(2013). Variation in the interpretation of scientific integrity in community-based participatory health research. Social Science & Medicine (1982), 97, 134–142. https://doi.org/10.1016/j.socscimed.2013.08.023

40.

Leykum

L. K.

Pugh

J. A.

Lanham

H. J.

Harmon

McDaniel

R. R.

(2009). Implementation research design: Integrating participatory action research into randomized controlled trials. Implementation Science, 4(1), 69. https://doi.org/10.1186/1748-5908-4-69

41.

Majno

(2012). From the model of El Sistema in Venezuela to current applications: Learning and integration through collective music education. Annals of the New York Academy of Sciences, 1252(1), 56–64. https://doi.org/10.1111/j.1749-6632.2012.06498

42.

McNeil

C. B.

Capage

L. C.

Bahl

Blanc

(1999). Importance of early intervention for disruptive behavior problems: Comparison of treatment and waitlist-control groups. Early Education and Development, 10(4), 445–454. https://doi.org/10.1207/s15566935eed1004_2

43.

Michener

Cook

Ahmed

S. M.

Yonas

M. A.

Coyne-Beasley

Aguilar-Gaxiola

(2012). Aligning the goals of community-engaged research: Why and how academic health centers can successfully engage with communities to improve health. Academic Medicine, 87(3), 285–291. https://doi.org/10.1097/ACM.0b013e3182441680

44.

Mikesell

Bromley

Khodyakov

(2013). Ethical community-engaged research: A literature review. American Journal of Public Health, 103(12), e7–e14. https://doi.org/10.2105/AJPH.2013.301605

45.

Minkler

(2004). Ethical challenges for the “outside” researcher in community-based participatory research. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 31(6), 684–697. https://doi.org/10.1177/1090198104269566

46.

Minkler

(2010). Linking science and policy through community-based participatory research to study and address health disparities. American Journal of Public Health, 100(Suppl 1), S81–S87. https://doi.org/10.2105/AJPH.2009.165720

47.

Miranda

. (2024). What is a RACI chart? How this project management tool can boost your productivity. Forbes Advisor, Business. https://www.forbes.com/advisor/business/raci-chart/

48.

Muhammad

Wallerstein

Sussman

A. L.

Avila

Belone

Duran

(2015). Reflections on researcher identity and power: The impact of positionality on community based participatory research (CBPR) processes and outcomes. Critical Sociology, 41(7–8), 1045–1063. https://doi.org/10.1177/0896920513516025

49.

O'Cathain

. (2018). A practical guide to using qualitative research with randomized controlled trials. Oxford University Press. https://doi.org/10.1093/med/9780198802082.001.0001

50.

Palmer-Wackerly

A. L.

Reyes

M. S.

Ali

S. H.

Gocchi Carrasco

Habecker

Houska

, … Dombrowski

(2021). Examining and evaluating multilevel communication within a mixed-methods, community-based participatory research project in a rural, minority-majority U.S. Town. Journal of Applied Communication Research, 49(2), 129–147. https://doi.org/10.1080/00909882.2020.1851042

51.

Peck

L. R.

(2020). Leveraging experimental evaluations for understanding causal mechanisms. New Directions for Evaluation, 2020(167), 145–160. https://doi.org/10.1002/ev.20422

52.

Pleger

Sager

(2018). Betterment, undermining, support and distortion: A heuristic model for the analysis of pressure on evaluators. Evaluation and Program Planning, 69, 166–172. https://doi.org/10.1016/j.evalprogplan.2016.09.002

53.

Querstret

Cropley

Fife-Schaw

(2018). The effects of an online mindfulness intervention on perceived stress, depression and anxiety in a non-clinical sample: A randomised waitlist control trial. Mindfulness, 9(6), 1825–1836. https://doi.org/10.1007/s12671-018-0925-0

54.

Sancassiani

Pintus

Holte

Paulus

Moro

M. F.

Cossu

, … Lindert

(2015). Enhancing the emotional and social skills of the youth to promote their wellbeing and positive development: A systematic review of universal school-based randomized controlled trials. Clinical Practice and Epidemiology in Mental Health : CP & EMH, 11(Suppl 1 M2), 21–40. https://doi.org/10.2174/1745017901511010021

55.

Schmidli

F. H.

Pleger

L. E.

Hadorn

(2023). Don’t you forget about me: Independence of evaluations from the perspective of US evaluation clients—An exploratory study. Evaluation, 29(1), 110–132. https://doi.org/10.1177/13563890221138876

56.

Scriven

(2008). A summative evaluation of RCT methodology: An alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11–24. https://doi.org/10.56645/jmde.v5i9.160

57.

Shalowitz

M. U.

Isacco

Barquin

Clark-Kauffman

Delger

Nelson

, … Wagenaar

K. A.

(2009). Community-based participatory research: A review of the literature with strategies for community engagement. Journal of Developmental & Behavioral Pediatrics, 30(4), 350–361. https://doi.org/10.1097/DBP.0b013e3181b0ef14

58.

Small

S. A.

(1995). Action-oriented research: Models and methods. Journal of Marriage and the Family, 57(4), 941–953. https://doi.org/10.2307/353414

59.

Small

S. A.

Uttal

(2005). Action-oriented research: Strategies for engaged scholarship. Journal of Marriage and Family, 67(4), 936–948. https://doi.org/10.1111/j.1741-3737.2005.00185.x

60.

Sobol-Goldberg

Rabinowitz

Gross

(2013). School-based obesity prevention programs: A meta-analysis of randomized controlled trials. Obesity, 21(12), 2422–2428. https://doi.org/10.1002/oby.20515

61.

Sperling

Lee

V. K.

Schmid

Gray

Kim

J. Y.

(2023). A randomized study of El Sistema-inspired youth music education programming: Evidence addressing executive function and social–emotional learning. Psychology of Aesthetics, Creativity, and the Arts. Advance online publication. https://doi.org/10.1037/aca0000596

62.

Stanley

(2007). Design of randomized controlled trials. Circulation, 115(9), 1164–1169. https://doi.org/10.1161/CIRCULATIONAHA.105.594945

63.

Strand

K. M.

(2003). Principles of best practice for community-based research. Michigan Journal of Community Service Learning, 9(3), 5–15. http://hdl.handle.net/2027/spo.3239521.0009.301

64.

Tedrow

V. A.

Zelaya

C. E.

Kennedy

C. E.

Morin

S. F.

Khumalo-Sakutukwa

Sweat

M. D.

Celentano

D. D.

(2012). No “magic bullet”: Exploring community mobilization strategies used in a multi-site community based randomized controlled trial: Project accept (HPTN 043). AIDS and Behavior, 16(5), 1217–1226. https://doi.org/10.1007/s10461-011-0009-9

65.

Trickett

E. J.

(2011). Community-based participatory research as worldview or instrumental strategy: Is it lost in translation(al) research? American Journal of Public Health, 101(8), 1353–1355. https://doi.org/10.2105/AJPH.2011.300124

66.

Viswanathan

Ammerman

Eng

Garlehner

Lohr

K. N.

Griffith

, … Whitener

(2004). Community-based participatory research: assessing the evidence: summary. Agency for Healthcare Research and Quality (US). https://www.ncbi.nlm.nih.gov/books/NBK11852/

67.

Wallerstein

N. B.

Duran

(2006). Using community-based participatory research to address health disparities. Health Promotion Practice, 7(3), 312–323. https://doi.org/10.1177/1524839906289376

68.

Wallerstein

Duran

(2010). Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(Suppl 1), S40–S46. https://doi.org/10.2105/AJPH.2009.184036

69.

Willis

G. B.

(2005). Cognitive interviewing. Sage Publications, Inc., https://doi.org/10.4135/9781412983655

70.

Wolff

(2000). Using randomized controlled trials to evaluate socially complex services: Problems, challenges and recommendations. The Journal of Mental Health Policy and Economics, 3(2), 97–109. https://doi.org/10.1002/1099-176x(200006)3:2<97::aid-mhp77>3.0.co;2-s

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB