Using Mixed Methods to Explore Variations in Impact Within RCTs: The Case of Project COMPASS

Abstract

Randomized controlled trials (RCTs) in education are common as the design allows for an unbiased estimate of the overall impact of a program. As more RCTs are completed, researchers are also noting that an overall average impact may mask substantial variation across sites or groups of individuals. Mixed methods can provide insight and help in unpacking some of the reasons for these variations in impact. This article contributes to the field of mixed methods research by integrating mixed methods into a recently developed conceptual framework for understanding variations in impact. We model the use of this approach within the context of an RCT for online courses that found differences in impact across courses.

Keywords

randomized controlled trials variations in impact technology online courses

Get full access to this article

View all access options for this article.

References

Arbaugh

J. B.

(2007). An empirical verification of the Community of Inquiry framework. Journal of Asynchronous Learning Networks, 11(1), 73-85. https://doi.org/10.24059/olj.v11i1.1738

Arbaugh

J. B.

Cleveland-Innes

Diaz

S. R.

Garrison

D. R.

Ice

Richardson

J. C.

Swan

K. P.

(2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. Internet and Higher Education, 11(3-4), 133-136. https://doi.org/10.1016/j.iheduc.2008.06.003

Bamberger

Rao

Woolcock

(2010). Using mixed methods in monitoring and evaluation: experiences from international development. The World Bank Development Research Group, Poverty and Inequality Team http://documents.worldbank.org/curated/en/884171468156574032/pdf/WPS5245.pdf

Bitler

M. P.

Gelbach

J. B.

Hoynes

H. W.

(2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review, 96(4), 988-1012. https://doi.org/10.1257/aer.96.4.988

Bloom

H. S.

Michalopoulos

(2010). When is the story in the subgroups? Strategies for interpreting and reporting intervention effects for subgroups. MDRC. https://www.mdrc.org/publication/when-story-subgroups

Connolly

Keenan

Urbanska

(2018). The trials of evidence-based practice in education: A Systematic review of randomised controlled trials in education research1980-2016. Educational Research, 60(3), 276-291. https://doi.org/10.1080/00131881.2018.1493353

Cragun

Pal

Vadaparampil

S. T.

Baldwin

Hampel

DeBate

R. D.

(2016). Qualitative comparative analyses: A hybrid method for identifying factors associated with program effectiveness. Journal of Mixed Methods Research, 10(3), 251-272. https://doi.org/10.1177/1558689815572023

Creswell

J. W.

Plano Clark

V. L.

Gutmann

M. L.

Hanson

W. E.

(2003). Advanced mixed methods research designs. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (pp. 209-240). Sage

de Salis

Tomlin

Toerien

Donovan

. (2008). Using qualitative research methods to improve recruitment to randomized controlled trials: The Quartet study. Journal of Health Services Research & Policy, 13(3), 92-96. https://doi.org/10.1258/jhsrp.2008.008028

10.

Donovan

Mills

Brindle

Frankel

Smith

Jacoby

Peters

Frankel

Neal

Hamdy

(2002). Improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. British Medical Journal, 325(7367), 765-768. https://doi.org/10.1136/bmj.325.7367.766

11.

Edmunds

J.A.

Gicheva

Thrift

Hull

(2021). High Tech, High Touch: The impact of an online course intervention on academic performance and persistence in higher education. The Internet and Higher Education, 49, 100790. https://doi.org/10.1016/j.iheduc.2020.100790

12.

Evens

T. M. S.

Handelman

(2006). The Manchester School: Practice and ethnographic praxis in anthropology. Bergahn Books.

13.

Garrison

D. R.

Anderson

Archer

(2001). Critical thinking, cognitive presence and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23. https://doi.org/10.1080/08923640109527071

14.

Gicheva

Edmunds

J. A.

Thrift

Hull

Bray

(2020). Conducting a randomized controlled trial in education: Experiences from an online postsecondary setting. SAGE Research Methods Cases. https://dx.doi.org/10.4135/9781529707533

15.

Gleason

Clark

Tuttle

C. C.

Dwyer

(2010). The evaluation of charter school impacts: Final report (NCEE 2010-4029). National Center for Educational Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/pubs/20104029/pdf/20104030.pdf

16.

Goldhaber

(2008). Teachers matter, but effective teacher quality policies are elusive. In Ladd

H. F.

Fiske

E. B.

(Eds.), Handbook of research in education finance and policy (pp. 146-165). Routledge.

17.

Gorard

Taylor

(2004). Combining methods in educational and social research. Open University Press.

18.

Hanley

Chambers

Haslam

(2016). Reassessing RCTs as the “gold standard”: Synergy not separatism in evaluation designs. International Journal of Research & Method in Education, 39(3), 287-298. https://doi.org/10.1080/1743727X.2016.1138457

19.

Hulleman

C. S.

Cordray

D. S.

(2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88-110. https://doi.org/10.1080/19345740802539325

20.

Institute of Education Sciences. (2005). Key items to get right when conducting a randomized controlled trial in education. https://files.eric.ed.gov/fulltext/ED531654.pdf

21.

Institute of Education Sciences. (2017). What works clearinghouse, procedures and standards handbook (Version 4.0). https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf

22.

Johnson

R. B.

Russo

Schoonenboom

(2019). Causation in mixed methods research: The meeting of philosophy, science, and practice. Journal of Mixed Methods Research, 13(2), 143-162. https://doi.dox.org/10.1177/1558689817719610

23.

Kao

Thompson

J. S.

(2003). Racial and ethnic stratification in educational achievement and attainment. Annual Review of Sociology, 29, 417-442. https://doi.org/10.1146/annurev.soc.29.010202.100019

24.

Lemons

C. J.

Fuchs

Gilbert

J. K.

Fuchs

L. S.

(2014). Evidence-based practices in a changing world: Reconsidering the counterfactual in education research. Educational Researcher, 43(5), 242-252. https://doi.org/10.3102/0013189X14539189

25.

Mannell

Davis

Akter

Jennings

Morrison

Kuddus

Fottrell

(2021). Visual participatory analysis: A qualitative method for engaging participants in interpreting the results of randomized controlled trials of health interventions. Journal of Mixed Methods Research, 15(1), 18-36.

26.

McCormick

M. P.

CapPella

O’Conner

Hill

J. L.

McClowery

(2016). Do effects of social-emotional learning programs vary by level of parent participation? Evidence from the randomized trials of INSIGHTS. Journal of Research on Educational Effectiveness, 9(3), 364-394. https://doi.org/10.1080/19345747.2015.1105892

27.

Miratrix

Furey

Feller

Grindal

Page

L. C.

(2018). Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness, 11(1), 133-162. https://doi.org/10.1080/19345747.2017.1379576

28.

Morrison

L. G.

Hargood

LIn

S. X.

Dennison

Joseph

Hughes

Michaelides

D. T.

Johnston

Michie

S., P.

Smith

P. W. F.

Weal

M. J.

Yardley

(2014). Understanding usage of a hybrid website and smartphone app for weight management: A mixed methods study. Journal of Medical Internet Research, 16(10), Article e201. https://doi.org/10.2196/jmir.3579

29.

Nielsen

Miraglia

(2016). What works for whom in which circumstances? On the need to move beyond the “what works” question in organizational intervention research. Human Relations, 70(1), 40-62. https://doi.org/10.1177/0018726716670226

30.

Nye

Konstantopolous

Hedges

L. V.

(2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237-257. https://doi.org/10.3102/01623737026003237

31.

O’Cathain

(2018). A practical guide to using qualitative research with randomized controlled trials. Oxford University Press.

32.

O’Cathain

Thomas

K. J.

Drabble

S. J.

Rudolph

Goode

Hewison

(2014). Maximising the value of combining qualitative research and randomized controlled trials in health research: The QUAlitative Research in Trials (QUART) study—A mixed methods study. Health Technology Assessment, 18, Article 38. https://doi.org/10.3310/hta18380

33.

Plano Clark

V. L.

Schumacher

West

Edrington

Dunn

L. B.

Harzstark

Melisko

Rabow

M. W.

Swift

P. S.

Miaskowski

. (2013). Practices for embedding an interpretive qualitative approach within a randomized clinical trial. Journal of Mixed Methods Research, 7(3), 219-242. https://doi.org/10.1177/1558689812474372

34.

Porta

(2008). Comparative analysis: Case-oriented versus variable-oriented research. In Porta

Keating

(Eds.), Approaches and methodologies in the social sciences: A pluralist perspective (pp. 198-222). Cambridge University Press.

35.

Raudenbush

S. W.

Bloom

H. S.

(2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475-499. https://doi.org/10.1177/1098214015600515

36.

Ross

Kena

Rathbun

KewalRamani

Zhang

Kristapovich

Manning

(2012). Higher education: Gaps in access and persistence study. Department of Education, National Center for Education Statistics. https://nces.ed.gov/pubs2012/2012046.pdf

37.

Sammons

Day

Kington

Stobart

Smees

(2007). Exploring variations in teachers’ work, lives, and their effects on pupils: Key findings and implications from a longitudinal mixed-method study. British Educational Research Journal, 33(5), 681-701. https://doi.org/10.1080/01411920701582264

38.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.

39.

Van Scoy

L. J.

Green

M. J.

Creswell

Thiede

Wiegand

I.S.

Lipnick

Johnson

Dimmock

A. E.

Foy

Lehman

. (2020). Generating a new outcome variable using mixed methods in a randomized controlled trial: The Caregiver Study—An Advance Care Planning investigation. Journal of Mixed Methods Research. Advance online publication. https://doi.org/10.1177/1558689820970686

40.

Weiss

M. J.

(2010). The implications of teacher selection and the teacher effect in individually randomized group treatment trials. Journal of Research on Educational Effectiveness, 3(4), 381-405. https://doi.org/10.1080/19345747.2010.504289

41.

Weiss

M. J.

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778-808. https://doi.org/10.1002/pam.21760

42.

Weiss

M. J.

Bloom

H. S.

Verbitsky-Savitz

Gupta

Vigil

A. E.

Cullinan

D. N.

(2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843-876. https://doi.org/10.1080/19345747.2017.1300719