A Framework for Approaching Mixed Methods Intervention Research to Address the Emotional and Behavioral Health Needs of Children

Abstract

The use of singular methodologies has dominated emotional and behavioral disorder (EBD) intervention research, and by extension, the field has not deployed mixed-methods research (MMR). The purpose of this article is to provide an overview of MMR and demonstrate its utility for conducting EBD intervention research. The basic tenants and design principles of MMR are discussed. Then, we present an aspirational framework that summarizes 17 areas where MMR can be applied to EBD intervention research using a school psychology study for illustration. The framework includes (a) research question development, (b) intervention development, (c) instrumentation development, (d) selection, (e) sampling, (f) multi-lens causality, (g) social validity, (h) acceptability, (i) stakeholder perspectives, (j) treatment integrity, (k) contamination, (l) carry-over effects, (m) outcome variability, (n) attrition, (o) replication, (p) data analysis and interpretation, and (q) dissemination. By referring to this framework as being aspirational, we mean to convey that very few social science studies or programs of research fully address all its elements; indeed, it may even be considered as a wish list. However, we contend that MMR approaches can help researchers to address the framework more fully within their lines of EBD inquiry.

Keywords

qualitative methods interventions experimental design disorders/disabilities framework mixed-methods research

Get full access to this article

View all access options for this article.

References

Anderson

J. A.

(2018). Challenges and opportunities: Mixed research for improving outcomes for students with emotional and behavior disabilities. Research in the Schools, 25(2), 44–55.

Baer

D. M.

Wolf

M. M.

Risley

T. R.

(1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97. https://doi.org/10.1901/jaba.1968.1-91

Bazeley

(2018). Integrating analyses in mixed methods research. SAGE.

Bierman

K. L.

Sanders

M. T.

(2021). Teaching explicit social-emotional skills with contextual supports for students with intensive intervention needs. Journal of Emotional and Behavioral Disorders, 29(1), 14–23. https://doi.org/10.1177/1063426620957623

Brantlinger

Jimenez

Klingner

(2005). Qualitative studies in special education. Exceptional Children, 71(2), 195–207. https://doi.org/10.1177/001440290507100205

Carter

S. L.

(2008). Further conceptualization of treatment acceptability. Education and Training in Developmental Disabilities, 43(2), 135–145.

Collins

(2017). Sampling designs in educational research. In Wyse

Smith

Suter

Selwyn

(Eds.), British educational research association (BERA)/SAGE handbook of educational research (Vol. 1, pp. 280–292). SAGE.

Collins

Onwuegbuzie

A. J.

Johnson

R. B.

(2012). Securing a place at the table: A review and extension of legitimation criteria for the conduct of mixed research. American Behavioral Scientist, 56(6), 849–865. http://doi.org/10.1177/0002764211433799

Common

E. A.

Lane

K. L.

(2017). Social validity assessment. In Luiselli

J. K.

(Ed.), Applied behavior analysis advanced guidebook: A manual for professional practice (pp. 73–92). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-811122-2.00004-8

10.

Conroy

Houchins

D. E.

Hitchcock

Hetrick

Anderson

(this issue). Using mixed methods in emotional/behavioral disorders intervention research. Behavior Disorders.

11.

Cook

B. G.

Cook

Landrum

T. J.

(2013). Moving research into practice: Can we make dissemination stick? Exceptional Children, 79(3), 163–180. https://doi.org/10.1177/001440291307900203

12.

Cook

B. G.

Cook

S. C.

(2013). Unraveling evidence-based practices in special education. The Journal of Special Education, 47(2), 71–82. https://doi.org/10.1177/0022466911420877

13.

Corr

Snodgrass

M. R.

Love

Scott

I. M.

Kim

Andrews

(2020). Exploring the landscape of published mixed methods research in special education: A systematic review. Remedial and Special Education, 42, 317–328. https://doi.org/10.1177/0741932520924030

14.

Corrigan

J. A.

Onwuegbuzie

A. J.

(2020). Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. The Qualitative Report, 25(3), 785–812. https://doi.org/10.46743/2160-3715/2020.3579

15.

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37(4), 244–253. https://doi.org/10.1177/0741932516648463

16.

Creamer

E. G.

Reeping

(2020). Advancing mixed methods in psychological research. Methods in Psychology, 3, 1–11. https://doi.org/10.1016/j.metip.2020.100035

17.

Creswell

J. W.

Plano Clark

V. L.

(2017). Designing and conducting mixed methods research (3rd ed.). SAGE.

18.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45.

19.

Deke

Chiang

(2017). The WWC attrition standard: Sensitivity to assumptions and opportunities for refining and adapting to new contexts. Evaluation Review, 41(2), 130–154. https://doi.org/10.1177/0193841X16670047

20.

Dellinger

A. B.

Leech

N. L.

(2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309–332. http://doi:10.1177/1558689807306147

21.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3/4), 327–350. https://doi.org/10.1007/s10464-008-9165-0

22.

Elliott

S. N.

(2017). The social validity of “acceptability of behavioral interventions used in classrooms”: Inferences from longitudinal evidence. Behavioral Disorders, 43(1), 269–273. https://doi.org/10.1177/0198742917739021

23.

Fabregues

Molina-Azorin

J. F.

(2017). Addressing quality in mixed methods research: A review and recommendations for a future agenda. Quality & Quantity, 51(6), 2847–2863. http://doi.org/10.1007/s11135-016-0449-4

24.

Farley-Ripple

May

Karpyn

Tilley

McDonough

(2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235–245. https://doi.org/10.3102/0013189X18761042

25.

Fetters

M. D.

Curry

Creswell

(2013). Achieving integration in mixed methods designs: Principles and practices. Health Services Research, 48(6), 2131–2156. https://doi.org/10.1111/1475-6773.12117

26.

Fetters

M. D.

Freshwater

(2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9, 115–117. https://doi.org/10.1177/1558689815581222

27.

Fetters

M. D.

Molina-Azorin

J. F.

(2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi.org/10.1177/1558689817714066

28.

Gage

N. A.

Stevens

R. N.

(2018). Rigor, replication, and reproducibility: Increasing the relevance of behavioral disorders research. Education & Treatment of Children, 41(4), 567–588. https://doi.org/10.1353/etc.2018.0029

29.

Garwood

J. D.

Peltier

Sinclair

Eisel

McKenna

J. W.

Vannest

K. J.

(2021). A quantitative synthesis of intervention research published in flagship EBD journals: 2010 to 2019. Behavioral Disorders, 47(1), 14–27. https://doi.org/10.1177/0198742920961341

30.

Gersten

Fuchs, L

Compton

Coyne

M. S.

Greenwood

Innocenti

(2005). Quality indicators for group experimental and quasi experimental research in special education. Exceptional Children, 71(2), 149–164. https://doi.org/10.1177/001440290507100202

31.

Greene

J. C.

(2007). Mixed methods in social inquiry. Jossey-Bass.

32.

Gresham

F. M.

Van

M. B.

Cook

C. R.

(2006). Social skills training for teaching replacement behaviors: Remediating acquisition deficits in at-risk students. Behavioral Disorders, 31, 363–377. http://doi.org/10.1177/019874290603100402

33.

Griffith

A. K.

Duppong Hurley

Hagaman

J. L.

(2009). Treatment integrity of literacy interventions for students with emotional and/or behavioral disorders: A review of literature. Remedial and Special Education, 30(4), 245–255. https://doi.org/10.1177/0741932508321013

34.

Hall

G. C.

Okazaki

(2002). Asian American psychology: The science of lives in context. American Psychological Association. https://doi.org/10.1037/10473-000

35.

Harry

Fenton

(2016). Risk in schooling. The contribution of qualitative research to our understanding of the overrepresentation of minorities in special education. Multiple Voices for Ethnically Diverse Exceptional Learners, 16(1), 17–28. https://doi.org/10.5555/2158-396X.16.1.17

36.

Hitchcock

J. H.

Horner

R. H.

Kratochwill

T. R.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2014). The what works clearinghouse single-case design pilot standards: Who will guard the guards? Remedial and Special Education, 35(3), 145–152. https://doi.org/10.1177/0741932513518979

37.

Hitchcock

Houchins

D. E.

Onwuegbuzie

A. J.

(this issue). Addressing the catch-22 of using mixed methods in Emotional and Behavioral Disorders intervention research. Behavior Disorders.

38.

Hitchcock

J. H.

Johnson

R. B.

Schoonenboom

(2018). Idiographic and nomothetic causal inference in special education research and practice: Mixed methods perspectives. Research in the Schools, 25(2), 56–67. https://doi.org/10.8553/135047246

39.

Hitchcock

J. H.

Onwuegbuzie

A. J.

(Eds.). (2022). The Routledge handbook for advancing integration in mixed methods research (1st ed.). Routledge. https://doi.org/10.4324/9780429432828

40.

Hitchcock

J. H.

Onwuegbuzie

A. J.

(2020). Developing mixed methods crossover analysis approaches. Journal of Mixed Methods Research, 14, 63–83. https://doi.org/10.1177/1558689819841782

41.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179. https://doi.org/10.1177/001440290507100203

42.

Houchins

D. E.

Marsh

R. J.

Tanner

(2022). An introduction to mixed methods special education research. In Farmer

Talbott

McMaster

Lee

Aceves

(Eds.), Handbook of special education research (Vol. I). Routledge. https://doi.org/10.4324/9781003156857

43.

Hunter

Brewer

(2015). Conundrums of multimethod research. In Hesse-Biber

Johnson

R. B.

(Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 616–623). Oxford University Press. http://doi.org/10.1093/oxfordhb/9780199933624.013.38

44.

Johnson

R. B.

Christensen

(2019). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). SAGE.

45.

Johnson

R. B.

Onwuegbuzie

A. J.

Turner

L. A.

(2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–133. https://doi.org/10.1177/1558689806298224

46.

Joyce

K. E.

Cartwright

(2020). Bridging the gap between research and practice: Predicting what will work locally. American Educational Research Journal, 57(3), 1045–1082. https://doi.org/10.3102/0002831219866687

47.

Kalkbrenner

(2021). A practical guide to instrument development and score validation in social sciences research: The MEASURE approach. Practical Assessment, Research & Evaluation, 26(1). https://scholarworks.umass.edu/pare/vol26/iss1/1/

48.

Kazdin

A. E.

(2021). Single-case research designs: Methods for clinical and applied settings (3rd ed.). Oxford University Press.

49.

Kern

Evans

S. W.

Lewis

T. J.

(2011). Description of an iterative process for intervention development. Education and Treatment of Children, 34(4), 593–617. http://www.jstor.org/stable/42900136

50.

Klingner

J. K.

Boardman

A. G.

(2011). Addressing the “Research Gap” in special education through mixed methods. Learning Disability Quarterly, 34(3), 208–218. https://doi.org/10.1177/0731948711417559

51.

Kratochwill

T. R.

Levin

J. R.

Horner

R. H.

(2018). Negative results: Conceptual and methodological dimensions in single-case intervention research. Remedial & Special Education, 39(2), 67–76. https://doi.org/10.1177/0741932517741721

52.

Lane

K. L.

Jolivette

Conroy

Nelson

C. M.

Benner

G. J.

(2011). Future research directions for the field of E/BD: Standing on the shoulders of giants. Education and Treatment of Children, 34(4), 423–443. http://doi.org10.1353/etc.2011.0029/

53.

Ledford

J. R.

Wolery

(2013). Procedural fidelity: An analysis of measurement and reporting practices. Journal of Early Intervention, 35(2), 173–193. https://doi.org/10.1177/1053815113515908

54.

Leko

M. M.

(2014). The value of qualitative methods in social validity research. Remedial and Special Education, 35(5), 275–286. https://doi.org/10.1177/0741932514524002

55.

Leko

M. M.

Hitchcock

Love

Houchins

D. E.

Conroy

. (in-press). Quality indicators for mixed methods research in special education. Exceptional Children.

56.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. SAGE.

57.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513

58.

Marsh

R. J.

Houchins

D. E.

Christenson

C. S.

(2022). Using an integrated approach when conducting juvenile justice special education mixed research. In Onwuegbuzie

A. J.

Hitchcock

(Eds.), The Routledge handbook for advancing integration in mixed methods research (pp. 425–451). Routledge.

59.

McKenna

J. W.

Garwood

Parenti

(2021). Inclusive instruction for students with Emotional/Behavioral Disorders: Service in the absence of intervention research. Intervention in School and Clinic, 56(5), 316–321. https://doi.org/10.1177/1053451220963084

60.

McKenna

J. W.

Solis

Brigham

Adamson

(2019). The responsible inclusion of students receiving special education services for emotional disturbance: Unraveling the practice to research gap. Behavior Modification, 43(4), 587–611. https://doi.org/10.1177/0145445518762398

61.

Mendenhall

A. N.

Fontanella

C. A.

Hiance

D. L.

Frauenholtz

(2014). Factors associated with treatment attrition for Medicaid-enrolled youth with serious emotional disturbances. Children & Youth Services Review, 40, 20–28. https://doi.org/10.1016/j.childyouth.2014.02.011

62.

Nastasi

B. K.

Hart

Naser

(2020). International handbook on child rights and school psychology. Springer. https://doi.org/10.1007/978-3-030-37119-7_10

63.

Nastasi

B. K.

Hitchcock

J. H.

Gutierrez

Oshrin

(2022). Promoting quality of mixed methods consultation research: A synergistic participatory framework. Journal of Education and Psychological Consultation, 32, 22–45. https://doi.org/10.1080/10474412.2021.1887743

64.

Nastasi

B. K.

Hitchcock

J. H.

Varjas

Jayasena

Sarkar

Moore

R. B.

Burden

Albrecht

(2010). School-based stress and coping program for adolescents in Sri Lanka: Using mixed methods to facilitate culture-specific programming. In Collins

K. M. T.

Onwuegbuzie

A. J.

Jiao

Q. G.

(Vol. Eds.), Toward a broader understanding of stress and coping: Mixed methods approaches. The research on stress and coping in education series (Vol. 5, pp. 305–342). Information Age Publishing.

65.

Newman

Hitchcock

J. H.

(2011). Underlying agreements between quantitative and qualitative research: The short and tall of it all. Human Resource Development Review, 10(4), 381–398. https://doi.org/10.1177/1534484311413867

66.

Newman

Houchins

D. E.

(2018). Conceptualizing mixed methods questions in special education research. Research in the Schools, 25(2), 23–33. https://doi.org/10.8553/135047243

67.

O’Cathain

(2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (2nd ed., pp. 531–555). SAGE.

68.

Odom

S. L.

(2021). Education of students with disabilities, science, and randomized controlled trials. Research & Practice for Persons with Severe Disabilities, 46(3), 132–145. https://doi.org/10.1177/15407969211032341

69.

Odom

S. L.

Brantlinger

Gersten

(2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71(2), 137–148.

70.

Onwuegbuzie

A. J.

Bustamante

R. M.

Nelson

J. A.

(2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56–78. https://doi.org/10.1177/1558689809355805

71.

Onwuegbuzie

A. J.

Collins

K. M. T.

(2017). The role of sampling in mixed methods-research: Enhancing inference quality. Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(2), 133–156. https://doi.org/10.1007/s11577-017-0455-0

72.

Onwuegbuzie

A. J.

Corrigan

J. A.

(2018). What is happening now? An overview of mixed methods applications in special education. Research in the Schools, 25(2), 1–22.

73.

Onwuegbuzie

A. J.

Hitchcock

J. H.

(2019). Using mathematical formulae as proof for integrating mixed methods research and multiple methods research approaches: A call for multi-mixed methods and meta-methods in a mixed research 2.0 Era. International Journal of Multiple Research Approaches, 11(3), 213–234. https://doi.org/10.29034/ijmra.v11n3editorial2

74.

Onwuegbuzie

A. J.

Johnson

R. B.

(2006). The validity issue in mixed research. Research in the Schools, 13(1), 48–63.

75.

Onwuegbuzie

A. J.

Johnson

R. B.

(2021). The Routledge reviewer’s guide to mixed methods analysis. Routledge. https://www.routledge.com/The-Routledge-Reviewers-Guide-to-Mixed-Methods-Analysis/Onwuegbuzie-Johnson/p/book/9781138305274

76.

Plano Clark

Walton

J. B.

Foote

L. A

. (2022). Designing for integration in mixed methods case studies. In Hitchcock

J. H.

Onwuegbuzie

A. J.

(Eds.), The Routledge handbook for advancing integration in mixed methods research (pp. 213–228). Routledge.

77.

Power

T. J.

Blom-Hoffman

Clarke

A. T.

Riley-Tillman

T. C.

Kellerher

Manz

(2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507. https://doi.org/10.1002/pits.20087

78.

Ridenour

C. S.

Newman

(2008). Mixed methods research: Exploring the interactive continuum. Southern Illinois University Press.

79.

Rumrill

P. D.

Jr. Cook

B. G.

Stevenson

N. A.

(2020). Research in special education: Designs, methods, and applications (3rd ed.). Charles C. Thomas Publisher.

80.

Sabornie

E. J.

Weiss

S. L.

(2014). Qualitative and mixed design research in emotional and behavioral disorders. In Walker

H. M.

Gresham

F. M.

(Eds.), Handbook of evidence-based practices for emotional and behavioral disorders: Applications in schools (pp. 537–551). The Guilford Press.

81.

Schoonenboom

Johnson

R. B.

(2017). How to construct a mixed methods research design. Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(2), 107–131. https://doi.org/10.1007/s11577-017-0454-1

82.

Schwandt

T. A.

(2000). Three epistemological stances for qualitative inquiry: Interpretivism, hermeneutics, and social constructionism. In Denzin

N. K.

Lincoln

Y. S.

(Eds.), Handbook of qualitative research (2nd ed., pp. 189–215). SAGE.

83.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

84.

Skrtic

T. M.

(1995). Special education and student disability as organization and change. In Skrtic

(Ed.), Disability and democracy: Reconstructing (special) education for postmodernity (pp. 5–22). Teachers College Press.

85.

Slavin

R. E.

Cheung

A. C. K.

(2017). Lessons learned from large-scale randomized experiments. Journal of Education for Students Placed at Risk, 22(4), 253–259. https://doi.org/10.1080/10824669.2017.1360774

86.

Thompson

Diamond

K. E.

McWilliam

(2005). Evaluating the quality of evidence from correlational research for evidence-based practice. Exceptional Children, 71(2), 181–194. https://doi.org/10.1177/001440290507100204

87.

Trainor

A. A.

(2011). Commentary: Using mixed methods to transform special education research. Learning Disability Quarterly, 34(3), 219–221. http://doi.org/10.1177/0731948711417560

88.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication research and special education. Remedial and Special Education, 37(4), 195–204. https://doi.org/10.1177/0741932516648462

89.

Whitlow

Cooper

Couvillon

(2019). Voices from those not heard: A case study on the inclusion experience of adolescent girls with Emotional Behavioral Disabilities. Children & Schools, 41(1), 45–54. https://doi.org/10.1093/cs/cdy027

90.

Witt

J. C.

Elliott

S. N.

(1985). Acceptability of classroom intervention strategies. In Kratochwill

T. R.

(Ed.), Advances in school psychology (Vol. 4, pp. 251–288). Lawrence Erlbaum.

91.

Wolf

M. M.

(1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214. https://doi.org/10.1901/jaba.1978.11-203

92.

Yardley

Bradbury

Morrison

(2021). Using qualitative research for intervention development and evaluation. In Camic

P. M.

(Ed.), Qualitative research in psychology: Expanding perspectives in methodology and design (2nd ed., pp. 263–282). American Psychological Association. https://doi.org/10.1037/0000252-013