Empirically Derived Single-Case-Design Effect Size Distributions of Engagement and Challenging Behavior in Early Childhood Research

Abstract

The purpose of this study is to present a set of empirically derived effect size distributions in order to provide field-based benchmarks for assessing the relative effects of interventions aimed at reducing challenging behavior or increasing engagement for young children with and without disabilities. We synthesized 192 single-case designs that represented data from 162 individuals and nine groups of individuals (e.g., classes) in 53 reports. We generated distributions of standardized mean difference and log-response ratio using 197 effect sizes for engagement and 146 effect sizes for challenging behavior. We examined benchmarks as a function of publication, disability, dependent variable (primary/secondary), and functional relation status and reported distributions separately by engagement and challenging behavior. Overall, the effect size distributions suggest considerable variability in the magnitude of change associated with school-based interventions for engagement and challenging behavior assessed in the context of single-case designs. Data suggest that expected magnitude of change for engagement and challenging behavior interventions may be difficult to predict and that study and effect size characteristics influence the variability of the distributions. Our results have direct implications for researchers relative to assessing the efficacy of interventions aimed at reducing challenging behavior or increasing engagement in young children with and without disabilities.

Get full access to this article

View all access options for this article.

References

Bakker

Cai

English

Kaiser

Mesa

Van Dooren

(2019). Beyond small, medium, or large: Points of consideration when interpreting effect sizes. Educational Studies in Mathematics, 102(1), 1–8. https://doi.org/10.1007/s10649-019-09908-4

Barton

E. E.

Pustejovsky

J. E.

Maggin

D. M.

Reichow

(2017). Technology-aided instruction and intervention for students with ASD: A meta-analysis using novel methods of estimating effect sizes for single-case research. Remedial and Special Education, 38(6), 371–386. https://doi.org/10.1177/0741932517729508

Bloom

H. S.

Hill

C. J.

Black

A. R.

Lipsey

M. W.

(2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072

Candbank

Chow

Bottema-Beutel

Woynaroski

(2021). Evaluating evidence-based practice in light of the boundedness and proximity of outcomes: Capturing the scope of change. Autism Research, 14(8), 1536–1542. https://doi.org/10.1002/aur.2527

Cheung

A. C.

Slavin

R. E.

(2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292. https://doi.org/10.3102/0013189X16656615

Chow

J. C.

(2020). Commentary: Classroom motivation and learning disabilities. Consensus points and recommendations. Learning Disabilities: A Multidisciplinary Journal, 25(2), 54–60. https://js.sagamorepub.com/ldmj/article/view/10734

Chow

J. C.

Ekholm

(2018). Do published studies yield larger effect sizes than unpublished studies in education and special education? A meta-review. Educational Psychology Review, 30(3), 727–744. https://doi.org/10.1007/s10648-018-9437-7

Chow

J. C.

Morse

Zhao

Kingsbery

Fisk

Soni

(2022). A systematic review of the characteristics of students with emotional disturbance in special education research. Remedial and Special Education. https://doi.org/10.1177/07419325221125890

Chow

J. C.

Sjogren

A. L.

Zhao

(2021). Reporting and reproducibility of meta-analyses in speech-language hearing research. Journal of Speech, Language, and Hearing Research, 64(7), 2786–2793. https://doi.org/10.1044/2021_JSLHR-21-00047

10.

Chow

Zhao

Sandbank

Bottema-Beutel

Woynaroski

(2021). Empirically-derived effect size distributions of interventions for young children on the autism spectrum. Journal of Clinical Child & Adolescent Psychology, 52(2), 1–13. https://doi.org/10.1080/15374416.2021.2007485

11.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.

12.

Cumming

M. M.

Bettini

Chow

J. C.

(2022). High-quality systematic literature reviews in special education: Promoting coherence, contextualization, generativity, and transparency. Exceptional Children. https://doi.org/10.1177/00144029221146576

13.

Ekholm

Chow

J. C.

(2018). Addressing publication bias in educational psychology. Translational Issues in Psychological Science, 4(4), 425–439. https://doi.org/10.1037/tps0000181

14.

Gage

N. A.

Cook

B. G.

Reichow

(2017). Publication bias in special education meta-analyses. Exceptional Children, 83(4), 428–445. https://doi.org/10.1177/0014402917691016

15.

Harris

D. N.

(2009). Toward policy-relevant benchmarks for interpreting effect sizes: Combining effects with costs. Educational Evaluation and Policy Analysis, 31(1), 3–29. https://doi.org/10.3102/0162373708327524

16.

Harris

P. A.

Taylor

Thielke

Payne

Gonzalez

Conde

J. G.

(2009). Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of biomedical informatics, 42(2), 377–381.

17.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x

18.

Huwaldt

J. A.

Steinhorst

(2015). Plot Digitizer (Version 2.6. 8) [Computer software]. https://sourceforge.net/projects/plotdigitizer

19.

Kelley

Preacher

K. J.

(2012). On effect size. Psychological Methods, 17(2), 137–152. https://doi.org/10.1037/a0028086

20.

Kraft

M. A.

(2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798

21.

Ledford

J. R.

Gast

D. L.

(2018). Single case research methodology: Applications in special education and behavioral sciences. Routledge.

22.

Ledford

J. R.

Lambert

J. M.

Pustejovsky

J. E.

Zimmerman

K. N.

Hollins

Barton

E. E.

(2022). Single-case-design research in special education: Next-generation guidelines and considerations. Exceptional Children. Advance online publication. https://doi.org/10.1177/00144029221137656

23.

Ledford

J. R.

Windsor

S. A.

(2021). Systematic review of interventions designed to teach imitation to young children with disabilities. Topics in Early Childhood Special Education, 42(2), 202–214. https://doi.org/10.1177/02711214211007190

24.

Lipsey

M. W.

Cullen

F. T.

(2007). The effectiveness of correctional rehabilitation: A review of systematic. The Annual Review of Law and Social Science, 3, 297–320.

25.

Luo

Reichow

Snyder

Harrington

Polignano

(2020). Systematic review and meta-analysis of classroom-wide social–emotional interventions for preschool children. Topics in Early Childhood Special Education. https://doi.org/10.1177/0271121420935579

26.

Maggin

D. M.

Cook

B. G.

Cook

(2019). Making sense of single-case design effect sizes. Learning Disabilities Research & Practice, 34(3), 124–132. https://doi.org/10.1111/ldrp.12204

27.

Pigott

T. D.

Polanin

J. R.

(2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90(1), 24–46. https://doi.org/10.3102/0034654319877153

28.

Pustejovsky

J. E.

(2018). Using response ratios for meta-analyzing single-case designs with behavioral outcomes. Journal of School Psychology, 68, 99–112. https://doi.org/10.1016/j.jsp.2018.02.003

29.

Pustejovsky

J. E.

(2019). Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures. Psychological Methods, 24(2), 217–235. https://doi.org/10.1037/met0000179

30.

Pustejovsky

J. E.

Swan

D. M.

(2018). SingleCaseES: A calculator for single-case effect sizes . R Package (Version 0.4, 3) [Computer software].

31.

R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. http://www.R-project.org

32.

Robertson

R. E.

Sobeck

E. E.

Wynkoop

Schwartz

(2017). Participant diversity in special education research: Parent-implemented behavior interventions for children with autism. Remedial and Special Education, 38(5), 259–271.

33.

Shadish

W. R.

Hedges

L. V.

Pustejovsky

J. E.

(2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. https://doi.org/10.1016/j.jsp.2013.11.005

34.

Tanner-Smith

E. E.

Durlak

J. A.

Marx

R. A.

(2018). Empirically based mean effect size distributions for universal prevention programs targeting school-aged youth: A review of meta-analyses. Prevention Science, 19(8), 1091–1101. https://doi.org/10.1007/s11121-018-0942-1

35.

Taylor

J. A.

Kowalski

S. M.

Polanin

J. R.

Askinas

Stuhlsatz

M. A.

Wilson

C. D.

Wilson

S. J.

(2018). Investigating science education effect sizes: Implications for power analyses and programmatic decisions. AERA Open, 4(3). https://doi.org/10.1177/2332858418791991

36.

Vannest

K. J.

Sallese

M. R.

(2021). Benchmarking effect sizes in single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 1–24. https://doi.org/10.1080/17489539.2021.1886412

37.

What Works Clearinghouse. (2020). What Works Clearinghouse standards handbook, Version 4.1. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB