Comparison of the What Works Clearinghouse Standards for Single-Case Research: Applications for Systematic Reviews

Abstract

Systematic reviews of single-case experimental research (SCER) in special education often use the What Works Clearinghouse (WWC) Standards to assess the methodological rigor of studies within a given literature base. While significant changes were made between the two most recent versions of the WWC standards, no research to date has evaluated the extent to which these standards would result in different evaluation outcomes. To examine potential differences, we applied version 4.1 and 5.0 of the standards to a sample database of SCER addressing the challenging behavior of students with autism in general education settings. Systematic search, screening, and review procedures resulted in a total of 20 articles included for this review. Findings indicated a 27% change in overall ratings across the studies, with version 5.0 leading to lower ratings than version 4.1, due to factors such as a lack of assessment of procedural fidelity, therapeutic baseline trends, and insufficient data points in the first baseline phase. We discuss implications for research based on our findings and recommend future directions for SCER in the field of special education.

Keywords

What Works Clearinghouse single case research quality indicators systematic review

Get full access to this article

View all access options for this article.

References

Akemoglu

Muharib

Meadan

(2020). A systematic and quality review of parent-implemented language and communication interventions conducted via telepractice. Journal of Behavioral Education, 29, 282–316.

Akiba

C. F.

Powell

B. J.

Pence

B. W.

Nguyen

M. X.

Golin

(2022). The case for prioritizing implementation strategy fidelity measurement: Benefits and challenges. Translational Behavioral Medicine, 12(2), 335–342.

*Bennett

Reichow

Wolery

(2011). Effects of structured teaching on the behavior of young children with disabilities. Focus on Autism and Other Developmental Disabilities, 26(3), 143–152. https://doi.org/10.1177/1088357611405040

*Blair

K.-S. C.

Umbreit

Dunlap

Jung

(2007). Promoting inclusion and peer participation through assessment-based intervention. Topics in Early Childhood Special Education, 27(3), 134–147. https://doi.org/10.1177/02711214070270030401

*Buggey

(2005). Video self-modeling applications with students with autism spectrum disorder in a small private school setting. Focus on Autism and Other Developmental Disabilities, 20(1), 52–63. https://doi.org/10.1177/10883576050200010501

Carroll

Patterson

Wood

Booth

Rick

Balain

(2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 1–9. https://doi.org/10.1186/1748-5908-2-40

*Casey

S. D.

Merical

C. L.

(2006). The use of functional communication training without additional treatment procedures in an inclusive school setting. Behavioral Disorders, 32(1), 46–54. https://doi.org/10.1177/019874290603200102

*Chan

J. M.

O’Reilly

M. F.

(2008). A Social Stories™ intervention package for students with autism in inclusive classroom settings. Journal of Applied Behavior Analysis, 41(3), 405–409. https://doi.org/10.1901/jaba.2008.41-405

Chia

G. L. C.

Anderson

McLean

L. A.

(2018). Use of technology to support self-management in individuals with Autism: Systematic review. Review Journal of Autism and Developmental Disorders, 5, 142–155.

10.

Council for Exceptional Children. (2014). Council for Exceptional Children: Standards for evidence-based practices in special education. Teaching Exceptional Children, 46(6), 206–212.

11.

Crockett

J. B.

Billingsley

Boscardin

M. L.

(Eds.). (2012). Leadership and instruction: Evidence-based practices in special education. In Crockett

J. B.

Billingsley

Boscardin

M. Lynn

(Eds.), Handbook of leadership and administration for special education (pp. 290–305). Routledge. https://doi.org/10.4324/9780203837313-25

12.

Crone

R. M.

Mehta

S. S.

(2016). Parent training on generalized use of behavior analytic strategies for decreasing the problem behavior of children with autism spectrum disorder: A data-based case study. Education and Treatment of Children, 39, 64–94. https://www.jstor.org/stable/44684095

13.

*Crozier

Tincani

(2007). Effects of social stories on prosocial behavior of preschool children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 37(9), 1803–1814. https://doi.org/10.1007/s10803-006-0315-7

14.

Emerson

Kiernan

Alborz

Reeves

Mason

Swarbrick

Mason

Hatton

(2001). The prevalence of challenging behaviors: A total population study. Research in Developmental Disabilities, 22(1), 77–93. https://doi.org/10.1016/S0891-4222(00)00061-5

15.

Fallon

L. M.

Collier-Meek

M. A.

Maggin

D. M.

Sanetti

L. M.

Johnson

A. H.

(2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case research. Exceptional Children, 81(2), 227–246.

16.

*Frea

W. D.

Arnold

C. L.

Vittimberga

G. L.

(2001). A demonstration of the effects of augmentative communication on the extreme aggressive behavior of a child with autism within an integrated preschool setting. Journal of Positive Behavior Interventions, 3(4), 194–198. https://doi.org/10.1177/109830070100300401

17.

Freeman

Sugai

(2013). Identifying evidence-based special education interventions from single-subject research. TEACHING Exceptional Children, 45(5), 6–12. https://doi.org/10.1177/004005991304500501

18.

Gregori

Drew

Lory

Kim

(2023). Grandparent-implemented interventions to reduce challenging behavior of an adult with autism: A pilot telehealth case study. Clinical Case Studies, 22, 439–452. https://doi.org/10.1177/15346501231159140

19.

*Haley

J. L.

Heick

P. F.

Luiselli

J. K.

(2010). Use of an antecedent intervention to decrease vocal stereotypy of a student with autism in the general education classroom. Child & Family Behavior Therapy, 32(4), 311–321. https://doi.org/10.1080/07317107.2010.515527

20.

Hill

H. C.

Erickson

(2019). Using implementation fidelity to aid in interpreting program impacts: A brief review. Educational Researcher, 48(9), 590–598.

21.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179. https://doi.org/10.1177/001440290507100203

22.

Jang

Dixon

D. R.

Tarbox

Granpeesheh

(2011). Symptom severity and challenging behavior in children with ASD. Research in Autism Spectrum Disorders, 5(3), 1028–1032. https://doi.org/10.1016/j.rasd.2010.11.008

23.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2013). Single-case intervention research design standards. Remedial and Special Education, 34(1), 26–38. https://doi.org/10.1177/0741932512452794

24.

Kratochwill

T. R.

Horner

R. H.

Levin

J. R.

Machalicek

Ferron

Johnson

(2023). Single-case intervention research design standards: Additional proposed upgrades and future directions. Journal of School Psychology, 97, 192–216.

25.

Ledford

J. R.

(2022). Concurrence on nonconcurrence in multiple-baseline designs: A commentary on slocum et al. (2022). Perspectives on Behavior Science, 45(3), 661–666. https://doi.org/10.1007/s40614-022-00342-1

26.

Ledford

J. R.

Lambert

J. M.

Pustejovsky

J. E.

Zimmerman

K. N.

Hollins

Barton

E. E.

(2023). Single-case-design research in special education: Next-generation guidelines and considerations. Exceptional Children, 89, 379–396. https://doi.org/10.1177/00144029221137656

27.

*Lee

G. T.

(2022). Using cooperative physical activities in inclusive settings to enhance social interactions for preschoolers with autism spectrum disorder in China. Journal of Positive Behavior Interventions, 24(3), 236–249. https://doi.org/10.1177/10983007211035135

28.

*Lee

Odom

S. L.

(1996). The relationship between stereotypic behavior and peer social interaction for children with severe disabilities. Journal of the Association for Persons with Severe Handicaps, 21(2), 88–95. https://doi.org/10.1177/154079699602100204

29.

*Loftin

R. L.

Odom

S. L.

Lantz

J. F.

(2008). Social interaction and repetitive motor behaviors. Journal of Autism and Developmental Disorders, 38, 1124–1135. https://doi.org/10.1007/s10803-007-0499-5

30.

*Macdonald

Trembath

Ashburner

Costley

Keen

(2018). The use of visual schedules and work systems to increase the on-task behaviour of students on the autism spectrum in mainstream classrooms. Journal of Research in Special Educational Needs, 18(4), 254–266. https://doi.org/10.1111/1471-3802.12409

31.

Maggin

D. M.

Barton

Reichow

Lane

K. L.

Shogren

K. A.

(2022). Commentary on the What Works Clearinghouse Standards and Procedures Handbook (v. 4.1) for the review of single-case research. Remedial and Special Education, 43(6), 421–433. https://doi.org/10.1177/07419325211051317

32.

Maggin

D. M.

Briesch

A. M.

Chafouleas

S. M.

(2013). An application of the What Works Clearinghouse standards for evaluating single-subject research: Synthesis of the self-management literature base. Remedial and Special Education, 34(1), 44–58. https://doi.org/10.1177/0741932511435176

33.

*Mancil

G. R.

Haydon

Whitby

(2009). Differentiated effects of paper and computer-assisted social stories^TM on inappropriate behavior in children with autism. Focus on Autism and Other Developmental Disabilities, 24(4), 205–215. https://doi.org/10.1177/1088357609347324

34.

Mason

R. A.

Lory

Gregori

Gerow

Rispoli

M. J.

David

Kim

S. Y.

Wang

(2022). Evidence-based practice in inclusive settings for students with autism: A best-evidence synthesis. Review Journal of Autism and Developmental Disorders, 1–17. https://doi.org/10.1007/s40489-022-00352-4

35.

*McCurdy

E. E.

Cole

C. L.

(2014). Use of a peer support intervention for promoting academic engagement of students with autism in general education settings. Journal of Autism and Developmental Disorders, 44, 883–893. https://doi.org/10.1007/s10803-013-1941-5

36.

McDonald

Moher

(2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1), 89. https://doi.org/10.1186/s13643-021-01626-4

37.

McKenna

J. W.

Brigham

Garwood

Zurawski

Koc

Lavin

Werunga

(2021). A systematic review of intervention studies for young children with emotional and behavioral disorders: Identifying the research base. Journal of Research in Special Educational Needs, 21(2), 120–145. https://doi.org/10.1111/1471-3802.12505

38.

Odom

S. L.

Brantlinger

Gersten

Horner

R. H.

Thompson

Harris

K. R.

(2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71(2), 137–148. https://doi.org/10.1177/001440290507100201

39.

Odom

S. L.

Duda

M. A.

Kucharczyk

Cox

A. W.

Stabel

(2014). Applying an implementation science framework for adoption of a comprehensive program for high school students with autism spectrum disorder. Remedial and Special Education, 35(2), 123–132. https://doi.org/10.1177/0741932513519826

40.

*Ozdemir

(2008). The effectiveness of social stories on decreasing disruptive behaviors of children with autism: Three case studies. Journal of Autism and Developmental Disorders, 38(9), 1689–1696. https://doi.org/10.1007/s10803-008-0551-0

41.

Peltier

Vannest

K. J.

Morin

K. L.

Sinclair

T. E.

Sallese

M. R.

(2020). A systematic review of teacher-mediated interventions to improve the mathematical performance of students with emotional and behavioral disorders. Exceptionality, 28(2), 121–141. https://doi.org/10.1080/09362835.2020.1771717

42.

Reichow

Barton

E. E.

Maggin

D. M.

(2018). Development and applications of the single-case design risk of bias tool for evaluating single-case design research study reports. Research in Developmental Disabilities, 79, 53–64. https://doi.org/10.1016/j.ridd.2018.05.008

43.

*Rispoli

M. J.

O’Reilly

M. F.

Sigafoos

Lang

Kang

Lancioni

(2011). Effects of presession satiation on challenging behavior and academic engagement for children with autism during classroom instruction. Education and Training in Autism and Developmental Disabilities, 46(4), 607–618.

44.

*Rosenbloom

Mason

R. A.

Wills

H. P.

Mason

B. A.

(2016). Technology delivered self-monitoring application to promote successful inclusion of an elementary student with autism. Assistive Technology, 28(1), 9–16. https://doi.org/10.1080/10400435.2015.1059384

45.

Schwartz

I. S.

Baer

D. M.

(1991). Social validity assessments: Is current practice state of the art? Journal of Applied Behavior Analysis, 24(2), 189–204. https://doi.org/10.1901/jaba.1991.24-189

46.

Slocum

T. A.

Joslyn

P. R.

Nichols

Pinkelman

S. E.

(2022). Revisiting an analysis of threats to internal validity in multiple baseline designs. Perspectives on Behavior Science, 45(3), 681–694. https://doi.org/10.1007/s40614-022-00351-0

47.

Snodgrass

M. R.

Chung

M. Y.

Meadan

Halle

J. W.

(2018). Social validity in single-case research: A systematic literature review of prevalence and application. Research in Developmental Disabilities, 74, 160–173. https://doi.org/10.1016/j.ridd.2018.01.007

48.

Steinbrenner

J. R.

Odom

S. L.

Hall

L. J.

Hume

(2020). Moving beyond fidelity: Assessing implementation of a comprehensive treatment program for adolescents with autism spectrum disorder. Exceptional Children, 86(2), 137–154. https://doi.org/10.1177/0014402919855321

49.

*Strain

P. S.

Wilson

Dunlap

(2011). Prevent-teach-reinforce: Addressing problem behaviors of students with autism in general education classrooms. Behavioral Disorders, 36(3), 160–171. https://doi.org/10.1177/019874291003600302

50.

Talbott

Maggin

D. M.

Van Acker

E. Y.

Kumm

(2018). Quality indicators for reviews of research in special education. Exceptionality, 26(4), 245–265. https://doi.org/10.1080/09362835.2017.1283625

51.

*Taylor

B. A.

Hoch

Weissman

(2005). The analysis and treatment of vocal stereotypy in a child with autism. Behavioral Interventions, 20(4), 239–253. https://doi.org/10.1002/bin.200

52.

Tekin-Iftar

Olcay

Sirin

Bilmez

Degirmenci

H. D.

Collins

B. C.

(2021). Systematic review of safety skill interventions for individuals with autism spectrum disorder. The Journal of Special Education, 54(4), 239–250. https://doi.org/10.1177/0022466920918247

53.

Watson

P. J.

Workman

E. A.

(1981). The non-concurrent multiple baseline across-individuals design: An extension of the traditional multiple baseline design. Journal of Behavior Therapy and Experimental Psychiatry, 12(3), 257–259. https://doi.org/10.1016/0005-7916(81)90055-0

54.

What Works Clearinghouse. (2020). What Works Clearinghouse^TM standards handbook (Version 4.1). https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf

55.

What Works Clearinghouse. (2022). What Works Clearinghouse^TM standards handbook(Version 5.0). https://ies.ed.gov/ncee/wwc/Docs/referenceresources/Final_WWC-HandbookVer5.0-0-508.pdf

56.

Wolery

(2013). A commentary: Single-case design technical document of the What Works Clearinghouse. Remedial and Special Education, 34(1), 39–43. https://doi.org/10.1177/0741932512468038

57.

Wolf

M. M.

(1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203–214. https://doi.org/10.1901/jaba.1978.11-203

58.

*Wu

Y.-C.

Chen

P.-Y.

Tsai

S.-P.

Tsai

S.-F.

Chou

Y.-C.

Chiu

C.-Y.

(2019). The effects of the class-wide function-related intervention teams on behaviors of an elementary student with autism spectrum disorder in an inclusive classroom in Taiwan. International Journal of Developmental Disabilities, 65(5), 368–377. https://doi.org/10.1080/20473869.2019.1647031