Same as It Ever Was: An Updated Review of Replication Studies in Special Education Journals

Abstract

Despite the importance of replication studies, previous reviews showed that they comprised less than 1% of publications in special education journals. This review conceptually replicates Lemons and colleagues’ previous review to provide updated rates of replication studies in special education journals. We identified 78 studies published in 44 special education journals between 2015 and 2022 that used the term replicat* and met our definition of replication, constituting 0.54% of all publications. Similar to previous findings, most replication studies were conceptual, successfully replicated previous results, and were conducted by one or more authors of the study being replicated. Replications with author overlap were significantly more likely to successfully reproduce the findings of the original study. Journal impact factor was significantly and positively related to journal-level replication rate. Publication rates of replications in special education journals remain similar to previous decades, suggesting that additional efforts are needed to increase the publication of replication research.

Keywords

quantitative research methodology evidence-based practice

Get full access to this article

View all access options for this article.

References

Adelson

J. L.

Barton

Bradshaw

Bryant

Cook

B. G.

Coyne

DeBettencourt

DeHaven

A. C.

Dymond

Esposito

Farmer

Flake

J. K.

Gage

N. A.

Kennedy

Kern

Lane

K. L.

Lee

Lembke

. . . Troia

G. A

. (2019, February 18). A Roadmap for transparent research in special education and related disciplines. https://doi.org/10.35542/osf.io/sqfy3

Altman

D. G.

(1990). Practical statistics for medical research. CRC Press.

Baker

(2016). Reproducibility crisis. Nature, 533(26), 353–366. https://doi.org/10.1038/533452a

Banerjee

(2022). Replication research in education: Is the tide turning? Educational Research and Evaluation, 27(1–2), 220–224. https://doi.org/10.1080/13803611.2021.2022311

Banerjee

Movahedazarhouligh

Millen

Luckner

J. L.

(2018). Building evidence in early childhood special education: A systematic review of replication intervention studies. Topics in Early Childhood Special Education, 37(4), 246–256. https://doi.org/10.1177/0271121417748197

Chhin

C. S.

Taylor

K. A.

Wei

W. S.

(2018). Supporting a culture of replication: An examination of education and special education research grants funded by the Institute of Education Sciences. Educational Researcher, 47(9), 594–605. https://doi.org/10.3102/0013189X18788047

Cook

B. G.

Collins

L. W.

Cook

S. C.

Cook

L. H.

(2016). A replication by any other name: A systematic review of replicative intervention studies. Remedial and Special Education, 37(4), 223–234. https://doi.org/10.1177/0741932516637198

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(Eds.). (2016). Replication in special education research [special issue]. Remedial and Special Education, 37(4).

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37(4), 244–253. https://doi.org/10.1177/0741932516648463

10.

Dettori

J. R.

Norvell

D. C.

(2020). Kappa and beyond: Is there agreement? Global Spine Journal, 10(4), 499–501. https://doi.org/10.1177/2192568220911648

11.

Exceptional Children. (n.d.). Submission guidelines. https://journals.sagepub.com/author-instructions/ecx

12.

Feinstein

A. R.

Cicchetti

D. V.

(1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L

13.

Fien

Nelson

N. J.

Smolkowski

Kosty

Pilger

Baker

S. K.

Smith

J. L. M.

(2021). A conceptual replication study of the enhanced core reading instruction MTSS-reading model. Exceptional Children, 87(3), 265–288. https://doi.org/10.1177/0014402920953763

14.

Frias-Navarro

Pascual-Llobell

Pascual-Soler

Perezgonzalez

Berrios-Riquelme

(2020). Replication crisis or an opportunity to improve scientific production? European Journal of Education, 55(4), 618–631. https://doi.org/10.1111/ejed.12417

15.

Gage

N. A.

Stevens

R. N.

(2018). Rigor, replication, and reproducibility: Increasing the relevance of behavioral disorders research. Education and Treatment of Children, 41(4), 567–588. https://www.jstor.org/stable/26535291

16.

Gwet

K. L.

(2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. https://doi.org/10.1348/000711006X126600

17.

Institute of Education Sciences. (n.d.). Research grants focused on systematic replication in special education. https://ies.ed.gov/funding/ncser_rfas/ncser_systematic_replications.asp

18.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://doi.org/10.2307/2529310

19.

Lemons

C. J.

King

S. A.

Davidson

K. A.

Berryessa

T. L.

Gajjar

S. A.

Sacks

L. H.

(2016). An inadvertent concurrent replication: Same roadmap, different journey. Remedial and Special Education, 37(4), 213–222. https://doi.org/10.1177/0741932516631116

20.

Lloyd

J. W.

Therrien

W. J.

(2018). Preview and introduction of open-science guidelines. Exceptional Children, 85(1), 6–9. https://doi.org/10.1177/0014402918795348

21.

Luckner

J. L.

Banerjee

Movahedazarhouligh

Millen

(2020). A systematic review of replicative self-determination intervention studies. The Journal of Special Education, 54(1), 29–39. https://doi.org/10.1177/002246691985018

22.

Maggin

D. M.

(2022). Journal editor and associate editor perspectives on research reproducibility and open science. Remedial and Special Education, 43(3), 135–146. https://doi.org/10.1177/07419325211017294

23.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513

24.

Makel

M. C.

Plucker

J. A.

Freeman

Lombardi

Simonsen

Coyne

(2016). Replication of special education research: Necessary but far too rare. Remedial and Special Education, 37(4), 205–212. https://doi.org/10.1177/0741932516646083

25.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688

26.

Mellor

(2021). Improving norms in research culture to incentivize transparency and rigor. Educational Psychologist, 56(2), 122–131. https://doi.org/10.1080/00461520.2021.1902329

27.

Nosek

B. A.

Errington

T. M.

(2020). What is replication? PLOS Biology, 18(3), Article e3000691. https://doi.org/10.1371/journal.pbio.3000691

28.

Page

M. J.

Moher

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

Loder

E. W.

Mayo-Wilson

McDonald

. . . McKenzie

J. E.

(2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372. Article 160. https://doi.org/10.1136/bmj.n160

29.

Perry

Morris

Lea

(2022). A decade of replication study in education? A mapping review (2011–2020). Educational Research and Evaluation, 27(1–2), 12–34. https://doi.org/10.1080/13803611.2021.2022315

30.

Plucker

J. A.

Makel

M. C.

(2021). Replication is important for educational psychology: Recent developments and key issues. Educational Psychologist, 56(2), 90–100. https://doi.org/10.1080/00461520.2021.1895796

31.

Root

J. R.

Cox

S. K.

Davis

Hammons

(2020). Contextualizing mathematical problem-solving instruction for secondary students with extensive support needs: A systematic replication. Research and Practice for Persons with Severe Disabilities, 45(4), 241–255. https://doi.org/10.1177/1540796920949448

32.

Root

J. R.

Henning

Jimenez

(2020). Building the early number sense of kindergarteners with autism: A replication study. Remedial and Special Education, 41(6), 378–388. https://doi.org/10.1177/0741932519873121

33.

Schwab

Aguinis

Bamberger

Hodgkinson

G. P.

Shapiro

D. L.

Starbuck

W. H.

Tsui

A. S.

(2023). How replication studies can improve doctoral student education. Journal of Management Scientific Reports, 1(1), 18–41. https://doi.org/10.1177/27550311231156880

34.

Steiner

P. M.

Sheehan

Wong

V. C.

(2023). Correspondence measures for assessing replication success. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000597

35.

Therrien

W. J.

Mathews

H. M.

Hirsch

S. E.

Solis

(2016). Progeny review: An alternative approach for examining the replication of intervention studies in special education. Remedial and Special Education, 37(4), 235–243. https://doi.org/10.1177/0741932516646081

36.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication research and special education. Remedial and Special Education, 37(4), 195–204. https://doi.org/10.1177/0741932516648462

37.

Vazire

(2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884

38.

Wong

V. C.

Anglin

Steiner

P. M.

(2022). Design-based approaches to causal replication studies. Prevention Science, 23, 723–738. https://doi.org/10.1007/s11121-021-01234-7

39.

Yates

(1934). Contingency table involving small numbers and the χ2 test. Supplement to the Journal of the Royal Statistical Society, 1(2), 217–235. https://doi.org/10.2307/2983604

40.

Zwaan

R. A.

Etz

Lucas

R. E.

Donnellan

M. B.

(2018). Making replication mainstream. Behavioral and Brain Sciences, 41, Article e120. https://doi.org/10.1017/S0140525X17001972

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB