Critical appraisal scales play an important role in evaluating methodological rigor (MR) of between-groups and single-case designs (SCDs). For intervention research this forms an essential basis for ascertaining the strength of evidence. Yet, few such scales provide classifications that take into account the differential weighting of items contributing to internal validity. This study aimed to develop an algorithm derived from the Risk of Bias in N-of-1 Trials (RoBiNT) Scale to classify MR and risk of bias magnitude in SCDs. The algorithm was applied to 46 SCD experiments. Two experiments (4%) were classified as Very High MR, 14 (30%) as High, 5 (11%) as Moderate, 2 (4%) as Fair, 2 (4%) as Low, and 21 (46%) as Very Low. These proportions were comparable to the What Works Clearinghouse classifications: 13 (28%) met standards, 8 (17%) met standards with reservations, and 25 (54%) did not meet standards. There was strong association between the two classification systems.
*BarkaiaA.StokesT. F.MikiashviliT. (2017). Intercontinental telehealth coaching by therapists to improve verbalizations by children with autism. Journal of Applied Behavior Analysis, 50(3), 582–589.
2.
BarlowD. H.NockM. K.HersenM. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Boston, MA: Pearson.
3.
*BroganK. M.FalligantJ. M.RappJ. M. (2017). Interdependent group contingencies decrease adolescents’ disruptive behaviors during group therapy: A practitioner’s demonstration. Behavior Modification, 41(3), 405–421.
4.
ByiersB. J.ReichleJ.SymonsF. J. (2012). Single-subject experimental designs for evidence-based practice. American Journal of Speech-Language Pathology, 21(4), 397–414.
5.
CookB.BuysseV.KlingnerJ.LandrumT.McWilliamR.TankersleyM.TestD. W. (2014). Council for exceptional children: Standards for evidence-based practices in special education. Teaching Exceptional Children, 46, 206–212.
6.
*DixonM. R.BelisleJ.StanleyC. R.SpeelmanR. C.RowseyK. E.KimeD.DaarJ. H. (2017). Establishing derived categorical responding in children with disabilities using the PEAK-E curriculum. Journal of Applied Behavior Analysis, 50(1), 134–145.
7.
*DoganR. K.KingM. L.FischettiA. T.LakeC. M.MathewsT. L.WarzakW. J. (2017). Journal of Applied Behavior Analysis, 50(4), 805–818.
8.
*DonnellyM. G.KarsteA. M. (2017). Effects of programmed teaching errors on acquisition and durability of self-care skills. Journal of Applied Behavior Analysis, 50(3), 511–528.
9.
GastD. L.BaekeyD. H. (2014). Withdrawal and reversal designs. In GastD. L.LedfordJ. R. (Eds.), Single case research methodology: Applications in special education and behavioral sciences (2nd ed., pp. 211–250). New York: Routledge.
10.
*GerencserK. R.HigbeeT. S.AkersJ. S.ConterasB. P. (2017). Evaluation of interactive computerized training to teach parents to implement photographic activity schedules with children with autism spectrum disorder. Journal of Applied Behavior Analysis, 50(3), 567–581.
11.
*GilesA.MarkhamV. (2017). Comparing book- and tablet-based picture activity schedules: Acquisition and preference. Behavior Modification, 41(5), 647–664.
12.
*HansenB. D.CaldarellaP.WilliamsL.WillsH. P. (2017). Managing student behaviour in dual immersion classrooms: A study of class-wide function-related intervention teams. Behavior Modification, 41(5), 626–646.
13.
HigginsJ. P. T.AltmanD. G. (2008). Assessing risk of bias in included studies. In HigginsJ. P. T.GreenS. (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 187–242). Chichester, England: John Wiley & Sons.
14.
*HigginsW. J.LuczynskiK. C.CarrollR. A.FisherW. W.MudfordO. C. (2017). Evaluation of a telehealth training package to remotely train staff to conduct a preference assessment. Journal of Applied Behavior Analysis, 50(2), 238–251.
15.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179.
16.
*JohnsonK. A.VladescuJ. C.KodakT.SidenerT. M. (2017). As assessment of differential reinforcement procedures for learners with autism spectrum disorder. Journal of Applied Behavior Analysis, 50(2), 290–303.
17.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.
18.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34(1), 26–38.
19.
KratochwillT. R.LevinJ. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15(2), 124–144.
20.
*LancioniG. E.SinghN. N.O’ReillyM. F. O.SigafoosJ.D/AmicoF.AddanteL. M.PintoK. (2017). Persons with advanced Alzheimer’s disease engage in mild leg exercise supported by technology-aided stimulation and prompts. Behavior Modification, 41(1), 3–20.
21.
*LeafJ. B.CihonJ. H.AlcalayA.MitchellE.Townley-CochraneD.MillerK …McEahinJ. (2017). Instructive feedback embedded within group instruction for children diagnosed with autism spectrum disorder. Journal of Applied Behavior Analysis, 50(2), 304–316.
22.
LedfordJ. R.LaneJ. D.ZimmermanK. N.ChazinK. T.AyresK. A. (2016). Single case analysis and review framework (SCARF). Retrieved from http://vkc.mc.vanderbilt.edu/ebip/scarf/
23.
LoganL. R.HickmanR. R.HarrisS. R.HerizaC. B. (2008). Single-subject research design: Recommendations for levels of evidence and quality rating. Developmental Medicine and Child Neurology, 50(2), 99–103.
24.
MagginD. M.BrieschA. M.ChafouleasS. M.FergusonT. D.ClarkC. (2014). A comparison of rubrics for identifying empirically supported practices with single-case research. Journal of Behavioral Education, 23, 287–311.
25.
MoseleyA.HerbertR. D.SherringtonC.MaherC. G. (2002). Evidence for physiotherapy practice: A survey of the Physiotherapy Evidence Database (PEDro). Australian Journal of Physiotherapy, 48, 43–49.
26.
*NottinghamC. L.VladescuJ. C.KodakT.KisamoreA. N. (2017). Incorporating multiple secondary targets into learning trials for individuals with autism spectrum disorder. Journal of Applied Behavior Analysis, 50(3), 653–661.
27.
Oxford Centre for Evidence-Based Medicine. (2011). The Oxford 2011 levels of evidence. Retrieved from www.cebm.net/index.aspx?o=5653
28.
*RadleyK. C.DartE. H.MooreJ. W.BattagliaA. A.LaBrotZ. C. (2017). Promoting accurate variability of social skills in children with autism spectrum disorder. Behavior Modification, 41(1), 84–112.
29.
ReichowB.BartonE. E.MagginD. M. (2018). Development and applications of the single-case design risk of bias tool for evaluating single-case design research study reports. Research in Developmental Disabilities, 79, 53–64.
30.
ReichowB.VolkmarF. R.CicchettiD. V. (2008). Development of the evaluative method for evaluating and determining evidence-based practices in autism. Journal of Autism and Developmental Disorders, 38(7), 1311–1319.
31.
*ScalzoR.DavisT. N. (2017). Analysis of behavioural indicators as a measure of satiation. Behavior Modification, 41(2), 308–322.
ShadishW. R.SullivanK. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43(4), 971–980.
35.
SimeonssonR.BaileyD. (1991). Evaluating programme impact: Levels of certainty. In MitchellD.BrownR. (Eds.), Early intervention studies for young children with special needs (pp. 280–296). London, England: Chapman and Hall.
36.
SmithV.JelenM.PattersonS. (2010). Video modeling to improve play skills in a child with autism: A procedure to examine single-subject experimental research. Evidence-Based Practice Briefs, 4, 1–11.
37.
*StasollaF.CaffoA. O.PerilliV.BoccasiniA.StellaA.DamianiR. …DamatoC. (2017). Microswitch-based program for promoting initial ambulation responses: An evaluation with two girls with multiple disabilities. Journal of Applied Behavior Analysis, 50(2), 345–356.
38.
TateR. L.McDonaldS.PerdicesM.TogherL.SchultzR.SavageS. (2008). Rating the methodological quality of single-subject designs and n-of-1 trials: Introducing the Single-Case Experimental Design (SCED) Scale. Neuropsychological Rehabilitation, 18(4), 385–401.
39.
TateR. L.PerdicesM. (2019). Single-case experimental designs for clinical research and neurorehabilitation settings: Planning, conduct, analysis and reporting. Abingdon, England: Routledge.
40.
TateR. L.PerdicesM.RosenkoetterU.ShadishW.BarlowD. H.HornerR., . . . WilsonB. (2016). The Single-Case Reporting guideline In BEhavioural Interventions (SCRIBE) 2016 Statement. Archives of Scientific Psychology, 4, 1–9.
41.
TateR. L.PerdicesM.RosenkoetterU.WakimD.GodbeeK.TogherL.McDonaldS. (2013). Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychological Rehabilitation, 23(5), 619–638.
42.
TateR. L.RosenkoetterU.WakimD.SigmundsdottirL.DoubledayJ.TogherL.McDonaldS.PerdicesM. (2015). The Risk of Bias in N-of-1 Trials (RoBiNT) Scale: An expanded manual for the critical appraisal of single-case reports. Sydney: Author.
43.
*ToussaintK. A.ScheithauerM. C.TigerJ. H.SaundersK. J. (2017). Teaching identity matching of Braille characters to beginning Braille readers. Journal of Applied Behavior Analysis, 50(2), 278–289.
44.
WendtO.MillerB. (2012). Quality appraisal of single-subject experimental designs: An overview and comparison of different appraisal tools. Education and Treatment of Children, 35(2), 235–268.
45.
WoleryM. (2013). A commentary: Single-case design technical document of the What Works Clearinghouse. Remedial and Special Education, 43(1), 39–43.
46.
*ZergerH. M.MillerB. G.ValbuenaD.MiltenbergerR. G. (2017). Effects of student pairing and public review on physical activity during school recess. Journal of Applied Behavior Analysis, 50(3), 529–537.
47.
ZimmermanK. N.LedfordJ. R.SeveriniK. E.PustejovskyJ. E.BartonE. E.LloydB. P. (2018). Single-case synthesis tools I: Comparing tools to evaluate SCD quality and rigor. Research in Developmental Disabilities, 79, 19–32.
