Overlap-based measures are increasingly applied in the synthesis of single-subject research. This article considers two criticisms of overlap-based metrics, specifically that they do not measure magnitude of effect and do not adequately correspond with visual analysis. It is argued that these criticisms are based on fundamental misconceptions regarding the nature of effect sizes and their appropriate interpretation in single-subject research. Suggestions for considerations in evaluating single-subject research studies are offered, including the need to separately consider experimental control and magnitude of effect.
AllisonD. B.GormanB. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behavior Research and Therapy, 31, 621-631. doi:10.1016/0005-7967(93)90115-B
2.
BeretvasS. N.ChungH. (2008). A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention, 2, 129-141. doi:10.1080/17489530802446302
3.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.
4.
BuskP. L.SerlinR. C. (1992). Meta-analysis for single-case research. In KratochwillT. R.LevinJ. R. (Eds.), Single-case research design and analysis: New directions for psychology and education. Hillsdale, NJ: Lawrence Erlbaum.
5.
CampbellD. T.StanleyJ. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
6.
CarterM. (2009). Effects of graphing conventions and response options on interpretation of small n graphs. Educational Psychology, 29, 643-658. doi:10.1080/ 01443410903204315
7.
CenterB. J.SkibaR. J.CaseyA. (1985). A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education, 19, 387-400. doi:10.1177/002246698501900404
8.
CritchfieldT. S.NewlandM. C.KollinsS. H. (2000). The good, the bad, and the aggregate. Behavior Analyst, 23, 107-115.
9.
DeProsperoA.CohenS. (1979). Inconsistent visual analyses of intrasubject data. Journal of Applied Behavior Analysis, 12, 573-579. doi:10.1901/jaba.1979.12-573
10.
FischG. S. (1998). Visual inspection of data revisited: Do the eyes still have it?Behavior Analyst, 21, 111-123.
11.
GanzJ. B.Earles-VollrathT. L.HeathA. K.ParkerR. I.RispoliM. J.DuranJ. B. (2011). A meta-analysis of single case research studies on aided augmentative and alternative communication systems with individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 60-74. doi:10.1007/s10803-011-1212-2
12.
GerstenR.FuchsL. S.ComptonD.CoyneM.GreenwoodC.InnocentiM. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149-164.
13.
HaardorferR. (2010). Concerns with using Cohen’s d and PND in single-case data analysis. Focus on Autism and Other Developmental Disabilities, 25, 125-127. doi:10.1177/1088357610371274
14.
HarveyS.BoerD.MeyerL.EvansI. (2009). Updating a meta-analysis of intervention research with challenging behaviour: Treatment validity and standards of practice. Journal of Intellectual & Developmental Disability, 34, 67-80. doi:10.1080/13668250802690922
15.
HattieJ. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, England: Routledge.
16.
HigginsJ. P. T.GreenS. (2009). Cochrane handbook for systematic reviews of interventions (Version 5.0.2). Retrieved from http://www.cochrane-handbook.org/
17.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.
18.
KahngS.ChungK. M.GutshallK.PittsS. C.KaoJ.GirolamiK. (2010). Consistent visual analyses of intrasubject data. Journal of Applied Behavior Analysis, 43, 35-45. doi:10.1901/jaba.2010.43-35
19.
KavaleK. A.FornessS. R. (1999). Efficacy of special education and related services. Washington, DC: AAMR.
20.
KazdinA. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York, NY: Oxford University Press.
21.
KennedyC. H. (2005). Single-case designs for education research. Boston, MA: Allyn & Bacon.
22.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
23.
LeongH. M.CarterM. (2008). Research on the efficacy of sensory integration therapy: Past, present and future. Australasian Journal of Special Education, 32, 83-99. doi:10.1080/10300110701842653
24.
LipsyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
25.
MaH. H. (2006). An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification, 30, 598-617. doi:10.1177/0145445504272974
26.
MaH. H. (2009). The effectiveness of intervention on the behavior of individuals with autism: A meta-analysis using percentage of data points exceeding the median of baseline phase (PEM). Behavior Modification, 33, 339-359. doi:10.1177/0145445509333173
27.
MatyasT. A.GreenwoodK. M. (1990). Visual analysis of single-case time series: Effects of variability, serial dependence and magnitude of intervention effects. Journal of Applied Behavior Analysis, 23, 341-351. doi:10.1901/jaba.1990.23-341
28.
OttenbacherK. (1986). Reliability and accuracy of visually analyzing graphed data from single-subject designs. American Journal of Occupational Therapy, 40, 464-469.
29.
ParkerR. I.Hagan-BurkeS. (2007). Median-based overlap analysis for single case data: A second study. Behavior Modification, 31, 919-936. doi:10.1177/ 0145445507303452
30.
ParkerR. I.Hagan-BurkeS.VannestK. (2006). Percentage of all non-overlapping data (PAND): An alternative to PND. Journal of Special Education, 40, 194-204.
31.
ParkerR. I.VannestK. J.BrownL. (2009). The improvement rate difference for single-case research. Exceptional Children, 75, 135-150.
32.
ParkerR. I.VannestK. J.DavisJ. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322. doi:10.1177/0145445511399147
33.
ParkerR. I.VannestK. J.DavisJ. L.SauberS. B. (2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284-299. doi:10.1016/j.beth.2010.08.006
34.
PrestonD.CarterM. (2009). A review of the efficacy of the picture exchange communication system intervention. Journal of Autism and Developmental Disorders, 39, 1471-1486. doi:10.1007/s10803-009-0763-y
35.
ReynhoutG.CarterM. (2011). Evaluation of the efficacy of social stories (TM) using three single subject metrics. Research in Autism Spectrum Disorders, 5, 885-900. doi:10.1016/j.rasd.2010.10.003
36.
SalzbergC. L.StrainP. S.BaerD. M. (1987). Meta-analysis for single-subject research: When does it clarify, when does it obscure?Remedial and Special Education, 8(2), 43-48.
37.
SchneiderN.GoldsteinH.ParkerR. (2008). Social skills interventions for children with autism: A meta-analytic application of percentage of all non-overlapping data (PAND). Evidence-Based Communication Assessment and Intervention, 2, 152-168. doi:10.1080/17489530802505396
38.
ScruggsT. E.MastropieriM. A.CastoG. (1987a). Reply to Owen White. Remedial and Special Education, 8(2), 40-42. doi:10.1177/074193258700800208
39.
ScruggsT. E.MastropieriM. A.CastoG. (1987b). The quantitative synthesis of single-subject research: Methodology and validation. Remedial and Special Education, 8(2), 24-33. doi:10.1177/074193258700800206
40.
SlavinR. E. (1987). Mastery learning reconsidered. Review of Educational Research, 57, 175-213.
41.
WhiteO. R. (1987). Some comments concerning “The quantitative synthesis of single-subject research.”Remedial and Special Education, 8(2), 34-39. doi:10.1177/074193258700800207
42.
WoleryM.BusickM.ReichowB.BartonE. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-28. doi:10.1177/0022466908328009
