A growing number of statistical analyses are being developed for single-case research. One important factor in evaluating these methods is the extent to which each corresponds to visual analysis. Few studies have compared statistical and visual analysis, and information about more recently developed statistics is scarce. Therefore, our purpose was to evaluate the agreement between visual analysis and four statistical analyses: improvement rate difference (IRD); Tau-U; Hedges, Pustejovsky, Shadish (HPS) effect size; and between-case standardized mean difference (BC-SMD). Results indicate that IRD and BC-SMD had the strongest overall agreement with visual analysis. Although Tau-U had strong agreement with visual analysis on raw values, it had poorer agreement when those values were dichotomized to represent the presence or absence of a functional relation. Overall, visual analysis appeared to be more conservative than statistical analysis, but further research is needed to evaluate the nature of these disagreements.
AllisonD. B.GormanB. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy, 31, 621-631.
2.
Arizona Software. (2008). GraphClick (Version 3.0) [Computer software]. Neuchâtel, Switzerland: Author. Available from http://www.arizona-software.ch
3.
BaerD.M. (1977). Perhaps it would be better not to know everything. Journal of Applied Behavior Analysis, 10, 167-172. doi: 10.1901/jaba.1977.10-167
4.
BeretvasS. N.HyewonC. (2008). A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention, 2, 129-141. doi:10.1080/17489530802446302
5.
BrossartD. F.ParkerR. I.OlsonE. A.MahadevanL. (2006). The relationship between visual analysis and five statistical analyses in a simple AB single-case research design. Behavior Modification, 30, 531-563.
6.
CenterB. A.SkibaR. J.CaseyA. (1985). A methodology for the quantitative synthesis of intra-subject design research. The Journal of Special Education, 19, 387-400.
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
FaithM. S.AllisonD. B.GormanB. S. (1996). Meta-analysis of single-case research. In FranklinR. D.AllisonD. B.GormanB. S. (Eds.), Design and analysis of single-case research (pp. 245-277). Hillsdale, NJ: Lawrence Erlbaum.
9.
HartmannD. P.BarriosB. A.WoodD. D. (2004). Principles of behavioral observation. In HaynesS. N.HeibyE. M. (Eds.), Comprehensive handbook of psychological assessment (Vol. 3, pp. 108-127). New York, NY: John Wiley.
10.
HedgesL. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
11.
HedgesL. V.PustejovskyJ. E.ShadishW. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224-239.
12.
HedgesL. V.PustejovskyJ. E.ShadishW. R. (2013). A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods, 4, 324-341. doi:10.1002/jrsm.1086
13.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.
14.
KazdinA. E. (2011). Single-case research designs: Methods for clinical and applied settings. Oxford, UK: Oxford University Press.
15.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26-38.
16.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
17.
MaH. H. (2006). An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification, 30, 598-617.
18.
MagginD. M.PustejovskyJ. E.JohnsonA. H. (2017). A meta-analysis of school-based group contingency interventions for students with challenging behavior: An update. Remedial and Special Education, 38, 353-370. doi:10.1177/0741932517716900
MoeyaertM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2014). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52, 191-211.
21.
ParkerR. I.Hagan-BurkeS. (2007). Median-based overlap analysis for single case data: A second study. Behavior Modification, 31, 919-936.
22.
ParkerR. I.Hagan-BurkeS.VannestK. J. (2007). Percentage of All Non-overlapping Data (PAND): An alternative to PND. The Journal of Special Education, 40, 194-204.
23.
ParkerR. I.VannestK. J. (2007). Pairwise data overlap for single case research. Unpublished manuscript.
24.
ParkerR. I.VannestK. J. (2009). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357-367.
25.
ParkerR. I.VannestK. J.BrownL. (2009). The improvement rate difference for single-case research. Exceptional Children, 75, 135-150.
26.
ParkerR. I.VannestK. J.DavisJ. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322.
27.
ParkerR. I.VannestK. J.DavisJ. L.SauberS. B. (2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284-299.
28.
PustejovskyJ. E. (2015, April). Operational sensitivities of non-overlap effect sizes for single-case designs. Poster presented at the annual meeting of the American Educational Research Association, Chicago, IL.
29.
PustejovskyJ. E. (2016). Scdhlm: A web-based calculator for between-case standardized mean differences (Version 0.3.1) [Web application]. Retrieved from: https://jepusto.shinyapps.io/scdhlm
30.
PustejovskyJ. E.HedgesL. V.ShadishW. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39, 368-393. doi:10.3102/1076998614547577
31.
RakapS.SnyderP.PasiaC. (2014). Comparison of nonoverlap methods for identifying treatment effect in single-subject experimental research. Behavioral Disorders, 39, 128-145.
32.
R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
33.
ScruggsT. E.MastropieriM. A.CastoG. (1987). The quantitative synthesis of single-subject research: Methodology and validation. Remedial and Special Education, 8, 24-33.
34.
ShadishW. R. (2014). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23, 139-146.
35.
ShadishW. R.HedgesL. V.HornerR. H.OdomS. L. (2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research (NCER 2015-002). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. Available from https://ies.ed.gov/ncser/pubs/2015002/
36.
ValentineJ. C.Tanner-SmithE. E.PustejovskyJ. E.LauT. S. (2016). Between-case standardized mean difference effect sizes for single-case designs: A primer and tutorial using the scdhlm web application. Oslo, Norway: Campbell Collaboration. doi:10.4073/cmdp.2016.1
37.
VannestK. J.NinciJ. (2015). Evaluating intervention effects in single-case research designs. Journal of Counseling & Development, 93, 403-411.
38.
WendtO. (2012). R-code for non-parametric effect size metrics applicable to single-subject design data. (Unpublished software). West Lafayette, IN: Purdue University.
39.
WhiteD. M.RuschF. R.KazdinA. E.HartmannD. P. (1989). Applications of meta analysis in individual-subject research. Behavioral Assessment, 11, 281-296.
40.
WhiteO. R.HaringN. G. (1980). Exceptional teaching. Columbus, OH: Charles E. Merrill.
41.
WoleryM.BusickM.ReichowB.BartonE. E. (2008, February). Quantitative synthesis of single subject research. Paper presented at the Conference on Research Innovations in Early Intervention, San Diego, CA.
42.
WoleryM.BusickM.ReichowB.BartonE. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44, 18-28.
43.
WolfeK.SeamanM. A.DrasgowE. (2016). Interrater agreement on the visual analysis of individual tiers and functional relations in multiple baseline designs. Behavior Modification, 40, 852-873.
