Single-case data analysis still relies heavily on visual inspection, and, at the same time, it is not clear to what extent the results of different quantitative procedures converge in identifying an intervention effect and its magnitude when applied to the same data; this is the type of evidence provided here for two procedures. One of the procedures, included due to the importance of providing objective criteria to visual analysts, is a visual aid fitting and projecting split-middle trend while taking into account data variability. The other procedure converts several different metrics into probabilities making their results comparable. In the present study, we expore to what extend these two procedures coincide in the magnitude of intervention effect taking place in a set of studies stemming from a recent meta-analysis. The procedures concur to a greater extent with the values of the indices computed and with each other and, to a lesser extent, with our own visual analysis. For distinguishing smaller from larger effects, the probability-based approach seems somewhat better suited. Moreover, the results of the field test suggest that the latter is a reasonably good mechanism for translating different metrics into similar labels. User friendly R code is provided for promoting the use of the visual aid, together with a quantification based on nonoverlap and the label provided by the probability approach.
AllisonD. B.GormanB. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy, 31, 621-631.
2.
BorckardtJ. J.NashM. R.MurphyM. D.MooreM.ShawD.O’NeilP. (2008). Clinical practice as natural laboratory for psychotherapy research: A guide to case-based time-series analysis. American Psychologist, 63, 77-95.
3.
BrossartD. F.VannestK.DavisJ.PatienceM. (2014). Incorporating nonoverlap indices with visual analysis for quantifying intervention effectiveness in single-case experimental designs. Neuropsychological Rehabilitation, 24, 464-491.
4.
BultéI.OnghenaP. (2008). An R package for single-case randomization tests. Behavior Research Methods, 40, 467-478.
5.
BultéI.OnghenaP. (2009). Randomization tests for multiple-baseline designs: An extension of the SCRT-R package. Behavior Research Methods, 41, 477-485.
6.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155-159.
7.
DanovS. E.SymonsF. J. (2008). A survey evaluation of the reliability of visual inspection and functional analysis graphs. Behavior Modification, 32, 828-839.
8.
FisherW. W.KelleyM. E.LomasJ. E. (2003). Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis, 36, 387-406.
9.
FisherW. W.LermanD. C. (2014). It has been said that, “There are three degrees of falsehoods: Lies, damn lies, and statistics.”Journal of School Psychology, 52, 243-248.
10.
GastD. L.SpriggsA. D. (2010). Visual analysis of graphic data. !In GastD. L. (Ed.), Single subject research methodology in behavioral sciences (pp. 199-233). London, England: Routledge.
11.
GlassG. V.McGawB.SmithM. L. (1981). Meta-analysis in social research. Beverly Hills, CA: SAGE.
12.
GoodmanL. A.KruskalW. H. (1954). Measures of association for cross-classification. Journal of the American Statistical Association, 49, 732-764.
13.
HedgesL. V.PustejovskyJ. E.ShadishW. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224-239.
14.
HedgesL. V.PustejovskyJ. E.ShadishW. R. (2013). A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods, 4, 324-341.
15.
HeyvaertM.OnghenaP. (2014). Analysis of single-case data: Randomisation tests for measures of effect size. Neuropsychological Rehabilitation, 24, 507-527.
16.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.
17.
HowickJ.ChalmersI.GlasziouP.GreenhaighT.HeneghanC.LiberatiA.… . . .,ThorntonH. (2011). The 2011 Oxford CEBM evidence table (Introductory document). Oxford Centre for Evidence-Based Medicine. Retrieved from http://www.cebm.net/index.aspx?o=5653
18.
HuitemaB. E.McKeanJ. W. (1991). Autocorrelation estimation and inference with small samples. Psychological Bulletin, 110, 291-304.
19.
JamiesonM.CullenB.McGee-LennonM.BrewsterS.EvansJ. J. (2014). The efficacy of cognitive prosthetic technology for people with memory impairments: A systematic review and meta-analysis. Neuropsychological Rehabilitation, 24, 419-444.
20.
JensonW. R.ClarkE.KircherJ. C.KristjanssonS. D. (2007). Statistical reform: Evidence-based practice, meta-analyses, and single subject designs. Psychology in the Schools, 44, 483-493.
21.
KratochwillT. R.HitchcockJ. H.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single case designs technical documentation. In What Works Clearinghouse: Procedures and standards handbook (Version 2.0). Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_scd.pdf
22.
LabelleK.-L.MihailidisA. (2006). The use of automated prompting to facilitate handwashing in persons with dementia. American Journal of Occupational Therapy, 60, 442-450.
23.
ManolovR.GastD. L.PerdicesM.EvansJ. J. (2014). Single-case experimental designs: Reflections on conduct and analysis. Neuropsychological Rehabilitation, 24, 634-660.
24.
ManolovR.JamiesonM.EvansJ. J.SierraV. (2015). Establishing empirical benchmarks for interpreting single-case effect sizes: An illustration of a method. Manuscript submitted for publication.
25.
ManolovR.SierraV.SolanasA.BotellaJ. (2014). Assessing functional relations in single-case designs: Quantitative proposals in the context of the evidence-based movement. Behavior Modification, 38, 878-913.
26.
ManolovR.SolanasA. (2012). Assigning and combining probabilities in single-case studies. Psychological Methods, 17, 495-509.
27.
ManolovR.SolanasA. (2013a). A comparison of mean phase difference and generalized least squares for analyzing single-case data. Journal of School Psychology, 51, 201-215.
28.
ManolovR.SolanasA. (2013b). Assigning and combining probabilities in single-case studies: A second study. Behavior Research Methods, 45, 1024-1035.
29.
MatyasT. A.GreenwoodK. M. (1991). Problems in the estimation of autocorrelation in brief time series and some implications for behavioral data. Behavioral Assessment, 13, 137-157.
30.
MercerS. H.SterlingH. E. (2012). The impact of baseline trend control on visual analysis of single-case data. Journal of School Psychology, 50, 403-419.
31.
MillerM. J. (1985). Analyzing client change graphically. Journal of Counseling & Development, 63, 491-494.
32.
MoeyaertM.FerronJ.BeretvasS.Van Den NoortgateW. (2014). From a single-level analysis to a multilevel analysis of since-case experimental designs. Journal of School Psychology, 52, 191-211.
33.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van Den NoortgateW. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48, 719-748.
34.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2014). The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental designs research. Behavior Modification, 38, 665-704.
35.
NinciJ.VannestK. J.WillsonV.ZhangN. (2015). Interrater agreement between visual analysts of single-case data: A meta-analysis. Behavior Modification, 39, 510-541.
36.
OttenbacherK. J. (1993). Interrater agreement of visual analysis in single-subject decisions: Quantitative review and analysis. American Journal of Mental Retardation, 98, 135-142.
37.
ParkerR. I.BrossartD. F.VannestK. J.LongJ. R.Garcia De-AlbaR.BaughF. G.SullivanJ. R. (2005). Effect sizes in single case research: How large is large?School Psychology Review, 34, 116-132.
38.
ParkerR. I.CryerJ.ByrnsG. (2006). Controlling baseline trend in single-case research. School Psychology Quarterly, 21, 418-443.
39.
ParkerR. I.VannestK. J. (2009). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357-367.
40.
ParkerR. I.VannestK. J.DavisJ. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322.
41.
ParkerR. I.VannestK. J.DavisJ. L. (2014). A simple method to control positive baseline trend within data nonoverlap. Journal of Special Education, 48, 79-91.
42.
ParkerR. I.VannestK. J.DavisJ. L.SauberS. B. (2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284-299.
43.
PfadtA.WheelerD. J. (1995). Using statistical process control to make data-based clinical decisions. Journal of Applied Behavior Analysis, 28, 349-370.
44.
R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/
45.
ReichowB.VolkmarF.CicchettiD. (2008). Development of the evaluative method for evaluating and determining evidence-based practices in autism. Journal of Autism and Developmental Disorders, 38, 1311-1319.
46.
ScruggsT. E.MastropieriM. A.CastoG. (1987). The quantitative synthesis of single-subject research: Methodology and validation. Remedial and Special Education, 8, 24-33.
47.
ShadishW. R. (2014). Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology, 52, 109-122.
48.
ShadishW. R.HedgesL. V.PustejovskyJ. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52, 123-147.
49.
ShadishW. R.RindskopfD. M.HedgesL. V.SullivanK. J. (2013). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 45, 813-821.
50.
ShadishW. R.SullivanK. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971-980.
51.
SmithJ. D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510-550.
52.
SolanasA.ManolovR.OnghenaP. (2010). Estimating slope and level change in N = 1 designs. Behavior Modification, 34, 195-218.
53.
SolanasA.ManolovR.SierraV. (2010). Lag-one autocorrelation in short series: Estimation and hypothesis testing. Psicológica, 31, 357-381.
54.
SolomonB. G. (2014). Violations of assumptions in school-based single-case data: Implications for the selection and interpretation of effect sizes. Behavior Modification, 38, 477-496.
55.
TateR. L.PerdicesM.RosenkoetterU.WakimaD.GodbeeK.TogherL.McDonaldS. (2013). Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychological Rehabilitation, 23, 619-638.
56.
TukeyJ. W. (1977). Exploratory data analysis. London, England: Addison-Wesley.
57.
WoleryM.BusickM.ReichowB.BartonE. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-29.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
