Multilevel modeling is an approach that can be used to summarize single-case experimental design (SCED) data. Multilevel models were developed to analyze hierarchical structured data with units at a lower level nested within higher level units. SCEDs use time series data collected from multiple cases (or subjects) within a study that allow researchers to investigate intervention effectiveness at the individual level and also to investigate how these individual intervention effects change over time. There is an increased interest in the field regarding how SCEDs can be used to establish an evidence base for interventions by synthesizing data from a series of intervention studies. Although using multilevel models to meta-analyze SCED studies is promising, application is often hampered by being potentially excessively technical. First, this article provides an accessible description and overview of the potential of multilevel meta-analysis to combine SCED data. Second, a summary of the methodological evidence on the performance of multilevel models for meta-analysis is provided, which is useful given that such evidence is currently scattered over multiple technical articles in the literature. Third, the actual steps to perform a multilevel meta-analysis are outlined in a brief practical guide. Fourth, a suggestion for integrating the quantitative results with a visual representation is provided.
BaekE. K.FerronJ. M. (2013). Multilevel models for multiple-baseline data: Modeling across-participant variation in autocorrelation and residual variance. Behavior Research Methods, 45, 65-74.
2.
BaekE. K.Petit-BoisM.Van den NoortgateW.BeretvasS. N.FerronJ. M. (2016). Using visual analysis to evaluate and refine multilevel models of single-case studies. The Journal of Special Education, 50, 18-26.
3.
BeesonP. M.RobeyR. R. (2006). Evaluating single-subject treatment research: Lessons learned from the aphasia literature. Neuropsychological Review, 16, 161-169.
4.
BlampiedN. M. (2017). Analyzing therapeutic change using modified Brinley plots: History, construction, and interpretation. Behavior Therapy, 48, 115-127.
5.
Bowman-PerrottL.BurkeM. D.ZhangN.ZainiS. (2014). Direct and collateral effects of peer tutoring on social and behavioral outcomes: A meta-analysis of single-case research. School Psychology Review, 43, 260-285.
6.
BuskP. L.MarascuiloL. A. (1988). Autocorrelation in single-subject research: A counterargument to the myth of no autocorrelation. Behavioral Assessment, 10, 229-242.
7.
ChowS.-M.HoijtinkH. (2017). Bayesian estimation and modeling: Editorial to the second special issue on Bayesian data analysis. Psychological Methods, 22, 609-615.
8.
DeclercqL.CoolsW.BeretvasS. N.MoeyaertM.FerronJ. M.Van den NoortgateW. (2019). MultiSCED: A tool for (meta-) analyzing single-case experimental data. Behavior Research Methods. doi: 10.3758/s13428-019-01216-2
9.
DedrickR. F.FerronJ. M.HessM. R.HogartyK. Y.KromreyJ. D.LangT. R.. . . LeeR. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79, 69-102.
10.
DixonM. A.CunninghamG. B. (2006). Data aggregation in multilevel analysis: A review of conceptual and statistical issues. Measurement in Physical Education and Exercise Science, 10, 85-107.
11.
DuvalS. J. (2005). The trim and fill method. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 127-144). Chichester, UK: John Wiley.
12.
DuvalS. J.TweedieR. L. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
13.
DuvalS. J.TweedieR. L. (2000b). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455-463.
14.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634.
15.
FerronJ. M.BellB. A.HessM. R.Rendina-GobioffG.HibbardS. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41, 372-384.
16.
FerronJ. M.FarmerJ. L.OwensC. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study for multilevel-modeling approaches. Behavior Research Methods, 42, 930-943.
17.
FoxJ. (2016). Applied regression analysis and generalized linear models (3rd ed.). London, England: SAGE.
18.
GageN. A.LewisT. J. (2014). Hierarchical linear modeling meta-analysis of single-subject design research. Journal of Special Education, 48, 3-16.
19.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65.
20.
HedgesL. V.VeveaJ. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
21.
HembryI.BunuanR.BeretvasS. N.FerronJ. M.Van den NoortgateW. (2015). Estimation of a nonlinear intervention phase trajectory for multiple-baseline design data. The Journal of Experimental Education, 83, 514-546.
22.
HeyvaertM.SaenenL.MaesB.OnghenaP. (2014). Systematic review of restraint interventions for challenging behaviour among persons with intellectual disabilities: Focus on effectiveness in single-case experiments. Journal of Applied Research in Intellectual Disabilities, 27, 493-510.
23.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.
24.
HowickJ.ChalmersI.GlasziouP.GreenhaighT.HeneghanC.LiberatiA.. . . ThorntonH. (2011). The 2011 Oxford CEBM evidence table (Introductory document). Oxford Centre for Evidence-Based Medicine. Retrieved from http://www.cebm.net/index.aspx?o=5653
25.
HoxJ. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
26.
HuitemaB. E. (1985). Autocorrelation in behavior analysis: A myth. Behavioral Assessment, 7, 107-118.
27.
HuitemaB. E.McKeanJ. W. (2000). Design specification issues in time-series intervention models. Educational and Psychological Measurement, 60, 38-58.
28.
JamshidiL.HeyvaertM.DeclercqL.Fernández CastillaB.FerronJ. M.MoeyaertM.. . . Van den NoortgateW. (2018). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities, 79, 97-115.
29.
JensonW. R.ClarkE.KircherJ. C.KristjanssonS. D. (2007). Statistical reform: Evidence-based practice, meta-analyses, and single subject designs. Psychology in the Schools, 44, 483-493.
30.
KennedyC. H. (2005). Single-case designs for educational research. Boston, MA: Pearson.
KratochwillT. R.LevinJ. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124-144.
33.
KreftI. G. G.de LeeuwJ.AikenL. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1-21.
34.
LedfordJ. R.GastD. L. (2014). Measuring procedural fidelity in behavioural research. Neuropsychological Rehabilitation, 24, 332-348.
35.
LevinJ. R.FerronJ. M.GafurovB. S. (2017). Additional comparisons of randomization-test procedures for single-case multiple-baseline designs: Alternative effect types. Journal of School Psychology, 63, 13-34.
36.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
37.
LoboM.MoeyaertM.Baraldi CunhaA.BabikI. (2017). Single-case experimental design, analysis, and quality assessment for intervention research. Journal of Neurologic Physical Therapy, 14, 187-197. doi:10.1097/NPT.0000000000000187
38.
MagginD. M.BrieschA. M.ChafouleasS. M.FergusonT. D.ClarkC. (2014). A comparison of rubrics for identifying empirically supported practices with single-case research. Journal of Behavioral Education, 23, 287-311.
39.
MagginD. M.LaneK. L.PustejovskyJ. E. (2017). Introduction to the special issue on single-case systematic reviews and meta-analyses. Remedial and Special Education, 38, 323-330.
40.
MagginD. M.O’KeefeB. V.JohnsonA. H. (2011). A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985–2009. Exceptionality, 19, 109-135.
41.
MagginD. M.PustejovskyJ. E.JohnsonA. H. (2017). A meta-analysis of school-based group contingency interventions for students with challenging behavior: An update. Remedial and Special Education, 38, 353-370.
42.
ManolovR. (2018). Linear trend in single-case visual and quantitative analyses. Behavior Modification, 42, 684-706.
43.
MoeyaertM.FerronJ. M.BeretvasS.Van den NoortgateW. (2014). From a single-level analysis to a multilevel analysis of since-case experimental designs. Journal of School Psychology, 52, 191-211.
44.
MoeyaertM.RindskopfD.OnghenaP.Van den NoortgateW. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods, 22, 760-778.
45.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS.Van den NoortgateW. (2013a). Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study. Behavior Research Methods, 45, 547-559.
46.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS.Van den NoortgateW. (2013b). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48, 719-748.
47.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2014a). The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental designs research. Behavior Modification, 38, 665-704.
48.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2014b). Three-level analysis of single-case experimental data: Empirical validation. The Journal of Experimental Education, 82, 1-21.
49.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van den NoortgateW. (2016). The misspecification of the covariance structures in multilevel models for single-case data: A Monte Carlo simulation study. The Journal of Experimental Education, 84, 473-509.
50.
OnghenaP.MichielsB.JamshidiL.MoeyaertM.Van den NoortgateW. (2018). One by one: Accumulating evidence by using meta-analytical procedures for single-case experiments. Brain Impairment, 19, 33-58.
51.
OwensC. M.FerronJ. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44, 795-805.
52.
ParkerR. I.CryerJ.ByrnsG. (2006). Controlling baseline trend in single-case research. School Psychology Quarterly, 21, 418-443.
53.
Petit-BoisM.BaekE. K.Van den NoortgateW.BeretvasS. N.FerronJ. M. (2016). The consequences of modeling autocorrelation when synthesizing single-case studies using a three-level model. Behavior Research Methods, 48, 803-812.
54.
PustejovskyJ. E.FerronJ. M. (2017). Research synthesis and meta-analysis of single-case designs. In KauffmanJ. M.HallahanD. P.PullenP. C. (Eds.), Handbook of special education (2nd ed., pp. 168-186). New York, NY: Routledge.
55.
PustejovskyJ. E.HedgesL. V.ShadishW. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39, 368-393.
56.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks, CA: SAGE.
57.
ReichowB.BartonE. E.MagginD. M. (2018). Development and applications of the single-case design risk of bias tool for evaluating single-case design research study reports. Research in Developmental Disabilities, 79, 53-64.
58.
SchlosserR. W. (2005). Meta-analysis of single-subject research: How should it be done?International Journal of Language & Communication Disorders, 40, 375-378.
59.
SchlosserR. W.LeeD. L.WendtO. (2008). Application of the percentage of non-overlapping data (PND) in systematic reviews and meta-analyses: A systematic review of reporting characteristics. Evidence-Based Communication Assessment and Intervention, 2, 163-187.
60.
ShadishW. R.HedgesL. V.PustejovskyJ. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52, 123-147.
61.
ShadishW. R.KyseE. N.RindskopfD. M. (2013). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18, 385-405.
62.
ShadishW. R.RindskopfD. M.HedgesL. V.SullivanK. J. (2013). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 45, 813-821.
63.
ShadishW. R.SullivanK. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971-980.
64.
SharpleyC. F.AlavosiusM. P. (1988). Autocorrelation in behavior data: An alternative perspective. Behavioral Assessment, 10, 243-251.
65.
ShawB. A.LiangJ. (2012). Growth models with multilevel regression. In NewsomJ. T.JonesR. N.HoferS. M. (Eds.), Longitudinal data analysis: A practical guide for researchers in aging, health, and social sciences (pp. 217-242). Hove, UK: Routledge.
66.
SmithJ. D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510-550.
67.
SnijdersT. A. B.BoskerR. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London, UK: SAGE.
68.
SolmiF.OnghenaP. (2014). Combining p-values in replicated single-case experiments with multivariate outcome. Neuropsychological Rehabilitation, 24, 607-633.
69.
SolomonB. G. (2014). Violations of assumptions in school-based single-case data: Implications for the selection and interpretation of effect sizes. Behavior Modification, 38, 477-496.
70.
TateR. L.PerdicesM.RosenkoetterU.WakimD.GodbeeK.TogherL.McDonaldS. (2013). Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychological Rehabilitation, 23, 619-638.
71.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ. M.Van den NoortgateW. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44, 1244-1254.
72.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ. M.Van den NoortgateW. (2014). Bias corrections for standardized effect size estimates used with single-subject experimental designs. The Journal of Experimental Education, 82, 358-374.
73.
Van den NoortgateW.OnghenaP. (2003a). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, & Computers, 35, 1-10.
74.
Van den NoortgateW.OnghenaP. (2003b). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63, 765-790.
75.
Van den NoortgateW.OnghenaP. (2005). Parametric and nonparametric bootstrap methods for meta-analysis. Behavior Research Methods, Instruments & Computers, 37, 11-22.
76.
Van den NoortgateW.OnghenaP. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence Based Communication Assessment and Intervention, 2, 142-151.
77.
VanderkerkenL.HeyvaertM.MaesB.OnghenaP. (2013). Psychosocial interventions for reducing vocal challenging behavior in persons with autistic disorder: A multilevel meta-analysis of single-case experiments. Research in Developmental Disabilities, 34, 4515-4533.
78.
WangS. Y.ParrilaR.CuiY. (2013). Meta-analysis of social skills interventions of single-case research for individuals with autism spectrum disorders: Results from three-level HLM. Journal of Autism and Developmental Disorders, 43, 1701-1716.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
