Multilevel modeling (MLM) is an approach for meta-analyzing single-case experimental designs (SCED). In this paper, we provide a step-by-step guideline for using the MLM to meta-analyze SCED time-series data. The MLM approach is first presented using a basic three-level model, then gradually extended to represent more realistic situations of SCED data, such as modeling a time variable, moderators representing different design types and multiple outcomes, and heterogeneous within-case variance. The presented approach is then illustrated using real SCED data. Practical recommendations using the MLM approach are also provided for applied researchers based on the current methodological literature. Available free and commercial software programs to meta-analyze SCED data are also introduced, along with several hands-on software codes for applied researchers to implement their own studies. Potential advantages and limitations of using the MLM approach to meta-analyzing SCED are discussed.
BaekE.FerronJ. J. M. (2020). Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data. Methodology, 16, 166–185.
2.
BaekE.BeretvasS. N.Van Den NoortgateW.FerronJ. M. (2020a). Brief research report: Bayesian versus REML estimations with noninformative priors in multilevel single-case data. Journal of Experiential Education, 88, 698–710.
3.
BaekE.LuoW.HenriM. (2020b). Issues and solutions in meta-analysis of single-case design with multiple dependent variables using multilevel modeling. Journal of Experiential Education, 90, 934–961. https://doi.org/10.1080/00220973.2020.1821342
4.
BaekE.Petit-BoisM.Van den NoortgateW.BeretvasT.FerronJ. (2016). Using visual analysis to evaluate and refine multilevel models of single-case studies. Journal of Special Education, 50, 18–26.
5.
BaekE. K.MoeyaertM.Petit-BoisM.BeretvasS. N.Van Den NoortgateW.FerronJ. M. (2014). The use of multilevel analysis for integrating single-case experimental design results within a study and across studies. Neuropsychological Rehabilitation, 24, 590–606.
6.
BarlowD. H.NockM. K.HersenM. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Pearson.
7.
BeretvasS. N.ChungH. (2008). A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention, 2(3), 129–141.
8.
BrowneW. (2008). MCMC estimation in MLwinN. Centre for Multilevel Modeling.
9.
FerronJ.DaileyR.YiQ. (2002). Effects of misspecifying the first-level error structure in two-level models of change. Multivariate Behavioral Research, 37, 379–403.
10.
FerronJ. M.BellB. A.HessM. R.Rendina-GobioffG.HibbardS. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modelingapproaches. Behavior Research Methods, 41, 372–384.
11.
FerronJ. M.FarmerJ. L.OwensC. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42, 930–943.
12.
GilksW. R.RichardsonS.SpiegelhalterD. J. (1996). Markov chain Monte Carlo in practice. Chapman and Hall/CRC.
GreenwoodK. M.MatyasT. A. (1990). Problems with the application of interrupted time series analysis for brief single-subject data. Behavioral Assessment, 12, 355–370.
15.
HoxJ. (1998). Multilevel modeling: When and why. In Classification, data analysis, and data highways (pp. 147–154). Springer, Berlin, Heidelberg.
16.
HoxJ. (2002). Multilevel analysis. Techniques and applications. Lawrence Erlbaum Associates.
17.
HuitemaB. E.MckeanJ. W. (2000). Design specification issues in time-series intervention models. Educational and Psychological Measurement, 60, 38–58.
18.
JamshidiL.HeyvaertM.Van Den NoortgateW. (2017, April). Review of systematic reviews and meta-analyses of single subject experimental studies [Paper presentation]. American Educational Research Association, San Antonio, TX.
19.
JamshidiL.HeyvaertM.DeclercqL.Fernández-CastillaB.FerronJ. M.MoeyaertM.BeretvasS. N.OnghenaP.Van Den NoortgateW. (2018). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities, 79, 97–115.
20.
JooS.-H.FerronJ. M. (2019). Application of the within- and between-series estimators to non-normal multiple-baseline data: Maximum likelihood and Bayesian approaches. Multivariate Behavioral Research, 54, 666–689. https://doi.org/10.1080/00273171.2018.1564877
21.
JooS. H.FerronJ. M.MoeyaertM.BeretvasS. N.Van Den NoortgateW. (2019). Approaches for specifying the level-1 error structure when synthesizing single-case data. Journal of Experiential Education, 87, 55–74.
22.
KazdinA. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). Oxford University Press.
23.
KenwardM. G.RogerJ. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997.
KwokO. M.WestS. G.GreenS. B. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42, 557–592.
27.
MagginD. M.O’KeeffeB. V.JohnsonA. H. (2011a). A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985–2009. Exceptionality, 19, 109–135.
28.
MagginD. M.SwaminathanH.RogersH. J.O’KeeffeB. V.SugaiG.HornerR. H. (2011b). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49, 301–321.
29.
MoerbeekM.van BreukelenG. J. P.BergerM. P. F. (2000). Design issues for experiments in multilevel populations. Journal of Educational and Behavioral Statistics, 25, 271–284.
30.
MoeyaertM.RindskopfD.OnghenaP.Van Den NoortgateW. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods, 22(4), 760–778.
31.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Den NoortgateW. V. (2014). Three-level analysis of single-case experimental data: Empirical validation. Journal of Experiential Education, 82, 1–21.
32.
MoeyaertM.UgilleM.FerronJ. M.BeretvasS. N.Van Den NoortgateW. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48, 719–748.
33.
MoeyaertM.UgilleM.FerronJ. M.OnghenaP.HeyvaertM.BeretvasS. N.Van Den NoortgateW. (2015). Estimating intervention effects across different types of single-subject experimental designs: empirical illustration. School Psychology Quarterly, 30, 50–63.
34.
NugentW. R. (1996). Integrating single-case and group-comparison designs for evaluation research. The Journal of Applied Behavioral Science, 32, 209–226.
35.
OwensC. M.FerronJ. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44, 795–805.
36.
Petit-BoisM. (2014). A Monte Carlo study: The consequences of the misspecification of the level-1 error structure when meta-analyzing single-case designs using a three-level model [Unpublished doctoral dissertation]. University of South Florida. http://scholarcommons.usf.edu/etd/5341
37.
Petit-BoisM.BaekE. K.Van Den NoortgateW.BeretvasS. N.FerronJ. M. (2016). The consequences of modeling autocorrelation when synthesizing single-case studies using a three-level model. Behavior Research Methods, 48, 803–812.
38.
PustejovskyJ. E.HedgesL. V.ShadishW. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39, 368–393.
39.
PyleK.FabianoG. A. (2017). Daily report card intervention and attention deficit hyperactivity disorder: a meta-analysis of single-case studies. Exceptional Children, 83(4), 378–395.
40.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. SAGE.
41.
RindskopfD. (2014a). Nonlinear Bayesian analysis for single case designs. Journal of School Psychology, 52, 179–189.
42.
RindskopfD. (2014b). Bayesian analysis of data from single case designs. Neuropsychological Rehabilitation, 24, 572–589.
43.
ShadishW. R. (2014). Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology, 52(2), 109–122.
44.
ShadishW. R.HedgesL. V.PustejovskyJ. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147.
45.
ShadishW. R.RindskopfD. M. (2007). Methods for evidence-based practice: Quantitative synthesis of single-subject designs. New Directions for Evaluation, 2007(113), 95–109.
46.
ShadishW. R.RindskopfD. M.HedgesL. V.SullivanK. J. (2013). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 45, 813–821.
47.
ShadishW. R.SullivanK. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971–980.
48.
SnijdersT. A. B.BoskerR. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. SAGE Publications.
49.
SolomonB. G. (2014). Violations of assumptions in school-based single-case data: Implications for the selection and interpretation of effect sizes. Behavior Modification, 38, 477–496.
50.
SwaminathanH.RogersH. J.HornerR. H. (2014). An effect size measure and Bayesian analysis of single-case designs. Journal of School Psychology, 52, 213–230.
51.
ToothakerL. E.BanzM.NobleC.CampJ.DavisD. (1983). N = 1 designs: The failure of ANOVA-based tests. Journal of Educational Statistics, 8(4), 289–309.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ.Van Den NoortgateW. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44, 1244–1254.
54.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ. M.Van Den NoortgateW. (2014). Bias corrections for standardized effect size estimates used with single-subject experimental designs. Journal of Experiential Education, 82, 358–374.
55.
Van Den NoortgateW.OnghenaP. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325–346.
56.
Van Den NoortgateW.OnghenaP. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods Instruments & Computers, 35, 1–10.
57.
Van Den NoortgateW.OnghenaP. (2007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8, 196–209.
58.
Van Den NoortgateW.OnghenaP. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention, 2(3), 142–151.
59.
WampoldB. E.SerlinR. C. (2000). The consequence of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425–433.
60.
WolfingerR. (1993). Covariance structure selection in general mixed models. Communications in Statistics - Simulation and Computation, 22, 1079–1106.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
