Multilevel meta-analysis is an innovative synthesis technique used for the quantitative integration of effect size estimates across participants and across studies. The quantitative summary allows for objective, evidence-based, and informed decisions in research, practice, and policy. Based on previous methodological work, the technique results in powerful, unbiased, and precise effect size estimates. However, its use in practice is limited and its full potential is not yet fully understood. This article aims to bring the multilevel meta-analytic model closer to the applied researcher by introducing the technique at a conceptual level and discussing its full potential and relevance to the field. The procedure of multilevel meta-analysis is illustrated using a recent single-case meta-analytic dataset. Software codes, output tables, interpretations, and graphical displays of effect size estimates are given such that the reader can repeat the analysis independently.
BaekE.FerronJ. M. (2013). Multilevel models for multiple-baseline data: Modeling across participant variation in autocorrelation and residual variance. Behavior Research Methods, 45, 65–74. doi:10.3758/s13428-012-0231-z
2.
BandaD. R.HartS. L.Liu-GitzL. (2010). Impact of training peers and children with autism on social skills during center time activities in inclusive classrooms. Research in Autism Spectrum Disorders, 4, 619–625. doi:10.1016/j.rasd.2009.12.005
3.
Barton-ArwoodS. M. (2003). Reading instruction for elementary-age students with emotional and behavioral disorders: Academic and behavioral outcomes (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3085747)
4.
BeretvasS. N.ChuY. (2013April). Handling count data outcomes trajectories in multiple-baseline design studies. Paper presented at the annual meeting of the American Educational Research Association, Philadelphia, PA.
5.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. Chichester, UK: John Wiley.
6.
CardN. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford.
7.
CoolsW.DeclercqL.BeretvasS. N.MoeyaertM.FerronJ. M.Van den NoortgateW. (2017). MultiSCED [Software]. Retrieved from http://52.14.146.253/MultiSCED/
8.
DeclercqL.JamshidiL.Fernández-CastillaB.BeretvasS.MoeyaertM.FerronJ.Van den NoortgateW. (2018). Analysis of single-case experimental count data using the linear mixed effects model: A simulation study. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-018-1091-y
9.
FerronJ. M.BellB. A.HessM. F.Rendina-GobioffG.HibbardS. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41, 372–384. doi:10.3758/BRM.41.2.372
10.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. doi:10.3102/0013189X005010003
11.
HedgesL. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128. doi:10.3102/10769986006002107
12.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press. doi:10.1080/00220973.2014.907231
13.
HembryI.BunuanR.BeretvasS. N.FerronJ.Van den NoortgateW. (2015). Estimation of a nonlinear intervention phase trajectory for multiple-baseline design data. Journal of Experimental Education, 83, 514–546. doi:10.1080/00220973.2014.907231
14.
HibbertD.KostinasG.LuiselliJ. K. (2002). Improving skills performance of an adult with mental retardation through peer-mediated instructional support. Journal of Developmental and Physical Disabilities, 14, 119–127. doi:10.1023/A:1015263329642
15.
JamshidiL.HeyvaertM.DeclercqL.Fernández Castilla FerronJ.MoeyaertM.BeretvasS. N.Van den NoortgateW. (2018). A systematic review of single-case experimental design meta-analyses: Characteristics of study designs, data, and analyses. Manuscript submitted for publication.
16.
JooS.FerronJ.MoeyaertM.BeretvasS.Van den NoortgateW. (2017). Approaches for specifying the level-1 error structure when synthesizing single-case data. Journal of Experimental Education. Advance online publication. doi:10.1080/00220973.2017.1409181
17.
KenwardM. G.RogerJ. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997. doi:10.2307/2533558
18.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single-case designs technical documentation. Retrieved from https://files.eric.ed.gov/fulltext/ED510743.pdf
19.
LeamerE. E. (1978). Specification searches. New York, NY: John Wiley.
20.
LenzA. S. (2013). Calculating effect size in single-case research: A comparison of nonoverlap methods. Measurement and Evaluation in Counseling and Development, 46, 64–73. doi:10.1177/0748175612456401
21.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
22.
LoftinR. L.OdomS. L.LantzJ. F. (2008). Social interaction and repetitive motor behaviors. Journal of Autism and Developmental Disorders, 38, 1124–1135. doi:10.1007/s10803-007-0499-5
23.
LorahE. R.GilroyS. P.HinelineP. N. (2014). Acquisition of peer manding and listener responding in young children with autism. Research in Autism Spectrum Disorders, 8, 61–67. doi:10.1016/j.rasd.2013.10.009
24.
MasonR.KampsD.TurcotteA.CoxS.FeldmillerS.MillerT. (2014). Peer mediation to increase communication and interaction at recess for students with autism spectrum disorders. Research in Autism Spectrum Disorders, 8, 334–344. doi:10.1016/j.rasd.2013.12.014
25.
MoeyaertM.KlingbeilD.RodabaughE.TuranM. (2018). Multilevel meta-analysis of peer-tutoring interventions to increase academic performance and social interactions for people with special needs. Manuscript submitted for publication.
26.
MoeyaertM.MagginD. M.VerkuilenJ. (2016). Reliability and validity of extracting data from image files in contexts of single-case experimental design studies. Behavior Modification, 40, 874–900. doi:10.1177/0145445516645763
27.
MoeyaertM.ManolovR.RodabaughE. (in press). Meta-analysis of single-case research via multilevel models: Fundamental concepts and methodological considerations. Behavior Modification.
28.
MoeyaertM.UgilleM.FerronJ.BeretvasS.Van den NoortgateW. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48, 719–748. doi:10.1080/00273171.2013.816621
29.
MoeyaertM.UgilleM.FerronJ.BeretvasS.Van den NoortgateW. (2014a). The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental design research. Behavior Modification, 38, 665–704. doi:10.1177/0145445514535243
30.
MoeyaertM.UgilleM.FerronJ.BeretvasS.Van den NoortgateW. (2014b). Three-level analysis of single-case experimental data: Empirical validation. Journal of Experimental Education, 82, 1–21. doi:10.1080/00220973.2012.745470
31.
MoeyaertM.UgilleM.FerronJ.OnghenaP.HeyvaertM.Van den NoortgateW. (2014). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly, 25, 191–211. doi:10.1037/spq0000068
32.
ParkerR. I.VannestK. (2008). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357–367. doi:10.1016/j.beth.2008.10.006
33.
PellegriniR. J.HicksR. A. (1972). Prophecy effects and tutorial instruction for the disadvantaged child. American Educational Research Journal, 9, 413–419. doi:10.3102/00028312009003413
34.
PlumerP. J. (2007). Using peers as intervention agents to improve the social behaviors of elementary-aged children with attention deficit hyperactivity disorder: Effects of a peer coaching package (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3275754)
35.
RaudenbushS. W.BrykA. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75–98. doi:10.3102/10769986010002075
RosenthalR. (1994). Parametric measures of effect size. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 231–244). New York, NY: Russell Sage Foundation.
38.
SchaaljeG. B.McBrideJ. B.FellinghamG. W. (2001). Approximations to distributions of test statistics in complex mixed linear models using SAS Proc MIXED. In Proceedings of the SAS Users Group International 26th annual conference (Paper 262–26). Retrieved from http://www2.sas.com/proceedings/sugi26/p262-26.pdf
39.
ShadishW. R.RindskopfD. M.HedgesL. V.SullivanK. J. (2012). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 45, 813–821. doi:10.3758/s13428-012-0282-1
40.
ShadishW. R.SullivanK. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971–980. doi:10.3758/s13428-011-0111-y
41.
ShogrenK. A.Fagella-LubyM. N.BaeJ. S.WehmeyerM. L. (2004). The effect of choice-making as an intervention for problem behavior. Journal of Positive Behavior Interventions, 6, 228–237. doi:10.1177/10983007040060040401
42.
SmithM. L.GlassG. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760. doi:10.1037/0003-066X.32.9.752
43.
SterneJ. A. C.EggerM. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046–1055. doi:10.1016/S0895-4356(01)00377-8
44.
SterneJ. A. C.HarbordR. M. (2004). Funnel plots in meta-analysis. The Stata Journal, 4, 127–141.
45.
SuttonA. J.AbramsK. R.JonesD. R.SheldonT. A.SongF. (2000). Methods for meta-analysis in medical research. Chichester, UK: John Wiley.
46.
TalbottE.MagginD. M.Van AckerE. Y.KummS. (2018). Quality indicators for reviews of research in special education. Exceptionality, 26, 245–265 doi:10.1080/09362835.2017.1283625
47.
TrembathD.BalandinS.TogherL.StancliffeR. J. (2009). Peer-mediated teaching and augmentative and alternative communication for preschool-aged children with autism. Journal of Intellectual & Developmental Disability, 34, 173–186. doi:10.1080/13668250902845210
48.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ.NoortgateW. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44, 1244–1254. doi:10.3758/s13428-012-0213-1
49.
UgilleM.MoeyaertM.BeretvasS. N.FerronJ.Van Den NoortgateW. (2013). Bias corrections for standardized effect size estimates used with single-subject experimental designs. Journal of Experimental Education, 45, 547–559. doi:10.1080/00220973.2013.813366
50.
Van den NoortgateW.OnghenaP. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325–346.
51.
Van den NoortgateW.OnghenaP. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, & Computers, 35, 1–10. doi:10.3758/BF03195492
52.
Van den NoortgateW.OnghenaP. (2007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8, 196–209. doi:10.1037/h0100613
53.
Van den NoortgateW.OnghenaP. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Inter-vention, 2, 142–151. doi:10.1080/17489530802505362
54.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. Retrieved from http://www.jstatsoft.org/v36/i03/
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
