Restricted accessReview articleFirst published online 2024-11
Four Best Practices for Meta-Analysis: A Systematic Review of Methodological Rigor in Mathematics Interventions for Students With or at Risk of Disabilities
Meta-analysis methodology has evolved with the development of more robust statistical techniques; however, few reviews in special education have focused specifically on methodological rigor in meta-analyses. In this study, we examined 29 meta-analyses of mathematics interventions published from 2000 to 2022 to determine the extent to which researchers have applied four best practices in meta-analyses focused on mathematics interventions for students with disabilities. Our findings were (a) studies used three primary moderator techniques: meta-regression (k = 10), subgroup analysis (k = 8), analysis of variance (k = 3), and both subgroup analysis and meta-regression (k = 1); (b) only one study considered small sample corrections for hypothesis tests; (c) few researchers handled the dependence between multiple effect sizes (k = 3); and (d) the funnel plot was commonly used to detect publication bias (k = 8). Based on our findings, we make recommendations for methodological considerations for future meta-analyses.
AhnS.AmesA. J.MyersN. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4), 436–476. https://doi.org/10.3102/0034654312458162
2.
*AlegreF.MolinerL.MarotoA.Lorenzo-ValentinG. (2019). Peer tutoring in mathematics in primary education: A systematic review. Educational Review, 71(6), 767–791. https://doi.org/gfv9tj
3.
*BakerS.GerstenR.LeeD. S. (2002). A synthesis of empirical research on teaching mathematics to low-achieving students. The Elementary School Journal, 103(1), 51–73. https://doi.org/10.1086/499715
4.
BeckerB. J. (2000). Multivariate meta-analysis. In TinsleyH. E.BrownS. D. (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Academic Press.
5.
BerkeyC. S.HoaglinD. C.MostellerF.ColditzG. A. (1995). A random-effects regression model for meta-analysis. Statistics in Medicine, 14(4), 395–411. https://doi.org/10.1002/sim.4780140406
6.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. John Wiley & Sons.
7.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2013). Comprehensive meta-analysis (Version 3.0) [Computer software]. Biostat.
8.
BorensteinM.HigginsJ. P. (2013). Meta-analysis and subgroups. Prevention Science, 14(2), 134–143.
9.
*BrowderD. M.SpoonerF.Ahlgrim-DelzellL.HarrisA. A.WakemanxyaS. (2008). A meta-analysis on teaching mathematics to students with significant cognitive disabilities. Exceptional Children, 74(4), 407–432. https://doi.org/gkkjvx
10.
BrykA. S.RaudenbuschS. W.CongdonR. T. (1994). HLM 2/3: Hierarchical linear modeling with the HLM/2L and HLM/3L programs. Scientific Software International.
11.
CheungA. C.SlavinR. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88–113. https://doi.org/10.1016/j.edurev.2013.01.001
12.
*ChoduraS.KuhnJ. T.HollingH. (2015). Interventions for children with mathematical difficulties: A meta-analysis. Zeitschrift für Psychologie, 223(2), 129–144. https://doi.org/10.1027/2151-2604/a000211
13.
CookB. G. (2014). A call for examining replication and bias in special education research. Remedial and Special Education, 35, 233–246. https://doi.org/10.1177/0741932514528995
14.
CooperH.HedgesL. V.ValentineJ. C. (Eds.). (2019). The handbook of research synthesis and meta-analysis (3rd ed.). Russell Sage Foundation.
15.
CooperH. M. (1998). Synthesizing research: A guide for literature reviews (Vol. 2). Sage.
16.
*DennisM. S.SharpE.ChovanesJ.ThomasA.BurnsR. M.CusterB.ParkJ. (2016). A meta-analysis of empirical research on teaching students with mathematics learning difficulties. Learning Disabilities Research & Practice, 31(3), 156–168. https://doi.org/10.1111/ldrp.12107
17.
DickersinK. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11–33). John Wiley & Sons.
18.
DowdyA.HantulaD. A.TraversJ. C.TincaniM. (2022). Meta-analytic methods to detect publication bias in behavior science research. Perspectives on Behavior Science, 45(1), 37–52. https://doi.org/10.1007/s40614-021-00303-0
19.
DuvalS.TweedieR. (2000). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89–98. https://doi.org/10.1080/01621459.2000.10473905
20.
*EnnisR. P.LosinskiM. (2019). Interventions to improve fraction skills for students with disabilities: A meta-analysis. Exceptional Children, 85(3), 367–386. https://doi.org/10.1177/0014402918817504
21.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629–634. https://doi.org/10.1177/0014402918817504
22.
GageN. A.CookB. G.ReichowB. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83(4), 428–445. https://doi.org/10.1177/0014402917691016
23.
*GerstenR.ChardD. J.JayanthiM.BakerS. K.MorphyP.FlojoJ. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research, 79(3), 1202–1242. https://doi.org/10.3102/0034654309334431
24.
GoertzM. E.BarnesC.MassellD.FinkR.FrancisA. T. (2013). State education agencies’ acquisition and use of research knowledge in school improvement strategies: A research report. Consortium for Policy Research in Education. https://files.eric.ed.gov/fulltext/ED547655.pdf
25.
HedgesL. V.TiptonE.JohnsonM. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5
26.
HigginsJ. P. T.ThompsonS. G. (2004). Controlling the risk of spurious findings from meta-regression. Statistics in Medicine, 23(11), 1663–1682. https://doi.org/b3psfv
27.
*HughesE. M.WitzelB. S.RiccominiP. J.FriesK. M.KanyongoG. Y. (2014). A meta-analysis of algebra interventions for learners with disabilities and struggling learners. The Journal of the International Association of Special Education, 15(1), 36–47.
28.
*HwangJ.RiccominiP. J.HwangS. Y.MoranoS. (2019). A systematic analysis of experimental studies targeting fractions for students with mathematics difficulties. Learning Disabilities Research & Practice, 34(1), 47–61. https://doi.org/gkqb
29.
*JitendraA. K.AlghamdiA.EdmundsR.McKevettN. M.MouanoutouaJ.RoessleinR. (2021). The effects of Tier 2 mathematics interventions for students with mathematics difficulties: A meta-analysis. Exceptional Children, 87(3), 307–325. https://doi.org/10.1177/0014402920969187
30.
*JitendraA. K.LeinA. E.ImS. H.AlghamdiA. A.HefteS. B.MouanoutouaJ. (2018). Mathematical interventions for secondary students with learning disabilities and mathematics difficulties: A meta-analysis. Exceptional Children, 84(2), 177–196. https://doi.org/10.1177/0014402917737467
31.
JohnsonB. T.HennessyE. A. (2019). Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses. Social Science & Medicine, 233, 237–251. https://doi.org/10.1016/j.socscimed.2019.05.035
32.
*KimS. A.BryantD. P.BryantB. R.ShinM.OkM. W. (2023). A multilevel meta-analysis of whole number computation interventions for students with learning disabilities. Remedial and Special Education, 44(4), 332–347. https://doi.org/10.1177/07419325221117293
33.
KnappG.HartungJ. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22(17), 2693–2710. https://doi.org/dncmwj
34.
*KongJ. E.YanC.SercekiA.SwansonH. L. (2021). Word-problem-solving interventions for elementary students with learning disabilities: A selective meta-analysis of the literature. Learning Disability Quarterly, 44(4), 248–260. https://doi.org/10.1177/0731948721994843
35.
*KroesbergenE. H.Van LuitJ. E. (2003). Mathematics interventions for children with special educational needs: A meta-analysis. Remedial and Special Education, 24(2), 97–114. https://doi.org/d9qd6z
36.
*KunschC. A.JitendraA. K.SoodS. (2007). The effects of peer-mediated instruction in mathematics for students with learning problems: A research synthesis. Learning Disabilities Research & Practice, 22(1), 1–12. https://doi.org/drdb5p
37.
*LeiQ.MasonR. A.XinY. P.DavisJ. L.DavidM.LoryC. (2020). A meta-analysis of single-case research on mathematics word problem-solving interventions for English learners with learning disabilities and mathematics difficulties. Learning Disabilities Research & Practice, 35(4), 201–217. https://doi.org/10.1111/ldrp.12233
38.
*LeinA. E.JitendraA. K.HarwellM. R. (2020). Effectiveness of mathematical word problem solving interventions for students with learning disabilities and/or mathematics difficulties: A meta-analysis. Journal of Educational Psychology, 112(7), 1388–1408. https://doi.org/10.1037/edu0000453
39.
LiX.DusseldorpE.SuX.MeulmanJ. J. (2020). Multiple moderator meta-analysis using the R-package meta-CART. Behavior Research Methods, 52, 2657–2673. https://doi.org/10.3758/s13428-020-01360-0
*LosinskiM. L.EnnisR. P.SandersS. A.NelsonJ. A. (2019). A meta-analysis examining the evidence-base of mathematical interventions for students with emotional disturbances. The Journal of Special Education, 52(4), 228–241. https://doi.org/gkqh
42.
MaagJ.LosinskiM. L. (2015). Thorny issues and prickly solutions: Publication bias in meta-analytic reviews in the social sciences. Advances in Social Sciences Research Journal, 2, 242–253. https://doi.org/10.14738/assrj.24.1044
43.
Marks-AnglinA.ChenY. (2020). A historical review of publication bias. Research Synthesis Methods, 11(6), 725–742. https://doi.org/10.1002/jrsm.1452
44.
McShaneB. B.BöckenholtU.HansenK. T. (2016). Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11(5), 730–749. https://doi.org/10.1177/1745691616662243
45.
MetheS. A.HojnoskiR.ClarkeB.OwensB. B.LilleyP. K.PolityloB. C.WhiteK. M.MarcotteA. M. (2011). Innovations and future directions for early numeracy curriculum-based measurement: Commentary on the special series. Assessment for Effective Intervention, 36(4), 200–209. https://doi.org/10.1177/1534508411414154
46.
*MoherD.LiberatiA.TetzlaffJ.AltmanD. G., . . . PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264–269. https://doi.org/bpq5
47.
MutluY. (2019). Math anxiety in students with and without math learning difficulties. International Electronic Journal of Elementary Education, 11(5), 471–475. https://doi.org/10.26822/iejee.2019553343
48.
National Center for Science and Engineering Statistics. (2019). Women, minorities, and persons with disabilities in science and engineering: 2019 (Special report NSF 15–311). National Science Foundation. http://www.nsf.gov/statistics/wmpd/
49.
NelsonG.CrawfordA.HuntJ.ParkS.LeckieE.DuarteA.BraffordT.Ramos-DukeM.ZarateK. (2022). A systematic review of research syntheses on students with mathematics learning disabilities and difficulties. Learning Disabilities Research & Practice, 37(1), 18–36. https://doi.org/10.1111/ldrp.12272
50.
*NelsonG.McMasterK. L. (2019). The effects of early numeracy interventions for students in preschool and early elementary: A meta-analysis. Journal of Educational Psychology, 111(6), 1001–1022. https://doi.org/ggskx9
51.
NelsonG.ParkS.BraffordT.HellerN. A.CrawfordA. R.DrakeK. R. (2022). Reporting quality in math meta-analyses for students with or at risk of disabilities. Exceptional Children, 88(2), 125–144. https://doi.org/10.1177/00144029211050851
52.
NelsonG.PowellS. R. (2018). A systematic review of longitudinal studies of mathematics difficulty. Journal of Learning Disabilities, 51(6), 523–539. https://doi.org/gbhg7q
53.
NewmanL.WagnerM.CametoR.KnokeyA.-M. (2009). The post-high school outcomes of youth with disabilities up to 4 years after high school. A report of findings from the National Longitudinal Transition Study-2 (NLTS2). SRI International.
54.
PageM. J.MoherD.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.MckenzieJ. E. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. British Medical Journal, 372, Article no. 160. https://doi.org/10.1136/bmj.n160
55.
ParkS.NelsonG. (2022). The quality of outcome measure reporting in early numeracy intervention studies. Psychology in the Schools, 59(9), 1721–1736. https://doi.org/10.1002/pits.22726
56.
*PeltierC.MorinK. L.BouckE. C.LingoM. E.PulosJ. M.SchefflerF. A.SukA.MathewsL. A.SinclairT. E.DeardorffM. E. (2020). A meta-analysis of single-case research using mathematics manipulatives with students at risk or identified with a disability. The Journal of Special Education, 54(1), 3–15. https://doi.org/ggqmrz
57.
*PeltierC. J.VannestK. J.MarbachJ. J. (2018). A meta-analysis of schema instruction implemented in single-case experimental designs. The Journal of Special Education, 52(2), 89–100. https://doi.org/gdzmw4
58.
PigottT. D.PolaninJ. R. (2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90(1), 24–46. https://doi.org/10.3102/0034654319877153
59.
PolaninJ. R.Tanner-SmithE. E.HennessyE. A. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067
60.
PustejovskyJ. E.FerronJ. M. (2017). Research synthesis and meta-analysis of single-case designs. In KauffmanJ. M.HallahanD. P.PullenP. C. (Eds.), Handbook of special education (2nd ed., pp. 168–186). Routledge. https://doi.org/10.4324/9781315517698-15
61.
PustejovskyJ. E.TiptonE. (2022). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3
62.
*RanH.KasliM.SecadaW. G. (2021). A meta-analysis on computer technology intervention effects on mathematics achievement for low-performing students in K-12 classrooms. Journal of Educational Computing Research, 59(1), 119–153. https://doi.org/10.1177/0735633120952063
63.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
64.
*RojoM.KingS.GersibJ.BryantD. P. (2022). Rational number interventions for students with mathematics difficulties: A meta-analysis. Remedial and Special Education, 44(3), 225–238. https://doi.org/10.1177/07419325221105520
RothsteinH. R. (2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4(1), 61–81. https://doi.org/10.1007/s11292-007-9046-9
67.
RothsteinH. R.McDanielM. A.BorensteinM. (2002). Meta-analysis: A review of quantitative cumulation methods. In DrasgowF.SchmittN. (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 534–570). Jossey-Bass.
68.
Sánchez-MecaJ.Marín-MartínezF. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological Methods, 13(1), 31–48. https://doi.org/10.1037/1082-989X.13.1.31
69.
ShadishW. R.HedgesL. V.PustejovskyJ. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. https://doi.org/10.1016/j.jsp.2013.11.005
70.
ShadishW. R.ZelinskyN. A. M.VeveaJ. L.KratochwillT. R. (2016). A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, 49(3), 1–18. https://doi.org/10.1002/jaba.308
71.
ShamE.SmithT. (2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47(3), 663–678. https://doi.org/10.1002/jaba.146
72.
*ShinM.BryantD. P.PowellS. R.JungP. G.OkM. W.HouF. (2021). A meta-analysis of single-case research on word-problem instruction for students with learning disabilities. Remedial and Special Education, 42(6), 398–411. https://doi.org/10.1177/0741932520964918
73.
*StevensE. A.RodgersM. A.PowellS. R.(2018). Mathematics interventions for upper elementary and secondary students: A meta-analysis of research. Remedial and Special Education, 39(6), 327–340. https://doi.org/ggx69c
74.
StephensM.ErberberE.TsokodayiY.FonsecaF. (2022). Changes between 2011 and 2019 in achievement gaps between high- and low-performing students in mathematics and science: International results from TIMSS (NCES 2022-041). U.S. Department of Education; National Center for Education Statistics, Institute of Education Sciences. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2022041
75.
TalbottE.MagginD.Van AckerE. Y.KummS. (2018). Quality indicators for reviews of research in special education. Exceptionality, 26, 245–265. https://doi.org/gftjmr
76.
*TempletonT. N.NeelR. S.BloodE.(2008). Meta-analysis of math interventions for students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 16(4), 226–239. https://doi.org/b8tpvc
77.
TherrienW. J.CookB. G.CookL. (2020). Utilizing meta-analyses to guide practice: A primer. Learning Disabilities Research & Practice, 35(3), 111–117. https://doi.org/gkqr
78.
TherrienW. J.ZamanM.BandaD. R. (2011). How can meta-analyses guide practice? A review of the learning disability research base. Remedial and Special Education, 32(3), 206–218. https://doi.org/10.1177/0741932510361266
79.
ThompsonS. G.HigginsJ. P. T. (2002). How should meta-regression analyses be undertaken and interpreted?Statistics in Medicine, 21(11), 1559–1573.
80.
TincaniM.TraversJ. C. (2022). Questionable research practices in single-case experimental designs: Examples and possible solutions. In O’DonohueW.MasudaA.LilienfeldS. O. (Eds.), Questionable research practices: Designing, conducting, and reporting sound research in clinical psychology (pp. 269–285). Springer.
81.
TiptonE. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011
82.
TiptonE.PustejovskyJ. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40(6), 604–634. https://doi.org/10.3102/1076998615606099
83.
TiptonE.PustejovskyJ. E.AhmadiH. (2019a). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10(2), 180–194. https://doi.org/10.1002/jrsm.1339
84.
TiptonE.PustejovskyJ. E.AhmadiH. (2019b). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10(2), 161–179. https://doi.org/10.1002/jrsm.1338
U.S. Department of Education. (2021). Science, technology, engineering, and math, including computer science. http://ed.gov/stem
87.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. https://doi.org/10.3758/s13428-012-0261-6
88.
Van den NoortgateW.López-LópezJ. A.Marín-MartínezF.Sánchez-MecaJ. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47(4), 1274–1294. https://doi.org/10.3758/s13428-014-0527-2
89.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48. https://doi.org/10.18637/jss.v036.i03
90.
WangA. H.FirmenderJ. M.PowerJ. R.ByrnesJ. P. (2016). Understanding the program effectiveness of early mathematics interventions for prekindergarten and kindergarten environments: A meta-analytic review. Early Education and Development, 27(5), 692–713.
91.
WilliamsR.CitkowiczM.MillerD. I.LindsayJ.WaltersK. (2022). Heterogeneity in mathematics intervention effects: Evidence from a meta-analysis of 191 randomized experiments. Journal of Research on Educational Effectiveness, 15(4), 1–51. https://doi.org/10.1080/19345747.2021.2009072
92.
YoungJ. (2017). Technology-enhanced mathematics instruction: A second-order meta-analysis of 30 years of research. Educational Research Review, 22, 19–33.
93.
ZelekeS. (2004). Differences in self-concept among children with mathematics disabilities and their average and high achieving peers. International Journal of Disability, Development and Education, 51(3), 253–269. http://doi.org/10.1080/1034912042000259224
94.
*ZhangD.XinY. P. (2012). A follow-up meta-analysis for word-problem-solving interventions for students with mathematics difficulties. The Journal of Educational Research, 105(5), 303–318. https://doi.org/gjp9pp
95.
*ZhengX.FlynnL. J.SwansonH. L. (2013). Experimental intervention studies on word problem solving and math disabilities: A selective analysis of the literature. Learning Disability Quarterly, 36(2), 97–111. https://doi.org/f45
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
