We replicated a study of a kindergarten mathematics intervention, ROOTS, delivered in the context of a research-based core program. We randomly assigned 62 classrooms to treatment (ROOTS) or a business-as-usual control. All classrooms implemented the research-based core program (Early Learning in Mathematics). Participants included 163 treatment students and 145 control students nested within classrooms. Key differences between the current replication study and the original study included geographical region, instructional context, and student initial skill. In contrast to the significant positive effects (Hedges’s g values of .30 to .38) found in the original study, no significant differences were found between the treatment and control conditions in this study (Hedges’s g values of –.09 to .12) Pretest skills did not moderate treatment effects. We discuss these results’ implications for replication research and evaluating intervention efficacy.
Get full access to this article
View all access options for this article.
References
1.
AkaikeH. (1973). Maximum likelihood identification of Gaussian autoregressive moving average models. Biometrika, 60(2), 255–265. https://doi.org/10.1093/biomet/60.2.255
2.
AllisonP. D. (2009). Missing data. In MillsapR. E.Maydeu-OlivaresA. (Eds.), The Sage handbook of quantitative methods in psychology (pp. 72–89). Sage. https://doi.org/10.4135/9780857020994.n4
3.
BakerS. K.GerstenR. M.LeeD.-S. (2002). A synthesis of empirical research on teaching mathematics to low- achieving students. Elementary School Journal, 103(1), 51–73. https://doi.org/10.1086/499715
4.
BodovskiK.FarkasG. (2007). Mathematics growth in early elementary school: The roles of beginning knowledge, student engagement, and instruction. Elementary School Journal, 108(2), 115–130. https://doi.org/10.1086/525550
5.
BryantD. P.BryantB. R.GerstenR. M.ScammaccaN. N.FunkC.WinterA.ShihM.PoolC. (2008). The effects of tier 2 intervention on the mathematics performance of first-grade students who are at risk for mathematics difficulties. Learning Disability Quarterly, 31(2), 47–63. https://doi.org/10.2307/20528817
6.
BurnhamK. P.AndersonD. R.HuyvaertK. P. (2011). AIC model selection and multimodel inference in behavioral ecology: Some background, observations, and comparisons. Behavioral Ecology and Sociobiology, 65(1), 23–35. https://doi.org/10.1007/s00265-010-1029-6
7.
ChardD. J.BakerS. K.ClarkeB.JungjohannK.DavisK. L. S.SmolkowskiK. (2008). Preventing early mathematics difficulties: The feasibility of a rigorous kindergarten mathematics curriculum. Learning Disability Quarterly, 31(1), 11–20. https://doi.org/10.2307/30035522
8.
ChardD. J.ClarkeB.BakerS. K.OtterstedtJ.BraunD.KatzR. (2005). Using measures of number sense to screen for difficulties in mathematics: Preliminary findings. Assessment for Effective Intervention, 30(2), 3–14. https://doi.org/10.1177/073724770503000202
9.
ChhinC. S.TaylorK. A.WeiW. S. (2018). Supporting a culture of replication: An examination of education and special education research grants funded by the Institute of Education Sciences. Educational Researcher, 47(9), 594–605. https://doi.org/10.3102/0013189X18788047
10.
ClarkeB.DoablerC. T.KostyD.Kurtz NelsonE.SmolkowskiK.FienH.TurturaJ. (2017). Testing the efficacy of a kindergarten mathematics intervention by small group size. AERA Open, 3(2), 1–16. https://doi.org/10.1177/2332858417706899
11.
ClarkeB.DoablerC. T.SmolkowskiK.BakerS. K.FienH.Strand CaryM. (2016). Examining the efficacy of a tier 2 kindergarten mathematics intervention. Journal of Learning Disabilities, 49(2), 152–165. https://doi.org/10.1177/0022219414538514
12.
ClarkeB.DoablerC. T.SmolkowskiK.TurturaJ.KostyD.Kurtz-NelsonE.FienH.BakerS. K. (2019). Exploring the relationship between initial mathematics skill and a kindergarten mathematics intervention. Exceptional Children, 85(2), 129–146. https://doi.org/10.1177/0014402918799503
13.
ClarkeB.DoablerC. T.Strand CaryM.KostyD. B.BakerS. K.FienH.SmolkowskiK. (2014). Preliminary evaluation of a tier-2 mathematics intervention for first grade students: Utilizing a theory of change to guide formative evaluation activities. School Psychology Review, 43(2), 160–177. https://doi.org/10.1080/02796015.2014.12087442
14.
ClarkeB.DoablerC. T.TurturaJ.SmolkowskiK.KostyD.SutherlandM.Kurtz NelsonE.FienH.BakerS. K. (2020). Examining the efficacy of a kindergarten mathematics intervention by group size and initial skill: Implications for practice and policy. Elementary School Journal, 121(1), 125–153. https://doi.org/10.1086/710041
15.
ClarkeB.ShinnM. R. (2004). A preliminary investigation into the identification and development of early mathematics curriculum-based measurement. School Psychology Review, 33(2), 234–248. https://doi.org/10.1080/02796015.2004.12086245
16.
ClarkeB.SmolkowskiK.BakerS. K.FienH.DoablerC. T.ChardD. (2011). The impact of a comprehensive tier 1 core kindergarten program on the achievement of students at risk in mathematics. Elementary School Journal, 111(4), 561–584. https://doi.org/10.1086/659033
17.
CollinsL. M.SchaferJ. L.KamC.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351. https://doi.org/10.1037/1082-989X.6.4.330
18.
CookB. G.BuysseV.KlingnerJ.LandrumT. J.McWilliamR. A.TankersleyM.TestD. W. (2014). CEC’s standards for classifying the evidence base of practices in special education. Remedial and Special Education. Advance online publication. https://doi.org/10.1177/0741932514557271
19.
CoyneM. D.CookB. G.TherrienW. J. (2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37(4), 244–253. https://doi.org/10.1177/0741932516648463
20.
CoyneM. D.Kame’enuiE. J.CarnineD. (2011). Effective teaching strategies that accommodate diverse learners (4th ed.). Pearson Education.
21.
CoyneM. D.SimmonsD. C.Hagan-BurkeS.SimmonsL. E.KwokO.-M.KimM.FogartyM.OslundE. L.TaylorA. R.Capozzoli-OldhamA.WareS.LittleM. E.RawlinsonD. A. M. (2013). Adjusting beginning reading intervention based on student performance: An experimental evaluation. Exceptional Children, 80(1), 25–44. https://doi.org/10.1177/001440291308000101
22.
DoablerC. T.CaryM. S.JungjohannK.ClarkeB.FienH.BakerS.SmolkowskiK.ChardD. (2012). Enhancing core mathematics instruction for students at risk for mathematics disabilities. TEACHING Exceptional Children, 44(4), 48–57. https://doi.org/10.1177/004005991204400405
23.
DoablerC. T.ClarkeB.KostyD. B.Kurtz-NelsonE.FienH.SmolkowskiK.BakerS. K. (2016). Testing the efficacy of a Tier 2 mathematics intervention: A conceptual replication study. Exceptional Children, 83(1), 92–110. https://doi.org/10.1177/0014402916660084
24.
DoablerC. T.ClarkeB.KostyD. B.Kurtz-NelsonE.FienH.SmolkowskiK.BakerS. K. (2019). Examining the impact of group size on the treatment intensity of a Tier 2 mathematics intervention within a systematic framework of replication. Journal of Learning Disabilities, 52(2), 168–180. https://doi.org/10.1177/0022219418789376
25.
DuncanG. J.DowsettC. J.ClaessensA.MagnusonK.HustonA. C.KlebanovP.PaganiL. S.FeinsteinL.EngelM.Brooks-GunnJ.SextonH.DuckworthK.JapelC. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428–1446. https://doi.org/10.1037/0012-1649.43.6.1428
26.
DysonN. I.JordanN. C.GluttingJ. (2013). A number sense intervention for low-income kindergartners at risk for mathematics difficulties. Journal of Learning Disabilities, 46(2), 166–181. https://doi.org/10.1177/0022219411410233
27.
FixsenD. L.NaoomS. F.BlaseK. A.FriedmanR. M.WallaceF. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.
FryeD.BaroodyA. J.BurchinalM.CarverS. M.JordanN. C.McDowellJ. (2013). Teaching math to young children: A practice guide (No. NCEE 2014-4005). National Center for Education and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/early_math_pg_111313.pdf
30.
FuchsD.FuchsL. S. (2018). On the importance of moderator analysis in intervention research: An introduction to the special issue. Exceptional Children, 85(2), 126–128. https://doi.org/10.1177/0014402918811924
31.
FuchsL. S.ComptonD. L.FuchsD.PaulsenK.BryantJ. D.HamlettC. L. (2005). The prevention, identification, and cognitive determinants of math difficulty. Journal of Educational Psychology, 97(3), 493–513. https://doi.org/10.1037/0022-0663.97.3.493
32.
GamseB. C.BloomH. S.KempleJ. J.JacobR. T.BoulayB.BozziL.CaswellL. J.HorstM.SmithW. C.St. PierreR. G.UnluF.HerlihyC.ZhuP.GreeneD.KimJ.LaLibertyD.LamK.MareeK.McCormickR., . . . WongE. (2008). Reading First impact study (Interim Report No. NCEE 2008-4016). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://files.eric.ed.gov/fulltext/ED501218.pdf
GerstenR. M.BeckmannS.ClarkeB.FoegenA.MarchL.StarJ. R.WitzelB. (2009). Assisting students struggling with mathematics: Response to intervention (RtI) for elementary and middle schools (Practice Guide Report No. NCEE 2009-4060). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/rti_math_pg_042109.pdf
35.
GerstenR. M.ChardD. J.JayanthiM.BakerS. K.MorphyP.FlojoJ. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research, 79(3), 1202–1242. https://doi.org/10.3102/0034654309334431
36.
GreenlandS.SennS. J.RothmanK. J.CarlinJ. B.PooleC.GoodmanS. N.AltmanD. G. (2016). Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3
37.
HanichL. B.JordanN. C.KaplanD.DickJ. (2001). Performance across different areas of mathematical cognition in children with learning difficulties. Journal of Educational Psychology, 93(3), 615–627. https://doi.org/10.1037/0022-0663.93.3.615
38.
Kame’enuiE. J.SimmonsD. C. (1999). Beyond effective practices to schools as host environments: Building and sustaining a school-wide intervention model in beginning reading for all children. Australasian Journal of Special Education, 23(2-3), 100–127. https://doi.org/10.1080/1030011990230204
39.
KaminskiR. A.CummingsK. D.Powell-SmithK. A.GoodR. H. (2008). Best practices in using dynamic indicators of basic early literacy skills for formative assessment and evaluation. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology (Vol. 5, pp. 1181–1204). NASP.
40.
LemonsC. J.FuchsD.GilbertJ. K.FuchsL. S. (2014). Evidence-based practices in a changing world: Reconsidering the counterfactual in education research. Educational Researcher, 43(5), 242–252. https://doi.org/10.3102/0013189X14539189
41.
LittleR. J. A.RubinD. B. (2002). Statistical analysis with missing data (2nd ed.). Wiley.
42.
MakelM. C.PluckerJ. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304–316. https://doi.org/10.3102/0013189X14545513
43.
MakelM. C.PluckerJ. A.FreemanJ.LombardiA.SimonsenB.CoyneM. (2016). Replication of special education research: Necessary but far too rare. Remedial and Special Education, 37(6), 205–212. https://doi.org/10.1177/0741932516646083
44.
MakelM. C.PluckerJ. A.HegartyB. (2012). Replications in psychology research: How often do they really occur?Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688
45.
MillerA. C.FuchsD.FuchsL. S.ComptonD.KearnsD.ZhangW.YenL.PattonS.KirchnerD. P. (2014). Behavioral attention: A longitudinal study of whether and how it influences the development of word reading and reading comprehension among at-risk readers. Journal of Research on Educational Effectiveness, 7(3), 232–249. https://doi.org/10.1080/19345747.2014.906691
46.
MorganP. L.FarkasG.WuQ. (2009). Five-year growth trajectories of kindergarten children with learning difficulties in mathematics. Journal of Learning Disabilities, 42(4), 306–321. https://doi.org/10.1177/0022219408331037
47.
MorganP. L.HillemeierM. M.FarkasG.MaczugaS. (2014). Racial/ethnic disparities in ADHD diagnosis by kindergarten entry. Journal of Child Psychology and Psychiatry, 55(8), 905–913. https://doi.org/10.1111/jcpp.12204
48.
MurrayD. M. (1998). Design and analysis of group-randomized trials. Oxford University Press.
49.
National Assessment Educational Progress. (2019). Results from the 2019 mathematics and reading assessments (NCES No. 2020012). National Center for Education Statistics, U.S. Department of Education, Institute of Education Sciences. https://www.nationsreportcard.gov/mathematics?grade=4
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. U.S. Department of Education.
53.
National Research Council. (2001). Adding it up: Helping children learn mathematics. Mathematics Learning Study Committee.
54.
National Research Council. (2009). Mathematics learning in early childhood: Paths toward excellence and equity. National Academies Press. http://www.nap.edu/catalog.php?record_id=12519
55.
National Science Foundation & Institute of Education Sciences. (2018). Companion guidelines on replication and reproducibility in education research: A supplement to the common guidelines for education research and development. National Science Foundation, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/pdf/CompanionGuidelinesReplicationReproducibility.pdf
56.
NelsonG.McMasterK. L. (2019). The effects of early numeracy interventions for students in preschool and early elementary: A meta-analysis. Journal of Educational Psychology, 111(6), 1001–1022. https://doi.org/10.1037/edu0000334
57.
Pro-Ed. (2007). Test of early mathematics ability–Third edition.
58.
SAS Institute. (2016). Base SAS 9.4 procedures guide: Statistical procedures (5th ed.).
SchmidtS. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
61.
ShinnM. R. (1989). Curriculum-based measurement: Assessing special children. Guilford Press.
62.
SimmonsD. C.Kame’enuiE. J.HarnB.CoyneM.StoolmillerM.SantoroL.SmithS.BeckC.KaufmanN. K. (2007). Attributes of effective and efficient kindergarten reading intervention: An examination of instructional time and design of instructional specificity. Journal of Learning Disabilities, 40(4), 331–347. https://doi.org/10.1177/00222194070400040401
63.
SoodS.JitendraA. K. (2013). An exploratory study of a number sense program to develop kindergarten students’ number proficiency. Journal of Learning Disabilities, 46(4), 328–346. https://doi.org/10.1177/0022219411422380
64.
SidmanM. (1960). Tactics of scientific research: Evaluating experimental data in psychology. Basic Books.
65.
WassersteinR. L.LazarN. A. (2016). The ASA statement on p-values: Context, process, and purpose. American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
66.
WattsT. W.DuncanG. J.SieglerR. S.Davis-KeanP. E. (2014). What’s past is prologue: Relations between early mathematics knowledge and high school achievement. Educational Researcher, 43(7), 352–360. https://doi.org/10.3102/0013189X14553660
