The present study evaluated the effectiveness of a non-randomized summer reading program offered by a school district meeting or exceeding all National Summer Learning Project recommendations. The district had 10 years of experience delivering the program to Grades 1–5 students with or at risk of reading disabilities. Voluntary participation had been declining in the three years prior to the pandemic and declined further during the pandemic. In 2021, 144 eligible students attended at least half the summer program and 1113 students were eligible but did not participate. All students were pre- and post-tested with a measure of overall reading ability. After propensity score weighting, the data from each grade were analyzed using multilevel models. Results revealed no statistically significant differences between the treatment and control groups, nor did special education and free or reduced-price lunch status significantly moderate outcomes. Implications for future summer reading programs are discussed.
AugustineC. H.McCombsJ. S.PaneJ. F.SchwartzH. L.SchweigJ.McEachinA.Siler-EvansK. (2016). Learning from summer: Effects of voluntary summer learning programs on low-income urban youth. RAND Corporation, RR-1557-WF. https://www.rand.org/pubs/research_reports/RR1557.html
2.
AustinP. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. https://doi.org/10.1080/00273171.2011.568786
3.
AustinP. C.StuartE. A. (2015). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661–79. https://doi.org/10.1002/sim.6607
4.
Barnard-BrakL.StevensT. (2021). Criteria for determining eligibility for extended school year services. The Journal of Special Education, 55(1), 3–10. https://doi.org/10.1177/0022466920911468
5.
BeachK. D.McIntyreE.PhilippakosZ. A.MrazM.PilonietaP.VintinnerJ. P. (2018). Effects of a summer reading intervention on reading skills for low-income black and Hispanic students in elementary school. Reading & Writing Quarterly, 34(3), 263–80. https://doi.org/10.1080/10573569.2018.1446859
6.
BeachK. D.WashburnE. K.GeselS. A.WilliamsP. (2021). Pivoting an elementary summer reading intervention to a virtual context in response to COVID-19: An examination of program transformation and outcomes. Journal of Education for Students Placed at Risk, 26(2), 112–34. https://doi.org/10.1080/10824669.2021.1906250
7.
BormanG. D.GoetzM. E.DowlingN. M. (2009). Halting the summer achievement slide: A randomized field trial of the KindergARTen summer camp. Journal of Education for Students Placed at Risk, 14(2), 133–47. https://doi.org/10.1080/10824660802427652
8.
CallenI.CarbonariM. V.DeArmondM.DeweyD.Dizon-RossE.GoldhaberD.IsaacsJ.KaneT. J.KuhfeldM.McDonaldA.McEachinA.MortonE.MurogaA.StaigerD. O. (2023). Summer school as a learning loss recovery strategy after COVID-19: Evidence from summer 2022(Working Paper No. 291-0823). CALDER. https://caldercenter.org/sites/default/files/CALDER%20WP%20291-0823.pdf
9.
ChristodoulouJ. A.CyrA.MurtaghJ.ChangP.LinJ.GuarinoA. J.HookP.GabrieliJ. D. E. (2017). Impact of intensive summer reading intervention for children with reading disabilities and difficulties in early elementary school. Journal of Learning Disabilities, 50(2), 115–27. https://doi.org/10.1177/0022219415617163
10.
ContesseV. A.CampeseT.KaplanR.MullenD.PicoD. L.GageN. A.LaneH. B. (2021). The effects of an intensive summer literacy intervention on reader development. Reading & Writing Quarterly, 37(3), 221–39. https://doi.org/10.1080/10573569.2020.1765441
11.
CooperH.CharltonK.ValentineJ. C.MuhlenbruckL. (2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development, 65(1), i-127. https://www.jstor.org/stable/3181549
12.
CooperH.NyeB.CharltonK.LindsayJ.GreathouseS. (1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66(3), 227–68. https://doi.org/10.2307/1170523
13.
CooperH.ValentineJ. C.CharltonK.MelsonA. (2003). The effects of modified school calendars on student achievement and on school and community attitudes. Review of Educational Research, 73(1), 1–52. https://doi.org/10.3102/00346543073001001
DilibertiM. K.SchwartzH. L. (2024). School was in session this summer, but less than half of eligible students enrolled: Selected findings from the fall 2023. American School District Panel Survey. RAND Corporation. https://www.rand.org/pubs/research_reports/RRA956-22.html
16.
DurlakJ. A.DuPreE. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–50. https://doi.org/10.1007/s10464-008-9165-0
EglitisE.MiatkeA.VirgaraR.MachellA.OldsT.RichardsonM.MaherC. (2024). Children’s health, wellbeing and academic outcomes over the summer holidays: A scoping review. Children, 11(3), 287–315. https://doi.org/10.3390/children11030287
19.
FongC.HazlettC.ImaiK. (2018). Covariate balancing propensity score for a continuous treatment: Application to the efficacy of political advertisements. The Annals of Applied Statistics, 12(1), 156–77. https://doi.org/10.1214/17-aoas1101
20.
GriffinB. A.SchulerM. S.CefaluM.AyerL.GodleyM.GreiferN.McCaffreyD. F. (2023). A tutorial for propensity score weighting for moderation analysis with categorical variables: An application examining smoking disparities among sexual minority adults. Medical Care, 61(12), 836–45. https://doi.org/10.1097/mlr.0000000000001922
21.
HarnB.ParisiD.StoolmillerM. (2013). Balancing fidelity with flexibility and fit: what do we really know about fidelity of implementation in schools?Exceptional Children, 79(3), 181–93. https://doi.org/10.1177/001440291307900204
22.
ImaiK.RatkovicM. (2014). Covariate balancing propensity score. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1), 243–63. https://doi.org/10.1111/rssb.12027
23.
IrwinV.WangK.TezilT.ZhangJ.FilbeyA.JungJ.Bullock MannF.DiligR.ParkerS. (2023). Report on the condition of education 2023 (NCES 2023-144). U.S. Department of Education, National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2023144
LekoM. M.RobertsC. A.PekY. (2015). A theory of secondary teachers’ adaptations when implementing a reading intervention program. The Journal of Special Education, 49(3), 168–78. https://doi.org/10.1177/0022466914546751
26.
LuncefordJ. K.DavidianM. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine, 23(19), 2937–60. https://doi.org/10.1002/sim.1903
National Center for Education Statistics. (n.d.). National assessment of educational progress (NAEP), various years, 1992-2022 reading assessments. U.S. Department of Education, Institute of Education Sciences. https://www.nationsreportcard.gov/reading/nation/groups
Rambo-HernandezK. E.McCoachD. B. (2015). High-achieving and average students’ reading growth: Contrasting school and summer trajectories. The Journal of Educational Research, 108(2), 112–29. https://doi.org/10.1080/00220671.2013.850398
31.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
32.
R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/
33.
ReedD. K.AloeA. M.ReegerA.FolsomJ. S. (2019). Defining summer gain among elementary students with or at-risk for reading disabilities. Exceptional Children, 85(4), 413. https://doi.org/10.1177/0014402918819426
34.
ReedD. K.AloeA. M.ReegerA. J.ParkS. (2021). Exploring the summer reading effect through visual analysis of multiple datasets. Journal of Research in Reading, 44(3), 597–616. https://doi.org/10.1111/1467-9817.12357
ReedD. K.MeginnisT.ParkS.GibbsA. (2023). Surveying elementary schools’ summer reading interventions in a state policy context. Remedial and Special Education, 44(1), 16–27. https://doi.org/10.1177%2F07419325221075729
RobertsG. J.CapinP.RobertsG.MiciakJ.QuinnJ. M.VaughnS. (2018). Examining the effects of afterschool reading interventions for upper elementary struggling readers. Remedial and Special Education, 39(3), 131–143. https://doi.org/10.1177/0741932517750818
39.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
von HippelP. T. (2023). Estimating learning when test scores are missing: The problem and two solutions. (EdWorkingPaper: 23-864). Annenberg Institute at Brown University. https://doi.org/10.26300/07bv-by90
43.
von HippelP. T.HamrockC. (2019). Do test score gaps grow before, during, or between the school years? Measurement artifacts and what we can know in spite of them. Sociological Science, 6(3), 43–80. https://doi.org/10.2139/ssrn.2745527
44.
WashburnE. K.GeselS. A.FitzgeraldM. S.BeachK. D.KingsberyC. R. (2023). The impact of a comprehensive, evidence-based approach to summer literacy intervention on the K-3 reading skills of economically and culturally diverse students. Reading & Writing Quarterly, 29(6), 510–529. https://doi.org/10.1080/10573569.2022.2147463
WilkinsC.GerstenR.DeckerL.GrundenL.BrasielS.BrunnertK.JayanthiM. (2012). Does a summer reading program based on Lexiles affect reading comprehension?(NCEE 2012-4006). U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. https://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_20124006.pdf
