Sage Journals: Discover world-class research

Abstract

Summer learning loss – decreased academic performance following an extended school break, typically during the period after one grade ends and before another grade starts – is a well-documented phenomenon in North America, but poorly described in sub-Saharan African contexts. In this article, we use the term ‘grade-transition break’ loss in lieu of ‘summer’ loss to refer to the period after one grade ends and before another grade starts. This study analyses data from early grade reading assessments in Malawi, estimating statistically significant average reductions of 0.38 standard deviations (SD) across several measures of reading and pre-reading skills during two grade-transition breaks. The data show the loss in reading skills during the extended breaks between grades 1 and 2 and between grades 2 and 3 in two consecutive years. The study found no gender-based differences in loss. The findings suggest a need for early grade reading interventions to develop and evaluate mitigation strategies lest significant proportions of within-year performance gains be lost over the break between academic years.

Keywords

Reading literacy Malawi Africa reading difficulties reading acquisition summer learning loss grade transition

Introduction

Remarkable strides have been made towards achieving universal primary education, largely the result of the enthusiasm brought about by the global Education for All movement and the second Millennium Development Goal. These accomplishments have been particularly exceptional in sub-Saharan Africa, where the enrolment rate increased from 52% in 1990 to 80% in 2015 (UNESCO Institute for Statistics (UIS), 2015). In Malawi, for example, primary school enrolment increased from 1.8 million to 3.2 million soon after the declaration of Free Primary Education in 1994 (National Statistical Office, 2011).

Despite these achievements, significant challenges to the quality of education in sub-Saharan Africa remain. Recent assessment data suggest that children in the region have alarmingly low levels of literacy. In Malawi, 98% of grade 2 students and 56% of grade 4 students were unable to read a single word in English (Dowd et al., 2010; UNESCO, 2012). Results elsewhere on the continent are equally disheartening, with disturbingly low literacy outcomes identified by many countries across sub-Saharan Africa (Gove and Cvelich, 2011).

A need to increase emphasis on improving learning outcomes has been identified by many countries in the region, including Kenya (MoEST, 2015), Ethiopia (MOE, 2013) and Malawi (Global Partnership for Education, 2016; MoEST, 2008). Recent research has shown a much stronger link between learning, and particularly literacy, and economic outcomes such as labour productivity and growth in the gross domestic product (Coulombe and Tremblay, 2006; National Institute for Literacy, 2010).

The link between education and economic growth, though well-established in the economic literature, typically has been estimated using quantitative measures of education, such as years of schooling, rather than cognitive skills or learning outcomes. Hanushek and Woessmann (2015) argued that tests of cognitive skills, such as international tests of student performance, are far more important determinants of economic growth than years of schooling. Initial income, years of schooling and cognitive skills explain 73% of the variation in economic growth, as opposed to 25% of the variation when cognitive skills are removed from the model (Hanushek and Woessmann, 2015). Like them, other researchers are now concluding that education quality as measured by learning is a more robust measure of educational output than the aforementioned quantity measures.

Thus, focusing on education outcomes in terms of learning rather than years of schooling is central to policy making (Glewwe and Muralidharan, 2016). Evidence suggests that more schooling does not necessarily correlate with education outcomes beyond that predicted by the cross-country relationship between per-capita income and learning outcomes (Singh, 2016). On the other hand, interventions that aim to improve pedagogy, such as teaching of literacy skills, have been found to produce consistently positive effects on learning outcomes, which in turn result in greater returns on spending (Snilstveit et al., 2015).

Policy makers in sub-Saharan Africa have rapidly mobilized to shift their focus from merely improving access, to ensuring that schooling translates to increases in learning outcomes. Over the past several years, real expenditure on education rose by 6% annually (Gove and Cvelich, 2011; UIS, 2011). Many countries are increasing their investments in evidence-based literacy programs. Among them are South Africa (Gauteng Primary Language and Mathematics Strategy – Fleisch and Schöer, 2014), Kenya (the Primary Math and Reading [PRIMR] Initiative and the Tusome national literacy program – Piper et al., 2015, 2014) and Gambia (Early Literacy in National Languages pilot program – UNESCO, 2014). Effects from these medium- to large-scale programs are encouraging, with effect sizes ranging from 0.34 to 0.58 standard deviations (SD).

This emphasis on education outcomes can also help policy makers look beyond school-based interventions to home- or community-based ones. This is because pedagogical interventions alone (such as the literacy- and numeracy-focused programs mentioned above) may not be enough to raise literacy rates to the levels that sub-Saharan African policy makers are targeting. To illustrate, community-based interventions were found to increase test scores ranging from 0.09 to 0.12 standardized mean differences, and were classified as having ‘high’ impact on learning outcomes compared to all other interventions in the meta-analysis (Snilstveit et al., 2015). Recent research from Save the Children showed that the time out of school was where educational inequities expanded most rapidly, and that children spent more time out of school than in school, when holidays, breaks, and afternoons and evenings were counted (Friedlander, Guajardo, et al., 2012). Literature from the US has demonstrated that, with respect to learning outcomes, students’ rate of cognitive gains decreases during the summer months (Cooper et al., 1996; Heyns, 1978). The summer breaks have consistently shown an increase in inequitable education growth, especially for students of low socioeconomic status (SES) (Alexander et al., 2001; Cooper et al., 1996), minorities (Kim, 2004) and low-SES minorities (O’Brien, 1999), and potentially for boys (Bakle, 2010; Guryan et al., 2014). As a result, summer loss is a well-established part of the educational literature, with nuanced understandings of which groups are most adversely affected by summer loss (Cooper et al., 1996).

In order to reduce this learning loss, several recent interventions have attempted to support the learning of students in the summers (Guryan et al., 2014). This research has increased attention to cost (McCombs et al., 2014) and to scalability in low-income settings in the US (Kim et al., 2016).

There is little evidence from sub-Saharan Africa regarding ‘summer’ or grade-transition learning loss. We are unclear as to whether such loss occurs, or whether the sub-Saharan African context is different from the US literature in ways that make grade-transition loss either not applicable or negligible in effect. Comparisons are complicated by the fact that countries structure their academic years differently, with some using terms and others semesters. We have limited literature on whether grade-transition loss in sub-Saharan Africa would be sensitive to intervention, as it is in the US, although Save the Children’s work on life-wide learning reported encouraging evidence of how community activities could support out-of-school learning in general, albeit not yet specific to grade-transition loss (Friedlander, Dowd, et al., 2012).

In order to examine grade-transition loss in a sub-Saharan African setting, the authors of this paper utilized the unique Malawi Learner Assessment Tool (LAT) data set. More than 10,000 LAT assessments, in the Chichewa language, were undertaken across several literacy tasks at 11 data points over 3 years. The four rounds of data collected before and after the long break between academic years allowed us to estimate rates of summer loss in Malawi.

Background

Summer loss in the US

Summer learning loss in the US is a measurable problem that has been well documented in academic literature, and studies examining seasonal learning rates date back over a century (White, 1906). The US summer holiday, with its roots in the needs of an agrarian society to have a long harvest season, can last up to 12 weeks and represents a substantial interruption to the rhythm of instruction (Kerry and Davies, 1998). In their meta-analysis of 13 studies on summer loss, Cooper et al. (1996) found that at best, students achieved no cognitive gains during the summer and on average, their test scores in reading (particularly spelling) and mathematics declined over the summer, amounting to nearly one month’s worth of loss on a grade-level scale.

Murnane (1975) suggested that some skills are more likely to deteriorate over the summer because they are based on conceptual or factual knowledge which requires a significant amount of practice, as opposed to conceptual knowledge that requires experience rather than practice (Cooper et al., 1996). Thus, the skills that are most likely to decay during the summer depend on ‘differences in opportunities to practice and the susceptibility of certain knowledge to be dissipated without practice’ (Cooper et al., 1996: 261). According to the self-teaching hypothesis, procedural skills such as letter-sound fluency, syllable fluency, fluency in decoding syllables, familiar words and oral reading fluency of connected text require practice and, specifically, successful practice, to deepen (Share and Stanovich, 1995).

Summer loss is not evenly distributed across the population and disproportionately affects low-income students (Allington et al., 2010). Cooper et al. (1996) estimated the annual reading achievement gap between students from low- and middle-income families to be around three months. This substantial difference could occur because opportunities to practise reading skills over the summer are closely linked to students’ socioeconomic background (Lareau, 2011). However, a significant gap remained between low- and high-SES students when researchers controlled for summer activities, estimated at 0.15 SD (Gershenson, 2013). Summer learning loss is also cumulative; the compounding losses between the primary grades were estimated to account for more than half of the achievement gap between high- and low-SES students in the ninth grade (Alexander et al., 2007).

There is also some evidence that summer break sets back minority students more than their white peers (Phillips and Chin, 2004). In fact, middle-class students were found to demonstrate modest gains in reading proficiency during the summer (Cooper et al., 2003). Therefore, a summer holiday can both cause learning losses generally and exacerbate inequities between students, mainly along socioeconomic and racial lines (Heyns, 1978).

Differences in summer loss by gender

While the evidence is clear that SES attenuates the magnitude and nature of summer loss, evidence on the role of gender in summer reading loss is mixed. Cooper et al. (1996) found that gender did not have a significant moderating effect on cognitive loss during the summer. Other studies, however, have suggested otherwise. Downey et al. (2004) found that an existing gender gap favouring girls in literacy rates grew faster when school was not in session. And among low-SES students with unusually high summer cognitive gains (comparable to their high-SES counterparts), girls had significantly better outcomes than boys (Slates et al., 2012).

Evidence from the US also suggests that girls may respond more favourably to summer literacy interventions. In one randomized controlled trial of a home-based reading intervention in which students received both books that matched their skill and interests and corresponding comprehension lessons, only third-grade girls showed significant treatment effects (Cohen’s d = 0.073), while boys had insignificant treatment effects (Cohen’s d = −0.024) (Guryan et al., 2014). Another study examining the impact of a summer-school program on language usage found no differences by gender in grade 2 or 3; however, gender had a significant main effect at fifth grade. For fourth-grade students, a significant interaction effect was identified between attending summer school and gender among fourth-grade students, where girls outperformed boys (Bakle, 2010).

Some studies were not designed to differentiate between whether the gender gap that existed during the school year simply carried into the summer months, or whether the magnitude of summer loss differed by gender (Slates et al., 2012). Basic gender gaps exist in many contexts, with girls outperforming boys in reading in the US (Willingham and Cole, 1997) and in sub-Saharan Africa (Gove and Cvelich, 2011). Possible explanations for the gender-differentiated summer loss results include girls’ more favourable attitudes towards recreational reading (Logan and Johnston, 2009), a higher tendency among girls to take home and read more books in the summer (Heyns, 1978) and evidence that their parents were more likely to take them to the library, although parents were found to spend more time reading to sons (Willingham and Cole, 1997).

Interventions that address summer loss

Several types of interventions have been devised to ameliorate summer loss. These include extending the school year or modifying the calendar (Cooper et al., 1996; Von Hippel, 2016). Card and Krueger (1992) found that when they controlled for other factors, such as teacher wages, term length had no effect on educational attainment. This finding is relevant for sub-Saharan Africa, since many countries – such as Malawi – utilize two shorter term breaks and one longer grade-transition break. Cooper and colleagues (2003) conducted a meta-analysis on 39 US school districts with modified calendars, where breaks were shorter than the standard summer vacation and equally sized, and the number of instructional days did not exceed approximately 180. The analysis revealed that schools with revised calendars had a small impact on student achievement, with stronger results among schools serving low-income students. The case for modified calendars was lessened, however, by the limited and contextual evidence, the high costs and the complex logistical planning (Cooper et al., 2003).

Other policy alternatives to mitigate summer loss generally fall into two categories: classroom-based and home-based interventions (Kim and Quinn, 2013). While the evidence suggests comparable impacts on learning outcomes in these interventions, there can be considerable differences in the costs (Kim and Quinn, 2013; McCombs et al., 2011). Classroom interventions involve instructional activities such as summer school, carried out by schoolteachers and university researchers, for example, and remedial or preparatory teacher-led efforts to increase student achievement. Positive impacts have been recorded for these types of interventions (Cooper et al., 2000), estimated at 0.09 SD (Kim and Quinn, 2013).

Home-based interventions are designed to respond to the literature showing that reading during the summer months is highly correlated to academic achievement (Chin and Phillips, 2004; Heyns, 1978). Positive impacts have also been recorded for interventions such as increasing access to (self-selected) high-quality books (Allington et al., 2010) and voluntary summer book reading programs with scaffolding (Kim and White, 2008), with effect sizes of 0.14 and 0.09 SD respectively. In their meta-analysis, Kim and Quinn (2013) estimated the average effect size of the home-based interventions at 0.12 SD. Costs for these programs are much lower than for classroom interventions, such that they have been put forward as scalable and cost-effective policy solutions to mitigate summer loss (McCombs et al., 2011).

Although many studies have invested in understanding whether literacy outcomes are sensitive to intervention, we could identify no evidence to help determine whether summer loss was affected by generic literacy interventions that had no specific components addressing summer loss. It might be that summer loss in literacy is a significant problem, but that interventions designed to improve literacy skills during the school year could reduce or remove that summer loss.

‘Summer’ loss in the global South

Research on ‘summer’ (‘grade-transition’ in low-latitude contexts) loss outside of the US and Canada is scant (Lindahl, 2001), and to our knowledge, there is no literature examining whether grade-transition loss exists in sub-Saharan Africa. Save the Children has increased its research into out-of-school factors and their relationship with learning outcomes in sub-Saharan Africa (Friedlander, Dowd, et al., 2012). Weiseman and Baker (2004) argued that the summer learning loss phenomenon might be unique to the US because in comparison to other industrial nations, the US has longer formal breaks (approximately 12 weeks) and shorter overall instructional time. However, other studies have shown that summer loss exists in Canada (Davies et al., 2016), Sweden (Lindahl, 2001), the Netherlands (Verachtert et al., 2009) and Germany (Meyer, 2014). In their study of summer learning loss in Canada, Davies et al. (2016) concluded that the phenomenon will be present in any setting where there is socioeconomic stratification.

Context

Malawi is a small, landlocked Anglophone country in southern Africa, with Chichewa as the national language. Its population density and average class size are among the highest in the world, with 182 people per square kilometre and nearly 90 students per classroom (UIS, 2015; World Bank, 2015). It is among the world’s least developed nations with roughly 80% of its 17 million people living in rural areas (UNDP, 2015). Learning outcomes are low; Malawi’s learning outcomes rank at the 17th percentile in comparison to other low- and middle-income countries (Education Policy and Data Center (EPDC), 2014). Recent nationally representative literacy assessments have shown that 92% of grade 1 and 47% of grade 3 students were unable to read a single word in English (Nagarajan et al., 2015).

Malawi’s primary education is free but not compulsory, and there are approximately 4 million students at the primary level. At the end of primary school, students sit for the Primary School Leaving Certificate Examination (PSLCE) (EPDC, 2014; Kananji, 2014). The school year runs from September to July, and the country has a trimester system with terms running from September to December, January to March/April, and April to July. Breaks between terms vary from three to seven weeks (EPDC, 2014); the 2014 and 2015 grade-transition breaks lasted approximately seven weeks.

The Malawi Early Grade Reading Activity was a USAID-funded intervention designed to support the Ministry of Education, Science and Technology (MoEST) to improve reading outcomes in Malawian public primary schools. It was implemented from 2013 to 2016 in 1600 schools across 11 educational districts. The intervention included teacher training, instructional coaching by MoEST primary education advisers, and provision of scripted lesson plans and learner books. It was rolled out in a phased approach, creating treatment and control groups. The Activity was implemented in both English and Chichewa, although this article focuses on the Chichewa results.

The program’s research design is depicted in Table 1. Teachers and students received Chichewa-specific training, instructional materials and coaching in grade 1 (‘Standard 1’ in Malawi) in the 2013–2014 school year, grade 2 in the 2014–2015 school year, and grade 3 in the 2015–2016 school year.

Table 1.

Implementation design for the Early Grade Reading Activity, by grade.

Grade	Sep 2013–Jul 2014	Sep 2014–Jul 2015	Sep 2015–Jul 2016
1
2
3

The intervention’s monitoring and evaluation strategy included periodic collection of student performance data using the LAT (see Table 2). LAT data were collected in October (near the beginning of the school year), January (the beginning of the second term), March (the end of the second term), and July (at the end of the school year) for each year of implementation.¹ The assessments were administered to different students for Chichewa and English in grades 1–3 for the first five LAT rounds (until January 2015). Beginning in the sixth LAT round (March 2015), the selected students were assessed in both Chichewa and English.

Table 2.

Students assessed in Chichewa for LAT 1 through LAT 11, by grade and gender.

LAT round	Grade 1		Grade 2		Grade 3		Total
LAT round	Boys	Girls	Boys	Girls	Boys	Girls	Total
LAT 1 – Jan 14	155	157	145	157	143	130	887
LAT 2 – Mar 14	168	163	150	163	160	160	964
LAT 3 – Jul 14	166	157	146	164	141	153	927
LAT 4 – Oct 14	170	169	166	171	165	165	1006
LAT 5 – Jan 15	170	171	151	159	164	180	995
LAT 6 – Mar 15	159	157	164	164	156	157	957
LAT 7 – Jul 15	169	165	165	165	167	161	992
LAT 8 – Oct 15	165	185	157	167	185	171	1030
LAT 9 – Jan 16	165	156	164	166	161	168	980
LAT 10 – Mar 16	163	159	163	165	162	162	974
LAT 11 – Jul 16	166	166	165	165	165	165	992
Total	1816	1805	1736	1806	1769	1772	10,704

The LAT instrument had items measuring skills in letter naming, phonemic awareness, syllable decoding, familiar word reading, oral reading fluency of connected text and reading comprehension, which we describe below.

The same LAT form was used for each assessment within a school year. Between the first and second school years, the grid items (i.e. letters, syllables or familiar words that were presented in rows and columns on a single sheet) were re-randomized. Between each school year, the reading passages and associated comprehension questions were modified. An equating study was conducted to assess the comparability of the reading passages, and a mean equating factor was applied so that oral reading fluency scores would be comparable (Kolen and Brennan, 2004).

Research questions

The literature reviewed above shows that summer loss in literacy exists in several Northern contexts. Summer loss is particularly large for the poor and for ethnic minorities, but the evidence remains mixed as to whether summer loss differs by gender, and whether summer loss is a phenomenon experienced only in the absence of instructional interventions. Our review of the research suggests that while this body of research is growing in the North, with more written on the US than elsewhere, it is a gap in the literature with respect to sub-Saharan Africa in general, and Malawi in particular. The increased emphasis on literacy outcomes in sub-Saharan Africa seems to not have resulted in increased emphasis on grade-transition loss. Therefore, in this study, we address the following research questions:

RQ1: Did students have lower literacy outcomes after the grade-transition break than before?

RQ2: Did the magnitude of grade-transition loss differ by gender?

RQ3: Did the magnitude of grade-transition loss differ by assignment to a generic literacy treatment group?

Methodology

Measures

The LAT measured a set of items that we describe below. The fluency scores were a function of both accuracy and automaticity (speed), as described.

Letter naming: A timed task in which the learner identified the names of letters presented in a grid consisting of 10 rows and 10 columns (10 × 10). Fluency scores were reported in correct letters per minute (clpm); there was no theoretical maximum.

Syllable reading: A timed task in which the learner read aloud the sounds represented by syllables presented in a 10 × 10 grid. Fluency scores were reported in correct syllable sounds per minute (csspm); there was no theoretical maximum.

Familiar word reading: A timed task in which the learner read aloud familiar words from a 5 × 10 grid. Fluency scores were reported in correct words per minute (cwpm); there was no theoretical maximum.

Oral reading fluency: A timed task in which the learner read aloud a simple passage of approximately 50–70 words in length. Fluency scores were reported in cwpm (orf).² There was no theoretical maximum.

Sampling

The Malawi LAT sampling frame used multi-stage random sampling. Three stages of sampling were executed at the zonal, school and grade levels. At the first stage, three intervention zones were selected at random from within districts. (There are between 9 and 15 zones per district.) One school was then selected at random from within each sampled zone. At each school, students were stratified by gender within grade, and equal numbers of girls and boys were randomly selected from each grade. For LAT rounds 1–5 (in which students were assessed separately for Chichewa and English), the target sample size was 1980 students (20 per grade per school); for LAT rounds 6–11, the target sample size was 990 students (10 per grade per school). The LAT sample of schools was maintained for one full school year, with a fresh sample drawn for the new school year. The sampling of LAT schools was done in early October (at the beginning of each school year).

Analysis

This section presents the design of the LAT study. Different students were sampled at each LAT round; thus, while individual students may have repeated a grade, at an aggregate level, cohorts of students progressed from one grade to another across years.

In the Malawi LAT data, students were grouped into cohorts. Cohort 1 students were in grade 1 during the 2013–2014 school year, grade 2 in 2014–2015, and grade 3 in 2015–2016. Cohort 2 students followed one year behind, and were in grade 1 during the 2014–2015 school year and grade 2 in 2015–2016. Cohort 3 students started in grade 2 during the 2013–2014 school year, moving to grade 3 in 2014–2015. Cohort 1 students faced two grade transitions, while Cohorts 2 and 3 each faced one grade transition.

We were interested in knowing whether grade-transition loss would differ by whether the school was included in an intervention, such as the Early Grade Reading Activity. We compared grade 2 and 3 Cohort 1 and 2 children with grade 2 and 3 Cohort 3 children, since teachers and learners in Cohort 3 were not exposed to the intervention.³

We estimated grade-transition loss in the following way. The delta (Δ) for the grade-transition loss estimate is simply the difference between the sample means pre- and post-grade-transition.

Δ = {\bar{X}}_{p o s t} - {\bar{X}}_{p r e}

In order to compare the magnitude of grade-transition loss, we calculated the Cohen’s d effect size (Cohen, 1988). This was estimated by dividing the absolute mean difference by the pooled SD of the two estimates, as

\frac{a b s (Δ)}{S}

where S is the pooled SD.

To determine whether there were differences in summer loss by gender, we fit a differences-in-differences (DiD) model comparing the fluency rates pre- and post-grade-transition for boys and girls, as per the following equation:

({\bar{X}}_{b o y s, p o s t} - {\bar{X}}_{b o y s, p r e}) - ({\bar{X}}_{g i r l s, p o s t} - {\bar{X}}_{g i r l s, p r e})

The gender difference is expressed in the DiD mean, and we present the Cohen’s d effect size (Cohen, 1988) in the Findings section below.

Findings

RQ1: Did students have lower literacy outcomes after the grade-transition break than before?

The first research question reflects our interest in determining whether there were differences in learning outcomes for literacy before and after Malawi’s long break between grades. The three cohorts of children assessed in LAT faced a grade-transition break four times. There were two transitions between grade 1 and 2, in both 2014 and 2015; and two transitions between grade 2 and 3, in both 2014 and 2015. Table 3 presents our findings. The table shows the mean scores before and after the four transitions for the four measures (letter sound fluency, syllable fluency, familiar word fluency and oral reading fluency in connected text). The table also shows the grade-transition loss, the results of the statistical test determining whether the loss was statistically significant and the Cohen’s d effect size of the loss.

Table 3.

Grade-transition loss estimates by LAT task. Standard errors in parentheses.

	Grade transition	Year	Before break	After break	Grade-transition loss	T score (p-value)	Effect size	Average effect size
Letter fluency	1 to 2	2015	23.5	12.3	11.2(1.35)	−8.32***	0.65
Letter fluency	1 to 2	2014	15.4	6.3	9.1(1.09)	−8.34***	0.66
Letter fluency	2 to 3	2015	37.6	24.9	12.7(1.83)	−6.92***	0.54
Letter fluency	2 to 3	2014	21.2	12.5	8.7(1.54)	−5.66***	0.45	0.58
Syllable fluency	1 to 2	2015	13.8	8.7	5.1(1.31)	−3.89***	0.31
Syllable fluency	1 to 2	2014	6.5	3.1	3.4(0.72)	−4.80***	0.38
Syllable fluency	2 to 3	2015	31.6	21.4	10.2(1.94)	−5.24***	0.41
Syllable fluency	2 to 3	2014	13.9	8.9	5.0(1.42)	−3.56***	0.28	0.35
Familiar word fluency	1 to 2	2015	5.5	3.8	1.73(0.7)	−2.46*	0.19
Familiar word fluency	1 to 2	2014	3.9	1.4	2.4(0.56)	−4.31***	0.34
Familiar word fluency	2 to 3	2015	16.9	11.2	5.7(1.18)	−4.80***	0.37
Familiar word fluency	2 to 3	2014	9.7	5.0	4.7(0.98)	−4.79***	0.38	0.32
Oral reading fluency	1 to 2	2015	5.0	3.4	1.6(0.68)	−2.39*	0.19
Oral reading fluency	1 to 2	2014	4.1	1.2	2.9(0.63)	−4.57***	0.36
Oral reading fluency	2 to 3	2015	17.6	12.2	5.4(1.3)	−4.18***	0.33
Oral reading fluency	2 to 3	2014	9.6	5.6	4.0(1.09)	−3.70***	0.29	0.29

Our results show statistically significant grade-transition loss for all four measures at each of the four transitions. For example, children transitioning from grade 2 to 3 in 2015 lost 12.7 letters per minute (p-value < .001), 10.2 syllables per minute (p-value < .001), 5.7 familiar words per minute (p-value < .001), and 5.4 words per minute on connected text (p-value < .001) in the 2015 grade transition. The absolute magnitude of the grade-transition loss was somewhat smaller for the grade 1 to 2 transition than for the grade 2 to 3 transition in many of the analyses; however, the relative loss was greater for the grade 1 to 2 transition than for the grade 2 to 3 transition. In any case, the results show statistically significant evidence for grade-transition loss in all 16 cases – that is, for both the grade 1 and 2 transition, in both 2014 and 2015, and for letter naming fluency, syllable fluency, familiar word fluency and oral reading fluency in connected text.

The magnitude of the grade-transition loss is presented in Table 3’s effect size column. Loss effect sizes are substantial in some of the comparisons. The results showed an average effect size of 0.58 SD for letter naming fluency, 0.35 SD for syllable fluency, 0.32 SD for familiar word fluency and 0.29 SD for oral reading fluency. Our results showed an average effect size of 0.38 SD for both the grade 1 to 2 and the grade 2 to 3 transition periods.

To better explain the nature of grade-transition loss, we present Figure 1. Figure 1 shows the mean rates of oral reading fluency of connected text for each of the three cohorts of children in each of the four LAT rounds per academic year (October, January, March and July). While the differences in mean oral reading fluency varied between LAT rounds, Figure 1 shows that all of the other, non-grade-transition breaks corresponded with positive changes in performance. Only at the grade-transition break, between July and October, was there a decline in learning outcomes. The figure also presents key information to understand grade-transition loss: the cwpm in July (before the break) and October (after the break), when the transition between the grades occurred. Asterisks show when each of these four comparisons was statistically significant. The figure also shows that modest but positive growth occurred for each cohort between each LAT round, with only the grade-transition comparison presenting as negative. Note that 27 of the 28 LAT round comparisons were statistically significant and positive, and only one comparison was not statistically significant.⁴

Figure 1.

Chichewa oral reading fluency, by student cohort and assessment time.

RQ2: Did the magnitude of grade-transition loss differ by gender?

Given that the literature on summer loss in North American and European contexts was not in agreement on whether summer loss results differed by gender, we present the results of our DiD models that determined whether the declines between the July and October time periods were larger for boys or girls.

Table 4 shows the magnitude of grade-transition loss for both boys and girls and the DiD comparison, with associated p-values and effect sizes. The table presents the same 16 comparisons as discussed above, with four comparisons before and after the grade-transition break for each of the four measures (letter naming fluency, syllable fluency, familiar word fluency and oral reading fluency on connected text). None of the 16 comparisons showed statistically significant differences between boys and girls. In fact, the effect size of the comparisons was small, with a statistically insignificant 0.17 SD identified for a 2.3 letters per minute difference (p-value .29), and 0.12 SD for a difference of 1.9 syllables per minute (p-value .46). All of the comparisons for familiar word and oral reading fluency were both statistically and substantively insignificant. This suggests no difference by gender in the magnitude of grade-transition loss.

Table 4.

Grade-transition loss DiD estimates by gender. Standard errors in parentheses.

LAT task	Grade transition	Year	Boys’ grade-transition loss	Girls’ grade-transition loss	DiD	p-value	T score	DiD effect size
Letter fluency	1 to 2	2015	10.9 (1.9)	11.6(1.9)	0.7(2.7)	0.81	−0.24	0.04
Letter fluency	1 to 2	2014	10.3 (1.6)	7.9(1.5)	2.3(2.2)	0.29	1.07	0.17
Letter fluency	2 to 3	2015	12.4 (2.5)	13.0(2.6)	0.6(3.7)	0.87	−0.16	0.02
Letter fluency	2 to 3	2014	9.3(2.1)	8.0(2.2)	1.3 (3.1)	0.67	0.43	0.07
Syllable fluency	1 to 2	2015	6.1(1.7)	4.2(2.0)	1.9 (2.0)	0.46	0.74	0.12
Syllable fluency	1 to 2	2014	3.6(1.1)	3.2(0.9)	0.4 (1.4)	0.78	0.28	0.04
Syllable fluency	2 to 3	2015	10.3 (2.6)	10.0(2.8)	0.4 (3.9)	0.92	0.10	0.01
Syllable fluency	2 to 3	2014	5.1(2.0)	4.9(2.0)	0.2 (2.8)	0.93	0.09	0.01
Familiar word fluency	1 to 2	2015	2.0(0.9)	1.5(1.1)	0.5 (1.4)	0.74	0.33	0.05
Familiar word fluency	1 to 2	2014	2.6(0.9)	2.3(0.7)	0.3 (1.1)	0.81	0.24	0.04
Familiar word fluency	2 to 3	2015	5.8(1.6)	5.6(1.7)	0.3 (2.4)	0.92	0.11	0.02
Familiar word fluency	2 to 3	2014	5.3(1.5)	4.1(1.3)	1.2 (1.9)	0.54	0.62	0.10
Oral reading fluency	1 to 2	2015	1.4(0.9)	1.9(1.0)	0.5 (1.4)	0.71	−0.38	0.04
Oral reading fluency	1 to 2	2014	3.2(1.0)	2.5(0.8)	0.7 (1.3)	0.57	0.57	0.09
Oral reading fluency	2 to 3	2015	5.8(1.8)	5.1(1.9)	0.7 (2.6)	0.79	0.27	0.04
Oral reading fluency	2 to 3	2014	4.1(1.7)	3.9(1.4)	0.2 (2.2)	0.93	0.03	0.01

RQ3: Did the magnitude of grade-transition loss differ by assignment to a generic literacy treatment group?

While the North American and European literature was largely silent as to whether summer loss would differ for children who were involved in an instructional improvement program similar to the ones implemented in Malawi and in several other sub-Saharan African countries, we fit models that determined whether there was a difference in grade-transition loss for children in a literacy intervention treatment program, implemented during the school year. Our DiD model compared the loss for children whose teachers were receiving the Early Grade Reading Activity treatment with another group of children whose teachers were not receiving the treatment. Our results showed no differences (p-value .98). In other words, whether one’s teachers were in a research-based instructional improvement program that provided books, training and support had no relationship with the magnitude of grade-transition loss.

Discussion

Our first research question addressed whether a grade-transition loss phenomenon analogous to North American ‘summer loss’ applied to Malawi. We found grade-transition loss in Malawi to be substantial, with an average effect size of 0.38 SD. The rich LAT data set allowed us to determine whether this identified loss was specific to the gap between grades, or was within the range of variation between two different LAT rounds. We found that all of the other (non-grade-transition) comparisons between LAT rounds showed either statistically significant improvements (27 transitions) or non-significant changes (one transition). The only negative comparisons were the four grade-transition comparisons that we presented in Figure 1 above.

Further, the magnitude of grade-transition loss in Malawi appears to have been substantial. Several countries in sub-Saharan Africa have undertaken literacy improvement programs, which have had varied effects. A recent article from New Directions for Child and Adolescent Developmentpresented the effect sizes of programs in Egypt, Guatemala, Jordan, Kenya, Liberia, Mozambique, Philippines, Rwanda and Senegal (Moore et al., 2017). The effect sizes for oral reading fluency ranged from 0.24 to 0.80. The average effect size identified in our analysis was a loss equivalent to 0.38 SD for both the grade 1 to 2 and the grade 2 to 3 transitions. It appears that grade-transition loss – a relatively ignored phenomenon in the recent increased investments in literacy – has as substantial a relationship to learning outcomes as do well-designed, well-implemented, well-funded and targeted interventions in literacy.

Our second research question asked whether the magnitude of grade-transition loss differed by gender. Studies have shown substantial gender differences in literacy outcomes, typically benefiting girls (Gove and Cvelich, 2011; Willingham and Cole, 1997). Recent investments have focused on girls’ education in general and literacy for girls specifically, such as the UK Department for International Development’s (DFID’s) Girls’ Education Challenge and USAID’s Let Girls Learn. Our analysis used a DiD estimate to determine whether there was an interaction between gender and grade-transition loss. We found that none of the 16 comparisons for grade-transition loss differed by gender. This means that girls’ results dropped by as much as boys’ did. This finding might seem somewhat surprising given previous research showing assigned tasks and duties differing by gender (Immajati, 2016; Jha and Pouezevara, 2016), and in ways that one could hypothesize would allow boys more time to practise the educational skills that otherwise would atrophy during the summer. However, features of the Malawian economic, cultural and social context may explain that finding, as indicated next.

The majority of Malawians live in rural areas and engage in subsistence farming (World Bank, 2010). As a result, typical Malawian families rely heavily upon their children’s labour for assistance with farming and for remuneration through ganyu (casual wage labour) arrangements. Younger children are more likely to be allocated unpaid work in the household, freeing up adults and older children to pursue income-generating activities outside the home (Colclough et al., 2000; Sunny et al., 2017; Westberg, 2010).

Discussions of how child labour supports Malawian households often draw connections to educational outcomes. For instance, Taniguchi (2015) found that the greater the burden of household chores, the greater the likelihood of grade repetition or dropout; however, this was significant for children in grade 7, but not in grade 5. Furthermore, it is not necessarily clear whether the relationship between household labour and schooling holds for children in grades 1–3. Among school heads queried for Malawi’s 2007 education management information system, 44% asserted that ‘family responsibilities’ were a main reason for boys dropping out of school, with 41% reporting it as a main reason for girls dropping out. However, when the pupils themselves were queried, only 1.3% to 1.8% of grade 1–3 respondents gave ‘had to work or help at home’ as a main reason for dropping out of school (World Bank, 2010).

Researchers have found gendered elements as to how children spend their out-of-school time. They established that girls were assigned the majority of household chores, whether one or both parents had died or under normal circumstances (Chimombo et al., 2000; Kadzamira and Rose, 2003; Munthali, 2002). Boys, on the other hand, were more likely to engage in ganyu (Kadzamira and Rose, 2003). In either case, a clear finding was that Malawian children were committing a very significant portion of their waking hours to supporting their households, and as such could be expected to have limited opportunities to engage in out-of-school academic activities. As Kadzamira and Rose (2003: 507–508) asserted:

Evidence also suggests that, although children in school spend approximately two to three hours per day working for the household, on average, children out of school spend an additional four and a half hours per day than a child in school working either for the household or in income-generating activities in both rural and urban areas. This suggests that children out of school provide an important contribution to the household which poorest households are unable to sacrifice even in the context of FPE [free primary education]. Furthermore, girls, both in and out of school, spend approximately one hour per day more than boys working for the household. (Rose, 2002)

We did not have data to examine whether children had particular resilience strategies that they could employ to reduce the grade-transition loss that one might expect given the extra household duties assigned to girls (Kadzamira and Rose, 2003) or the extra income-generating activities assigned to boys. Further research would also be necessary to determine whether the daily extra hour of household-supporting labour provided by girls results in gender-based differences in how much out-of-school time children devote to academic pursuits. In any case, given the magnitude of the general grade-transition loss, effective girl-focused educational interventions should not ignore the phenomenon.

Contextual data collected alongside a nationally representative National Reading Assessment (NRA) commissioned by USAID and the MoEST in June–July 2014 showed that a higher percentage of boys took books home with them, although the overall percentage was low (Nagarajan et al., 2015).⁵ This finding matters because regression results from those same data showed that taking books home was associated with higher literacy outcomes, except for grade 1 boys (Nagarajan et al., 2015). However, the NRA was intended to give ‘an overview of education across Malawi rather than just of education in the districts affected by the USAID interventions’ (Nagarajan et al., 2015: vii). In other words, all the assessments described in this article (LATs and NRA) took place at school, at times when school was in session; and only a portion of them were administered within the context of a literacy intervention. Thus, the causal link between learning outcomes and books available at home might differ in the school year and in the break between grades.

Our final research question examined whether literacy interventions reduced grade-transition loss. The Early Grade Reading Activity intervention that we evaluated focused on improving the literacy environment, with additional literacy materials given to pupils, training on improved literacy methods provided to teachers, and support given to teachers on utilizing those methods and materials. The Activity intervention did not have a grade-transition loss-reduction component, but we hypothesized that the increased interest in literacy and more frequent access to materials within the program might either create greater resistance to grade-transition loss, or change behaviours in the break between grades to allow higher access to materials and training. Data from a community mobilization study conducted internally in 2015 for the Activity found that 83% of parents surveyed while school was in session said that their children brought books home in the previous week; however, more work may need to be done to increase children’s access to materials outside of normal academic interventions.

Our results show that the magnitude of grade-transition loss was the same regardless of whether a child’s teacher was receiving a literacy intervention. If one assumes that the Malawi Early Grade Reading Activity intervention was somewhat similar to other large-scale interventions currently in place across sub-Saharan Africa (Moore et al., 2017), it begs the question whether children enjoying these programs would similarly have no resiliency against loss. More research is necessary to understand this, but at least in Malawi we note that as designed, this literacy intervention was unable to reduce grade-transition loss.

Given that the literacy intervention in Malawi had no impact on grade-transition loss, whereas several US-based interventions specifically targeting summer loss have been effective, it might be possible to conceive a grade-transition loss-reduction intervention for Malawi. More research is necessary to determine whether the design of the grade-transition loss-reduction interventions for a Malawi-type context would be similar to or different from the US-based ones, and whether school-based or community-based grade-transition loss-reduction interventions would be more effective.

Limitations

This section presents the threats to validity for our analyses, and our attempts to address them. One possible threat to validity relates to the potential for students to be assessed more than once, given the random sampling at the school level during each LAT round. This means it is possible that individual students were assessed more than once, and that a learning effect could have occurred. A student tested during the LAT data collection immediately before the break between grades (July 2014 or July 2015) who had been tested at least once before might exhibit stronger performance because of increased familiarity with the assessment, particularly because the same assessment was used for every round within an academic year. If many children were assessed repeatedly, and a learning effect occurred, this in turn would bias estimates of the magnitude of learning loss between that round and the first round of the subsequent academic year.

To address this threat to validity, we used a binomial probability analysis that considered the schools’ enrolment by grade and gender, and the number of students sampled each round, to estimate the likelihood that a student assessed during the July round was seeing the test for the first time. For the 2013–2014 school year, the average probability across schools was 88%; for the 2014–2015 school year, the average probability across schools was 75%.⁶ Put another way, less than an eighth of the assessed students in 2014, or less than a quarter in 2015, were likely to have been assessed in previous rounds. Given that literacy assessment results from the same children were higher by 20% at the second round of data collection, and operating under the assumption that approximately 25% of children were assessed more than once, our estimates might have overestimated the size of summer loss by up to 1 cwpm (for an analysis of similar repeated test-takers in Nepal, see RTI International, 2016b).

A second limitation relates to the timing of assessment. Our research questions focused on estimating the magnitude of grade-transition loss; ideally, data collection would have been done at the very end of the instructional time in July and at the start of the academic year in October. The July data collections occurred within a couple of weeks of the end of the term, and in Malawi, much of that time is spent on examinations, so little learning occurs after the assessment. However, there is a larger time difference at the beginning of the academic year. The school years began on September 8 in 2014 and September 7 in 2015, while LAT data collection began on October 6 in 2014 and October 27 in 2015. Given that data collection extended over two weeks in 2014 and one week in 2015, this means that students were assessed between four and eight weeks after the term began. This suggests that the magnitude of the grade-transition loss might be somewhat underestimated, since students would have learned (or relearned) material in that first month or two of school.

A third limitation relates to grade repetition, where children fail to advance to the next grade. A substantial number of children in grades 1–3 in Malawi repeat grades. This inefficiency is well attested in the Malawian education literature (Hewett et al., 2009; Khasu et al., 2014; Sabates et al., 2010). Our analyses were unable to account for the influence of this repetition on the magnitude of grade-transition loss, since we did not follow an actual cohort of children from grade to grade. We are also unable to know whether being unable to identify repeaters biased the results negatively or positively, although each sample was likely to have included some pupils who had previously repeated.

Finally, a fourth threat to validity relates to whether the inclusion or exclusion of non-readers (children who read zero words per minute on the oral reading fluency task) influenced grade-transition loss. Because the July (pre-break) and October (post-break) assessments were administered to different children, the ‘loss’ cannot be assumed to have been driven entirely by a decrease in mean scores. An alternative explanation might be that the loss was due to imbalances in the proportion of readers and non-readers. The section below considers some possible implications of this threat.

Table 5 presents one approach to understanding how grade-transition loss might have been affected by the proportion of non-readers. The entire sample is presented first. This shows that, for all four grade-transition comparisons, there were more non-readers in the post-break assessment than in the pre-break assessment. These differences ranged from 8 percentage points more in the grade 2 to 3 transition for Cohort 1 in the 2015 break (30% to 38%) to 31 percentage points more in the grade 2 to 3 transition for Cohort 3 in the 2014 break (55% to 86%).

Table 5.

Grade-transition changes in fluency scores and proportion of non-readers for the entire LAT sample, for a sample restricted to readers at 1 cwpm or more, and for a hypothetical sample of readers in the pre-break assessment.

		Entire sample			Readers only			Hypothetical sample
		Jul	Oct	Grade-transition change	Jul	Oct	Grade-transition change	Jul	Oct	Grade-transition change
Break 2014(Grade 1→2)	cwpm	4.11	1.15	−2.97	15.1	8.4	−6.7	15.10	4.39	−10.71
	% zeros	73%	86%	13%	0	0		0	48%
	N	323	337	14	88	46		88	46 + (42 zeroes)
Break 2014(Grade 2→3)	Cwpm	9.62	5.26	−4.36	21.3	15.1	−6.2	21.30	12.40	−8.90
	% zeros	55%	86%	31%	0	0		0	18%
	N	310	330	20	140	115		140	115 + (25 zeroes)
Break 2015(Grade 1→2)	Cwpm	4.75	3.46	−1.29	11.2	14.1	2.9	11.20	7.81	−3.39
	% zeros	57%	75%	18%	0	0		0	45%
	N	328	314	−14	139	77		139	77 + (62 zeroes)
Break 2015(Grade 2→3)	Cwpm	16.66	12.16	−4.50	23.9	19.6	−4.3	23.90	17.71	−6.19
	% zeros	30%	38%	8%	0	0		0	10%
	N	327	332	5	228	206		228	206 + (22 zeroes)

To illustrate how this significant number of non-readers might have affected the magnitude of grade-transition loss, the ‘readers only’ columns of Table 5 present the fluency rates for the subpopulation of the July and October samples that read at least one word correctly (i.e., had an orf >= 1). The ‘readers only’ calculations indicate that the number of non-zero readers was lower in the post-break assessment for all four comparisons. It also shows that fluency rates for those non-zero readers were lower after the break for three of the four comparisons. (The fourth comparison group—the Cohort 2 children transitioning from grade 1 to 2—recorded an orf 2.9 correct words per minute higher in the post-break period than the pre-break period.)

The problem with an approach that ignores children who recorded a fluency rate of zero is that it ignores the possibility that children who were reading before the break had forgotten how to read after the break. The final columns in Table 5 (‘hypothetical sample’) illustrate what grade-transition loss might be if the sample sizes before and after each break were the same, assigning an orf of 0 to each hypothetical child added to balance the samples.

Figure 2 provides a graphic comparison of these calculations. First, our interpretation of these data is that the high number of non-readers in Malawi somewhat reduces apparent grade-transition loss, since there is nothing lower than zero. The ‘hypothetical sample’ calculations illustrate a case where grade-transition loss could range from 6.2 cwpm (for the grade 2 to 3 transition in 2015) to 10.7 cwpm (for the grade 1 to 2 transition in 2014). The inability to detect grade-transition loss due to non-readers indicates that the analysis of the full sample presented in this article might underestimate grade-transition loss by 38% or more (derived from the difference between the loss observed in the entire sample and that stipulated in the hypothetical sample population).

Figure 2.

Grade-transition loss in the entire sample, the readers-only subsample and the hypothetical sample.

Conclusion

This study shows that grade-transition loss exists in Malawi and suggests that policy makers interested in improving learning outcomes should actively address this phenomenon. In Malawi, the grade-transition-loss effect was impervious to an intervention that did not directly target it; however, the US-based research shows that out-of-school interventions can improve learning outcomes. Unfortunately, there is little rigorous evidence as to whether reading camps or literacy interventions that organizations like Pratham and Save the Children do during term breaks would be more or less effective than the targeted, home-based summer-loss interventions that have been shown to be successful in lower-income contexts in the US.

In Malawi, interventions that take a community-focused approach to strengthening learning outcomes have yet to produce rigorous, actionable evidence of program effects at scale. Save the Children has been implementing various small-scale iterations of the Literacy Boost intervention throughout southern Malawi since 2009.⁷ In the most recent version documented to date, the Tiwerenge Ndi Ana Athu (TiANA) program provided community-produced reading materials, used high school graduates as volunteers to run reading camps, and broadcast recordings developed at the reading camps over a radio station serving the area. Impact was modest, however, and the endline report did not present the program’s learning gains net of the within-year learning effects of the school-based teaching, so it is difficult to gauge its success (Save the Children International, 2014).

One add-on element of the Malawi Early Grade Reading Activity was a small-scale study (31 schools) of a roughly six-month social and behaviour-change communication (SBCC) pilot intervention in two zones of Ntcheu district. The multichannel approach used a radio program, radio spots, listening groups, posters hung in public areas, community meetings and community theatre performances to inform parents about how they could support their children’s reading acquisition. While results of the evaluation were positive, they were not linked to children’s learning outcomes (Schmidt et al., 2016).

A lamentably under-documented and underreported aspect of the pilot was a book-lending program conducted during the grade-transition break. Head teachers made arrangements for school staff to be available at the school to lend books to children from the schools’ Reading Tools in a Box⁸ mini libraries. Participating children took the books home, read them (sometimes with their parents) and then exchanged them for others. Unfortunately, rigorous data on the frequency and duration of the book borrowing were not collected (Dr. Elizabeth Randolph, personal communication, 12 March 2017). In communities whose schools have been provided with large quantities of books by donor-funded interventions, such borrowing programs would have promise and merit further investigation.

A major limiting factor to any intervention aimed at children when school is not in session is the economic reality of the average Malawian family. As quoted above from Kadzamira and Rose (2003), ‘children out of school provide an important contribution to the household which poorest households are unable to sacrifice even in the context of FPE’ (p. 507). When poor rural families rely heavily upon even young children for farm-based labour and income-generation activities – for up to seven hours a day, when school is not in session – significant sensitization efforts would likely be needed to convince parents that the potential long-term benefit of any educational activity would outweigh the short-term opportunity cost in lost wages or harvest productivity (Kadzamira and Rose, 2003).

Large-scale literacy interventions, spurred on by increased evidence of low learning achievement, have been burgeoning across the developed world. The designs of those interventions have focused on classroom-based, within-school interventions. The magnitude of grade-transition loss in Malawi should give pause to those supervising, designing and implementing these large-scale interventions, and prompt them to consider including investments to reduce grade-transition loss. An examination of the within-year rate of reading acquisition suggests that, on average, between one and two months of instruction may be required to regain pre-break levels of performance. To combat these steps backward, we recommend that targeted interventions be undertaken that are both low cost and potentially scalable, to determine whether school-based, home-based or community-based interventions are more promising in these contexts. Policy makers should pay careful attention to those results so that the improvements identified during the school year will not be lost just as quickly during the break between grades.

Footnotes

Acknowledgements

The authors wish to thank the United States Agency for International Development (USAID) for funding and direction in the implementation of the Early Grade Reading Activity, and for approving the release of these data; Malawi’s Ministry of Education, Science, and Technology (MoEST) and the staff of the Early Grade Reading Activity for their support in the implementation of the Activity; and RTI International (especially Erin Newton) for analytical, writing and editorial support.

Declaration of Conflicting Interests

One co-author served as the Chief of Party for the Malawi Early Grade Reading Activity and another co-author served as the Deputy Chief of Party for the same Activity. The authors declare no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: These analyses are based on data generated through and collected by the Malawi Early Grade Reading Activity, USAID Contract AID-612-C-13-00002.

Notes

Author biography

Timothy S Slade is the Deputy Chief of Party for the Kenya Tusome Early Grade Reading Activity implemented by RTI International, having previously served as DCOP for the Malawi Early Grade Reading Activity. Since joining RTI his work has focused on the design, delivery, and evaluation of reading interventions in various African contexts. He graduated from Alma College with a B.S. in Exercise & Health Sciences and a B.A. in French Language, and earned a Master’s degree in International Studies from North Carolina State University.

Dr. Benjamin Piper is the Senior Director, Africa Education for RTI International based in Nairobi. Dr. Piper provides technical support to RTI’s work across Sub-Saharan Africa and supervises Tusome, the national literacy program of Kenya, funded by USAID and DFID, and the Tayari early childhood development (ECD) program, sponsored by CIFF. Tusome is being implemented in all public primary schools across Kenya, and Dr. Piper served as the Chief of Party from Tusome’s inception in 2014 to 2016. Dr. Piper led the PRIMR Initiative, the National Tablet Programme, and the PRIMR Rural Expansion Programme from 2011-2015. PRIMR tested low-cost, scalable approaches to improving literacy and numeracy. His dissertation research investigated the impact of Ethiopian in-service programs on teacher and student outcomes. He has worked with RTI, World Bank, DFID, UNICEF, and Save the Children.

Dr. Zikani Kaunda was the Chief of Party for the Malawi Early Grade Reading Activity implemented by RTI International. Dr. Kaunda’s education-sector expertise lies in leading programs focused on early grade reading, girls’ education, orphans and other vulnerable children (OVCs), expansion of education access, and improvement of education quality. Outside the field of education, he has expertise in HIV/AIDs prevention and mitigation; promotion of gender equality; elimination of gender-based violence; elimination of child labor; agriculture; and community-based natural resource management. He earned a Bachelor’s degree in agriculture from the University of Malawi, a Master’s degree in Public Development and Management from Witwatersrand University, and a Doctor of Philosophy degree in Comparative International Development and Education at the University of Minnesota.

Simon King is a Research Statistician with interests in complex survey methodology and analysis. Mr. King has been lead statistician on many early grade reading projects, conducted capacity-building activities and led four technical workshops on sampling, data management, weighting, and analysis. Prior to working for RTI, Mr. King was involved in K–12 education in the UK, Switzerland, USA, Zambia and Channel Islands, notably as a principal of a charter school and as a volunteer teacher working for DFID in rural Zambia. He received his B.S. in Mathematics from Royal Holloway College, University of London, and his M.S. in Statistics from Texas A&M University.

Hibatalla Ibrahim is an Ed.M. candidate in International Education Policy at Harvard Graduate School of Education. She is interested in literacy, mother-tongue instruction, gender and education in conflict settings with particular emphasis on disadvantaged populations in the Middle East and Africa. She obtained her undergraduate degree from Monash University.

References

Alexander

Entwisle

Olson

(2001) Schools, achievement, and inequality: A seasonal perspective. Educational Evaluation and Policy Analysis 23(2): 171–191.

Alexander

Entwisle

Olson

(2007) Lasting consequences of the summer learning gap. American Sociological Review 72(2): 167–180.

Allington

McGill-Franzen

Camilli

Williams

Graff

Zeig

Zmach

Nowak

(2010). Addressing summer reading setback among economically disadvantaged elementary students. Reading Psychology 31(5): 411–427.

Bakle

(2010) Summer learning loss: The influence of summer school programs on student achievement in language usage, math, and reading. PhD dissertation, Indiana State University, USA. ERIC Number: ED517121.

Card

Krueger

(1992) Does school quality matter? Returns to education and the characteristics of public schools in the United States. Journal of Political Economy 100(1): 1–40.

Chimombo

Chibwanna

Dzimadzi

Kadzamira

Kunkwenzu

Kunje

Namphota

(2000) Classroom, school and home factors that negatively affect girls education in Malawi. Report prepared for UNICEF. Lilongwe: Centre for Educational Research and Training (CERT), University of Malawi. Available at: www.unicef.org/spanish/evaldatabase/files/MLW_2000_005.pdf (accessed 25 February 2017).

Chin

Phillips

(2004) school reproduction and child-rearing practices: Social class, children’s agency, and the summer activity gap. Sociology of Education 77(3): 185–210.

Cohen

(1988) Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

Colclough

Rose

Tembon

(2000) Gender inequalities in primary schooling: The roles of poverty and adverse cultural practice. IDS Working Paper No. 78. Brighton, UK: Institute of Development Studies, University of Sussex. Available at: http://pt.ircwash.org/sites/default/files/Colclough-2000-Gender.pdf (accessed 12 March 2017).

10.

Cooper

Charlton

Valentine

Muhlenbruck

(2000) Making The Most Of Summer School: A Meta-Analytic And Narrative Review. Monographs of the Society for Research in Child Development 65(1): i-vi+1–127.

11.

Cooper

Nye

Charlton

Lindsay

Greathouse

(1996) The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research 66(3): 227–268.

12.

Cooper

Valentine

Charlton

Melson

(2003) The effects of modified school calendars on student achievement and school community attitudes: A research synthesis. Review of Educational Research 73(1): 1–52.

13.

Coulombe

Tremblay

J-F

(2006) Literacy and growth. Topics in Macroeconomics 6(2): 1–34.

14.

Davies

Aurini

Milne

(2016) Contextualizing summer learning inequality: The view from Canada. In: Alexander

Pitcock

Boulay

(eds) The Summer Slide: What We Know and Can Do About Summer Learning Loss. New York: Teachers College Press.

15.

Dowd

Mabeti

(2011) Literacy Boost Malawi: Year 2 report. Washington, DC: Save the Children. Available at: https://globalreadingnetwork.net/eddata/literacy-boost-malawi-year-2-report (accessed 5 March 2017).

16.

Dowd

Wiener

Mabeti

(2010) Malawi Literacy Boost 2009: Year 1 report. Washington, DC: Save the Children. Available at: https://globalreadingnetwork.net/eddata/literacy-boost-malawi-2009-year-1-report (accessed 5 March 2017).

17.

Downey

Von Hippel

Broh

(2004) Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review 69(5): 613–635.

18.

Education Policy and Data Center (EPDC) (2014) Malawi: National Education Profile. Washington, DC: FHI 360. Available at: www.epdc.org/sites/default/files/documents/EPDC%20NEP_Malawi.pdf (accessed 1 November 2017).

19.

Fleisch

Schöer

(2014) Large-scale instructional reform in the global South: Insights from the mid-point evaluation of the Gauteng primary language and mathematics strategy. South African Journal of Education 34(3): 1–12.

20.

Friedlander

Dowd

Borisova

Guajardo

(2012) Life-Wide Learning: Supporting All Children to Enjoy Quality Education. Prepared for the United Nations Global Thematic Consultation ‘Addressing Inequalities: The Heart of the Post-2015 Development Agenda and the Future We Want for All’. Washington, DC: Save the Children. Available at: https://static1.squarespace.com/static/57ffc29c414fb543385340da/t/580b998d1b631bb287e9333a/1477155217838/Friedlander+Dowd+etal_2012_Life-wide+learning-+Supporting+all+children+to+enjoy+quality+education+-+Friedlander.pdf (accessed 4 November 2017).

21.

Friedlander

Guajardo

Williams Vasulu

Joaki

Chasanga

Polepole

(2012) Malawi Literacy Boost: Baseline report. Washington, DC: Save the Children and World Vision. Available at: https://resourcecentre.savethechildren.net/sites/default/files/documents/literacy_boost_world_vision_malawi_baseline_report-_september_2012.pdf (accessed 12 March 2017).

22.

Gershenson

(2013) Do summer time-use gaps vary by socioeconomic status? American Educational Research Journal 50(6): 1219–1248.

23.

Glewwe

Muralidharan

(2016). Chapter 10. Improving education outcomes in developing countries: Evidence, knowledge gaps, and policy implications. In: Hanushek

Machin

Woessmann

(eds) Handbook of the Economics of Education, Vol. 5. Amsterdam: North Holland, pp.543–743.

24.

Global Partnership for Education (2016) Education Sector Plan 2008–2017: Malawi. A Statement. Lilongwe, Malawi: Ministry of Education, Science and Technology. Available at: www.globalpartnership.org/content/malawi-education-sector-plan (accessed 1 November 2017).

25.

Gove

Cvelich

(2011) Early Reading: Igniting Education for All. A Report by the Early Learning Community of Practice. Revised Edition. Research Triangle Park, NC: RTI International. Available at: http://www.rti.org/pubs/early-reading-report-revised.pdf (accessed 1 November 2017).

26.

Guryan

Kim

Quinn

(2014) Does Reading During the Summer Build Reading Skills? Evidence from a Randomized Experiment in 463 Classrooms. Working Paper 20689. Cambridge, MA: National Bureau of Economic Research. Available at: www.nber.org/papers/w20689 (accessed 1 November 2017).

27.

Hanushek

Woessmann

(2015) The economic impact of educational quality. In: Dixon

Humble

Counihan

(eds) Handbook of International Development and Education. Cheltenham, UK: Edward Elgar Publishing, pp.6–19.

28.

Hewett

Mensch

Lloyd

Chimombo

Rankin

(2009) Assessing the Impact of Primary School Quality on Adolescent Educational Outcomes in Rural Malawi. Presentation given at 24th International Population Conference of the International Union for the Scientific Study of Population, Marrakech, Morocco, 27 September–2 October. Available at: www.popcouncil.org/pdfs/events/2009IUSSP_Hewett.pdf (accessed 1 November 2017).

29.

Heyns

(1978) Summer Learning and the Effects of Schooling. New York: Academic.

30.

Immajati

(2016) Follow-On Gender Study from USAID PRIORITAS Mid-Term Evaluation. Prepared for USAID under the Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) project, Contract No. AID-497-C-12–00003. Research Triangle Park, NC: RTI International. Available at: http://pdf.usaid.gov/pdf_docs/PA00MG4H.pdf (accessed 1 November 2017).

31.

Jha

Pouezevara

(2016) Measurement and Research Support to Education Strategy Goal 1: Boys’ underachievement in education: A review of the literature with a focus on reading in the early years. Prepared for USAID under the Education Data for Decision Making (EdData II) project, Task Order No. AID-OAA-12-BC-00003 (RTI Task 20, Activity 13). Research Triangle Park, NC: RTI International. Available at: http://pdf.usaid.gov/pdf_docs/PBAAF451.pdf.

32.

Kadzamira

Rose

(2003) Can free primary education meet the needs of the poor? Evidence from Malawi. International Journal of Educational Development 23: 501–516.

33.

Kananji

(2014) School Examinations and National Assessment in Malawi. Presentation given at UNESCO Institute for Statistics Regional Workshop on Measurement of Learning Outcomes, Dakar, Senegal. Available at: www.uis.unesco.org/StatisticalCapacityBuilding/Workshop%20Documents/Education%20workshop%20dox/Dakar%202014/05_Malawi_EN.pdf.

34.

Kerry

Davies

(1998) Summer learning loss: The evidence and a possible solution. Support for Learning 13(3): 118–122.

35.

Khasu

Mapondera

Shawa

Chiyani

(2014) Report for Study on Student Repetition and Attrition in Primary Education in Malawi. Prepared by D R Consulting Services (DRCS) for USAID/Malawi. Available at: http://pdf.usaid.gov/pdf_docs/PA00K2HF.pdf (accessed 1 November 2017).

36.

Kim

(2004) Summer reading and the ethnic achievement gap. Journal of Education for Students Placed at Risk 9(2): 169–188.

37.

Kim

Quinn

(2013) The effects of summer reading on low-income children’s literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions. Review of Educational Research 83(3): 386–431.

38.

Kim

White

(2008) Scaffolding voluntary summer reading for children in grades 3 to 5: An experimental study. Scientific Studies of Reading 12(1): 1–23.

39.

Kim

Guryan

White

Quinn

Capotosto

Kingston

(2016) Delayed effects of a low-cost and large-scale summer reading intervention on elementary school children’s reading comprehension. Journal of Research on Educational Effectiveness 9(suppl 1): 1–22.

40.

Kolen

Brennan

(2004) Test Equating, Linking, and Scaling: Methods and Practices (2nd ed.). New York: Springer-Verlag.

41.

Lareau

(2011) Unequal Childhoods: Class, Race, and Family Life. Berkeley, CA: University of California Press.

42.

Lindahl

(2001) Summer Learning and the Effect of Schooling: Evidence from Sweden. IZA Discussion Paper Series, No. 262. Bonn, Germany: Forschungsinstitut zur Zukunft der Arbeit [Institute for the Study of Labor]. Available at: http://ftp.iza.org/dp262.pdf (accessed 1 November 2017).

43.

Logan

Johnston

(2009) Gender differences in reading ability and attitudes: Examining where these differences lie. Journal of Research in Reading 32(2): 199–214.

44.

McCombs

Augustine

Schwartz

Bodilly

McInnis

Lichter

Cross

(2011) Making Summer Count: How Summer Programs Can Boost Children’s Learning. Santa Monica, CA: RAND Corporation. Available at: www.rand.org/content/dam/rand/pubs/monographs/2011/RAND_MG1120.pdf (accessed 1 November 2017).

45.

McCombs

Pane

Augustine

Schwartz

Martorell

Zakaras

(2014) Ready for Fall? Near-Term Effects of Voluntary Summer Learning Programs on Low-Income Students’ Learning Opportunities and Outcomes. Santa Monica, CA: RAND Corporation. Available at: www.rand.org/pubs/research_reports/RR815.html (accessed 1 November 2017).

46.

Meyer

(2014) The Summer Learning Effect in Germany: Home and School Contributions to Summer Learning Patterns in the Communities of Two Primary Schools. Unpublished doctoral dissertation, University of Auckland, New Zealand.

47.

Ministry of Education (MOE) [Ethiopia] (2013) Mother Tongue Syllabus, Grades 1–4. Addis Ababa: Ministry of Education.

48.

Ministry of Education, Science and Technology (MoEST) [Kenya] (2015) National Education Sector Plan (NESP). Volume One: Basic Education Programme Rationale and Approach (2013–2018). Nairobi: Ministry of Education, Science and Technology.

49.

Ministry of Education, Science and Technology (MoEST) [Malawi] (2008) National Education Strategic Plan (NESP) 2008–2017. Lilongwe: Ministry of Education, Science and Technology.

50.

Moore

Gove

Tietjen

(2017) Great expectations: A framework for assessing and understanding key factors affecting student learning of foundational reading skills. In: McCardle

Mora

Gove

(eds) Progress Toward a Literate World: Early Reading Interventions in Low-Income Countries. New Directions for Child and Adolescent Development 155: 13–30.

51.

Munthali

(2002) Adaptive Strategies and Coping Mechanisms of Families and Communities Affected by HIV/AIDS in Malawi. Report prepared for the United Nations Research Institute for Social Development’s (UNRISD’s) HIV/AIDS and Development project. Grahamstown, South Africa: Department of Anthropology, Rhodes University. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.511.918&rep=rep1&type=pdf (accessed 1 November 2017).

52.

Murnane

(1975) The Impact of School Resources on the Learning of Inner City Children. Cambridge, MA: Ballinger.

53.

Nagarajan

Carneiro

Chiappetta

Fuller

Gonzales

Sinclair

Mapondera

(2015) Assessment: 2014 National Reading Assessment. Prepared for USAID/Malawi under Contract No. AID-612-C-13-00001. Washington, DC: Social Impact. Available at: http://pdf.usaid.gov/pdf_docs/PA00KHCK.pdf (accessed 1 November 2017).

54.

National Institute for Literacy (NIL) [USA] (2010) Adult Literacy Development and Economic Growth. Washington, DC: NIL. Available at: https://lincs.ed.gov/publications/pdf/AdultLiteracyDevEcoGrowth.pdf (accessed 1 November 2017).

55.

National Statistical Office [Malawi] (2011) Malawi Demographic and Health Survey 2010. Zomba, Malawi: National Statistical Office; and Calverton, MD: ICF Macro. Available at: www.dhsprogram.com/pubs/pdf/FR247/FR247.pdf (accessed 1 November 2017).

56.

O’Brien

(1999) Three essays on early academic achievement of minority and disadvantaged students. Unpublished doctoral dissertation, University of Texas at Dallas.

57.

Phillips

Chin

(2004) How families, children, and teachers contribute to summer learning and loss. In: Borman

Boulay

(eds) Summer Learning: Research, Policies and Programs. Mahwah, NJ: Lawrence Erlbaum Associates, pp.53–69.

58.

Piper

Jepkemei

Kibukho

(2015) Pro-poor PRIMR: Improving early literacy skills for children from low-income families in Kenya. Africa Educational Review 12(1): 67–87.

59.

Piper

Zuilkowski

Mugenda

(2014) Improving reading outcomes in Kenya: First-year effects of the PRIMR Initiative. International Journal of Educational Development 37: 11–21.

60.

Rose

(2002). Cost-Sharing in Malawian Primary Schooling: From the Washington to the Post-Washington Consensus. Unpublished PhD thesis, Institute of Education, University of Sussex, Brighton, UK.

61.

RTI International (2016a) [Malawi] Early Grade Reading Activity, Quarterly Report, Fiscal Year 2016, Period of Performance: April 1, 2016 – June 30, 2016. Prepared for USAID under Contract No. AID-612-C-13-00002. Research Triangle Park, NC: RTI.

62.

RTI International (2016b) USAID’s Early Grade Reading Program (EGRP) in Nepal: Baseline report. Prepared for USAID/Nepal, Task Order No. AID-367-TO-15-00002 of the Assistance to Basic Education–All Children Reading (ABE-ACR) program. Research Triangle Park, NC: RTI.

63.

Sabates

Akyeampong

Westbrook

Hunt

(2010) School Dropout: Patterns, Causes, Changes and Policies. Background paper prepared for the Education for All Global Monitoring Report 2011, The Hidden Crisis: Armed Conflict and Education. Report No. 2011/ED/EFA/MRT/PI/08. Available at: http://unesdoc.unesco.org/images/0019/001907/190771e.pdf (accessed 1 November 2017).

64.

Save the Children International (2014) Basic education program: TiANA final evaluation report for St. Anthony and Namadidi Education Zones, Zomba Rural, Malawi. Prepared for USAID under the ‘Tiwerenge Ndi Ana Athu’ (TiANA) project. Washington, DC: Save the Children.

65.

Schmidt

DeStefano

Cummings

(2016) Malawi social and behavior change communications pilot: Endline report. Prepared for USAID under the Education Data for Decision Making (EdData II) project, Measurement and Research Support to Education Strategy Goal 1, Task Order No. AID-OAA-BC-12-00003 (RTI Task 20, Activity 11). Research Triangle Park, NC: RTI International. Available at: http://pdf.usaid.gov/pdf_docs/PBAAF442.pdf (accessed 12 March 2017).

66.

Stanovich

(1995) Cognitive processes in early reading development: Accommodating individual differences in a model of acquisition. Issues in Education 1(1): 1–57.

67.

Singh

(2016) Learning More with Every Year: School Year Productivity and International Learning Divergence. Job Market Paper. London: Department of Economics, University College. Available at: https://weblearn.ox.ac.uk/access/content/group/34d393d2-bc7a-46b7-8fa9-bf2c2ce8f65e/2016_17/Job%20Market/Job%20Market%20Folders/Singh_Abhijeet_JMP.pdf (accessed 12 March 2017).

68.

Slates

Alexander

Entwisle

Olson

(2012) Counteracting summer slide: Social capital resources within socioeconomically disadvantaged families. Journal of Education for Students Placed at Risk 17(3): 165–185.

69.

Snilstveit

Stevenson

Phillips

Vojtkova

Gallagher

Schmidt

Jobse

Geelen

Grazia Pastorello

Eyers

(2015) Interventions for Improving Learning Outcomes and Access to Education in Low- and Middle-Income Countries: A Systematic Review. 3ie Final Review. London: International Initiative for Impact Evaluation (3ie). Available at: www.3ieimpact.org/media/filer_public/2015/12/31/education-review-finalreport.pdf (accessed 12 March 2017).

70.

Sunny

Elze

Chihana

Gondwe

Crampin

Munkhondya

Kondowe

Glynn

(2017) Failing to progress or progressing to fail? Age-for-grade heterogeneity and grade repetition in primary schools in Karonga district, northern Malawi. International Journal of Educational Development 52: 68–80.

71.

Taniguchi

(2015) Determinants of grade repetition in primary school in sub-Saharan Africa: An event history analysis for rural Malawi. International Journal of Educational Development 45: 98–111.

72.

UNDP (2015) Briefing Note for Countries on the 2015 Human Development Report: Work for Human Development, Malawi. Available at: http://hdr.undp.org/sites/all/themes/hdr_theme/country-notes/MWI.pdf (accessed 1 November 2017).

73.

UNESCO (2012) At Least 250 Million Children of Primary School-Age Are Failing to Learn the Basics. Technical note prepared for the Education for All Global Monitoring Report 2012, Youth and Skills: Putting Education to Work. Report No. 2012/ED/EFA/MRT/PI/49/REV. Paris: UNESCO. Available at: http://unesdoc.unesco.org/images/0021/002193/219349E.pdf (accessed 1 November 2017).

74.

UNESCO (2014) Gambia: Education for All 2015 National Review. Prepared by the Ministry of Basic and Secondary Education in collaboration with the Gambia National Commission for UNESCO, for use at the World Education Forum, Incheon, Republic of Korea, 19–22 May 2015. Available at: http://unesdoc.unesco.org/images/0023/002314/231425e.pdf (accessed 1 November 2017).

75.

UNESCO Institute for Statistics (UIS) (2011) Financing Education in Sub-Saharan Africa: Meeting the Challenges of Expansion, Equity and Quality. Montreal: UNESCO-UIS. Available at: http://unesdoc.unesco.org/images/0019/001921/192186e.pdf (accessed 1 November 2017).

76.

UNESCO Institute for Statistics (UIS) (2015) School Resources and Learning Environments in Africa: Key Results from a Regional Survey on Factors Affecting Quality of Education (July 2015). Presentation. Montreal: UNESCO-UIS. Available at: http://uis.unesco.org/sites/default/files/school-resources-and-learning-environment-in-africa-2016-en/school-resources-and-learning-environment-in-africa-2016-en.pdf (accessed 1 November 2017).

77.

Verachtert

Van Damme

Onghena

Ghesquiere

(2009) A seasonal perspective on school effectiveness: Evidence from a Flemish longitudinal study in kindergarten and first grade. School Effectiveness and School Improvement 20(2): 215–233.

78.

Von Hippel

(2016) Year-round school calendars: Effects on summer learning, achievement, parents, teachers, and property values. In: Alexander

Pitcock

Boulay

(eds) Summer Learning and Summer Learning Loss. New York: Teachers College Press.

79.

Weiseman

Baker

(2004) The American summer learning gap from an international perspective. In: Borman

Boulay

(eds) Summer Learning: Research, Policies and Programs. Mahwah, NJ: Lawrence Erlbaum Associates, pp.53–69.

80.

Westberg

(2010) Girls Versus Boys? Factors Associated with Children’s Schooling in Rural Malawi. Master’s thesis, Department of Economics and Resource Management, Norwegian University of Life Sciences, Ås, Norway. Available at: http://www.sv.uio.no/esop/english/research/news-and-events/news/2014/westberg-thesis.pdf (accessed 1 November 2017).

81.

White

(1906) Reviews before and after vacation. American Education 19: 185–188.

82.

Willingham

Cole

(1997) Gender and Fair Assessment. Mahwah, NJ: Lawrence Erlbaum Associates.

83.

World Bank (2010) The Education System in Malawi. Working Paper No. 182. Washington, DC: International Bank for Reconstruction and Development/World Bank. Available at: https://doi.org/10.1596/978-0-8213-8198-4 (accessed 25 February 2017).

84.

World Bank (2015) World Data Bank: World Development Indicators, Malawi. Available at: http://data.worldbank.org/country/malawi (accessed 1 November 2017).

Is ‘summer’ reading loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from grade-transition breaks

Abstract

Keywords

Introduction

Background

Summer loss in the US

Differences in summer loss by gender

Interventions that address summer loss

‘Summer’ loss in the global South

Context

Research questions

Methodology

Measures

Sampling

Analysis

Findings

RQ1: Did students have lower literacy outcomes after the grade-transition break than before?

RQ2: Did the magnitude of grade-transition loss differ by gender?

RQ3: Did the magnitude of grade-transition loss differ by assignment to a generic literacy treatment group?

Discussion

Limitations

Conclusion

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

Notes

Author biography

References