Efficacy of Grade Retention in Macao

Abstract

Grade retention is widely used in Macao at the elementary and secondary levels. While many teachers and students believe retention gives low-performing students the opportunity to catch up, there is little empirical evidence to support such a claim. Using Programme in International Student Assessment 2009 (pisa, 2009) data, we examine the effect of grade retention on students’ learning time, learning strategy, metacognition, and academic achievement. We also analyze the influence of school policies to determine the net effect of retention. Our findings suggest that, contrary to conventional thinking, grade retention has a highly negative effect on the above factors. Students who repeated a grade did not benefit from this second chance, but rather were substantially held back in their learning trajectory. We suggest that schools in Macao shift their focus to designing programs that will help students with greater needs, rather than focusing exclusively on identifying such students.

* Funding Note: University of Macau Multi-Year Research Grant MYRG201500052FED.

Macao is a small autonomous city located in the southeast of mainland China that has experienced rapid economic and social development since the former Portuguese colony returned to the People’s Republic of China (prc). Its education system is distinctive to that of mainland China in several ways, most notably, in the high rate of grade retention at elementary and secondary level. In recent years, this retention has developed into a social issue, due to rising contention from parents and students. Despite efforts by the government to find solutions, official statistics show that the annual repetition rates remain relatively high. The latest official statistics suggest that at elementary, lower secondary, and upper secondary levels, 3.6, 11.1, and 4.6 percent of students repeated a grade in the 2012-13 school year. 1 When the accumulative retention rate is considered, Macao is ranked the second highest, just after Tunisia, among all countries and regions participating in the triennial Programme for International Student Assessment (pisa), conducted by the Organization for Economic Co-operation and Development (oecd). Over 40% of 15-year-olds in Macao reported a previous retention experience. 2

Grade retention is widely practiced in Macao and is typically viewed positively by educators. It is regarded as synonymous with educational quality assurance. 3 More specifically, grade retention is a practice used to help students strengthen their academic foundation and to keep them motivated, according to many teachers. 4 It is believed that retaining students early gives them a second chance at learning, which is considered more desirable than promoting academically weak students who typically find it hard to keep up in subsequent years. Thus, grade retention, or the threat of grade retention, is used to make students perform, possibly motivating them to work harder. However, such beliefs have not been scrutinized empirically. Evidence-based research on the process of identifying students for retention and its effects on various aspects of student learning is scarce in Macao. It is unclear how low achieving students benefit from retention, which in some cases occurs multiple times.

In this paper, we analyze the Macao data from pisa 2009. With close-to-census coverage of the 2009 cohort of 15-year-olds we are able, for the first time, to examine the effect of retention on four related factors—engagement in learning, learning strategies, metacognition, and academic performance—and compare retained students with promoted students in these aspects. In the following section, we first review the relevant literature on grade retention. Second, we introduce the data and variables used in the analysis. Third, referring to such data, we present the results of the statistical analysis and discuss the effect of grade retention on students with regard to the four aspects mentioned and the implications of such findings for education policy. Our research indicates that, contrary to popular beliefs widely circulated in Macao, grade retention is not an effective intervention strategy for enhancing student learning.

Literature Review

Effectiveness of Grade Retention

Grade retention is not a new practice within the educational community. 5 In many education systems, advocacy for grade retention is associated with a call to end social promotion, due to deteriorating educational standards. While under social promotion, students are automatically promoted to the next grade regardless of their learning outcomes, practitioners of grade retention require a student to remain in the particular grade for the subsequent school year if she/he fails to meet explicit criteria. Proponents of grade retention see it as an effective intervention strategy for students with academic and/or behavioral difficulties. That is, it is believed that grade retention is an effective solution to academic failure or school maladjustment. 6

There are some empirical findings to support such claims. For instance, the reading proficiency of retained students is found to have improved one and two years after the retention. 7 Despite this, opponents are against retention practices for at least two reasons. First, some opponents point out that the potential social and emotional consequences of grade retention have been overlooked and that the negative effects of grade retention on students’ socioeconomic well-being outweigh any potential academic benefits. 8 For example, compared with their promoted counterparts, retained students have lower self-esteem, poor academic self-concept, poor relationships with their peers, and engage less in academic and school activities. 9 Second, others doubt the potential academic benefits supposedly associated with grade retention. They argue that grade retention could at most lead to a temporary or very short-term academic advantage, 10 and that grade retention is actually an ineffective strategy for improving students’ academic performances. 11 While there are pros and cons to practicing grade retention, empirical analyses derived from a number of important meta-analyses have thrown its effectiveness into serious doubt. 12 In addition, grade retention has been found to be a significant predictor of whether a student will drop out of high school. 13 It seems that grade retention is not merely an ineffective learning strategy for helping students to remain in the education system, but a practice that results in students leaving the education system altogether.

From this well-developed literature, we see that the focus of the discussion has shifted over time from debating the academic efficacy of grade retention, to exploring the effect of the timing of grade retention on students’ potential, 14 discussing what should constitute the basis of practicing grade retention, 15 and eventually to examining new substitute intervention strategies for students with academic and behavioral difficulties. 16

The Macao Context

Macao was ruled by the Portuguese from 1887 to 1999. During the colonial era there was no public school system for the local Chinese and the provision of basic education was shouldered by private schools run by the civil society. 17 Almost all major civil organizations run their own schools. Even after the handover to the prc, this aspect of the education system remained. Today, more than 95% of students are studying in private schools, with no common standardized examination at any level. The government’s responsibility lies more in funding and regulation, with schools in Macao still enjoying autonomy in a wide variety of areas, such as staffing, promotion policy, curriculum design, and student recruitment. 18

The survival of private schools depends heavily on student enrollments, as the Macao government provides a de facto voucher system to finance basic education (K-12). For each class of a certain number of students a school takes, the government transfers a fixed amount of subsidies to that school. However, without common/standardized examinations in Macao, private schools do not have readily available indicators to demonstrate their quality to potential students. Thus, many schools rely on their criteria for retention. Given the autonomy of private schools in setting their own criteria for grade retention, schools seem to consider that the stricter the criteria, the better the school will appear to students and their parents.

The lack of consensus and common guidelines on retention criteria result in substantially different school-level policies. Wong 19 analyzed the retention criteria from 43 schools and found that they set different pass marks and retained students for failing a different number of subjects, ranging from a unit of a core subject to six subjects altogether. Wong and Zhou (under review) further analyzed the retained student body and found that the students who were most likely to be retained were boys from low socioeconomic families. Grade retention has been such a common practice that it is only recently, since the pisa data revealed that the retention rate in Macao far surpassed many less developed nations and regions, that it has been considered controversial. Although grade retention is seen in a positive light, retained students actually do not get additional assistance, academic or otherwise. Educators and the general public are seemingly satisfied with the existing treatment of retained students, whereby they are placed in regular classrooms without receiving any remedial instruction. 20 Surprisingly, there is little discussion about the treatment of retained students in Macao. Although emerging studies are beginning to identify student characteristics and school policies associated with retention, little is known about its efficacy in Macao.

Gaps in the Literature and Research Questions

By 9th grade, about 40-50% of us students have been retained, so the cumulative rate of retention in the usa is as high as that of Macao. 21 Most studies of retention are conducted in the United States and the applicability of their findings to Macao is unclear. In addition to the direct effects of retention in Macao, we are also concerned with the possible spillover effect of retention on the overall mean achievement of students, an area where there is little consensus within the existing literature. Thus, there is a pressing need for an empirically based overview of the efficacy of grade retention in Macao before any serious discussion can be carried out and before any sensible educational policy can be implemented. In this paper, we seek to answer the question, what are the effects of retention on student learning? Put simply, does grade retention deliver what is promised for students in Macao? We compare four factors between students who have had previous retention experiences and those who have not:

1. engagement in learning;

2. learning strategies;

3. metacognition; and

4. academic performance.

These four factors are important measures of student learning. By comparing engagement in learning and learning strategies, we seek to understand whether previously retained students learn differently from their peers. Metacognition and academic performance are used as learning outcomes, providing us with concrete yet varied measurements. Our premise is that if grade retention helps students to catch up, then we should observe small positive differentials in these aspects from students who have past retention experience over those who do not. That is, retained students should be more engaged in learning and use more effective learning strategies, and therefore be capable of a higher level of thinking, eventually performing better academically.

Data, Research Design & Statistical Model

We used data from pisa 2009 in our analysis. pisa is currently one of the largest international assessment programs and has been conducted by the oecd every three years since 2000. For each program, students aged 15 years who are still attending school are assessed on their reading, mathematics, and science skills. In addition, a student survey is conducted. Macao has participated in pisa since 2003. Unlike other participating countries and regions in pisa, which use systematic Probability Proportional to Size (pps) sampling, Macao includes every secondary school in the survey due to its relatively small population. 22 Within each school, nearly all students participated in the test and answered the survey, with statistical methods used to adjust for non-respondents. As a result, 98% of the target population responded to the survey and completed the tests.

Variables

Descriptions of the variables are listed in Table 1. We explain the construct of each variable in this section.

Dependent Variables

pisa includes students’ self-reports of the amount of time (in minutes) they spent every week on the three tested subjects: test language, 23 mathematics, and science. We estimated learning time by multiplying the number of minutes spent on average on the particular subject class by the number of class periods of that subject per week. 24 We used this information as an indication of student engagement in learning at school.

In addition to the time devoted to each subject, we also looked at the particular learning strategies used by students; specifically, the use of memorization, control, and elaboration strategies in their study method. These three continuous indices were constructed by the oecd team using the one-parameter item response model, or Rasch model. The memorization strategy indicates the degree to which students memorize the material without demonstrating understanding of it. In contrast, the control strategy can be regarded as a more effective learning strategy, as it requires students to figure out what they are trying to learn and what information they need to enhance understanding. By comparison, the elaboration strategy can indicate a higher level of learning, as it requires students to make connections between what they have learnt and both their prior and real-world knowledge. Higher values on these indices indicate a greater use of that learning strategy.

table 1

Variable description

Metacognition measures the level of thinking ability, involving active control over the cognitive processes engaged in learning. In this sense, metacognition can be used as a learning outcome. Two sets of measures of metacognition were used in this study, understanding and memorization of material, and the ability to summarize such material. Each index is a continuous number that ranges from 0 to 1. The construction of these indices involved first letting each student complete a reading task and then asking them to rate the usefulness of a set of strategies given to them. The ratings they gave were expressed as order relations and the consistency compared between the student order relations and expert order relations, which were predetermined by each nation’s expert center. According to the oecd, the score could be interpreted “as the proportion of the total number of expert pair-wise relations that are consistent with the student ordering.” 25

Lastly, we used test scores. pisa uses a testing design called plausible values, 26 in which each student is given five possible test scores. We used all five plausible values at the student level and the appropriate statistical procedure to take into account the design of the test. 27

All of the variables are continuous. The descriptive statistics are reported in Table 1.

Independent Variables

The retention variable was the main independent variable, which we created using item No.7 of the student questionnaire, and defined as binary. It takes a value of one if the student ever repeated a grade at primary or lower secondary level, otherwise it is zero. Other independent variables included students’ gender, socioeconomic status (ses), family structure, immigration status, whether from a foreign background (i.e., non-Chinese), whether they had siblings, attitude toward school, and their perception of the class’s disciplinary climate. The attitude variable, ses, and disciplinary climate variable were constructed by the pisa team. They were all measured on a continuous scale and standardized with a mean of zero and a standard deviation of one. The choice of these variables was based on an empirical consideration of the Macao context and other empirical findings. 28

Research Design

As Hong and Raudenbush summarized, 29 most research utilizing cross-sectional data has used the “same-grade” comparison. Using such a design, researchers have compared the outcomes of children repeating a grade with those who entered the grade for the first time. The same-grade comparison allows researchers to compare the “standing of retainees relative to their classmates before and after the retention.” 30 However, studies that have used this design have not produce consistent findings regarding the learning outcomes of the retained students and their promoted peers.

Because the target population of the pisa studies is 15-year-olds, we were able to use the “same-age” comparison. Under this design, we compared students of the same age group who were either retained at lower grades or promoted to the subsequent grade. The same-age comparison implicitly controls for student maturation because all of the students are from the same age group. In past studies, researchers were limited by the fact that the content of assessment was correlated with the curriculum to a certain extent. Therefore, by comparing students from different grades, the effect of curricula inevitably confounded the effect of grade retention. The advantage of the pisa data is that the assessments are explicitly designed to be independent of curricula, measuring student proficiency in specific knowledge domains and skills.

Guided by this design, we limited our comparison to those who, (a) had never had a retention experience and (b) had been retained and were currently in a lower grade (grade 7 through 9). 31 We excluded students who had retention experience but progressed faster than their regular peers. 32 In an alternative specification of the sample for sensitivity analysis, we excluded those who were never retained, but who stayed in lower grades for the purpose of redshirting. 33 The distribution of student grades is listed in Table 2.

table 2

Grade distribution by retention status

Statistical Model

We used three sets of comparisons between retained and non-retained students. The first was a simple t-test to compare the sets of dependent variables, which we termed naïve comparison. The second was a series of single level multiple regression models using the student level independent variable as controls. The regression model is specified in Equation 1.

Under this specification, dv describes the series of dependent variables. They include time devoted to learning, learning strategy, metacognition, and achievement scores. Subscripts refer to the index student, i, in school, s. The independent variables are the same as those listed in Table 1.

In the Macao context, schools typically set their own rules with regard to learning time and organize activities within or outside of the classroom. Thus, the school effect may confound the individual effect if the factors that cause such differences in retention practice across schools are not explicitly accounted for. However, as pisa does not collect school-level information on this particular issue, analysts could not easily control for the influence of school-level policy. To address this problem, we used the fixed effect approach to control for the variation of the dependent variable due to the school characteristics. 34 This approach enabled us to obtain estimates beyond any school effect. We thus specify our third comparison in Equation 2. The dummy variable represents the school fixed effect.

We follow the recommendation of the oecd to use proper statistical weights to account for non-respondents. 35

Analysis

In Table 3, we present the naïve comparison between retained and non-retained students. The result is straightforward in that students who have previous retention experience at primary or lower secondary level lagged behind in all of the areas we measured. On average, they spent 8 minutes less on reading, 15 minutes less on mathematics, and 104 minutes less on science per week. They used less effective learning strategies, as reflected by the 0.18-0.4 standard deviation differentials compared to their peers in the non-retained group. Their metacognitive measures were 0.26 and 0.33 standard deviations lower than those of their peers. The differences in achievement were equally substantial, with 69 points less in reading, 77 less in mathematics, and 65 points less in science. Considering the standard deviation in achievement score is 100, these effects reflected more than half a standard deviation difference.

Because the naïve estimator does not take into consideration the influence of personal background, the observed difference might be due to the presence of individual characteristics. Next, we fit a series of regression models using the specification stated in Equation 1. The results are presented in Table 4, with the findings summarized in the row labeled “Retention.” The results are qualitatively similar, as the sign of the estimates does not change. The statistical significance pertains to all models, showing confidently that students who were previously retained are statistically different from their peers in these aspects. Other variables such as attitude toward school and ses also display the expected correlation with these learning measures. We caution readers to interpret the coefficients as associational rather than causal.

table 3

Descriptive statistics

In the last specification (Equation 2), we explicitly control for school context. This is important because the enrollment pattern and school retention policy might correlate with an individual’s previous retention experience. Previous studies have reported diverse school-level retention policies. 36 We did not have any associated information on school policy to merge with the student data, neither could we identify these schools. Thus, we chose to use the fixed effects technique to control for all of the variation in the outcome variables that is induced by school factors. The results are summarized in Table 5. The layout of Table 5 is identical to that of Table 4. The estimates are considerably smaller for the retention variable compared to the estimates in Table 4. This is understandable because the school fixed effects should have absorbed some of the variation in outcome measures. Again, the quality of the coefficients remains the same and the significance unchanged. To put it into perspective, when school background was accounted for, students who were previously retained spent less time studying language, mathematics, and sciences in particular. Their use of learning strategies of various types was less frequent by 0.11 to 0.28 of a standard deviation. Similarly, their metacognitive ability was lower, though not to as large an extent. The differentials are 0.12 and 0.21 of a standard deviation. These are all translated into the achievement gap. In the three subjects tested, retained students achieved relatively lower scores. The differences are approximately 13 points in reading, 16 points in mathematics, and 12 points in science.

Discussion

Retention is a common practice widely used within the education system in Macao. As the high number of students who are retained at elementary and secondary levels has persisted in recent years, tension has arisen among educators, parents, and students. The Macao community is contesting the efficacy of retention in improving student learning at school, yet much of the debate is without empirical evidence to support claims on either side. In this study, we provide the first empirical evidence, using high quality near-census data, on the effect of retention on students’ engagement in learning, learning strategies, and learning outcomes, with our findings consistent in all of these aspects.

In short, students who repeated a grade or more did not benefit from such learning experiences. In fact, our statistical analysis shows that they were engaged to a lesser degree, expressed by a shorter amount of time devoted to the learning of language, mathematics, and science subjects. The differences were most pronounced in science learning, with retained students spending less than 100 minutes per week on this subject. This implies that retained students in Macao do not receive additional treatment, let alone special treatment, during retention. While reduced learning time might not be undesirable in itself, when coupled with the less effective use of learning strategies we observed from the pisa data, the possible fallout should alarm the education community. Perhaps it is understandable that retained students used the memorization strategy less than their promoted peers as they had learnt the material before. However, what is worrying is that compared with their promoted peers, retained students were less likely to use an effective learning strategy (control strategy) or a higher level learning strategy (elaboration strategy). Consequently, their metacognitive ability, in both aspects, was considerably lower and lagging behind their promoted peers. Such differences could have long-term effects on learning processes. In addition, these differences translated into worse test scores.

table 4

Comparing retained and non-retained students using regression with student-level control

a Single-parent family as reference group

b Native student as reference group

Standard errors in parentheses

=”+ p<0.10 * p<0.05

table 5

Comparing retained and non-retained students using regression with school fixed-effects

a Single-parent family as reference group

b Native student as reference group

Standard errors in parentheses

=”+ p<0.10 * p<0.05 ** p<0.01 ***p<0.001”

We offer some preliminary explanations for what might cause such differences. First of all, the school context plays an important role here. When we controlled for the school context, the differences in the above measures shrank considerably between retained and non-retained students. This suggests that considerable variation exists among schools. When we analyzed within schools, the differences were smaller, but still substantial. This finding suggests a need to look into the particular school-level policy that leads to retention decisions and also to identify school-based remedial programs that may help the student to improve.

This is where we believe current practice falls short. According to another study by the authors that utilized qualitative information from administrators and teachers, schools in Macao are more fixated on identifying students who are not fit for promotion instead of designing measures to help them succeed. 37 The purpose of the implementation of such a policy is beyond the scope of this paper, but it is worth noting that our interviews with retained students showed that they did not receive any extra help either from individual teachers or from the school. When retained, they were simply placed in regular classrooms without additional help, let alone tailor-made assistance. Some even reported being intentionally ignored and discouraged in classrooms. Their social experience deteriorated due to alienation from peers who were removed from them. It is plausible to hypothesize that this may hamper students’ motivation to learn, causing them to eventually lose confidence. Considering that many retained students suffer from such sink-or-swim treatment in schools, it becomes apparent why retained students do not use more effective learning strategies. If a tough retention policy helps schools to establish a reputation within the education market, the unintended consequence is that a large number of students who struggle with schoolwork are falling behind their peers.

Macao is unique in the sense that retention is used on such a large scale. If we look at the student population as a whole, on average students who have been previously retained score more than two thirds of a standard deviation lower than those who have not. This means that if these students reached the average achievement level of the non-retention students, the average performance by Macao students would greatly improve. This is consistent with other education systems where cross-national studies have found that school systems with higher retention rates also have lower student performance. 38 We also fear that with retention used so frequently, there is a possible negative spillover effect on the rest of the population. Thus, the discussion on the modification of retention policy should not focus only on individuals, but should be targeted at the systemic level.

We acknowledge some limitations of this current study. While pisa offers representative data on the entire student population, we still lack information on the treatment of retained students in schools. We do not know exactly who their classmates are and how they interact with teachers. Future studies could use observation and other means to analyze specific practices used by schools.

In addition, we did not have measures of previous learning status and therefore could prevent reverse causation. It is possible that retained students were already worse in these aspects before retention compared to their promoted counterparts. In such a scenario, retention would alleviate some of the learning obstacles that shrink the gap between these students and non-retained students. Yet, if this is true, it would suggest that the achievement gap would be much wider if retention was not in place. Future studies require a careful evaluation of the situation, involving longitudinal data on student characteristics.