Abstract
How can a bonus program affect the performance of teachers in classrooms? This paper collected data from teachers employed in Brazilian state-run schools to examine the key mechanisms through which a teacher bonus scheme can have a causal effect on the academic achievements of students. Using mixed-methods research and data triangulation from questionnaires and semi-structured interviews with teachers, this study finds limited empirical evidence supporting the bonus program of the state of São Paulo. The bonus program created no additional incentive for educators to start/continue working in state schools, and it was not responsible for the improvement of teaching activities.
Introduction
The use of pay-for-performance (PFP) programs in schools is seen by some public actors as a key tool to improve professional engagement in classrooms and, consequently, the performance of students (Barrera-Osorio & Raju, 2017; Loyalka et al., 2019). Therefore, these programs have been implemented in several countries to correct eventual market efficiency problems deriving from traditional fixed-salary schedules (Balch & Springer, 2015; Liu et al., 2016). This is because many studies have indicated that years of experience and educational degrees of teachers alone are not strongly correlated with students’ academic achievements (see Kane et al., 2011; Rivkin et al., 2005).
Despite the increasing number of merit pay programs, there is very limited evidence in the literature showing how PFP programs can directly affect student outcomes (Hanushek & Woessmann, 2011; Nyberg et al., 2018). In theory, the change in teaching practices is a necessary condition for a causal relationship between these two outcomes (Jones, 2013), but a more insightful understanding of the mechanisms behind this association is required to support public decisions concerning the optimum design of PFP programs in the education system (Stephens, 2015). Therefore, researchers have appealed to the academic community to expand the empirical investigation to teachers’ perceptions related to merit pay (Marsh, 2014; Viscardi, 2014).
The current paper will follow this guidance and present the studied teachers’ perspectives in relation to one merit pay program in the Brazilian education system. Internationally comparative studies have already shown that the performance of students and the salaries of teachers in Brazil are low compared with other countries. In the PISA 2018, for example, 15-year olds in Brazil score 413 points in reading literacy compared to an average of 487 points in OECD countries, and similar gaps were found also for Math and Science (see OECD, 2018). According to the 2018 Global Teacher Status Index (see Dolton et al., 2018), the average annual salary of a Brazilian starting secondary school teacher was USD 13,000 (adjusted for purchasing power parity, PPP). Therefore, Brazil sits in the eighth position in the ranking of countries with the lowest paid teachers.
Against this background, many school districts in the country started to link additional remuneration for teachers with the improvement of academic achievements among students. However, despite the increasing number of teacher bonus programs in Brazil, scholars have encountered substantial difficulties in gathering reliable data to assess the impact of these PFP programs, since most of them have been implemented in ways that hinder the creation of statistically valid comparison groups (Bresolin, 2014). To the best of my knowledge, in Brazil, no empirical evaluation began in advance of the implementation of the teacher bonus programs—thus rendering impossible the use of randomized control trials to investigate their impact.
As a consequence, the scientific monitoring and evaluation of teacher bonus programs in Brazil are rarely reported. To the best of my knowledge, only three studies have been able to overcome the existing data limitations and thus examined empirically the impacts of the teacher bonus scheme in Brazilian schools. While Lepine (2016) and Oshiro et al. (2015) evaluate the effects of PFP programs for teachers vis-à-vis the academic achievements of students, Bresolin (2014) researches questions similar to those investigated in this paper. The author investigates the impact of teacher bonus programs on: (i) teachers’ pedagogic practices; (ii) absenteeism and teachers’ rotation; and (iii) interlocution with family members to ensure students’ presence at school. For the study, Bresolin (2014) applies the data from the Prova Brasil (Brazil Test) questionnaire—which was answered by principals, teachers, and students—and a propensity score matching methodology to calculate the differences in outputs between the states that implemented the PFP programs for teachers and those who did not. According to the author, the bonus program had a positive impact on the teachers’ absenteeism and on the frequency with which they correct students’ homework.
To counterbalance the current data limitations, this paper has carried out an own data collection in Brazil and applied a mixed-methods design to investigate the effects of a teacher bonus program on the professional practices and behaviors of educators. To address the research questions in a rigorous and systematic manner, I create a research approach integrating quantitative surveys and qualitative interviews. Using an explanatory sequential design—which is considered the most straightforward approach in undertaking mixed-methods research (John & Creswell, 2000)—I first collect quantitative data for the descriptive survey research, and then, I include qualitative information to generate more in-depth explanations of the initial findings.
The use of mixed-methods research for the investigation of PFP programs in the education system is becoming increasingly common in the literature. 1 Therefore, conceptual similarities can be found between this research project here and existing studies. Brasington (2016) utilizes a mixed-methods design—including online surveys, qualitative interviews, and analysis of district documents—to investigate changes in educators’ motivation and job performance with the implementation of a PFP program in a suburban school district of Michigan. While the online survey was designed to address school educators’ perceptions related to the merit pay program, the group and individual interviews were conducted to enhance and elaborate on data collected from the surveys. Also for the United States, Marsh (2014) uses a mixed-method approach for the investigation of teacher perceptions concerning the performance-based pay of the Pandora School District in Missouri. He first drew on a questionnaire to flesh out the opinions and perceptions of teachers on the topic, and then from those respondents, he selected a subsample for in-depth and intensive interviewing. Similarly, Liu et al. (2016) applies quantitative and qualitative data to investigate the attitudes of Chinese teachers toward performance-related pay, and the underlying factors influencing their expressed attitudes.
The data for the current paper stem from teachers who worked in the state elementary schools in Campinas during the 2020 school year. Campinas is a Brazilian municipality with around 1.2 million inhabitants located in the state of São Paulo. In 2008, the state implemented a PFP program for all staff employed in the São Paulo state education network (SEE-SP). This is a collective bonus program that rewards staff according to the achievement of preset targets related to academic performance and the pass rate of students, the so-called Education Development Index of the State of São Paulo (IDESP). All employees from the SEE-SP are eligible for the bonuses, which is paid once a year and ranges in amount from between 0% and 20% of the individual’s annual salary depending on the success of the school in achieving the IDESP in the previous year (Castro & Lopes, 2016).
The remainder of the paper is structured as follows. Theoretical Motivation and Hypotheses describes the theoretical motivation and hypotheses for the research. The Brazilian Context presents background information on the Brazilian education system. The subsequent section explains the methodological design and procedures. Empirical Findings reports the empirical results, and Conclusion concludes.
Theoretical Motivation and Hypotheses
As in other countries around the world, the implementation of teacher bonuses in Brazil was also justified by the expectation of an increase in student performance (Oshiro et al., 2015). According to Yuan et al. (2013), two main theoretical frameworks can help explain the link between performance incentives and teachers’ motivation: expectancy theory (Vroom, 1964) and goal-setting theory (Locke, 1968). Vroom’s expectancy theory suggests that the motivation of teachers in relation to the bonus program is dependent on three main factors: expectancy, instrumentality, and valence. Expectancy because the amount of effort that teachers expect to meet the program targets is dependent on the belief that their higher personal effort will yield student benefits in terms of academic achievement. Instrumentality refers, meanwhile, to the genuine likelihood of receiving the bonus in case of meeting performance expectations; valence is the private desirability of attaining the bonus. Locke’s goal-setting theory complements this expectancy theory, stating that teachers’ motivation will be also affected by their individual perception of the fairness of the bonus program, and by their understanding of the program’s content-related design and goals.
Since the success of implementing PFP programs in the education system depends heavily on their support by teachers in the classroom (see Ballou & Podgursky, 1993; Liu et al., 2016), this paper integrates the expectancy and goal-setting theories into the evaluation of bonuses offered in the Brazilian education system. It strives to identify the individual knowledge base, acceptance, and preferences of teachers in relation to the PFP program in which they are involved. Therefore, the first hypothesis is:
The PFP program implemented in state schools of São Paulo ties the bonuses to measures of student achievement, as a public policy tool to improve education quality. Therefore, to understand the dynamic behind bonus programs in the education system, it is necessary to highlight the main effects that are expected to take place when teachers receive the opportunity to connect salary improvements with the improvement of student outcomes.
First, performance-related-pay programs have the capacity to reform the single salary schedule, providing financial incentives for more productive workers to enter/stay in the teaching profession (Goodman & Turner, 2013; Podgursky & Springer, 2007). Wage differentiations between high- and low-performing staff is an elementary condition for the intrinsic motivation of teachers (Muralidharan & Sundararaman, 2011) since hard-working ones tend to become dissatisfied if they see no difference in earned income between them and those who shirked their duties (Kremer et al., 2005). Several studies of teacher turnover have found that high-ability teachers are more likely to leave the profession than their less-able peers (see Murnane & Olsen, 1990; Podgursky et al., 2004). As such, the second hypothesis can be derived as follows:
Work attendance is another possible mechanism linking the bonus program to student performance. Given the inherent difficulty of quantifying teachers’ effort in the classroom, researchers have used work attendance as a proxy for the evaluation of the effectiveness of PFP programs in the education system (Goodman & Turner, 2013). In the case of the bonus program in São Paulo, the absenteeism rate of teachers enters directly into the calculation of the bonuses paid. This is done by penalizing every single absence with a reduction in additional payments (Oshiro & Scorzafave, 2011). For this reason, the third hypothesis concerns the link between bonuses and work absenteeism:
The most common argument cited in the literature in favor of teacher bonuses is that financial incentives provide these individuals with an additional motivation to put more effort into the instruction process (Gneezy et al., 2011; Lavy, 2009). In the literature, there is ample evidence supporting the argument that PFP programs change teacher behavior. However, supporters and opponents of PFP programs in the education system generally disagree about the reason for this change occurring. While the former argue that teachers will undertake changes to promote a broad capital acquisition by their students (see Lavy, 2009; Muralidharan & Sundararaman, 2011), skeptics point out that teachers are able to manipulate student performance by short-term strategies focusing efforts on skills and actions that inflate student scores in the standardized tests used for the bonus calculation (see Behrman et al., 2015; Glewwe et al., 2010; Jacob, 2005). From this background, the fourth hypothesis relates to evidence for changes in teaching practices due to bonuses:
Opponents of merit pay programs in schools frequently cite the principal-agent theory to claim that performance-based compensation can result in less collaborative school environments. One of the most frequently expressed criticisms in this regard is “teaching to the test,” a teacher practice aimed at narrowing the curriculum to artificially increase the performance of students per the government-run tests used for bonus calculations (Podgursky & Springer, 2007).
Furthermore, other examples of manipulative behavior by teachers have already been reported in the literature. Behrman et al. (2015) demonstrates that the reason for the higher test scores in schools with bonus programs can be explained, in part, by the higher rate of student cheating in the administrative exams used for the calculation of bonus payments. Jacob (2005) shows that teachers tend to exclude weak students from testing. Jacob & Levitt (2003) points to dishonest teacher actions to inflate test scores, such as providing correct answers to students or changing the latter’s responses on submitted answer sheets. Against, this background the fifth and final hypothesis of this paper concerns increased attempts at cheating by teachers to inflate student test scores with regard to the standardize academic tests used for bonus calculations:
The Brazilian Context
The Brazilian constitution shares the responsibility for the country’s educational system between all three levels of the Federation. While the federal government’s role is limited to the setting of curriculum guidelines and the provision of financial and technical assistance to the subnational levels, states and municipalities are primarily responsible for maintaining public education. Consequently, these two subnational authorities operate the overwhelming majority of public schools in the country and are the key players behind the wage policies for teachers (Becker, 2014).
Up until the start of this century, the remuneration of teachers in Brazil was based on a uniform pay system that ensured the same salary for those with the same years of experience and education level. But after the turn of the millennium, the idea of merit pay for educators would grow in popularity with the proliferation of the (large-scale) standardized tests for the evaluation of student achievement (Brooke, 2016). Then, public authorities began to introduce performance-monitoring frameworks for schools to improve the quality of education system, and to use teacher bonuses as an accountability tool to influence educational outputs (Bresolin et al., 2018). 2
Starting with Rio de Janeiro in 2001, several school districts in Brazil would implement accountability programs in which the payment of financial bonuses to teaching staff is dependent on the achievement of targets related to students’ academic performance (Brooke, 2016). 3 Apart from the teacher bonus scheme of the city of Sobral—which pays out specifically based on individual performance—all other PFP programs are group incentive-based ones that use performance measures at the aggregate level (normally, school) to reward the efforts of educators in a collective manner (Brooke, 2013).
This is the case also for the teacher bonus program of São Paulo, Brazil’s richest and most populous state. With approximately 4.3 million students, 5,300 schools, and 220,000 teachers, the SEE-SP is the largest education system in Brazil (SEE-SP, 2020a). At the time of the bonus scheme’s implementation (2008), the base salary for teachers working full time (40 hr per week) in elementary state schools was BRL 1,309.17 (ca. PPP USD 663.69) per month—equivalent to 2.8 times higher than the national minimal wage at the time (Barbosa & Fernandes, 2016). 4 Compared to the salaries paid in the other 26 Brazilian states, the wage paid in São Paulo held 11th position in the ranking of highest teacher salaries nationwide (APEOC, 2010).
In November 2008, the state of São Paulo launched the “School Quality Program” (SQP), with the purpose of (i) establishing clear and objective criteria for the performance evaluation of schools and (ii) setting the targets need to promote quality standards in the education system (Castro & Lopes, 2016). The SQP was made responsible for the implementation of the IDESP, which is a composite indicator of school quality based on two main components: the pass rate, indicating the share of students who have completed the respective educational levels on time; and the performance rate, which is measured as the average scores of students per the government-run compulsory test called the “Evaluation System of Learning Achievement of São Paulo” (SARESP).
The IDESP establishes annual and individual quality targets for each school and educational level: lower primary education (Grades 1–5), upper primary education (Grades 6–9), and secondary education (Grades 10–12) (Brooke, 2016). At the beginning of the school year, the SEE-SP publishes the IDESP for the current year (used as a target) and the IDESP achieved in the previous one, thereby enabling the different stakeholders to track the progress of the school in achieving these goals (Castro & Lopes, 2016).
Then, teacher bonuses are calculated according to the degree of IDESP improvement reached by the school in which the staff is employed and proportional to their work attendance in the year in question; no bonuses are paid to teachers with a work absenteeism rate higher than one-third of classes missed. 5 The maximum bonus paid to teachers corresponds to 2.9 times their (monthly) salary and is reached when the absenteeism is zero and the target achievement is equal to 120% (or more). In case of no work absenteeism and an IDESP improvement of 100%, the additional payment is equivalent to 2.4 times the monthly salary, and so on (e.g., 50% improvement = 1.2 times the salary gain) (Oshiro et al., 2015). In March 2009, the SEE-SP paid out teacher bonuses for the first time, disbursing around BRL 590.6 million related to the achievement of the IDESP in the year 2008 (Lepine, 2016).
Methodology
This paper follows Liu et al. (2016) and Burns et al. (2009) in developing a mixed-methods research approach that includes questionnaire surveys and interviews to investigate the impact of the teacher bonus program.
Instrumentation
For data collection, this paper applied two independent but complementary research instruments. For the quantitative research, I created a structured questionnaire—which was based on the empirical investigation of Soto-Pérez et al. (2020); Liu et al. (2016); Burns et al. (2009)—containing 40 questions that examine individual attitudes toward the teaching profession, levels of acceptance of PFP programs, and experience with the IDESP. 6 Following Liu et al. (2016), these questions asked individuals about their level of agreement with a series of statements related to the topics of investigation, and were measured on a 5-point Likert scale ranging from strongly disagree (1) to completely agree (5). To reduce the time spent on filling out the questionnaire and consequently to increase the likelihood of reliable responses, the author created an online survey and sent it to the schools through email in the 30th calendar week of 2020. 7
Since structured surveys are not efficient in providing an in-depth understanding of the statements and perspectives of respondents (Weiss, 1998), the paper applied qualitative (remote) interviews to complement and supplement the data gathered in the survey. For this, I used a semi-structured interview guide that contains open-ended questions—thus allowing participants to express their own narratives embedded in their personal experiences related to the bonus program. Besides the introduction and concluding thoughts, the interview guide—which was based on the work of Burns et al. (2009) and can be found in the online Supplementary Material of this paper—highlights two main topics related to PFP programs: expectancy theory and teaching practices in the classroom. For the latter, two different effects of the teacher bonus program were investigated: the “good” practices in the learning process that may have changed with the bonus scheme’s introduction and the dishonest behavior of teachers regarding inflating students’ academic scores per the government-run tests used for the calculation of bonuses.
Between August and September 2020, the author conducted the semi-structured, open-ended interviews with the teachers. All the data collection was strictly confidential, anonymous, and taken remotely using video conferencing software (Microsoft Teams and Skype). Then, the resulting interviews, each between 20 and 43 min long, were tape-recorded, fully transcribed, and analyzed to illuminate the main aspects vis-à-vis the research questions. 8
Sample Selection
The empirical research was conducted in the Brazilian city of Campinas with teachers from state primary schools. With a population of over 1.2 million people, Campinas is the third most populated municipality in the state of São Paulo. According to the 2018 school census, the school district of Campinas enrolls approximately 124,000 students in 658 schools, from which 175 (28.8%) are state-run institutions where 4,729 teachers are employed. Of these 175 state schools, 112 provide primary education—in which 71,509 students are enrolled and 1,932 teachers employed (INEP, 2019).
Using the online search tool for schools from the SEE-SP, 9 I collected the contact details of the state primary schools in Campinas and phoned the principals of all 112 to explain about the research project and to identify a willingness to cooperate. Next, I sent an email to the principals with a formal invitation to participate in the survey. This email also included the link to the online questionnaire, to be forwarded to the teachers employed in the school. Finally, 2 weeks after the email, I contacted the principals by phone once again to ask about the forwarding of said email to the teachers. 10
For the qualitative interviews, the paper applied a common practice used in mixed-method analysis and derived the subsample for the qualitative research from the overall quantitative sample (Thompson, 2004). For this purpose, I used the online questionnaire as a starting point for the selection of those to participate in the interviews. In the introduction to the survey, the author informed the participants about the two forms of data collection for the research project and mentioned the existence of the subsequent interviewing. Then, those with an interest in participating in the semi-structured interviews were asked to declare this willingness in question 37 of the questionnaire.
Empirical Findings
This section summarizes and discusses the main results, divided into three parts. The first applies a descriptive-survey research design to report the quantitative findings related to the questionnaire. The second part complements this with individual insights generated by the interviews with open-ended questions. The third and final part triangulates the data from both methods in order to provide support or additional evidence for the preference patterns of respondents.
Quantitative Results
Table 1 reports the responses from the (online) structure questionnaire and summarizes the responses from the 53 teachers employed in 25 schools who completed the online questionnaire. 11 For clarity, the table groups the questions by their respective hypotheses and reports the percentage of responses for each of the 5-point Likert scale alternatives.
Quantitative Results from Structured Questionnaire.
Notes. Data collection was carried out through structured questionnaires with 53 teachers from the SEE-SP in the municipality of Campinas (see questionnaire in the online Supplementary Material). Data are not representative of the whole population. The questions ask teachers about their level of agreement with the respective statements, and were measured on a 5-point Likert scale ranging from strongly disagree (1) to completely agree (5). Questions 41–55 are answered only if the answer to Q5 (worked in SEE-SP in the pre-bonus period) was “Yes.” For Q35, response 1 is “Yes” and 2 “No”. IDESP = Education Development Index of the State of São Paulo; SEE-SP = São Paulo state education network.
Source. Author’s own data collection, 2020.
The sample was composed of 35 female and 18 male teachers, on average 46 years old and with 13 years of experience. Only 2% of the teachers have no tertiary education, while 73% have a bachelor’s degree as their highest educational level, 19% a master’s, and 6% a doctorate. Some 74% of the respondents exclusively dedicate themselves to teaching in the SEE-SP, while 26% work also in the private or municipal education system.
The quantitative findings presented in Table 1 indicate reduced support for PFP programs among teachers (see Q15–Q19). H2—asserting that the teacher bonus program might have created an additional incentive for the work in the SEE-SP—can be rejected categorically per the majority of the teachers’ responses (see Q21–Q25).
In relation to the impact of the teacher bonus program on work absenteeism (H3), Table 1 suggests a small but positive impact (see Q41 and Q52). The empirical evidence for H4—which investigates the changes in teaching practices and behaviors due to the bonus program—is mixed. For the questions related to teaching activities (Q42–Q48), the answers are relatively evenly distributed over all five alternatives. But the data indicated a higher level of brainstorming between teachers since the implementation of the bonus program (see Q49) and a higher willingness to help their work colleagues (see Q50). Finally, the H5 investigates the cheating of teachers in relation to the assessment basis for the bonus program. In this case, the quantitative survey confirms that cheating is an element present within the bonus program (see Q30–Q36).
Qualitative Results
The research project conducted remote interviews with 11 teachers from nine different schools of the SEE-SP with the aim of collecting qualitative data related to the bonus program. 12 Table 2 lists the main data of interviewed teachers. 13
Interview Participants.
Notes. Interviews were conducted between August and September 2020. Teachers were randomly assigned to numbers between 1 and 11, and schools to letters between A and I. Experience is expressed in years of teaching and duration of the interview in minutes.
Source. Author’s own data collection, 2020.
The resulting 5.8 hours of recorded material were fully transcribed and then evaluated qualitatively. In order to consolidate the major statements and issues arising from the open-ended questions, I followed Heneman (1998) and applied an analytic framework to transform the personal narratives of the participants into declarative statements. Table 3 reports the main statements that arose from the interviews, and their respective degrees of support among the teachers. 14
Content Analysis of Interviews.
Notes. Data collection was carried out through remote interviews with 11 teachers from the SEE-SP in the municipality of Campinas (see interview guide in the online Supplementary Material). Data analysis using the software f4analyse. ”. IDESP = Education Development Index of the State of São Paulo; SEE-SP = São Paulo state education network.
Source. Author’s own data collection, 2020.
The qualitative interviews provided complementary insights into teacher perceptions of the bonus program. All the interviewees are quite opposed to the bonus program as it is currently structured (see I01 and I02). This low support for the bonus program can be explained by the lack of expectancy and instrumentality, which are—according to expectancy theory (Vroom, 1964)—necessary conditions for being personally motivated to get behind such a scheme. Because the teacher bonus is awarded on the basis of a collective PFP program at school level, the individual effort of a teacher in the classroom will not lead necessarily to successful goal attainment: in this case, student learning. In addition, the bonus program of the SEE-SP violates the assumptions of Locke’s goal-setting theory (Locke, 1968), since the teachers do not consider the program fair (I04) and they have tremendous difficulty in understanding the formula used in the bonus scheme—and so to prove its calculation was indeed carried out correctly (I03).
Table 3 also provides in-depth data validating H2. There seems to be a broad consensus that teachers’ salary in the SEE-SP is very low (I12), and the money from the bonus program is greatly appreciated—but it is too low to play a fundamental role in career plans (I14). Since the bonus amount is proportional to the gross wage of the teachers, which again depends on their years of experience, those at the beginning of their careers generally receive a very modest bonus (I17). Consequently, they do not see it as an incentive to enter into the SEE-SP either (I18).
In relation to work absenteeism (H3), the interviews revealed divided and diverse views. Some teachers believe that the bonus program can help to decrease work absenteeism, while for others it has no impact (I20). The majority of respondents see, however, other labor rights such as seniority leave and remuneration per year of service as more efficient mechanisms to combat absences at work than the bonus scheme (I21).
Referring to H4, respondents indicated that the bonus program had not really changed their teaching practices and behaviors in the classroom (I24). The teachers do not see the bonus program as an incentive for the improvement of their teaching (I25), but see the SARESP as a good instrument for the evaluation of student performance (I06). Based on its results, the teachers can create collectively an action plan to improve student achievement in their schools (I27). This teamwork—which aims at optimizing teaching activities (I28) and, consequently, seeks to improve the learning outcomes—was mentioned by several teachers as an extremely helpful tool for the improvement of teaching practices (I28).
Finally, the qualitative research presented complementary insights into the cheating of teachers regarding the SARESP (H5). Although most of the teachers believe that the cheating vis-à-vis the SARESP creates a bias in the performance results (I33), not all the illicit actions listed in the interview guide occur with the same intensity. According to many respondents, the SARESP structure itself already helps to combat cheating during the test (I40), because no teacher applies the test in the same school as the one that he/she works in. Nevertheless, respondents confirmed three instances of cheating with regard to the SARESP that they had personally witnessed.
A relatively common practice in schools is the distribution of gifts or awards for participation in the SARESP (I34). The second cheating practice reported in the interviews is the use of measures to exclude low-performing students from the SARESP (I35). In this case, these pupils are advised to stay at home on the day of the test or they are temporarily withdrawn from the class some minutes before the SARESP application. Two respondents mentioned during the interviews that they witnessed cases in which the teacher provided the answers for students during the test (I38), but in sequence, they complemented that by saying that, in their view, this cheating may have been an isolated incident since they would experience it only once during their entire professional life. The interviews also found evidence for mock tests (I37) and extra preparation for the SARESP (I36), confirming once again the “teaching to the test” idea.
Another frequently cited consequence of the implementation of the school targets (IDESP) was the shift of the teaching focus overwhelmingly to Math and Portuguese skills (I39). Since only these two subjects are tested by the SARESP, teachers from all other subjects (such as Geography, Biology, Philosophy, and the like) needed to change their programmatic content in order to develop students’ Math and Portuguese skills in an integrated and interdisciplinary manner (I30).
Data Triangulation
Since teacher opposition is the main reason for the failure of merit pay programs (Ballou & Podgursky, 1993), I start by investigating the support among respondents for the PFP program of the SEE-SP. As already described, the quantitative and qualitative research shows a very low level of agreement with the bonus scheme. Respondents seem to have lost trust in the SEE-SP providing a fair wage for its teachers. Therefore, the attempts at reforming the payment structure—as the case of the bonus represents—are perceived as palliative measures that do not resolve the real wage problem. Another reason for the frustration of teachers in relation to the bonus program is the methodology used for its calculation. The teachers know that the IDESP results are not free of bias, and see this pressure for results as a negative for the learning process itself.
H2 was aimed at identifying whether the bonus program has created an additional incentive for staff to enter/continue working in the SEE-SP. As already mentioned, the results from Tables 1 and 3 reject categorically this hypothesis; the response of a 43-year-old teacher helps us to understand why the bonus program plays practically no role in long-term career planning. “The bonus is not official. If the government chooses to stop paying the bonus, it can stop at any time. The bonus is not a labor right like the salary that we know that we will receive every month and the money that we can count on. We never know whether we will receive the bonus or not, and how much it will be.” (ID 7, § 64)
Besides this legal uncertainty regarding the payment scheme, many teachers stated that the values of the bonuses paid have strongly declined in the past few years, thus reducing even further the marginal returns offered by the additional payment. All these factors have contributed to the teacher bonus program not having the motivational potential that it might do. A male teacher who has worked since 2012 in the SEE-SP summarizes the importance of the teacher bonus program as follows: “I am not sure if the bonus has nowadays a relevance to creating a negative or positive impact. I think that it became so outdated in relation to what it idealizes and what it really practices that the teachers became indifferent to this program.” (ID 1, § 63)
It is difficult to imagine that the bonus program created an additional incentive for young teachers to start working in the SEE-SP since, as described in the previous section, the value of the bonus payments to this group is especially low. When asked if the bonus played a role in the decision to become a teacher in state schools, a 34-year-old female respondent said: “The bonus is much less important for me and for all young teachers. I do not count on the bonus, no one does. The bonus is relevant only for teachers with extensive experience in the SEE-SP. To me it does not mean shit.” (ID 2, § 89)
H3 was interested in identifying whether the implementation of the bonus program has had a positive impact on work absenteeism; responses related to it in Quantitative Results and Qualitative Results provide a mixed picture here. There seems to be a consensus that the bonus program did not affect the absence of the respondents themselves, given that they declared having no problem with work absenteeism. However, respondents believed that the bonus program may have had a positive impact on the absence of their work colleagues, especially those with high absenteeism before the scheme’s introduction. This view was confirmed by a 61-year-old teacher with 7 years of experience as an educational coordinator. “Those people that were absent without any reason and were not compromised with the school, they started to avoid, to think better about, their work absenteeism with the introduction of the bonus program.” (ID 5, § 54)
The interviews helped also to clarify the mixed empirical evidence related to H4 found in Quantitative Results. Although some teachers confirmed in the structured questionnaires certain modifications to their teaching since the implementation of the bonus scheme (see Table 1), the interviews made clear that these changes have not resulted from the program itself but rather from the accounting system created with the IDESP. When asked whether the bonus could have provided teachers an additional motivation to put more effort into their teaching practices, a 32-year-old Philosophy teacher remarked: “I don’t feel encouraged to improve my teaching activities due to the bonus alone, and I think that this is also the case with many of my colleagues.” (ID 9, § 70)
Despite this lack of motivating power on the part of the bonus, H4 cannot be entirely rejected—since the teachers confirmed that SARESP results are used to identify learning deficits within the school, and, consequently, to create a better learning plan for the students (see statements I06, I27, and I28 in Table 3). As this work plan is normally created jointly by all educators of the school, teachers can benefit from a spillover effect triggered by the joint discussion of the continuous improvement of learning practices. In describing this association, a teacher with 8 years of experience said: “The only changes—I made—were due to the collective learning plan. When we receive the SARESP results, the Regional Board of Education demands of us an action plan that should be created and implemented by the school unit as a whole.” (ID 1, § 54)
Another key issue related to teaching that arose through the interview process was the focus on Math and Portuguese skills—as already reported in the previous sections with relation to dishonest behavior (H5). Some teachers view this evolution with a highly critical eye, and hold the SARESP mainly responsible for that. As one teacher contended: “For me, is very clear that this focus contributes very little to the general improvement of the academic achievement of students and much more to inflating the school’s performance with regard to the SARESP. Then we [the teachers] receive the bonus, and the government utilizes this improvement as data.” (ID 9, § 29)
This need for generating “good” statistical data was also mentioned in answer to the questions related to other cheating practices vis-à-vis the SARESP. According to many teachers, the IDESP has genuinely created a process of continuous improvement in schools, but this pressure for results is also responsible for the cheating.
As already mentioned in Quantitative Results and Qualitative Results, this research found clear evidence for the notion of educators “teaching to the test.” Most of the teachers highlighted, however, that in theory, at least extra preparation for the SARESP would not be necessary, since the specific knowledge requirements for the SARESP are in line with the programmatic content of the SEE-SP which is communicated to the schools at the beginning of the school year. However, the teachers reported to have difficulties in applying all the programmatic content during the school year, since they have students with very different learning capacities in the same class. This requires extra time for teaching, and consequently, some content runs behind schedule.
Conclusion
This paper applied own data collection and mixed-methods research—combining information from 53 questionnaire respondents and 11 qualitative interviews—to investigate the effects of the implementation of a teacher bonus program on the professional practices and behaviors of teachers in Brazil. Since a sustainable solution to the problem of students’ poor academic performance passes necessarily through the improvement of teaching practices (Woessmann, 2011), this paper has examined the key mechanisms through which the bonus program can generate improved academic achievement.
By and large, the quantitative and qualitative findings highlight the ineffectiveness of the teacher bonus program of the state of São Paulo. This paper has shown that the bonus scheme did not create an additional incentive for teachers to start/continue working in the SEE-SP, and it was not directly responsible for the improvement of teaching activities. The results do point to certain positive effects of the program: namely the reduction of work absenteeism and the monitoring of student achievement. However, both effects are achievable also without the bonus program. As mentioned by the respondents themselves, other programs—such as seniority leave and remuneration per year of service—seem to be more efficient in combating work absenteeism than teacher bonuses. In the same vein, the use of performance tests and their associated targets (respectively, the SARESP and IDESP), do not necessarily have to be connected to the bonus program.
The in-depth interviews set out to explore the reasons for the inefficiency of the program, and delivered compelling results: Although teacher support is an essential condition for the success of any merit pay program implemented in schools (Ballou & Podgursky, 1993), the educators employed in the state schools of São Paulo are strongly against the bonus program of the SEE-SP as it is currently structured. Further, they do not see the bonus scheme as an effective mechanism to improve their teaching practices. Despite some respondents having indicated that they had a favorable attitude toward PFP programs in general, they were unanimous in recognizing that the bonus scheme of the SEE-SP fails to reward individual efforts made in the classroom.
The research project also confirmed for Brazil the existence of negative side effects to the bonus scheme. Respondents indicated that practices related to “teaching to the test,” such as extra preparation and mock tests for the SARESP, have become very common in schools. In addition, they confirmed the existence of cheating vis-à-vis the SARESP, such as the exclusion of low-performing students from the test and the distribution of solutions during the course of it. Although these two practices do not seem to be in widespread use in schools, many teachers were aware of them at least.
In closing, it is important to stress that this research is only a limited study; as with most works, it has certain limitations to it. First, the most obvious shortcoming of this study is the low response rate of the (online) survey. Only 53 of the around 2,000 teachers employed in the state (primary) schools in Campinas answered the questionnaire. Therefore, a higher participation rate would have been desirable. Likewise, only 11 of the 53 contacted teachers agreed to be interviewed. Additional interviews would have been of great value to identify more personal experiences related to the bonus program. Second, the word limitation imposed by academic papers led to only a selection of verbatim quotations to be used in this study. I was very careful here to avoid selection bias since many other interesting statements in relation to the bonus program were made during the 5.8 hr of interviews. Despite all these precautions in processing the responses and the use of theoretical foundations for the choice of the quotations, problems of selection bias cannot be excluded. And, last but not least, I was challenged all the time to isolate my own biases and perspectives from the interpretation of the findings; of course, all my decisions taken may be questionable.
Supplemental Material
sj-docx-1-ref-10.1177_10567879221076082 - Supplemental material for Addressing Changes in Professional Behavior by Teacher Bonuses: A Mixed-Methods Approach
Supplemental material, sj-docx-1-ref-10.1177_10567879221076082 for Addressing Changes in Professional Behavior by Teacher Bonuses: A Mixed-Methods Approach by Tharcisio Leone in International Journal of Educational Reform
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Friedrich-Ebert-Stiftung (grant number 1766180).
Data Accessibility
Supplemental Material
Supplemental material for this article is available online.
Notes
Author Biography
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
