Sage Journals: Discover world-class research

Abstract

Purpose

The purpose of this study was to examine the impact of accountability mechanisms on the social and emotional learning (SEL) competencies of students in schools in Suzhou, China.

Design/Approach/Methods

The study utilized survey data from schools in Suzhou, China, to explore how various inspection schemes and accountability measures differentially influence students’ SEL competencies. The study employed multilevel statistical analysis to examine the relationship between accountability measures and SEL outcomes.

Findings

The study's results suggest that while some accountability measures are associated with improvements in certain SEL skills, most have no effect on or negatively impact SEL competencies.

Originality/Value

This study is one of the few existing studies on the impact of accountability mechanisms on SEL competencies in schools. The findings provide valuable insights for policymakers, organizational scholars, and practitioners. Notably, the findings challenge the assumption that accountability mechanisms necessarily improve educational outcomes and suggest the need for more nuanced approaches to educational accountability policies.

Keywords

Accountability educational policy school reform social and emotional learning

Introduction

School accountability is widely debated in educational research. Some scholars view accountability policies as solutions to improving student outcomes, whereas others warn of potential unintended consequences (Mintrop & Sunderman, 2009). Non-cognitive skills, including social and emotional learning (SEL), have become the focal point of this debate (Bae, 2018). However, there is little evidence on the effects of school accountability on non-cognitive outcomes, such as SEL.

This study investigates the impact of accountability on SEL outcomes using data from the Organisation for Economic Co-operation and Development (OECD) Survey on Social and Emotional Skills (SSES) in Suzhou, China. Having implemented accountability measures for SEL, Suzhou provided a unique opportunity to investigate how various accountability strategies affect the seven domains of SEL. This study aimed to understand how different types of accountability measures, specifically inspection and accountability schemes in Suzhou schools, might either enhance or impede the development of students’ social-emotional competencies.

Literature review

The use of accountability systems to hold educators responsible for student outcomes has become a prevalent approach in educational policies worldwide. Almost all countries and regions have established accountability systems to reassure the public that schools provide quality and equitable education. Over the past decade, students’ SEL competencies have been gradually recognized as key predictors of their life success. Consequently, researchers and policymakers worldwide are exploring the possibility of using accountability systems as a strategy to continuously improve SEL education in schools. In this review, we examine the literature on both school accountability and SEL education.

School accountability policy and SEL

The concept of accountability originated in industrial models. A typical school accountability system comprises four components: (a) information about the organization's performance, (b) standards for judging the quality of organizational performance, (c) significant consequences to the organization, and (d) an agent who judges the extent to which standards have been met and distributes rewards and sanctions (Newmann et al., 1997). Harris (2007) noted that “the idea of accountability for performance has a firm grip on education policy in virtually every industrialized democracy” (p. 41). For example, in the United States, the No Child Left Behind Act (NCLB) of 2002 highlighted the responsibility of states, districts, and schools to improve students’ math and reading achievement (Mandinach et al., 2006). Similarly, in China, while the Gaokao (the higher education entrance examination) can be viewed as a form of educational accountability, a more formalized accountability framework was established in 2012 when the Chinese State Council issued the National Regulations on Educational Inspection. This new regulation provides a government-led mechanism for inspecting educational activities in schools, including direct assessments and evaluations by specialized experts, thus solidifying its role as an official form of educational accountability (Rasmussen & Zou, 2014).

Proponents of accountability contend that accountability policies positively impact student outcomes. Several studies have identified correlations between strict accountability measures and improved academic achievement. For instance, multiple studies comparing student performance before and after the implementation of the NCLB have discovered that proficiency levels have consistently increased in most states (Chudowsky et al., 2007). Furthermore, a plethora of studies have provided strong evidence that stricter state accountability systems correlate with higher student performance gains, even after accounting for various background variables (Braun, 2004; Hanushek & Raymond, 2005; Rosenshine, 2003).

In contrast, an increasing number of researchers have voiced concerns about the real impact of accountability on student achievement (Amrein & Berliner, 2002; Darling-Hammond et al., 2014). As Goodlad (1979) has argued, “the problems and injustices in contemporary approaches to educational accountability stem from the fact that all the richness, shortcomings, successes, and failures of human effort are reduced to a few figures, much as one records profits and losses in a ledger book” (p. 61). Research has demonstrated that teachers in states that do not mandate testing in social studies allocate less time to this subject (Fitchett et al., 2014). Additionally, accountability policies have been found to inhibit the development of high-ability students in low-achieving schools (Krieg, 2008). Sheldon and Biddle (1998) warned that the introduction of performance-based sanctioning systems might have adverse effects on teachers’ behavior and motivation, potentially leading to an authoritarian and detached approach to teaching, demotivation, and job dissatisfaction. This could result in superficial adherence to standards and an increased likelihood of teachers exiting the profession.

In recent years, SEL has been increasingly recognized as a crucial element of education worldwide. For instance, China's newly established quality monitoring framework has incorporated students’ psychological health into one of the six areas to be continually evaluated (Rasmussen & Zou, 2014). Similarly, in the United States, the Every Student Succeeds Act of 2015, which succeeded the NCLB, encourages states to collect SEL indicators for more holistic education. This shift in legislation could be part of a larger trend toward broadening the definition of student success and quality education. However, owing to limited evidence on how accountability systems may intersect with SEL, the use of students’ SEL indicators for school accountability in the United States has remained highly constrained (Mckown, 2017; West et al., 2018).

SEL education in China

Existing literature suggests that SEL can greatly impact students’ educational attainment, general well-being, mental and physical health, civic engagement, and social integration and reduce crime and other anti-social behaviors in society (Furnham et al., 2003; Jensen-Campbell & Malcolm, 2007; Poropat, 2009; Rahafar et al., 2016; Strickhouser et al., 2017). As the global awareness of SEL education grows, multiple international bodies have embraced different SEL frameworks and programs. For example, the Collaborative for Academic, Social, and Emotional Learning is a leading organization in the development of frameworks and standards for evaluating and promoting SEL in school settings.

SEL in China: Mental health education as a proxy

In recent decades, increasing public concern has been raised regarding students’ mental health problems in China (Tang et al., 2018). These issues have been linked to various educational and social problems, such as high school dropouts, bullying, Internet addiction, and issues with left-behind children (H. Huang et al., 2012; X. Huang et al., 2010; Shi et al., 2016; Wang et al., 2015). In response, the Ministry of Education (MOE) of the People's Republic of China established standards for mental health education in schools in 2002; later, the MOE further emphasized the importance of these standards and integrated them into regular inspections in a 2012 amendment.

Therefore, although “SEL” is an umbrella term (Harvard EASEL Lab, 2022) without an official national curriculum in China, many Chinese educators are familiar with it because of its similarities to mental health education, which has been extensively taught in Chinese classrooms for over a decade. Hence, although direct research on SEL development in China may not be available, SEL in China can still be examined using existing work on mental health education. Specifically, because this discipline is seen as China's equivalent to SEL and covers a range of student psychological attributes in the social and emotional realms, we can use it to explore SEL in China within the OECD SSES frameworks.

Recently, China's Office of National Education Inspection (ONEI) added mental health education to the National Compulsory Education Quality Monitoring Program as an accountability measure (MOE, 2021). This initiative, signaling a new wave in China's education accountability, tasks local governments with inspecting mental health development in schools to measure students’ mental health outcomes. It is noteworthy that even before the ONEI announcement, cities such as Suzhou implemented local accountability measures for student mental health based on the 2012 Amendment to the Guidelines. Suzhou introduced a three-tiered system in 2017 and 2018, holding educators at the city, district, and school levels accountable for students’ mental health outcomes. These measures involved school self-reports in March, random external inspections in April, and re-inspections in December (Suzhou Municipal Education Bureau, 2017, 2018).

The SSES project and the OECD's primary findings on SEL in Suzhou

In recent years, the Programme for International Student Assessment (PISA) has become a blockbuster for the OECD. However, it has been criticized for its narrow focus on cognitive areas and for promoting an unhealthy “arms race” among nations. To address this, the OECD launched a new international survey in 2017, the SSES, a cornerstone of the OECD Learning Compass 2030. Suzhou participated in the SSES.

The OECD's initial report on the first round of the SSES in 2021 revealed that the intraclass correlation coefficient (ICC) for Suzhou was higher than that for the other participating cities (OECD, 2021). In Suzhou, the variation observed in elementary student SEL outcomes at the school level (5.2%) was twice as high as the average variation of other participating cities (2.5%). In contrast, a study by Wu et al. (2020), using PISA results, found that 21% of the total variance in student science achievement was attributed to differences between schools. This suggests that (a) schools are less effective in promoting SEL than cognitive skills and (b) the effect of schooling on SEL is not random—there may be more significant differences in how schools in Suzhou shape their students’ SEL outcomes than in how other cities do the same. Therefore, further investigation into how Suzhou's school-level policies and practices impact student SEL outcomes is warranted.

Challenges of incorporating SEL into school accountability frameworks

The rise of SEL education could signal a paradigm shift in the educational system, introducing new challenges for teaching and learning and necessitating innovation in educational administration and leadership. Should we allow past experiences in cognitive domains to guide our current approach to SEL domains? In the United States, while an increasing number of states have incorporated SEL components into their learning standards, few have linked student SEL outcomes to high-stakes accountability systems (Mckown, 2017). It is also challenging for school leaders to integrate SEL into existing school structures, routines, and practices (Bailey & Weiner, 2022). There is an urgent need to transform accountability systems to accommodate SEL.

Melnick et al. (2017) raised key questions for researchers and policymakers regarding the incorporation of SEL into current accountability systems, including: What does the indicator measure? Can the data lead to significant improvements? How can we effectively incorporate SEL into school accountability systems? What was the measure designed for, and how might data be distorted if attached to high-stakes consequences? Based on their review, Melnick et al. (2017) advised against using measures of students’ social-emotional competencies in accountability systems, at least for now, and suggested that decision makers at the state, district, and local levels need to collaborate to build a system that focuses on continually informing schools and teachers to improve SEL programs or curricula.

Melnick et al. (2017) provided a valuable perspective on the cautious application of current accountability models to student SEL outcomes. However, their recommendation is theoretical and based on the United States’ practices and lacks empirical evidence to guide the transformation of accountability systems for SEL education in other countries. This study aimed to address this gap by offering new insights based on the Suzhou case study.

Conceptual framework and research questions

In this study, we explored the compatibility between SEL education and accountability systems by investigating whether current accountability systems can benefit or hinder SEL education. Our research draws upon two areas of the literature: SEL education and school accountability systems. While most accountability systems focus on academic performance, the importance of SEL is often overlooked and undervalued. Although some studies suggest that holding educators accountable can effectively improve student cognitive outcomes, it is unclear whether accountability can enhance student SEL performance. Fortunately, Suzhou has implemented accountability systems in its schools, providing a valuable case study to investigate the relationship between accountability and student SEL outcomes.

Our study aims to answer two research questions: First, to what extent does the type of school assessment of student SEL achievement (e.g., formal assessment, informal assessment, no assessment) affect differentiated student SEL performance? Second, to what extent does the use of inspection results (e.g., internal accountability and external accountability) affect differentiated student SEL performance? These questions are central to understanding how inspection and accountability schemes in school accountability systems can improve students’ SEL outcomes.

Methodology

Data source and sample

The target population for this study was students from schools in Suzhou, China. A stratified two-stage cluster sampling scheme was employed for the SSES. In the first stage, 76 elementary schools were selected randomly from a pool of 387. Subsequently, 3800 elementary students were sampled from the selected schools.

This study focused on a younger cohort (10-year-old elementary school students) for two reasons. First, high school education in China is not part of the compulsory education system, resulting in fewer enforced inspections and accountability measures. Second, the 2012 Amendment to the Guidelines does not provide as detailed skill standards for middle and high school students as it does for elementary students. In simpler terms, the guidelines specify the skills expected of a 10-year-old student, but not of a 15-year-old student.

To include both school-level and student-level variables in the multilevel models, we matched student data with school data based on their cohort ID and school ID from the SSES student and principal questionnaires. After matching, 3581 students from 75 elementary schools were included.

Variables

Outcome variables

As previously mentioned, one of the challenges of this study was that domestic mental health education did not fully align with the SSES framework. To accurately assess the impact of the accountability measure, we first needed to correlate the SEL skills taught in Chinese mental health education with the SEL skills from the SSES framework. As the accountability measure in Suzhou is based on the 2012 Amendment to the Guidelines (MOE, 2012), we examined detailed descriptions of skills/behaviors in the 2012 guidelines and compared them with the detailed descriptions of skills/behaviors in the SSES framework.

One advantage of the 2012 Amendment to the Guidelines is that, at the elementary level, it offers detailed descriptions of teaching expectations for each age group. For instance, first and second graders are expected to develop a sense of safety, self-control, self-discipline, and friendship and time-management skills. Third graders are expected to develop self-awareness, confidence, collectivism, sociability, and persistence skills. As the SSES targets 10-year-old students, who are predominantly in the third grade, we narrowed our focus to skills from the first to third grade according to the 2012 guidelines.

By comparing the skills described in the SSES framework with those in the 2012 Amendment to the Guidelines, we identified seven SEL skills that were measured by the SSES and inspected in the Suzhou accountability system: persistence, self-control, emotional control, empathy, cooperation, sociability, and achievement motivation. The OECD calculated the scores for these SEL areas for each student based on their responses to the questionnaire. Thus, for all outcome variables that evaluated student performance in the above seven areas, we directly used the OECD scores calculated from the SSES student data.

Key predictors

To investigate the impact of inspection schemes on student SEL outcomes, we selected the item PRQM01501 from the SSES school data. This question asked the principals whether their school evaluated students’ achievement in social and emotional skills (1 = No, we did not evaluate these skills; 2 = Yes, we used informal evaluations; 3 = Yes, we used formal evaluations). We further dummy-coded this variable, using “we did not evaluate these skills” as the reference group.

To examine the effect of accountability schemes on student SEL outcomes, we selected the items PRQM01601 and PRQM01602. These questions asked whether the evaluation of students’ achievement in social and emotional skills was used for (a) an internal evaluation of school performance and (b) an external evaluation of school performance. Responses to the two variables were binary (0 = No; 1 = Yes).

Control variables

For student demographics, we extracted five variables from the SSES data: gender (female), age (in months), academic performance in mathematics, academic performance in reading, and socioeconomic status (SES; a composite score created by the OECD).

In terms of school and principal demographics, we drew school and principal background variables from the SSES principal data. These included the age and gender of the principal, teacher–student ratio, percentage of students from socioeconomically disadvantaged homes, and school size.

Finally, we applied weights at both the student and school levels using the student weight variable, WT19, and the school weight variable, WT19PR. This methodology ensured that our sample of students and schools was representative of the target population of Suzhou (Muthén & Muthén, 2017). As our primary focus was on Level 2 predictors, we centered all variables at the grand mean in all statistical models (Enders & Tofighi, 2007).

Statistical procedure

The hierarchical structure of our dataset necessitates the use of multilevel modeling. As presented in Table 1, the ICCs for the seven SEL areas ranged from 4.1% to 6.5%. According to LeBreton and Senter (2008), an ICC of 0.05 represents a medium effect; therefore, in our study, the ICCs indicated moderate to large effects.

Table 1.

ICCs of seven SEL areas.

	Persistence	Self-control	Emotional control	Empathy	Cooperation	Sociability	Motivation
ICC	5.3%	5.7%	4.1%	5.8%	6.5%	4.4%	5.6%

In our model, students represent the first level and schools form the second level. We opted not to include the teacher level because, in the sampling stage, students were not strictly nested within a specific classroom or under a specific teacher. The algorithm was integrated with the maximum likelihood with robust standard errors estimation. The final model was expressed as follows:

S t u d e n t l e v e l : (S E L)_{i j} = β_{0 j} + \sum_{n = 1 k j}^{5} β_{k j} X_{k i j} + r_{i j} r_{i j} \sim N (0, σ^{2}) .

S c h o o l l e v e l : β_{0 j} = γ_{00} + \sum_{k = 1}^{5} γ_{0 k} W_{k i j} + γ_{6 j} F o r m_{i j} + γ_{7 j} I n f o r m_{i j} + u_{0 j} μ_{0 j} \sim N (0, τ_{00})

For the student-level equation, SEL _ij is student i's SEL scores in each of the seven areas in school j; β_0j is the average SEL score; X_kij is the kth student demographic variable, including gender (girl), age, performance in mathematics, performance in reading, and SES. r_ij represents the student-level random effect with variance σ².

As for the school level equation, γ₀₀ is the grand mean of student SEL scores in each of seven areas, W_kj represents the kth school-level covariates, including the age of the principal, the gender of the principal, the teacher–student ratio, the percent of students from socioeconomically disadvantaged homes, and school size. γ_0k is the effect of the school background on student average SEL outcomes. γ_6j and γ_7j represent the effects of formal and informal inspections, respectively. u_0j represents the school-level random effect with variance τ₀₀_.

Similar modeling schemes have been used to evaluate different types of accountability. The full school-level model is expressed as follows:

S c h o o l l e v e l : β_{0 j} = γ_{0 j} + \sum_{k = 1}^{5} γ_{0 k} W_{k i j} + γ_{6 j} I n t_{i j} + γ_{7 j} E x t_{i j} + u_{0 j}

μ_{0 j} \sim N (0, τ_{00})

where γ_6j ∼ γ_7j represent the effects of using students’ SEL achievements for internal and external school evaluations, respectively.

Findings

Descriptive statistics

According to Suzhou's policy documents, inspections are conducted in three ways: school self-reports, random external inspections, and external re-inspections. Logically, these three forms led to different school SEL inspection schemes (Table 2). Specifically, 25.33% of schools did not inspect students’ SEL performance, 57.33% utilized informal inspections, and 17.33% employed formal inspections. In terms of accountability schemes, among schools that implement formal inspection schemes, 61.54% use inspection results for internal school evaluations, while 53.85% use the inspection results for external school evaluations. Table 2 also shows that the correlation between external and internal school evaluations is small and statistically insignificant.

Table 2.

Descriptive statistics of inspection schemes and accountability schemes.

	Variable	N	Percent	1	2
Inspection scheme	Non-inspection	19	25.33
	Informal inspection	43	57.33
	Formal inspection	13	17.33
Accountability scheme	Internal school	8	61.54	1
Accountability scheme	External school	7	53.85	0.220	1

Note. *** p < .001; ** .001 < p < .01; * .01 < p < .05.

The effect of school SEL inspection schemes on student SEL

Table 3 presents the results of the impacts of the three types of inspection schemes (non-inspection, informal inspection, and formal inspection) on the SEL performance of elementary students. Contrary to common perceptions, the findings suggest that there are no significant differences in most SEL areas between students from schools with informal inspections and those without inspections or between students from schools with formal inspections and those without inspections. However, emotional control is an exception.

Table 3.

Impact of school inspection schemes on student SEL (formal inspection vs. informal inspection vs. non-inspection).

	Persistence		Self-control		Emotional control		Empathy		Cooperation		Sociability		Motivation
	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
Student level
Gender (female)	4.37	3.89	2.79	3.45	−6.78	4.22	17.56***	4.02	12.69**	4.25	3.87	3.76	5.16	3.25
Age	−4.50	6.76	3.28	5.17	−8.96	7.22	−8.79	5.88	−4.02	5.04	2.12	5.93	−7.04	5.55
Mathematics	0.77***	0.20	0.59**	0.19	0.27	0.21	0.30	0.20	0.34	0.19	0.42**	0.16	0.60**	0.18
Reading	0.68**	0.20	0.68**	0.20	0.59**	0.22	0.78***	0.21	0.73***	0.20	0.93***	0.20	1.05***	0.18
SES	16.68**	2.42	15.66***	2.43	15.35***	3.21	16.85***	2.50	15.38***	2.57	16.66***	2.50	15.47***	2.15
School level
Percent of students with low SES	−19.57	11.42	−16.71	11.99	−7.29	15.02	−12.89	12.81	−11.31	15.49	−8.03	9.55	−12.12	11.58
Principal (female)	−4.76	6.37	−2.06	5.86	−5.31	7.59	−4.38	7.01	−8.67	7.14	−8.21	5.55	−3.73	5.62
School size	0.34	0.22	0.36	0.19	0.04	0.27	0.28	0.25	0.42	0.22	0.18	0.20	0.27	0.21
Principal age	1.62**	0.58	0.81	0.53	1.53*	0.72	1.64*	0.67	1.55*	0.63	2.13***	0.53	1.72**	0.53
Teacher–student ratio	−1.61	0.95	−1.46	0.82	−0.87	1.00	−1.55	0.83	−1.53	0.85	−0.99	0.69	−1.44	0.79
Informal inspection	−6.38	6.86	−5.05	6.21	−9.69	9.24	−6.45	8.03	−3.94	7.98	−3.08	6.34	−5.52	6.51
Formal inspection	−5.18	9.84	−6.86	10.11	−22.49*	10.60	−7.13	10.35	−4.57	11.72	3.83	8.92	−5.48	8.64
Intercepts	651.38***	2.88	613.48***	2.71	596.33***	3.36	675.49***	3.12	675.28***	3.25	637.63***	2.54	632.45	2.57

Note. *** p < .001; ** .001 < p < .01; * .01 < p < .05.

As shown in Table 3, students from schools with informal evaluations performed worse on average than students from schools without evaluations across all seven SEL areas. However, these differences were not statistically significant. Similarly, students from schools with formal evaluations performed worse on average across all seven SEL areas. Notably, students from schools with formal evaluations significantly underperformed compared to students from schools without evaluations by 22.49 points (β= −22.49, p < .05) in the emotional control area.

Consistent with previous research, at the student level, both student performance in reading and student SES were positively associated with student performance in all seven SEL areas. At the school level, the principal's age appeared to be positively related to the school's average performance in SEL, with the exception of self-control.

The effect of school SEL accountability schemes on student SEL

Next, we investigated the impact of two types of accountability schemes (i.e., internal and external evaluations of school performance) on elementary students’ SEL performance. Our findings (Table 4) reveal that the use of inspections for the internal evaluation of school performance was negatively associated with a school's average performance across all seven areas of SEL. Specifically, schools that use internal evaluations may significantly reduce student performance in terms of self-control, sociability, and achievement motivation by 33.69, 39.72, and 23.39 points, respectively.

Table 4.

Impact of school accountability schemes on student SEL (using inspection for external school evaluation vs. for internal school evaluation).

	Persistence		Self-control		Emotional control		Empathy		Cooperation		Sociability		Motivation
	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE	Estimate	SE
Student level
Gender (female)	4.62	9.16	11.91	9.65	−8.85	10.49	22.33**	8.97	22.36**	8.97	4.62	9.16	10.92	6.77
Age	16.50	6.26	−15.65	9.85	−33.03***	10.26	−16.21	11.57	6.17	12.16	16.50**	6.26	−24.58	13.42
Mathematics	0.55	0.45	0.41	0.24	−0.51	0.35	−0.37	0.36	−0.16	0.31	0.55	0.45	0.41	0.27
Reading	0.94	0.64	0.16	0.45	1.05	0.56	1.11*	0.56	0.60	0.53	0.94	0.64	0.53	0.45
SES	12.27	8.20	14.754***	4.23	5.33	6.34	13.14	7.56	12.24	8.41	12.27	8.20	11.83*	5.77
School level
Percent of students with low SES	−5.45	31.43	−87.02***	25.36	1.80	20.14	−22.04	39.40	−24.15	42.52	−5.45	31.43	−37.48	26.49
Principal (female)	−15.08	21.81	35.79**	16.34	1.90	12.40	4.91	24.61	−7.02	27.68	−15.08	21.81	7.71	17.62
School size	1.24	1.30	−1.34	1.34	2.54*	1.13	1.00	2.01	0.49	2.22	1.24	1.30	0.28	1.31
Principal age	1.74	4.47	−8.41**	3.14	−0.10	2.65	−1.46	5.21	0.44	5.53	1.74	4.47	−1.39	2.98
Teacher–student ratio	0.05	4.63	−13.36***	4.51	−0.84	4.19	−4.54	6.86	−5.44	8.02	0.05	4.63	−6.67	4.42
Internal school	−28.72	11.73	−33.69**	11.87	−10.11	9.86	−13.04	18.34	−39.72*	20.40	−28.72**	11.73	−23.39*	11.76
External school	25.17	25.40	65.94***	17.34	21.95	14.46	27.82	29.79	36.75	29.82	25.17	25.40	37.60*	17.73
Intercepts	639.28***	4.90	607.15***	4.22	580.51***	3.79	669.23***	6.31	670.31***	7.14	639.28***	4.90	627.24***	4.26

Note. *** p < .001; ** .001 < p < .01; * .01 < p < .05.

Conversely, inspections of the external evaluation of school performance were positively associated with a school's average performance across all seven areas of SEL. Notably, schools employing external evaluations may significantly enhance students’ performance in emotional control and achievement motivation by 65.94 and 37.60 points, respectively.

Discussion

SEL education has emerged as a key theme in education worldwide. In this study, we empirically investigated the relationship between student SEL performance and school accountability systems using recent data from a large-scale OECD survey. Given the overlap between the SSES framework and the Chinese mental health education framework in areas such as persistence, self-control, emotional control, empathy, cooperation, sociability, and achievement motivation, we explored two key questions related to school accountability systems: (a) How should schools/educators be inspected to enhance student SEL? (b) Who should be held accountable for the inspection results? Our findings offer new evidence and policy implications for researchers and policymakers considering the integration of SEL with current accountability systems.

Inspection schemes

Formal inspections play a vital role in evaluating cognitive areas, including reading, mathematics, and science. However, the question remains as to whether they are equally indispensable in non-cognitive areas. Furthermore, the extent to which existing knowledge and experience garnered from inspections of cognitive domains are translatable and applicable to inspections of non-cognitive areas remains ambiguous. In this study, we compared three types of inspection schemes: non-inspection, informal inspection, and formal inspection. Our findings demonstrate that, for the most part, Suzhou elementary schools either give up inspections altogether (27.9%) or use informal inspections (56.5%); only a minority of schools conduct formal inspections (15.6%). Surprisingly, our findings show that formal inspections may not necessarily improve student performance in six SEL areas when compared to non-inspection schemes. Moreover, formal inspections may have an unfavorable impact on students’ emotional control.

These results address the first question asked by Melnick et al. (2017) regarding the measures and efficiency of the data, leading to a meaningful improvement. Our study indicates that most schools in Suzhou either do not utilize indicators (non-inspection) or rely solely on arbitrary subjective measures (informal inspection) to evaluate students’ SEL development. Consequently, the measurement of SEL in Suzhou schools lacks objective criteria. However, despite teachers and principals relying on personal perceptions, preferences, or other non-quantifiable methods to assess student SEL development, these do not seem to result in inferior student SEL outcomes.

A causal relationship between formal school inspections and student SEL outcomes could not be directly inferred from our data and analyses. Nonetheless, it is pertinent to explore why schools with formal inspections are unable to yield better student SEL outcomes than those without. A plausible explanation for this may lie in the technical differences in the measurement of SEL from those of academic subjects such as science, mathematics, and reading. SEL is a complex system comprising multiple layers and components, and there is no existing evaluation system for it comparable to standardized tests for academic ability. Schools with or without formal SEL inspections have adequate resources and the capacity to develop a scientifically sound assessment system for SEL at the school level. Consequently, teachers and school administrators cannot rely on assessment data to develop effective evidence-based programs to improve student SEL. This limitation curtails schools’ efforts to improve their students’ social and emotional well-being.

Accountability schemes

The second and third questions raised by Melnick et al. (2017) revolve around how SEL can be fully integrated into school accountability systems and whether data may be distorted if associated with high-stakes consequences. Our study of Suzhou's accountability schemes indicates that when inspection data are utilized for internal accountability, there is a significant decrease in nearly all categories of student SEL outcomes, particularly self-control, sociability, and achievement motivation. Conversely, when inspection data are employed for external accountability, there is a significant increase in all areas of student SEL, particularly self-control and achievement motivation.

According to many organizational theorists, it is worth noting that internal accountability is independent of external accountability. Newmann et al.'s (1997) seminal work on internal and external accountability asserts that internal accountability can exist with or even in opposition to external accountability. Our results corroborate this claim, as we found that internal accountability produced contrasting outcomes for student SEL compared to external accountability. Contrary to conventional beliefs, our findings highlight the critical role of external accountability in enhancing students’ SEL outcomes.

Why does an increase in schools’ commitment toward student SEL potentially result in decreased student SEL outcomes, whereas increased district controls improve student SEL outcomes? This can be explained from two theoretical perspectives.

The first theory focuses on the relationship between organizational innovation and accountability. Accountability systems are typically self-reinforced. Instead of preventing the diffusion of incorrect practices and knowledge, internal accountability can strengthen such practices. Over the past few decades, we have witnessed a trend toward increasing standardization in schools’ everyday work. This trend has led to a relative decrease in teachers’ autonomy and innovation, thereby justifying and stimulating further standardization across all fields of education, including SEL. However, SEL education represents a paradigm shift from academically focused education. Past knowledge and experience from academically focused education cannot be applied directly to SEL education. Therefore, even if a school has high internal accountability, it may be heading in the wrong direction because of misleading signals from an immature and misaligned accountability system.

Our second theoretical explanation emphasizes the institutional nature of school organizations. It is important to acknowledge that school administrators can potentially misjudge or unduly influence teachers’ work. However, it is equally crucial to note that individual teachers are not direct targets of district-level controls. Furthermore, external accountability pressures from the district are often absorbed at the school level and rarely affect individual teachers. As observed by Qian and Walker (2019), Chinese school principals are tasked with maintaining a delicate balance between navigating politics, upholding professional expertise, and fostering sincere relationships with teachers.

Furthermore, Peter Senge (2006, p. 19) posited, “When people in organizations focus solely on their position, they often lack a sense of responsibility for the outcomes produced by the interaction of all positions” (p. 19). Given that individual teachers are not subject to district controls, they might exhibit fewer reservations about collaborating toward a common objective, such as a school-level SEL initiative. As a result, district external accountability could stimulate increased cooperation and collegiality among teachers, which is vital for nurturing SEL education.

Conclusion

Over four decades, discourse on school accountability has yielded diverse views. Some scholars propose that accountability can enhance educational systems, whereas others warn of its potential pitfalls. The advent of SEL has introduced a new and significant dimension to this debate. In this study, we used data from the OECD SSES to examine the effects of accountability on student SEL outcomes. Our preliminary findings suggest that accountability systems do not better predict students’ SEL outcomes in Suzhou. This study contributes new evidence and perspectives to existing knowledge on school accountability, offering implications for researchers, policymakers, and practitioners.

First, SEL education prompts a rethinking of the mechanism of school accountability, necessitating considerable innovation and reform in school accountability measurement. Traditional systems rely on easily quantifiable data, such as test scores and graduation rates. However, SEL's multifaceted and multilayered nature means that the existing one-size-fits-all solutions from traditional accountability systems may not be suitable. The added complexity and challenges call for educational researchers to devise a more comprehensive yet personalized approach for evaluating student SEL development. This approach should include a well-aligned and practical framework for SEL measurement; diverse assessments with multiple formative evaluation points; diverse data sources, including family and community inputs; and professional SEL education training for teachers.

Second, policymakers must understand that attaching high-stakes consequences to students’ SEL outcomes is inappropriate. Penalizing teachers or schools for poor student SEL outcomes could lead to confusion and demoralization. Instead, policymakers might consider deeply integrating SEL with existing standards, including professional teachers and curriculum/program development standards. Additionally, policymakers should allocate more economic and political resources to SEL research and program development and support school SEL networks and school-community partnership programs.

Third, as SEL is a key competency in the 21st century, teachers and staff must update their mental models. School practitioners should shift from an academically focused mindset to a whole-child mindset, recognizing that student SEL is as important as academic development. Teaching SEL is a long-term commitment that requires consistent effort and ongoing dedication from teachers through both direct and indirect educational channels. School leaders should empower teachers; foster a learning atmosphere for SEL; and implement democratic decision-making processes involving teachers, parents, and communities.

Our study has three limitations that underscore areas for future research. First, the sample from the SSES 2019 international survey was not customized for the Suzhou school system. Second, although the 2012 Amendment to the Guidelines identified SEL areas for 3rd-grade students, no official documents explicitly articulated the consequences for schools or teachers who failed to meet these standards. For instance, the implications for Suzhou schools and teachers who do not meet the standards in the December re-inspections remain unclear. Consequently, our study was limited in its ability to correlate the level of accountability with student SEL outcomes. Third, owing to the scope of this study, the limitations of the dataset, and its explorative nature, we did not include parent- or teacher-level variables in our analysis. We recommend that future research be more customized, comprehensive, or qualitative, and that these factors be included at the school, family, and teacher levels.

Footnotes

Contributorship

Xingyuan Gao, Qi Lin, and Jiangang Xia contributed to manuscript development and writing. Xingyuan Gao was responsible for writing the abstract and bulk of the main body, finalizing the paper, and responding to the reviewers’ comments. Qi Lin and Jiangang Xia contributed to various aspects of the paper, including theorizing and analyzing the datasets. All authors worked collaboratively throughout the writing process and provided feedback and suggestions to arrive at a final manuscript that was both comprehensive and coherent.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical statement

The present study is based on an analysis of an international database by the OECD, and no primary data were collected from human subjects. The database used in this study has undergone an ethical review and is publicly available. According to the OECD guidelines, all participants included in this database provided informed consent. The researchers had no access to any information that could identify the individual participants during or after data collection.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Office for Education Sciences Planning (NOESP) (grant number DAA220427).

ORCID iDs

Xingyuan Gao

Qi Lin

References

Amrein

A. L.

Berliner

D. C.

(2002). High-stakes testing, uncertainty, and student learning. Education Policy Analysis Archives, 10(18). https://doi.org/10.14507/epaa.v10n18.2002

Bae

(2018). Redesigning systems of school accountability: A multiple measures approach to accountability and support. Education Policy Analysis Archives, 26(8). https://doi.org/10.14507/epaa.26.2920

Bailey

Weiner

(2022). Interpreting social-emotional learning: How school leaders make sense of SEL skills for themselves and others. School Leadership Review, 16(2), 4.

Braun

(2004). Reconsidering the impact of high-stakes testing. Education Policy Analysis Archives, 12(1). https://doi.org/10.14507/epaa.v12n1.2004

Chudowsky

Kober

(2007). Answering the question that matters most: Has student achievement increased since No Child Left Behind? Center on Educational Policy.

Darling-Hammond

Wilhoit

Pittenger

(2014). Accountability for college and career readiness: Developing a new paradigm. Education Policy Analysis Archives, 22(86). https://doi.org/10.14507/epaa.v22n86.2014

Enders

C. K.

Tofighi

(2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138. https://doi.org/10.1037/1082-989X.12.2.121

Fitchett

P. G.

Heafner

T. L.

VanFossen

(2014). An analysis of time prioritization for social studies in elementary school classrooms. Journal of Curriculum and Instruction, 8(2), 7–35. https://doi.org/10.3776/joci.2014.v8n2p7-35

Furnham

Chamorro-Premuzic

McDougall

(2003). Personality, cognitive ability, and beliefs about intelligence as predictors of academic performance. Learning and Individual Differences, 14(1), 47–64. https://doi.org/10.1016/j.lindif.2003.08.002

10.

Goodlad

J. I.

(1979). What schools are for. Phi Delta Kappa Educational Foundation.

11.

Hanushek

E. A.

Raymond

M. E.

(2005). Does school accountability lead to improved student performance? Journal of Policy Analysis and Management, 24(2), 297–327. https://doi.org/10.1002/pam.20091

12.

Harris

(2007). The current crisis in leadership: Threat or opportunity. School Leadership and Management, 27(2), 105–107. https://doi.org/10.1080/13632430701237065

13.

Harvard EASEL Lab. (2022). Taxonomy project. https://easel.gse.harvard.edu/taxonomy-project

14.

Huang

Hong

J. S.

Espelage

D. L.

(2012). Understanding factors associated with bullying and peer victimization in Chinese schools within ecological contexts. Journal of Child and Family Studies, 22(7), 881–892. https://doi.org/10.1007/s10826-012-9647-4

15.

Huang

Zhang

Wang

Zhang

Tao

(2010). Mental health, personality, and parental rearing styles of adolescents with Internet addiction disorder. Cyberpsychology Behavior and Social Networking, 13(4), 401–406. https://doi.org/10.1089/cyber.2009.0222

16.

Jensen-Campbell

L. A.

Malcolm

K. T.

(2007). The importance of conscientiousness in adolescent interpersonal relationships. Personality and Social Psychology Bulletin, 33(3), 368–383. https://doi.org/10.1177/0146167206296104

17.

Krieg

J. M.

(2008). Are students left behind? The distributional effects of the No Child Left Behind Act. Education Finance and Policy, 3(2), 250–281. https://doi.org/10.1162/edfp.2008.3.2.250

18.

LeBreton

J. M.

Senter

J. L.

(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642

19.

Mandinach

Honey

Light

Brunner

(2006). An exploration of educator’s uses of data in assessment for learning. Journal of Educational Computing Research, 35(2), 215–243. https://doi.org/10.2190/YHH7-HNWA-W1KG-LNWE

20.

McKown

(2017). Social-emotional assessment, performance, and standards. The Future of Children, 27(1), 157–178. https://doi.org/10.1353/foc.2017.0008

21.

Melnick

Cook-Harvey

C. M.

Darling-Hammond

(2017). Encouraging social and emotional learning in the context of new accountability. Learning Policy Institute.

22.

Ministry of Education of the People’s Republic of China. (2012). Amendment to the guidelines for mental health education in primary and secondary schools [in Chinese]. http://www.moe.gov.cn/srcsite/A06/s3325/201212/t20121211_145679.html

23.

Ministry of Education of the People’s Republic of China. (2021). Amendment to the National Compulsory Education Quality Monitoring Program [in Chinese]. http://www.moe.gov.cn/srcsite/A11/moe_1789/202109/t20210926_567095.html

24.

Mintrop

Sunderman

G. L.

(2009). Predictable failure of federal sanctions-driven accountability from school improvement—and why we may retain it anyway. Educational Researcher, 38(5), 353–364. https://doi.org/10.3102/0013189X09339055

25.

Muthén

L. K.

Muthén

B. O.

(2017). Mplus: Statistical analysis with latent variables: User’s guide (version 8). Muthén & Muthén.

26.

Newmann

King

M. B.

Rigdon

(1997). Accountability and school performance: Implications from restructuring schools. Harvard Educational Review, 67(1), 41–75. https://doi.org/10.17763/haer.67.1.14141916116656q6

27.

OECD. (2021). Beyond academic learning: First results from the Survey of Social and Emotional Skills. OECD Publishing. https://doi.org/10.1787/92a11084-en

28.

Poropat

A. E.

(2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322. https://doi.org/10.1037/a0014996

29.

Qian

Walker

(2019). Reconciling top-down policy intent with internal accountability: The role of Chinese school principals. Educational Assessment, Evaluation and Accountability, 31(4), 495–517. https://doi.org/10.1007/s11092-019-09309-4

30.

Rahafar

Maghsudloo

Farhangnia

Vollmer

Randler

(2016). The role of chronotype, gender, test anxiety, and conscientiousness in academic achievement of high school students. Chronobiology International, 33(1), 1–9. https://doi.org/10.3109/07420528.2015.1107084

31.

Rasmussen

Zou

(2014). The development of educational accountability in China and Denmark. Education Policy Analysis Archives, 22(121). https://doi.org/10.14507/epaa.v22.1693

32.

Rosenshine

(2003). High-stakes testing: Another analysis. Education Policy Analysis Archives, 11(24). https://doi.org/10.14507/epaa.v11n24.2003

33.

Senge

(2006). The fifth discipline: The art and practice of the learning organization. Doubleday.

34.

Sheldon

Biddle

(1998). Standards, accountability, and school reform: Perils and pitfalls. Teachers College Record, 100(1), 164–180. https://doi.org/10.1177/016146819810000110

35.

Shi

Bai

Shen

Kenny

Rozelle

(2016). Effects of parental migration on mental health of left-behind children: Evidence from Northwestern China. China & World Economy, 24(3), 105–122. https://doi.org/10.1111/cwe.12161

36.

Strickhouser

J. E.

Zell

Krizan

(2017). Does personality predict health and well-being? A metasynthesis. Health Psychology, 36(8), 797. https://doi.org/10.1037/hea0000475

37.

Suzhou Municipal Education Bureau. (2017). Suggestions on strengthening student mental health education [in Chinese]. https://www.suzhou.gov.cn/szsrmzf/bmwj/201701/HUL6YSY8BHQ4OABBBOK1XH1GANVUK65K.shtml

38.

Suzhou Municipal Education Bureau. (2018). Notice on special supervision of mental health education in primary and secondary schools of the city [in Chinese]. http://www.suzhou.gov.cn/szsrmzf/bdwglywwjq/201803/BNCC3MXDRTNH07FDRLR6NZ5YS740IQWE.shtml

39.

Tang

Wang

Dai

Yang

(2018). Mental health and psychosocial problems among Chinese left-behind children: A cross-sectional comparative study. Journal of Affective Disorders, 241, 133–141. https://doi.org/10.1016/j.jad.2018.08.017

40.

Wang

Yang

Shi

Rozelle

Chu

(2015). Mental health and dropout behavior: A cross-sectional study of junior high students in northwest rural China. International Journal of Educational Development, 41, 1–12. https://doi.org/10.1016/j.ijedudev.2014.12.005

41.

West

M. R.

Buckley

Krachman

S. B.

Bookman

(2018). Development and implementation of student social-emotional surveys in the CORE districts. Journal of Applied Developmental Psychology, 55(July 2017), 119–129. https://doi.org/10.1016/j.appdev.2017.06.001

42.

Gao

Shen

(2020). Principal leadership effects on student achievement: A multilevel analysis using Programme for International Student Assessment 2015 data. Educational Studies, 46(3), 316–336. https://doi.org/10.1080/03055698.2019.1584853

School Accountability and Student Social-Emotional Learning: Evidence From the OECD's Survey on Social and Emotional Skills

Abstract

Purpose

Design/Approach/Methods

Findings

Originality/Value

Keywords

Introduction

Literature review

School accountability policy and SEL

SEL education in China

SEL in China: Mental health education as a proxy

The SSES project and the OECD's primary findings on SEL in Suzhou

Challenges of incorporating SEL into school accountability frameworks

Conceptual framework and research questions

Methodology

Data source and sample

Variables

Outcome variables

Key predictors

Control variables

Statistical procedure

Findings

Descriptive statistics

The effect of school SEL inspection schemes on student SEL

The effect of school SEL accountability schemes on student SEL

Discussion

Inspection schemes

Accountability schemes

Conclusion

Footnotes

Contributorship

Declaration of conflicting interests

Ethical statement

Funding

ORCID iDs

References