Are Changes in Reported Social-Emotional Skills Just Noise? The Predictive Power of Longitudinal Differences in Self-Reports

Abstract

Prior work has shown that students’ self-reported levels on social-emotional measures predict achievement levels and gains, but we have little evidence on whether within-student changes in self-reports on social-emotional surveys are predictive of changes in theoretically related academic and behavioral outcomes. We use large-scale data from the California CORE districts to examine whether changes in individual reports on social-emotional measures from one school year to the next predict changes in state math and English language arts test scores and attendance. We show that changes in self-reported social-emotional measures predict changes in both achievement and attendance. These results are robust across model specifications. Moreover, the relationships between SEL and achievement and attendance outcomes are consistent across student subgroups.

Keywords

social-emotional learning student surveys measurement achievement attendance

Introduction

Recent evidence shows that social-emotional learning (SEL) is strongly predictive of educational and career success, even after controlling for differences in students’ experiences and achievement (Deming, 2017; Duckworth et al., 2010; Farrington et al., 2012; Kautz et al., 2014; Yeager & Dweck, 2012). As a consequence, school systems are increasingly seeking ways to measure students’ SEL. The most common approach to measurement, given its administrative ease and low cost, is the use of self-report student surveys. Yet questions remain about the validity of using students’ responses to these surveys to track their social-emotional development.

It is well documented that student self-reports on SEL surveys do predict theoretically related academic and behavioral outcomes (Claro & Loeb, 2019a, 2019b; Duckworth et al., 2010; Soland, 2019; West et al., 2020). Although these correlations, in theory, could reflect other factors that predict both students’ responses and their other outcomes, research provides some evidence that these responses do capture differences across students in the assessed constructs (Gehlbach & Hough, 2018). At the same time, studies have not comprehensively assessed the validity of changes in student responses over time for monitoring student social-emotional development. Changes over time could measure real changes in student SEL, or they could simply be noise with students selecting responses based on their daily mood or level of concentration. If changes in reports were noise, we would not expect them to predict changes in other valued outcomes, such as achievement. But if changes in reported SEL were capturing real changes in students, those changes could provide useful information to schools and districts. School improvement decisions require making sense of data in real time to guide interventions at a variety of levels, from individual students or entire organizations (Schildkamp, 2019). Changes in reported SEL could be used to evaluate the efficacy of programs or interventions or for monitoring student well-being.

In this paper, we use data from five school districts to examine whether changes in individual students’ self-reported SEL from one school year to the next predict changes in state test scores and attendance. The districts are members of the CORE districts, a collaborative of California school districts that since 2014–15 has administered a common set of surveys to all students in Grades 4 through 12. Our results provide evidence of the relationship between within-student changes in levels of self-reported social-emotional constructs and within-student changes in attendance and academic achievement. Our measures of academic achievement are state standardized tests, thereby extending previous work done studying the relationship between within-student changes in self-management (Duckworth et al., 2010) and self-efficacy (Soland, 2019) and other measures of achievement. To our knowledge, our study is the first to examine the relationship between changes in multiple SEL constructs and changes in outcomes simultaneously.

The CORE SEL survey seeks to measure four aspects of students’ social-emotional development: (a) growth mindset, the belief that one’s abilities can improve with effort (Dweck, 1999); (b) social awareness, the ability to take the perspective of and empathize with others from diverse backgrounds and cultures, to understand social and ethical norms for behavior, and to recognize family, school, and community resources (Collaborative for Academic Social and Emotional Learning [CASEL], n.d.); (c) self-efficacy, the belief in one’s own ability to succeed in achieving an outcome or reaching a goal (Bandura, 1997); and (d) self-management, the ability to regulate one’s emotions, thoughts, and behaviors effectively in different situations, such as managing stress, delaying gratification, motivating oneself, and setting and working toward personal and academic goals (CASEL, n.d.).

Although the importance of SEL is well established, validity evidence surrounding the use of self-report survey measures of SEL to monitor students’ social-emotional development is still emerging. Known limitations of self-report measures (Duckworth & Yeager, 2015; Melnick et al., 2017; Soland et al., 2013) raise doubts about their ability to inform practitioners and policymakers about students’ development. Students may lack sufficient knowledge to evaluate themselves accurately, their responses may be skewed by social desirability bias, and their reference points for evaluating themselves may depend on their classroom or school environment (Duckworth & Yeager, 2015; Soland, et al., 2013; West et al., 2016). Duckworth and Yeager (2015) hypothesize specifically that students’ responses may be insensitive to short-term changes, preventing self-report ratings from capturing changes over time.

However, since 2017, the CORE-PACE Research Partnership has produced a number of studies exploring the measurement properties and use of the SEL surveys. This research has shown that the survey items were developed based on research evidence and that they have broad buy-in from school and district leaders (Allbright & Marsh, 2020; Krachman et al., 2016; West et al., 2018) and, further, that the items are generally reliable measures of their respective constructs (Meyer et al., 2018). The SEL survey scores have also been shown to be predictive of measures of student achievement and behavior such as test scores, grade point average, attendance, and suspensions (Claro & Loeb, 2019a, 2019b; West et al., 2020). In addition, although SEL growth varies widely within schools and classrooms (Fricke et al., 2019; Soland et al., 2019), recent research provides initial evidence that teachers and schools may contribute to students’ growth in SEL measures over time, as student SEL growth varies systematically across schools and across classrooms within schools (Fricke et al., 2019; Meyer et al., 2019).

Although the extant body of work makes strides toward establishing the validity and reliability of the CORE SEL measures for monitoring students’ SEL development, the question of whether students’ academic outcomes improve alongside increases in their SEL is yet unanswered, particularly for SEL dimensions such as growth mindset and social awareness. Establishing this connection is useful for two reasons. First, districts will benefit from appropriately sensitive measures of change in addition to levels if they aim to use the collected data to assess students’ social-emotional development over time. Second, districts will be better served by measures of SEL development that predict achievement gains or attendance if they aim to implement SEL interventions as a means to improve these other outcomes. Assessing the predictive validity of SEL measures based on levels alone leaves open the possibility that student SEL is correlated with unobserved student characteristics. In contrast, assessing whether changes in SEL are predictive of changes in other academic outcomes within students accounts for the influence of unobserved factors that are fixed over time.

Other studies using the CORE data indicate that grade-to-grade changes in students’ responses to the CORE SEL survey do not simply reflect measurement error around a stable student-specific mean. In particular, studies have shown that the SEL measures vary systematically across grades and that this variation differs across constructs. For example, West et al. (2020), using 2 years of the CORE districts’ student surveys, document that students’ self-reported scores on self-efficacy and social awareness decline markedly between Grades 4 and 12, whereas growth mindset scores increase over time. Self-management appears to increase between Grades 4 and 6, decline between Grades 6 and 8, and then increase slightly during high school. Fahle and colleagues (2019) show that the decline in self-efficacy over time is observed among all major demographic groups within the CORE districts and is especially pronounced for girls. These papers speak to the measures’ internal consistency but do not assess their ability to predict distal outcomes.

In this study, we consider the changes in students’ relative position in growth mindset, self-efficacy, self-management, and social awareness in the distribution of their peers and their relationship with the changes in the student’s relative position in the distribution of their peers for math achievement, English language arts (ELA) achievement, and attendance rate. To our knowledge, only two studies have assessed the relationship between within-student changes in SEL and changes in other student-level outcomes. The first focused specifically on self-management skills and grade point average (GPA; Duckworth et al., 2010) and the second on self-efficacy and scores on a district-administered standardized test (Soland, 2019). We are not aware of studies documenting the relationship between changes in self-reported growth mindset and social awareness and changes in other student outcomes. Moreover, because three of these four SEL constructs are highly correlated with each other (Meyer et al., 2018), estimating the relationship between each SEL construct’s change and other outcomes independently may result in omitted-variable bias. As such, it is important to consider all four of these focal constructs simultaneously while also recognizing that this may not fully solve the omitted-variable problem. We found no prior work assessing the relationship between SEL and achievement changes that takes into account changes in other SEL constructs. It is likely, however, that individual stakeholders may examine and interpret changes in SEL measures over time, making the collection of validity evidence around the interpretation of these changes paramount for sound decision making. Although this study cannot establish causal relationships between SEL constructs and other outcomes, it lays the groundwork for causal analyses and the evaluation of specific interventions that seek to build SEL schools as a means to support students’ academic and behavioral success.

The present study uses administrative data and self-reported social-emotional measures for middle school students enrolled in the CORE districts between 2015 and 2017. We address two specific questions: Are within-student changes in self-reported social-emotional measures, measured relative to other students in same grade cohort, predictive of changes in academic and behavioral outcomes? Do the relationships between SEL changes in self-reported social-emotional measures and changes in other outcomes vary by student characteristics such as gender, race, economic disadvantage, prior academic achievement, and prior levels on self-reported social-emotional measures?

Method

Sample and Measures

We used longitudinal data from students in the CORE districts for school years 2014–15 (SY15), 2015–16 (SY16), and 2016–17 (SY17). The CORE districts together serve over 1 million students and represent 20% of all students in California. We constructed our analytic sample based on the requirements of our most restrictive model. We began with the 218,851 students who were in fourth to sixth grade during SY15 and were enrolled in the school districts of Fresno, Long Beach, Los Angeles, San Francisco, and Santa Ana. After filtering out students not observed in all three study years, we were left with 109,335 unique students. We dropped students who appeared in more than one school in a given year, who were missing one or more of the four SEL measured constructs in any of the years analyzed, or for whom we did not have each of the outcomes available each year. This process resulted in retaining 45% of students observed for 3 years, leaving an analytical sample of 49,216 unique students and 147,648 individual observations. Ensuring that each model was fit to the same sample of students was key, as the results of all three of our model specifications were meant to be interpreted alongside each other, and we wanted to remove the possibility that interpretations across models were driven by data availability differences. Table 1 provides a demographic summary of the students included in the main analysis. The sample was predominantly Latinx (73%) and eligible for free or reduced-priced lunch (85%).

Table 1

Demographic Summary of Students in the Analytical Sample

Category
Total students, N	49,216
Percentage female	51.22
Percentage African American	6.71
Percentage Latinx	73.07
Percentage Asian	7.10
Percentage White	65.97
Percentage English learners	13.61
Percentage receiving free/reduced-price lunch	84.60
Percentage whose parents had no high school diploma	22.38

Note. The sample consisted of fourth to sixth grade students in SY15 who completed social-emotional learning (SEL) surveys and had available achievement and attendance data between SY15 and SY17.

The CORE districts began fielding surveys of students to collect information on SEL in spring 2015. The surveys, conducted each spring, aim to measure four SEL constructs by asking students to rate themselves on a series of items in five-level Likert scales. The four SEL measures are the following:

Growth mindset, adapted by Transforming Education from Farrington et al. (2013) and Dweck (1999), measures the extent to which students believe their intelligence is malleable (as opposed to fixed). Students rate how true is each of four statements using a 5-category Likert scale (5 = very true, 1 = not at all true).

Social awareness, adapted by Transforming Education (2016) from the American Institutes for Research (AIR) and the Collaborative for Academic, Social, and Emotional Learning (CASEL) tool “Student Self-Report of Social and Emotional Competencies,” measures perceived interpersonal abilities such as empathizing with others and listening to others’ points of view. Each item has its own 5-category scale.

Self-efficacy, adapted from Farrington et al. (2013), measures how students perceive their abilities to perform academic tasks and succeed in classes. Students rate how confident they are with statements such as “I can do well on all my tests, even when they are difficult” using a 5-category Likert scale (5 = completely confident, 1 = not at all confident).

Self-management represents the “ability to regulate one’s emotions, thoughts, and behaviors effectively in different situations” (Transforming Education, 2016, p. 5), measured through nine items adapted from Park et al. (2017). Students rate how often they behaved as the item described “during the past 30 days,” using a 5-category Likert scale (5 = almost all the time, 1 = never or almost never).

Following Meyer et al. (2018), we used SEL scores constructed by fitting a polytomous item response theory (IRT) model called the generalized partial credit model. Using this method, Meyer et al. (2018) show that the four measures have relatively high reliability, with the exception of growth mindset at lower grades where reliability is lower and there is possible rating scale confusion due to negatively worded items (Bolt et al., 2019). Meyer et al. also provide evidence that the instruments measure distinct constructs.

We focus on three main student outcomes: math and ELA scores from the Smarter Balanced Assessment Consortium (SBAC), a standardized computer-adaptive test administered in Grades 3 through 8, and attendance percentage (days attended divided by days enrolled). The SBAC is also scaled using IRT, with scores placed on a vertical scale. We focus on test scores and attendance percentage because of their availability, importance to practitioners, and interpretability. Table 2 reports mean scores and standard deviations of each SEL construct and outcome by year and cohort. As a shorthand, we refer to growth mindset, social awareness, self-efficacy, and self-management as $S E L_{G M}, S E L_{S A}, S E L_{S E}, S E L_{S M}$ , respectively. Our outcome measures of SBAC ELA, SBAC math, and attendance percentage are denoted as $S B A C_{E L A}, S B A C_{M a t h}, and A T T - P$ , respectively.

Table 2

Score Means (Standard Deviations) by Cohort-Year

SY	Grade	n	$S E L_{G M}$	$S E L_{S A}$	$S E L_{S E}$	$S E L_{S M}$	$S B A C_{E L A}$	$S B A C_{M a t h}$	$A T T - P$
15	4	22,744	0.04	0.09	0.02	0.11	2434.91	2446.79	0.97
			(1.28)	(1.26)	(1.05)	(1.15)	(86.95)	(74.73)	(0.03)
15	5	7982	–0.1	0.07	–0.01	0.07	2474.47	2467.16	0.97
			(1.16)	(1.23)	(1.02)	(1.12)	(87.95)	(79.91)	(0.03)
15	6	18,482	0.07	0.12	0.05	0.16	2502.04	2499.37	0.98
			(1.22)	(1.19)	(1.01)	(1.09)	(88.29)	(93.91)	(0.03)
15	7	8	–0.53	0.42	–0.47	–0.19	2427.25	2412.62	0.96
			(1.91)	(1.97)	(1.53)	(1.37)	(47.02)	(77.45)	(0.04)
16	4	65	–0.36	0.06	–0.35	–0.35	2390.55	2424.49	0.96
			(1.14)	(1.32)	0.85	(1.06)	(70.15)	(56.86)	(0.03)
16	5	22,688	0.08	0.13	0.03	0.14	2489.78	2477.81	0.97
			(1.18)	(1.23)	(1.02)	(1.13)	(91.39)	(85.29)	(0.03)
16	6	7980	0.02	0.09	0.00	0.07	2511.49	2499.66	0.97
			(1.13)	(1.17)	(0.96)	(1.07)	(87.87)	(95.62)	(0.03)
16	7	18,482	0.07	0.12	0.05	0.14	2534.60	2519.85	0.97
			(1.16)	(1.16)	(1.00)	(1.08)	(95.29)	(105.51)	(0.03)
16	8	1	–0.95	0.12	–1.05	–0.32	2532.00	2445	0.95
			—	—	—	—	—	—	—
17	5	76	–0.38	–0.15	–0.19	–0.34	2434.95	2440.41	0.96
			(1.12)	(1.18)	(0.90)	(0.93)	(76.47)	(68.48)	(0.03)
17	6	22,675	0.00	–0.08	0.01	0.04	2510.80	2501.26	0.97
			(1.18)	(1.19)	(1.01)	(1.12)	(95.54)	(106.13)	(0.03)
17	7	7981	0.03	0.06	–0.04	0.03	2530.44	2511.94	0.97
			(1.12)	(1.16)	(0.97)	(1.08)	(97.25)	(106.81)	(0.04)
17	8	18,484	0.00	–0.07	–0.01	–0.02	2554.07	2536.88	0.97
			(1.11)	(1.15)	(1.00)	(1.11)	(97.73)	(118)	(0.03)

Note. SY = school year; SEL = social-emotional learning; GM = growth mindset; SA = social awareness; SE = self-efficacy; SM = self-management; SBAC =Smarter Balanced Assessment Consortium; ATT-P = attendance percentage. Low sample size within some grade-year cells is due to students retained in the same grade over 2 years.

Table 3 shows correlations between SEL constructs and outcomes. Note that social awareness, self-efficacy, and self-management are highly intercorrelated (r = .44–.54), whereas growth mindset is moderately correlated with the other three constructs (r = .20–.34). Additionally, SBAC ELA and SBAC math are correlated with each other (r = .80), whereas attendance percentage is not well correlated with any of the other measures (r < .2).

Table 3

Correlations Between SEL Constructs, SBAC Scores, and Attendance Percentage

	$S E L_{G M}$	$S E L_{S A}$	$S E L_{S E}$	$S E L_{S M}$	$S B A C_{E L A}$	$S B A C_{M a t h}$	$A T T - P$
$S E L_{G M}$	—	—	—	—	—	—	—
$S E L_{S A}$	.20	—	—	—	—	—	—
$S E L_{S E}$	.34	.44	—	—	—	—	—
$S E L_{S M}$	.26	.54	.48	—	—	—	—
$S B A C_{E L A}$	.39	.16	.28	.33	—	—	—
$S B A C_{M a t h}$	.37	.14	.31	.31	.80	—	—
$A T T - P$	.05	.04	.08	.09	.14	.18	—

Note. SEL = social-emotional learning; SBAC =Smarter Balanced Assessment Consortium; GM = growth mindset; SA = social awareness; SE = self-efficacy; SM = self-management; ELA = English language arts; ATT-P = attendance percentage. All correlations were significant at the p < .01 significance level.

The SEL change and achievement change constructs are measures of the change in students’ position in their peer distributions over time. To create these measures, we standardize each SEL measure and outcome variable to have a mean of zero and a standard deviation of one within each grade-year, converting our SEL measures to standard deviations from the cohort mean for a given year. Within-grade-year standardization is essential, as West et al. (2018) document large differences across grades in the average change in SEL scores within the CORE districts. Students’ growth in academic achievement as measured on the SBAC vertical scale also varies systematically across grades, as do average changes in attendance rates. Across-grade differences in SEL could in part reflect differences in student response styles or contexts across grades, whereas differences in outcomes could be influenced by characteristics of the school system unrelated to social-emotional development (e.g., more effective teachers in a given grade). Rather than attempt to link these two phenomena, we take the more conservative approach of relating changes in students’ relative position within the distribution of social-emotional measures among their grade-level peers to changes in their outcomes—also measured relative to their peers.

Figure 1 illustrates the density of the distribution of year-to-year changes in position relative to the peer distribution across all SEL constructs and outcome variables. With the exception of growth mindset, the modal change in each distribution is zero. For growth mindset, the modal change is slightly negative. All measures have year-to-year changes beyond 2½ SD, and changes at or above 1 SD are not uncommon in any measure. Overall, we see that the SEL change distributions have slightly longer tails than the SBAC change distributions. The attendance percentage change distributions have extremely long tails, but most observations are also clustered around zero.

Figure 1.

Distributions of year-to-year changes in SEL and outcome measures. SEL, social-emotional learning; SBAC, Smarter Balanced Assessment Consortium; ELA, English language arts.

We use student demographic variables, such as indicators for economic disadvantage, special education, English learner (EL) status, homelessness, and race/ethnicity, to assess the heterogeneity of results. Table 1 gives the proportion of students in these demographic categories.

Analysis

Our goal is to describe the relationship between changes in SEL and changes in other student outcomes. To provide a benchmark for our preferred estimates, we begin by modeling the relationship between SEL levels and student achievement changes as described by Equation 1:

Y_{i t} = β_{0} + β_{1} S E L_{i t} + β_{2} Y_{i, t - 1} + ε_{i t},

(1)

where $Y_{i t}$ represents an outcome measure (test scores, GPA, or attendance) for student i at time t; $S E L_{i t}$ is the student’s year-cohort standardized level of one of the four SEL variables at time t; and $ε_{i t}$ is the stochastic component of the model. $β_{1}$ is our target estimand, corresponding to the correlation between a SEL variable and the outcome, conditional on the previous measure of that same outcome. By controlling for the outcome measure in the previous period, this model relates change in the outcome to change in SEL. Because the model does not control for within-student time-invariant characteristics, however, it is possible that the relationship estimated through this model reflects unobserved factors related to both the SEL measure and the outcome.

To address this concern, we exploit the panel structure of our data (i.e., the ability to connect observations of the same student across years) to estimate a second model that uses within-student variation to assess whether outcomes change when SEL changes. Equation 2 describes this second model:

Y_{i t} = β_{1} S E L_{i t} + ω_{i} + ε_{i t},

(2)

where $Y_{i t}$ is our outcome measure for student $i$ at time $t$ ; $S E L_{i t}$ is the measurement of a particular SEL construct for student i at time t; and $ω_{i}$ is a student-specific fixed effect. When there are only two time periods per student, this approach is equivalent to a first-differences model that regresses change in outcome on change in SEL (first-difference estimator), which differences out the time-invariant individual student effect. Because we observe each student three times, the student fixed effect in Equation 2 captures the average levels of the outcome variable and SEL construct but allows for the flexible estimation of an intercept that corresponds to a specific student instead of the shared intercept across all periods in Equation 1. A positive value of $β_{1}$ will show that, within individuals, in years when a student’s self-reported SEL is higher relative to their grade cohort, the student’s relative standing in the outcome will also be higher. Hence, the model permits us to establish a general relationship between within-person changes in self-reported SEL and our outcomes.

Equation 2 models the within-student relationship between changes in SEL and changes in outcomes, but we might also be interested in the relationship between changes in SEL and higher order trends in outcomes. That is, how does the change in SEL predict differences in student learning (as measured by test score gains) across years? To address this question, we present a third model:

Δ Y_{i t, t - 1} = β_{1} S E L_{i t} + ω_{i} + ε_{i t},

(3)

where $Δ Y_{i t, t - 1}$ is the change in the standardized value of the outcome measure between the previous time period the current time period. All other values are as in Model 2. This third model, therefore, estimates the association between changes in the SEL construct and changes in the changes of the outcome. Positive values of $β_{1}$ indicate that when a student moves up in their peer distribution for a SEL construct, they can expect that the same-period change in their relative standing on the outcome will be larger than the previous period’s change. We therefore consider this third model complementary to Model 2, instead of better or worse. A disadvantage of this model is that it requires observations to include a lagged achievement variable, so each student’s 3 years of data are treated as only two observations.

We choose the models represented in Equations 2 and 3 instead of structural equation model (SEM) or hierarchical linear model (HLM)-based approaches (e.g., Soland, 2019, and Duckworth et al., 2010, respectively), because specifying a SEM requires making significant assumptions about causal pathways between variables and the items used to measure them and because HLM approaches treat the individual student intercepts as random variables, only estimating their mean and variance. Our sample was large enough that the efficiency gains from estimating random effects were outweighed by the flexibility of estimating individual student fixed effects. We are agnostic about the directionality of the causal relationships between changes in the four SEL constructs and changes achievement. Additionally, SEL constructs in the CORE districts survey appear to follow diverse and changing growth trajectories along school grades (West et al., 2018).

We estimate all models using the plm package v2.2-0 (Croissant & Millo, 2008) in R v3.6.1 (R Foundation, n.d.). We estimate standard errors using cluster-robust variance estimators (Millo, 2017). Additionally, because the correlation between same-period changes in growth mindset are not well correlated with changes in the other three constructs ( $r < . 12$ ) and same-period changes in the other three SEL constructs are only moderately correlated ( $r < . 4$ ), we consider models with a single SEL construct as well as models that include all four SEL constructs.

Results

Overall, we find positive and statistically significant associations between the SEL constructs and the outcome variables across all three model specifications. In all tables for all models, we report robust standard errors clustered by student. Our figures display 95% confidence intervals constructed from robust standard errors adjusted for student-level clustering. In general, we observe that changes in growth mindset and self-management are the strongest predictors of test performance. Additionally, growth mindset, along with self-efficacy, is a predictor of changes in individual attendance percentage.

SBAC ELA

Table 4 presents the results for each of the three models for SBAC ELA. We first look at Columns (a) through (e) under Model 1. Because we control for the previous period’s outcome, this model uses 3 years of data to produce two observations per student in the sample. Results demonstrate that, controlling for prior ELA scores, SEL is related to current ELA scores. Taken individually, a difference of 1 SD in SEL is associated with an expected ELA score from .044 SD (for social awareness) to .098 SD (for growth mindset) higher, controlling for previous SBAC ELA achievement. When combining all four SEL constructs, growth mindset and self-management dominate (b = .083, SE = .002 and b = .073, SE = .002, respectively). The coefficient on self-efficacy becomes trivial in magnitude (b = .009, SE = .003), whereas the coefficient on social awareness becomes negative (b = –.009, SE = .002). This may suggest that social awareness and self-efficacy are predictive of achievement when examined independently only due to their correlations with growth mindset and self-management.

Table 4

Relationship Between Changes in SEL and Changes in ELA Test Scores, Diverse Models

	Model 1: SBAC ELA Predicted by SEL and lagged SBAC ELA					Model 2: SBAC ELA Predicted by SEL and Student FE					Model 3: Delta SBAC ELA Predicted by SEL and Student FE
	SBAC_ELA,t					SBAC_ELA,t					ΔSBAC_ELA,t,t-1
	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)
SEL_GM,t	.098***				.083***	.032***				.028***	.036***				.031***
	(.002)				(.002)	(.002)				(.002)	(.004)				(.004)
SEL_SA,t		.044***			–.009***		.021***			.006**		.031***			.016***
		(.002)			(.003)		(.002)			(.003)		(.004)			(.004)
SEL_SE,t			.059***		.009***			.019***		.002			.024***		.005
			(.002)		(.003)			(.002)		(.003)			(.004)		(.004)
SEL_SM,t				.087***	.073***				.037***	.031***				.040***	.028***
				(.002)	(.003)				(.002)	(.003)				(.004)	(.004)
SBAC_ELA,t-1	.787***	.820***	.811***	.801***	.770***
	(.002)	(.002)	(.002)	(.002)	(.002)
Constant	.012***	.013***	.014***	.011***	.010***
	(.002)	(.002)	(.002)	(.002)	(.002)
Student FE	No	No	No	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Observations	98,432	98,432	98,432	98,432	98,432	147,648	147,648	147,648	147,648	147,648	98,432	98,432	98,432	98,432	98,432
R ²	.691	.685	.686	.69	.696	.004	.002	.001	.004	.008	.002	.001	.001	.002	.004
Adjusted R²	.691	.685	.686	.69	.696	–.494	–.498	–.498	–.494	–.489	–.997	–.998	–.999	–.996	–.993

Note. SEL = social-emotional learning; ELA = English language arts; SBAC =Smarter Balanced Assessment Consortium; FE = fixed effects; GM = growth mindset; SA = social awareness; SE = self-efficacy; SM = self-management. Standard errors clustered by student are shown in parentheses.

p < .1. **p < .05. ***p < .01.

Results for Model 2 are presented in the next set of Columns (a) through (e). This model predicts SBAC ELA including student fixed effects. The inclusion of student fixed effects removes time-invariant student characteristics and, as a result, estimates the relationship between within-student changes in SEL and ELA. Thus, we interpret Model 2 as describing the relationship between changes: Positive coefficients show that upward movement in SEL is accompanied by concurrent upward movement in ELA. The coefficient estimates from Model 2 are smaller than those for Model 1, ranging between .019 and .032. Associations in the four-construct model are, once again, dominated by self-management and growth mindset (b = .031, SE = .002 and b = .028, SE = .002, respectively).

Overall, we take this to show that within students, positive changes in SEL constructs are significantly associated with same-period positive changes in SBAC ELA performance. Additionally, note that for all Model 2 specifications, the sample size is reported as 50% larger than in Models 1 and 3. This occurs because it does not include a lagged outcome, which, from the model’s perspective, allows it to see three observations per student, compared with two per student in the other models. Despite the difference in reported observations, all models are estimated using the same pool of students. Of note are the negative values for the adjusted $R^{2}$ . Adjusted $R^{2}$ incorporates a parsimony penalty that does not confine it to the unit interval. The fixed-effects specification requires estimating a number of model parameters on the order of the number of students, which is heavily penalized by this metric. The unadjusted $R^{2}$ is also near zero, so despite observed associations, outcome prediction at the individual level would be unreliable.

Finally, the third set of Columns (a) through (e) present results for Model 3, where the outcome variable is change in SBAC ELA, and we include student fixed effects. Similar to Model 2, Model 3 focuses on changes within students over time. It differs from the specification in Model 2 in that it relates changes in SEL to changes in the changes in SBAC ELA. Here, positive coefficients describe a situation where a positive change in SEL from time $t$ to time $t + 1$ is accompanied by a larger change in ELA from time $t$ to time $t + 1$ than was observed in time $t - 1$ to time $t$ . We observe that individual constructs are also positively associated with score gains in SBAC ELA, ranging from .024 to .040. As in previous models, associations are dominated by growth mindset and self-management (b = .031, SE = .002 and b = .028, SE = .005, respectively). In this model, however, the coefficient on social awareness is larger and more comparable to the coefficients on growth mindset and self-management (b = .016, SE = .004). As such, increases in all SEL constructs are associated with larger positive changes in SBAC ELA achievement over time.

SBAC Math

Results for SBAC math are similar to those for ELA. Table 5 presents results for each of the three models with both single-construct and four-construct specifications. The estimates presented in the first set of Columns (a) through (e) show the results for Model 1. They demonstrate that controlling for previous SBAC math, students’ position in the SEL distribution of their grade cohort is related to their position in the SBAC math distribution. As with ELA, the four-construct model results in attenuation of estimated coefficients, and social awareness becomes negatively associated with SBAC math achievement. Coefficients in the single-construct models range from .040 to .084, and the four-construct model is dominated by growth mindset and self-management (b = .069, SE = .002 and b = .060, SE = .002, respectively).

Table 5

Relationship Between Changes in SEL and Changes in Math Test Scores, Diverse Models

	Model 1: SBAC Math Predicted by SEL and Lagged SBAC Math					Model 2: SBAC Math Predicted by SEL and Student FE					Model 3: Delta SBAC Math Predicted by SEL and Student FE
	SBAC_Math,t					SBAC_Math,t					ΔSBAC_Math,t,t-1
	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)
SEL_GM,t	.084***				.069***	.033***				.029***	.035***				.032***
	(.002)				(.002)	(.002)				(.002)	(.004)				(.003)
SEL_SA,t		.040***			–.010***		.024***			.008***		.026***			.016***
		(.002)			(.002)		(.002)			(.003)		(.004)			(.004)
SEL_SE,t			.061***		.021***			.024***		.008***			.014***		–.003
			(.002)		(.003)			(.002)		(.003)			(.004)		(.004)
SEL_SM,t				.076***	.060***				.035***	.026***				.030***	.022***
				(.002)	(.003)				(.002)	(.003)				(.004)	(.004)
SBAC_Math,t-1	.823***	.851***	.838***	.835***	.807***
	(.002)	(.002)	(.002)	(.002)	.000
Constant	.001	.001	.002	–.001	–.002
	(.001)	(.001)	(.001)	(.001)	(.001)
Student FE	No	No	No	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Observations	98,432	98,432	98,432	98,432	98,432	147,648	147,648	147,648	147,648	147,648	98,432	98,432	98,432	98,432	98,432
R ²	.729	.724	.725	.728	.733	.005	.002	.002	.005	.009	.002	.001	.0003	.001	.003
Adjusted R²	.729	.723	.725	.728	.733	–.493	–.497	–.497	–.493	–.486	–.996	–.998	–.999	–.997	–.994

Note. SEL = social-emotional learning; SBAC =Smarter Balanced Assessment Consortium; ELA = English language arts; FE = fixed effects; GM = growth mindset; SA = social awareness; SE = self-efficacy; SM = self-management; ATT-P = attendance percentage. Standard errors clustered by student are shown in parentheses.

p < .1. **p < .05. ***p < .01.

The second set of Columns (a) through (e) in Table 5 report results for the individual fixed effects Model 2). Here again, Model 2 is a model describing the relationship between changes: Positive coefficients show that upward mobility in the SEL distribution is accompanied by concurrent upward mobility in the SBAC math distribution. Here again, the associations for social awareness and self-efficacy reduce in magnitude in the four-construct model, and growth mindset and self-management are the dominant SEL constructs (b = .029, SE = .002 and b = .026, SE = .002, respectively), although positive changes in any of the SEL constructs have significant positive associations with changes in SBAC performance. Table 5 presents the results for Model 3 in the third set of Columns (a) through (e). Recall that positive coefficients indicate that positive SEL changes are associated with increased SBAC math gains.

Each single-construct model shows a significant positive relationship between the change in SEL and the change in SBAC math achievement over time. In the four-construct model, the same three constructs are positively associated with improving math achievement, but the association with self-efficacy is not significantly different from zero. This result is qualitatively similar to the observed relationship between self-efficacy and SBAC ELA from the same model, where the coefficient was also not significantly different from zero. Increases in growth mindset (b = .032, SE = .004) dominate this model, but changes in social awareness and self-management are positively associated with increasing math achievement gains across time with similar magnitudes (b = .016, SE = .004 and b = .022, SE = .004, respectively).

Attendance

We also look at the behavioral outcome of attendance. We standardize attendance rate (percentage of days attending) within cohort-year, so all changes are again relative to their peer distribution. In Table 6, the first set of Columns (a) through (e) show attendance regressed on SEL and lag attendance. Coefficients on SEL in the single-construct models are similar in magnitude to those in the SBAC models, ranging from .024 to .035. The coefficient on social awareness is no longer significantly different from zero in the four-construct model, and self-efficacy and self-management, instead of growth mindset and self-management, emerge as dominant (b = .025, SE = .003 and b = .033, SE = .003, respectively).

Table 6

Relationship Between Changes in SEL and Changes in Attendance Percentage, Diverse Models

	Model 1: ATT-P Predicted by SEL and Lagged ATT-P					Model 2: ATT-P Predicted by SEL and Student FE					Model 3: Delta ATT-P Predicted by SEL and Student FE
	ATT-P_t					ATT-P_t					ΔATT-P_t,_t-1
	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)	(a)	(b)	(c)	(d)	(e)
SEL_GM,t	.028***				.011***	.013***				.010***	.020***				.016***
	(.002)				(.002)	(.002)				(.002)	(.005)				(.005)
SEL_SAt		.026***			–.004*		.014***			.005*		.013**			.004
		(.002)			(.003)		(.002)			(.003)		(.005)			(.006)
SEL_SE,t			.043***		.025***			.021***		.015***			.027***		.023***
			(.002)		(.003)			(.002)		(.003)			(.005)		(.006)
SEL_SM,t				.045***	.033***				.016***	.008***				.013***	.002
				(.002)	(.003)				(.002)	(.003)				(.005)	(.006)
ATT-P_t-1	.650***	.650***	.648***	.647***	.646***
	(.006)	(.006)	(.006)	(.006)	(.006)
Constant	.016***	.017***	.017***	.015***	.015***
	(.002)	(.002)	(.002)	(.002)	(.002)
Student FE	No	No	No	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Observations	98,432	98,432	98,432	98,432	98,432	147,648	147,648	147,648	147,648	147,648	98,432	98,432	98,432	98,432	98,432
R ²	.412	.412	.414	.414	.415	.0003	.0004	.001	.0005	.001	.0003	.0001	.001	.0001	.001
Adjusted R²	.412	.412	.414	.414	.415	–.499	–.499	–.499	–.499	–.498	–.999	–1.000	–.999	–1.000	–.999

Note. SEL = social-emotional learning; ATT-P = attendance percentage; FE = fixed effects; GM = growth mindset; SA = social awareness; SE = self-efficacy; SM = self-management. Standard errors clustered by student are shown in parentheses.

p < .1. **p < .05. ***p < .01.

The second set of Columns (a) through (e) in Table 6 present Model 2 with individual fixed effects. Here, coefficients on SEL are generally smaller than those observed in the SBAC models, ranging from .013 to .021. In the four-construct model, self-efficacy is the dominant construct (b = .015, SE = .003) and self-management takes a back seat to growth mindset, whereas the coefficient on social awareness is again not significantly different from zero. Here we see that increasing self-efficacy is associated with the largest increases in attendance.

The final set of Columns (a) through (e) in Table 6 show the results for Model 3. The estimated coefficients for the individual SEL construct models are lower in magnitude than those for the SBAC models (.013–.027). In the four-construct model, growth mindset and self-efficacy dominate (b = .016, SE = .005 and b = .023, SE = .006, respectively).

Taking the three sets of results together, we observe that changes in SEL are less strongly associated with changes in attendance than they are with changes in achievement. This pattern holds even though SEL levels are more strongly associated with attendance levels than are SBAC scores. One possible explanation is that students’ relative position in the attendance distribution is less stable year to year than relative position in the SBAC distributions (as evidenced in Figure 1). Overall, we find that changes in self-management and growth mindset are stronger predictors of changes in test performance, whereas changes in self-efficacy and growth mindset are stronger predictors of changes in attendance. Changes in self-management also predict changes in attendance, but the estimates for this construct are less consistent.

Heterogeneity

The relationship between changes in SEL and achievement may differ depending on students’ learning opportunities (Yeager et al., 2019), academic level (Yeager et al., 2019), previous levels of SEL, or other student characteristics. We explore heterogeneity in three primary ways: by demographic subgroups, by previous achievement and attendance, and by starting SEL.

Demographic Subgroups

We fit models within each subgroup to assess heterogeneity in the relationship between SEL and outcomes by subgroup. By plotting coefficients from these models side by side, we can visually diagnose heterogeneity. Heterogeneity here may be an indication of different relationships between changes in SEL constructs and outcomes of interest for different subgroups or differences in what the SEL survey items are measuring in different groups. Figure 2 shows the coefficient on each SEL construct from a model that regresses SBAC ELA on all four SEL measures under the fixed-effects specification from Model 2. The figure displays the 95% confidence intervals from cluster-robust standard errors (clustered at the individual level). The logic of this approach is that by looking for where confidence intervals do (and do not) overlap with each other, we have a visual indicator of observed heterogeneity. All four constructs in Figure 2 display overlapping confidence intervals, indicating that the estimates may not be distinguishable from each other at traditional significance levels. One exception is that students who are eligible for free or reduced-price lunch (FRL) appear to have a larger association between changes in growth mindset and changes in SBAC ELA scores than do students who are not eligible for subsidized lunch (non-FRL). However, this finding may be spurious given the considerable number of subgroups examined. Overall, the relationship between improving relative position in the SEL distribution and improving relative position in the distributions of academic and behavioral outcomes is roughly the same for all groups of students. (For details, see Table A1 in the supplemental materials.)

Figure 2.

Change in SBAC ELA per unit change in SEL by subgroup and construct. SBAC, Smarter Balanced Assessment Consortium; ELA, English language arts; SEL, social-emotional learning; EL, English learner; FRL, free or reduced-price lunch; HS, high school.

Figure 3 shows the coefficient on SEL under Model 2 when the outcome variable is SBAC math performance. Here, we see overlapping confidence intervals within each construct across all subgroups, but EL students seem to have a lower association between changes in self-management and changes in math achievement than non-EL students, which may be spurious due to the large number of comparisons being conducted. (For details, see Table A2 in the supplemental materials.)

Figure 3.

Change in SBAC math per unit change in SEL by subgroup and construct. SBAC, Smarter Balanced Assessment Consortium; SEL, social-emotional learning; EL, English learner; FRL, free or reduced-price lunch; HS, high school.

Figure 4 shows the coefficient on SEL under Model 2 for change in attendance percentage. Although there are generally overlapping confidence intervals across all subgroups and constructs, we see that students classified as English language learners (EL) have a significantly lower association between change in self-efficacy and change in attendance when compared to their non-EL peers. (For details, see Table A3 in the supplemental materials.)

Figure 4.

Change in attendance percentage per unit change in SEL by subgroup and construct. SEL, social-emotional learning; SBAC, Smarter Balanced Assessment Consortium; ELA, English language arts; EL, English learner; FRL, free or reduced-price lunch; HS, high school.

Heterogeneity by Previous Achievement and Attendance

Figures 5 and 6 present the estimated change in outcome per unit change in SEL, binned by quintile of the starting (SY15) value of the outcome, using the fifth specification of Model 2. Results from the earlier models suggest that self-management and growth mindset will likely be the dominant SEL constructs. Overall, we observe that self-management and growth mindset improvements predict higher achievement for all students on both outcomes, but their relative magnitude and ordering vary by achievement quintile.

Figure 5.

Change in SBAC ELA per unit change in SEL by SBAC ELA quintile. SBAC, Smarter Balanced Assessment Consortium; ELA, English language arts; SEL, social-emotional learning.

Figure 6.

Change in SBAC math per unit change in SEL by SBAC math quintile. SBAC, Smarter Balanced Assessment Consortium; SEL, social-emotional learning.

Figure 5 shows the estimated change in SBAC ELA per unit change in SEL, binned by quintile of ELA. We see that growth mindset and self-management dominate across all SBAC quintiles. The coefficient on self-efficacy is low across all quintiles except the highest, where the point estimate for the coefficient on self-efficacy edges out that of growth mindset, although they are not significantly different. Finally, we observe that with the exception of the lowest quintile, the association between change in growth mindset and change SBAC ELA decreases as starting SBAC ELA increases, whereas the relationship between change self-management and change SBAC ELA increases as starting SBAC ELA increases. (For details, see Table A4 in the supplemental materials.)

Figure 6 shows the same analysis for SBAC math. Here, we see a similar pattern for growth mindset and self-management as observed in Figure 5, whereas social awareness and self-efficacy are now more often significantly different from zero. Again, self-efficacy trumps growth mindset for students starting in the highest SBAC math quintile (although these estimates are, again, not significantly different), whereas increases in social awareness distribution are associated with increases in SBAC math across the middle three starting SBAC math quintiles. (For details, see Table A5 in the supplemental materials.)

Figure 7 shows this analysis for attendance percentage. This sheds some light on why we may have observed attenuated coefficients compared with our SBAC models. All of the action, perhaps not surprisingly, is in the lowest quintile of attendance. This is because there is a hard ceiling on attendance rate, and most of the density in change in attendance percentage is near zero (see Figure 1). The association between changes in social awareness and changes in attendance is small across the distribution. From the second quintile onward, the estimated change in attendance per unit change in the other three SEL constructs is on the order of half the magnitude observed in the first quintile and often is not significantly different from zero. (For details, see Table A6 in the supplemental materials.)

Figure 7.

Change in attendance percentage per unit change in SEL by mean attendance quintile. SEL = social-emotional learning.

Heterogeneity by Starting SEL

In our final analyses, we assess differences in the relationship between changes in SEL and other outcomes across quintiles of SEL. Figures 8, 9, and 10 show the association between change in SEL and change in SBAC performance and attendance percentage, broken out by quintile of starting (SY15) student SEL. Using starting SEL to create the quintiles ensures that individual students remain in the same quintile across all three observation years. The plots use the individual fixed-effects specification from Model 2, fitting single-construct models within each of the 20 construct-quintiles. We run single-construct models instead of including all SEL constructs in the same model because an individual student can appear in different quintiles for different constructs. For example, within each of the growth mindset quintiles, we run a separate regression of the outcome of interest on growth mindset with individual fixed effects. The figures allow comparisons of the estimated change in the outcome per unit change in SEL across the quintiles. Thus, each figure summarizes 20 independent model fits and includes 95% confidence intervals computed using cluster-robust standard errors clustered at the individual level.

Figure 8.

Change in SEL and SBAC ELA by starting SEL quintile. SEL, social-emotional learning; SBAC, Smarter Balanced Assessment Consortium; ELA, English language arts.

Figure 9.

Change in SEL and SBAC Math by starting SEL quintile. SEL, social-emotional learning; SBAC, Smarter Balanced Assessment Consortium.

Figure 10.

Change in SEL and attendance percentage by starting SEL quintile. SEL = social-emotional learning.

Students in the lowest quintile show a stronger relationship between SEL changes and ELA and math changes, particularly for self-management and social awareness. The relationship decreases the higher the average SEL of a student but remains significantly different from zero, with the exception of self-efficacy for students with high self-efficacy. Changes in self-management and growth mindset appear to be the most predictive of changes in achievement for the majority of students. Differences across groups defined by average SEL are less clear when predicting attendance. In what follows, we describe each outcome in more detail.

Figure 8 shows the relationship between change in SEL and change in SBAC ELA across SEL construct-quintile. With two exceptions, we observe a fairly consistent association between upward mobility in the SEL distribution and upward mobility in the SBAC ELA distribution. First, the association between self-management and SBAC ELA is of the largest magnitude closer to the middle quintile, and second, associations for all constructs except growth mindset are generally smaller in the highest quintile. (For details, see Table A7 in the supplemental materials.)

Figure 9 shows the relationship between change in SEL and change in SBAC math by SEL quintile. We see approximately the same pattern observed for ELA—consistent magnitudes across the SEL quintiles with self-management cresting at the middle quintile but without the general drop-off in coefficient magnitude in the highest SEL quintiles. (For details, see Table A8 in the supplemental materials.)

Figure 10 presents the same analysis for change in attendance. The coefficients on each of the constructs are more tightly grouped across the quintiles. The clear overlap of the confidence intervals shows that the ordering of the magnitudes of the associations is less clear-cut than with the SBAC figures. This overlap is, in part, a function of the overall smaller coefficients for these models. The lowest SEL quintile has all constructs except self-management not significantly different from zero and a slight decline in coefficient magnitude as starting SEL increases. (For details, see Table A9 in the supplemental materials.)

Discussion

In this paper, we use large-scale data from California to examine whether changes in individual students’ reports on social-emotional measures from one school year to the next, relative to other students in the same grade cohort, predict relative changes in theoretically related academic and behavioral outcomes. Although the importance of SEL is well established, the understanding of survey-based measures of social-emotional constructs collected at scale is still emerging. Prior work has shown that self-reported SEL levels predict student achievement levels as well as student achievement gains. However, prior work had not investigated whether changes in student reports of SEL predict changes in other factors. These analyses are useful because districts need appropriately sensitive measures of change in SEL if they are to use the data to assess students’ social-emotional development over time. Moreover, districts will be better served by measures of SEL development that predict achievement gains if they aim to use social-emotional interventions as a means to improve these other outcomes.

Our analyses confirm that changes in a student’s self-reported SEL predict changes in both their ELA and their math achievement. These associations are strongest for growth mindset and self-management. Although the estimated effects are modest, they are educationally meaningful and robust across model specifications. It is important to note from Table 2 that we observe a mean year-to-year increase in SBAC scores that varies between grade levels and by cohort. Additionally, 1 SD in the grade-year SBAC distribution varies from year to year and cohort to cohort. Putting this in context, a student experiencing a 1 SD increase in growth mindset as she moved from fourth to fifth grade in SY16 increased her SBAC score by .028 SD, the equivalent of 2.56 points, a 4.66% higher increase than the mean annual growth in SBAC that year for that cohort (which was 54.87 points). Depending on the year and cohort, we find that a 1 SD change in growth mindset is associated with additional SBAC ELA gains from 4.66% to 11.58% of average annual learning gains and additional SBAC math gains from 7.97% to 25.06% of average annual learning gains. A 1 SD change in self-management is associated with SBAC ELA gains from 5.16% to 15.63% of average annual learning gains and SBAC math gains from 7.15% to 22.47% of average annual learning gains. As expected, gains associated with social awareness and self-efficacy are much smaller, ranging from 0.33% to 3.03% of average annual learning gains for SBAC ELA and 2.2% to 6.91% of average annual learning gains for SBAC math. Taking a typical school year to be 180 days, we find that a 1 SD increase in SEL is associated with attending between .03 and .11 additional days of school, depending on the construct and cohort-year.

In summary, assuming (naively) that learning is uniformly distributed across the 180 days of school, we find that a 1 SD change in growth mindset or self-management is associated with SBAC gains the equivalent of between 8 and 45 days of learning. Additionally, these estimates may underreport the true effects of SEL given that the measures are likely to be measured with some error and therefore suffer from attenuation bias (Loeb et al., 2019; Meyer et al., 2018). Although the sample in which we observe these trends is diverse across multiple demographic characteristics, including socioeconomic status, race, and language status, the relationships between SEL and both achievement and attendance outcomes are stable across groups. Even when grouping students by SEL quintile and performing the analysis within these groups, we estimate positive point estimates that are statistically significant across a wide range of SEL quintiles.

These results complement previous research on the relationship between within-student changes in SEL constructs and academic measures (Duckworth et al., 2010; Soland, 2019), by adding an analysis for growth mindset and social awareness as well as outcomes such as attendance and state tests. In addition, we estimate the relationship between SEL constructs and achievement using all four SEL constructs simultaneously. This multiple-measures model is an important distinction from previous work, because the SEL constructs correlate with one another, which invites the possibility of omitted variable bias in models that do not take all four into account. This study also complements recent evidence showing that the average changes in student SEL scores relative to grade-level peers vary substantially across schools (Jackson et al., 2020; Loeb et al., 2018). Loeb et al. (2018) estimate that variation across schools on SEL outcomes can range from .09 to .24 SD for the four constructs examined in this study. Our results suggest that these changes may be associated with improved learning (especially in the case of self-management and growth mindset) and better attendance (especially in the case of growth mindset and self-efficacy).

Despite the large size of our sample and associated statistical power, this study has limitations. In particular, our approach does not allow us to convincingly estimate causal relationships. Although we go further than prior studies in controlling for other factors that could mask the relationship between SEL changes and changes in other outcomes, we are not able to eliminate all possible biases. Causal inference is challenging due to issues of reverse causality and timing. Although we document a robust relationship between changes in SEL and changes in other outcomes, we do not know the causal pathway.

We also lean heavily on linear models and individual fixed-effects specifications. These specifications have two specific drawbacks. First, they make assumptions about functional forms that may not, in practice, be true. For example, there may be heterogeneity in the magnitude of associations across the SEL bands or by direction of change. Second, fixed-effects estimators only control for time-invariant features of the individual. Time-varying factors, like changes in mental health or cognitive ability and interventions at the school or district level, may confound results. Fortunately, our data were all collected prepandemic and thus are free from pandemic-related exogenous shocks. Additionally, researchers may be interested in heterogeneity in associations at the school, district, or cohort level. These questions are better answered using an HLM specification and present a good line of inquiry for future work with these data.

Our sample is limited, and we advise caution when attempting to generalize these findings. We consider only students in the middle grades, preventing us from generalizing more broadly to elementary and high school students. The CORE districts are demographically unlike many districts across the United States, and the samples we use from these districts are also heavily restricted to only retain complete cases. Because of this, they may not be accurate representations of their own districts, as there may be systematic patterns in the students who were excluded from the final sample. We only have SEL measurements that were conducted once yearly, in the spring of each school year, so further work would be needed to understand the relationship between within-student changes in self-reported SEL on much shorter time scales (such as from fall to spring) and academic outcomes.

Even with these limitations, this study provides additional validity evidence for the use of the CORE SEL measures to monitor students’ improvement on SEL dimensions and may serve as an impetus to expand survey-based data collection on SEL. Our analytic approach of looking at changes of position within grade-year distributions is also a promising foundation for the analysis of data collected from other districts or age groups, as this approach controls for changes in the saliency of the SEL constructs and the individual items used to measure them as students mature. This can allow both researchers and practitioners to observe individual trajectories and better connect them to other outcomes, academic and otherwise.

Supplemental Material

sj-docx-1-ero-10.1177_23328584241233277 – Supplemental material for Are Changes in Reported Social-Emotional Skills Just Noise? The Predictive Power of Longitudinal Differences in Self-Reports

Supplemental material, sj-docx-1-ero-10.1177_23328584241233277 for Are Changes in Reported Social-Emotional Skills Just Noise? The Predictive Power of Longitudinal Differences in Self-Reports by Klint Kanopka, Susana Claro, Susanna Loeb, Martin West and Hans Fricke in AERA Open

Footnotes

Acknowledgements

Hans Fricke worked on this project before joining Amazon.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Klint Kanopka

Susana Claro

Supplemental Material

Supplemental material for this article is available online.

Authors

KLINT KANOPKA is an assistant professor of applied statistics in the Steinhardt School of Culture, Education, and Human Development at New York University, 246 Greene St, Floor 3, New York, NY 10003; email: klint.kanopka@nyu.edu. His research focuses on the integration of psychometric methods and machine learning.

SUSANA CLARO is an assistant professor in the School of Government at Pontificia Universidad Católica de Chile, Vicuña Mackenna 4860, Edificio Mide UC, 3rd floor, Macul, Santiago, Chile; email: susanaclaro@uc.cl. Her research focuses on the inequality of opportunities among students and on teachers, motivation, and socio-emotional skills.

SUSANNA LOEB is a professor at the Stanford University Graduate School of Education, 482 Galvez Mall, Stanford, CA 94305; email: sloeb@stanford.edu. Her research focuses broadly on education policy and its role in improving educational opportunities for students.

MARTIN WEST is academic dean and Henry Lee Shattuck Professor of Education at the Harvard Graduate School of Education, 13 Appian Way, Cambridge, MA 02138; email: martin_west@gse.harvard.edu. His research focuses on K-12 education policy and politics in the U.S.

HANS FRICKE is an economist at Amazon, 410 Terry Ave. N Seattle, WA 98109; email: hans.fricke@gmail.com. His research focuses on economics of education, education policy, behavioral interventions in education, and causal analysis.

References

Allbright

T. N.

Marsh

J. A.

(2020). Policy narratives of accountability and social-emotional learning. Educational Policy. Advance online publication. https://doi.org/10.1177/0895904820904729

Bandura

(1997). Self-efficacy: The exercise of control. Freeman.

Bolt

D. M.

Wang

Y. C.

Meyer

R. H.

Pier

(2019, October). An IRT mixture model for rating scale confusion associated with negatively worded items in measures of social-emotional learning [Working paper]. Policy Analysis for California Education. https://www.edpolicyinca.org/publications/irt-mixture-model-rating-scale-confusion

Claro

Loeb

(2019a, September). Self-management skills and student achievement gains: Evidence from California’s CORE districts [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/self-management-skills-and-student-achievement-gains-evidence-california-core-districts

Claro

Loeb

(2019b, October). Students with growth mindset learn more in school: Evidence from California’s CORE school districts [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/mindset-effect-academic-achievement

Collaborative for Academic, Social, and Emotional Learning (CASEL). (n.d.). Core SEL competencies. https://casel.org/core-competencies/

Croissant

Millo

(2008). Panel data econometrics in R: The plm package. Journal of Statistical Software, 27(2). https://doi.org/10.18637/jss.v027.i02

Deming

D. J.

(2017). The growing importance of social skills in the labor market. Quarterly Journal of Economics, 132(4), 1593–1640. https://doi.org/10.1093/qje/qjx022

Duckworth

A. L.

Tsukayama

May

. (2010). Establishing causality using longitudinal hierarchical linear modeling: An illustration predicting achievement from self-control. Social Psychological and Personality Science, 1(4), 311–317. https://doi.org/10.1177/1948550609359707

10.

Duckworth

A. L.

Yeager

D. S.

(2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251. https://doi.org/10.3102/0013189X15584327

11.

Dweck

C. S.

(1999). Self-theories: Their role in motivation, personality and development. Psychology Press.

12.

Fahle

E. M.

Lee

M. G.

Loeb

(2019, October). A middle school drop: Consistent gender differences in students’ self-efficacy [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/gender-differences-students-self-efficacy

13.

Farrington

Levenstein

Nagaoka

(2013). “Becoming effective learners” survey development project. Society for Research on Educational Effectiveness. https://eric.ed.gov/?id=ED563049

14.

Farrington

C. A.

Roderick

Allensworth

Nagaoka

Keyes

T. S.

Johnson

D. W.

Beechum

N. O.

(2012, June). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance—A critical literature review. University of Chicago Consortium on Chicago School Research. https://consortium.uchicago.edu/publications/teaching-adolescents-become-learners-role-noncognitive-factors-shaping-school

15.

Fricke

Loeb

Meyer

Rice

Pier

(2019, May). Stability of school contributions to student social-emotional learning gains [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/sites/default/files/WP_Fricke_May19_1.pdf

16.

Gehlbach

Hough

H. J.

(2018). Measuring social emotional learning through student surveys in the CORE districts: A pragmatic approach to validity and reliability [Report]. Policy Analysis for California Education. https://edpolicyinca.org/publications/measuring-social-emotional-learning-through-student-surveys-core-districts

17.

Jackson

K. C.

Porter

S. C.

Easton

J. Q.

Blanchard

Kiguel

(2020, February). School effects on socio-emotional development, school-based arrests, and educational attainments [NBER Working paper 26759]. National Bureau of Economic Research. https://www.nber.org/papers/w26759

18.

Kautz

Heckman

J. J.

Diris

ter Weel

Borghans

(2014, December). Fostering and measuring skills: Improving cognitive and non-cognitive skills to promote lifetime success [NBER Working paper 20749]. National Bureau of Economic Research. https://www.nber.org/papers/w20749

19.

Krachman

S. B.

Arnold

LaRocca

(2016, September). Expanding the definition of student success: A case study of the CORE districts. Transforming Education. https://www.transformingeducation.org/core-case-study/

20.

Loeb

Christian

M. S.

Hough

H. J.

Meyer

R. H.

Rice

A. B.

West

M. R.

(2018, May). School effects on social-emotional learning: Findings from the first large-scale panel survey of students [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/school-effects-social-emotional-learning

21.

Loeb

Christian

M. S.

Hough

Meyer

R. H.

Rice

A. B.

West

M. R

. (2019). School differences in social-emotional learning gains: Findings from the first large-scale panel survey of students. Journal of Educational and Behavioral Statistics, 44(5), 507–542. https://doi.org/10.3102/1076998619845162

22.

Melnick

Cook-Harvey

C. M.

Darling-Hammond

(2017, April). Encouraging social and emotional learning in the context of new accountability [Report]. Learning Policy Institute. http://learningpolicyinstitute.org/product/sel-new-accountability

23.

Meyer

Pier

Mader

Christian

Rice

Loeb

Fricke

Hough

(2019, October). Can we measure classroom supports for social-emotional learning? Applying value-added models to student surveys in the CORE districts [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/can-we-measure-classroom-supports-sel

24.

Meyer

R. H.

Wang

Rice

A. B.

(2018, May). Measuring students’ social-emotional learning among California’s CORE districts: An IRT modeling approach [Working paper]. Policy Analysis for California Education. https://edpolicyinca.org/publications/measuring-students-social-emotional-learning-among-californias-core-districts-irt

25.

Millo

(2017). Robust standard error estimators for panel models: A unifying approach. Journal of Statistical Software, 82(3). https://doi.org/10.18637/jss.v082.i03

26.

Park

Tsukayama

Goodwin

G. P.

Patrick

Duckworth

A. L.

(2017). A tripartite taxonomy of character: Evidence for intrapersonal, interpersonal, and intellectual competencies in children. Contemporary Educational Psychology, 48, 16–27. http://dx.doi.org/10.1016/j.cedpsych.2016.08.001

27.

R Foundation. (n.d.). The R project for statistical computing. https://www.R-project.org/

28.

Schildkamp

(2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273.

29.

Soland

(2019). Modeling academic achievement and self-efficacy as joint developmental processes: Evidence for education, counseling, and policy. Journal of Applied Developmental Psychology, 65, Article 101076. https://doi.org/10.1016/j.appdev.2019.101076

30.

Soland

Hamilton

L. S.

Stecher

B. M.

(2013, November). Measuring 21st century competencies: Guidance for educators. RAND Corporation. https://www.rand.org/pubs/external_publications/EP50463.html

31.

Soland

Kuhfeld

Wolk

(2019). Examining the state-trait composition of social-emotional learning constructs: Implications for practice, policy, and evaluation. Journal of Research on Educational Effectiveness, 12(3), 550–577. https://doi.org/10.1080/19345747.2019.1615158

32.

Transforming Education. (2016, April). Measuring MESH: Student and teacher surveys curated for the CORE districts. https://www.transformingeducation.org/wp-content/uploads/2017/04/160406_MeasuringMESH_ForRelease2.pdf

33.

West

M. R.

Kraft

M. A.

Finn

A. S.

Martin

R. E.

Duckworth

A. L.

Gabrieli

C. F. O.

Gabrieli

J. D. E.

(2016). Promise and paradox: Measuring non-cognitive traits of students and the impact of schooling. Educational Evaluation and Policy Analysis, 38(1), 148–170.

34.

West

M. R.

Buckley

Krachman

S. B.

Bookman

(2018). Development and implementation of student social-emotional surveys in the CORE districts. Journal of Applied Developmental Psychology, 55, 119–129. https://doi.org/10.1016/j.appdev.2017.06.001

35.

West

M. R.

Pier

Fricke

Loeb

Meyer

R. H.

Rice

A. B.

(2020). Trends in student social-emotional learning: Evidence from the CORE districts. Educational Evaluation and Policy Analysis, 42(2), 279–303.

36.

Yeager

D. S.

Dweck

C. S.

(2012). Mindsets that promote resilience: When students believe that personal characteristics can be developed. Educational Psychologist, 47(4), 302–314. https://doi.org/10.1080/00461520.2012.722805

37.

Yeager

D. S.

Hanselman

Walton

G.M.

, et al. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573, 364–369. https://doi.org/10.1038/s41586-019-1466-y

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB