Abstract
This study shows how receptivity and responsiveness can influence the efficacy of an intervention helping adolescents reappraise worry or uncertainty following the difficult transition to middle school. The intervention was implemented at-scale in a diverse sample of sixth-grade public school district students followed through eighth grade (N = 1,180; 41% Black or Latinx; 44% low socioeconomic status). Results suggest the intervention’s effects on grade point average are confined to a racially and socioeconomically diverse subgroup of adolescents who had high teacher evaluations of their classroom behaviors in kindergarten that declined over the early elementary school years (i.e., “Disengagers”). These findings suggest that adolescents’ past school experiences with educators may bound the extent to which interventions can promote success in school.
Keywords
T
In this article, I help explain how past experiences shape student involvement with educational interventions by introducing the “receptivity–responsiveness” theoretical framework. I do so by describing that framework in detail and applying it to an intervention targeting adolescents’ worry about belonging following the transition to middle school (i.e., “the belonging intervention”). 1 This intervention, which helps adolescents reappraise uncertainty about the transition to middle school, may resolve worry about belonging in a positive direction among adolescents and ultimately increase their academic achievement (Borman et al., 2019; Goyer et al., 2019; Pyne & Borman, 2020). Administered early in the school year, the intervention’s two exercises present vignettes from older peers reassuring participants that worry about belonging in middle school is normal, and that most peers worry much less about belonging by seventh grade. Participants then complete writing prompts, reflecting on why they might worry about middle school at first, why they might worry less over time, and why they might do well in middle school despite worrying. Important to understanding the degree to which positively resolving such worry helps new middle school students is how their earlier experiences influence the ways they feel about school in the first place.
In the following sections, I first describe the proposed receptivity–responsiveness framework, including how to consider its relationship to individuals’ past experiences. I then introduce the belonging intervention under study and survey the literature on brief interventions in which it is situated to understand what is known about how context and individual history influence such interventions. Next, I describe studies similar to the current one reporting positive academic main effects of the belonging intervention, the theory explaining why the intervention works, and how I add to this robust empirical and theoretical literature by applying the receptivity–responsiveness framework. Following a discussion of the data, methods, and results of this study, I close by discussing the ways in which this study may be generative for future empirical and theoretical literature on these sorts of interventions.
The Receptivity and Responsiveness Framework
An educational intervention’s success is likely tied to participants’ past and current experiences. For example, it is likely the belonging intervention under study can be influenced by current experiences—including the context in which it is implemented (Walton & Brady, 2017; Walton & Yeager, 2020). However, individuals’ past experiences are not always obvious to policymakers and researchers, and it remains unclear the degree to which personal histories also influence whether and for whom it works. I propose that knowing how someone’s history affects educational interventions involves understanding receptivity and responsiveness. A person must first engage with such an intervention and understand its message. Having done so, if they consciously or unconsciously draw on their past and decide to believe the intervention’s message, they have receptivity. People’s histories influence receptivity to the intervention because those who align with the values of a setting will more easily adapt than others who have been perceived as unaligned with those values (Bourdieu, 1984, 2005; Merleau-Ponty, 1962).
If someone’s past shapes receptivity positively, the intervention can succeed; however, a person also needs responsiveness—or the capacity to change one’s thinking in a way that affects an observable change of interest. Real-world, policy-relevant responses, like improved academic achievement, are many steps removed from immediate, internal responses people undergo during interventions. For example, among adolescents in the belonging intervention under study, some may underperform academically due to worry about belonging, while for others resolving that worry may be unrelated to their current level of achievement. Thus, low responsiveness on academic achievement occurs when someone’s history either prevents, or renders inert, the processes allowing those internal responses to develop into impacts on achievement.
Prior Teacher Perceptions of Students
The complimentary ideas of receptivity and responsiveness help explain how people’s past experiences shape their current circumstances. I contend that teacher perception of past classroom behavior is one way to capture an adolescent’s history of school experiences, and thus helps clarify how receptivity and responsiveness to educational interventions may work. In many elementary schools, teachers regularly evaluate each child’s behavior in class by responding to standardized report card prompts using a numbered scale, much like they do to evaluate children’s math or reading. Like many academic achievement measures, classroom behavior measures such as these have subjective, teacher-reported components to them (but certainly also may capture actual behaviors). Just as educators define what an “A” means in their class, they also define what being “well-behaved” means, often with expectations for behavior explicitly reviewed and posted in class. As with letter grades, teacher perceptions of behavior are not neutral, evenly distributed, or readily understood among students (Calarco, 2018). Although partially subjective, teachers’ perceptions of student behavior are consequential for how well adolescents do in school, including how they will feel about school for years to come (Okonofua, Paunesku, & Walton, 2016; Okonofua, Walton, & Eberhardt, 2016; Pyne & Musto, 2022).
In differentiating participants by whether teachers see them as exhibiting relatively high levels of classroom behavior in kindergarten, I seek to determine whether teacher perceptions of their behavior subsequently change over early elementary school. Doing so reveals four distinct groups of adolescents: Consistent Engagers, who have high teacher ratings at school entry that remain high through early elementary school; Rising Engagers, who have low teacher ratings at school entry that increase through early elementary school; Disengagers, who have high initial teacher ratings that decline over time; and Nonengagers, who have had relatively low teacher ratings of their classroom behaviors throughout early elementary school. According to the receptivity and responsiveness framework I develop here, those with positive early experiences should have receptivity to the belonging intervention’s messages. However, among those students, only those who must be reminded that belonging is possible will have responsiveness in terms of academic achievement. This situates Disengagers as most likely to benefit from the intervention, as they have much ground to regain with their engagement in school.
Context and Personal History in Wise Belonging Interventions
The focal intervention in this study is part of a constellation of social-psychological interventions considered psychologically “wise”—because they help enhance people’s ideas about (or make them “wise to”) how individuals and places function (Walton & Wilson, 2018). Among these interventions, it is not always clear why, where, and for whom they work (Walton & Yeager, 2020; Yeager et al., 2018). For example, to function properly, wise interventions must treat adolescents with dignity and respect (Yeager et al., 2018). When they do, they succeed by motivating participants to make positive changes—provided change can happen (Yeager, Romero et al., 2016). Even then, however, context threatens an intervention’s success. For example, adolescents report higher belonging when they enjoy the school context and find it useful (Gillen-O’Neel & Fuligni, 2013). Yet enjoyment and utility come not only from the setting but also from people’s relationships and experiences there. Convincing adolescents they can belong is thus an “empty charade” if not accompanied by the respect and care of others (Walton & Brady, 2017, p. 287). Walton and Yeager (2020) further explain how social context shapes adolescents’ psychological capacities (or “affordances”) to make positive change, through the metaphor of “seed and soil.” As they explain, the psychological change made by an intervention is simply a “seed” planted in an individual, and just as a seed needs the right soil to grow, individuals need the right contextual affordances for the intervention’s change to flourish.
Such psychological changes occur beyond the immediate context, however. How people have been perceived, labeled, and treated in the past may also affect how they think about a wise intervention’s message, beyond how they feel in their current setting. A seed needs proper conditions to grow. But where has the seed come from, what places it there, and how? In the same way, the psychological change the seed represents in Walton and Yeager’s (2020) metaphor depends on past situations not always obvious to researchers.
Unknowns about how a person’s history affects wise belonging interventions may be explained through receptivity and responsiveness (Figure 1). Receptivity affects the intervention in question through teachers’ perceptions of students’ past classroom behaviors. Adolescents with positive past school experiences have more receptivity to messages assuring them that worry about belonging is temporary than do their peers with negative ones, because the former likely begin school receiving positive feedback from teachers that their behaviors are aligned with the school’s expectations (Alwin & Arland, 1984; Bourdieu & Passeron, 1990; Calarco, 2018; Lareau, 2003). Those who instead receive feedback that their behaviors do not align with school values and expectations have less receptivity to later belonging messages because they have likely been conditioned from an early age to believe that they do not fit in at school.
Receptivity and responsiveness to intervention. Receptivity refers to an individual’s belief in the intervention’s message. Responsiveness refers to the changes an individual makes due to the intervention’s message.
However, not all receptive to an intervention’s message are also responsive. Unresponsiveness could occur because the intervention happens in a place unaccommodating to belonging (Walton & Brady, 2017; Walton & Yeager, 2020) or when a person cannot change in the desired direction (Yeager, Romero et al., 2016). Adolescents who receive consistently positive feedback from teachers (i.e., those who teachers see as “Consistent Engagers”) may be receptive but not responsive to messages resolving worry about belonging in terms of academic achievement, because they likely regard school fondly and may have few worries about fitting into the new setting. Thus, Consistent Engagers likely have receptivity but not responsiveness, even if they believe the intervention’s message and have room for greater school success. This means responsiveness is not the same as a ceiling effect on an intervention’s outcome of interest. Even when individuals still have room to increase their academic achievement, they may yet be unresponsive to an intervention targeting worry about belonging. Put simply, if underperforming students are confident they will eventually feel like they belong in middle school, worry about belonging is likely not the reason they could perform better academically—and resolving worry about belonging may thus not lead to greater success in school.
Similarly, those with relatively negative experiences early in school, but better ones over time (i.e., who teachers see as “Rising Engagers”), may have receptivity to the intervention because their increasingly positive school experiences reinforce messages resolving worry about belonging. These students might also be unresponsive, however, because they have likely already begun to regard school positively over the years, and the intervention does not improve that progress. By contrast, those who instead have positive experiences when they begin school, but have more negative experiences over time, may be considered by teachers as “Disengagers.” Disengagers are poised to be both receptive and responsive to these messages of the belonging intervention. Teachers used to regard them positively, and they likely used to feel belonging as a result. But because teachers view them less favorably over time, Disengagers stand to benefit academically from resolving worry about belonging in a positive direction. Thus, the intervention may help Disengagers reappraise their worry about belonging by evoking how positively teachers perceived them earlier on in school. The remaining type of students in this conceptual framework consists of those with few positive school experiences over time (i.e., viewed by teachers as “Nonengagers”). Nonengagers may have neither receptivity nor responsiveness to the belonging intervention, because they have likely never felt belonging in school—in part due to consistent negative feedback from educators about their classroom behaviors. Thus, Nonengagers may not believe the message based on their negative experiences.
In sum, recent theory (i.e., Walton & Brady, 2017; Walton & Yeager, 2020; Yeager, Romero et al., 2016, 2018) suggests the immediate context can influence whether a wise intervention succeeds. However, only knowing the qualities of the current setting and how students feel in it may not clarify which students have both receptivity and responsiveness. One way to understand receptivity and responsiveness to the intervention in question is to understand how teachers’ perceptions of classroom behaviors early in schooling change how adolescents come to regard the messages in the intervention. Such early repeated reports of behavior may provide the needed data to identify those with past experiences that make them receptive and responsive to the belonging intervention.
The Current Study
I revisit experimental data from a wise belonging intervention reported by Borman and colleagues (2019) to understand how receptivity and responsiveness can influence an intervention’s success. Based on an intervention fielded by Goyer and colleagues (2019) and adapted from one made for college-aged students (Walton & Cohen, 2007), the intervention’s exercises highlight two key aspects of school belonging: worry about the work and social aspects of middle school. Borman and colleagues (2019) find robust main treatment effects on postintervention sixth-grade grade point average (GPA; d = 0.09), fewer incidents of office disciplinary referrals (d = −0.14), decreased school absence (d = −0.13), and increased well-being across multiple self-report measures compared with those in the control group. A direct-replication follow-up study involving 2,000 middle school students in a different school district confirmed main effects on GPA (d = 0.06; Pyne & Borman, 2020).
Research on many kinds of psychologically wise interventions helps explain how the belonging intervention in question works. The intervention seeks to normalize participants’ worry about belonging in a new school because social threats have less impact when people understand worry is common and short-lived (Walton & Brady, 2017). In these and other similar interventions, resulting positive attitudes likely then afford participants greater focus on social and academic pursuits rather than worrying about fitting in (Beilock et al., 2017). As engagement in school increases, affected students miss less school, or act out less when they do attend, as they align with the academic and social demands of educators (Okonofua, Paunesku, & Walton, 2016). When higher achievement follows an intervention, it should reinforce positive school attitudes recursively (Yeager & Walton, 2011). Improved achievement leads to sustained positive school attitudes and better relationships, which feed into even higher achievement (Beilock et al., 2017; Okonofua, Walton, & Eberhardt, 2016; Walton & Wilson, 2018).
However, neither Borman and colleagues (2019) nor Pyne and Borman (2020) find effect heterogeneity by gender, race or ethnicity, or participation in subsidized school meal programs. Undetected moderator effects in both of those large-scale studies run counter to college-student-aged belonging intervention findings, which claim the intervention should work mostly for socioeconomically disadvantaged students or those from historically underserved racial and ethnic groups (Brady et al., 2020; Murphy et al., 2020; Walton & Cohen, 2007, 2011; Yeager, Walton, et al., 2016). In addition, those studies report moderate to large effect sizes in these subgroups of participants, reporting Cohen’s d values from 0.25 to 1.19 on academic and labor market outcomes of disadvantaged and marginalized college students. Borman and colleagues (2019) point to the more universal nature of the middle school transition as an explanation. Unlike worry or uncertainty when transitioning to college—which only some adolescents experience—nearly all adolescents attend middle school and contend with its many social and developmental challenges (see also Atteberry et al., 2022; Eccles, 2004).
Although more universal than moving to college, adolescents’ challenges with the middle school transition should still vary, meaning a wise intervention addressing worry about belonging should affect some adolescents differently than others. I draw on multiple teachers’ perceptions of early elementary classroom behaviors to assess how they can influence students’ later receptivity and responsiveness to the intervention’s messages addressing worry about belonging following the transition to middle school. Educators are powerful actors in schools who define what qualifies as good and bad behaviors—and their assessments of behavior have real, tangible consequences for students’ school success (Carter, 2005; Fine, 1991; Okonofua, Walton, & Eberhardt, 2016; Pyne, 2019; Rios, 2011). Thus, teachers’ perceptions of a student’s classroom behaviors over time represent preexisting, prevailing, and at times adversarial views educators have of students in schools (Oknonofua, Walton, et al., 2016; Pyne & Musto, 2022). Focusing on teacher perceptions of student behavior over time effectively differentiates participants by the starting points and trajectories of those early behavioral reports.
I focus my study on each participant’s GPA as the dependent variable of interest, in keeping with the focal and policy-relevant achievement outcomes of prior middle school belonging interventions (e.g., Borman et al., 2019; Pyne & Borman, 2020). I propose testing two main hypotheses based on the theoretical framing above:
Method
Participants
The original study’s intent-to-treat (ITT) sample includes 1,408 sixth-grade students recruited across all 11 middle schools in the participating district who returned signed parental consent and student assent forms prior to implementing the first intervention exercise (a 75% district-wide response rate). In the ITT sample, 702 students are assigned to the intervention group and 706 students to the control group, block-randomized within the district’s 11 middle schools. The current analytic sample retains those who remain in the district through their eighth-grade year. Attrition from the ITT sample results in the removal of 228 adolescents (N = 1,180); but the control–treatment difference in attrition from the ITT sample is negligible (χ2 = .04, p = .85). I classify students with missing classroom behaviors from kindergarten to third grade in a category called “Early Movers.” Nearly all leaving the district by eighth grade are Early Movers, who by definition have missing or incomplete data on early school behaviors.
Table 1 displays participants’ sixth-grade descriptive statistics. In the first column representing all students in the analytic sample, 47% participate in the district’s free or reduced-price lunch program (FRL) and just under half are female. White students comprise half of the sample, Black students a fifth, Latinx students another fifth, and Asian students comprise 7% of the sample. One percent report another race or ethnicity. Fourteen percent receive special education services and 15% are English language learners. The average score on the state’s fifth-grade math test is 496 (SD = 55).
Descriptive Statistics of Sample
Note. This table displays descriptive statistics of participants at the time of the intervention in sixth grade, unless otherwise indicated (N = 1,180). FRL = free or reduced-price lunch participation; GPA = grade point average.
Percentages in this row do not include “Early Movers.” Early Movers comprise 23% of the total sample.
Because the theoretical framework I propose suggests treatment effects may be concentrated within certain teacher-reported behavioral groupings, my main analyses will focus on differential characteristics among those groups, whether within-group treatment–control covariate balance appears maintained, and whether treatment effects are influenced by pretreatment patterns. Thus, Table 1 also reports descriptive statistics within the four teacher-reported elementary school classroom behavior groups I describe in the next section. Among those with data through early elementary school, 39% are Consistent Engagers, whose teachers have consistently perceived their classroom behaviors as high from kindergarten through third grade. They are disproportionately higher income, female, and White compared with the average student in the sample, and are much less likely to be Black, Latinx, in a special education program, or an English language learner compared with the full sample. Their fifth-grade math test scores are typically about half of a standard deviation above the sample average. Their control group’s average sixth-grade GPA is 3.60 (SD = 0.40).
Rising Engagers, whose teachers perceive their classroom behaviors as relatively low in kindergarten but increasing by third grade, comprise a third of the sample. Disengagers are those who begin kindergarten with high teacher perceptions of classroom behavior that decline over time, and comprise just under a fifth of the sample for whom full elementary school data are available. On average, Rising Engagers and Disengagers look very much like the typical student in the sample based on race/ethnicity, family income, gender, and English language learner status. Both groups are roughly half White, a quarter Latinx, a 10th Asian, and a 100th are of another race or ethnicity. A fifth of Rising Engagers and 14% of Disengagers are Black. Their typical fifth-grade spring math test scores are average for the sample and half of a standard deviation below the average Consistent Engager’s score. The average sixth-grade GPA in both Rising Engager and Disengager control groups is about 3.00 (SD = 0.66 and 0.71, respectively).
Finally, Nonengagers, whose teachers consistently perceive them as having poor classroom behaviors from kindergarten through third grade, comprise roughly a 10th of the sample observed since kindergarten and are 64% low income, 64% male, 36% Black, and 29% in a special education program. White students comprise a third and Asian students comprise 4% of Nonengagers. Twenty-seven percent are Latinx and a fifth are English language learners. Nonengagers’ typical fifth-grade math test scores are about 60% of a standard deviation below the sample average and over a full standard deviation below the average Consistent Engager’s score. Control group average sixth-grade GPA is 2.56 (SD = 0.68).
Using a power of (1 – β) = 0.80, the minimum detectable effect size (MDES) of the GPA outcome variable among Disengagers (n = 159) is 0.22 SD (0.15 grade points), among Consistent Engagers (n = 359) is 0.15 SD (0.10 grade points), among Rising Engagers (n = 302) is 0.16 SD (0.11 grade points), and among Nonengagers (n = 100) is 0.28 SD (0.20 grade points). The MDES to estimate the overall average effect across the entire analytic sample (N = 1,180) is 0.08 SD.
Procedure
This was a double-blinded randomized controlled trial implemented by sixth-grade teachers in class and as a part of instructional time. Each school had a site liaison who coordinated but did not administer the exercises to students. The site liaison received exercise materials from the research team and distributed the materials to teachers. Teachers were unaware of treatment condition and students received packets with identical covers. Teachers administered exercises to sixth-grade students twice. The first exercise happened in late September and manipulated students’ test-taking worry in middle school. Page 1 of the treatment exercise informed participants of a seventh-grade student survey asking respondents to express how they felt the year prior about taking important tests. The exercise then informed participants that survey results suggested almost all seventh graders worried about tests starting in sixth grade, and almost all now worry much less about test taking in seventh grade. The second page displayed three vignettes, ostensibly by some of those seventh graders surveyed, explaining their past worries about tests and how they currently worried less. On the third page, students received three writing prompts: (a) “Name 1 or 2 reasons
The second exercise, addressing worry about fitting in among peers, was administered in November. On the first page was a summary of results from a seventh-grade survey suggesting that almost all current seventh-grade students had worried about fitting in and belonging at their new school starting in sixth grade and that almost all of them eventually said that they knew they fit in or belonged. The second page displayed three vignettes as examples of seventh-grade responses to the survey. On the third page, students received two writing prompts: (a) “Name 1 or 2 reasons
Measures
Dependent Variables
The main outcome of interest is the district’s calculation of GPA, in keeping with previous work using the same middle school belonging intervention (e.g., Borman et al., 2019; Pyne & Borman, 2020). The GPA variable is a standardized average of grades in School Terms 2 through 4 when the student is in sixth grade and School Terms 1 through 4 when in seventh and eighth grades—all of which occur after participants complete both exercises. In addition, four survey item outcomes help test whether the intervention’s manipulation of reducing worry about middle school was consistent across the different groups. Each item is measured on a 5-point Likert-type scale in which 1 = worried very little and 5 = worried a lot. Two “Worried Before” items (one for each of the two exercises) measure the degree of worry participants believe last year’s students have at the beginning of their middle school experience. In theory, treatment group participants should report greater worry at the beginning of middle school than control group students because the intervention should activate worry by making belonging uncertainty salient. However, that worry should be substantially reduced by the end of each exercise. Thus, two “Worried Now” items measure the degree of worry participants believe last year’s students have at the end of their first year of middle school. Treatment group participants should report less worry on these items than do control group students as a sign that the manipulation is fully activated. Ultimately, the main manipulation of interest in each exercise is the change in worry students perceive, or the difference between “Worry Now” and “Worried Before” items for each exercise.
Treatment and Moderator Variables
The treatment variable is a binary indicator of whether a participant was randomly assigned to the treatment condition (“1”) or control condition (“0”). The treatment moderator of interest is teacher perception of students’ elementary school classroom behaviors, measured over time using multiple teacher evaluations. Elementary school teachers in the district report on 13 three-point classroom behavior items as part of report cards sent home twice a year (e.g., completes assignments on time, demonstrates self-control, works independently, participates in classroom activities, persists in tasks until completion, works cooperatively with others; α = .90). Response categories ask teachers to report their perceptions of how often each child engages in the specified behavior: 1 = rarely, 2 = some of the time, 3 = most of the time. Exploratory factor analysis suggests all items load onto a single factor (see Supplementary Table A1 in the online version of the journal).
These teacher perceptions of classroom behavior are highly skewed, making their use as continuous measures problematic. For example, teachers perceive about 55% of the sample’s students as having a perfect kindergarten classroom behavior score of “3” on this averaged measure of the 13 items, while teachers perceive another 20% as having a score between “2.85” and “2.95.” In third grade, teachers perceive that 73% of students have a score of 2.85 or higher. Even when log-transformed, the data remain highly skewed. Using these items as measures of adolescents’ school histories is thus limited, having substantial ceiling effects and limited variability across the three response categories per item (e.g., most children’s average score suggests they engage in the rated behaviors more frequently than “some of the time” in class). However, when examining trends over time to consider both kindergarten starting points and early grade trajectories, these measures of teacher perceptions of classroom behavior are useful for creating a four-category measure consistent with the receptivity–responsiveness framework.
I create the four-category measure of teacher perceptions of classroom behavior in three steps using actual observed teacher report card scores at the beginning of kindergarten and the end of third grade. First, using beginning-of-year kindergarten teacher evaluations, I create an intercept cutoff for initial teacher perceptions of classroom behaviors. Students above the cutoff have “high” teacher perceptions of classroom behaviors in kindergarten. This threshold (an average of 2.85 over the 13 items) is a half of a standard deviation from the top score of “3” and is the same as saying that teachers believe students are following classroom expectations “most of the time” at the beginning of kindergarten by receiving the highest possible score on 11 of 13 items rated. Second, I subtract beginning-of-year kindergarten teacher reports from end-of-year third-grade teacher reports to determine the overall slope of students’ classroom behavior scores in early elementary school. Finally, using the intercept cutoff along with the slope calculations completes construction of the four categories. Those above the kindergarten cutoff with a null or positive slope are students whom teachers perceive as “Consistent Engagers,” while those below the kindergarten cutoff with a positive slope are those teachers perceive as “Rising Engagers.” Those above the initial-year cutoff with a negative slope are considered “Disengagers,” while those below the cutoff with a null or negative slope are “Nonengagers.” 2
Figure 2 shows observed differences in reported teacher perceptions of student behavior for students in the analytic sample. Notably, although the scale ranges from a low of “1,” a midpoint of “2,” and a high of “3,” all the lines and trajectories displayed in Figure 2 fall within the top half of that range (i.e., between “2” and “3”; see Supplementary Figure A1 in the online version of the journal for a fully scaled graph). For example, students who fall relatively low on the scale still engage in the rated behaviors more than “some of the time.” This limited range of values suggests teachers believe most students are behaving as expected a good deal of the time they are in school, which is perhaps unsurprising as we have no reason to believe teachers think many children “rarely” behave as expected in class. Even with this limited range, however, diverging trends over elementary school may still differentiate students in meaningful ways. For example, relatively small differences in childhood can grow into much larger differences in early adolescence—the developmental stage at which these students are being asked to participate in the belonging intervention.
Patterns of teacher perceptions of student behavior through third grade.
Despite its limited variation in group trajectories, we can see that the lines in Figure 2 correspond remarkably well to the four proposed engagement groups, and follow the theorized patterns of trajectories. For example, Consistent Engagers stay consistently near a perfect rating of “3” from kindergarten through third grade, while Disengagers drop from a score of nearly 3 to about 2.73 by the end of third grade. Even more dramatically, Nonengagers start at about a 2.68 on these scales and drop to a 2.35 score by the end of third grade (see also subgroup estimates reported in Table 1). In results not shown, I find that making reasonable changes to the kindergarten cut points (e.g., an increase or decrease in the cut-point by a 1/2 SD) does little to change the trends presented in Figure 2, or in the main results I present below.
Covariates
Other independent variables included to increase precision of treatment effect estimates include pretreatment indicators of race or ethnicity, gender, FRL participation, special education status, English language learner status, and prior achievement. I code race or ethnicity the same as Borman and colleagues (2019), in which Black, Latinx, and other non-Asian minority students comprise a category called “historically underserved racial minority students,” with White and Asian students comprising a second category. “Female” is coded with a “1” for female and “0” for male. I code students participating in the district’s FRL a “1” and nonparticipating students “0.” Preintervention student achievement comes from fifth-grade state standardized test scale scores, as the district does not record student GPA prior to sixth grade.
Strategy and Design
My main aim is to identify which groups of adolescents have both receptivity and responsiveness to the belonging intervention in question. However, the sampling design used in the original study does not account for subgroup analyses by teacher-reported classroom behavior types, and randomization was not blocked by K–3 teachers’ perceptions of participants classroom behavior patterns. Thus, subgroup analyses require an assessment of treatment–control balance on baseline and time-invariant observables. I assess balance on each baseline variable first using the normalized difference approach from Imbens and Rubin (2015), represented as follows:
in which µt is the preintervention mean for the treatment group, µc represents the preintervention mean for the control group, σt2 is the variance associated with µt and σc2 is the variance associated with µc. What Works Clearinghouse (WWC) defines
Next, I examine the treatment effects of the intervention on manipulation check items and GPA across teacher-perceived classroom behavior categories in the following model:
in which the outcome Y for adolescent i in grade g is a function of the treatment indicator (Treat), an indicator of the behavioral engagement group they are a part of (Group, with indicators across K – 1 groups and with Disengagers as the reference group), interactions between treatment indicator and each K – 1 teacher-reported classroom behavior type, other covariates, and randomly distributed error. The model includes controls for prior achievement, FRL participation, racial or ethnic group, gender, special education designation, and limited English proficiency (all represented in vector
Finally, I present a series of alternative models using additional teacher-reported behavior moderators. I test point-in-time indicators of classroom behaviors reported among kindergarten and third-grade teachers separately, reasoning that point-in-time measures may be more efficient indicators of differential treatment effects than teacher perceptions requiring multiple years of elementary school data. In addition, if kindergarten or third-grade teacher perceptions of classroom behaviors are as good or better signals of responsiveness to treatment than patterns of teacher reports over all of early elementary school, that would weaken claims about early elementary school patterns influencing receptivity and responsiveness to intervention. Thus, in another alternative model, I remove the four behavioral engagement categories from the models and replace them with binary indicators of (a) kindergarten teacher reports of classroom behavior and (b) third-grade teacher reports of classroom behavior. Any student who is half a standard deviation below perfect (33% and 26% of students, respectively) is considered “lower engagement” by teachers and marked “1,” while any student within a half of a standard deviation of a perfect score is considered “high engagement” and marked a “0.” In a final alternative model, I consider a moderator variable that simply averages each student’s kindergarten and third-grade teachers’ behavior scores.
Results
Full-sample balance checks suggest there is adequate balance based on patterns of teacher-reported behavioral engagement categories, although the treatment group has slightly fewer Consistent Engagers and Rising Engagers and slightly more Disengagers and Nonengagers than the control group. Treatment and control group balance differs notably by third-grade behavior score (∆ tc = −.10) and gender (∆ tc = .10), which WWC characterizes as adequate balance but requiring statistical adjustment to satisfy equivalence. All other pretreatment variables are well-balanced (see Supplementary Table A2 in the online version of the journal for details).
Manipulation Check Treatment Effects by Behavioral Engagement Type
I next consider manipulation checks within each of the four groups, assigning “Disengagers” as the reference group and accounting for all reported covariates and school fixed effects (Table 2). The results are consistent with theoretical expectations of how the intervention should affect participants. For example, treated Disengagers report having more heightened worries at the beginning of sixth grade than control group Disengagers in both exercises (b = .15, p = .366 in Exercise 1; b = .46, p = .005 in Exercise 2) and then report lower levels of worry than control group Disengagers when considering belonging in seventh grade (b = −.41, p = .012 in Exercise 1; b = −.08, p = .631 in Exercise 2). Despite a relatively small sample size, two of the four estimates reveal statistically significant results. Most importantly, among Disengagers are large and statistically significant manipulation effects on the change in worry from “before” to “now” on both exercises (b = –.44, p = .008 in Exercise 1; b = −.42, p = .009 in Exercise 2), suggesting effective exercise manipulations in reducing worry in the treatment group much more so than in the control group. No interaction effects are statistically significant in that model, suggesting no manipulation effects among other groups show detectable differences from effects among Disengagers.
Manipulation Check Effects by Engagement Type
Note. The dependent variables are standardized manipulation check items; therefore, “Δ Worry” coefficients do not equal the difference between “Worried Before” and “Worry Now” coefficients because the scale scores are first differenced prior to standardization. Due to some noncompletion of manipulation checks following the implementation of the exercises, N = 1,068. Models include controls for race/ethnicity, gender, free or reduced-price lunch participation, special education designation, English language proficiency, prior achievement, and school fixed effects. Standard errors are in parentheses.
p < .05. **p < .01. ***p < .001.
GPA Treatment Effects by Behavioral Engagement Type
To measure the effect of adolescents’ school histories on receptivity and responsiveness to the belonging intervention, I investigate the degree to which the intervention affects students’ postintervention middle school GPA by teacher perceptions of early elementary school classroom behavior. Table 3 shows full model results for the main analyses and Figure 3 shows post hoc linear combination test results from each individual grade-level model, the latter of which are treatment–control differences within each of the four behavioral engagement groups. In Figure 3, positive coefficients above the horizontal bar at zero imply that treatment group students in a given behavioral engagement group have higher postintervention GPAs than control group students in the same group.
Heterogeneous Treatment Effects on GPA From Sixth Through Eighth Grades (N = 1,180)
Note. The dependent variable is standardized GPA. Models also include controls for school fixed effects. Standard errors are in parentheses. GPA = grade point average.
p < .05. **p < .01. ***p < .001.
Treatment effects on GPA from sixth through eighth grades, by types of teacher perceptions of student behavior (N = 1,180).
Results support Hypothesis 1, suggesting that only those who teachers perceive as Disengagers have statistically significant intervention effects on sixth-, seventh-, and eighth-grade GPA. While typical treatment group Disengagers experience a third of a standard deviation gain in GPA over control group Disengagers by the end of sixth grade (b = 0.31, SE = 0.01, p = .001) and nearly half a standard deviation gain by the end of seventh grade (b = 0.46, SE = 0.10, p < .001), the average treatment effect weakens to about a fifth of a standard deviation in eighth grade (b = 0.22, SE = 0.11, p = .037). When pooled and averaged across all three grade levels, the average GPA treatment effect among Disengagers through middle school is 35% of a standard deviation (b = 0.35, SE = 0.10; p < .001).
By contrast, average treatment group students who teachers perceive as Consistent Engagers experience no positive effects of intervention on GPA, while treatment effects for average students perceived as Rising Engagers are relatively small and not distinguishable from zero. Students considered Nonengagers by teachers in the treatment group have higher but statistically insignificant GPAs over their control group counterparts in sixth grade—although the estimate is very close to the standard p < .05 threshold for statistical significance (b = 0.24, SE = 0.12, p = .053). However, descriptive GPA differences among Nonengagers quickly diminish by the end of seventh grade (b = 0.09, SE = 0.13, p = .484) and vanish by the end of eighth grade (b = 0.00, SE = 0.14, p = .971).
Treatment effects among Disengagers are also meaningfully different from those of treated students in the other groups. Specifically, between-group tests suggest the Disengager–Consistent Engager treatment effect difference is statistically significant in sixth grade (b = 0.34, SE = 0.12, p = .004), seventh grade (b = 0.51, SE = 0.12, p < .001), and eighth grade (b = 0.26, SE = 0.13, p = .049). The Disengager–Rising Engager difference in treatment effects is statistically significant in sixth grade (b = 0.26, SE = 0.12, p = .030) and seventh grade (b = 0.44, SE = 0.12, p < .001) but not in eighth grade (b = 0.12, SE = 0.13, p = .38). Finally, the Disengager–Nonengager difference in treatment effects is not statistically significant in sixth grade (b = 0.07, SE = 0.16, p = .64) or in eighth grade (b = 0.22, SE = 0.17, p = .21) but is statistically significant in seventh grade (b = 0.37, SE = 0.16, p = .021). When averaging GPA across all three grade levels, treatment effects for Disengagers are higher than and statistically significant compared with Consistent Engagers (b = 0.39, SE = 0.12, p = .001) and Rising Engagers (b = 0.28, SE = 0.12, p = .018), but not Nonengagers (b = 0.25, SE = 0.16, p = .115).
Alternative Model Specifications
I next address Hypothesis 2 by reporting potential heterogeneous treatment effects in alternative models using point-in-time or averaged teacher perceptions of classroom behaviors. Table 4 suggests there are no statistically significant or substantively large treatment-by-moderator interaction effects of the intervention on sixth-grade GPA for any of the groups differentiated. Column 1 suggests that the sixth-grade treatment effect coefficient for those whose teachers believe they have low kindergarten behavioral engagement (i.e., the “Treatment” row) is 0.06 SD and not statistically significant (b = 0.06, SE = 0.05, p = .24). The additional treatment effect for those whose teachers believe they have high kindergarten classroom behavior (i.e., interaction effect row) is 0.00 SD (b = 0.00, SE = 0.08, p = .98), implying that teachers’ kindergarten behavior reports alone do not explain the treatment effect over and above the main effect coefficient. Column 2 shows interaction effects based on high or low third-grade teacher perceptions of classroom behavior. The treatment effect coefficient, representing low teacher perceptions of third-grade classroom behaviors, is 0.08 SD and is not statistically significant (b = 0.08, SE = 0.06, p = .20). The treatment-by-low third-grade classroom behavior coefficient is small and not statistically significant (b = −0.04, SE = 0.07, p = .55). In Column 3, showing averaged kindergarten and third-grade teacher perceptions of classroom behaviors, there is much the same trend as in Column 2, with a null treatment effect coefficient (b = 0.05, SE = 0.06, p = .41) and interaction coefficient (b = −0.04, SE = 0.09, p = .67). All results on alternative specifications are consistent in seventh- and eighth-grade GPA as well (results not shown).
Alternative Classroom Teacher Perception Moderators of Sixth-Grade GPA
Note. These models exclude Early Movers, resulting in a sample size of n = 923. Models also control for gender, special education status, English language proficiency, prior achievement, and school fixed effects. For KG and Grade 3 engagement behaviors, 3 is the highest average score a student can receive out of 13 teacher-reported prompts. Fifty-five percent of students in the sample received the highest score in third grade. GPA = grade point average; KG = kindergarten.
p < .05. **p < .01. ***p < .001.
Even if focusing only on the combined point estimates in each of these models, no effects are either statistically significant or as large as the 0.31 SD effect size observed among Disengagers in sixth grade. Null results imply none of the alternative moderators detect discernible treatment effects beyond the main treatment effects of the intervention. Patterns of teacher perceptions of classroom behaviors from kindergarten to third grade, represented as four teacher-reported categories in Figure 2, are thus much more informative moderators for the effects of treatment on these outcomes than several other plausible point-in-time moderators shown in Table 3. I report null moderator treatment effects by race/ethnicity, FRL participation, and gender in Supplemental Table A3 in the online version of the journal.
Additional Robustness Checks for the Disengager Subsample
Due to the quite sizable treatment effects among the relatively small group of Disengagers (n = 159), I perform several additional robustness checks only on the Disengager subgroup. First, I test treatment–control balance on preintervention observables among Disengagers to test for pretreatment imbalances in the subsample. Normalized difference indicators between treatment and control Disengagers suggest fifth-grade math test score, kindergarten and third-grade behavior score, gender, and Black or Latinx identification satisfy baseline equivalence, while English proficiency, subsidized school meal participation, and special education status are adequate but may require statistical adjustment to satisfy baseline equivalence. In addition, p values for all baseline indicators are not statistically significant, and all prior achievement and behavior indicators have p values of .65 to .75. These results suggest no detrimental imbalances are observed in the Disengager subsample (see Supplementary Table A4 in the online version of the journal).
Another concern is that covariate relationships in the full sample and Disengager subsample are different, which might improperly influence treatment effect estimates among Disengagers. Supplemental Table A5 in the online version of the journal presents model results for Disengagers only, suggesting covariate relationships in the full model are marginally different from those in the Disengagers subsample, as Disengager treatment coefficients vary by at most 3% of a standard deviation compared with analyses of the full sample. Treatment coefficients in the sixth, seventh, and combined sixth- to eighth-grade GPA outcomes remain statistically significant at the p < .05 level, while the eighth-grade treatment coefficient is statistically significant at the p < .10 level. Relatedly, Supplemental Table A6 in the online version of the journal tests whether values of covariates in these models drive results among Disengagers by artificially altering effect size estimates. Results suggest covariates correct treatment effect sizes downward by a modest 5% to 6% of a SD for GPA at each grade level and averaged across all three grade levels.
I next account for potential outliers in the Disengager subsample by removing observations with a GPA greater than 2 SDs from the mean. Supplemental Table A7 in the online version of the journal shows removing potential outliers produces similar results to those presented in Supplemental Table A5 in the online version of the journal. Finally, I consider whether controlling for past teacher perceptions influences the treatment effects among Disengagers. Supplemental Table A8 in the online version of the journal demonstrates that including kindergarten and third-grade behavior scores as covariates barely changes any of the GPA treatment effect estimates.
Discussion
I have investigated the degree to which teacher perceptions of adolescents’ past school behaviors help explain who has receptivity and responsiveness to messages resolving worry about belonging from their peers following the transition to middle school. I have done so by reanalyzing data from a brief, randomized experiment implemented at-scale and seeking to help adolescents reappraise feelings of worry or uncertainty about the difficult transition to middle school (see Borman et al., 2019). Participants read about how older peers resolve their belonging uncertainty in a positive direction, and they then reflect in writing on those stories and their own feelings of uncertainty or worry. This process can help sixth-grade adolescents reappraise worry they encounter in the new middle school environment (Walton & Brady, 2017). Although these teacher-reported measures may be both subjective and based on real student actions, educators’ beliefs affect adolescents’ feelings about school greatly (Okonofua, Paunesku, & Walton, 2016; Okonofua, Walton, & Eberhardt, 2016; Pyne, 2019).
Longitudinal measures of teacher perceptions of student behavior differentiate four types of students—those teachers perceive as Consistent Engagers, Rising Engagers, Disengagers, and Nonengagers (see Figure 2). Teachers perceive Consistent Engagers as those students having consistently high levels of classroom behaviors from kindergarten through third grade, while Rising Engagers are perceived as having initially low but increasingly positive classroom behaviors over the same period. Disengagers are those perceived as having initially high but declining reports of classroom behaviors, while Nonengagers are perceived as having relatively low classroom behaviors across the early elementary school grades.
Results suggest that the intervention has moderate effects on the GPA of adolescents who were perceived by teachers as Disengagers in elementary school. This racially and socioeconomically diverse subgroup of students experiences benefits as they move from sixth through eighth grade—with about a 0.35 SD increase in GPA across the three grades due to the intervention. I find no consistent or statistically distinguishable effects on the GPAs of similarly diverse students whose teachers consider them to be Rising Engagers, and the intervention appears to do nothing for those perceived as Consistent Engagers, who are predominantly White and more socioeconomically privileged than the sample as a whole. The receptivity–responsiveness framework I propose suggests that null effects among Rising and Consistent Engagers may be due to a lack of responsiveness to the intervention, not a lack of receptivity to it. That is, engaged students are likely quite receptive to ideas about fitting in and doing well at a new school, as their past school experiences likely reinforce that fitting in and doing well in school are possible—and potentially expected. However, because their continued success in school is unlikely to hinge on positively resolving their worry about fitting in, they are likely not responsive to the intervention.
Null effects on GPA among Rising Engagers and Consistent Engagers are also not likely due to ceiling effects. Drawing on descriptive GPA means by group (Table 1), an average control or treatment group Rising Engager has a GPA between 3.03 and 3.08 (SD ~ 0.66). This is roughly the same average as among control group Disengagers (GPA = 2.96, SD = 0.71). Given the treatment appears to improve Disengagers’ GPA by about 0.25 grade points from sixth to eighth grade, such a shift or smaller should be possible and detectable among Rising Engagers as well. Although ceiling effects are more plausible among Consistent Engagers, they are still unlikely. An average control or treatment group Consistent Engager has a sixth-grade GPA of about 3.60 (SD ~ 0.40). About 5% of Consistent Engagers have a perfect 4.00 sixth-grade GPA and another 20% have a sixth-grade GPA between 3.9 and 3.99. This leaves ample room for the remaining 75% of Consistent Engagers to make GPA gains at or above the MDES of 0.10 grade points (0.15 SD). Instead, treatment coefficients among Consistent Engagers are effectively zero.
Findings among students who teachers perceive as Nonengagers are less clear, however. While the sixth-grade treatment effect among Nonengagers is a quarter of a standard deviation in size, it is not statistically significant. However, the reported p value for the estimate is only negligibly higher than the conventional threshold, at p = .053, and the effect may be undetectable simply because the subsample is slightly underpowered (MDES = 0.28 SD). If choosing to interpret the treatment effect among Nonengagers, it is notable for two reasons. First, the magnitude of the treatment effect among Nonengagers in sixth grade is only slightly smaller than the treatment effect among those considered Disengagers in sixth grade. Second, the effect among Nonengagers greatly diminishes by seventh grade and vanishes by eighth grade. It may be useful to think of these patterns among Nonengagers through Walton and Yeager’s (2020) framing of the importance of educational contexts for interventions to be effective. Just as seeds need the right soil to grow, effective interventions need the right environment to succeed. In this intervention, those who teachers consider to be Nonengagers may have a great seed but the wrong soil, meaning they may be initially responsive to the intervention, much like Disengagers are, but the intervention’s message of reappraising and eventually overcoming worry does not adequately materialize in their daily school experience—and over time, Nonengagers eventually reject the intervention’s messages.
Alternative analyses support the study’s overall findings. Specifically, teacher perceptions of elementary school classroom behavior patterns do a better job detecting a GPA treatment effect compared with teachers’ perceptions of classroom behavior measured only at either kindergarten or third grade, or when averaging perceptions across the grades. These robustness checks suggest that patterns of teacher-reported classroom behaviors over early elementary school grades predict intervention effects better than do point-in-time measures of classroom behaviors. In addition, results among Disengagers are robust to many different model specifications targeted only at this subgroup. I conclude that treated Disengagers appear to enjoy a 35% of a standard deviation increase in postintervention middle school GPA—three times larger than the main effect reported by Borman and colleagues (2019).
Moderation effects among categories of teacher-perceived classroom behaviors highlight how people’s past relationships affect receptivity and responsiveness to this brief belonging intervention. Early in life, people acquire the practices of those closest to them—their temperaments, mannerisms, vocabulary, nonverbal expressions, and more. Over time, individuals internalize the social habits and behaviors they see in those around them (Bourdieu, 1984; Elias, 1939/1994; Mauss, 1973; Merleau-Ponty, 1962). If an intervention’s setting reflects the values and beliefs of their home environment, they may not have much belonging uncertainty upon transitioning to the new setting, where those in positions of power interpret behaviors through a similar subjective lens. Thus, those perceived as showing behaviors aligned with their teachers’ expectations stand to regard the new environment more positively than those who are perceived more negatively by teachers.
The findings presented here also suggest adolescents’ school histories can affect how this belonging intervention helps them reappraise worry and adversity when transitioning to a new school. If children are well-regarded by educators and peers early on in school, they are poised to be receptive to later messages of middle school belonging. If those same children go on to have more negative encounters in school over time, they are poised to be responsive to the intervention’s messages because they have some ground to regain in academic achievement that can be influenced by resolving worry about belonging in a positive direction. Even though encounters in school shape how teachers perceive students in the Disengager group for the worse over time, those encounters do not appear to permanently change these students’ beliefs about school more generally in ways that would prevent them from improving academically. Individuals consciously or unconsciously draw on their earlier life experiences when they decide whether they believe the stories others tell about their ability to fit in (Bourdieu, 1984, 2005). Past teachers believe that students in the Disengagers group conform to the values and expectations of school early on, meaning educators’ perceptions at kindergarten entry help adolescent Disengagers believe the intervention’s messages. When peers tell Disengagers they will eventually worry less about belonging in middle school, because of their teachers’ past perceptions of them, they likely believe it.
The receptivity–responsiveness framework may also help explain the null effects among Consistent Engagers and the inconsistent treatment effects among Rising Engagers and Nonengagers. Consistent Engagers and Rising Engagers, as shown by more recent teacher perceptions of classroom behaviors, likely already know they belong, as their teachers signal to them that they do. They are thus unresponsive to the intervention in terms of academic achievement, even if they believe the intervention’s messages. And when peers tell Nonengagers—students who teachers have consistently perceived as having poor classroom behaviors—that they will someday worry less about belonging in school, it is likely they either do not believe it or are not adequately awarded for doing so.
Limitations and Future Directions
Several questions arise from this study for those hoping to help adolescents with their feelings about school. First, and most importantly: given these findings, how should future belonging intervention research proceed? The current study is exploratory, not conducted with the current research question in mind, and not preregistered prior to implementation. Preregistration accommodates exploratory analyses like the ones presented here; future studies seeking to replicate these results should consider designing preregistration protocols with hypotheses in mind about treatment effects differing by assessments of children’s behaviors.
Relatedly, randomization was not blocked by prior patterns of teacher-reported classroom behaviors. Although balance statistics are adequate for baseline equivalence of treatment and control groups both overall and within the Disengagers group, future studies may consider randomizing the intervention within each of the four categories presented here. That future work would benefit from larger subsample sizes, particularly among Nonengagers (who may have been slightly underpowered in this study). Those studies would also benefit from considering other relevant outcomes that could be related to both the intervention and past patterns of teacher-reported classroom behavior, such as school attendance and disciplinary involvement.
In addition to replications with the present study’s goals in mind, future studies may also use school histories to study how adolescents engage with the intervention’s exercises. For example, Disengagers may consider the exercises’ messages more genuine or relatable than might Nonengagers. Consistent Engagers could report that the intervention’s messages ring true and are useful for others, but do not apply to them. Work uncovering the motivations and beliefs of different groups of adolescents can be carried out first in laboratory studies—provided researchers have access to consistent records of participants’ school histories. Then, larger field studies could ask similar questions as part of closing items included in the exercises (e.g., items included with existing manipulation checks at the end of the exercises) or by studying writing responses across behavioral engagement groups.
Second, how practical is it to collect and use repeated measures of teacher-reported classroom behaviors to predict who later responds to wise interventions in middle school? Classroom behavior measures like these have been validated and used in studies of nationally representative student data (e.g., DiPrete & Jennings, 2012; Downey et al., 2019) and commonly appear in children’s report cards alongside mathematics and reading evaluations from teachers. Applying these types of teacher-reported measures—essentially number scores for what teachers perceive as children’s in-class positive behaviors—is no more complicated than calculating math or reading test score growth. Even though the reports have subjective components, what teachers think about students matters a great deal to their achievement, engagement, and feelings in school (Carter, 2005; Fine, 1991; Okonofua, Paunesku, & Walton, 2016; Okonofua, Walton, Eberhardt, 2016; Pyne, 2019; Rios, 2011). Even so, researchers who are less interested in how teacher perceptions influence academic achievement may seek out other more objective measures of actual student behavior.
Another challenge of using such teacher-reported behavior measures is that schools do not universally or consistently collect these types of evaluations in elementary school, nor are school districts generally required to do so in a standardized way for accountability purposes. This means schools could simply collect classroom behavior measures only in some grade levels, only for some groups of students, or not at all. The discretionary nature of these measures within school districts is certainly a limitation here because the school district under study does not collect teacher-reported classroom behavior measures after third grade. Not having fourth- and fifth-grade measures of teacher perceptions of students’ classroom behaviors might introduce error into the assignment of children to different behavior categories. For example, a student who does very well in kindergarten could experience life events that lead to rapid declines in teacher-reported classroom behaviors in second and third grades, followed perhaps by recovery in fourth and fifth grades. That can incorrectly categorize such participants as “Disengagers” rather than “Consistent Engagers,” due to unobserved classroom behaviors in fourth and fifth grades. However, patterns reported in Figure 2 give some confidence that those types of anomalies are uncommon; trends in classroom behavior reports from Grades K–3 are remarkably linear (especially when comparing only fall-to-fall or spring-to-spring trends in scores), and it is unclear whether teachers change their perceptions of students’ classroom behaviors in fourth or fifth grade relative to the patterns already evident in these earlier grades. In any case, these potential anomalies, which occur prior to randomization, would presumably be evenly distributed between control and treatment groups, and the misattribution would likely serve to mute the Disengager treatment effect, given the near-zero effect among Consistent Engagers.
Third, how feasible is implementing the intervention in schools? The belonging intervention is quite easy and inexpensive to administer—at about $1.35 per student for two 15-minute reading and writing exercises (Borman et al., 2019). This makes administration to all middle school–aged adolescents in a school district feasible. However, as the intervention may have little to no effect at all on the plurality of students consistently engaged throughout elementary school, researchers and educators could be tempted to find even greater cost-effectiveness by identifying subgroups of adolescents who they believe stand to benefit the most (e.g., only administering the exercises to those they perceive to be Disengagers). This choice could prove counterproductive; identifying and targeting specific subgroups of students could fundamentally change the intervention’s message. The exercises help adolescents understand that everyone feels worry or uncertainty during the transition to middle school. If targeted students were to realize only they and a small number of their peers were completing the exercises, they could understandably question not only whether those feelings of worry or uncertainty were truly “normal,” but why they were being singled out for help and remediation and not others. Studies show such design effects impact how well wise interventions work. For example, treatment effects of a growth mind-set intervention are smaller when high school students are told the exercises will help them (Yeager, Romero, et al., 2016), confirming work on the importance of more indirect and stealthy approaches (Sherman et al., 2009; Walton & Brady, 2017). Administering the current intervention to all students maintains an indirect approach. In addition, the belonging intervention may benefit adolescents transitioning to middle school beyond their GPA (see Borman et al., 2019). Denying participation based simply on moderated GPA effects may prove to be a myopic approach to helping adolescents feel less worry about transitioning to middle school—simply for efficiency’s sake.
Fourth, what—if anything—can policymakers and practitioners do to help unengaged students attach to and succeed in school? Evidence presented here suggests children who teachers perceive as unengaged are not all the same, and the strategies for addressing needs of Disengagers differ from strategies to help Nonengagers. Some who teachers perceive to disengage over time may feel encouraged by their peers’ reassurances about belonging uncertainty in middle school, because they need reminding that resolving such feelings of worry in a positive direction is possible. Others may have no reference point for fitting in at school; for them, such reassurances may seem like an “empty charade” (Walton & Brady, 2017, p. 287).
Often, federal, state, and local educational policies cannot address issues of schooling in a “one size fits all” approach. This study both affirms the complexity of implementing school belonging interventions at-scale across entire urban school districts and offers the conceptual framework of receptivity–responsiveness to consider for whom this intervention is likely to improve academic outcomes. Doing so helps clarify whether pulling policy levers to implement interventions at-scale in districts, states, and across the country are feasible and will have the expected impacts on students. This framework also helps policymakers think through whether at-scale interventions reach the populations or subpopulations they intend them to.
While this study offers reasonable bounds on the impacts of the belonging intervention, it does not clarify how to approach nonengagement—or those students who teachers perceive as having low engagement throughout school. However, results reported here still may help policymakers anticipate a wise interventions’ reach. For example, can an institution that has made someone feel rejected since childhood convince them that it will now help? Walton and Yeager (2020) write that not only must the intervention create a psychological change (i.e., “the seed”), but the setting (i.e., “the soil”) must also allow for the intervention to work (or “to grow”). The school environment perhaps could (and should) change so it is more attentive to the needs of those labeled as Nonengagers by first regaining and maintaining trust from teachers and among students (Bryk & Schneider, 2004; Okonofua, Paunesku, & Walton, 2016). But some work must fall on other social institutions. Rather than turn to our school system as a panacea for all social ills—part of the “Education Gospel” pervading discourse of inequalities (Grubb & Lazerson, 2005)—understanding and using our complex system of social institutions may improve the lives of children underserved in schools.
Fifth, how do the results reported here relate to those in many other belonging interventions, which report effects on moderators such as race, gender, and social class? Future work could extend the receptivity–responsiveness framework developed here by describing how it relates to stereotyping, discrimination, and disadvantage based on social identity. For example, researchers could consider receptivity a part and product of an individual’s habitus, or the system of behaviors, attitudes, and dispositions attributable to a person that is typically unconscious, acquired early in life, long-lasting, and culturally derived (see Bourdieu, 1984, 2005; Bourdieu & Passeron, 1990). Habitus connects the concepts of agency and social structure in a framework of culture and power, and helps clarify broader power dynamics based on social class, race, gender, or combinations of these and other attributes of individuals (Hiller & Rooksby, 2005). Thus, individuals who are perceived as embodying the culturally derived habitus valued by teachers will have a greater advantage than others when they are coping with worry about “fitting into” institutions like schools. Future work could more completely explore this framing between receptivity and identity than I can here.
Finally, how can researchers continue to evaluate the success of wise interventions based on participants’ receptivity and responsiveness? The receptivity–responsiveness framework offers many avenues forward for studying this and other wise interventions. In addition to exploring receptivity and responsiveness using GPA or other policy-relevant outcomes (e.g., school suspensions, truancy, or attainment), a crucial area of further study is understanding the mechanisms through which receptivity and responsiveness work to affect such distal outcomes. For example, strategies to help uncover some of those mechanisms include well-crafted measures on students’ motivating factors for increasing achievement in school, detailed questions regarding student thoughts on the extent to which they believe peers’ reassurances of eventually fitting in, and repeated measures of students’ worry about belonging.
The framework may also be applied in other wise interventions. For example, patterns of teacher perceptions of students’ behaviors may be good indicators of a student’s receptivity and responsiveness to utility value interventions. If teachers have rewarded students with high praise in the past, we might expect those students to believe school helps them achieve their goals. Those with high classroom behavior reports in the past that decline over time may thus respond well to utility value interventions because they have ground to regain compared with those who find value in schooling. Disengagers (and potentially Nonengagers, to some degree) may be responsive to utility value interventions in terms of improved GPA, high school completion, or college matriculation, as such students do not enjoy positive teacher ratings but could be open to believing school can instrumentally help them achieve their future goals anyway. 3
The receptivity–responsiveness framework can inform intervention research even more broadly. Future work should consider alternate data collection approaches to capture receptivity and responsiveness in other measures and for other outcomes. This may include better understanding additional nuances to the framework I am unable to describe here—for instance, by uncovering not only who has both receptivity and responsiveness, but who has just receptivity and who has neither. Such continued work on developing the receptivity–responsiveness framework will help situate it as a useful conceptual and policy-relevant tool for understanding psychologically wise interventions, and perhaps educational interventions more broadly.
Supplemental Material
sj-pdf-1-epa-10.3102_01623737231154605 – Supplemental material for Teacher Perceptions of Past Classroom Behaviors Influence Adolescents’ Receptivity and Responsiveness to a Belonging Intervention
Supplemental material, sj-pdf-1-epa-10.3102_01623737231154605 for Teacher Perceptions of Past Classroom Behaviors Influence Adolescents’ Receptivity and Responsiveness to a Belonging Intervention by Jaymes Pyne in Educational Evaluation and Policy Analysis
Footnotes
Acknowledgements
Special thanks to Geoffrey Borman for his comments on an early version of the manuscript and for his support developing this project; to Christopher S. Rozek and Paul Hanselman for their helpful ongoing comments and suggestions about this manuscript; to Eric Grodsky, Linn Posey-Maddox, Jordan Conwell, and Katherine Magnuson for their productive comments and suggestions on early versions of this manuscript; and to Taylor Dyste Pyne for designing the conceptual chart presented in the paper. Finally, I would like to thank the anonymous reviewers and editorial team, and Drs. Allison Atteberry and Joseph Cimpian in particular, for their valuable guidance in strengthening this manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by U.S. Department of Education (Grant Numbers R305A110136, R305C050055) and the Spencer Foundation (Grant Number 201500044). The content is solely the responsibility of the author and does not necessarily represent the official views of the supporting agencies.
Notes
Author
JAYMES PYNE, PhD, is a senior research associate at Stanford University. Through his research, he studies the nature and consequences of engagement, punishment, and inequality in social institutions.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
