Abstract
Recent randomized studies suggest brief social-psychological interventions can help students reappraise common social and academic worries during the difficult transition to middle school and, in turn, improve school performance. We conducted a preregistered student-level randomized controlled trial to assess the replicability of these findings for sixth-grade students transitioning to middle school in three Texas schools (n = 604). Hypothesized main effects for the preregistered confirmatory academic and behavioral outcomes did not replicate. However, exploratory analyses revealed that treatment students with greater numbers of disciplinary referrals during the transition to middle school experienced larger reductions in referrals after intervention than those with fewer baseline referrals. Also, students of color showed greater improvements in their grade point averages after intervention than their white and Asian peers. Non-replicated main effects may be explained by an unusual district context and by evidence suggesting that the intervention mitigated students’ academic worries but did not resolve social worries.
Social belonging is an essential human need (Baumeister & Leary, 1995). When individuals transition from the familiar to the unknown—such as the typical move from elementary to middle school—they are at risk of experiencing belonging uncertainty, a psychological state whereby people perceive the surrounding environment as potentially threatening (Walton & Cohen, 2007). In this psychological state, individuals associate ambiguous or negative cues in their environments (e.g., getting one bad grade, having an argument with a friend) with the idea that they do not belong and often attribute the cause of belonging uncertainty to internal and permanent characteristics instead of external and changeable conditions (Murphy et al., 2007; Sekaquaptewa, 2011; Smith et al., 2013; Thoman et al., 2014; Walton & Cohen, 2007). In a school-based setting, this attribution error can initiate a negative feedback loop, in turn, confirming feelings that students do not belong academically and socially and hindering their academic engagement and effort (Thoman et al., 2013). It also causes anxiety that reroutes mental processing to stress management, ultimately reducing students’ capacity for academic tasks (Schmader & Johns, 2003; Schwarzer, 1986). Finally, feelings of social rejection cause declines in self-regulation (Baumeister et al., 2005), and can place adolescents at greater risk for both internalizing and externalizing problems (Newman et al., 2007).
Attending to the academic and social well-being of students transitioning to middle school is urgent. In this study, we conducted a direct or, as termed by Hudson (2023), an “exact” replication of a brief middle-school belonging intervention in a new district context. Previous research reported by Borman et al. (2019) and Pyne and Borman (2020) has confirmed that this social-psychological intervention, which is designed to mitigate students’ social and academic worries associated with the transition to middle school, can have positive impacts on transitioning students’ school success. Here, we utilize the same intervention materials and procedures as used in the two prior experiments, thereby directly replicating the independent variable, while also collecting the same dependent variables (i.e., students’ grade point averages, counts of D and F grades, school attendance rates, and disciplinary referral and suspension counts). We apply the same methods within a new district context “to establish the efficiency of a specific treatment or intervention” (Stroebe & Strack, 2014). Our exact replication, as Cesario (2014) noted, has further value both in more carefully determining the magnitude of the intervention impacts and providing an indicator of whether the impacts may actually be the product of type one error. Finally, as Crandall and Sherman (2016) suggested, this practice of direct or exact replication can be particularly important when the findings might lead to policy recommendations regarding wider use of a program or practice.
While all school transitions are difficult, the transition to middle school presents an especially potent risk for belonging uncertainty. Just as they make this transition, middle school students are experiencing an early adolescent period development that comes with considerable physiological, psychological, social, and cognitive changes and is characterized by a heightened sensitivity to social comparison and acceptance (National Academies of Sciences, Engineering, and Medicine, 2019). Moreover, their new environment poses unique challenges for students at this dynamic developmental stage. At a time when social acceptance is of great concern, middle school students are entering schools that are generally larger, feel less personal, and interrupt prior peer networks. As they are developing their academic identity, their new environment promotes detrimental social comparisons through, for example, the initiation of letter grades and more separation of students by academic ability (Eccles, Midgley, et al., 1993; Eccles, Wigfield, et al., 1993). Students are also more likely to experience fewer close, positive relationships with their teachers, as discipline and control is prioritized (Okonofua et al., 2016; Wentzel, 1997). Further, African American and Latinx students are at additional risk for suffering social exclusion in school given negative societal stereotypes about their academic performance (Cook et al., 2012; Sherman et al., 2013; Steele & Aronson, 1995).
This “stage-environment mismatch” (Eccles, Midgley, et al., 1993; Eccles, Wigfield, et al., 1993), along with associated belonging uncertainty, contributes to a range of negative outcomes for students transitioning into middle school. Compared to students in K–8 schools, students who transition to middle school demonstrate sizable declines in achievement (Rockoff & Lockwood, 2010; Schwerdt & West, 2013), including one study that reports a 3.5- to 7-month dip in expected learning over the course of a school year (West & Schwerdt, 2012). Other research documents associations between the transition to middle school and declining grade point averages (GPAs), identification with school, and intrinsic motivation, plus increasing mental health problems and increasing disciplinary infractions, with more pronounced declines in achievement for African American and Latinx students (Cook et al., 2012; Sherman et al., 2013; Shim et al., 2008).
School Belonging Interventions
Because students experiencing belonging uncertainty misattribute the cause of their nonbelonging to internal and stable characteristics, causing psychological and academic harm, one way to promote belonging is through interventions that help students to normalize and reappraise their adversity. Indeed, prior research on social-psychological interventions designed to mitigate belonging uncertainty in students transitioning into college supports this approach. For example, Wilson and Linville (1982, 1985) studied an intervention that was designed to help students reattribute academic difficulties as external and changeable, by teaching them that early academic challenges are normal and transitory. The study demonstrated increased retention and academic achievement among those students who were initially struggling with poor academic performance. Walton and Cohen (2007, 2011) studied an intervention that normalized difficulties concerning making friends and fitting in at the beginning of college with a prompt expressing that all students worry at first about belonging in college but that their sense of belonging increases over time. The researchers found that African American students, whose group may experience particular worries of marginalization, earned higher grades during the next 3 years, halving the racial achievement gap and reported better health outcomes 3 years later.
These approaches are related to other recent social-psychological strategies to reframe or reappraise anxiety as a positive form of arousal that may have functional benefits that help students improve their academic performance (Jamieson et al., 2010, 2016, 2022). In this case, though, the middle school belonging mindset intervention used in the current study is based on normalizing adversity. The intervention focuses on both academic and social sources of adversity, making it a combination of strategies applied in the previous work by Walton and Cohen (2007, 2011) and Wilson and Linville (1982, 1985). The key messages of the middle-school belonging intervention are that worries about belonging are normal, that they are short-lived, and that support is available (Borman et al., 2019). It is based on the understanding that internalizing such a message, through reading and writing about social and academic adversity, students are more capable of reappraising belonging uncertainty as less threatening (Walton & Brady, 2017), thus creating conditions in which they can more easily succeed academically (Thoman et al., 2013; Walton & Cohen, 2007, 2011).
More specifically, normalizing these challenges, while noting that teacher and peer support is available, is theorized to help students interpret negative or ambiguous cues as external and changeable. As shown in Figure 1, these reappraisals of adversity can promote belonging and stronger identification with school (Dweck, 2006), while reducing evaluative anxieties, which frees up mental capacity for academic tasks (Jacobs & Gross, 2014). Further, students’ perceived support from teachers and peers may promote help-seeking behavior (Newman & Schwager, 1993; Ryan et al., 1998) and social and psychological well-being (Eliot et al., 2010). In addition, greater perceived fit at school supports academic engagement, attendance, and other positive behavioral change by reducing internalizing and externalizing problems associated with belonging uncertainty and alienation (Newman et al., 2007; Okonofua et al., 2016). Over time, the intervention initiates a positive recursive cycle whereby shifts in student beliefs produce short-term wins, which then reinforce those positive beliefs, and so on, ultimately producing stronger academic achievement (Beilock et al., 2017; Cohen et al., 2009; Okonofua et al., 2016).
Middle-School Belonging Intervention Theory of Change.
Previous Research on the Middle School Belonging Intervention
Borman et al. (2019) first studied the middle-school belonging intervention for transitioning sixth graders (n = 1,304) across all 11 middle schools in the Madison Metropolitan School District. Using a double-blind experimental approach, the researchers found that, relative to control students, the intervention improved students’ GPAs (d = 0.09), reduced failing grades (d = 0.11), reduced sixth-grade disciplinary incidents by 34%, and increased attendance by 12% (Borman et al., 2019). Further, the intervention improved students’ school trust, social belonging, and identification with school, while lowering evaluation anxiety (Borman et al., 2019). A non-experimental mediation analysis suggests that these attitudinal and behavioral changes explained improvements in students’ academic performance, as measured by increased GPAs and a reduced number of failing grades. In contrast to prior intervention work focused on the transition to college by Walton and Cohen (2007, 2011), which facilitated the transition for African American students, and Wilson and Linville (1982, 1985), which helped those who struggled with low achievement during the transition, the positive middle-school belonging intervention effects were statistically indistinguishable regardless of racial/ethnic group or baseline academic performance. This finding suggests that the middle school transition may be difficult for everyone and that the intervention can help all students who receive it.
Pyne and Borman (2020) conducted a direct replication of this Madison study, with a preregistered project that aimed to confirm the prior study’s main effects on GPA, at scale, in all seven middle schools across the Paradise Valley (Arizona) Unified School District. This study confirmed the achievement findings of the Madison study, both in terms of increasing GPA (d = 0.06) and reducing the number of failing grades (d = −0.06). Manipulation checks also confirmed the theoretical mechanism of the intervention, as treatment students reported less worry about academic and social belonging in the future compared to their control-group peers. As in the Madison study, Pyne and Borman (2020) found that the intervention effects were not moderated by students’ race/ethnicity.
These results demonstrate consistent intervention impacts across contexts that are quite different. For example, the Paradise Valley study took place in a district in which the transition to middle school occurs between sixth and seventh grade, while the transition in Madison is between fifth and sixth grade. This difference may have accounted for the slightly weaker effects found in Paradise Valley, given that students who are a year older should be somewhat more physically, psychologically, and socially prepared for the structure and expectations of middle school (Schwerdt & West, 2013; West & Schwerdt, 2012). In addition, the two districts differ demographically with Madison representing a more racially and socioeconomically diverse student population, compared to Paradise Valley’s majority white, relatively affluent composition. These consistent academic impacts across these two diverse school districts at two distinct transition points—Grades 6 and 7—offer important generalizable evidence of the effectiveness of this middle school belonging intervention.
The Significance of Replication in Education Research
In the context of a so-called “replication crisis” in both the psychology and education research fields (Makel & Plucker, 2014, 2015; Makel et al., 2012, 2016; Plucker & Makel, 2021; Pridemore et al., 2018), this direct replication across two different sites is meaningful and promising. Further, the Pyne and Borman (2020) study was implemented as a preregistered direct replication at scale. Such replications are extremely rare in education research, with just 0.13% of published studies being replications, of which 28% are direct replications (Makel & Plucker, 2014, 2015; Makel et al., 2012, 2016; Plucker & Makel, 2021; Pridemore et al., 2018).
Even while asserting the importance of the replicated results for the middle school belonging intervention, prior scholarship suggests that one replication is rarely enough to confirm an intervention’s efficacy (Hedges & Schauer, 2019). Further replication of this intervention would help to increase confidence in the prior results, particularly if implemented in a new context. Indeed, replication helps to validate evidence and control for sampling errors, establishes fact over novelty, and ultimately identifies repeatable results that can justify the investment of scarce public resources (Plucker & Makel, 2021; Tyson, 2014). Further, while many education studies are conducted at a relatively small scale, replicating with larger samples helps guard against false positives and inflated effect sizes, which are more likely with small, nonreplicated studies (Camerer et al., 2018; Slavin & Smith, 2009).
The Current Study
Given the significance of replication, the promise of the prior two studies, along with the need to address students’ needs during the transition to middle school, the purpose of the current preregistered study (see https://osf.io/45nek/) is to gather evidence that would allow us to generalize the results of the prior two studies beyond those specific contexts. The current replication took place in a Texas school district, Killeen (TX) Independent School District (KISD), that differs in a number of ways from the prior Madison and Paradise Valley sites. Madison is a school district that is racially (44% white, 19% Latinx, 18% African American, and 10% Asian or other group) and socioeconomically diverse (48% in the free or reduced-price lunch program). The analytic sample for the Paradise Valley study includes a majority white (60%) and relatively affluent (37% receive free or reduced-price lunch) group of students. In contrast, the sample of students from Killeen is diverse with respect to race/ethnicity, with nearly three quarters of students identifying as non-white and nearly three quarters qualifying for free or reduced-price meals.
Killeen is somewhat unusual, in that about one in four students is from a military family. Although children from military families have different childhood experiences compared to their civilian peers, such as frequent relocations, indirect exposure to and awareness of conflict, and extended separation from parents or siblings due to deployment, prior research has generally concluded that military children report similar mental health outcomes relative to nonmilitary connected children (Williamson et al., 2018). Indeed, Easterbrooks et al. (2013) argued that the unique experiences of children from military families may help them build assets, such as a capacity to socialize and move fluidly into new social situations, an intergenerational commitment to character and service and a devotion to something beyond themselves, knowledge of different cultures and traditions, and, inspired by their parents’ commitments, a strong sense of purpose that motivates them to take on leadership and social action roles. Such assets and sources of resilience may help military students navigate the middle-school transition and may further encourage a positive transition for their civilian peers. As military children move and change schools three times more often than nonmilitary children—experiencing six to nine school changes from the start of kindergarten to high school graduation (Astor, 2011; Berg, 2008; Kitmitto et al., 2011; Sherman & Glenn, 2011)—they are certainly familiar with the process of transitioning to a new school.
Our preregistered analyses assessed whether student assignment to the social belonging intervention has an effect on the confirmatory outcomes: GPA, D and F grade counts, attendance, behavioral referrals, and suspensions. We also conducted preregistered exploratory analyses of potential interaction effects of treatment by students’ potentially-stereotype-threatened status. Finally, we estimated non-registered exploratory analyses of potential interaction effects of treatment by students’ baseline performance on each outcome. These latter two exploratory questions are motivated by the stronger impacts for Black students found by Walton and Cohen (2007) and for initially poorly performing students, as suggested by Wilson and Linville (1982, 1985), for college-level samples. Specifically, our analyses addressed the following research questions:
Does assignment to the school belonging intervention impact sixth-grade students’ GPAs, D and F grades, and behavioral outcomes (i.e., attendance, suspensions, and disciplinary referrals)? (Preregistered confirmatory);
Does student race/ethnicity moderate impacts on GPAs, D and F grades, and behavioral outcomes (i.e., attendance, suspensions, and disciplinary referrals)? (Preregistered exploratory).
Does student baseline performance for each outcome moderate impacts on GPAs, D and F grades, and behavioral outcomes (i.e., attendance, suspensions, and disciplinary referrals)? (Non-registered exploratory).
Does assignment to treatment impact students’ reported future academic and social worries? (Manipulation check).
Method
Study Design and Intervention Implementation
We implemented a multisite randomized controlled trial (RCT) to examine the impact of a school belonging intervention on academic and behavioral outcomes among sixth-grade students. The study was conducted during the 2018–19 academic year in three middle schools within KISD. A total of 808 sixth-grade students were randomly assigned to either the treatment or control conditions within each school. Due to mobility out of the school district, a total of 604 of the 808 students completed the school year in KISD, had complete baseline and outcome data, and were retained for analysis.
The intervention involved two 15-minute in-class exercises addressing belonging uncertainty in beginning middle school students. The exercises prompted students to reflect on circumstances where they might feel uncertain about belonging. The prompts included quotes and stories from previous middle school students who initially faced challenges but eventually overcame them. The treatment materials convey two core messages: (1) it is common that middle school students experience initial struggles socially and academically and (2) with time and support from peers and teachers, students come to realize that they belong. Students were asked to reflect on this information, consider ways they may navigate their difficulties, and envision how school will ultimately get better. They were then prompted to provide written responses addressing their concerns about test-taking and fitting in with peers to internalize the intervention message. In the control condition, students also completed two writing exercises of equal length, but the prompts focused on neutral middle school experiences unrelated to belonging uncertainty, including dealing with a noisy lunchroom and learning about politics.
The writing exercises were designed to be administered as routine assignments within the sixth-grade English language arts classes. Teachers provided students with copies of the prompts and instructions. Students were instructed to complete the exercises, return them, and continue with their regular classwork. The packets given to treatment and control students had identical cover sheets, and both teachers and students were unaware of the students’ treatment status. The first exercise, which focused on students’ worries about academic performance, was administered early in the school year (September) to address belonging uncertainty before potential negative cues (e.g., report cards) could reinforce a sense of not belonging. The process was then repeated approximately 6 weeks later (November), coinciding with a period of potential stress for students, such as before an important district-wide assessment or midterm exam. This second exercise dealt with students’ worries about social belonging in school.
To ensure intervention fidelity, we developed training protocols based on the prior work of Borman et al. (Borman et al., 2019; Pyne & Borman, 2020). At the start of the 2018–19 school year, the research team provided all teachers with a 30-minute webinar session and supporting written materials and guides. These resources familiarized teachers with the intervention script, addressed common student questions, and provided instructions for completing the in-class exercise fidelity report. Aside from some students missing the assigned exercises due to absence, no significant problems were noted by teachers on the fidelity reports.
Sample
We engaged in significant outreach and recruitment efforts to identify a district partner that was demographically and geographically different from those that have participated in prior work on school belonging in middle school. We found strong support for the study from KISD leadership, who in turn, identified three of its 13 middle schools that were willing to take part in the research and support implementation of the intervention. The three participating middle schools are public schools with grade spans of sixth through eighth grade, enrollments of approximately 750 to 850 students, and student mobility rates of greater than 25%. The district did not indicate that any active measures were being implemented to help students manage middle-school transitions, nor was there discussion of the prevalence of military families. Although we actively sought out a district context that was geographically and demographically distinctive from those identified for prior studies, the sample of three KISD schools is best defined as a convenience sample.
The study included a baseline sample of 808 sixth-grade students from three middle schools in KISD, Texas. We randomized only those students who were on the original rosters at the beginning of the school year prior to distribution of the first student writing exercise. For the confirmatory analyses, we identified a complete-case analytical sample composed of 604 of the baseline sample of 808 students with complete data on all baseline student covariates and the five administrative outcomes (i.e., GPA, the number of D and F grades, suspensions, behavioral referrals, and attendance rate). The district required no consent and there were no opt-outs. The only attrition occurred through missing data. Missing data occurred due to students leaving the district and having incomplete baseline and/or outcome data. In this case of relatively low data attrition occurring only through missing administrative information, we opted to use the most conventional What Works Clearinghouse ([WWC], 2022) strategy recommended for handling missing data in randomized trials: listwise deletion. This was also our preregistered method for dealing with missing data. Among the 604 students from the analytic sample, there were 294 treatment students and 310 control group students. The sample consisted of diverse students, with 72% of students identifying as members of racial or ethnic minority groups that are potentially susceptible to negative stereotypes in school settings, 72% experiencing economic disadvantage, 11% being English learners, 17% eligible for special education services, and 24% from military families. All students in the KISD transition from elementary to middle school in sixth grade.
We consider our final analysis sample of 604 students nested in three schools as an intention-to-treat (ITT) sample aligned with Lachin’s (2000) conceptualization, which emphasizes that all randomized individuals should be included in the analysis, regardless of their adherence to the treatment or protocol. As such, and aligned with the recommendations of Alshurafa et al. (2012), we confirm that students were analyzed in the groups to which they were randomized with no post-randomization exclusions. Lachin (2000, p. 183) further suggests that “a true intent-to-treat analysis requires the inclusion of all patients randomized to the extent possible.” This suggestion to include all sample members “to the extent possible” acknowledges that there may be situations where including all individuals in the analysis is not feasible, such as when data for a person is completely missing.
As suggested by Alshurafa et al. (2012), we provide a second and independent statement concerning how we have handled missing data from our student sample. Specifically, we have not excluded students due to non-compliance with the treatment/control regime and we have included all students randomized “to the extent possible.” However, we do acknowledge that we applied a complete case analysis, which omitted students with incomplete administrative data, and this may reduce power and increase the Type I error rate slightly relative to other methods applying imputation of missing information.
Student-Level Covariates
Demographic Data
From the school district, we obtained information regarding students’ race/ethnicity (i.e., American Indian, Asian, Black, Latinx, Native Hawaiian, and white) and sex (male or female). Based on the race/ethnicity data, we also developed a dichotomous variable to explore students’ potentially stereotyped status within school as a moderator of treatment effects (Potentially stereotyped: Black, Latinx, Native Hawaiian, and American Indian. Not potentially stereotyped: white and Asian). The literature suggests that schools are “chronically evaluative” and that stereotype threats are routinely activated in the “real world” of schools and classrooms (Cook et al., 2012. Therefore, threatened status need not be artificially primed, as in some laboratory-based studies. It is effectively induced as students navigate their typical educational programs, but we do refer to students’ status as “potentially threatened” and “potentially non-threatened” because we rely on school conditions rather than active priming to be the source of threat.
In addition, the district provided student-level data for free or reduced-price lunch eligibility status (as a proxy for poverty), English language learner status, special education participation status, and the family’s military status. The student-level covariates, including race/ethnicity, sex, free or reduced-price lunch eligibility status, English language learner status, and special education participation status, were specified in our preregistered plan. We preregistered the study prior to recruitment of KISD and the family military status covariate was not considered in our preregistered plan, but sensitivity analyses suggested that exclusion of this covariate did not alter our estimated treatment effects and substantive conclusions. All of these dichotomous covariates were dummy coded, with a value of 1 indicating the presence of the attribute and a value of 0 indicating the absence of the attribute.
Outcome Measures
Grade Point Average (GPA)
We calculated a student’s GPA using administrative data provided by the school district. The letter grades were converted into a numerical score (i.e., A = 4, B = 3, C = 2, D = 1, F = 0), and the average score was computed. The analysis focused on core academic courses only, including math, science, English language arts, and history/social studies. The GPA values have a minimum of 0.00 for a student who failed every class and a maximum of 4.00 for a straight-A student. Because the intervention exercises occurred in the first quarter of the academic year, we created the GPA outcome measure using the post-treatment GPA by averaging students’ GPAs from the final three quarters of the school year. The first term GPA served as the baseline measure.
Number of Ds and Fs
The number of Ds and Fs for each student represents the sum of the number of such grades received in core academic courses. Similar to the GPA measure, we created outcome measures for the number of Ds and Fs using post-treatment failing grades for Terms 2–4 by summing the number of Ds and Fs for the final three quarters of the school year, excluding Term 1. The baseline outcome was measured using students’ GPAs from the first quarter.
Student Suspensions
Suspensions are incidents of exclusionary discipline when students are temporarily removed from their classrooms. In our study, we adopted the definition of suspension provided by KISD, which includes in-school and out-of-school suspensions, placement in a disciplinary alternative education program (DAEP), and partial day placement in an intensive care unit (ICU). We measured student suspension as the total number of suspensions accumulated after the date of the second intervention administration during the 2018–19 school year. The baseline outcome was measured as the number of suspensions that occurred before the date of the second exercise administration.
Student Behavioral Referrals
Behavioral referrals encompass a range of disciplinary actions that involve reporting or documenting when a student’s behavior violates school rules or policies. All suspensions are preceded by a behavioral referral, but some referrals in KISD indicate less severe actions, including detention, probation, hearings (campus or district level), bus suspension, and others. We measured student behavioral referrals as the total number of disciplinary referrals after the date of the second intervention administration during the 2018–19 school year. The baseline behavioral referrals were indexed as the count of referrals that occurred prior to the date of the second intervention.
Student Attendance Rate
Attendance rate was measured as the total number of days absent for each student, recorded by the district, divided by the number of days enrolled. The attendance rate on or after the second administration date served as the post-intervention outcome, while the attendance rate before the second exercise administration was used as the baseline measure.
Manipulation Check Data
Manipulation checks were conducted to determine whether the intervention produced the hypothesized student responses. Manipulation checks are a common and long-standing procedure in psychological lab and field experiments that help researchers evaluate whether the treatment group participants understood the intervention’s intended message after experiencing the experimental manipulation (Festinger, 1953; Hauser et al., 2018). After participants received the manipulation—in our case, the school belonging intervention or the neutral control materials—we verified that the experimental manipulation produced the assumed changes. Our manipulations should cause treatment group students to report stronger expectations than control group students that academic and social worries will be resolved with time. Once we establish that treatment group students reappraised the common academic and social worries targeted by the intervention materials, we can be more confident that this theorized mechanism was enacted by the treatment and in proceeding to examine the academic and behavioral effects of the intervention.
We used five-point rating scales (from “Not at all” to “A Lot”) placed at the end of both exercises to determine whether the intervention appears to change students’ worries about either taking tests (Exercise 1) or fitting in socially at school (Exercise 2). Exercise 1 items address worries about tests in school (How much do you think sixth graders last year worried about taking important tests in middle school?; How much do you think those same students worry now about taking important tests as seventh graders?). Exercise 2 manipulation check measures address worries about fitting in at school (How much do you think sixth graders last year worried about whether they “fit in” or “belonged” at your school?; How much do you think those same students worry now as seventh graders about whether they “fit in” or “belong” at your school?). We administer these manipulation checks to both treatment and control students, thus allowing a test of the experimental impact on this proximal outcome.
Students’ Written Responses to the Exercises
We also investigated whether the content of students’ written responses to the exercises referenced four theorized psychological mechanisms that may have helped them reevaluate adversity associated with the middle school transition. After students read the series of vignettes that described past students’ initial struggles and ultimate successes with the middle school transition, they were asked to provide one or two short answers that were generally one sentence in length. For Exercise 1, which dealt with academic worries, the questions asked students to provide one or two reasons “why a sixth grader like you might worry less about taking tests after a little bit of time,” and one or two reasons “why a sixth grader like you might do well on tests even if you worry about taking tests.” In response to Exercise 2, which focused on social worries, students were asked for one or two reasons “why sixth graders like you might feel more sure that they “fit in” or “belong” at (school name) after a little bit of time.” Although control students did not read vignettes that explicitly addressed academic and social worries, all students were asked this same series of questions. We hypothesized that treatment students would provide responses that were more likely to reflect four themes: (1) worries of transitioning students are common and shared; (2) these worries will be positively resolved with time; (3) adversities may be overcome through positive experiences with teachers, administrators, or school in general; and (4) adversities may be overcome through positive experiences with their peers.
We coded the responses of both treatment and control students to the two writing exercises. Although control students completed exercises that were not designed to elicit reappraisal of adversity and acknowledgement of support from school staff and peers, it is possible for them to engage in such reappraisals without specific prompting. The coding rubric identified four types of student responses: (1) acknowledgement that struggles are common and experienced by nearly everyone (“shared experience”); (2) statements that communicate students’ understanding that these struggles improve with some time (“improve with time”); (3) acknowledgement that struggles may be overcome through positive experiences shared with peers (“social belonging”); and (4) responses communicating that positive experiences with staff at school can attenuate adversity (“group belonging”).
For example, when a student wrote “it might get easier over time,” regarding getting used to taking difficult tests, the student response was coded as “improve with time.” When a student stated that they may feel more confident and have a sense of fitting in or belonging at their school after a short period of time “because they have tons of friends,” the student response was coded as “social belonging.” Each student-provided short answer was coded for all four potential mechanisms: 1 if the short answer referenced the mechanism and 0 if it did not. It is further possible for a short answer to include references to multiple mechanisms. For instance, when a student wrote “everyone worries about tests, but teachers are there to help,” this response was coded as both “shared experience” and “group belonging.”
All student responses were independently coded by two trained coders who were unaware of the students’ treatment conditions. The coders began by coding the same random sample of approximately 10% of the student short answers, with half being treatment and half being control student responses. All coding discrepancies were resolved through discussions between the two coders. After this initial step, another random sample of approximately 10% of the short answers was double-coded in order to assess interrater reliability. Across the four codes, Cohen’s κ for writing Exercise 1 ranged from 0.87 to 0.98, and for writing Exercise 2, it ranged from 0.94 to 0.98. These values indicate nearly perfect interrater consistency (Altman, 1999; Landis & Koch, 1977).
Statistical Analysis
The primary preregistered ITT research questions focused on the impact of the social belonging intervention on academic (GPAs and D and F grades) and behavioral (suspension, disciplinary referrals, and attendance) outcomes for sixth-grade students. Our goal is to estimate the super-population student-level average treatment effects, as defined by Miratrix et al. (2021). To accomplish this, we applied a linear regression estimation strategy with school fixed effects and cluster robust standard errors (CRSEs), notation
Yet, as Imbens and Kolesar (2012) pointed out, there is a large body of literature suggesting that conventional CRSEs perform poorly if applied to a small sample of clusters like ours. As a solution, we applied CR2 CRSEs with Bell and McCaffrey (2002) degrees of freedom adjustments, which are particularly useful when dealing with data from a small number of clusters. As noted by Miratrix et al. (2021), CRSEs assume entire clusters (schools) are randomly selected from a hypothetical, infinitely large population (the “super population”) within which each observation is independent and identically distributed. Miratrix et al. (2021, p. 286) suggested that CRSEs effectively “treat the sites as integral clusters and give super-population estimates of uncertainty” while “potential correlations among individuals within sites, as well as heteroskedasticity, are accounted for by aggregating the site-specific patterns of residuals within site.”
Our ordinary least squares regression model investigated whether assignment to the social belonging intervention has an effect on the confirmatory outcomes, as follows:
This model allows us to assess the impact of treatment assignment, represented by
Furthermore, we examined whether the impact of the school belonging intervention varied based on students’ background characteristics. Given that prior research has shown that the middle school transition can be particularly difficult for potentially negatively stereotyped racial/ethnic minority students (Cook et al., 2012; Sherman et al., 2013; Shim et al., 2008) and that prior college-level impacts have been restricted to students of color (Walton & Cohen, 2007) and students who are experiencing initial struggles academically (Wilson & Linville, 1982, 1985), we also conducted a preregistered exploratory analysis, which interacted treatment by students’ potentially stereotyped status (1 = Black, Hispanic, Native Hawaiian, and American Indian, 0 = White and Asian) along with a non-preregistered exploratory analysis interacting treatment by students’ baseline status on each outcome. The former interaction effect assessed whether the intervention effect on student outcomes differed between potentially threatened and potentially nonthreatened students. The latter exploratory interaction effect investigated whether the treatment effects depended on students’ initial performance on each premeasure. Similar to the work by Wilson and Linville (1982, 1985), the hypotheses tested by these latter analyses were that those students struggling the most academically and behaviorally may have realized greater intervention effects than those experiencing fewer struggles.
The preregistered treatment-by-potentially-threatened interaction effects tell us whether potentially threatened and non-threatened students respond differently to treatment. To determine whether the intervention had an impact on potentially threatened students, we estimate marginal effects of treatment. We transform the parameter estimates from the impact model to estimate the average effect of the belonging intervention for the students who are potentially subject to stereotype threat and the students who are not subject to stereotype threat, at the average values of each racial/ethnic group’s covariates. We estimate these margins at the group means because the preexisting differences on some of the covariates are large. Using the Stata margins dydx command, we then calculated intervention effect sizes using the overall unadjusted standard deviations of the outcomes.
We also evaluated treatment-control differences for two additional outcomes: the manipulation check and the coded student responses to the exercise writing prompts. We used an ordinary least squares regression analysis of the same specification as applied to the academic and behavioral outcomes with the same vector of preintervention covariates including baseline term 1 GPA. These analyses evaluated both students’ reports of current academic and social worries and their anticipated future academic and social worries. We hypothesized that there would be no statistically significant treatment-control differences for the current academic and social worries. For reports of future worries, we hypothesized that treatment students reappraised the common academic and social worries targeted by the intervention materials and anticipated statistically significantly lower levels of future anxiety relative to control students.
For the analyses of students’ coded responses to the exercises, we conducted a total of eight chi-square tests of independence, with one analysis for each of the four coded responses to the two exercises: shared experience (1, 0); improve with time (1, 0); social belonging (1, 0); and group belonging (1, 0). These analyses tested whether the observed frequencies for each of the four dichotomous codes are statistically significantly different from the frequencies expected if treatment assignment is unrelated to students’ coded responses. These simple descriptive comparisons did not account for student covariates or school fixed effects and did not apply CR2 standard errors.
Results
Attrition Analysis
To assess potential bias resulting from sample attrition, we followed the standards outlined in the What Works Clearinghouse (WWC) handbook, Version 5.0 (WWC, 2022). Sample attrition occurs when the final analytical sample differs from the randomly assigned sample due to missing data. In this study, students with missing covariates or outcomes were excluded from the analytic sample. The WWC standards determine whether the combination of overall attrition (i.e., the rate of attrition for the entire sample) and differential attrition (i.e., the difference in the rates of attrition for the treatment and control groups) is categorized as high or low and whether the expected bias due to attrition rate is tolerable or unacceptable based on optimistic or cautious assumptions. 1
With an overall attrition rate of 25.2% and a differential attrition rate of 4.0 percentage points, the sample falls into the WWC’s low attrition category under both optimistic and cautious assumptions. Further, the difference between the attrition rates of treatment (27.2%) and control (23.3%) students is not statistically significant,
Baseline Equivalence
For the final complete-case sample of 604 students included in our primary confirmatory and exploratory analyses, we examined whether the treatment and control groups had equivalent baseline characteristics. We assessed baseline equivalence on all covariates, including students’ baseline outcomes. Table 1 shows the mean differences between the treatment and control groups and the standardized mean differences for each baseline covariate (Hedges’ g for continuous variables and Cox index for binary variables) for the analytic sample. As shown in Table 1, the results suggest that the differences between the treatment and control groups are below 0.25 standardized units for all variables, except for Asian (SMD = 0.52), Native Hawaiian (SMD = 0.55), and special education status (SMD = −0.32). These chance differences, especially for Asian American and Native Hawaiian students, are most likely explained by relatively small sample sizes, as each of these racial/ethnic groups composed approximately 1% of the overall sample.
Student Sample Characteristics and Baseline Equivalence for Academic and Behavioral Outcome Sample (N = 604)
Note. SD = standard deviation.
Although the results at the bottom of Table 1 show that the preintervention differences between the treatment and control groups for the outcomes, attendance rate, total referrals, total suspensions, and core course GPA, were well below the What Works Clearinghouse (2022) baseline equivalence standard of less than 0.25 standard deviations, it is regrettable that the baseline differences for referral and suspension counts were 0.10 and 0.12 standard deviations, respectively. 2 Our impact models do statistically control for these baseline measures, as suggested by the What Works Clearinghouse (2022) guidance. Although the guidance does not explicitly suggest that chance baseline demographic differences, such as those found for Asian, Native Hawaiian, and special education students, should be accounted for by covariate adjustment, we do include these and all other background characteristics as covariates as well.
Manipulation Check Outcomes
Our first analysis, shown in Table 2, assesses whether the manipulations in the experiment (i.e., reducing students’ worries about test taking and fitting in socially) performed as expected. To perform this analysis, we analyze the student responses to the manipulation checks included at the end of each exercise. Exercise 1 items address worries about tests in school (How much do you think sixth graders last year worried about taking important tests in middle school?; How much do you think those same students worry now about taking important tests as seventh graders?). The results shown for “Worry Before” for Exercise 1 in Table 2 indicate no statistically significant difference between students in the intervention and control groups in reported levels of anxiety that sixth graders, like them, worried about taking important tests at the beginning of sixth grade. When asked about future worries (see results for “Worry Now”) about important tests, students in the intervention group reported less anxiety compared to those in the control group (β = −0.41, p < .001). In response to the Exercise 2 manipulation checks (How much do you think sixth graders last year worried about whether they “fit in” or “belonged” at your school?; How much do you think those same students worry now as seventh graders about whether they “fit in” or “belong” at your school?), the results in Table 2 show that students in the intervention group expressed statistically significantly greater levels of current social worries compared to those in the control group (β = 0.08, p = .037). Unexpectedly, there was not a statistically significant difference between treatment and control groups in terms of future social worries. Overall, our results indicate the intervention produced the intended proximal social-psychological effect concerning reducing worries about academic outcomes, but contrary to our expectations, the intervention seem to increase treatment students’ current social worries while having no effect on future worries relative to their control-group peers. 3
Intervention Effects on Exercises 1 and 2 Manipulation Checks (N = 444)
Note. CR2 cluster robust standard errors are noted in parentheses; Dependent variables are z-scored manipulation check questions asked at the end of both Exercises 1 and 2.
p < 0.001, ** p < 0.01, * p < 0.05.
Confirmatory Preregistered Impact Results
The confirmatory preregistered analyses examined the main effects of the school belonging intervention on students’ academic and behavioral outcomes. The results of these analyses are presented in Table 3. The main outcomes of interest are the treatment impact estimates for each of the Terms 2 through 4 outcomes, including GPA, number of D and F grades, number of suspensions, number of behavioral referrals, and attendance rate. As the treatment coefficients for each outcome suggest, we found no statistically significant main effects of treatment on any academic or behavioral outcomes. Interestingly, in addition to the Term 1 premeasures for each of the Terms 2 through 4 outcomes, there were several student characteristics that were statistically significant predictors of the outcomes. Those students who identified as female tended to have higher GPAs and receive fewer D and F grades. Free or reduced price lunch students received lower GPAs, greater numbers of D and F grades, and greater suspension and behavioral referral counts. Black students tended to have lower GPAs, Latinx students had lower suspension counts, American Indian students had higher attendance rates, and Hawaiian students had higher GPAs than students of other racial/ethnic categories. Students from military families had fewer suspensions and behavioral referrals than those from non-military families. Finally, there were differences on all five outcomes among the three participating schools.
Intention-to-Treat Impacts on Terms 2, 3, and 4 Grades; Number of Ds and Fs; Suspensions; Behavioral Referrals; and Attendance Rate
Note. CR2 cluster robust standard errors are noted in parentheses.
p < 0.001, **p < 0.01, *p < 0.05.
Moderating Effects of Students’ Race/Ethnicity
A second set of preregistered exploratory research analyses investigates whether the impact of the school belonging intervention on student academic and behavioral outcomes varied based on students’ racial/ethnic status. Specifically, we explored whether those racial/ethnic minority students who tend to be threatened by negative stereotypes within the academic domain may have, in particular, benefited from the intervention. We defined those students who identified as Black, Latinx, Native Hawaiian, and American Indian as potentially threatened students, while white, Asian, and multiracial students were classified as potentially nonthreatened. Table 4 indicates no statistically significant interaction effects between the treatment and students’ potentially threatened status on the behavioral outcomes. However, treatment effects on post-intervention GPAs revealed statistically significant differences based on students’ potentially threatened status (β = 0.07, p < .014).
Interactions Between Potentially Threatened Status and Treatment on Terms 2, 3, and 4 Grades; Number of Ds and Fs; Suspensions; Behavioral Referrals; and Attendance Rate
Note. CR2 cluster robust standard errors are noted in parentheses.
p < 0.001, **p < 0.01, *p < 0.05.
This interaction effect tells us that potentially threatened treatment students’ GPAs showed more positive improvements relative to their non-potentially threatened counterparts. To determine whether the intervention had a statistically significant impact on potentially threatened students, we estimate marginal effects of treatment. We transform the parameter estimates from the impact model to estimate the average effect of the belonging intervention for the students who are potentially subject to stereotype threat and the students who are not subject to stereotype threat, at the average values of each racial/ethnic group’s covariates. For non-threatened students, the marginal mean difference between treatment and control students, expressed an effect size divided by the overall standard deviation, is d = −0.020 (SE = 0.070, p = 0.781). For potentially threatened students, the marginal mean difference between treatment and control is d = 0.046 (SE = 0.045, p = 0.300). Thus, although the interaction effect is statistically significant, neither of the marginal effect estimates revealed a statistically significant treatment-control difference.
Moderating Effects of Baseline Outcomes
A set of non-preregistered exploratory analyses assessed whether students’ baseline outcomes moderated the intervention effect. In other words, did students with worse first-term grades, relatively higher numbers of suspensions or behavioral referrals, or poor attendance respond differently after treatment than those with more favorable term 1 preintervention outcomes? The estimated treatment-by-baseline-outcome interaction effects, shown at the bottom of Table 5, indicate a significant interaction effect for behavioral referrals (β = −0.81, p = 0.020), but no significant interaction effects were found for GPA, the number of Ds and Fs, suspensions, and attendance rate.
Interactions Between Baseline Outcomes and Treatment Group on Terms 2, 3, and 4 Grades; Number of Ds and Fs; Suspension; Behavioral Referral; and Attendance Rate
Note. CR2 cluster robust standard errors are noted in parentheses.
p < 0.001, **p < 0.01, *p < 0.05.
The statistically significant negative interaction effect between baseline behavioral referrals and treatment status on post-intervention behavioral referrals reveals that treatment students with a higher number of behavioral referrals during term 1 (baseline) tended to experience fewer referrals over the remainder of the school year relative to their control-group peers. Figure 2 provides a direct comparison of the unadjusted means of the treatment and control groups based on the number of baseline behavioral referrals students received. The figure shows that those treatment students with zero to two term 1 behavioral referrals did not have fewer postintervention behavioral referrals than their control group peers. However, for those students who had three or more term 1 behavioral referrals, treatment students experienced fewer postintervention terms 2–4 behavioral referrals compared to control group students. Figure 2 suggests that the intervention was particularly effective in reducing behavioral referrals for students who were struggling with a higher number of disciplinary incidents early in the academic year prior to completing the two school belonging interventions.
Unadjusted Post-Intervention Behavioral Referrals by Treatment Status and Term 1 Behavioral Referrals.
Although the interaction and figure suggest that treatment students experiencing more baseline disciplinary incidents may have realized impacts, to determine whether the intervention had a statistically significant impact for those with referrals at the beginning of the year we estimated effects of treatment for this subgroup. Specifically, we created a subgroup of treatment and control students who had experienced at least one referral during the baseline period and estimated a main effect of treatment by applying the same model for main effects as previously described. For those with one or more pre-intervention disciplinary incidents, the difference between treatment and control students divided by the pooled standard deviation resulted in a large but non-statistically significant effect size of d = −0.953 (SE = 1.523, p = 0.644). This is a substantial reduction in post-intervention disciplinary referrals, but it is likely that low statistical power to detect the effect for this subgroup analysis contributed to the non-statistically-significant treatment effect.
Analyses of Students’ Writing Exercise Responses
We also investigated whether students’ written responses reflected internalization of the intervention’s theorized psychological mechanisms and potential reevaluation of adversity. Coding the responses of both treatment and control students to the two writing exercises, we hypothesized that treatment students would provide responses that were more likely to reflect four themes: (1) worries of transitioning students are common and shared (shared experience); (2) these worries will be positively resolved with time (improve with time); (3) adversities may be overcome through positive experiences with teachers, administrators, or school in general (group belonging); and (4) adversities may be overcome through positive experiences with their peers (social belonging). Although the control exercises were not designed to elicit these reappraisals of adversity, it is possible for students to provide such reappraisals without specific prompting. Our hypothesis was that if treatment students were more likely than control students to write about reappraising adversity, they would be more likely to internalize the intervention’s intended message and thus benefit more from it academically and behaviorally.
We cross-tabulated outcomes for each of the four dichotomous codes by the students’ assigned treatment condition. With these two dichotomous variables—the student’s coded outcome and treatment status—we developed a two-by-two contingency table for each of the four outcomes showing the number of observations in each of the four possible cells. Table 6 presents the counts and percentages of respondents categorized into each code for the writing exercises, along with the results of the chi-square tests of independence. We found statistically significant associations between treatment status and students’ coded responses for all analyses except the group belonging outcome for Exercise 2, which discussed social adversity at school.
Contingency Tables for Coded Responses to Exercises Tabulated by Treatment Status
In both Exercises 1 and 2, a statistically significantly higher percentage of treatment students compared to control students described experiencing academic or social adversity as common (shared experience), improving with time (improve with time), and something that can be overcome with peer support (social belonging). Specifically, 35% of treatment students acknowledged adversity as a common experience, while only 11% of control students reported the same. For social worries, 66% of treatment students and 34% of control students reported them as a common and shared experience. Further, 60% to 62% of treatment students mentioned that academic and social worries resolve over time, but only 1% to 28% of control students did. Peer support in dealing with adversity was mentioned by 11% to 41% of treatment students, whereas only 1% to 28% of control students mentioned the same. When examining outcomes for group belonging, the proportion of students citing positive experiences with teachers, administrators, or school in general differed by treatment condition for the first writing exercise (χ2(1, N = 435) = 43.74, p = 0.000), but not for the second exercise (χ2(1, N = 435) = 2.79, p = 0.10). Specifically, after completing the first exercise, 32% of treatment students mentioned overcoming adversity through positive experiences with teacher, administrators, or the school compared to only 4% of control students. For Exercise 2, though, a similar number of treatment students (13%) and control students (20%) cited an example of group belonging as a means of overcoming adversity.
Discussion
Although two prior double-blind randomized controlled trials yielded replicated impacts at scale within two distinct district contexts, the current study did not uncover the same main effects on students’ academic or behavioral outcomes. In this case, there were no statistically significant main effects on the preregistered student academic and behavioral outcomes. However, additional exploratory analyses revealed that those students who experienced greater behavioral challenges during the transition to middle school benefitted from intervention more than those who experienced fewer or no disciplinary incidents. Similarly, potentially negatively stereotyped students who struggled with poorer grades during the transition showed larger improvements than their non-stereotyped peers in their terms 2–4 GPAs after intervention. Although these two interaction effects appear promising, follow-up analyses of the marginal effects for neither outcome revealed statistically significant evidence of impact. These mixed results may be attributed to several possible explanations.
First, unlike prior studies of the middle school belonging intervention, it is interesting that only those subgroups, composed of students experiencing initial challenges and those students belonging to potentially negatively stereotyped racial/ethnic groups, responded more positively to intervention than their counterparts. These results mirror the college-level findings revealing stronger effects for students of color (Walton & Cohen, 2007, 2011) and for those students who are initially underperforming in school (Wilson & Linville, 1982, 1985). It is possible that contextual differences between the current site and the prior two districts in which the middle school belonging intervention was administered contributed to these interaction effects and an inability to replicate positive main effects. The GPA interaction effect for potentially negatively stereotyped students is also important, as GPAs depend heavily on past academic performance and are difficult to change. Indeed, comprehensive, schoolwide behavioral interventions can take five to seven years to produce changes in academic outcomes (Madigan et al., 2016).
Empirical indicators of which differences may have contributed to the current results are difficult to pinpoint. One of our findings that may have implications is that students from military families are less likely than those from civilian families to be suspended or receive a disciplinary referral. Indeed, suspensions and referrals in our sample, and in general, are relatively infrequent. As such, these outcomes are difficult to change for students who rarely receive a referral or suspension, which makes the positive moderation result for reducing disciplinary referrals among those treatment students experiencing greater numbers of term 1 referrals predictable. Indeed, national data from the 2020–21 school year indicate that 28% of suspended students were suspended more than once in the same school year, suggesting that many students who are disciplined in school do not receive the needed support to realize lasting behavioral change (Jain et al., 2024). In this case, the middle school belonging intervention seems to have had stronger effects for those students struggling most with disciplinary problems.
The large population of military families in KISD along with the relatively high percentages of economically disadvantaged and non-white students are other clear characteristics that distinguish the current site from the prior two. Although we find no evidence that the middle school belonging intervention affected the military students in our sample differently than civilian students, the overall culture of the community and schools in Killeen is certainly impacted by the significant presence of the many United States Army families and Fort Cavazos, which employs more than five times the number of people than the next largest employer, KISD, in the area (City of Killeen, 2021).
Given the significant presence of the Army base and large population of military children, it is likely that the military families of Killeen contribute a great deal to the culture of the community and schools. Indeed, Wertsch (1991) suggests that military towns, like Killeen, are highly influenced by military culture. Some research suggests that military children have high resilience, perseverance, and are quite experienced and comfortable assimilating and welcoming newcomers across their many moves and school changes (Easterbrooks et al., 2013). Military children tend to share bonds with one another through common experiences that transcend race, religion, and nationality and embrace a strong culture of antiracism within the armed services (Wertsch, 1991). It is quite possible that these characteristics eased students’ middle school transitions in Killeen and caused them to see the school transition as relatively normative, which quelled belonging uncertainty. The Killeen middle school environments may have already been well situated to welcome transitioning students and, thus, the middle school belonging intervention may have had limited additional impact aside from, potentially, the students who were particularly struggling with poor academic performance and behavior during their middle school transition.
In addition, although responses of students to the manipulation check items regarding worries about academics signaled that the treatment did effectively mitigate these adversities, the responses to the manipulation check regarding social worries did not elicit the hypothesized reappraisals that we expected. Treatment students expressed future worries about social belonging no differently than control students. These manipulation check outcomes contrast those found in the prior studies of the middle school belonging intervention by Borman et al. (2019) and Pyne and Borman (2020), which showed consistent positive reappraisals concerning both academic and social worries. This failure to prompt the theorized mechanisms of positive reappraisals of the social challenges faced by transitioning middle school students is likely to explain, in part, the limited effects of the intervention.
Finally, our coding of the student exercises, which was not a task completed for either of the previous two studies of the middle school belonging intervention, revealed some potential insights. Treatment students’ written responses, relative to those of control students, did tend to more often reflect the important messages that academic adversity during the middle school transition is something commonly experienced and is relatively short-lived. However, smaller percentages of treatment students indicated that these adversities may be overcome through positive experiences with peers, teachers, administrators, or school in general. Only 11% of treatment students indicated that academic struggles could be overcome with the help of their peers (social belonging), and only 21% indicated teachers, administrators, or other staff (group belonging) might provide support. These frequencies, though, were statistically significantly greater than those from control students, of whom less than 1% indicated such responses regarding social and group belonging. Concerning social worries about fitting in and belonging, treatment students did report statistically greater frequencies of social belonging (41%), relative to control students (25%).
In contrast, treatment students were not more likely than control students to report teachers, administrators, and other school resources as positive sources of support. In addition to the primary message of normalizing adversity, the intervention prompts do mention that support is available—with an additional goal of trying to induce students’ help-seeking behaviors. It is possible that the intervention’s messages concerning the availability of support were not matched by actual support available in participating schools. If students actually encountered a lack of support, that could undermine the intervention message.
Because similar coding of the exercises was not conducted in the prior studies of the middle school belonging intervention, it is not possible to assess potential differences across replications. One known prior study has conducted similar coding of adolescents’ written responses to a high-school belonging intervention. Williams et al. (2020) successfully fielded an intervention that eased minoritized students’ transition to high school. The intervention focused on two themes akin to those emphasized here: (1) social challenges are normative and (2) they improve with time. Coded results of the written responses suggested that 41% of the students mentioned one or both of these theorized mechanisms for overcoming adversity during the transition. In our sample, students had the opportunity to write about either of these two themes in response to the two exercises and did so at rates of 35% to 66%. Although treatment students’ acknowledgements of social and group belonging are relatively lower, the intervention appears to have activated the two key theorized mechanisms associated with reappraisal of adversity—transitional challenges are normal and improve with time—at a rate that is similar or higher to those found for a high-school belonging intervention that achieved large positive impacts on minoritized students’ academic and behavioral outcomes.
Conclusion
Replications of educational interventions, and especially social-psychological interventions, are highly important to advance understanding of what works and under what conditions. The typical response to the second student exercise involving social worries about belonging in school was not found in this setting. Our manipulation check results suggest that future worries about fitting in socially were not mitigated relative to the control condition. Also, treatment students’ responses to the exercise cited their peers and adults in the school as social resources for overcoming belonging uncertainty at a relatively low overall rate compared to reappraising their academic and social worries as normative and transient. Perhaps students in this context are generally more experienced in handling adversities associated with school transitions and were able to easily overcome them without intervention. Perhaps students also tend to depend less on others in their schools to help and, instead, feel sufficient resilience and agency to manage their worries on their own. Such higher levels of agency and resiliency are reported among children who come from military families and communities (Easterbrooks et al., 2013).
On the other hand, with the second exercise targeting social worries concerning belonging and “fitting in” with peers administered in November, it is possible that the intervention messages concerning social worries may have come too late. Another consideration may be to present the exercises in the opposite order, or randomize their order and empirically evaluate the alternate sequencing of the two exercise. We, ultimately, do find some promising evidence, though, that minoritized students and students who experienced significant early struggles with their behavior benefited more from intervention.
Improving the overall contextual conditions that make the transition to middle school difficult especially for students facing existent transition risk related to their identities and/or school challenges is also an important endeavor. The middle school belonging intervention can help students cope with the status quo, but does little to change these conditions. Other strategies to encourage students’ help-seeking behavior and efforts to prompt teachers to provide the support needed for students to feel heard and supported would certainly complement the belonging intervention. Students may feel stronger connections to school when they experience more positive classroom management climates, tolerant disciplinary policies, and smaller school sizes (McNeely et al., 2002). Finally, initiatives to improve the overall school climate, such as implementing restorative practices to promote conflict resolution skills and strengthen community bonds, can also contribute to students’ positive academic and social development—particularly for those groups disproportionately affected by exclusionary discipline (Darling-Hammond, 2023).
It is interesting that, in comparison, similar belonging interventions helping negatively stereotyped or under-represented college students transition from high school have realized replicated success in at least a dozen contexts (Walton & Brady, 2021). Prior evidence from two replications suggests that the middle school transition is difficult for everyone and that all students tend to benefit from intervention (Borman et al., 2019; Pyne & Borman, 2020). Our current results may represent an anomaly or chance null finding. Transitioning to the middle school context and the adversities it generates may be more variable relative to the college transition. More evidence and more replications are needed to better understand the boundary conditions under which belonging interventions can help secondary students.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported here was supported by a grant from the Institute of Education Sciences, U.S. Department of Education (R305A160060). The content is the responsibility of the authors and does not necessarily represent the views of the supporting agency.
Notes
Note: This manuscript was accepted under the editorial team of Kara S. Finnigan, Editor in Chief.
Authors
GEOFFREY D. BORMAN is the Alice Wiley Snell Endowed Professor at Arizona State University, Mary Lou Fulton College, H.B. Farmer Education Building, 1050 S Forest Mall, Tempe, AZ 85281; email:
TRISHA H. BORMAN is a principal researcher at Measured Decisions, Inc., 8105 N. 47th Street, Paradise Valley, AZ 85253. email:
SO JUNG PARK is a senior researcher at the American Institutes for Research, 1400 Crystal Drive, 10th Floor, Arlington, VA 22202; email:
BO ZHU is a senior researcher at the American Institutes for Research, 1400 Crystal Drive, 10th Floor, Arlington, VA 22202; email: