Abstract
Despite growing economic opportunities in “middle-skills” science, technology, engineering, and mathematics (STEM) professions, the majority of community college STEM students leaves the STEM pipeline or withdraws from college altogether. We tested an intervention that addressed one reason why students abandon STEM: psychosocial barriers, including identity threats, a lack of belonging, and a mismatch between students’ values and those prevalent in STEM. Our intervention leveraged behavioral science (“nudging”) strategies as part of a 7-week summer campaign to encourage reenrollment, delivered via text message to 1,367 randomly selected first-year students at three community colleges. The intervention increased overall fall reenrollment by 7 percentage points; the effect was primarily observed among STEM students (+10 percentage points) and students taking summer courses (+10 percentage points). Our results indicate that interventions that target psychosocial barriers experienced by community college STEM students can increase retention and should be considered alongside broader reforms.
While community college enrollments have consistently declined since 2010 (Juszkiewicz, 2017), programs in science, technology, engineering, and mathematics (STEM) continue to expand (National Student Clearinghouse Research Center, 2017)—and for good reason. STEM careers are growing twice as fast as non-STEM careers, and STEM workers earn nearly 50% more on average than their non-STEM counterparts (Fayer, Lacey, & Watson, 2017). Given that half of these STEM jobs—“middle-skills” positions such as web developer, petroleum technician, and computer network support specialist—require less than a bachelor’s degree (Fayer et al., 2017; Rothwell, 2013), studying STEM at a community college is an increasingly attractive postsecondary option. In fact, more than 20% of community college students major in STEM at some point (Chen & Soldner, 2013). The majority, however, will not earn a STEM degree. According to the National Science Foundation (2018), only 13% of community college STEM students graduate within 3 years, and over 40% leave college without a degree. Moreover, approximately 30% of these students earn a credential in a non-STEM major (Chen & Soldner, 2013).
Stereotypes Within STEM
There are multiple reasons why so many community college students abandon STEM, not least of which is that STEM majors and careers are rife with stereotypes about who belongs there and who does not. Psychologists have long documented how stereotype threat—anxiety that one’s performance is judged through the lens of a negative cultural stereotype and may serve to confirm that stereotype—undermines academic performance among marginalized groups in higher education, such as racial and ethnic minorities (Steele & Aronson, 1995), students from lower income backgrounds (Croizet & Clare, 1998; Harrison, Stevens, Monty, & Coakley, 2006), and women participating in science and mathematics (Murphy, Steele, & Gross, 2007; Shapiro & Williams, 2012; Spencer, Steele, & Quinn, 1999). Students from underrepresented racial and ethnic groups also face racism embedded in STEM education, such as the concept of “mathematics as Whiteness” (Battey & Leyva, 2016) and various microaggressions that reinforce the perceived inferiority of these students (Grossman & Porche, 2014).
There are stereotypes about STEM, however, that speak less to who should be in STEM and more so why one should be in STEM. Notably, the goal congruity perspective argues that many students are dissuaded from STEM because they hold erroneous stereotypes about STEM education and careers, which are incongruent with their personal values and goals (Diekman, Brown, Johnston, & Clark, 2010). One such stereotype is that STEM research is indifferent toward the communal good (Smith, Brown, Thoman, & Deemer, 2015), yet many students want to pursue a path where they can make a difference in the world. Highlighting the societal benefits of STEM research, however, can increase students’ motivation to study STEM (Brown, Smith, Thoman, Allen, & Muragishi, 2015). Another stereotype is that scientific work is largely isolated and agentic, a turnoff for many students who prefer to work collaboratively. Reframing STEM careers as more communal increases students’ interest in STEM and eliminates the gender gap in STEM motivation (Diekman, Clark, Johnston, Brown, & Steinberg, 2011).
These motivational reasons for departure from the STEM pipeline, however, are not independent of identity. Women, racial and ethnic minority students, and students from lower income backgrounds tend to be more attracted to communal and collectivist careers than male, White, and middle class students (Stephens, Fryberg, Markus, Johnson, & Covarrubias, 2012), and goal congruency has been posited as one explanation for gender gaps in STEM success (Diekman et al., 2010). But any student, regardless of gender, race, ethnicity, or educational pathway, may withdraw from their program of study—or higher education altogether—if they experience a clash between their values and those embedded in the educational environment. In the current study, therefore, we targeted psychosocial barriers to success in order to boost reenrollment among community college students and, in particular, STEM students.
Wise Interventions
The current study follows closely the idea of “wise interventions,” which are programs designed to create behavior changes that endure past the end of treatment (Walton, 2014). Instead of attempting to directly change a maladaptive behavior, a wise intervention targets the attitude, belief, or mind-set underlying that behavior. Modifying that cognition can then trigger a recursive psychological process in which a person’s newly minted beliefs are reinforced by their environment. The three approaches central to our intervention—social belonging, values affirmation, and a self-transcendent purpose for learning—have all been shown to increase success by targeting students’ perspectives on college and learning, thus creating the foundation for long-lasting changes in behavior and improved educational outcomes, such as persistence.
Social Belonging
Decades of research have affirmed social identities as a key factor in students’ success in both college and in STEM (Syed, Azmitia, & Cooper, 2011). But as described earlier, stereotype threat can undermine students’ sense of belonging, depress their motivation, and harm their performance (Cohen & Garcia, 2009; Good, Rattan, & Dweck, 2012; Steele & Aronson, 1995). One way to combat this threat is to modify the perceived norms attached to those social identities. For example, women in introductory chemistry had significantly higher grades at the end of the semester (a B vs. a C+ average) after reading a letter, purportedly from a female graduate student, which normalized struggle and a lack of belonging in early college science courses (Herrmann et al., 2016). More generally, first-year, African American college students who read testimonials framing adversity in college as both common and short-lived showed improved grade-point averages (GPAs) over the 3 years of college following the initial intervention (Walton & Cohen, 2007, 2011). In both studies, dissociating students’ social identities from obstacles faced in college proved effective in mitigating performance detriments commonly observed among students from higher risk backgrounds.
Values Affirmation
Another way to mitigate identity threat is to provide students with the opportunity to reestablish their self-worth. Like a cast that protects a broken bone while it heals, focusing students on sources of resilience, such as social support, religious beliefs, or other talents and interests, helps maintain academic motivation while the identity threat passes (Harackiewicz & Priniski, 2018). For example, the gender gap in performance typically observed in an introductory college physics course completely disappeared after students wrote brief essays about their most important values (Miyake et al., 2010). A similar intervention improved undergraduates’ performance on a final exam in introductory biology (Jordt et al., 2017). These brief exercises may have helped students process any threatening experiences in their STEM courses that otherwise would have derailed their academic progress.
Self-Transcendent Purposes for Learning
College students’ persistence is partly determined by their motivation, and their motivation is strongly influenced by their reasons for pursuing a degree. Students who more strongly endorse self-transcendent purposes for learning, such as wanting to help other people and make a difference in society, demonstrate higher levels of academic diligence, self-control, grit, and, most important, persistence (Yeager et al., 2014). When ninth-grade students wrote about self-transcendent purposes for learning, their GPAs in science courses significantly improved during that quarter. And as described earlier, framing STEM research as benefiting the common good increased college students’ positivity toward, and motivation to participate in, a STEM career (Brown et al., 2015). Connecting college and STEM to students’ self-transcendent motives may be a powerful way to encourage them to stay enrolled.
Delivering Interventions via Text Messaging
Perhaps as important as what makes up an intervention is how it is delivered. With mobile phone ownership in the United States over 95% and approaching 100% among 18- to 29-year-olds (Pew Research Center, 2018), text messaging is a highly efficient, scalable, and direct means of reaching college students. Moreover, connecting with students by text message is an effective way to influence their behavior. For example, text message campaigns to help college-intending high school graduates complete prematriculation requirements (e.g., financial aid, registration) can decrease “summer melt” by up to 7 percentage points (Castleman & Page, 2015; Page & Gehlbach, 2017). Similarly, a text message campaign to help first-year college students renew their financial aid increased persistence at community colleges by at least 12 percentage points (Castleman & Page, 2016).
Aside from pragmatic guidance, colleges can also provide socioemotional support over text messaging. A text-based intervention to support 2- and 4-year undergraduates nearing the end of college increased persistence and graduation rates among those students at highest risk to withdraw (Mabel, Castleman, & Bettinger, 2017). In the United Kingdom, a text message campaign that connected college students with additional sources of on-campus support improved students’ attendance, performance, and persistence (Deighton et al., 2017). And various mHealth interventions have employed text messaging to change college students’ health behaviors, such as exercising more, drinking less, and quitting smoking (e.g., Lewis et al., 2018; Müssener et al., 2016; Patrick et al., 2009). Together, these studies demonstrate that text messaging is a highly effective means of conveying guidance and support to college students.
The Current Study
This study is part of a larger project (“Nudging to STEM Success”) coordinated by Jobs for the Future, a national nonprofit organization whose mission is to align the U.S. workforce and education systems to ensure access to economic advancement for all. The overall goal of Nudging to STEM Success was to increase persistence and graduation rates among community college students participating in STEM pathways. Here we report on the results of an experimental study conducted in the summer of 2017 with the specific aim of increasing student reenrollment after the first year of college. We tested our intervention during summer because this is a particularly ripe time for students to leave college, as many have disengaged from academics to focus on work and family. Moreover, transition periods like summer are an efficacious time to influence students’ decision making around college enrollment (Castleman & Page, 2015; Page & Gehlbach, 2017).
In this randomized controlled trial, we used text messaging to deliver interactive messages rooted in behavioral science principles—what we refer to as “nudges”—to first-year community college students over the summer. The primary purpose of our nudges was to help nonenrolled students register for classes and to address the psychosocial barriers that may cause both nonenrolled and enrolled students to withdraw from college. We hypothesized that students who received nudges would be more likely to return for their second year of community college compared with students who received business-as-usual support from their college. We also explored the differential impact of this intervention on reenrollment among subgroups, specifically STEM students, women, students of color, and students enrolled in summer courses.
Method
Participants
Participants were 2,759 students enrolled in the Spring 2017 term at three different U.S. community colleges. Students were selected from class lists with defined majors, with one college restricting participation to students in STEM pathways. As displayed in Table 1, 38% of students in this study identified as men and 62% as women, and the majority identified as either White (72%) or as Black/African American (17%). Moreover, 70% of students were enrolled in STEM pathways and 47% of students were enrolled in summer courses. Selected students were randomly assigned into experimental conditions by each college using Microsoft Excel’s RAND function, resulting in 1,367 students in the intervention arm and 1,392 students in the control arm.
Descriptive Statistics Overall and by Treatment Status
Intervention
Nudges (examples of which are provided in the appendix) were developed by the Behavioral Research Team at Persistence Plus, an educational company that collaborates with higher education institutions to increase student success via behavioral science, in consultation with student support specialists at the three community colleges. Students receiving the intervention were automatically enrolled in the text messaging service (Bergman, Lasky-Fink, & Rogers, 2017) using mobile phone numbers on file with the college, but could opt out of the nudges at any time over the summer by replying with “STOP” or “UNSUBSCRIBE.” Beginning on June 28, 2017, Persistence Plus proactively delivered two nudges per week to students in the treatment group via text message. When a student responded to a question asked by Persistence Plus, it would elicit a dialogue between that student and the platform that would provide additional advice and support. Only when a student asked a question outside the scope of the Persistence Plus system did a human intervene to respond.
All students in the intervention arm received nudges encouraging them to return for the Fall 2017 term. Students who were not enrolled for Fall 2017 (based on data received from each college at the beginning of the intervention period) received alternative nudges designed to diagnose why they had not reenrolled (e.g., issues with financial aid, confusion over the registration process), motivate them to leverage campus resources in order to resolve their issue, and remind them about the benefits of continued enrollment in community college. The intervention ended on August 18, 2017, just prior to the start of the Fall 2017 term at each college.
Analysis
We first tested equivalence of the treatment and control groups with a linear regression in which each of the baseline demographics were regressed on treatment assignment. For this and subsequent analyses, race categories were combined into one binary race variable (0 = White, 1 = student of color) due to the small numbers of students in the non-White race categories.
Reenrollment, defined as a student remaining enrolled in at least one class past their college’s census date during the Fall 2017 semester, was the outcome of interest in this study. To test the efficacy of our summer nudge intervention on the binary outcome of reenrolled or not reenrolled in Fall 2017, we estimated and report both linear probability and logit models using intent-to-treat estimates in which all students who were assigned to experimental conditions were included in the analysis, regardless of their engagement postrandomization. Predictors included experimental condition, student sex (0 = male, 1 = female), race, summer enrollment, whether they majored in a STEM pathway, and overall GPA. All demographic and outcome variables were derived from official registrar data provided by each college in September 2017. To further understand for whom the nudges were most successful, we tested our treatment intervention effect separately for subgroups of students based on each demographic factor for which we found a significant main effect. Finally, we explored the heterogeneity of treatment effects based on students’ levels of engagement with the intervention.
Results
Baseline Equivalence
As displayed in Table 2, combining the treatment and control categories from each of the three schools resulted in well-balanced groups, with one exception: a significantly greater percentage of STEM students were randomized into the control group (72%) compared with the treatment group (68%). We judge this not to be of concern for two reasons. First, this is the only difference between groups that we observed, and the magnitude of the difference is not overwhelming. Second, because our subsequent analyses show that the intervention worked primarily for students enrolled in STEM pathways, the direction of this difference would likely attenuate the impact of the treatment, making ours a more conservative test and less likely to produce a Type I error.
Assessing Balance of Baseline Demographics in Randomization
Note. Tests of baseline equivalence based on linear regression models. Robust standard errors in parentheses.
p < .05.
Students’ Interaction With Nudges
Table 3 details the engagement of students in the treatment group with the intervention. Overall, nearly all students targeted for treatment received nudges (96%). The remaining 4% had phone numbers that did not receive text messages for various reasons (e.g., landlines, cell phones without service). Only a small portion of students unsubscribed from the nudges immediately on the first day (5%), and over the course of the summer the percentage of students who opted out remained small (cumulatively 15%). Students received approximately 12 nudges each, including 8 questions on average, and answered 25% of the time. Nineteen percent of students responded to at least one nudge, and the most active student sent 12 replies. Just under half of all students in the treatment group were enrolled in summer courses, and compared with students who were not enrolled during the summer, these students were less likely to unsubscribe (13% vs. 17%), χ2(1) = 3.88, p < .05, and more likely to respond at least once (28% vs. 14%), χ2(1) = 49.84, p < .001. We found no significant differences between students in STEM pathways and those not in STEM regarding engagement or number of students who opted out of the intervention.
Student Engagement With Persistence Plus Nudges
p < .05. **p < .001.
Treatment Effects
Linear probability and odds ratios estimates examining main effects are presented in Table 4. Linear probabilities are highlighted below for brevity and ease of interpretation, but all significant effects are demonstrated in both linear and logit models. We first examined the impact of assignment to the treatment group relative to the control group on retention and observed that receiving nudges improved reenrollment by 7 percentage points compared with the control group rate of 62%. Additionally, higher reenrollment was observed among students participating in a non-STEM pathway and students enrolled in summer courses. Race and gender were not significant predictors of retention and were therefore omitted from subsequent linear and logit models examining the effect of the intervention on student subgroups.
Effect of Nudging Intervention and Individual Factors on Fall Reenrollment
Note. Robust standard errors in parentheses.
p < .05. **p < .001.
We examined effects separately for the two binary demographic variables for which we found significant impact on retention: STEM pathway participation and summer course enrollment (Table 5). As illustrated in Figure 1A, we found significant effects of the intervention on retention among students participating in STEM, with those who received nudges reenrolling at a rate 10 percentage points higher than the control group rate of 58%. In fact, STEM students who received nudges reenrolled at nearly the same rate as non-STEM students, closing the observed retention gap based on program of study. No treatment impact was found for students who were not enrolled in STEM pathways. The intervention also improved retention by 10 percentage points above the control group mean of 71% for students who were enrolled in courses during the summer (Figure 1B). Although we observed a difference of nearly 4 percentage points in the predicted direction among students not enrolled in summer courses, this effect failed to reach statistical significance.
Effect of Nudging Intervention on Fall Reenrollment Among Student Subgroups
Note. Robust standard errors in parentheses.
p < .05. **p < .001.
Interactions Between Experimental Condition and (A) Participation in a STEM Pathway and (B) Summer Enrollment, on Fall Term Reenrollment.
Heterogeneity of Treatment Effects by Student Engagement
To explore heterogeneity of treatment effects based on student engagement, we first divided students into three groups based on the level of treatment received: those who unsubscribed on the first day of the intervention (i.e., no treatment; n = 74), those who unsubscribed after the first day but before the end of the intervention (i.e., partial treatment; n = 132), and those who remained subscribed throughout the intervention (i.e., full treatment; n = 1,161). Moreover, we subdivided students who received full treatment into those who never responded to a nudge (n = 898) and those who responded at least once (n = 263). Because only 20 students responded to at least one nudge and then later unsubscribed, we could not subdivide those students in a similar fashion.
For students who unsubscribed on the first day, we observed a 26-percentage-point lower rate of reenrollment compared with the control group rate of 62%, p < .001. Students who unsubscribed after the first day (64% reenrollment rate) did not significantly differ from the control group. Finally, students who remained subscribed for the duration of the intervention were more likely to reenroll than the control group: A difference of 6 percentage points for students who never responded (p = .004) and a difference of 24 percentage points for students who responded at least once (p < .001).
Discussion
In this randomized controlled trial, we examined whether a summer nudge campaign designed to address common psychosocial challenges among community college students, and specifically STEM students, could bolster retention. We found that students in STEM pathways and students enrolled in summer courses were more likely to return to community college in the fall when nudged by text message. Non-STEM students were not affected by the intervention, and although students not enrolled during the summer showed a small gain in retention, it failed to reach statistical significance.
One explanation for the observed impact of our intervention on STEM students that was absent among non-STEM students is baseline retention. STEM students were 20% less likely to return for the fall term than non-STEM students and, therefore, may have been more in need of additional support. Retention for non-STEM students, hovering around 70%, may have hit a plateau at which our wise interventions had less effect. Another explanation is that STEM students may have been more receptive to the kinds of nudges we provided, which focused on alleviating identity threats. STEM students face many psychosocial threats that challenge whether they belong in a STEM program, or in college at all (e.g., Battey & Leyva, 2016; Diekman et al., 2010; Shapiro & Williams, 2012). Moreover, evidence suggests that the “cultural mismatch” experienced by many students at 4-year universities (Stephens et al., 2012) is not felt as strongly at 2-year colleges (Tibbetts, Priniski, Hecht, Borman, & Harackiewicz, 2018), possibly dampening the effects of our wise interventions on the general student population.
More perplexing was our finding that nudges boosted retention for students enrolled in the summer, but not significantly so for those away from college, despite students enrolled in summer courses being over 36% more likely to return than their peers. Prior interventions have focused on summertime, in part, to keep college salient when students are disengaged from an academic mind-set (e.g., Castleman & Page 2015; Page & Gehlbach, 2017). Perhaps nudges addressing psychosocial barriers failed to resonate with students who were not actively experiencing those kinds of situations. Moreover, some students may have chosen not to take summer courses because they experienced those types of identity threats and may have already decided to withdraw. Students currently taking college courses, on the other hand, may be more likely to be facing identity threats and other psychosocial pressures, making our nudges more effective in influencing their decision whether to reenroll.
Finally, we examined heterogeneity of our treatment effects based on students’ engagement with the intervention. Students who unsubscribed on the first day were nearly half as likely to reenroll as students in the control group. Given that these groups of students had the same experience, save for one introductory text message, this result suggests that unsubscribing simply flagged students already on the cusp of withdrawal. In a sense, this model of support could provide an additional “early alert” for students at risk to leave. Recall, however, that only 5% of students unsubscribed immediately and thus this result should be interpreted cautiously. For students who received partial treatment, one could interpret the nonsignificant difference with the control group as evidence that the intervention failed when not delivered in full. However, given the much lower reenrollment rate of students who unsubscribed from the intervention immediately, partial treatment may have had the desired effect of motivating some of the most at risk group of students to reenroll. Finally, students who passively received the full treatment without responding reenrolled at nearly the same rate as the overall treatment effect, providing evidence that the intervention works in the absence of active responding. Although responding was associated with an even higher reenrollment rate, it is likely this effect is largely attributable to unmeasured third variables driving both student engagement and reenrollment.
Implications for Community Colleges
These results offer several implications for how community colleges retain students, both generally and within the STEM pipeline. All three community colleges in this research have implemented guided STEM pathways, in which students progress through a course of study with predesignated classes delivered in a predetermined order (Jenkins, Lahr, & Fink, 2017), yet retention of their STEM students was still bolstered by our intervention. Even as more colleges pursue programmatic reforms such as the guided pathways model, our results suggest students can still benefit from supports that address identity threats—provided via text messages, as we and others have done (e.g., Deighton et al., 2017; Herrmann et al., 2016), as well through classroom-based interventions (Miyake et al., 2010) and orientation programs (Stephens, Hamedani, & Destin, 2014). Moreover, although our intervention did not directly address the structural inequities that produce identity threats and deter many students from pursuing STEM, the efficacy of these wise interventions may help guide community colleges as they design comprehensive efforts to create a diverse and inclusive STEM pipeline. Finally, as many community colleges are already aware, encouraging summer enrollment appears to be an important part of retention. Students in summer classes were not only much more likely to return for Fall 2017 compared with students not enrolled, but our intervention was more effective at motivating reenrollment among summer enrolled students.
Limitations
There are several limitations to the study. Given prior research on the impact of wise interventions on first-generation students (e.g., Harackiewicz et al., 2014; Stephens et al., 2014; Tibbetts et al., 2018), examining generational status as a moderator would have shed additional light on the effect of our intervention. Each college, however, had unique definitions of first-generation student and we could not reconcile them into a single, meaningful variable. Additionally, our data only included whether students returned to the same community college they attended in Spring 2017. We could not account, therefore, for successful transfers to other community colleges or 4-year universities. Second, our intervention took an omnibus approach in which we used several techniques to address multiple psychosocial barriers to success that community college students face. Future studies could isolate different mechanisms to determine what works best with which populations of students. Moreover, experimental designs could determine better than our analysis of heterogeneity whether differing treatment amounts and the interactivity of nudges have an impact on student behavior. Finally, we had a defined aim in the current study to boost reenrollment over the summer and help more community college students stay on track toward a credential. Future studies should examine how these kinds of psychosocial interventions could influence other academic metrics (e.g., GPA; credits earned) as well as long-term outcomes, namely graduation rates.
Conclusions
We demonstrated that a brief, scalable program of nudges, built on evidence-based wise interventions (Walton, 2014) and delivered via text message, could improve retention among STEM students at three U.S. community colleges. Given that over 1 million STEM jobs could become available in the United States by 2024 (Fayer et al., 2017), keeping the 70% of community college STEM students who leave the pipeline (Chen & Soldner, 2013; National Science Foundation, 2018) engaged with STEM is imperative for the continued growth of community colleges and their local economies. While much attention and work is needed to rectify the underlying structural inequities that foster the identity threats that lead many STEM students to withdraw—particularly underrepresented students like women, racial and ethnic minorities, and students from lower income backgrounds—this study adds to the bourgeoning array of interventions that can aid these students immediately (e.g., Brown et al., 2015; Diekman et al., 2011; Herrmann et al., 2016; Jordt et al., 2017; Miyake et al., 2010). Community colleges should explore both psychosocial strategies for student support, as well as the efficacy of text messaging as a communication method, alongside broader reform efforts such as guided pathways (Jenkins et al., 2017) aimed to keep students enrolled in college and participating in STEM fields.
Footnotes
Appendix
Acknowledgements
The authors would like to thank the Leona M. and Harry B. Helmsley Charitable Trust, and especially Ryan Kelsey, for supporting this research; Jobs for the Future, and in particular Barbara Endel, Amy Girardi, and Lisa Soricone, for their leadership on this project; our dedicated colleagues at our partner colleges for their collaboration on this research; Dr. Gerardo Ramirez for his expert guidance on intervention design; and the rest of our team at Persistence Plus for their essential contributions.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors prepared the work as employees of Persistence Plus, LLC.
Author
ROSS E. O’HARA is a behavioral researcher at Persistence Plus. He uses his expertise in behavioral science to develop nudge interventions to help more college students graduate.
BETSY SPARROW is a behavioral researcher at Persistence Plus. She leverages data science to reveal new insights about college students’ challenges and how institutions can help students graduate.