Sage Journals: Discover world-class research

Abstract

Technology-facilitated interventions following high school graduation have shown promise for increasing the likelihood of college matriculation, but we know little about how to fine-tune these tools. I conducted an experiment in which college-intending Tennessee high school graduates received informational messages in distinct behavioral frames: business-as-usual, in which they received the same messages as the prior cohort; loss aversion, which emphasized what students would lose if they did not act; reduction of ambiguity, which provided details on necessary actions and anticipated completion times; and peer support, which encouraged students to work with friends on enrollment tasks. There was no main effect of the treatment frames. Heterogeneity analyses suggest that, at certain eligibility checkpoints, a loss aversion frame may negatively affect men and the peer support frame may negatively affect first-generation and Black participants. I situate the findings in the literature and recommend future directions for research on informational intervention delivery.

Keywords

behavioral framing college access summer melt text messaging free college randomized control trial

Since 2015, Tennessee has promised resident students tuition-free community or technical college after high school graduation. The statewide last-dollar scholarship program, Tennessee Promise (TN Promise), funds the remainder of a student’s tuition and fees after federal and other state financial aid awards have been deducted from their bill. TN Promise students, regardless of high school achievement or family circumstances, do not pay out of pocket for up to five semesters of postsecondary training.

However, “summer melt” among would-be scholarship recipients suggests that TN Promise students may benefit from more than financial support during the college transition (see Castleman & Page, 2014; Ceja, 2013, for detail on summer melt, the phenomenon of nonmatriculation among college-intending students). Among all individuals who complete TN Promise eligibility requirements throughout their senior year of high school, only half enroll in college the semester after graduation (Tennessee Higher Education Commission [THEC], 2017). Maintaining TN Promise eligibility through graduation is no easy feat: High school seniors must submit a TN Promise application, complete the notoriously complex Free Application for Federal Student Aid (FAFSA), attend two “team meetings” to get scholarship information and meet their volunteer mentor, and apply to college.

To facilitate navigation of the scholarship requirements and the postsecondary transition, the TN Promise program provides informational supports to supplement the guidance of school-based counselors. Through nonprofit partners such as tnAchieves, which in 2019–2020 serves students in 90 of Tennessee’s 95 counties, high school seniors receive direct communication about TN Promise via email and in person from both nonprofit-based counselors and volunteer mentors. After graduation, tnAchieves ramps up communication, adding twice-monthly text messages through the end of students’ first year of college.

Policymakers and practitioners are eager to improve transition supports for college-intending Tennesseans, and informational text messages during the college transition offer low-cost opportunities to test changes in practice. The texting campaigns mounted by tnAchieves and other college-focused nonprofits are crafted based on the empirical evidence that such interventions promote college matriculation (e.g., Castleman, 2015). However, text message content may offer little guidance on how to complete required actions or fail to make salient the consequences of inaction. Without detail regarding what is at stake, how and when to take action, or social motivation for doing so, college-intending students may not prioritize completion of required tasks.

At least three theories suggest ways to increase the likelihood that students act on information. First, messages may motivate action by emphasizing what students stand to lose by failing to do so (Kahneman & Tversky, 1979). Alternatively, providing clarifying next steps may make decision-making less complex and increase the likelihood that individuals will take action (Milkman et al., 2012; Nickerson & Rogers, 2010). A third theory encourages students to work with peers on academics and college-related logistical tasks to build socio-academic integration (Deil-Amen, 2011). Strategic framing of messages may induce students to take action on directives and lead to matriculation.

I partnered with tnAchieves to test this hypothesis. In May 2017, I randomized college-intending class of 2017 Tennessee high school graduates into four groups that all received informational messages, albeit in distinct behavioral frames: business as usual, in which they received the same messages from the prior year’s informational campaign; loss aversion, which emphasized what students would lose if they do not act on the information; reduction of implementation ambiguity, which provided details on the requisite steps for action, as well as the amount of time it would take to complete the task; and peer support, which encouraged students to work with friends to ensure they acted, promoting socio-academic integration (Deil-Amen, 2011). I answer the following questions: Do TN Promise–eligible, college-intending high school graduates who receive informational text messages in various behavioral frames enroll and succeed in college at different rates? Does the impact of text message framing vary by student characteristics?

This study considers the relative efficacy of distinct behavioral frames in a statewide text messaging intervention in a tuition-free college environment. I find no evidence that text messages crafted in strategic behavioral frames have a differential effect on scholarship maintenance relative to receipt of business-as-usual text messages. Heterogeneity analyses suggest that, at some eligibility checkpoints, certain frames may negatively affect particular subgroups relative to the business-as-usual frame. These results contribute to an ongoing conversation about how to scale and fine-tune text messaging campaigns as the field of text-based, college transition interventions enters adolescence.

Promise Programs and the Tennessee Context

Tuition-free guarantees for 2- or 4-year college are gaining ground across the country. While “free” college programs are not novel, they have grown increasingly popular since the Great Recession amid reports of soaring college debt and postsecondary-centric forecasts for the future of work. “Promise” programs range in size and scope (Perna & Leigh, 2017): some are city-based, privately funded programs, others provide state grants for workforce-oriented training, and some fund 4-year scholarships for in-state residents to pursue a bachelor’s degree and remain in state for work. Over a dozen states now cover college tuition for some students, and nearly two dozen state legislatures considered “free college” bill this past legislative session (Quinton, 2019). Both Republican- and Democrat-controlled state legislatures have enacted Promise programs, and a number of Democratic presidential candidates have expressed support for a federal tuition-free college program. Empirical evidence suggests that tuition-free college programs have limited success addressing social inequities (e.g., Deming & Walters, 2017; Murphy et al., 2017), but they continue to grow in popularity.

Tennessee’s last-dollar Promise scholarship guarantees a tuition-free 2-year postsecondary education for Tennesseans who have just graduated with a high school diploma and who enroll at a community or technical college (known as Tennessee Colleges of Applied Technology, or TCATs). TN Promise, which in 2019–2020 is serving its fifth entering cohort, has limited eligibility criteria to ensure broad access. High school seniors who graduate from an eligible Tennessee high school, complete a Tennessee home school program, or, prior to their 19th birthday, obtain a General Education Development (GED) or High School Equivalency Test (HiSET) diploma, must apply for the scholarship by the November deadline of their senior year or the year in which they will obtain the GED/HiSET. Students must submit the FAFSA in January of their senior year, communicate with a volunteer mentor, and attend mandatory informational meetings. To maintain eligibility, students must enroll in college full-time, complete their FAFSA, participate in the mentoring program, and perform 8 hours of community service prior to each term the award is received. In addition to providing last-dollar tuition and fee support, TN Promise was designed to facilitate access to information about college transition and success. The state’s nonprofit partners share information with students via text and email and focus in particular on providing advising support during the months between high school graduation and college matriculation.

Economic theory suggests that Tennessee should expect increases in enrollment at community and technical colleges by making postsecondary education tuition-free. By lowering the personal cost of attending college, the state reduces the cost of attendance to the opportunity cost of lost wages and time during enrollment. Indeed, enrollment gains in the early cohorts of TN Promise suggest that some barriers to entry are mitigated by TN Promise’s financial support or the simplification of messaging around the cost of college (Carruthers & Fox, 2016). Since Promise scholarship implementation in 2015–2016, first-time enrollment in public postsecondary institutions has increased by 13% (THEC, 2017). Across the first three cohorts, roughly 80% of high school seniors applied for the TN Promise by the fall application deadline and roughly 90% of Promise-eligible students filed their FAFSA by the January deadline (THEC, 2017).

However, only 28% of initial TN Promise applicants and 30% of FAFSA-completers enroll for the fall semester (THEC, 2017). The high rate of college pipeline attrition among college-intending, Promise-eligible students reinforces the notion that nonfinancial factors prevent students from matriculating. While there is likely natural attrition as students elect to pursue other education or work opportunities, the burden of logistical and procedural tasks may contribute to this attrition among students who appear college-intending during their senior year. This experiment tested whether strategic framing of informational text messages could increase the power of this postsecondary transition support to mitigate summer melt.

Prior Research on Mitigating Summer Melt

Estimates by the U.S. Department of Education and leading researchers contend that somewhere between 10% and 40% of college-intending high school graduates do not arrive on campus in the fall (Castleman & Page, 2014; Ceja, 2013). Summer melt can be considered a manifestation of the information gaps that have been well-documented by researchers and practitioners. Melt disproportionately affects low-income and first-generation college students (Arnold et al., 2009; Castleman & Page, 2014), so mitigating summer melt represents both a moral and an economic imperative that warrants intervention and research. Fortunately, innovative programming and individualized, automated supports have been shown to ameliorate summer melt (Castleman & Page, 2015).

At the same time, a body of research in behavioral economics has gathered substantial evidence of the importance and efficacy of the framing of information in decision-making processes (see Thaler & Sunstein, 2008). In the postsecondary context, suboptimal performance on matriculation-related tasks hinders enrollment. The provision of appropriately timed information has been shown to mitigate rates of summer melt. Researchers contend that interventions that encourage students to complete requisite steps for enrollment have increased student success by crafting choice architecture in a way that optimizes transition-related behaviors (see Castleman, 2015). Such interventions hold promise for increasing college going among students at the margins of enrollment, particularly for first-generation college students.

Practitioners, researchers, and policymakers have explored various methods for increasing the likelihood of completion of these important tasks. Interventions designed to mitigate summer melt often focus on providing information and building social connections to individuals with college-going social capital. Advising and mentoring have long been popular place-based interventions, aimed at connecting college-intending students with a social connection to a professional or a peer who can provide information about important tasks and emotional support (Avery et al., 2014). Some empirically tested interventions have explored the utility of principles of behavioral economics in encouraging action on the part of students in transition.

Behavioral Economics and Text Messaging as an Informational Intervention

While the provision of information is important to reducing disparities in postsecondary success between student populations and institutional sectors, the delivery and construction of information has increasingly been a focus of attention in the academic literature. Research has presented compelling evidence that the way in which information is presented affects decision making. This work in psychology and economics, pioneered by Richard Thaler and Cass Sunstein (2003), has shown that humans act on the same information in different ways depending on its presentation. Strategic, low-cost changes in choice architecture can make information more transparent and personalized. Researchers have empirically demonstrated the validity of this theoretical work across a number of fields, including finance, healthcare, and education (e.g., Beshears et al., 2009; Hastings & Weinstein, 2008). Proponents of choice architecture maintain that the framing of information can optimize its use for choices that prove to be more beneficial for individuals. How information is delivered, accessed, and framed matters.

The ubiquity of cell phone ownership has provided a low-cost, high-leverage platform for the deployment of strategies grounded in behavioral economics. There is mounting evidence that text messaging is a viable platform for delivering interventions to 21st-century youth given the popularity of SMS (short message service) communication. For example, in a study of automated text messaging “nudges,” Castleman and Page (2015) found that overall enrollment in 2-year institutions rose by over 3 percentage points and 4-year college enrollment increased by 4.5 percentage points. Castleman and Page (2014) also built on earlier findings (Bos et al., 2012; Carrell & Sacerdote, 2013) regarding the efficacy of near-peer advising by pairing an informational text message intervention with near-peer outreach to encourage matriculation. Text messaging interventions have also been successful outside of the domain of college enrollment, encouraging individuals to act on intentions to save money (Karlan et al., 2016), receive vaccinations (Stockwell et al., 2012), and increase involvement in their child’s schooling (Kraft & Dougherty, 2013).

Research in this domain is predicated on the notion that individuals intend to complete tasks but fail to do so because they are inattentive to the future consequences. Reminders may reduce the attentional failure or make more salient the time needed to complete tasks in advance of deadlines (Karlan et al., 2016). In the postsecondary context, text messages can focus students on important benchmarks and encourage proper time management during the matriculation process. Furthermore, students who are the first in their families to attend college may be less likely to receive these reminders from parents and advisors.

In spite of early positive evidence suggesting positive enrollment effects, new research has shown less consistent and positive effects on enrollment (e.g., Bergman et al., 2019; Phillips & Reber, 2018). In addition to inconsistent average effects, new estimates suggest that messages may be differentially effective by institutional context (e.g., community versus 4-year colleges; Castleman & Page, 2016) or student background (e.g., language spoken at home; Phillips & Reber, 2018). Furthermore, it remains unclear whether text message interventions contribute to student persistence and degree attainment. Consequently, there is ongoing scrutiny and debate of informational text messaging as a policy instrument, particularly for decisions that involve personal costs (e.g., college enrollment).

The inconsistent empirical evidence on the utility of informational text messaging begs further refinement of intervention strategies. One way in which researchers and practitioners have endeavored to increase the leverage of texting interventions is through optimizing the framing of information delivery. Early evidence exploring the relative efficacy of various strategic frames for prospective college students has shown little promise for positively affecting practice (Bergman et al., 2019; Bird et al., 2019). Bird et al. (2019) investigate the impact of both a national- and a state-level campaign to encourage financial aid application. The researchers find no evidence that different frames affect intervention efficacy (Bird et al., 2019).

Bergman et al. (2019) also test the relative efficacy of varied informational interventions on college affordability at a large scale. Their campaign intended to change the salience of tax benefits for three samples totaling one million prospective college-going Texans: rising high school seniors, enrolled postsecondary students, and students who had previously applied to college but were not enrolled. Bergman et al. (2019) do not find that their treatment arms change student outcomes. The nascent and inconclusive evidence on the relative efficacy of different informational frames warrants additional exploration.

This study considers the relative efficacy in the behavioral framing of messages, particularly for students who are eligible to attend community or technical college tuition-free. The intervention was designed to test the hypothesis that recipients may perceive informational messages as more or less attractive depending on how content highlights certain aspects of a decision. Examples of such theoretical frames include loss aversion (which maintains that the pain of losing is more powerful than the pleasure of gaining; Kahneman & Tversky, 1979; Smith, 1759/2010), reduction of implementation ambiguity (which contextualizes actions in a time frame and provides details on process before asking students to articulate their plans for completion; Nickerson & Rogers, 2010), and peer support (which draws on students’ connections to social networks, deepens campus integration, and signals behavioral expectations; Deil-Amen, 2011). Taking as given the short-term efficacy of informational messaging in general, this experiment aims to contribute to the literature around the relative impact of distinct behavioral frames on college transition outcomes.

Loss Aversion

Loss aversion is a well-documented phenomenon through which the magnitude of a person’s negative reaction to losing a quantity is far greater than the magnitude of their positive reaction to gaining the same amount (Kahneman & Tversky, 1979; Smith, 1759/2010). Individuals generally place a higher value on things that they “own” than they would put on an item prior to ownership, and they are wont to give up that possession once they have it. Consequently, representing ongoing eligibility for the TN Promise scholarship more saliently may induce student action in multiple ways. A loss aversion frame may both increase the salience of the magnitude of scholarship benefits, as well as amplify feelings of ownership, such that loss of eligibility would be perceived as a loss of money that previously “belonged” to the student. With consistent framing that represents scholarship eligibility or achievement of grades as something to be lost, rather than gained, students may be encouraged toward task completion and postsecondary success.

Reduction of Ambiguity

One reason that students may fail to complete logistical and procedural tasks necessary for college matriculation is lack of clarity or direction about how to complete the tasks. This lack of clarity may be exacerbated by natural procrastination: Individuals hesitate to make sacrifices at present in the interest of reaping returns at a later date (O’Donoghue & Rabin, 1999). Empirical research has shown that concretized planning information can increase the likelihood that individuals will take action, whether for voter turnout or preventative medical screenings (e.g., Milkman et al., 2012; Nickerson & Rogers, 2010). Students may plan to go to college but, for instance, fail to complete FAFSA verification or attend campus orientation. They may believe that enrolling is the right choice for their future, but their commitment wavers when the time comes to dedicate the time and energy to do so. Informational text messaging may combat the ambiguity of implementation and curtail the influence of time inconsistency through the provision of more concrete planning information. Students may be most responsive to text message interventions when they provide concrete, actionable information (Castleman & Page, 2017). By defining the amount of time it is likely to take to complete a task and pointing students toward resources that will facilitate the task’s completion, an informational intervention may induce timely, successful completion.

Peer Support

There is a long-standing literature that contends that the support of peers contributes to the academic adjustment of college students (Astin, 1993; Deil-Amen, 2011; Tinto, 1993). Community and technical college-intending students are disproportionately first-generation postsecondary students and may have less access to information about the college transition and success from family and friends (Bailey et al., 2005). Tinto (1993) explains that when students drop out or fail academically, it is due to lack of belonging, or integration with the college community. Interacting with peers about academic matters is important to academic performance and persistence. Deil-Amen (2011) refers to these opportunities for peer support as socio-academic integrative moments. In a qualitative study of community college students, Deil-Amen (2011) finds that such moments not only induce feelings of attachment and belonging but also enhance students’ acquisition of knowledge, enabling them to make more effective academic and procedural choices for college success. Framing informational messages to encourage students to engage with their peers about academic success and logistical tasks may present an opportunity for meaningful socio-academic contact and bear fruit in the form of inducing action and encouraging student enrollment and persistence.

This study builds on the body of research on informational interventions by examining how behavioral framing of informational text messages affects students’ transition to publicly funded postsecondary education. This research will contribute to the literature on tuition-free college programs and informational interventions for college-intending youth, support the state’s attainment of ambitious postsecondary completion goals, and inform the work of school and community partners focused on college access and attainment.

Research Design and Sample

During the 2017–2018 academic year, I collaborated with tnAchieves to conduct a text-based informational campaign to determine whether their delivery of enrollment- and persistence-focused text messages could be optimized through strategic framing. The intervention was designed in March and April of 2017 and began that May. I preregistered my analyses with the editors of the AERA Open Registered Reports special issue in January 2018, after the intervention was underway but before the first round of outcomes data were made available in March 2018.

Data and Sample

The text message intervention focused on college-intending high school graduates with whom tnAchieves was working to facilitate postsecondary enrollment and success. Students resided in 84 of Tennessee’s 95 counties. The analytic sample includes prospective students who had, through high school graduation, maintained eligibility for the TN Promise scholarship and had expressed their intention to enroll in college by indicating an institution of enrollment on the state’s student aid portal. I also restricted the sample to individuals who consented to receive the text messaging intervention and for whom tnAchieves had a cell phone number on file. Across the state, the experimental sample includes roughly 18,400 college-intending high school graduates.

I use de-identified administrative data derived from tnAchieves’ records and state of Tennessee records reported to tnAchieves. These records include demographic and prior academic achievement data as well as ongoing scholarship eligibility information. In particular, I observe gender and race/ethnicity, high school grade point average, and ACT score. I also observe expected family contribution (EFC) as calculated by the U.S. Department of Education on completion of the FAFSA.

As of May 2017, there were just over 24,000 class of 2017 graduates who were eligible for TN Promise in the 84 counties served by tnAchieves at that time. However, roughly 5,700 students did not provide tnAchieves with cell phone numbers or permission to send informational text messages. These students constitute a nonrandom sample of individuals who did not receive treatment but who are part of the population to which I would optimally like to generalize the results.

In the interest of both describing the sample and considering the degree to which the sample is representative of the whole, Table 1 shows descriptive statistics for all eligible students and students in the experimental sample. There are no significant differences when the eligible students and experimental sample group means are compared using t tests. Fifty-two percent of the experimental sample students are female. The majority of students in the experiment are White (71.2%), and roughly one fifth are African American or Black (20.8%). The proportion of students who are White is higher among the TN Promise–eligible high school graduates and the experimental sample than among TN Promise applicants and the high school cohort overall (THEC, 2019). The plurality of students in the experimental sample are the first in their families to attend college (45.8%), which differs from the proportion of TN Promise applicants who identified as first generation (36.8%) and class of 2017 students who ultimately enrolled in college with Promise (40.6%). More than one third of students in the experimental sample have an EFC of 0 (35.3%). Just over two thirds of students (66.9%) intend to enroll in a community college.

Table 1

Descriptive Statistics for the Experimental Sample and all TN Promise–Eligible Class of 2017 High School Graduates

Variable	Experimental Sample, % (N = 18,400)	All TN Promise–Eligible HS Graduates, % (N = 24,080)
Female	52.1	52.0
Asian/Pacific Islander	1.1	1.1
Black/African American	20.8	19.8
Hispanic/Latino	4.5	4.8
White	71.2	72.1
First generation	45.8	46.5
EFC of zero	35.3	35.9
Intention to enroll in CC	66.9	66.7
Intention to enroll in TCAT	13.0	13.4
Intention to enroll in 4-year	16.5	16.3

Note. HS = high school; EFC = Expected Family Contribution; CC = community college; TCAT = Tennessee Colleges of Applied Technology. All data derived from measures obtained from tnAchieves. Students included in column 2 did not provide tnAchieves with a mobile phone number. Sample sizes rounded to the nearest 10. Notation of statistical significance refers to t-test comparisons between sample and eligibility group means.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

Measures

I estimate the effect of the three treatment arms on community service completion, initial enrollment, and persistence outcomes. I derive outcome measures from tnAchieves data detailing TN Promise scholarship eligibility loss date and eligibility loss reason for all students in the experimental sample. Students who do not lose eligibility at the criterion checkpoint are coded as successfully meeting that deadline (see Appendix A for eligibility checkpoints). The first outcome is completion of summer community service; students were required to submit 8 hours of community service by July 1, 2017. The next outcome of interest is Fall 2017 eligibility, which is dependent on both fall term postsecondary matriculation and enrollment in a TN Promise eligible program (i.e., 2-year degree program at an eligible public or private institution). Students who lose eligibility at this checkpoint may have done so either if they did not enroll or if they enrolled in a noneligible degree program. To approximate Fall 2017 enrollment, the next outcome of interest, I code students as having enrolled in the fall if they maintained Promise eligibility (by enrolling in an eligible program) or if the stated reason for their eligibility loss is enrollment in a noneligible program. Although this enrollment indicator is not National Student Clearinghouse verified, it is based on state financial aid disbursement and, thus, can be considered a reliable approximation of fall enrollment. The next eligibility checkpoint and outcome of interest is completion of Fall 2017 community service by December 5, 2017. Similar to the summer deadline, students are coded as successfully meeting this deadline if they submitted hours to retain their ongoing Promise scholarship eligibility.

The final two outcomes are Spring 2018 eligibility and Fall 2018 eligibility, which I have again approximated using Promise scholarship eligibility. For these two outcomes, students are coded as either remaining eligible by returning to a TN Promise eligible program for the spring term and fall term, or as losing eligibility due to attrition or enrollment in a different, noneligible program. I cannot see in the data whether students who lost eligibility for the fall semester due to noneligible program enrollment returned to their campuses for their second and third term, so these indicators are limited in their utility and any estimates must be interpreted with caution.

The explanatory variables of interest are the indicators for experimental intervention. The business-as-usual condition is the reference group against which students in the loss aversion, reduction of ambiguity, and peer support conditions are compared. I leverage student-level demographic, background, and academic information as covariates to increase the precision of the estimates in my statistical models. Student-level demographic and background information includes gender, race, EFC, and I use cumulative ACT scores as a measure of prior academic achievement.

Intervention Design

The randomized control trial began in May 2017. In partnership with tnAchieves, I implemented a text-based informational intervention with roughly 18,400 individuals from 452 public, private, and home-based high schools who were identified as eligible for the TN Promise at the time of May 2017 high school graduation. May 2017 scholarship eligibility depended on successful TN Promise application, FAFSA completion, and mentor meeting attendance before established checkpoints during the academic year.

tnAchieves was committed to ensuring that all eligible students receive important transition information via text message after they no longer have access to their high school counselor. Due to the organization’s concern about the efficacy of text campaigns for previous cohorts, our goal was to determine whether the informational intervention would be improved by strategically framing information to induce action.

All students who provided a cell phone number and consent for contact received text messages. Students were randomized at the individual-level to one of three treatment arms or the business-as-usual group, which received the same message content as prior cohorts. Messages covered a number of broad topics, including enrollment, admissions, financial aid, scholarship eligibility, and retention/success. Messages were sent on the 1st and the 15th of each month from May 2017 to December 2017. The business-as-usual messages contained the same text as messages sent by tnAchieves to the 2016 graduating cohort during the prior year. The treatment condition messages were framed to be consistent with the literature on loss aversion, reduction of ambiguity, and peer support. Appendix B provides the text message content for the four randomized conditions. The text-messaging service, SMS Magic, allows for two-way texting. tnAchieves advisors were able to monitor incoming responses and answer student questions directly.

Randomization and Baseline Equivalence

The 18,400 eligible students who consented to receive text messages and provided a cell phone number were randomized at the individual level to balanced groups. In Table 2, I present the mean values of baseline covariates by treatment arm. In no instance was I able to detect significant differences in mean values of baseline covariates between each treatment group and the business-as-usual group.

Table 2

Assessment of Balance in Covariates Across the Three Treatment Groups

Variable	Business-As-Usual, % (N = 4,600)	Loss Aversion, % (N = 4,600)	Implementation, % (N = 4,600)	Peer Support, % (N = 4,600)
Female	51.6	53.1	51.9	51.9
Asian/Pacific Islander	0.9	1.2	1.2	1.2
Black/African American	19.8	21.3	21.3	21.0
Hispanic/Latino	4.4	4.6	4.5	4.5
White	72.3	70.9	71.0	70.7
First generation	46.2	45.8	45.1	46.4
EFC of zero	33.9	35.8	35.8	35.6
Intention to enroll in CC	66.6	66.7	66.3	67.9
Intention to enroll in TCAT	13.4	13.4	13.1	12.2
Intention to enroll in 4-year	16.5	16.7	16.6	16.1
ACT score	18.71	18.63	18.71	18.68

Note. EFC = Expected Family Contribution; CC = community college; TCAT = Tennessee Colleges of Applied Technology. All values except ACT score are in percentage. Cells report group means. Sample sizes rounded to the nearest 10. Notation of statistical significance refers to comparisons between control and each of the individual treatment group means.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

Power Analysis

I conducted an a priori power analysis using PowerUp’s MDES Calculator for Individual Random Assignment (IRA) Designs for randomized controlled trials (Maynard & Dong, 2013). Assuming power of 0.8 and alpha of .05, the minimum detectable effect size (MDES) for the main model is 0.058, which aligns with empirically estimated effect sizes in previous evaluations of text message interventions. Thus, the total sample of 18,400 (4,600 by treatment arm) should be adequate for the main models of the study. To determine whether it is appropriate to run heterogeneity analyses by limiting the sample by subgroup (rather than adding interaction terms to the baseline model), I also ran post hoc power analyses for the relevant subgroups by gender, race, and intended postsecondary sector. The sample size for even the smallest subgroup among those to be tested, TCAT-intending students (N = 2,390), is sufficiently large to detect a MDES of 0.099 (Maynard & Dong, 2013). I proceed by assuming that with power of 0.8 and alpha of .05, the sample is sufficiently large to detect hypothesized effects for the main model and subgroups of interest.

Prespecified Analytic Strategy

To estimate the effect of treatment, I use a linear probability model, which estimates the intent-to-treat effect of each of the treatment arms relative to the business-as-usual condition. I also test the sensitivity of the results to a logistic regression modeling approach. The primary model is an intent-to-treat model in the following form:

{OUTCOME}_{i j} = β_{0} + β_{1} T X_{i j} + X_{γ} + ϵ_{i j}

(1)

where for student i in high school j, OUTCOME_ij is an indicator for the enrollment-related outcomes defined above. The $T X_{i j}$ term is a categorical indicator for the three experimental conditions to which students were randomly assigned, with the business-as-usual sample serving as the reference group against which I estimate the effect of treatment. $X_{γ}$ serves as a vector of student-level covariates. The coefficient of interest is $β_{1}$ , which represents the causal effect of a text message framing condition on the outcome of interest. I cluster standard errors by high school to account for potentially unobserved correlations in the error terms across students who attended the same high school.

The higher education research suggests that transition to postsecondary education presents particular obstacles for some student populations (Braxton, 2000; DesJardins et al., 2002). To assess the equitability of the intervention, I estimate heterogeneity of treatment by gender, race, and intended postsecondary sector, dimensions along which empirical research has shown there to be substantive differences in experience and success in postsecondary education. In my preregistered analytic plan, I proposed heterogeneous treatment effect models that interacted key characteristics with the treatment groups, as seen in Equation 2:

\begin{array}{l} {OUTCOME}_{i j} = β_{0} + β_{1} C C_{i j} + β_{2} T X_{i j} \\ + β_{3} T X_{i j} + C C_{i j} + β_{4} T X_{i j} \\ * T C A T_{i j} + β_{5} T X_{i j} * 4 Y R_{i j} \\ + X_{γ} + ϵ_{i j} \end{array}

(2)

In this model, $β_{3}$ represents the causal effect of being assigned to a text messaging intervention among students who plan to enroll at a community college, $β_{4}$ represents the effect of that assignment for college of applied technology–intending students, and $β_{5}$ represents the effect for would-be 4-year institution enrollees. I ran these interaction models as additional checks on the exploratory analysis described below; I report the interaction estimates in Appendix C (Tables C1 –C3).

Exploratory Analysis

The interaction models to test heterogeneity of effects for this multiarm intervention yield estimates with limited utility for policy analysis. For ease of interpretability of the heterogeneity of effects analysis, I also run the main model with the sample limited to the preregistered subgroups of students. I present this exploratory analysis as the primary heterogeneity analysis in the results.

Results

Descriptive Analysis

First, I examine the proportion of the sample that maintained eligibility for the scholarship at each of the eligibility checkpoints. As shown in Table 3, the percentage of individuals meeting the summer community service eligibility checkpoint in the experimental sample (78.6%) is marginally higher than the overall percentage in the cohort of prospective TN Promise students across the state (74.4%; THEC, 2017). However, there are not significant differences in scholarship eligibility maintenance between the business-as-usual group and each of the treatment groups.

Table 3

Percentage of Sample Maintaining TN Promise Eligibility by Treatment Group

Treatment	July 2017 Community Service, %	Fall 2017 Semester Eligibility, %	Fall 2017 Community Service, %	Spring 2018 Enrollment, %	Fall 2018 Semester Eligibility, %
Business-as-usual (N = 4,600)	78.8	69.3	57.8	46.6	31.1
Loss aversion (N = 4,600)	78.7	68.6	57.8	47.0	29.6
Reduction of ambiguity (N = 4,600)	78.9	68.4	57.8	47.4	30.2
Peer support (N = 4,600)	77.9	67.7	57.9	45.6	29.7
Total (N = 18,400)	78.6	68.5	57.5	46.9	30.1

Note. TN Promise = Tennessee Promise. Cells report group means. Sample sizes rounded to the nearest 10. Eligibility updated through December 2018. Eligibility status can be corrected or appealed. Notation of statistical significance refers to comparisons between control and treatment group means.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

Intervention Effect Estimation

In Table 4, I estimate the effect of the intervention on successful maintenance of TN Promise eligibility through six scholarship checkpoints, including three early outcomes on the path to college success: completion of Summer 2017 community service hours, Fall 2017 semester scholarship eligibility, and completion of Fall 2018 community service hours (columns 1–3). The treatment arms of the informational campaign appear not to have affected scholarship eligibility maintenance behaviors relative to the business-as-usual text messaging condition. Baseline intent-to-treat estimates regressing the outcome on treatment arm show no effect of the texting frames relative to the business-as-usual frame, loss aversion, and reduction of ambiguity, on summer community service completion (Table 4). When considering the estimated effect of the treatment arms on Summer Community Service completion, I find small, nonsignificant positive estimates for the loss aversion and reduction of ambiguity conditions and a small, nonsignificant negative estimate for the peer support condition, relative to the business-as-usual condition. Results are qualitatively similar when modeled via logistic regression (see Appendix C, Table C1).

Table 4

Overall Impact of Text Intervention on Community Service, Fall Scholarship Eligibility, and Persistence Outcomes

Treatment	July 2017 Community Service	Fall 2017 Semester Eligibility	Fall 2017 Enrollment	Fall 2017 Community Service	Spring 2018 Enrollment	Fall 2018 Semester Eligibility
Treatment	(1)	(2)	(3)	(4)	(5)	(6)
Loss aversion	0.011(0.012)	0.007(0.012)	−0.006(0.010)	0.013(0.014)	0.007(0.014)	−0.011(0.012)
Reduction of ambiguity	0.005(0.012)	−0.001(0.012)	−0.009(0.010)	0.002(0.014)	0.003(0.014)	−0.005(0.013)
Peer support	−0.007(0.012)	−0.013(0.012)	−0.005(0.010)	0.002(0.014)	−0.003(0.014)	−0.007(0.013)
Business-as-usual group rate	0.606	0.597	0.787	0.522	0.476	0.311
N	18,400	18,400	18,400	18,400	18,400	18,400
R ²	.033	.031	.017	.052	.059	.065

Note. Coefficients estimated from linear probability models. Approximated enrollment Fall 2017 estimates count individuals as enrolled if they began college with Tennessee Promise or lost scholarly eligibility due to enrollment in a noneligible degree program. Heteroskedasticity robust standard errors.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

To determine whether there are later-emerging effects of the intervention, which ran throughout the 2017–2018 academic year, I estimate the effect of the intervention on within-year scholarship maintenance, or students’ TN Promise eligibility in the spring semester, and between-year scholarship maintenance, or eligibility in the fall term of the second year (Table 4, columns 5 and 6). It does not appear that the treatment arms had an effect on student persistence relative to the business-as-usual condition. Looking at student outcomes by condition across all eligibility checkpoints, the Peer support condition has a fairly consistent negative, nonsignificant point estimate, while the loss aversion and reduction of ambiguity conditions generally have positive, nonsignificant point estimates.

Most of the estimates in Table 4 are based on measured TN Promise scholarship eligibility maintenance. However, students who enroll in postsecondary training programs that are not TN Promise–eligible, namely, baccalaureate programs, are not captured as fall semester enrollees because they lose scholarship eligibility. To better model the effect of the intervention on fall semester enrollment, I estimate whether there is an effect of the intervention on enrollment at any Tennessee postsecondary institution in the fall of 2017 (Table 4, column 3). The point estimates are negative relative to the business-as-usual condition, but not significant. I do not find an effect of any of the treatment arms on fall semester enrollment.

Heterogeneity Analysis

The main models may mask heterogeneity in the effect of the text message frames by aspects of participant identity or postsecondary plans. I tested for heterogeneity by gender, race, and intended postsecondary sector by estimating models in which the sample was limited to subgroups of interest; I ran interaction models as a secondary check on these estimates.¹ Comparing the upper and lower panels of Table 5 shows that men were negatively affected by the loss aversion condition on the magnitude of roughly 2 percentage points at time of fall 2017 eligibility and enrollment, fall 2017 community service submission, and fall 2018 eligibility. The treatment frames did not differentially affect women.

Table 5

Estimated Effects of Treatment Frames by Gender

Treatment	July 2017 Community Service	Fall 2017 Semester Eligibility	Fall 2017 Enrollment	Fall 2017 Community Service	Spring 2018 Enrollment	Fall 2018 Semester Eligibility
Men
Loss aversion	−0.013(0.013)	−0.029^*(0.014)	−0.024^{^}(0.014)	−0.025^{^}(0.015)	−0.023(0.015)	−0.023^{^}(0.014)
Reduction of ambiguity	0.001(0.013)	−0.011(0.014)	−0.008(0.014)	−0.013(0.015)	0.001(0.015)	−0.019(0.013)
Peer support	−0.009(0.013)	−0.017(0.014)	−0.017(0.014)	−0.013(0.015)	−0.013(0.015)	−0.012(0.014)
Business-as-usual group rate	0.765	0.681	0.701	0.504	0.453	0.288
N	8,800	8,800	8,800	8,800	8,800	8,800
R²	.002	.005	.004	.003	.004	.004
Women
Loss aversion	0.008(0.011)	0.011(0.013)	0.004(0.013)	0.015(0.014)	0.008(0.014)	−0.010(0.014)
Reduction of ambiguity	0.001(0.011)	−0.009(0.013)	−0.006(0.013)	0.002(0.014)	−0.006(0.014)	−0.002(0.014)
Peer support	−0.010(0.011)	−0.017(0.013)	−0.015(0.013)	−0.018(0.014)	−0.027^{^}(0.014)	−0.017(0.014)
Business-as-usual group rate	0.810	0.705	0.738	0.540	0.498	0.332
N	9,600	9,600	9,600	9,600	9,600	9,600
R²	.003	.005	.002	.005	.007	.002

Note. Coefficients estimated from linear probability models. Approximated enrollment Fall 2017 estimates count individuals as enrolled if they began college with Tennessee Promise or lost scholarly eligibility due to enrollment in a non-eligible degree program. Heteroskedasticity robust standard errors.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

I also estimate the effects of the text messaging frames on TN Promise scholarship eligibility by participant race or ethnicity. Table 6 shows that White participants were generally not differentially affected by the frames, with the exception of two marginally significant negative point estimates on the magnitude of 2.0 to 2.5 percentage points at the time of Fall 2017 community service and Spring 2018 enrollment. Black participants randomized to the peer support condition were less likely to complete July 2017 community service (−5.4 percentage points), maintain Fall 2017 semester eligibility (−6.1 percentage points), and enroll in Fall 2017 (−6.4 percentage points).

Table 6

Estimated Effects of Treatment Frames by Race

Treatment	July 2017 Community Service	Fall 2017 Semester Eligibility	Fall 2017 Enrollment	Fall 2017 Community Service	Spring 2018 Enrollment	Fall 2018 Semester Eligibility
White
Loss aversion	−0.002(0.010)	−0.010(0.011)	−0.010(0.011)	−0.012(0.013)	−0.013(0.013)	−0.013(0.012)
Reduction of ambiguity	0.005(0.010)	−0.012(0.011)	−0.010(0.011)	−0.011(0.013)	−0.005(0.013)	−0.009(0.012)
Peer support	−0.000(0.010)	−0.005(0.011)	−0.004(0.011)	−0.020^{^}(0.013)	−0.025^{^}(0.013)	−0.012(0.012)
Business-as-usual group rate	0.818	0.734	0.755	0.577	0.533	0.355
N	12,440	12,440	12,440	12,440	12,440	12,440
R²	.000	.001	.001	.002	0.004	0.001
Black
Loss aversion	−0.007(0.022)	−0.006(0.023)	−0.016(0.023)	0.023(0.023)	0.014(0.022)	−0.032^{^}(0.018)
Reduction of ambiguity	−0.007(0.022)	0.004(0.023)	0.001(0.023)	0.029(0.023)	0.018(0.022)	−0.004(0.018)
Peer support	−0.054^*(0.022)	−0.061^**(0.024)	−0.064^**(0.023)	−0.014(0.023)	−0.016(0.022)	−0.026(0.018)
Business-as-usual group rate	0.710	0.573	0.631	0.362	0.312	0.185
N	3,640	3,640	3,640	3,640	3,640	3,640
R²	.002	.002	.003	.001	0.001	0.001

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

When modeling heterogeneity by intended postsecondary sector, I examine whether the treatment arms differentially affect participants intending to attend community colleges, technical colleges, and baccalaureate institutions. As shown in Table 7, I find that TCAT-intending participants randomized to the peer support condition were less likely to complete July 2017 community service (−5.5 percentage points), maintain Fall 2017 semester eligibility (−7.0 percentage points), and enroll in Fall 2017 (−7.1 percentage points). The estimates do not suggest differential effects for community college–intending and baccalaureate institution–intending students.

Table 7

Estimated Effects of Treatment Frames by Intended Postsecondary Sector

Treatment	July 2017 Community Service	Fall 2017 Semester Eligibility	Fall 2017 Enrollment	Fall 2017 Community Service	Spring 2018 Enrollment	Fall 2018 Semester Eligibility
Community college–intending
Loss aversion	−0.002(0.009)	−0.010(0.010)	−0.010(0.011)	−0.007(0.013)	−0.007(0.013)	−0.011(0.012)
Reduction of ambiguity	0.009(0.009)	−0.005(0.011)	−0.004(0.011)	0.005(0.012)	0.008(0.013)	0.000(0.012)
Peer support	−0.000(0.009)	−0.007(0.011)	−0.007(0.011)	−0.001(0.012)	−0.015(0.013)	−0.009(0.012)
Business-as-usual group rate	0.846	0.785	0.785	0.606	0.555	0.362
N	12,290	12,290	12,290	12,290	12,290	12,290
R²	.002	.001	.001	.002	.003	.001
TCAT-intending
Loss aversion	0.015(0.024)	0.026(0.026)	0.025(0.026)	0.011(0.029)	0.002(0.028)	−0.043^{^}(0.025)
Reduction of ambiguity	0.003(0.025)	−0.005(0.027)	−0.007(0.027)	−0.021(0.029)	−0.012(0.029)	−0.047^{^}(0.025)
Peer support	−0.055^**(0.026)	−0.070^**(0.028)	−0.071^**(0.028)	−0.037(0.029)	−0.042(0.029)	−0.039(0.026)
Business-as-usual group rate	0.759	0.689	0.690	0.501	0.455	0.279
N	2,390	2,390	2,390	2,390	2,390	2,390
R²	.004	.006	.006	.001	.001	.002
4-Year intending
Loss aversion	−0.030(0.025)	−0.034(0.026)	−0.042(0.026)	−0.006(0.024)	−0.014(0.023)	−0.019(0.020)
Reduction of ambiguity	−0.029(0.025)	−0.009(0.026)	−0.015(0.026)	−0.013(0.024)	−0.022(0.023)	−0.011(0.020)
Peer support	−0.021(0.025)	−0.022(0.026)	−0.022(0.026)	−0.024(0.024)	−0.035(0.023)	−0.027(0.020)
Business-as-usual group rate	0.655	0.472	0.549	0.312	0.276	0.200
N	3,030	3,030	3,030	3,030	3,030	3,030
R²	.001	.001	.001	.001	.001	.001

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

Discussion

In spite of lowered financial barriers to entry, many college-intending Tennesseans who are eligible for the TN Promise scholarship at the time of high school graduation do not matriculate for the fall semester. In an effort to mitigate summer melt, I implemented a randomized control trial in partnership with nonprofit tnAchieves to test whether the strategic framing of a text message intervention affects the maintenance of scholarship eligibility. Estimates show no main effect of any of the treatment frames on student transition, first-semester, or persistence outcomes. I estimated differential effects of strategic informational framing on students by gender, race, and intended postsecondary sectors. Men in the loss aversion condition were less likely to maintain Fall 2017 eligibility, enroll in Fall 2017, complete Fall 2017 community service, and maintain Fall 2018 eligibility. This finding is consistent with the tendency for men to be less risk averse than women (e.g., Rau, 2014).

Black participants and TCAT-intending participants who receive peer support framing were less likely to complete summer service, maintain fall semester eligibility, or enroll in the fall semester. For Black participants, the negative effect of the peer support frame runs counter to the prevailing evidence in higher education that social engagement and sense of belonging yield better collegiate outcomes (Kuh et al., 2011; Strayhorn, 2018). Community college students are less likely to have expectations of college social engagement (Deil-Amen, 2011); however, there is no theoretical or empirical work to suggest that peer support framing would yield negative outcomes.

Indeed, the importance of peer socio-academic integration or sense of belonging has long been a backbone of higher education persistence research (Hurtado & Carter, 1997; Tinto, 1993). However, in the context of this intervention, messages that evoke support for peers had a negative effect relative to other frames. The support of professionals or those with college experience may be more important than the support of peers at the time of enrollment, with peer support gaining importance for persistence and attainment.

Overall, the results of this experiment do not suggest that strategic framing of informational messages is appreciably more effective than providing informational text messages at all. This is consistent with the mixed evidence in studies that have compared the relative efficacy of behavioral frames in text messaging interventions, as well as the null effects found in other studies that have taken text messaging interventions to state and national scale (e.g., Bird et al., 2019; Gurantz et al., 2019; Hyman, 2020). The mixed results in the literature raise questions regarding the way in which informational interventions are personalized and by whom they are delivered. The differential framing in this study may not have been sufficiently fine-tuned or targeted to induce greater student action toward scholarship eligibility maintenance. Different forms of personalization, for example, based on dimensions of student identity or personality, might yield better targeting. It is possible that consistent messaging in one frame is not an advantage, particularly if that frame is poorly aligned with a particular student’s orientation or outlook. For example, an anxious student randomized to the loss aversion treatment arm would receive a string of messages that all emphasize the potential to lose their scholarship. Rather than inducing action, biweekly receipt of such messages could incite panic. Alternatively, personalization of delivery at particular times or in particular locations may better align reminders with the recipient’s availability or mind-set. There are many options to tailor an informational messaging intervention rather than using a blunt, universal instrument.

Familiarity with and expectations for the text message deliverer may also play a role in the efficacy of text messaging interventions. In qualitative data collected for the broader study of which this experiment was a part, TN Promise focus group participants expressed reservations about receiving frequent text messages from parties with whom they have little familiarity and virtually no in-person contact. Participants conveyed a disconnect between their expectations for modes of professional contact and receipt of text messages from advising organizations. Additionally, widespread use of such interventions given prior evidence of text message campaign efficacy may contribute to feelings of text intervention oversaturation and feed discomfort with texting as a medium for what recipients perceive as impersonal reminders from relatively unknown entities. Expectations for the deliverer and text messaging as a mode of contact may have implications for large-scale interventions overall or for the perception of certain behavioral frames in particular.

At present, it is unclear whether text message reminders are beneficial for “nudging” college-intending students toward enrollment and, ultimately, degree attainment (e.g., discussion in Castleman & Page, 2019; Greene, 2019). Inconsistent average results between studies and heterogeneity of treatment estimates in this study and others (e.g., Phillips & Reber, 2018) suggest that text messaging may be differentially beneficial between populations and contexts. With the established popularity of texting interventions to mitigate summer melt and support students during the academic year, this study sought to contribute evidence as to whether strategic framing of such messages could serve as a budget-neutral, scalable improvement for organizations and institutions that already implement text messaging campaigns. The null main effects and small, positive subgroup effects do not yield definitive, actionable evidence, but contribute to the picture of the potential utility of strategic framing and personalization in texting interventions.

There are a number of factors that limit the internal and external validity of this experiment. The comparative treatment design reduces the size of each treatment arm and increases the likelihood of type II error. However, power analyses do not raise cause for concern, identifying an MDES of 0.058. The multiple treatment arms and repeated measures also raise the possibility of issues related to multiplicity. To address multiple hypothesis testing, I make multiplicity adjustments as outlined by Romano and Wolf (2005; Clarke, 2016). With adjustments, the estimated significant effects for Black and TCAT-intending students become marginally significant, with p values between .05 and .1. The marginal significance after multiplicity adjustments necessitates caution in the interpretation of the subgroup effect estimates. Rather than being interpreted independently in their magnitude and significance, the estimates presented in this article should be considered as part of a nascent literature on the potential for differential framing of informational interventions.

Generalizability is limited by the study context and sample. Models of college choice and persistence consistently identify financial burden and environmental context as important factors in student postsecondary transition and success. A study of the enrollment and first semester success of college aspirants under Tennessee’s tuition-free community and technical college scholarship program is limited to students whose choices are uniquely informed by TN Promise’s financial and social supports. Moreover, tnAchieves was the state’s partner for TN Promise support services in 84 of 95 counties; the effect of this intervention cannot be generalized to students in the remaining 11 counties.

Another limitation of the intervention is lack of knowledge of whether students read the messages. The text messaging provider reported that more than 95% of text messages were delivered to the mobile numbers provided; however, it does not track whether or not students accessed the information. I estimate the intent-to-treat effect of the intervention rather than estimating the effect of treatment-on-the-treated, which is unknowable given the provider’s data collection.

The overall intervention may not have been maximally leveraged by recipients. Data collected in on-campus focus groups during the spring semester revealed that many students were unaware that the text messaging platform was capable of two-way communication. Students should be informed and reminded of the full range of functions of the text messaging service to ensure that they can fully leverage opportunities for professional support of college enrollment and scholarship eligibility maintenance.

These limitations are balanced by the contributions of this study. Informational text messaging has shown promise not only in its estimated impacts but also in its accessibility to target individuals (given the ubiquity of cell phones and popularity of SMS communication among teens and young adults), affordability to implement, and ease of scale. This experiment examines a way in which a widely used behavioral intervention may be fine-tuned by leveraging potential mechanisms for choice architecture. In so doing, it builds on a budding literature that considers the relative efficacy of informational framing of informational interventions for prospective postsecondary students. In particular, this study builds on the work of Bird et al. (2019) and Bergman et al. (2019). This study is unique in the timing of the intervention, testing the relative efficacy of frames during the summer between high school and college; the intended sectors of enrollment of the sample, community and technical college; and the tuition-free college context. This study is the first to consider the effect of a technology-based informational intervention scaled to a statewide tuition-free college program. The null effects suggest that the provision of informational text messaging, which is supported by empirical evidence, may not be improved by the strategic behavioral frames tested in this study. The null effects in the main model are consistent with Bergman et al.’s (2019) findings that differential messaging regarding tax benefits did not affect college enrollment and Bird et al.’s (2019) estimates of null effects in statewide and national FAFSA reminder campaigns. Future work testing framing and personalization will be valuable to building the base of evidence and, ultimately, determine how to improve the utility of informational interventions for matriculation support.

Conclusion

This study contributes to a growing body of literature about informational campaigns for prospective college students, generally, and more specifically, the relative efficacy of varied framing of information. This study poses questions about mechanisms that contribute to acting on enrollment intentions in the unique context of a tuition-free college environment. Informational text messaging is being implemented to achieve policy goals in a variety of contexts and its mechanisms and the potential to optimize delivery warrant additional exploration.

Low-touch text messaging interventions may not be optimizable through strategic behavioral framing, but informational messaging can be personalized for recipients. The estimated differential effects of messaging frames for individuals along social dimensions joins other evidence suggesting the potential advantages of targeting messages to students. This text messaging experiment does not provide sufficient evidence on heterogeneity to make particular recommendations for the tailoring of informational interventions, but it suggests that future research in this area may be productive.

While a clear main effect did not emerge in this randomized control trial, there is some evidence that groups of prospective and current students may react differently to text message frames. This study contributes to an emerging body of literature that endeavors to determine the best content, mode, and messenger for informational interventions during the college transition. The continued efforts of practitioners and researchers may allow for optimization of informational campaigns for student success.

Footnotes

Appendix A

Tennessee (TN) Promise Eligibility Checkpoints for the Class of 2017

Deadline	Action
November 1, 2016	Complete TN Promise application
Fall 2016	Attend first team meeting hosted by tnAchieves
January 17, 2017	File the 2018–2019 Free Application for Federal Student Aid
Spring 2017	Apply to an eligible TN Promise institution
Spring 2017	Attend mandatory team meeting hosted by tnAchieves
July 1, 2017	Complete and submit 8 hours of community service
August 1, 2017	Complete financial aid verification, if selected
Summer 2017	Ensure that chosen college is listed on TN Student Assistance Corporation Portal
August 2017	Enroll as a full-time student in a TN Promise eligible program (minimum 12 credit hours or full-time at technical college)
Fall 2017	Attend third team meeting hosted by tnAchieves
December 1, 2017	Complete and submit 8 hours of community service

Appendix B

Appendix C

Table C3

Impact of Treatment Frames by Condition and Intended Postsecondary Sector

Treatment	Summer Community Service	Fall Semester Eligibility	Fall Community Service	Spring Enrollment	Second-Year Enrollment
Loss aversion, 4-year	−0.03(0.025)	−0.034(0.025)	−0.006(0.024)	−0.014(0.023)	−0.019(0.020)
Loss aversion, CC or TCAT	0.177^***(0.018)	0.293^***(0.019)	0.273^***(0.019)	0.256^***(0.018)	0.131^***(0.016)
Reduction of ambiguity, 4-year	−0.029(0.025)	−0.009(0.026)	−0.013(0.024)	−0.022(0.023)	−0.011(0.020)
Reduction of ambiguity, CC or TCAT	0.185^***(0.018)	0.292^***(0.019)	0.278^***(0.019)	0.268^***(0.018)	0.141^***(0.017)
Peer support, 4-year	−0.021(0.025)	−0.022(0.026)	−0.024(0.024)	−0.035(0.023)	−0.027(0.020)
Peer support, CC or TCAT	0.169^***(0.018)	0.281^***(0.019)	0.263^***(0.019)	0.244^***(0.018)	0.136^***(0.016)
CC- or TCAT-intending participants	0.176^***(0.018)	0.296^***(0.019)	0.277^***(0.019)	0.262^***(0.018)	0.148^***(0.017)
Business-as-usual, 4-year intending	0.655	0.472	0.312	0.276	0.136
N	17,710	17,710	17,710	17,710	17,710
R ²	.035	.065	.046	.043	.016
F test	0.001	0.001	0.001	0.001	0.001

Note. Coefficients estimated from linear probability models. Heteroskedasticity robust standard errors. Sample sizes rounded to the nearest 10.

p < .10. ^*p < .05. ^**p < .01. ^***p < .001.

ORCID iD

Jenna W. Kramer

1.

Results are qualitatively similar when I interact treatment variables with these factors, as prespecified in January 2018. I report these estimates in Appendix C (Tables C2–).

Author

JENNA W. KRAMER is an associate policy research at the RAND Corporation. Her research focuses on institutional and governmental interventions that facilitate college transition and success.

References

Arnold

Fleming

DeAnda

Castleman

B. L.

Warman

K. L.

(2009). The summer flood: The invisible gap among low-income students. Thought & Action, 25, 23–34. http://whp-template.nea.org/assets/docs/HE/TA09SummerFlood.pdf

Astin

A. W.

(1993). What matters in college? Four critical years revisited (Vol. 1). Jossey-Bass.

Avery

Howell

J. S.

Page

L. C.

(2014). A review of the role of college counseling, coaching, and mentoring on students’ postsecondary outcomes (Research brief). College Board.

Bailey

Jenkins

Leinbach

(2005, January). Community college low-income and minority student completion study: Descriptive statistics from the 1992 High School Cohort. Community College Research Center. https://ccrc.tc.columbia.edu/media/k2/attachments/low-income-minority-completion-study.pdf

Bergman

Denning

J. T.

Manoli

(2019). Is information enough? The effect of information about education tax benefits on student outcomes. Journal of Policy Analysis and Management, 38(3), 706–731. https://doi.org/10.1002/pam.22131

Beshears

Choi

J. J.

Laibson

Madrian

B. C.

(2009). The importance of default options for retirement saving outcomes: Evidence from the United States. In Brown

J. R.

Liebman

J. B.

Wise

D. A.

(Eds.), Social security in a changing environment (pp. 167–195). University of Chicago Press. https://doi.org/10.7208/chicago/9780226076508.003.0006

Bird

K. A.

Castleman

B. L.

Denning

J. T.

Goodman

Lamberton

Rosinger

K. O.

(2019). Nudging at scale: Experimental evidence from FAFSA completion campaigns (NBER Working Paper No. 26158). https://doi.org/10.3386/w26158

Bos

J. M.

Berman

Kane

T. J.

Tseng

F. M.

(2012, November 8–10). The impacts of SOURCE: A program to support college enrollment through near-peer, low-cost student advising [Conference session]. Association of Public Policy Analysis and Management Annual Conference, Baltimore, MD, United States.

Braxton

J. M.

(Ed.). (2000). Reworking the student departure puzzle. Vanderbilt University Press.

10.

Carrell

S. E.

Sacerdote

(2013). Late interventions matter too: The case of college coaching New Hampshire (NBER Working Paper No. 19031). National Bureau of Economic Research. https://www.nber.org/papers/w19031.pdf

11.

Carruthers

C. K.

Fox

W. F.

(2016). Aid for all: College coaching, financial aid, and post-secondary persistence in Tennessee. Economics of Education Review, 51, 97–112. https://doi.org/10.1016/j.econedurev.2015.06.001

12.

Castleman

B. L.

(2015). The 160-character solution: How text messaging and other behavioral strategies can improve education. Johns Hopkins University Press.

13.

Castleman

B. L.

Page

L. C.

(2014). Summer melt: Supporting low-income students through the transition to college. Harvard Education Press.

14.

Castleman

B. L.

Page

L. C.

(2015). Summer nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates? Journal of Economic Behavior & Organization, 115, 144–160. https://doi.org/10.1016/j.jebo.2014.12.008

15.

Castleman

B. L.

Page

L. C.

(2016). Freshman year financial aid nudges: An experiment to increase FAFSA renewal and college persistence. Journal of Human Resources, 51(2), 389–415. https://doi.org/10.3368/jhr.51.2.0614-6458R

16.

Castleman

B. L.

Page

L. C.

(2017). Parental influences on postsecondary decision making: Evidence from a text messaging experiment. Educational Evaluation and Policy Analysis, 39(2), 361–377. https://doi.org/10.3102/0162373716687393

17.

Castleman

B. L.

Page

L. C.

(2019). A response to “Texting nudges harm degree completion.” EducationNext. https://www.educationnext.org/response-texting-nudges-harm-degree-completion/

18.

Ceja

(2013, July 12). Summer melt. Homeroom. https://blog.ed.gov/2013/07/summer-melt/

19.

Clarke

(2016). RWOLF: Stata module to calculate Romano-Wolf stepdown p-values for multiple hypothesis testing (Statistical Software Components S458276). Boston College Department of Economics. https://ideas.repec.org/c/boc/bocode/s458276.html

20.

Deil-Amen

(2011). Socio-academic integrative moments: Rethinking academic and social integration among two-year college students in career-related programs. Journal of Higher Education, 82(1), 54–91. https://doi.org/10.1353/jhe.2011.0006

21.

Deming

D. J.

Walters

C. R.

(2017). The impact of price caps and spending cuts on US postsecondary attainment (NBER Working Paper No. w23736). National Bureau of Economic Research. https://doi.org/10.1353/jhe.2002.0042

22.

DesJardins

S. L.

Ahlburg

D. A.

McCall

B. P.

(2002). A temporal investigation of factors related to timely degree completion. Journal of Higher Education, 73(5), 555–581. https://doi.org/10.1353/jhe.2002.0042

23.

Greene

J. P.

(2019). Texting nudges harm degree completion. EducationNext. https://www.educationnext.org/texting-nudges-harm-degree-completion/

24.

Gurantz

Howell

Hurwitz

Larson

Pender

White

(2019, May). Realizing your college potential? Impacts of College Board’s RYCP campaign on postsecondary enrollment (EdWorkingPaper No. 19-40). https://doi.org/10.26300/nqn3-sp29

25.

Hastings

J. S.

Weinstein

J. M.

(2008). Information, school choice, and academic achievement: Evidence from two experiments. Quarterly Journal of Economics, 123(4), 1373–1414. https://doi.org/10.1162/qjec.2008.123.4.1373

26.

Hurtado

Carter

D. F.

(1997). Effects of college transition and perceptions of the campus racial climate on Latino college students’ sense of belonging. Sociology of Education, 70(4), 324–345. https://doi.org/10.2307/2673270

27.

Hyman

(2020). Can light-touch college-going interventions make a difference? Evidence from a statewide experiment in Michigan. Journal of Policy Analysis and Management, 39(1), 159–190. https://doi.org/10.1002/pam.22155

28.

Kahneman

Tversky

(1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263–292. https://doi.org/10.2307/1914185

29.

Karlan

McConnell

Mullainathan

Zinman

(2016). Getting to the top of mind: How reminders increase saving. Management Science, 62(12), 3393–3411. https://doi.org/10.1287/mnsc.2015.2296

30.

Kraft

M. A.

Dougherty

S. M.

(2013). The effect of teacher–family communication on student engagement: Evidence from a randomized field experiment. Journal of Research on Educational Effectiveness, 6(3), 199–222. https://doi.org/10.1080/19345747.2012.743636

31.

Kuh

G. D.

Kinzie

Schuh

J. H.

Whitt

E. J.

(2011). Student success in college: Creating conditions that matter. John Wiley.

32.

Maynard

R. A.

Dong

(2013). PowerUp! A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24–67. https://doi.org/10.1080/19345747.2012.673143

33.

Milkman

K. L.

Beshears

Choi

J. J.

Laibson

Madrian

B. C.

(2012). Following through on good intentions: The power of planning prompts (NBER Working Paper No. 17995). National Bureau of Economic Research. https://doi.org/10.3386/w17995

34.

Murphy

Scott-Clayton

Wyness

(2017). The end of free college in England: Implications for quality, enrolments, and equity (NBER Working Paper No. w23888). National Bureau of Economic Research. https://doi.org/10.3386/w23888

35.

Nickerson

D. W.

Rogers

(2010). Do you have a voting plan? Implementation intentions, voter turnout, and organic plan making. Psychological Science, 21(2), 194–199. https://doi.org/10.1177/0956797609359326

36.

O’Donoghue

Rabin

(1999). Doing it now or later. American Economic Review, 89(1), 103–124. https://doi.org/10.1257/aer.89.1.103

37.

Perna

L. W.

Leigh

E. W.

(2017). Understanding the promise: A typology of state and local college promise programs. Educational Researcher, 47(3), 155–180. https://doi.org/10.3102/0013189X17742653

38.

Phillips

Reber

(2018, May 11). When “low touch” is not enough: Evidence from a random assignment college access field experiment (UCLA CCPR Population Working Paper PWP-CCPR-2018-008). California Center for Population Research, University of California, Los Angeles. http://papers.ccpr.ucla.edu/index.php/pwp/article/view/1213/596

39.

Quinton

(2019, March 5). “Free college” is increasingly popular—and complicated for states. Pew Trusts. https://www.pewtrusts.org/en/research-and-analysis/blogs/stateline/2019/03/05/free-college-is-increasingly-popular-and-complicated-for-states

40.

Rau

H. A.

(2014). The disposition effect and loss aversion: Do gender differences matter? Economics Letters, 123(1), 33–36. https://doi.org/10.1016/j.econlet.2014.01.020

41.

Romano

J. P.

Wolf

(2005). Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association, 100(469), 94–108.

42.

Smith

(2010). The theory of moral sentiments. Penguin Classics. (Original work published 1759)

43.

Stockwell

M. S.

Kharbanda

E. O.

Martinez

R. A.

Vargas

C. Y.

Vawdrey

D. K.

Camargo

(2012). Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: A randomized controlled trial. JAMA Journal of the American Medical Association, 307(16), 1702–1708. https://doi.org/10.1001/jama.2012.502

44.

Strayhorn

T. L.

(2018). College students’ sense of belonging: A key to educational success for all students. Routledge.

45.

Tennessee Higher Education Commission. (2017). Tennessee Promise: Annual report. https://www.tn.gov/content/dam/tn/thec/bureau/research/promise/2017_TN_Promise_Report.pdf

46.

Tennessee Higher Education Commission. (2019). Tennessee Promise: Annual report. https://www.tn.gov/content/dam/tn/thec/bureau/research/promise/TN%20Promise%20Report%202019%20Final%20Revised%2006192019.pdf

47.

Thaler

R. H.

Sunstein

C. R.

(2003). Libertarian paternalism. American Economic Review, 93(2), 175–179. https://doi.org/10.1257/000282803321947001

48.

Thaler

R. H.

Sunstein

C. R.

(2008), Nudge: Improving decisions about health, wealth and happiness. Yale University Press.

49.

Tinto

(1993). Building community. Liberal Education, 79(4), 16–21. https://doi.org/10.1177/105382599301600204

Experimental Evidence on the Effects (or Lack Thereof) of Informational Framing During the College Transition

Abstract

Keywords

Promise Programs and the Tennessee Context

Prior Research on Mitigating Summer Melt

Behavioral Economics and Text Messaging as an Informational Intervention

Loss Aversion

Reduction of Ambiguity

Peer Support

Research Design and Sample

Data and Sample

Measures

Intervention Design

Randomization and Baseline Equivalence

Power Analysis

Prespecified Analytic Strategy

Exploratory Analysis

Results

Descriptive Analysis

Intervention Effect Estimation

Heterogeneity Analysis

Discussion

Conclusion

Footnotes

Appendix A

Appendix B

Appendix C

ORCID iD

1.

Author

References