Sage Journals: Discover world-class research

Abstract

Mental health disorders are prevalent among college students and increasing in frequency and severity. However, there is a significant gap between those who need treatment and those who engage in treatment. Given the documented efficacy of financial incentives for promoting health behavior change and engagement in treatment, financial incentives may help, along with nonfinancial behavioral incentives such as motivational messaging, gamification, and loss aversion techniques. We compared brief (28-day) use of two versions of a behavioral economics-inspired digital mental health app, NeuroFlow: (1) the full app including financial incentives and nonfinancial behavioral incentives (treatment group) and (2) a version of the app with nonfinancial behavioral incentives only (control group). In our intent-to-treat analyses, in order to examine the primary outcome of app engagement, a one-way analysis of variance (ANOVA) (treatment vs. control) was conducted, and to examine the secondary outcomes (depression, anxiety, emotion dysregulation, and wellbeing), a two-way repeated measures ANOVAs (treatment vs. control × baseline vs. post-trial) were conducted. We found that there were no differences between treatment groups on app engagement or the change in the mental health/wellness outcome measures. There was a main effect of timepoint on symptoms of anxiety and emotion dysregulation, such that there were significantly lower self-reported symptoms at post-trial relative to baseline. Our results suggest that financial incentives in digital mental health apps over and above nonfinancial behavioral incentives do not have an impact on app engagement or mental health/wellness outcomes.

Keywords

Digital mental health behavioral economics financial incentives anxiety emotion dysregulation college students

Introduction

Mental health disorders are prevalent among college students and increasing in frequency and severity.^1,2 Approximately one out of every four or five young people will experience a mental illness in any given year.³ Furthermore, the onset of mental health disorders during college, a challenging transition, has significant impacts on academic success, relationships with others, productivity, and substance use.^4,8 Anxiety disorders are the most prevalent psychiatric disorder in college students, with approximately 12% of college students diagnosed with some type of anxiety disorder and, on a larger scope, impacts an estimate of 48 million adults in the United States.^9,11 Depression is also prevalent in college students, with one large-scale study finding that 25% of college students had experienced depression in the past year^12,13 and another showing that approximately 80% of college students they surveyed experienced depression since starting college.¹⁰ The onset and maintenance of depression and anxiety has been found to be positively predicted by difficulties in emotion regulation, such that individuals who experience difficulties in downregulating negative emotions and upregulating positive emotions are more likely to have these mental health concerns.^14,16 Therefore, examination and support of emotion regulation and affective experience is critical in populations with relatively high rates of anxiety and depression, such as college students.

Concern over college students’ mental health and affective experience has been further exacerbated due to the COVID-19 pandemic. One study suggests that the pandemic has relayed a modest but persistent negative impact on the mood and the wellness behaviors of university freshman.¹⁷ Another study of university students during the pandemic found a large percentage of respondents showed moderate-to-severe depression, anxiety, and/or suicidal thoughts.¹⁸ Research illustrates that, ordinarily, 10% of adults in the US report symptoms of depression or anxiety (a nonzero score on the depression and anxiety screeners), but from 2019 to 2021, this figure was 40%, representing a 30% uptick in adults reporting symptoms of these disorders.¹⁹

Despite the availability of mental health services and treatment, relatively few people use these resources.²⁰ Large-scale epidemiological surveys demonstrate low rates of mental health service use relative to the amount of mental health services available.²¹ Specifically, one study reported that only approximately 35% of people who were diagnosed with common mental health problems received aid and another found that approximately 80% of those who suffer from a mental illness or who know someone who suffers from a mental illness cited cost concerns as a barrier to seeking treatment.^22,23 Other barriers to mental health treatment include a lack of awareness of mental illness symptoms, time constraints, stigma and language barriers.^24,27

The rise of digital mental health

As mental health awareness and intervention have gained more traction, mobile apps have emerged to facilitate mental health care for those who are unable to afford or access alternative mental health care methods.²⁸ This has been especially critical during the pandemic as many families are experiencing economic difficulties such as unemployment and quarantine requirements, making it even more difficult for individuals to afford and access in-person professional mental health services.²⁹ As a combination of these factors, there has been increased momentum for the use of/need for digital mental health and wellness apps in recent years.

Research has also illustrated the effectiveness of many digital mental health online programs delivered from personal computers and laptops.³⁰ In line with this, more traction for smartphone and tablet digital mental health apps centered around mental help has emerged.³¹ Randomized controlled trials on digital mental health apps demonstrate efficacy in their ability to support general mental wellness.^30,32,33 Thus, evidence indicates that the benefits of digital mental health apps are realized in practice. However, while digital mental health apps show promise, over 95% of them have not actually been studied and there is a lack of research in this field regarding efficacy, especially relative to studies on digital mental health online programs.^30,34 Additionally, the majority of the available research examined user adoption, which focuses on optimizing user-app experience, leaving a gap in research regarding user engagement and interest postinstallation in relation to mental health promoting apps.³⁵ As a whole, there is little evidence that digital mental health apps can effectively engage clients.³⁶ Thus, despite some promising evidence for aiding in the reduction of mental health symptoms and user adoption, not much is known regarding sustained user engagement and retention postinstallation.

The potential of financial and nonfinancial behavioral incentives: Increasing user engagement

To address the identified gap between those who need treatment and those who engage in treatment, behavioral economics principles related to motivating adult behavior change have been applied to healthcare in the form of incentives. Behavioral economic principles seek to understand behavior and decision making through a dual lens of behavioral science integrated with economic principles. In relation to digital mental health, research suggests that behavioral economics incentives can increase engagement in healthcare— including engagement with a virtual interfaces such as apps designed to help increase positive health behaviors.^37,39 Specifically, one nonfinancial incentive is gamification, which refers to attempts to redesign behavior through implementing aspects of game design to daily activities, such as fitness and nutrition.^40,41 For example, gamification includes the use of point accrual as an incentive for the user to complete a task and studies suggest its promise in sustaining user engagement in mobile health applications.^42,45 Furthermore, gamification research illustrates the effectiveness of gamification as a technique to increase health-related, suggesting the potential benefits of incorporating gamification as a behavioral economics incentive.⁴⁶

Financial incentives are recognized tools of behavior economics capable of influencing behavior as they present a means to inspire positive changes in behavior.^47,48 In the healthcare sphere, research indicates that financial incentives are effective in promoting health behavior change and aid in decreasing healthcare costs in the long run.⁴⁹ Regarding mental health treatments, a recent meta-analysis showed that financial incentives increase treatment engagement with medium to large effect sizes.⁵⁰ For example, one study found that financial incentives improved daily antidepressant medication adherence compared with a control group during the critical first 6 weeks of treatment.⁵¹ However, there is little research regarding the implementation of behavioral incentives in digital mental health and wellness apps and their respective impact on short and long-term user engagement.⁵²

Current study

The current study examined the additive effect of financial incentives over and above other behavioral economics incentives in the digital mental health app, NeuroFlow. The NeuroFlow app is a digital mental health smartphone app that uses behavioral economic principles to increase motivation and engagement through nonfinancial behavioral economics incentives (motivational messages, gamification, and loss aversion techniques) and financial incentives (see Methods section for more details on NeuroFlow). We conducted a pilot two-arm randomized controlled trial, comparing two versions of the app: (1) the full NeuroFlow app with nonfinancial behavioral economics incentives and financial incentives (treatment group) and (2) a version of the NeuroFlow app with nonfinancial behavioral economics incentives only (control group). We examined whether the app version (i.e. treatment group) influenced app engagement (primary outcome) and mental health and wellness, including anxiety symptoms, depressive symptoms, and emotion regulation difficulties which are all directly targeted in the app, in addition to wellbeing (secondary outcomes). We hypothesized that financial combined with nonfinancial behavioral incentives would lead to increased app engagement relative to nonfinancial behavioral incentives only (hypothesis 1) and that financial combined with nonfinancial behavioral incentives would lead to more of a positive effect on mental health and wellness relative to nonfinancial incentives (hypothesis 2). We also explored whether there was any change in secondary outcomes from before to after the trial period of app use across treatment groups. We hypothesized improvements in mental health and wellness from baseline to post-trial (hypothesis 3).

Methods

Participants

Recruitment

Recruitment for this study took place via SONA systems, an online system used by the University of Pennsylvania to manage and schedule research projects within the university. Students enrolled in select courses at the University of Pennsylvania can participate in research projects and receive course credit in exchange for their participation. In addition, emails were sent out to University of Pennsylvania professors who taught classes registered in the SONA system in order to recruit more participants.

Enrollment

Eligible participants were college students at the University of Pennsylvania who had access to a smartphone and agreed to download and use the NeuroFlow app. See Figure 1 for a flow diagram that depicts the phases of parallel randomized trial of the study following the CONSORT flow diagram template. Please refer to Supplemental Figure 1 for a flow chart of the weekly user loss.

Figure 1.

CONSORT flow diagram of the parallel randomized controlled trial. Note: Duplicate = two accounts were created for the same participant. Drop Out = participant declared that they no longer wished to participate in this study pre allocation. Discontinued Intervention = participant declared that they no longer wished to participate in this study post allocation.

Demographics

All participants were between the ages of 18 and 29. As shown in Table 1, the treatment groups were matched on all participant demographic variables (age, gender, college grade level, household income, race, and non-Hispanic/Hispanic distribution) indicating that the randomization to the treatment versus control groups was successful.

Table 1.

Participant characteristics.

Characteristics	Control	Treatment	Comparison statistics
Characteristics	M/N (SD/%)	M/N (SD/%)	Comparison statistics
Age
Years	19.4 (1.1)	19.7 (1.8)	t(81) = −0.913, p = 0.36
Biological sex
Female	29 (69.0%)	23 (53.5%)	X2 (1, N = 83) = 1.49, p = 0.22
Male	13 (31%)	18 (41.9%)
Missing	0	2 (4.7%)
Race
Asian	15 (35.7%)	15 (34.9%)	X2 (4, N = 83) = 5.39, p = 0.25
White	19 (45.2%)	19 (44.2%)
Black or African American	4 (9.5%)	6 (14.0%)
Pacific Islander	0	1 (2.3%)
Other	4 (9.5%)	0
Missing	0	2 (4.7%)
Hispanic/Latino
Non-Hispanic/Latino	4 (9.5%)	2 (4.7%)	X2 (1, N = 83) = .67, p = 0.41
Hispanic/Latino	38 (90.5%)	39 (90.7%)
Missing	0	2 (4.7%)
Grade
Freshman	13 (31.0%)	9 (20.9%)	X2 (3, N = 83) = 4.52, p = 0.21
Sophomore	13 (31.0%)	21 (48.0%)
Junior	9 (21.4%)	4 (9.3%)
Senior	7 (16.7%)	7 (16.3%)
Missing	0	2 (4.7%)
Income
$10,000–$25,000	2 (4.8%)	6 (14.0%)	X2 (3, N = 83) = 2.75, p = 0.43
$ 25,000–$45,000	5 (11.9%)	5 (11.6%)
$45,000–60,000	3 (7.1%)	4 (9.3%)
Above $60,000	32 (76.2%)	26 (60.5%)
Missing	0	2 (4.7%)

Digital mental health app: NeuroFlow

NeuroFlow is a digital mental health app that utilizes behavioral economics to better mental wellness. NeuroFlow is implemented in several use cases and care settings, including NeuroFlow by health systems, health payers, government (e.g. military), and other organizations in support of the delivery of mental health programming, in combination with the delivery of new and existing program goals. Each use case is configured to provide relevant, available, and geographic specific resources for the target population. This use case was configured for a college population. The NeuroFlow app delivers resources on resilience training, mindfulness-based strategies and cognitive behavioral therapy (CBT) activities to adults looking to support their mental health and wellbeing, and draws from CBT frameworks to include such strategies as psychoeducation, goal setting, behavioral activation, and CBT worksheets (e.g. ABC worksheets). The specific “journey” (i.e. track or tracks of as activities, videos, trackers, and journaling) is tailored to individuals based on depression and anxiety screeners administered upon signing up for the app. All participants were placed on the “resilience” track which included emotion regulation strategies based on mindfulness-based stress reduction. Those also on the “depression” or “anxiety’ tracks were provided with CBT activities relevant to the identified condition(s), based on a cutoff score of 10 from the Generalized Anxiety Disorder-7 (GAD-7) and Patient Health Questionnaire-8 (PHQ-8) (see Secondary outcomes section). Therefore, tracks are not mutually exclusive, dependent on one another, or limited to one track assignment per participant.

Each journey consisted of a 4-week program which averaged one assigned activity (e.g. journaling) every 2–3 days. The length of all the video content was consistent with each video between 2 and 5 min. Participants were able to rewatch videos, find additional psychoeducation videos in the video library, and had access to all the worksheets and materials provided to be revisited or completed again. On a daily basis, students were prompted to provide daily mood and sleep ratings through push notifications and additional tracking options for stress and pain were available.

The nonfinancial behavioral economics strategies that NeuroFlow uses include motivational messages, gamification (point accrual for activity completion, daily app usage “streaks” with congratulatory messages and cartoon badges assigned for consecutive app use), and loss aversion techniques (notifications that refer to the user's points such as “complete an assessment now before your points will expire”). The financial incentives included are redeemable points accrued for activity completion for gift cards at popular outlets (e.g. 1000 points which is the maximum per month, the duration of the study =$10 gift card). In this study, two versions of the app were used: (1) all of the nonfinancial and financial behavioral economics incentives described above (the treatment group) and the nonfinancial behavioral economics incentives only (the control group). The control group app version was the same as the treatment group app version except the points accrued were not redeemable for financial incentives.

Track assignment

Figure 2 shows the track distribution of the control and treatment groups for resilience, anxiety, depression, and both anxiety and depression of the participants who completed at least one assessment. As the treatment groups are relatively well matched in the tracks, this again indicates that randomization to the control versus treatment groups was successful.

Figure 2.

Neuroflow track assignment. Note. All participants on the anxiety, depression, and anxiety and depression tracks were also assigned the resilience track.

Outcome measures

The baseline and post-app usage surveys were administered via the NeuroFlow app (which is Health Insurance Portability and Accountability Act (HIPAA) compliant) to determine which wellbeing/mental health journey participants were assigned to (depression, anxiety, or resilience, as described above).

Primary outcome

App engagement

Engagement level was measured through the total activity rate. The total activity rate was calculated through dividing the number of completed activities per participant during the duration of the study by the total activities assigned to that participant and is expressed as a percentage.

Secondary outcomes

Depression

Depression symptoms were measured using the 8-item Patient Health Questionnaire (PHQ-8),⁵³ on a 4-point Likert scale ranging from 0 (“not at all”) to 3 (“nearly every day”). The PHQ-8 is a diagnostic and severity measure for assessing depression disorders in large clinical settings.⁵³ The total score is reported on a scale of 0 to 24. Higher scores suggest a greater severity of depressive symptoms. A clinical cutoff score of 10 indicates clinically significant depressive symptoms. The PHQ-8 has high internal consistency reliability (Cronbach α = 0.82; Pressler, 2011) and high validity; 96.5% of those who score at the clinical cutoff or higher have been found to have a depression diagnosis.⁵³

Anxiety

Anxiety symptoms were measured using the GAD-7.⁵⁴ The GAD-7 consists of 7 items rated on a 4-point Likert scale ranging from 0 (“not at all”) to 3 (“nearly every day”) with the same options as in the PHQ-8. Scores on the items are summed to create a total severity score ranging from 0 to 21, with higher scores indicating greater symptom severity. A clinical cutoff score of 10 indicates clinically significant anxiety symptoms. The GAD-7 has been found to have internal consistency reliability (Cronbach's α = 0.89) and high validity, with moderate correlations with related constructs such as depression and self-esteem.⁵⁵

Wellbeing

Wellbeing was measured using the World Health Organization—five wellbeing index (WHO-5).⁵⁶ The WHO-5 consists of five items using a 6-point Likert scale ranging from 0 (“At no time”) to 5 (“All of the time”). Specifically, participants rate the frequency of their symptoms using these scale options: “all of the time,” “most of the time,” “more than half the time,” “less than half the time,” “some of the time,” and “at no time.” The total score is calculated by aggregating the scores of each item. A greater total score suggests greater wellbeing and quality of life. A clinical cutoff score of 50 or below indicates poor mood and 28 or below indicates probable depression. The WHO-5 has been found to have high internal consistency reliability (e.g. Cronbach's α = .81–.90), be valid in screening for depression, have strong construct validity among adolescents and adults, and be sensitive to change in interventions.^56,58

Emotion dysregulation

Emotion regulation difficulties were measured using the Difficulties in Emotion Regulation Scale (DERS).^59,60 The DERS is a self-report measure that includes 6 subscales, with 5 to 8 items each, with 36 items overall. Items are rated on a 6-point Likert scale ranging from 0 (“At no time”) to 5 (“All of the time”). Scores on each item are summed to create a total score with a range of 36 to 180, with higher scores suggesting greater difficulties with emotion regulation. The DERS is a validated measure for assessing emotion regulation problems among adolescents and adults that possesses high internal consistency (Cronbach's α = .93) and strong test–retest reliability.

Procedure

This study was approved by the University of Pennsylvania IRB and is registered on clinicaltrials.gov (NCT05121675). Random assignment of participant to treatment group was accomplished using a block randomization procedure to assign subjects to either the control (no financial incentives) or treatment (financial incentives) group.

After providing informed written consent to participate in the study, participants were required to complete an initial set-up meeting at baseline. Prior to the initial set-up meeting, participants were asked to complete baseline surveys of depressive and anxious symptoms, wellbeing, and emotion regulation difficulties. The initial set-up meeting was conducted with a research assistant at the beginning of this study to ensure that all participants were set up on the app, aware of what the study entailed and required of them, ensure that they had filled out the baseline measures, and answer any questions regarding the study. Participants who did not have access to a model device, were unwilling to download and use the NeuroFlow app, were not above 18 years of age, did not schedule an initial set-up meeting where app download and set-up instructions given, and/or did not download the app, were excluded from this study. All participants who met the criteria were included.

For the duration of the study, participants were asked to complete the NeuroFlow daily check in for 28 consecutive days. Participants were given a suggested minimum of 5 min a day and no maximum time limit to engage with the app. Participants were sent a push notification each day through the app to remind them to complete their daily activities, which included surveys, videos, or skill-based practice materials. Through the completion of these activities, participants accrued points—which participants in the treatment group could then exchange for gift cards.

The day the participant completed the baseline surveys was considered day 1 of the 28-day trial period for each participant. Following the conclusion of the trial period, participants were prompted to again complete the same set of measures. Participants signed up for a 30-min virtual exit meeting that was at minimum 30 days from the initial set-up meeting date to allow at least 2 days to complete the outcome measures before the meeting. The purpose of the exit meeting was to gauge user feedback on the NeuroFlow app for our app partners, ensure that participants completed the post-study measures and surveys, and inquire about which factor they thought to be most influential in increasing app engagement. “What did the NeuroFlow app do well” and “what can be improved on for the NeuroFlow app” are examples of questions asked in accordance with these goals.

Data analysis

Data was first analyzed for skewness, kurtosis, and outliers. All variables were normally distributed and therefore parametric tests were used. As shown in Table 1, the treatment and control groups were not significantly different in any demographic comparison statistics. All analyses that were run were intent to treat analyses as we included all participants who were randomized and downloaded the app regardless of whether participants received the app for the entire duration of the study (we did not have any data on app engagement if they did not download the app). Secondary outcome measure analyses included fewer participants for this reason as they did not complete those measures, but those participants were still included in the primary outcome analysis on app engagement.

To test hypothesis 1 (financial incentives increase app engagement), we ran a one-way analysis of variance (ANOVA) (2 groups: treatment vs. control) on activity completion rate. To test hypothesis 2 (those financial incentives have a positive effect on mental health and wellness), we ran a two-way repeated measures ANOVA (2 groups (treatment, control) × 2 timepoints (baseline, post-trial)) on change in depression (PHQ-8), anxiety (GAD-7), wellbeing (WHO-5), and emotion dysregulation (DERS) survey scores. To test exploratory hypothesis 3 (users will have better mental health and wellness at the end of the study period), we examined the timepoint main effect (baseline and post-trial) in the two-way repeated measures ANOVA described above on change in depression, anxiety, wellbeing, and/or emotion dysregulation survey scores. Qualification for this analysis was primarily based on participant completion of both the baseline and post-trial surveys (and therefore did not include any participants who were lost-to follow-up). Given that multiple comparisons were run on the secondary outcome measures, Bonferroni corrections were applied.

Results

Results reported below map onto the sequence of the hypotheses stated above.

Hypothesis 1: Impact of financial incentives on application engagement

We found that there was no treatment group effect on app engagement as measured by completion rate, F(1, 84) = 0.023, p = .881, η2 = .000 (95% CI of η2 = .000–.036) and therefore our hypothesis 1 was not supported. Specifically, there was no significant difference between the average completion rate percentage in the control versus treatment group, as presented in Figure 3.

Figure 3.

Neuroflow activity completion rate. Note. Average completion rate in percentage of control (no financial incentives) and treatment (financial incentives) groups. Error bars represent standard error and the number within the bar represents the mean.

Hypothesis 2: Impact of financial incentives on mental health/wellness

We found that there were no treatment group × timepoint interaction effects on any of the mental health or wellness variables, including depression symptoms, F(1, 57) = 0.083, p = .775, η2 = .001; anxiety symptoms, F(1, 58) = 1.627, p = .207, η2 = .027; wellbeing, F(1, 67) = .084, p = .773, η2 = .001; and emotion regulation difficulties, F(1, 62) = 0.335, p = .565; η2 = .005 and therefore our hypothesis 2 was not supported.

Hypothesis 3: Impact of NeuroFlow app on mental health/wellness

We found no significant main effect of time point for depression symptoms, F(1, 57) = 0.564, p = .456, η2 = .010, or wellbeing, F(1, 67) = 0.610, p = .437, η2 = .009. However, there was a significant main effect of timepoint for anxiety symptoms, F(1, 58) = 11.923, p = .001, η2 = .171, and emotion regulation difficulties, F(1, 62) = 7.096, p = .010, η2 = .103, such that scores decreased over time across treatment groups. Therefore, hypothesis 3 was partially supported. The means and standard deviations of each group's secondary outcomes scores are presented in Figure 4.

Figure 4.

Baseline to post-trial data on mental health/wellness outcomes variables. Note. Baseline versus post-trial for control and treatment groups. (a) depression symptoms as measured by the PHQ-8; (b) anxiety symptoms as measured by the GAD-7; (c) wellbeing as measured by the WHO-5; and (d) emotion dysregulation as measured by the DERS. See Methods section for details on each questionnaire. Error bars represent standard error. Significance level is indicated by the following scale: * p < 0.05 and ** p < 0.01. DERS: Difficulties in Emotion Regulation Scale; GAD-7: Generalized Anxiety Disorder-7; WHO-5: World Health Organization—five wellbeing index.

To elucidate whether the differences from baseline to post-trial in depression and anxiety were driven by participants that screened negative for anxiety and depression and were affected by regression to the mean, we examined the main effect of timepoint (baseline, post-trial) within participants who screened positive on their baseline depression and anxiety symptoms only (and thus were assigned to the Depression or Anxiety track, respectively), with a post-hoc repeated measures one-way ANOVA. Within this subgroup analysis we again found no significant main effect of timepoint for depression symptoms (n = 14; baseline: M = 13.50, SD = 3.70; post-trial: M = 11.13, SD = 4.12), F(1, 13) = 1.63, p = .24, η2 = .19. However, there was again a significant main effect of timepoint for anxiety symptoms (n = 17; baseline: M = 13.75, SD = 2.09; post-trial: M = 8.92, SD = 4.14), F(1, 16) = 23.78, p < .001, η2 = .68. Therefore, the findings of this subsample matched those from the whole group.

Discussion

The main aim of this study was to understand how financial incentives encourage engagement with a digital mental health app and mental health/wellness outcomes relating to app use. As a secondary focus, we explored users’ anxiety symptoms, depressive symptoms, wellbeing, and emotion regulation difficulties outcomes prior to and after engagement with either version of the NeuroFlow app.

Our results suggested that financial incentives alone do not significantly increase app engagement and do not significantly impact users’ anxiety symptoms, depressive symptoms, wellbeing, and emotion regulation difficulties. These results diverge from current literature on the topic given the use of behavioral economics through financial incentives as a means of invoking positive changes in behavior and engagement in treatment has been documented.^47,48,50 Similarly, additional research finds that this is observed in the healthcare sphere where behavioral changes led to health benefits.⁴⁹ However, these studies did not specifically look at the impacts of financial incentives on digital mental health app engagement and the benefits derived from implementing financial incentives on mental health and wellness. There are several potential confounders to consider, including the attainment of class credit for study participation. Furthermore, as this study was not comprised of a clinical population, a future direction for this research would be to investigate the impacts of the NeuroFlow app on users’ anxiety symptoms, depressive symptoms, wellbeing, and emotion regulation difficulties for clinical populations.

Despite the absence of impact of financial incentives on user engagement, we did find that participants reported statistically significant reductions in symptoms of anxiety and decreases in emotion regulation difficulties over the 28-day period of the study. Our findings for reductions in anxiety is consistent with previous research regarding digital mental health interventions among college students.³² Emotion regulation difficulties have not been extensively investigated in the literature,⁶¹ therefore this finding is relatively novel. However, we acknowledge that our study was not designed to examine the positive impact of the app on secondary outcomes due to a lack of an active control group and lack of inclusion criteria regarding mental health diagnoses. Therefore, the findings are suggestive but not definitive of the possible impact of digital mental health apps on reducing anxiety and emotion regulation difficulties over a relatively short period of time (1 month). The results are consistent with other studies which found similar results in digital mental health intervention demonstrated the potential to aid in mental wellbeing typically spanned even shorter periods of time from 1 h to 2 weeks or longer durations from 2 to 6 months.^62,65

Reductions in self-reported depression symptoms and increase in self-reported wellbeing were not statistically significant. Contrary to our results, a recent systematic review concluded that depression and wellbeing are effectively improved through digital mental health interventions.^61,66,67 It may be that depression symptoms and wellbeing do not respond well to digital mental health treatments applied in such a short time span (1 month). Indeed, most studies required a more extended period of intervention.^62,65 Future research should address whether there is a positive impact of the NeuroFlow app on depression symptoms and wellbeing over a longer time span using an active control group with individuals with a depression diagnosis. However, it must be noted that in our subgroup analysis of examining outcomes on individuals on the depression track, based on their corresponding symptoms meeting screening measure thresholds, the pattern of results identified was similar to the results from the whole group, with no improvements in depression symptoms over the study period. By contrast, in individuals on the anxiety track, anxiety symptoms decreased during the study period, also in line with the findings of the whole sample. As mentioned above, future well-designed research with active controls and clinical inclusion criteria is needed to determine NeuroFlow's clinical impact.

Overall completion rate across the groups was low (around 50%), however this is consistent with previous research that has found low (even lower) study completion rates in college students (e.g. 14%–32%).⁶⁸ Nevertheless, this result poses the question about how best to optimize and individualize the use of behavioral incentives in digital mental health apps to increase user engagement and health benefits. For example, understanding which constellation of behavioral incentives work best from all those available (e.g. point accrual, tailored reminders, feedback messages, etc.) and for whom (e.g. what works best for all college students vs. adults with mental health diagnoses).

Another valid direction for research is to understand the payment structure sustainability of financial incentives, including whether insurance companies and government/public payers will invest in financial incentives to improve mental health outcomes. Given that some research has identified that financial incentives decrease healthcare costs in the long run,⁴⁹ the return on investment may attract payers to pursue this route. However, more research is needed in the area of financial incentives in digital mental health before this determination can be made for digital mental health.

Limitations

Our study is not without limitations. First, although the NeuroFlow app in our study is differentiated into two groups, control (no financial incentives), and treatment (financial incentives) the use of nonfinancial behavioral incentives (gamification, motivational messaging, and loss aversion techniques) is present in both groups. A systematic review found conflicting results regarding the efficiencies of use of gamification.⁶⁹ Therefore, it may be worthwhile to replicate the study with four groups, control (no incentives), and three treatment conditions (nonfinancial incentives, financial incentives, and nonfinancial incentives and financial incentives combined), instead of two, to observe the impacts, if any, of nonfinancial incentives such as gamification on engagement with a mental health app. Future research should include these four arms to further investigate the relative impact of nonfinancial and financial behavioral economics incentives of app engagement and mental health and wellness outcomes.

Second, while we had a relatively low attrition rate such that we did have app use data that could be used for the primary outcome analysis, many participants did not complete the post-trial measures for the secondary outcome analyses. Third, another limitation of the study was the use of course credit as a means of incentivizing participants to enroll, which introduces a confounding variable as our study sought to investigate the impacts of behavioral economics in the form of financial incentives on app engagement (however both treatment groups received course credit, which minimizes the impact on the group comparison results). Fourth, there is potential low salience of the financial incentive administered in the treatment group as our participant pool stemmed from a relatively well-resourced population (University of Pennsylvania students). Future research should resolve these limitations and expand the study to include a larger sample of adolescents, adults, and more diverse less resourced or under-resourced populations. In addition, participants were unable to be blinded to treatment allocation as the participants in the treatment group received financial incentives directly through the application and self-reported on secondary measure outcomes.

Finally, we did not target recruit students experiencing clinical levels of depression and/or anxiety. In turn, we acknowledge that we cannot comment on the efficacy of the NeuroFlow app or addressing clinical levels of depression and/or anxiety without the presence of a sufficient sample of participants who meet the threshold for depression and/or anxiety on the PHQ-8 and GAD-7, respectively. Note, as we were not using the treatment effect as the primary hypothesis, this concern more so contributes to a promising future direction for this field of research.

Conclusion

Our results suggest that financial incentives do not have a positive impact on app engagement or mental health and wellness.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231170693 - Supplemental material for Money can’t buy happiness: A randomized controlled trial of a digital mental health app with versus without financial incentives

Supplemental material, sj-docx-1-dhj-10.1177_20552076231170693 for Money can’t buy happiness: A randomized controlled trial of a digital mental health app with versus without financial incentives by Cheryl Chang, Emma Palermo, Sky Deswert, Alyssa Brown and Heather J Nuske in DIGITAL HEALTH

Footnotes

Acknowledgements

We would like to acknowledge all the incredible staff at NeuroFlow that supported the implementation of this trial, including but not limited to (in alphabetical order) Amanda Brooks, Jarrett Button, Maggie Hanson, Karm Kaur, Jung Kim, Ellen McGeoch, Matt Miclette, and Adam Pardes.

Contributorship

HJN acquired funding for the study and conceived the study. HJN, CC, and EP were involved in the protocol development. CC and HJN gained ethical approval. CC recruited participants, researched literature and wrote the first draft of the manuscript. CC and HN conducted the data analysis. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The ethics committee of the University of Pennsylvania approved this study (IRB number: 849894).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: NeuroFlow.

Guarantor

HJN.

ORCID iD

Heather J Nuske

Supplemental material

Supplemental material for this article is available online.

References

Hunt

Eisenberg

. Mental health problems and help-seeking behavior among college students. J Adolesc Health 2010; 46: 3–10.

Kitzrow

. The mental health needs of today’s college students: challenges and recommendations. NASPA J 2003; 41: 167–181.

Patel

Flisher

Hetrick

, et al. Mental health of young people: a global public-health challenge. Lancet 2007; 369: 1302–1313.

Angst

. Comorbidity of mood disorders: a longitudinal prospective study. Br J Psychiatry Suppl 1996; 168: 31–37.

Kessler

Foster

Saunders

, et al. Social consequences of psychiatric disorders, I: educational attainment. Am J Psychiatry 1995; 152: 1026–1032.

Kessler

Walters

Forthofer

. The social consequences of psychiatric disorders, III: probability of marital stability. Am J Psychiatry 1998; 155: 1092–1096.

Wang

Simon

Avorn

, et al. Telephone screening, outreach, and care management for depressed workers and impact on clinical and work productivity outcomes: a randomized controlled trial. JAMA 2007; 298: 1401–1411.

Weitzman

. Poor mental health, depression, and associations with alcohol consumption, harm, and abuse in a national sample of young adults in college. J Nerv Ment Dis 2004; 192: 269–277.

National Alliance on Mental Health [NAMI]. 2020 Mental Health by the Numbers: Youth and Young Adults, https://nami.org/mhstats (2020).

10.

Blanco

Okuda

Wright

, et al. Mental health of college students and their non-college-attending peers: results from the national epidemiologic study on alcohol and related conditions. Arch Gen Psychiatry 2008; 65: 1429–1437.

11.

Thompson

. Emotion regulation: a theme in search of definition. Monogr Soc Res Child Dev 1994; 59: 25–52. 250–283.

12.

Lindsey

Fabiano

Stark

. The prevalence and correlates of depression among college students. Coll Stud J 2009; 43: 999–1014.

13.

Westefeld

. Suicide prevention and psychology: a call to action. Prof Psychol Res Pract 2019; 50: 1–10.

14.

Berking

Wupperman

. Emotion regulation and mental health: recent findings, current challenges, and future directions. Curr Opin Psychiatry 2012; 25: 128–134.

15.

Cisler

Olatunji

Feldner

, et al. Emotion regulation and the anxiety disorders: an integrative review. J Psychopathol Behav Assess 2010; 32: 68–82.

16.

Min’er

Dejun

. Emotion regulation and depression of college students. Chin Ment Health J 2001; 15: 438–441.

17.

Copeland

McGinnis

Bai

, et al. Impact of COVID-19 pandemic on college student mental health and wellness. J Am Acad Child Adolesc Psychiatry 2021; 60: 134–141. e2.

18.

Wang

Hegde

Son

, et al. Investigating mental health of US college students during the COVID-19 pandemic: cross-sectional survey study. J Med Internet Res 2020; 22: e22817.

19.

Panchal

Kamal

, Feb 10 RGP , et al. The Implications of COVID-19 for Mental Health and Substance Use. KFF, https://www.kff.org/coronavirus-covid-19/issue-brief/the-implications-of-covid-19-for-mental-health-and-substance-use/ (2021, accessed 5 March 2022).

20.

Beidas

Volpp

Buttenheim

, et al. Transforming mental health delivery through behavioral economics and implementation science: a project protocol (Preprint). JMIR Res Protoc 8. Epub ahead of print 5 September 2018. DOI: 10.2196/12121.

21.

Mental health statistics: people seeking help. Mental Health Foundation, https://www.mentalhealth.org.uk/statistics/mental-health-statistics-people-seeking-help (2018, accessed 7 February 2022).

22.

Lubian

Weich

Stansfield

, et al. Chapter 3: mental health treatment and services. In: McManus

Bebbington

Jenkins

Brugha

(eds) Mental health and wellbeing in England: adult psychiatric morbidity survey 2014. 3, Leeds, UK: NHS Digital, 2016, pp.69–69.

23.

Chamberlin

. Survey says: more Americans are seeking mental health treatment, https://www.apa.org, https://www.apa.org/monitor/julaug04/survey (2004, accessed 7 February 2022).

24.

Tucker

Davison

. Waiting to see the doctor: the role of time constraints in the utilization of health and behavioral health services. In: Bickel WK and Vuchinich RE (eds). Reframing health behavior change with behavioral economics. Mahwah, NJ: Lawrence Erlbaum Associates Publishers, 2000, pp.219–264.

25.

Pedersen

Paves

. Comparing perceived public stigma and personal stigma of mental health treatment seeking in a young adult sample. Psychiatry Res 2014; 219: 143–150.

26.

Gearing

MacKenzie

Ibrahim

, et al. Stigma and mental health treatment of adolescents with depression in Jordan. Community Ment Health J 2015; 51: 111–117.

27.

Dear

Johnston

, et al. An internet survey of emotional health, treatment seeking and barriers to accessing mental health treatment among Chinese-speaking international students in Australia. Couns Psychol Q 2014; 27: 96–108.

28.

Huckvale

Nicholas

Torous

, et al. Smartphone apps for the treatment of mental health conditions: status and considerations. Curr Opin Psychol 2020; 36: 65–70.

29.

Blustein

Duffy

Ferreira

, et al. Unemployment in the time of COVID-19: a research agenda. J Vocat Behav 2020; 119: 103436.

30.

Lecomte

Potvin

Corbière

, et al. Mobile apps for mental health issues: meta-review of meta-analyses. JMIR MHealth UHealth 2020; 8: e17458.

31.

Marshall

Dunstan

Bartik

. Clinical or gimmickal: the use and effectiveness of mobile mental health apps for treating anxiety and depression. Aust N Z J Psychiatry 2020; 54: 20–28.

32.

Mak

Chio

Chan

, et al. The efficacy of internet-based mindfulness training and cognitive-behavioral training with telephone support in the enhancement of mental health among college students and young working adults: randomized controlled trial. J Med Internet Res 2017; 19: e6737.

33.

Titov

Dear

Staples

, et al. Mindspot clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv Wash DC 2015; 66: 1043–1050.

34.

Torous

Nicholas

Larsen

, et al. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018; 21: 116–119.

35.

Huang

H-Y

Bashir

. Users’ adoption of mental health apps: examining the impact of information cues, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5508115/ (2017, accessed 28 January 2022).

36.

Mohr

Weingardt

Reddy

, et al. Three problems with current digital mental health research … and three things we can do about them. Psychiatr Serv 2017; 68: 427–429.

37.

Loewenstein

Asch

Volpp

. Behavioral economics holds potential to deliver better results for patients, insurers, and employers. Health Aff Proj Hope 2013; 32: 1244–1250.

38.

Mierlo

Hyatt

Fournier

. Behavioral economics holds potential to deliver better results for patients, insurers, and employers | health affairs. JMIR Publications, https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2012.1163 (2016, accessed 18 May 2022).

39.

Purnell

Thompson

Kreuter

, et al. Behavioral economics: “nudging” underserved populations to be screened for cancer. Prev Chronic Dis 2015; 12: E06.

40.

Calvillo-Gámez

Cairns

Cox

. Assessing the core elements of the gaming experience. In: Bernhaupt

(ed.) Game user experience evaluation. Cham: Springer International Publishing, 2010, pp.37–62.

41.

Egel

. Reinforcer variation: implications for motivating developmentally disabled children. J Appl Behav Anal 1981; 14: 345–350.

42.

Cotton

Patel

. Gamification use and design in popular health and fitness mobile applications. Am J Health Promot AJHP 2019; 33: 448–451.

43.

Cugelman

. Gamification: what it is and why it matters to digital health behavior change developers. JMIR Serious Games 2013; 1: e3139.

44.

Looyestyn

Kernot

Boshoff

, et al. Does gamification increase engagement with online programs? A systematic review. PLoS One 2017; 12: 1–19.

45.

Cronk

. Using gamification to increase student engagement and participation in class discussion. In: Amiel

Wilson

(eds) Proceedings of EdMedia + innovate learning. Denver, Colorado: Association for the Advancement of Computing in Education (AACE), 2012, pp.311–315.

46.

Morford

Witts

Killingsworth

, et al. Gamification: the intersection between behavior analysis and game design technologies. Behav Anal 2014; 37: 25–40.

47.

Linnemayr

Rice

. Insights from behavioral economics to design more effective incentives for improving chronic health behaviors, with an application to adherence to antiretrovirals. J Acquir Immune Defic Syndr 1999 2016; 72: e50–e52.

48.

Vlaev

King

Darzi

, et al. Changing health behaviors using financial incentives: a review from behavioral economics. BMC Public Health 2019; 19: 1059.

49.

Lunze

Paasche-Orlow

. Financial incentives for healthy behavior: ethical safeguards for behavioral economics. Am J Prev Med 2013; 44: 659–665.

50.

Khazanov

Morris

Beed

, et al. Do financial incentives increase mental health treatment engagement? A meta-analysis. J Consult Clin Psychol 2022; 90: 528.

51.

Marcus

Reilly

Zentgraf

, et al. Effect of escalating and deescalating financial incentives vs usual care to improve antidepressant adherence: a pilot randomized clinical trial. JAMA Psychiatry 2021; 78: 222–224.

52.

Rudd

Beidas

. Digital mental health: the answer to the global mental health crisis? JMIR Ment Health 2020; 7: e18472.

53.

Kroenke

Strine

Spitzer

, et al. The PHQ-8 as a measure of current depression in the general population. J Affect Disord 2009; 114: 163–173.

54.

Spitzer

Kroenke

Williams

, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166: 1092–1097.

55.

Löwe

Decker

Müller

, et al. Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population. Med Care 2008; 3: 266–274.

56.

Topp

Østergaard

Søndergaard

, et al. The WHO-5 well-being index: a systematic review of the literature. Psychother Psychosom 2015; 84: 167–176.

57.

Lara-Cabrera

Betancort

Muñoz-Rubilar

, et al. Psychometric properties of the WHO-5 well-being index among nurses during the COVID-19 pandemic: a cross-sectional study in three countries. Int J Environ Res Public Health 2022; 19: 10106.

58.

Sischka

Costa

Steffgen

, et al. The WHO-5 well-being index–validation based on item response theory and the analysis of measurement invariance across 35 countries. J Affect Disord Rep 2020; 1: 100020.

59.

Gratz

Roemer

. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess 2004; 26: 41–54.

60.

Difficulties in Emotion Regulation Scale (DERS). NovoPsych. https://novopsych.com.au/assessments/formulation/difficulties-in-emotion-regulation-scale/ (2021, accessed 7 February 2022).

61.

Lattie

Adkins

Winquist

, et al. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: systematic review. J Med Internet Res 2019; 21: e12869.

62.

Cukrowicz

Joiner

. Computer-based intervention for anxious and depressive symptoms in a non-clinical population. Cogn Ther Res 2007; 31: 677–693.

63.

Day

McGrath

Wojtowicz

. Internet-based guided self-help for university students with anxiety, depression and stress: a randomized controlled clinical trial. Behav Res Ther 2013; 51: 344–351.

64.

Ellis

Campbell

Sethi

, et al. Comparative randomized trial of an online cognitive-behavioral therapy program and an online support group for depression and anxiety. J CyberTherapy Rehabil 2011; 4: 461–467.

65.

Fitzpatrick

Darcy

Vierhile

. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): a randomized controlled trial. JMIR Ment Health 2017; 4: e7785.

66.

Davies

Morriss

Glazebrook

. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J Med Internet Res 2014; 16: e3142.

67.

Firth

Torous

Nicholas

, et al. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord 2017; 218: 15–22.

68.

LaRose

Tsai

. Completion rates and non-response error in online surveys: comparing sweepstakes and pre-paid cash incentives in studies of online behavior. Comput Hum Behav 2014; 34: 110–119.

69.

Cheng

VWS

Davenport

Johnson

, et al. Gamification in apps and technologies for improving mental health and well-being: systematic review. JMIR Ment Health 2019; 6: e13717.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB