Abstract
Demand for the use of mobile apps in mental health interventions has grown in recent years, particularly among adolescents who experience elevated levels of distress. However, there is a scarcity of evidence for the effectiveness of these tools within this population. The aim of this study was to test the effectiveness of CopeSmart, a mental health mobile app, using a multicentre cluster randomised controlled trial design. Participants were 15–18-years-olds (N = 560) recruited from 10 schools randomly assigned to an intervention or control condition. Intervention participants used the app over a 4-week period. Multi-level modelling analyses revealed no significant changes in the intervention group from pre-test to post-test, when compared to the control group, in terms of emotional distress, well-being, emotional self-awareness or coping strategies. Findings suggest that a 4-week app-based intervention may not be enough to elicit intra-personal changes in mental health outcomes in a general adolescent population.
Introduction
The term ‘mobile health’ or ‘mHealth’ refers to the use of mobile technologies in health contexts, 1 which has increased significantly in recent years due to technological advances and the widespread uptake of smartphone ownership.2,3 The potential for mHealth in mental health care is increasingly being recognised, 4 particularly in adolescence when there is typically a rise in the prevalence of mental health difficulties.5–7
A key feature of smartphones is the ability to run software applications, known as ‘apps’. 8 Apps can provide almost constant support where and when users need it, compared to interventions that can only be delivered at specific times or locations.9–12 They provide a means of accessing support anonymously, 13 which may facilitate help-seeking in people for whom stigma remains an obstacle to accessing mental health services,14,15 and thus provide a unique platform for the delivery of self-management interventions. The World Health Organisation refers to self-management as a means of putting individuals in control of managing their own health conditions, focusing on problem-solving, goal setting, identifying triggers of deteriorating health and responding to these. 16 Self-management interventions have improved health status for patients experiencing a range of physical conditions 17 and have potential to play a role in mental health. For example, a self-management mobile app called PRISM, 18 used with adults with Bipolar Disorder, prompted users to self-monitor mood states and provided personalised coping strategies based on this. Users reported significant reductions in depressive symptoms compared to controls. 19 However, MONARCA, an app also used with adults with Bipolar Disorder, 20 where users monitored mood and factors such as sleep, activities, stress, and alcohol showed no effect in improving mental health outcomes compared to controls. The authors suggest that as both groups also received psychopharmacological treatment and face-to-face therapy, there may not have been potential to improve outcomes above and beyond treatment as usual by adding an app-based component.
Looking to non-clinical populations, a mobile app called Mobiletype, 21 was evaluated using a randomised controlled trial (RCT) with adolescents who reported elevated levels of distress.21,22 Users monitored mood, stress, coping strategies, activities, eating, sleeping, exercise and substance use on the app. Those who used the app in conjunction with visits to their general practitioner (GP) showed significant increases in emotional self-awareness (ESA), which was associated with reductions in depressive symptoms. An RCT with adults evaluated an app called MoodHacker, where users tracked daily mood and activities and received content-promoting positive coping behaviours. This app reduced depressive symptoms in adults exhibiting mild-to-moderate depressive symptoms at baseline, compared to controls. 23 Another app called Health Mashups required adult users to self-report data about food, exercise, mood and pain 24 and using other data from the phone (e.g. step count, sleep, weight) presented personalised feedback on their patterns of health behaviours. The developers postulated that reflecting on these patterns would help users make informed choices about their health behaviours, leading to improved well-being. Results indicated that using Health Mashups was associated with increased well-being after 3 weeks, which was maintained after 3 months of use.
All these tools share the commonality of eliciting engagement in emotional self-monitoring through mobile apps. This falls under the category of self-management and helps people to identify periods where they may be struggling with their emotions, providing insight into what the triggers of these periods might be. Kauer et al. 21 describe how emotional self-monitoring increases ESA by teaching individuals to recognise, identify and differentiate between various emotional states, which allows them to respond to early symptoms of distress before they become more severe, serving as a protective factor against development or progression of mental health difficulties.25–28
In line with this, the majority of these studies saw direct or indirect reductions in depressive symptoms or improvements in well-being over time,19,21,23,24 suggesting technologies involving self-monitoring of mood states may be efficacious in reducing negative affect and improving well-being. However, these studies have a number of limitations. For example, the study on Health Mashups did not use a control group; thus, the extent to which changes can be attributed to the intervention 24 is limited. In addition, the evaluation of Mobiletype did not find a direct effect on depressive symptoms as a result of the intervention, only an indirect effect through ESA. 21
A feature evident in many of these self-management interventions is the promotion of positive strategies for coping with distress.19,29 Use of positive coping strategies is associated with positive mental health outcomes in young people and is considered to be an important protective factor in adolescence.30–32 A systematic review of online youth mental health promotion and prevention interventions 33 identified 28 technology-based interventions, 16 of which focused on the promotion of positive coping strategies in some format (such as relaxation techniques, emotional regulation, cognitive restructuring and problem-solving). These interventions were linked to beneficial outcomes including reductions in anxiety and depression, improved stress management, increased well-being, increased positive thinking, reduced anxiety and reduced psychological distress in secondary school and college age young people.
While these findings suggest promise for the use of technology-based coping interventions, the authors found that the majority of studies included were of moderate-to-weak quality, highlighting the need for more methodologically rigorous research in this area. Furthermore, these interventions were delivered on computers rather than mobile devices. In light of the fact that promoting positive coping is a feature of many mental health mobile apps for adults,19,34–36 there is currently a gap in the literature examining the effectiveness of such interventions delivered through a mobile app to an adolescent population.
While there are some promising preliminary findings, concerns have been raised about the evidence base for mobile apps in mental health, with the majority of studies in this area underpowered and lacking methodological rigour.3,13,37–40 However, demands for such technologies continue to grow41,42 and there is a clear need for rigorous research to bridge the gap between consumer desire for mobile mental health interventions and empirically supported options available for them.3,13,43 Furthermore, although there is a high prevalence of mental health difficulties in adolescents, there is a scarcity of high-quality studies testing the effectiveness of e-resources for mental health self-management with this age group, particularly in non-clinical samples, 44 despite the plethora of mental health apps that are widely available to the general public. 4
The present study
The aim of this study was to address the gaps in the literature by testing the effectiveness of CopeSmart, a mental health mobile app which promotes self-management through emotional self-monitoring and the use of positive coping strategies. Adolescent users were encouraged to engage with the app regularly in their own personal environments during the course of their normal day-to-day activities. The app allowed them to rate how happy, angry, sad, stressed or worried they felt on a scale of 1–10 and users were able to review their ratings over time. The app provided a series of positive coping tips informed by the ‘Think Positively’ school-based course for positive coping in adolescence. 45 A tip was randomly selected to display each time a user completed a mood rating and all tips were available for users to browse at their leisure. The app also listed contact details for local mental health support services. A more detailed description of the app and its development are presented in the study by Kenny et al. 46
It was expected promotion of emotional self-monitoring through the app would lead to increased ESA; thus, the primary outcome under study was change in ESA as a result of the intervention. It was expected that promotion of positive coping strategies would result in increased use of positive coping strategies and decreased use of negative coping strategies. To validate that ESA and coping strategies were appropriate mechanisms to target, their relationship to emotional distress and well-being was tested prior to testing the primary and secondary hypotheses. Secondary outcomes were changes in the use of positive and negative coping strategies, help-seeking behaviour, general well-being and emotional distress.
1. Intervention group participants will report increases in ESA over time in comparison to control group participants.
Intervention group participants will report the following:
2. Increases in their use of functional coping strategies,
3. Decreases in their use of dysfunctional coping strategies,
4. Decreases in distress,
5. Increases in well-being and
6. Increased willingness to seek help
over time in comparison to control group participants.
Method
Sampling
A multicentre cluster RCT design was employed. Clusters were defined as the second-level schools in the Dublin Mid-Leinster Health Service Executive (HSE) Region (population of approximately 1.3 million at the time of recruitment (May 2014–January 2015)).47,48
G*Power 3.1 software was used to determine sample size required for a conventional parallel-group trial as N = 156 based on the following parameters: a moderate effect size of 0.25, alpha of 0.05, power of 0.8, two groups, three test occasions and correlation among repeated measures as 0.6. 49 To account for lack of independence between participants within clusters, this estimate was increased using the variance inflation factor, defined as 1 + (n – 1)ρ, where n is the average cluster size and ρ is the intraclass correlation coefficient (ICC; a statistical measure of the degree of outcome correlation within clusters). 50 The average cluster size was estimated as 42 participants per school based on previous research. 5 Previously reported ICCs for mental health outcomes in secondary schools range from 0.004 to 0.01;51–53 thus, the ICC was estimated as 0.01. This gave a variance inflation factor of 1.41 which, multiplied by the original sample size estimate, gave an adjusted minimum sample size of N = 220 for a sufficiently powered study.
As lower socioeconomic status (SES) is linked to higher risk of emotional problems in adolescents,54–56 the SES of the school was considered as a potential covariate (whether or not schools had a DEIS (‘delivering equality of opportunity in schools’) status, that is, nationally identified as socially or economically disadvantaged). 57 The literature has suggested school environment may differ between single-sex and co-educational schools; 58 therefore, school gender mix was also considered as a potential covariate.
Participants
Eligible participants were all Senior Cycle students (fourth to sixth year). The exclusion criterion was failure to return parental consent. Overall, 10 of the 44 randomly selected schools (23%) agreed to participate and were assigned to the control or intervention condition using a random binary computer-generated sequence (six in the intervention condition and four in the control condition). Figure 1 shows participant recruitment. Demographic characteristics are presented in Table 1.
Flowchart of RCT participants.
Demographic characteristics of sample.
SD: standard deviation; DEIS: delivering equality of opportunity in schools.
Fourteen percent of participants reported they did not know their mother’s highest education level and 21 percent of participants reported that they did not know their father’s highest education level. Six percent of participants did not answer the question about whether or not they were seeing a mental health professional. Participants who responded ‘do not know’ or ‘did not answer’ were not included in further analyses involving these variables.
The junior certificate is a state examination taken mid-way through second-level education.
The leaving certificate is a state examination taken at the end of second-level education.
Outcome measures
The Emotional Self-Awareness Scale
The Emotional Self-Awareness Scale (ESAS) measures ESA. 21 It has displayed high internal reliability (α = 0.83) and shown sensitivity to change following a mobile app intervention in an adolescent sample. 21 In this study, ESA was defined in line with the literature on this topic59–62 as ‘the recognition and identification of emotions in one’s self as they occur’; thus, the recognition, identification and contextualisation subscales were used to form a total ESA score for each participant. This comprised a combination of 21 positively and negatively worded statements to which participants respond on a 5-point scale ranging from ‘Never’ to ‘A Lot’, with higher scores indicating a higher level of ESA.
The Coping Strategies Inventory – Short Form
The 32-item Coping Strategies Inventory (CSI) 63 assesses functional or ‘engagement’ coping strategies and dysfunctional or ‘disengagement’ coping strategies in young people. Participants were asked to indicate how much they use these strategies to cope with problems on 5-point scale ranging from ‘Never’ to ‘Very Often’. Across subscales, the CSI has shown satisfactory internal consistency (α = 0.69–0.94)63,64 and good test–retest reliability (r = 0.67–0.83) in young adults 63 and has shown concurrent validity with other measures of coping. 65
Depression Anxiety and Stress Scale – Short Version
The Depression Anxiety and Stress Scale-21 (DASS-21) 66 assesses psychological distress. It is composed of three subscales representing depression, anxiety and stress. Responses are made on a 4-point scale based on the participants’ experiences of the past week, ranging from ‘Did Not Apply to Me At All’ to ‘Applied to Me Most of the Time’. The DASS has high internal reliability as an overall measure of distress (α = 0.93) in adolescent samples.67–71
WHO-Five Well-Being Index
The WHO-Five Well-Being Index (WHO-5) 72 is a five-item measure of positive psychological well-being. Participants rate how much these positive feelings have been present for them in the last 2 weeks on a 6-point scale ranging from ‘At No Time’ to ‘All of the Time’. The scale’s one-factor structure has been confirmed, satisfactory concurrent validity has been established with other mental health measures73,74 and it has consistently shown high internal reliability (α = 0.82–0.89) in adolescent samples.73,75
General Help-Seeking Questionnaire
The General Help-Seeking Questionnaire (GHSQ) 76 was used to assess help-seeking intentions. Participants were asked ‘If you were having a personal or emotional problem, how likely is it that you would seek help from the following people/places?’ and were presented with a list of 12 sources, including both informal sources (e.g. mother, father, friend) and formal sources (e.g. psychologist, phone help-line, teacher). Participants’ responses were made on a 5-point scale ranging from ‘Very Unlikely’ to ‘Very Likely’. The GHSQ has been found to have satisfactory test–retest reliability (r = 0.86–0.92), internal reliability (α = 0.70–0.88) and convergent validity with measures of established help-seeking correlates in adolescent samples.77,78
Procedure
Ethical approval was granted by the Human Research Ethics Committee in the authors’ university. Data collection took place from October 2014 to May 2015. Participation was voluntary. Students in the intervention condition completed questionnaires at baseline (Time 1) and were given instructions on downloading and using the CopeSmart app and asked to engage with it each day over the following 4 weeks. Participants completed Time 2 questionnaires 4–5 weeks after baseline measures. Follow-up Time 3 questionnaires were completed 8–10 weeks after Time 2 questionnaires. Students in the control condition (no intervention) completed the same questionnaires at the same time intervals. To link data across time points, students were required to generate an Anonymous Identification Code, which they completed at all three time points and, where applicable, input to the app upon initial download.
Missing data
Follow-up data were collected from participants who were present in class on the day of data collection. In total, 91 percent of participants (n = 511) completed questionnaires at two time points and 67 percent (n = 373) completed questionnaires at three time points. Analyses indicated no differences between those who completed questionnaires at all time points and those who completed questionnaires at one or two time points, with the exception of school year group χ2(2, N = 560) = 17.30, p < 0.001, where fourth-year students fourth-year students were more likely to have not completed a questionnaire at one of the time points (standardised residual (SR) = 2.3). This is likely attributable to fourth-year students participating in extra-curricular activities 79 outside school.
Missing data points also occurred within individual cases, where a question was skipped or the response was unintelligible. Greater amounts of missing values occurred for items that fell towards the end of the questionnaire. Fourth-year students were more likely to be missing data on the CSI Disengagement Scale (SR = 2.5) and the GHSQ (SR = 2.3). Missingness on the CSI Disengagement Scale was also more likely to occur among lower SES participants, as measured by school DEIS status (SR = 3.0) and lower maternal education status (SR = 3.3). Thus, these variables were included as covariates in the corresponding models run to test the study hypotheses.
Analysis
All analyses were conducted using IBM SPSS statistics Version 20.0.0. Linear regressions tested whether the key intervention mechanisms (ESA and coping strategies) predicted mental health outcomes. Where multiple comparisons were conducted simultaneously, p values were adjusted using the false discovery rate correction for multiple comparisons. 80
To test the primary study hypotheses, taking into account the hierarchical nature of data in cluster RCTs by allowing parameters within models to vary across higher-level groups or clusters, 81 and allowing for correlated repeated measures longitudinally within each participant, we fitted a linear mixed model, using the SPSS MIXED procedure. Specifically, we fitted a random intercept for schools and modelled repeated measures marginally using the REPEATED command.
For each outcome, in order to initially assess variance sources, a baseline model with no fixed predictors was estimated, with schools as random clusters and students as random effects nested within schools, which provided a partitioning of the variance between level 1 (student level) and level 2 (cluster level) observations. 82 For all outcomes, the initial baseline model indicated significant variability at the level of the student, but not at school level, suggesting only a small portion of the variance occurred due to schools as a cluster effect.
To assess the covariance structure of the data, for each outcome, a model was run with an unstructured covariance matrix over time, and contrasting models with a more restricted structure (compound symmetric and first-order auto-regressive, both with and without heterogeneous variances). These models were compared to each other using the Akaike Information Criterion, a finite-sample corrected AIC (AICC), a Bayesian Information Criterion (BIC) and likelihood ratio tests, to determine which structure best fit the data for that outcome, and guide the primary models (as above) for that outcome, with the appropriate covariance structure. A compound symmetric covariance structure was deemed appropriate for all outcomes except emotional distress. For emotional distress, a heterogeneous compound symmetric structure was the best fit and was used for estimating models for this outcome. However, When school DEIS status and school gender mix were included as covariates in the models for Emotional Distress, they did not adequately converge with a heterogeneous compound symmetric structure; thus, the more general unstructured covariance matrix was used.
To test the hypotheses, a model for each outcome was specified which included (1) Time, (2) Group Assignment (i.e. intervention vs control) and (3) the interaction between Time and Group as fixed effects. A random effect of school was also included, and the residuals for each student were allowed to be correlated over time, within the selected covariance structure, blocked within schools. In line with best practice,83,84 these models were run as both intention-to-treat (ITT) and per-protocol (PP) analyses. PP was defined as having used the app on more than 1 day during the intervention period (N = 132). PP, models compared intervention participants who met minimum usage requirements (PP), intervention participants who did not (not PP) and control group participants. No differences in Time × Group interaction effects emerged between ITT and PP analyses across outcomes; thus, additional models for each outcome, with school DEIS status and school gender mix included as fixed effects, were run on an ITT basis. Where relevant, variables associated with missingness were included as fixed effects.
Results
Descriptive statistics for study outcome measures are presented in Table 2. Linear regressions of the data collected from participants at Time 1 indicated that ESA predicted lower emotional distress (β = −0.42, p < 0.001, adjusted r2 = 0.18) and higher well-being (β = 0.48, p < 0.001, adjusted r2 = 0.23). Similarly, use of functional coping strategies predicted lower distress (β = −0.27, p < 0.001, adjusted r2 = 0.07) and higher well-being (β = 0.41, p < 0.001, adjusted r2 = 0.17), whereas use of dysfunctional coping strategies predicted higher distress (β = 0.61, p < 0.001, adjusted r2 = 0.37) and lower levels of well-being (β = −0.41, p < 0.001, adjusted r2 = 0.16).
Descriptive statistics for study outcome measures (N = 560).
ESAS: Emotional Self-Awareness Scale; CSI: Coping Strategies Inventory; DASS: Depression Anxiety and Stress Scale; WHO-5: WHO-Five Well-Being Index; GHSQ: General Help-Seeking Questionnaire.
In the intervention condition, 64 percent (N = 247) of participants reported downloading the app and 36 percent (N = 138) did not. Chi-square analyses indicated no differences in demographic characteristics or baseline measures of distress between those who reported downloading the app and those who did not. Of those who reported downloading the app, backend data indicated that 153 participants subsequently used it, engaging with it on approximately 6.56 separate days within the 28-day intervention period (standard deviation (SD) = 5.42). Of these, 132 participants used the app more than once within the 28 days and were included in PP analyses. Pearson’s correlations revealed no relationship between the number of days participants used the app and changes in outcomes from pre-test (Time 1) to post-test (Time 2) (rESA = 0.117, rCSI-Engagament = −0.115, rCSI-Disengagement = −0.066, rDASS = −0.063, rWHO5 = −0.090, rGHSQ = −0.101); thus, number of days of use of the app was not considered as a covariate in subsequent analyses.
Primary outcome
ESA
Omnibus F values and estimated marginal means for ESA models are presented in Table 3. Scores over time for the intervention and control groups for all outcomes are visually represented through boxplots in Figure 2. No significant interaction effects between Time and Group were observed. A main effect was observed for school DEIS status (Model 1c). Looking to the estimated means suggests those in non-DEIS schools reported higher ESA than those in DEIS schools.
Linear mixed models for ESA.
ESA: emotional self-awareness; DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request. School year group was included as a fixed effect for all models.
Significant at the 0.05 level.
Clustered boxplots visually illustrating comparisons of key outcomes over time.
Secondary outcomes
Functional coping
Omnibus F values and estimated marginal means are presented in Table 4. No significant interaction effects between Time and Group emerged across models. Statistically significant time effects were observed for Models 2a and 2b and for the model in which school DEIS status was included as a covariate (Model 2c). Looking to the estimated means indicates functional coping appears to be higher at Time 3 than at Times 1 and 2. A main effect for DEIS status was also evident, with estimated marginal means suggesting those in non-DEIS schools reported higher levels of functional coping than those in DEIS schools.
Linear mixed models for functional coping.
DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request.
Significant at the 0.05 level.
Dysfunctional coping
Omnibus F values and estimated marginal means are presented in Table 5. As school DEIS status, school year group and mother’s education were associated with missingness on the CSI Dysfunctional Coping Subscale; these variables were included as fixed effects for all models for this outcome. No significant effects were observed for Models 3a–3c. In Model 3d, a significant interaction effect emerged between time, group assignment and school gender mix. Looking to the estimated means indicated that in the intervention group, co-educational schools were higher in dysfunctional coping at Time 1 and subsequently declined in dysfunctional coping, whereas single-sex schools were lower at Time 1 and subsequently increased. Both types of school displayed similar levels of dysfunctional coping at Time 3. In the control group, single-sex schools had higher levels of dysfunctional coping to begin with, which subsequently declined and co-educational schools had lower levels of dysfunctional coping to begin with, which subsequently increased. At Time 3, both types of school had similar levels of dysfunctional coping.
Linear mixed models for dysfunctional coping.
DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request.
Significant at the 0.05 level.
Emotional distress
Omnibus F values and estimated marginal means are presented in Table 6. No statistically significant effects were observed across models for emotional distress.
Linear mixed models for emotional distress.
DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request.
Well-being
Omnibus F values and estimated marginal means are presented in Table 7. No significant interaction effects between Time and Group were observed across models. A significant time effect was observed in the PP model, whereby well-being increased over time.
Linear mixed models for well-being.
DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request.
Significant at the 0.05 level.
Help-seeking
Omnibus F values and estimated marginal means are presented in Table 8. As school year group was associated with missingness on the GHSQ, it was included as a fixed effect for all models. No significant interaction effects between Time and Group were observed. In Model 3d, a significant interaction effect emerged between time and school gender mix. Looking to the estimated means indicates those in single-sex schools showed small decreases in scores from Time 1 to Time 2 before showing an increase again at Time 3. Conversely, those in co-educational schools showed an increase in scores from Time 1 to Time 2 and a decrease again at Time 3.
Linear mixed models for help-seeking intentions.
DEIS: delivering equality of opportunity in schools; T1: Time 1; T2: Time 2; T3: Time 3; PP: per protocol; Coed: co-educational.
Fixed and random effect estimates for the analytic models are not shown but are available upon request.
Significant at the 0.05 level.
Discussion
The aim of this study was to evaluate the effectiveness of CopeSmart, a mental health self-monitoring mobile app, using a cluster RCT methodology. ESA and use of functional coping strategies were linked to lower distress and higher well-being, and use of dysfunctional coping strategies was linked to higher distress and lower well-being, suggesting that the mechanisms targeted by the intervention were valid, that is, that ESA and promotion of positive coping strategies were appropriate constructs to target in order to promote positive mental health in young people.
The primary trial hypothesis, that intervention group participants would report increases in ESA over time in comparison to control group participants, was not supported. Similarly, the secondary trial hypotheses that intervention group participants would report increases in the use of functional coping strategies, well-being and willingness to seek help over time and decreases in the use of dysfunctional coping strategies and emotional distress over time in comparison to controls were not supported. Thus, findings indicate the app-based intervention was not effective in improving mental health outcome among young people.
These findings are in contrast to those of Kauer et al.’s 21 RCT of the Mobiletype app with adolescents. However, participants in the Mobiletype study were attending GP services; thus, these young people had already chosen to actively seek help, as opposed to this study where they were recruited through schools and may have participated simply because everyone else did. Thus, Kauer et al.’s participants may have been more invested in the intervention and willing to engage than those in this study. Furthermore, the participants in Kauer et al.’s study also received treatment as usual from their GPs, which may have been a participating factor in the positive outcome obtained from the trial.
A number of possible reasons for the present findings can be considered. First, the low level of app engagement recorded suggests that students may not have interacted enough with the intervention to elicit change. However, correlations between app engagement and change scores from pre- to post-test suggest any changes that did occur were not linked to intervention adherence. Another explanation is that the target population may not have had high enough levels of distress to begin with in order for a mental health intervention to have utility in eliciting improvements for them.
Across the sample in this study, improvements in functional coping over time were observed and improvements in well-being were observed in the PP model for this outcome. It may be that being involved in a study focusing on coping strategies caused adolescents to think about and improve the way they were coping or they may naturally have developed more positive coping strategies over time as part of their personal development throughout the school year, which translated to more positive well-being.
While statistically significant three-way interactions between school gender mix, group assignment and time were identified for dysfunctional coping, looking to estimated means suggests these interactions were not meaningful in the context of the intervention. For example, in the intervention group, single-sex schools had lower dysfunctional coping at Time 1 and then increased, whereas co-educational schools had higher dysfunctional coping at Time 1 and then decreased, while the means across all groups emerged as similar at follow-up. Given the intervention overall was not effective, it is unlikely that the intervention is impacting on this interaction, suggesting these effects may be spurious or influenced by other external mental health covariates not accounted for (such as peer relationships or experiencing bullying for example). Similarly, the statistically significant interaction between time and school gender identified for help-seeking does not appear to be meaningful, with only marginal differences in estimated means between time points.
In terms of covariates, those in non-DEIS schools reported higher ESA and functional coping than those in DEIS schools. This is in line with the literature, which suggests that those from lower socioeconomic backgrounds have less coping resources available to them and thus are less equipped to engage in functional coping strategies.85–87
Strengths and limitations
To the authors’ knowledge, this is the first cluster RCT to examine the effectiveness of a self-monitoring mobile app-based intervention in a general sample of adolescents. Key strengths include a stratified sampling frame, the use of random allocation to assign schools to treatment arms, the use of both PP and ITT analyses, a detailed consideration of missing data and the use of a linear mixed model to account for possible clustering effects and test the effect of potential confounding variables on outcomes.
The lack of control the researchers had over how frequently the participants used the app and the low levels of engagement subsequently recorded were key limitations of this study that should be acknowledged. Two other key limitations of this study were that allocation concealment and blinding were not employed. It is possible that participants’ knowledge of their treatment group may have influenced their responses, and it is also possible that knowledge of the treatment groups may have influenced the researchers’ approach to interacting with participants and analysing the data. However, inadequate allocation concealment and lack of blinding are generally linked to an exaggeration of intervention effectiveness;88,89 as this study did not find effectiveness for the intervention, it is unlikely to have been affected by bias in this way.
Conclusion and implications
Although this study did not find efficacy for a stand-alone mental health mobile app in a general sample of adolescents, these technologies may have utility in other mental health contexts. Future research should consider exploring the efficacy of such tools with more targeted samples who experience higher level of emotional distress. For example, if used as supports for therapeutic interventions, they can provide real-time summaries of client data to therapists which may lead to improved therapeutic outcomes. 3 In light of the low levels of intervention adherence in this study, future studies should also consider ways of maximising participants’ engagement with these types of interventions. For example, engagement may be encouraged by providing personalised feedback from a mental health professional as was employed in Kauer et al.’s 21 Mobiletype evaluation, to ensure participants interact with the intervention enough to elicit improvements in outcome measures.
It must also be acknowledged that in spite of their intuitive attractiveness, it is possible that mobile apps are not an appropriate platform for providing mental health information to adolescents and other platforms, such as websites, may be more appropriate. While this is possible, further research is necessary before any definitive conclusions can be drawn.
While the results suggest a 4-week mobile app intervention may not be enough to elicit changes in outcomes in a general adolescent population, findings should not discourage researchers from adopting new technologies in mental health as it is undeniable such technologies open up a world of opportunity for improving access to care. Instead, these findings should serve as a caution for researchers; as Olff 13 points out, it is easy to be seduced by exciting, state-of-the art technologies; however, at the end of the day, the young person has to actually want to engage with the end product. Going forward, we must bring together the training and expertise of professionals with the voice and insights of adolescents, in order to develop innovative and effective ways to utilise technology to create the best possible well-being outcomes for young people.
Footnotes
Acknowledgements
The author would like to thank Serena Carberry, Caoimhe Doyle, Shane Mulligan and Nicola Mitchell for assistance with data collection and Conall Tunney for his assistance with inter-rater reliability analysis. This trial is registered at ClinicalTrials.gov (identifier: NCT02265978).
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported by the Irish Research Council (Scholarship ID Number: GOIPG/2013/450).
