Sage Journals: Discover world-class research

Abstract

Background:

Despite the growing popularity of mobile app interventions, specific engagement components of mobile apps have not been well studied.

Methods:

The objectives of this scoping review are to determine which components of mobile health intervention apps encouraged or hindered engagement, and examine how studies measured engagement.

Results:

A PubMed search on March 5, 2020 yielded 239 articles that featured the terms engagement, mobile app/mobile health, and adult. After applying exclusion criteria, only 54 studies were included in the final analysis.

Discussion:

Common app components associated with increased engagement included: personalized content/feedback, data visualization, reminders/push notifications, educational information/material, logging/self-monitoring functions, and goal-setting features. On the other hand, social media integration, social forums, poor app navigation, and technical difficulties appeared to contribute to lower engagement rates or decreased usage. Notably, the review revealed a great variability in how engagement with mobile health apps is measured due to lack of established processes.

Conclusion:

There is a critical need for controlled studies to provide guidelines and standards to help facilitate engagement and its measurement in research and clinical trial work using mobile health intervention apps.

Introduction

Enhancing participant engagement is considered a key priority for wellness and health care, especially as health care undergoes a shift toward the integration of digital technologies (e.g., mobile apps, health care monitors, and online portals with their consumer interfaces).^1,2 Technological systems play a critical role in enhancing participant engagement.^1,2 Among urban and low-income mothers, the use of smart-device technology for communication was a particularly important contributor to higher retention in longitudinal studies.³ Providing digital health tools has not only led to an increase in study participation adherence rates,⁴ but it has also contributed to measurable improvements in health care outcomes across several conditions. For instance, greater patient activation in their health care improved patient adherence to treatment prescriptions.⁵ Participants' use of web portals to augment treatment of diabetes demonstrated improved glycemic control across multiple studies.^6
–8 Other studies have seen improvements in participants with HIV,⁹ with coronary artery disease,¹⁰ and with depression,^11
–13 highlighting how impactful the implementation of these tools can be across different clinical populations.

Schoeppe et al.¹⁴ emphasized common strategies that successful mobile interventions often use, such as goal setting, self-monitoring, and performance feedback in their app design. To our knowledge, however, there has not been a scoping review of the specific components of mobile intervention apps that increase engagement. Common across all digital health tools are the focus on increased patient engagement and “empowerment,” which is a result of several qualities inherent in these tools. Most of these technological systems improve patients' communication with and access to health care providers,^1,2,15 and provide patients with more comprehensive information about their health on demand.^2,15 While these qualities are common across successful tools and play a large part in improving patient self-management and decreasing stress,² improved engagement is no guarantee.

Furthermore, measuring engagement is a challenge that has likely contributed to our lack of knowledge on app components that effectively increase this important metric. There are now several measures that quantify the amount of engagement that patients feel toward the digital tools and apps that are being developed,^2,15 but these are not widely used and engagement measurements are not standardized across studies. Some examples of such measures are the Patient Activation Measure (PAM¹⁶), Mobile App Rating Scale (MARS¹⁷), and the Patient Health Engagement scale (PHE-s¹⁸). These measures create a quantifiable standardized method by which researchers can measure the phenomenon of user engagement during program development, and are important considerations when creating new digital tools for patients and clinical research participants.

In an effort to support the shift toward mobile interventions and the benefits of using mobile apps, this review article aims to address the following questions:

What are the components or elements of mobile interventions that successfully increase participant engagement, and those that may hinder engagement?

How do studies measure engagement?

By addressing these questions, we can inform how future work may be able to standardize this effort with apps or app features.

Methods

A PubMed search with the following criteria was conducted on March 5, 2020: (engagement[Title/Abstract]) AND (mobile app) OR (mobile health) AND (adult). To be included in the analysis, articles must have recruited participants who used a mobile intervention, and articles must have examined the usage of specific aspect(s) or component(s) of the mobile app, whether through measurable app metrics, through participant feedback, or through author conjecture. The participant population must have also consisted of patients or of individuals seeking treatment for a condition; articles examining health care providers, administrators, or employees as the participant population were not included in the analysis. Ineligible articles included articles that did not have participants use a mobile app intervention (i.e., design/protocol/methods-only articles were not included), did not provide insight into which particular feature(s) were engaging for participants, were not written in English, and/or were duplicates. This review article followed PRISMA guidelines for scoping review articles. There is no review protocol for this article. See Figure 1 for the PRISMA-based flow chart.

Fig. 1.

PRISMA flow chart of articles included in this review. From: Moher et al.⁸⁰

Before the original PubMed search, the process for data collection and analysis was agreed upon by the study team so that each article was screened for the same information and that data were collected the same way. Inclusion and exclusion criteria were also standardized as described above, and all reasons for exclusion were recorded specific to each article that was excluded from analysis. Data were extracted from reading each of the published articles' manuscripts. Information on the article's country of origin was collected based on the countries listed for each author. We also classified whether the article's outcome measure(s) for engagement were test based or opinion based: test-based outcome measures were measures that examined app usage in a measurable way (i.e., had outcomes that were quantifiable and based on measurements rather than on participant feedback) and opinion-based outcome measures were measures that did not have a quantifiable outcome (i.e., had outcomes that were based on subjectivity, participants' opinions, or qualitative measures). Additional information collected on each article included whether the study was randomized or not (as a binary yes/no), the study's participant sample size, and the length of app usage in the study. Length of app usage was converted into weeks of usage to standardize reporting. For instance, we converted articles that had reported usage time in months into usage time in weeks by dividing the number of months by 12 and then multiplying that number by 52 (the number of weeks in a year). We converted articles that reported usage time in days into usage time in weeks by dividing the number of days by 7.

To examine the features that were determined as “engaging” for participants and the methods by which each article examined engagement, we collected details on engagement measurements, retention rates, clinical changes, and the specific components identified as “engaging” or “not engaging.” We categorized the article's methods for measuring engagement into one of the following categories: did not measure, based on participant self-report (meaning no measurements were taken by the study team), app usage logs, log-in frequency, survey/lesson completion rate, number/length of app sessions, use of MARS¹⁷, use of Systems Usability Scale (SUS¹⁹), or “Other.” The study's retention rates were reported (as a percent) based on the article's reported retention/attrition rate. If it was presented in another format or missing, we used the enrollment numbers to calculate the overall retention rate as a percentage, with the number of participants who were reportedly enrolled to use the mobile app treated as the total number enrolled, or the denominator, and the number of participants who were still enrolled at the last time point in the study was treated as the numerator. We also recorded whether the study reported or found any clinical changes by classifying it into one of the following categories: not reported, no differences, N/A, majority of participants self-reported the app as helpful/useful, trend, or yes, if differences were found. And lastly, we listed out the important app components that were most used by participants in the study and/or were associated with an increase in app engagement.

There was inconsistent reporting on studies' potential biases as not all articles assessed were randomized trials; however, as described above, the outcome measures used, whether or not the studies were randomized, and the methods of measuring engagement were all collected to inform the quality of each article's results.

Results

In total, 236 articles resulted from the search criteria. After removing articles based on exclusion criteria (Fig. 1), 54 articles were assessed for this review. Table 1 provides details regarding each study and its characteristics.

Table 1.

Summary List of Articles Included in the Review and Their Characteristics

AUTHORS	DISEASE INDICATION OR HEALTH REALM	MEASURABLE OR OB ENGAGEMENT METRICS?	RANDOMIZED?	SAMPLE SIZE	LENGTH OF USAGE (IN WEEKS)	STUDY RETENTION, %	CLINICAL CHANGES?	MEASURED ENGAGEMENT?
Puddephatt et al.²⁰	Alcohol/substance abuse	OB	No	29	4	N/A	Majority reported app was helpful/useful	No
Westergaard et al.²¹	Alcohol/substance abuse	OB	No	19	39	78.9	Not reported	Based on self-report
Bergman et al.²²	Alcohol/substance abuse	OB	No	123	Variable	N/A	Majority reported app was helpful/useful	Log-in frequency
Davis et al.²³	Asthma	OB	No	20	One session	N/A	Not reported	Adapted usability scale²⁴
Cohen et al.²⁵	Breast cancer prevention	M	No	86	91	79.07	Not reported	Number/length of app sessions
Michaelides et al.²⁶	Diabetes management	M	No	43	24	64.46	Yes	Survey/lesson completion rate
Park et al.²⁷	Diabetes management	OB	No	28	21.67	84.8	Majority reported app was helpful/useful	No
Conway et al.²⁸	Diabetes management	Both	No	234	One session	59	Not reported	No
Koot et al.²⁹	Diabetes management	Both	No	100	24	80	Yes	App usage logs
Kato-Lin et al.³⁰	Diet	OB	Yes	375	17.33	64	Trend	No
Kerr et al.³¹	Diet	M	Yes	247	26	89	No differences	No
Graetz et al.³²	Electronic health records access	M	No	18,529	52	N/A	Not reported	Log-in frequency
Lee et al.³³	General health monitoring	M	No	1,439	78	54	Not reported	App usage logs
Harzand et al.³⁴	Heart disease	Both	No	21	12	72.22	Yes	Messages sent
Dillingham et al.³⁵	HIV	M	No	77	52	40	Yes	App usage logs
Cho et al.³⁶	HIV	OB	Yes	38	13	N/A	Not reported	No
Toro-Ramos et al.³⁷	Hypertension	M	No	50	24	80	Yes	Weight/blood pressure loggings
Fuller-Tyszkiewicz et al.³⁸	Mental health	OB	No	15	12	N/A	Not reported	Based on self-report
Pratap et al.¹³	Mental health	M	Yes	348	12	14	Yes	Survey/lesson completion rate
Lehto et al.³⁹	Mental health	OB	No	11	4	N/A	Not reported	No
Bauer et al.⁴⁰	Mental health	Both	No	17	8	35	Majority reported app was helpful/useful	Survey/lesson completion rate
Cheung et al.⁴¹	Mental health	M	No	1,514	16	21	Trend	Number/length of app sessions
Mohr et al.⁴²	Mental health	M	No	99	8	90.1	Yes	Number/length of app sessions
Forchuk et al.⁴³	Mental health	OB	No	394	Variable	N/A	Not reported	No
Mackintosh et al.⁴⁴	Mental health	Both	Yes	58	6	48.27	No differences	No
Glover et al.⁴⁵	Mental health	OB	No	100	12	19	Majority reported app was helpful/useful	App usage logs
Bidargaddi et al.⁴⁶	Mental health	M	Yes	1,255	12.71	Not reported	Not reported	Whether user has “charted” in the app within 24 h of a push notification
McCauley et al.⁴⁷	Neurological diseases	Both	No	28	12	N/A	Not reported	App usage logs
Greiner et al.⁴⁸	Neurological diseases	OB	No	42	6	92.8	Not reported	No
Selter et al.⁴⁹	Pain management	Both	No	93	13	38	Not reported	Interactions with daily self-reports
Druce et al.⁵⁰	Pain management	Both	No	270	4.27	91	Not reported	App usage logs
Reade et al.⁵¹	Rheumatoid arthritis	Both	No	20	8.57	68	Not reported	Number/length of app sessions
Amorim et al.⁵²	Physical activity	Both	Yes	68	26	81	Trend	Survey/lesson completion rate
Reyes et al.⁵³	Physical activity	OB	No	N/A	One session	N/A	N/A	MARS
Tong et al.⁵⁴	Physical activity	OB	No	55	26	81.82	Not reported	Based on self-report
Wang et al.⁵⁵	Physical activity	Both	Yes	67	6	91.04	Yes	Length of time participants wore wearable devices
Tong et al.⁵⁶	Physical activity	Both	No	55	26	82	Yes	App usage logs, SUS
Baretta et al.⁵⁷	Weight loss	OB	No	20	2	85	Not reported	Based on self-report
Bush et al.⁵⁸	Pregnancy	M	No	85	26	N/A	Yes	App usage logs
Soh et al.⁵⁹	Rehabilitation	Both	Yes	42	Not given	90	No differences	SUS
Pavliscsak et al.⁶⁰	Rehabilitation	OB	Yes	95	Variable	N/A	Not reported	App usage logs
Choi and Paik⁶¹	Rehabilitation	Both	Yes	24	2	Not reported	Yes	No
Hoeppner et al.⁶²	Smoking cessation	Both	No	30	3	97	Majority reported app was helpful/useful	App usage logs
Nash et al.⁶³	Smoking cessation	M	No	141,429	Variable	N/A	Not reported	Log-in frequency
Marler et al.⁶⁴	Smoking cessation	Both	No	319	Variable	39.5	Yes	App usage logs
Kim et al.⁶⁵	Weight loss	M	Yes	60	4	50	Yes	Based on self-report
Alnasser et al.⁶⁶	Weight loss	Both	No	240	17	16.667	Yes	Updates per week
Kim et al.⁶⁷	Weight loss	M	No	301	26	N/A	Yes	App usage logs
Serrano et al.⁶⁸	Weight loss	M	No	12,427,196	Variable	N/A	Not reported	App usage logs
Svetkey et al.⁶⁹	Weight loss	M	Yes	365	104	86	No differences	Log-in frequency
Patel et al.⁷⁰	Weight loss	M	Yes	105	12	76	No differences	App usage logs
Dolan et al.⁷¹	Weight loss	Both	No	10	4.28	90	Not reported	Survey/lesson completion rate
Partridge et al.⁷²	Weight loss	Both	No	200	26	81	Not reported	Based on self-report
Morrison et al.⁷³	Weight loss	Both	No	13	4	100	Yes	App usage logs

M, measurable usage metrics; MARS, Mobile App Rating Scale; OB, opinion-based usage metrics; SUS, System Usability Scale.

More than half, 56% (n = 29) of articles, were from the USA, 7.8% (n = 4) were from the United Kingdom, 15.7% (n = 8) were from Australia, 3.9% (n = 2) from Canada, 7.8% (n = 4) from South Korea, 2% (n = 1) from Scotland, 2% (n = 1) from Singapore, and 3.9% (n = 3) involved more than one country (Australia/USA, U.K./Italy, and U.K./Saudi Arabia). There was a wide variety of health realms targeted by the mobile interventions (Table 1), including: Asthma, Rheumatoid Arthritis, Breast Cancer prevention, Heart disease, Pain management, Neurological diseases (one multiple sclerosis and one dementia), Diabetes management, Diet, Electronic health records access, General health monitoring, HIV, Hypertension, Mental health, Physical Activity, Pregnancy, Smoking cessation, Rehabilitation, Alcohol/Substance abuse, and Weight loss (involving both diet and physical activity). There was a near-even split between studies that examined the use of the app based on quantifiable measures or “test-based” (33.3%, n = 18), qualitative measures or “opinion-based” (31.5%, n = 17), and studies that used both test-based and opinion-based outcomes (35.1%, n = 19). However, only 25.9% (14/54) articles were randomized studies. The median participant sample size was 77, with a wide range from 10 participants up to 12,427,196.

Comparison Of Suggested Design Element/App Components

Table 2 outlines the app components that were associated with more participant engagement. Across health conditions, three major areas emerged that were associated with greater engagement: (1) Diaries (logging) and feedback: meal, blood pressure, medications, and weight loggings, visualization of participant's health data over time, personalized feedback based on questionnaires or from health care providers; (2) Coaching and education: goal-setting tools, health coach/provider messaging, personalized content, educational modules or lessons; and (3) Reminders: reminder tests and/or app notifications at a limited frequency. App components that hindered participation included: reminders that were too frequent, social forums or integration with social media, daily surveys (or redundant surveys), and technical problems (such as problems with Wi-Fi connection, app navigation, or problems logging in).

Table 2.

Important Elements of Applications That May Increase and Decrease Usage by Disease Indication or Health Realm

DISEASE INDICATION OR HEALTH REALM (NO. OF ARTICLES)	APP COMPONENTS ASSOCIATED WITH INCREASED ENGAGEMENT	APP COMPONENTS USED LEAST, AND/OR ASSOCIATED WITH DECREASED ENGAGEMENT
Alcohol/substance abuse (n = 3)	Personalized content²⁰	None reported
	Real-time feedback²⁰
	Text-message prompting use of app²⁰
	Reminders to take medications and attend appointments²¹
	Daily meditation prompts²²
	Live online video meetings²²
	Discussion boards²²
	Diversity of resources under the concept that this “might help engage individuals at various recovery stages (e.g., less than 1 year and greater than 1 year)”²²
Rheumatoid arthritis (n = 1)	Data visualization, particularly a 10-segment motif interface instead of a list of questions⁵¹	Mobile app drained smartphone battery⁵¹
Rheumatoid arthritis (n = 1)	Daily alerts⁵¹	Smartphone memory problems due to accelerometer's large files⁵¹
Asthma (n = 1)	Reminders²³	None reported
	Asthma resources and educational information²³
	Ability to connect with others²³
	Goal-setting tools and assistants²³
Breast cancer prevention (n = 1)	Upcoming procedure list²⁵	None reported
	Upcoming procedure detail²⁵
	Navigation²⁵
Diabetes management (n = 4)	Meal logging^26,29	Communicating with other patients²⁷
	Weight monitoring/“Weekly weigh-ins”^26,29	Social media integration²⁸
	Nutrition information of food eaten²⁷
	Blood glucose level tracking^27 –29
	Insulin logging²⁸
	Patient education²⁸
	Health coach messaging²⁹
Electronic health records access (n = 1)	Mobile public health record access³²	None reported
General health monitoring (n = 1)	Self-monitoring function (tracking/recording health information)³³	Medication function³³
	Access to electronic medical record information from chart³³
	Outpatient support service to make reservations³³
Heart disease (n = 1)	Reminders³⁴	None reported
	Goal-setting³⁴
	Electronic health diary³⁴
	Secure app messaging with a coach³⁴
	Educational modules³⁴
HIV (n = 2)	Customizable push-notification medication reminders³⁶	Flashing lights and beeping of electronic pill bottle³⁶
	Discreteness of electronic pill bottle³⁶
	Blood pressure logging³⁴
	Weight logging³⁴
	Education modules³⁴
	Health-related messages to coach³⁴
Hypertension (n = 1)	Weigh-ins³⁷	None reported
	Meal logging³⁷
	Educational articles³⁷
	Targeted text messages³⁷
Mental health (n = 10)	Personalization of app content^38 –40	Videogame-inspired cognitive intervention¹³
	Push notifications at 12:30 pm any day, or at 7:30 pm on weekends (vs. other times of the day)⁴⁶
	Tailored health message notifications (vs. standard push notifications) are associated with a small increase in likelihood to engage with an app within 24 h⁴⁶
	Graphical representation of mood states over time³⁸	“Some participants appreciated the badges and reinforcements they received when they completed their check-in surveys, whereas others felt patronized by the motivational language”⁴⁰
	Prompts to use app³⁸
	Internet-based problem-solving therapy¹³
	Daily health tips^13,45
	Daily surveys^40,45
	Alerts in response to daily surveys⁴⁰
	Direct visualization of their own data⁴⁰
	Using a hub recommender app⁴¹
	Expectation to swap apps in and out of use rotation⁴²
	Low-intensity coaching⁴²
	Skills training through brief app sessions⁴²
	Appointment reminders⁴³
	Tracking functions⁴³
	Anger frequency/intensity/cues logging⁴⁴
	Behavioral strategies suggestions⁴⁴
	Individually tailored anger management plan prompts⁴⁴
Neurological diseases (n = 2)	Photo multimedia content for individuals with dementia⁴⁷	Video content for individuals with dementia⁴⁷
	Personal media for individuals with dementia⁴⁷
	“Reminiscing screens” for individuals with dementia⁴⁷
	Sharing information with doctor⁴⁸
Pain management (n = 2)	Daily exercise notifications⁴⁹	Event marker button was difficult for those with dexterity and hand function issues⁵⁰
	“uMotif” interface design⁵⁰
	Passive data collection⁵⁰
	Personalizing time reminders are sent⁵⁰
	End-of-study report that detailed their sleep, average pain, fatigue, and wellbeing scores⁵⁰
	Study support (for app problems, etc.)⁵⁰
Physical activity (n = 5)	Using FiMit^52,55,56	Weekly surveys⁵²
	Health coaching⁵²	3+ text message reminders per day were too frequent⁵⁵
	Gamification and score sharing through social media⁵³	Social forum and private messages⁵⁶
	Customization features (changing color components of the app, score sharing options, smartphone vibration, etc.)⁵³
	Self-monitoring of behavior⁵⁴
	Goal setting⁵⁴
	Feedback on behavior⁵⁴
	Social comparison⁵⁴
	Similarity and familiarity between users⁵⁴
	Participation from other users in the network⁵⁴
	Automation and personalization⁵⁴
Pregnancy (n = 1)	Health milestones⁵⁸	None reported
Pregnancy (n = 1)	Personalized “What's Happening this Week” screen⁵⁸	None reported
Rehabilitation (n = 3)	Exercise tracking⁵⁹	Wi-Fi connection issues⁵⁹
	Peer patients' information⁵⁹
	Receiving feedback from questionnaires⁶⁰
	Immediate feedback from the patient's movement⁶¹
Smoking cessation (n = 3)	“Happiness exercises”⁶²	Social support app functionality⁶²
	“Interactive Tobacco Tracker”⁶³	No human intervention contact⁶⁹
	Cost savings calculator⁶³
	Quitting plan behaviors⁶³
	Taking daily breath samples⁶⁴
	Cigarette logging⁶⁴
	Wearable usage⁶⁵
Diet and weight loss (n = 12)	Feedback about calorie intake and consumption^57,72,73	Redundant surveys⁷¹
	Calorie counter⁶⁶	Peer support³⁰
	Step counter⁶⁶	Mobile-based visual diary³⁰
	Contributing posts on a group⁶⁷	Difficulty logging in⁷²
	Information/reading articles^67,73	A hard to navigate app⁷²
	Customized recipes⁶⁸
	Diet, fluid, and/or protein tracking⁷⁰
	Push notification reminders (i.e., of goals, to drink and walk frequently)^71,73
	Nutritional information⁷¹
	Image-based dietician support³⁰
	Personalized text messaging^31,72
	Personal data entry (i.e., fluid intake logging)^71 –73
	Goal-setting⁷³
	Food lists⁷³

Clinical Outcomes Of Assessed Apps

Only 55% (30 out of 54) articles included in the analysis examined differences in clinical outcomes due to the mobile intervention, of which 53% (n = 16) found significant differences, 10% (n = 3) showed a positive trend due to the intervention, 20% (n = 6) had a majority of participants report the app as helpful and/or useful, and 17% (n = 5) found no impact.

Measuring Engagement

About a fifth of articles (20%; n = 11) did not use any measurement method and drew conclusions about engaging features based on conjecture or opinions from the study team (rather than from collected study data), whereas 11% (n = 6) of articles used participant self-report of usage to measure engagement with the app. Of the remaining 37 articles that did measure engagement in a quantifiable way, 41% (n = 15) looked at app usage logs, 14% (n = 5) relied on survey/lesson completion rates, 11% (n = 4) relied on log-in frequency, 11% (n = 4) examined the number and/or length of app sessions, and only 8% (n = 3) used an established scale to measure engagement (n = 1 used the MARS and n = 2 used the SUS). Nineteen percent (n = 7) of these 37 articles used another way to measure engagement, which included: participant updates per week, the length of time participants wore wearable devices, interactions with daily self-reports, weight/blood pressure loggings, whether or not participants “charted” in the app within 24 h of a push notification, and an “adapted usability scale.”²⁴

Length Of App Usage

The review analysis showed that one article did not report the length of app usage, three were completed in one-time sessions, and six had not set a specific period of time for the intervention (participants used it a variable amount of time and were neither told to start or stop using the app). Of the remaining 44 articles, the usage duration ranged from 2 to 104 weeks, and the average length of usage was 21 weeks (SD = 22.7 weeks, median = 12.8 weeks, mode = 26 weeks). Many articles also reported a drop-off in usage after the first week of the study,^29,41,62,73 and/or reported an initially high level of participation that slowly diminished over the course of the study.^{29,32,40
–42,56,62,73}

Study Retention Rates

Retention rates were not applicable or not reported in 31% (n = 17) of the evaluated studies. In the remaining studies, the retention rate ranged from 14% to 100%, with an average 68% (SD 25%) and a median of 79%.

Discussion

The use of mobile apps in health care is gaining ground, however, research geared toward understanding patient/app interactions (engagement) is still nascent. This is evident by the small number (236) of articles matching the broad keywords used in the initial literature search. The PRISMA process then reduced this number to only 54 articles (∼23%) eligible for engagement analysis.

There was little commonality in how studies measured participant engagement, and a wide range in the length of app usage in these studies. Most studies, depending on the app utility objectives, used longitudinal or cohort-testing approaches. Several studies explored ways to improve engagement within their own app prototype; of note was Bidargaddi et al.⁴⁶ who implemented a “micro-randomized” clinical trial of push notifications to determine the effects they had on engagement with a mobile health app. This study approach was one of very few that used an experimental approach to address design choices and their effects on engagement directly. They found that sending a tailored health message at 12:30 pm on any day, or at 7:30 pm on weekends, made participants almost 9% more likely to use the app.

Mobile app developers utilize measure scales such as SUS and MARS to gain insight on the usability, functionality, and satisfaction from users. It was surprising to find that only three articles followed the MARS¹⁷ and SUS¹⁹ to evaluate engagement outcome measures, and none used the PAM¹⁶ or PHE-s.⁵ For the rest of the studies, excluding the ones that did not directly measure engagement (n = 11), and those that relied on participant's report for engagement results (n = 6), the most common quantitative approach to measure engagement was through app usage logs (n = 15) from which the study teams were able to quantify minutes of app usage, the frequency of log-ins, and the frequency of app components that were used.

A host of app components affected engagement level, but the most engaging mobile interventions provided participants with the ability to view and/or interact with their health data. This aligned with other findings that the return of information to participants is an especially important aspect of creating engaging and impactful digital tools,^74,75 as doing so increases participant self-efficacy by involving individuals in making their own well-informed health care decisions.⁷⁶ Moreover, providing results and information tailored to an individual's gender, needs, characteristics, and interests has been shown to both produce more online activity and increase retention on follow-up surveys.⁷⁷ The findings from this review support these statements.

It appears that the first week marks an important time point in continuing usage of the app, as many studies reported the largest drop-off in usage after the first week of the study, regardless of study duration (ranged from 3 to 26 weeks). Through app usage logs, researchers were able to track app engagement and assess drop-off in participation (Table 1). This pattern was not reported in studies that only relied on subjective measures of engagement, demonstrating that measurable methods to track app engagement provide a fuller picture of app usage. The analysis suggests that app engagement plays an important role in study retention. Once past the first week, retention rate was very high averaging around 68% among participants who stayed on the study. Study duration, disease indication, study duration, and participant incentives affected retention rate that ranged widely from 46%⁷⁸ to 86%.⁷⁹ The retention rates of most of the examined mobile interventions fall well within the expected range of longitudinal studies, with only n = 12 studies (22.2%) examined reporting retention rates lower than 60%. Of the n = 11 studies (20.4%) that reported retention rates of 85% or greater, the apps featured the following components that were popular among participants: reminders/push notifications (especially personally tailored messages), ability to communicate with doctors and/or care teams through messaging features, self-reporting or “logging” of symptoms, and easy access to health information.

Incorporating digital tools and mobile apps in the management plan of patients' health is rather a new approach in health care. Research to generate evidence for the utility and efficacy of these tools in health care intervention and management is still nascent. We recognize that this analysis bears several limitations due to (1) the small number of published studies (articles) addressing engagement to be included in analysis, (2) many studies' lack of processes around measurement of engagement, and (3) hence, great variability in results. This review may not be able to accurately capture the true app components necessary to increase engagement with apps due to articles' lack of robust methods for measuring engagement, and due to the limited number of studies that can be properly compared by underlying health conditions.

Conclusion

In conclusion, despite the growing popularity of mobile app interventions, the specific engaging components have not been well defined. This article provides insight into existing methods and tools that may encourage participant engagement with future mobile app interventions in various disease indications. The information provided in this study is intended to help clinical teams, researchers, and clinical trialists design better mobile applications that will provide participants and patients with greater satisfaction and ultimately better outcomes. Future work is needed to develop common guidelines to support specific components or activities that lead to increased engagement with mobile health applications. There is also a tremendous need for cross-disciplinary agreement on standards for engagement measurement to provide greater generalizability. These two elements could help minimize app design and user testing periods and lead to greater success with mobile health interventions in the future.

Footnotes

Authors' Contributions

This article was written and edited by J.S.O., I.O.G., R.Y., and M.L. J.S.O. and I.O.G. wrote the initial draft of the article. R.Y. and M.L. provided substantial edits to the article and approved of the final version to be published. I.O.G. determined the scope of the review article and methods. J.S.O. performed the initial search and article screening.

Disclosure Statement

No competing financial interests exist.

Funding Information

While not directly supported with funding, the idea for this article arose from work funded by the National Cancer Institute of the National Institutes of Health under contract numbers HHSN261201700030C and HHSN261201800010C. Ingrid Oakley-Girvan, Reem Yunis, and Michelle Longmire serve as coinvestigators for the former and Ingrid Oakley-Girvan is the PI for the latter. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References

Barello

, Triberti

, Graffigna

, et al. eHealth for patient engagement: A systematic review. Front Psychol, 2016; 6:2013.

Dendere

, Slade

, Burton-Jones

, Sullivan

, Staib

, Janda

. Patient portals facilitating engagement with inpatient electronic medical records: A systematic review. J Med Internet Res, 2019; 21:e12779.

Beering

, Sun

, Cummings

, et al. Factors affecting retention and compliance in a longitudinal study of connected, low income, urban, primiparous mothers. Available at https://scholarworks.iupui.edu/handle/1805/9682

Dubad

, Winsper

, Meyer

, Livanou

, Marwaha

. A systematic review of the psychometric properties, usability and clinical impacts of mobile mood-monitoring applications in young people. Psychol Med, 2018; 48:208–228.

Graffigna

, Barello

, Bonanomi

. The role of Patient Health Engagement Model (PHE-model) in affecting patient activation and medication adherence: A structural equation model. PLoS One, 2017; 12:e1079865.

Lau

, Campbell

, Tang

, Thompson

, Elliott

. Impact of patient use of an online patient portal on diabetes outcomes. Can J Diabetes, 2014; 38:17–21.

Osborn

, Mayberry

, Mulvaney

, Hess

. Patient web portals to improve diabetes outcomes: A systematic review. Curr Diab Rep, 2010; 10:422–435.

Tang

, Overhage

, Chan

, et al. Online disease management of diabetes: Engaging and motivating patients online with enhanced resources-diabetes (EMPOWER-D), a randomized controlled trial. J Am Med Informatics Assoc, 2013; 20:526–534.

Crouch

PCB

, Rose

, Johnson

, Janson

. A pilot study to evaluate the magnitude of association of the use of electronic personal health records with patient activation and empowerment in HIV-infected veterans. PeerJ, 2015; 3:e852.

10.

Toscos

, Daley

, Heral

, et al. Impact of electronic personal health record use on engagement and intermediate health outcomes among cardiac patients: A quasi-experimental study. J Am Med Inform Assoc, 2016; 23:119–128.

11.

Anguera

, Jordan

, Castaneda

, Gazzaley

, Areán

. Conducting a fully mobile and randomised clinical trial for depression: Access, engagement and expense. BMJ Innov, 2016; 2:14–21.

12.

Arean

, Hallgren

, Jordan

, et al. The use and effectiveness of mobile apps for depression: Results from a fully remote clinical trial. J Med Internet Res, 2016; 18:e330.

13.

Pratap

, Renn

, Volponi

, et al. Using mobile apps to assess and treat depression in Hispanic and Latino populations: Fully remote randomized clinical trial. J Med Internet Res, 2018; 20:e10130.

14.

Schoeppe

, Alley

, Van Lippevelde

, Bray

, Williams

, Duncan

, Vandelanotte

. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: A systematic review. Int J Behav Nutr Phys Act, 2016; 13:127.

15.

Triberti

, Barello

. The quest for engaging AmI: Patient engagement and experience design tools to promote effective assisted living. J Biomed Inform, 2016; 63:150–156.

16.

Hibbard

, Stockard

, Mahoney

, Tusler

. Development of the Patient Activation Measure (PAM): Conceptualizing and measuring activation in patients and consumers. Health Serv Res, 2004; 39:1005–1026.

17.

Stoyanov

, Hides

, Kavanagh

, Zelenko

, Tjondronegoro

, Mani

. Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth, 2015; 3:e27.

18.

Graffigna

, Barello

, Bonanomi

, Lozza

. Measuring patient engagement: Development and psychometric properties of the Patient Health Engagement (PHE) scale. Front Psychol, 2015; 6:274.

19.

Lewis

. The system usability scale: Past, present, and future. Int J Hum Comp Interact, 2018; 34:577–590.

20.

Puddephatt

J-A

, Leightley

, Palmer

, et al. A qualitative evaluation of the acceptability of a tailored smartphone alcohol intervention for a military population: Information About Drinking for Ex-Serving Personnel (InDEx) app. JMIR Mhealth Uhealth, 2019; 7:e12267.

21.

Westergaard

, Genz

, Panico

, et al. Acceptability of a mobile health intervention to enhance HIV care coordination for patients with substance use disorders. Addict Sci Clin Pract, 2017; 12:11.

22.

Bergman

, Kelly

, Hoeppner

, Vilsaint

, Kelly

. Digital recovery management: Characterizing recovery-specific social network site participation and perceived benefit. Psychol Addict Behav, 2017; 31:506–512.

23.

Davis

, Peters

, Calvo

, Sawyer

, Foster

, Smith

. “Kiss myAsthma”: Using a participatory design approach to develop a self-management app with young people with asthma. J Asthma, 2018; 55:1018–1027.

24.

Lewis

. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int J Hum Comp Interact, 1995; 7:57–58.

25.

Cohen

, Scherr

, Nixon

. An iPhone application intervention to promote surveillance among women with a BRCA mutation: Pre-intervention data. J Genet Couns, 2018; 27:446–456.

26.

Michaelides

, Raby

, Wood

, Farr

, Toro-Ramos

. Weight loss efficacy of a novel mobile Diabetes Prevention Program delivery platform with human coaching. BMJ Open Diabetes Res Care, 2016; 4:e000264.

27.

Park

, Burford

, Nolan

, Hanlen

. The role of digital engagement in the self-management of type 2 diabetes. Health Commun, 2016; 31:1557–1565.

28.

Conway

, Campbell

, Forbes

, Cunningham

, Wake

. mHealth applications for diabetes: User preference and implications for app development. Health Informatics J, 2016; 22:1111–1120.

29.

Koot

, Goh

PSC

, Lim

RSM

, et al. A mobile lifestyle management program (GlycoLeap) for people with type 2 diabetes: Single-arm feasibility study. JMIR Mhealth Uhealth, 2019; 7:e12965.

30.

Kato-Lin

, Padman

, Downs

, Abhishek

. Evaluating consumer m-health services for promoting healthy eating: A randomized field experiment. AMIA Annu Symp Proc, 2015; 2015:1947–1956.

31.

Kerr

, Harray

, Pollard

, et al. The connecting health and technology study: A 6-month randomized controlled trial to improve nutrition behaviours using a mobile food record and text messaging support in young adults. Int J Behav Nutr Phys Act, 2016; 13:52.

32.

Graetz

, Huang

, Brand

, Hsu

, Reed

. Mobile-accessible personal health records increase the frequency and timeliness of PHR use for patients with diabetes. J Am Med Inform Assoc, 2019; 26:50–54.

33.

Lee

, Kwon

, Lee

, et al. Effect of self-monitoring on long-term patient engagement with mobile health applications. PLoS One, 2018; 13:e0201166.

34.

Harzand

, Witbrodt

, Davis-Watts

, et al. Feasibility of a smartphone-enabled cardiac rehabilitation program in male veterans with previous clinical evidence of coronary heart disease. Am J Cardiol, 2018; 122:1471–1476.

35.

Dillingham

, Ingersoll

, Flickinger

, et al. PositiveLinks: A mobile health intervention for retention in HIV care and clinical outcomes with 12-month follow-up. AIDS Patient Care STDS, 2018; 32:241–250.

36.

Cho

, Flynn

, Saylor

, Gradilla

, Schnall

. Use of the FITT framework to understand patients' experiences using a real-time medication monitoring pill bottle linked to a mobile-based HIV self-management app: A qualitative study. Int J Med Inform, 2019; 131:103949.

37.

Toro-Ramos

, Kim

, Wood

, et al. Efficacy of a mobile hypertension prevention delivery platform with human coaching. J Hum Hypertens, 2017; 31:795–800.

38.

Fuller-Tyszkiewicz

, Richardson

, Klein

, et al. A mobile app–based intervention for depression: End-user and expert usability testing study. JMIR Ment Health, 2018; 5:e54.

39.

Lehto

, Heeter

, Allbritton

, Wiseman

. Hospice and palliative care provider experiences with meditation using mobile applications. Oncol Nurs Forum, 2018; 45:380–388.

40.

Bauer

, Iles-Shih

, Ghomi

, et al. Acceptability of mHealth augmentation of Collaborative Care: A mixed methods pilot study. Gen Hosp Psychiatry, 2018; 51:22–29.

41.

Cheung

, Ling

, Karr

, Weingardt

, Schueller

, Mohr

. Evaluation of a recommender app for apps for the treatment of depression and anxiety: An analysis of longitudinal user engagement. J Am Med Inform Assoc, 2018; 25:955–962.

42.

Mohr

, Tomasino

, Lattie

, et al. IntelliCare: An eclectic, skills-based app suite for the treatment of depression and anxiety. J Med Internet Res, 2017; 19:e10.

43.

Forchuk

, Reiss

, O'Regan

, Ethridge

, Donelle

, Rudnick

. Client perceptions of the mental health engagement network: A qualitative analysis of an electronic personal health record. BMC Psychiatry, 2015; 15:250.

44.

Mackintosh

M-A

, Niehaus

, Taft

, Marx

, Grubbs

, Morland

. Using a mobile application in the treatment of dysregulated anger among veterans. Mil Med, 2017; 182:e1941–e1949.

45.

Glover

, Schueller

, Winiarski

, Smith

, Karnik

, Zalta

. Automated mobile phone–based mental health resource for homeless youth: pilot study assessing feasibility and acceptability. JMIR Ment Health, 2019; 6:e15144.

46.

Bidargaddi

, Almirall

, Murphy

, et al. To prompt or not to prompt? A microrandomized trial of time-varying push notifications to increase proximal engagement with a mobile health app. JMIR Mhealth Uhealth, 2018; 6:e10123.

47.

McCauley

, Bond

, Ryan

, et al. Evaluating user engagement with a reminiscence app using cross-comparative analysis of user event logs and qualitative data. Cyberpsychol Behav Soc Netw, 2019; 22:543–551.

48.

Greiner

, Sawka

, Imison

. Patient and physician perspectives on MSdialog, an electronic PRO diary in multiple sclerosis. Patient, 2015; 8:541–550.

49.

Selter

, Tsangouri

, Ali

, et al. An mHealth app for self-management of chronic lower back pain (Limbr): Pilot study. JMIR Mhealth Uhealth, 2018; 6:e179.

50.

Druce

, Dixon

, McBeth

. Maximizing engagement in mobile health studies: Lessons learned and future directions. Rheum Dis Clin North Am, 2019; 45:159–172.

51.

Reade

, Spencer

, Sergeant

, et al. Cloudy with a chance of pain: Engagement and subsequent attrition of daily data entry in a smartphone pilot study tracking weather, disease severity, and physical activity in patients with rheumatoid arthritis. JMIR Mhealth Uhealth, 2017; 5:e37.

52.

Amorim

, Pappas

, Simic

, et al. Integrating Mobile-health, health coaching, and physical activity to reduce the burden of chronic low back pain trial (IMPACT): A pilot randomised controlled trial. BMC Musculoskelet Disord, 2019; 20:71.

53.

Reyes

, Qin

, Brown

. A standardized review of smartphone applications to promote balance for older adults. Disabil Rehabil, 2018; 40:690–696.

54.

Tong

, Coiera

, Laranjo

. Using a mobile social networking app to promote physical activity: A qualitative study of users' perspectives. J Med Internet Res, 2018; 20:e11439.

55.

Wang

, Cadmus-Bertram

, Natarajan

, et al. Wearable sensor/device (Fitbit One) and SMS text-messaging prompts to increase physical activity in overweight and obese adults: A randomized controlled trial. Telemed J E Health, 2015; 21:782–792.

56.

Tong

, Coiera

, Tong

, et al. Efficacy of a mobile social networking intervention in promoting physical activity: Quasi-experimental study. JMIR Mhealth Uhealth, 2019; 7:e12181.

57.

Baretta

, Perski

, Steca

. Exploring users' experiences of the uptake and adoption of physical activity apps: Longitudinal qualitative study. JMIR Mhealth Uhealth,, 2019; 7:e11636.

58.

Bush

, Barlow

, Echols

, Wilkerson

, Bellevin

. Impact of a mobile health application on user engagement and pregnancy outcomes among Wyoming Medicaid members. Telemed J E Health, 2017; 23:891–898.

59.

Soh

, Lee

, et al. A mobile phone–based self-monitoring tool for perioperative gastric cancer patients with incentive spirometer: Randomized controlled trial. JMIR Mhealth Uhealth, 2019; 7:e12204.

60.

Pavliscsak

, Little

, Poropatich

, et al. Assessment of patient engagement with a mobile application among service members in transition. J Am Med Inform Assoc, 2016; 23:110–118.

61.

Choi

, Paik

. Mobile game-based virtual reality program for upper extremity stroke rehabilitation. J Vis Exp, 2018; 8:56241.

62.

Hoeppner

, Hoeppner

, Carlon

, et al. Leveraging positive psychology to support smoking cessation in nondaily smokers using a smartphone app: Feasibility and acceptability study. JMIR Mhealth Uhealth, 2019; 7:e13436.

63.

Nash

, Vickerman

, Kellogg

, Zbikowski

. Utilization of a Web-based vs integrated phone/Web cessation program among 140,000 tobacco users: An evaluation across 10 free state quitlines. J Med Internet Res, 2015; 17:e36.

64.

Marler

, Fujii

, Utley

, Tesfamariam

, Galanko

, Patrick

. Initial assessment of a comprehensive digital smoking cessation program that incorporates a mobile app, breath sensor, and coaching: Cohort study. JMIR Mhealth Uhealth, 2019; 7:e12609.

65.

Kim

, Ryu

, Cho

, et al. Impact of personal health records and wearables on health outcomes and patient response: Three-arm randomized controlled trial. JMIR Mhealth Uhealth, 2019; 7:e12070.

66.

Alnasser

, Kyle

, Aloumi

, Al-Khalifa

, Marais

. The Twazon Arabic Weight Loss App: App-based intervention for Saudi women with obesity. JMIR Mhealth Uhealth, 2019; 7:e10923.

67.

Kim

, Ray

, Veluscek

. Complementary support from facilitators and peers for promoting mhealth engagement and weight loss. J Health Commun, 2017; 22:905–912.

68.

Serrano

, Coa

, Yu

, Wolff-Hughes

, Atienza

. Characterizing user engagement with health app data: A data mining approach. Transl Behav Med, 2017; 7:277–285.

69.

Svetkey

, Batch

, Lin

, et al. Cell phone intervention for you (CITY): A randomized, controlled trial of behavioral weight loss intervention for young adults using mobile technology. Obesity (Silver Spring), 2015; 23:2133–2141.

70.

Patel

, Hopkins

, Brooks

, Bennett

. Comparing self-monitoring strategies for weight loss in a smartphone app: Randomized controlled trial. JMIR Mhealth Uhealth, 2019; 7:e12209.

71.

Dolan

, Afaneh

, Dakin

, Pomp

, Yeo

. Lessons learned from developing a mobile app to assist in patient recovery after weight loss surgery. J Surg Res, 2019; 244:402–408.

72.

Partridge

, Allman-Farinelli

, McGeechan

, et al. Process evaluation of TXT2BFiT: A multi-component mHealth randomised controlled trial to prevent weight gain in young adults. Int J Behav Nutr Phys Act, 2016; 13:7.

73.

Morrison

, Hargood

, Lin

, et al. Understanding usage of a hybrid website and smartphone app for weight management: A mixed-methods study. J Med Internet Res, 2014; 16:e201.

74.

Christofides

, Stroud

, Tullis

, O'Doherty

. Improving dissemination of study results: Perspectives of individuals with cystic fibrosis. Res Ethics, 2019; 15:1–14.

75.

National Academies of

Sciences

, Engineering, and Medicine; Health and Medicine Division; Board on Health Sciences Policy; Committee on the Return of Individual-Specific Research Results Generated in Research

Laboratories

. The return of individual-specific research results from laboratories: Perspectives and ethical underpinnings. In: Downey

, Busta

, Mancher

, Botkin

, eds. Returning individual research results to participants: Guidance for a new research paradigm. USA: National Academies Press, 2018.

76.

Armstrong

, Rueda

, Gronseth

, Mullins

. Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expect, 2017; 20:3–10.

77.

Couper

, Alexander

, Maddy

, et al. Engagement and retention: Measuring breadth and depth of participant use of an online intervention. J Med Internet Res, 2010; 12:e52.

78.

Booker

, Harding

, Benzeval

. A systematic review of the effect of retention methods in population-based cohort studies. BMC Public Health, 2011; 11:249.

79.

Robinson

, Dennison

, Wayman

, Pronovost

, Needham

. Systematic review identifies number of strategies important for retaining study participants. J Clin Epidemiol, 2007; 60:757–765.

80.

Moher

, Liberati

, Tetzlaff

, Altman DG; The PRISMA

Group

. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 2009; 6:e1000097.

What Works Best to Engage Participants in Mobile App Interventions and e-Health: A Scoping Review

Abstract

Background:

Methods:

Results:

Discussion:

Conclusion:

Introduction

Methods

Results

Comparison Of Suggested Design Element/App Components

Clinical Outcomes Of Assessed Apps

Measuring Engagement

Length Of App Usage

Study Retention Rates

Discussion

Conclusion

Footnotes

Authors' Contributions

Disclosure Statement

Funding Information

References