Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach

Abstract

Background

Mobile health applications are increasingly used to support the delivery of health care services to a variety of patients. Based on data obtained from a pragmatic trial of the electronic Patient Reported Outcome (ePRO) app designed to support goal-oriented care primary care, this study aims to (1) examine how patient-reported usability changed over the one-year intervention period, and (2) explore participant attrition rate of the electronic Patient Reported Outcome app over one year study period.

Methods

We performed a secondary analysis of 44 older adults with complex chronic needs enrolled in the electronic Patient Reported Outcome-digital health intervention. App usage and attrition were measured using device-generated usage logs; usability was measured using the patient-reported post-study system usability questionnaire collected at 3, 6, 9, and 12 months. Research memos were used to interpret potential contextual contributing factors to patients’ overall usage and usability score pattern. A data triangulation method of both quantitative and qualitative data was used to analyze and interpret study findings.

Results

While there was gradual attrition in the use of the ePRO app, patients’ usability scores remained consistent throughout the study period. Qualitative memos suggested patients’ encounters with technical difficulties and relationship dynamics with primary providers influenced patients’ adherence to the ePRO app.

Conclusion

This study highlights that the patient–provider relationship is a key determining factor that influences complex patients’ continued engagement with a Mobile health app. The finding calls attention to the measurement of usability of a Mobile health app, its impact on attrition, and contributing factors that influence patients’ attrition. Trial registration: Clinicaltrials.gov Identified NCT02917954.

Keywords

Mobile applications mobile health usability attrition patients with complex chronic conditions

Background

In recent years, there has been a shift towards a more integrated and person-centered health system as a means to empower patients and their caregivers to be engaged partners in their care.^1,2 A key aspect of this partnership is to make available information and resources needed to help patients and families play this more active role.³ Furthermore, patients have higher satisfaction and utilize health care more effectively when they have guidance in managing their conditions.^4,5 Guidance and partnership may be particularly important for patients with complex care needs who may be managing multiple health and social challenges at a time.⁶ These individuals with multiple chronic conditions and biopsychosocial problems have challenges that can lead to functional decline, poor quality of life, and high healthcare utilization.⁷

A person-centered goal-oriented approach has been regarded as a preferred approach to caring for individuals with complex care needs.^5,8,9 Health authorities are looking to adopt person-centered care developing care plans that are attuned to patients’ long-term health-related goals¹⁰ and to deliver such care providers are increasingly adopting information and communication technologies (ICT) in their practices.^11,12 Among the ICT-based solutions being explored, mobile health (mHealth) apps have emerged as valuable to support the delivery of integrated and person-centered care and support patients’ health goals.^13–15 Some studies have shown mHealth solutions to be effective in developing complex patients’ self-management skills and improve their quality of life.^16,17

Several mHealth apps have successfully supported patients’ achievement of health goals related to weight management, physical activity, diet, and smoking cessation.^14,18 Despite promising results, interventions like these experience a high attrition rate¹⁹^,²⁰ Attrition in this context is defined as a phenomenon in which one initially commits to a web-based (or mHealth) intervention but subsequently discontinues using the platform.²¹ Among mHealth research studies, high rates of attrition have a substantial impact on the integrity of collected data, the amount of missing data, and biased responses. While a common challenge, there is a limited number of research studies conducted on the determinants of abandonment of web-based, or mHealth-platform²²^,²³

One potential driver of attrition is system usability.²¹ Nielson²⁴ defines usability as a quality attribute that assesses how easy it is to use an interface of a web-based system or platform. In current literature, the majority of usability studies evaluate usability based on operational ease, learnability, and understandability of apps, and this concept is often measured at the end of the intervention.²⁵ Lessons learned from previous reviews identify that combining multiple usability tools, adopting iterative measurement of usability, and going beyond the questionnaire method to test usability are needed to understand usability in a meaningful way.^25,26

This paper explores the links between usability and attrition by adopting a novel blended approach that combines data from surveys (capturing usability) and the mHealth system usage-logs (capturing attrition). In addition, the presented study draws on qualitative research memos to provide relevant contextual data regarding patient engagement with technology in a real-world and complex environment.

Research objective

This is a sub-study of a pragmatic stepped-wedge evaluation of the electronic Patient Reported Outcomes (ePRO) app, a goal-oriented mobile application for patients with complex conditions.^27–29 This sub-study aims to answer the following research questions: (1) how does patient-reported usability change over a one-year intervention period?; (2) what is the participant attrition rate of the ePRO app over a one-year study period?

Methods

Study design

The evaluation of the ePRO tool involves a pragmatic trial using a stepped-wedge randomized design⁹ with an embedded case study. The trial was conducted across Ontario, Canada in six primary care practices over 15 months. Based on a random number generator, each site was assigned to either the early intervention group (n = 3) or the late intervention group (n = 3). Sites that were assigned in the early intervention group remained in the control period for 3 months followed by a 12 month intervention period. Sites that were assigned in the late intervention group remained in the control period for 6 months, followed by a 9-month intervention period. Figure 1 shows the stepped-wedge research design.

Figure 1.

Stepped-wedge research design.

We conducted a secondary data analysis of the intervention arm of the trial data. For this sub-study, patient attrition is measured using device-generated usage logs; usability is measured using the patient-reported post-study system usability questionnaire (PSSUQ) collected once every 3 months during the intervention period. Research memos capturing patients’ and their providers’ interactions with the research team are used to contextualize the usability-related study data. The research memos contain information about patients’ or their providers’ experiencing technical difficulties, observational data from patient–provider interactions.

Description of the intervention: The ePRO app

The ePRO app is a mobile health application that supports collaborative goal setting and monitoring of outcomes for patients enrolled in interdisciplinary, primary care practice. The app was developed following multiple iterative phases via a user-centered co-design method by involving complex patients and their caregivers during the exploratory trial of the ePRO app.^5,9,27,28 The developed app during the exploratory trial was used in the pragmatic trial to assess its usability. During the 12 months of pragmatic trial, the ePRO interface and functionalities remained consistent and no further change was made. The ePRO app includes two key features. First, The My Goal Tracker feature, helps patients and providers to set specific, measurable, attainable, realistic, and time-specific (SMART) goals relevant to the patient. Patient-centered goals are set-up during an in-person consultation with a provider on the care team. Progress towards goal achievement is tracked between visits through the mobile function that monitors outcomes and symptoms. Second, The Health Journal feature can be used to track other outcomes important to patients and providers as appropriate. Outcomes include pain, mobility, mental health symptoms, and overall wellness.

During the initial pilot phases, it was found that the ePRO app increases patients’ feelings of ownership over their health, and providers were able to leverage the app as a means of focusing on goal setting and obtaining an overview of goal progress with their patients.²⁷

Study setting

The study was conducted in six primary care practices across Ontario, Canada. All six primary care practices in the trial has team-based care models called Family Health Teams (FHTs) that offer comprehensive primary care delivered by an interprofessional team.³⁰ In this care setting patients typically have a lead primary care provider, either a physician or nurse practitioner; however, patients may receive care from any provider in the team (registered nurses, a social worker, a dietitian, and a diabetes nurse educator).

Target population

Eligible patients for the trial met all of the following inclusion criteria: (1) 60 years or older; (2) have two or more diagnosed chronic conditions, and 10 or more visits to their primary health care provider within the last 12 months; (3) physical capability to use a smartphone or a caregiver who can use the tablet on their behalf; (4) ability to read and write in English or the availability of a caregiver who can do so on their behalf; and (5) be in the contemplative or ready to change behavior mode for self-management of their chronic conditions.

Measures used in this sub-study

The ePRO study represents a complex intervention that involves multiple interacting components and mechanisms that can influence outcomes (such as patients’ quality of life) as well as users’ overall experience with the app.³¹ Consistent with recommendations on how to evaluate and test complex interventions using multiple data sources,^29,32,33 the following data sources were used to answer the research question of this study: (1) Device generated data to measure actual usage; (2) The 2002 version of PSSUQ³⁴ to measure the usability of the app; (3) Research memos to interpret usage and usability of ePRO.

System usage logs to measure a monthly number of interactions with the ePRO app

The first source of data was the ePRO system-generated usage logs which we used to measure the number of interactions users had with the ePRO system. An Interaction was defined as any log-in or data entered into the ePRO system. Any patient who had zero interactions in a given month was considered as a non-active participant of ePRO, whereas patients who had at least one complete interaction in the given month were considered as active participants.

Participants were categorized by the number of months they used the ePRO app. Previous mHealth studies performed similar segmentation activities based on patients’ health conditions, mHealth usage, and decision-making styles to understand user behavior.³⁵^,³⁶ For this study, patients were categorized into three groups:

Non-users: The participants who did not use the ePRO app after their initial onboarding.

Short-term users: The participants who used the ePRO app for less than 3 months consecutively or non-consecutively.

Long-term users: Participants who used the ePRO app for 3 or more months.

A 3-month cut-off time was used as this is the point in which the ePRO app experienced its first sharp decline in the active number of participants (Figure 2) for both groups (early and late intervention groups). Previous mHealth studies have shown a similar drop in the number of active participants after 3 months, and have divided participants into long- and short-term users according to that shift in usage.³⁶

Figure 2.

Comparison of patient-reported post-study system usability questionnaire (PSSUQ) sub-scale scores between two groups over the intervention period.

PSSUQ to measure usability of the ePRO app

The PSSUQ survey³⁴ was used to measure the usability of the ePRO app for this trial and had been used in all previous piloting and exploratory trial phases of the study.²⁸ Please see Supplemental Appendix 1 for PSSUQ. Each participant received the survey every 3 months after they started using the app. The PSSUQ is a questionnaire to assess the perceived usability of technology, which has been demonstrated to be valid and reliable even for small sample sizes.^34,37 This questionnaire contains 18 items on a 7-point Likert scale,³⁴ providing an overall global score (averaging all 18 items) and three sub-scores: system usefulness (seven items), information quality (six items), and interface quality (three items). Health technologies with lower PSSUQ scores indicate higher/better usability.

Research memos to contextualize study data

Research memos were used to interpret the usability and usage of ePRO. In qualitative research, a research memo is defined as recorded documentation that aims to convey the meaning of a phenomenon.³⁸ Using a research memo in this usability study helps the researchers to answer the question “what is happening with the data?”³⁹ During the 15 months of the study period, many interactions with the participants of the study occurred including patients’ onboarding experience, phone calls regarding technical difficulties, patient–provider interaction during patient visits, and patients’ concerns regarding surveys or the overall study. Those events were defined as study incidents and were captured using detailed research memos documented by the research team.

Supplemental Appendix 2 and Table 1 summarize the description of the study measures, data sources, description of each data source, time of collected data, and analysis of each measure.

Table 1.

Characteristics of the patients.

Variable		Late group (N = 21)	Total (N = 44)
Age, mean (SD)	68.65 (7.10)	71.98 (6.20)	70.32 (6.65)
Sex, n (%)
Female	15 (65.22)	7 (33.33)	22 (48.88)
Smartphone comfort level, n (%)^a
Very comfortable	5 (21.74)	7 (38.89)	12 (26.09)
Somewhat comfortable	14 (60.27)	5 (27.78)	19 (41.30)
Not comfortable	1 (4.35)	0	1 (2.17)
Very uncomfortable	1 (4.35)	2 (11.11)	3 (6.52)
Don't know	2 (8.70)	4 (22.22)	6 (13.04)
Missing	2 (8.06)	1 (5.55)	3 (6.818)
Total	23 (100)	18 (100)	44 (100)
# of chronic condition (means, SD)	5.95 (2.72)	4.5 (1.98)	5.27 (2.46)
Chronic disease management program, n (%)
Yes	6 (26.09)	1 (4.76)	7 (15.55)
No	1 (4.35)	1 (4.76)	2 (44.44)
Gross income, n (%)
$0–29K	7 (30.43)	1 (4.76)	8 (17.77)
$30–$59K	7 (30.43)	5 (23.81)	12 (26.66)
$60–$89K	2 (8.70)	8 (38.10)	10 (22.22)
> $90K	7 (30.43)	7 (33.33)	14 (31.11)
Education, n (%)
Less than high-school	4 (17.39)	1 (4.76)	5 (11.11)
High-school	4 (17.39)	1 (4.76)	5 (11.11)
Some college/university	9 (39.13)	4 (19.05)	13 (28.8)
University (undergraduate, graduate	6 (26.09)	15 (71.43)	21 (46.66)

The range of the smartphone comfort level score is 1–5. A higher score indicates a higher comfort level with the smartphone.

Statistical analyses

Participants’ demographic information was analyzed using descriptive statistics. Descriptive analysis was also conducted for the PSSUQ scores stratified by participants’ group (early or late intervention groups), subscales of PSSUQ (system usefulness, interface quality, and information quality), and participant demographics (age, sex, number of chronic conditions). Mean usability scores (standard deviation [SD]) were calculated for the total PSSUQ and its subscales. The scores were calculated based on intention to treat, meaning that all patients were included in their original groups regardless of their compliance to the ePRO intervention and withdrawal from the study. Mean usability score of each usage group (non-users, short-term users, and long-term users), was calculated. The attrition rate of participants was defined as the percentage of active participants per group each month. The following formula was used to calculate the percentage of active participants per group:

Percentage of active participants per group = (Number of active participants (per month)/Total number of individuals per group per month).

Qualitative memo analysis

In total, 15 researcher memos were analyzed narratively using an inductive approach⁴⁰ to identify contextual information of the patients’ experience with the technology and how it impacts patients’ ePRO user experience. To analyze the research memo,⁴¹ we used a thematic analysis approach involving four steps: (1) breaking down each memo into incidents; (2) coding the incidents into categories based on this study's focus of inquiry which was factors that contributed to ePRO usability and patient attrition; (3) Establishing relationships between categories; and (4) creating a coherent explanatory description.^41,42

Ethics

This study was approved by the University of Toronto Health Sciences Research Ethics Board and the special Research Ethics Board of three participating primary care practices.

Results

Patient demography

Of the 46 patient participants who initially enrolled in the intervention arm of the ePRO trial, one participant dropped out before getting initial training, the remaining (n = 44) patients’ data were analyzed. Table 1 shows the demographic characteristics of ePRO participants. Baseline patient characteristics were derived from participants’ demographic form and their EMR.

Usability of ePRO

The comparison of PSSUQ subscales scores between participants from early intervention sites and late intervention sites is presented in Figure 3, which shows a decrease (meaning better usability) in PSSUQ score from 3 to 6 months for both groups (except overall satisfaction for the early intervention group and information quality for the late intervention group) and then the average scores increase (i.e. poorer usability) up to 9 months.

Figure 3.

Gradual attrition of electronic Patient Reported Outcomes (ePRO) patients.

The response rate of the PSSUQ survey for each time point is noted in Table 2. Table 2 shows that the number of PSSUQ respondents declined over time. After 6 months of the intervention, there were only seven respondents in both groups. However, the average PSSUQ scores of overall satisfaction and system usefulness decreases after 12 months for early intervention group participants as compared to their reports after being on the system for 3, 6, and 9 months. There were only four respondents at 12 months (Table 2).

Table 2.

Number and percentage of respondents at each time point.

Participant response rate, n (%)	Months on intervention
Participant response rate, n (%)	3	6	9	12
Early intervention group	15(65.22)	7(45.45)	11(55.00)	4(20.00)
Late intervention group	7(33.33)	7(33.33)	12(66.67)	^a

Late intervention group was only on intervention for 9 months.

Participant attrition and comparison of long-term users versus short term users versus non-users

To better understand the unusual pattern of PSSUQ scores throughout the intervention period, we compared this data to user engagement patterns over time. The overall percentage of active participants decreased steadily with the longer follow-up period (Figure 2).

Table 3 shows the distribution of long, short, and non-users of ePRO between the early and late intervention groups. Of the 44 participants who received the intervention and training, 21 (48%) are long term users, using the app for more than three months, 5 (11%) participants are short-term users (using the app for less than 3 months), and 17 (39%) participants are non-users (did not use the app after their initial onboarding training). The late intervention group had a fewer number of long-term users and a larger number of non-users comparing to the early intervention group.

Table 3.

Participant distribution in three user groups.

Usage category n (%)	Early intervention group	Late intervention group	Total
Long term	16 (66)	5 (26)	21 (48)
Short term	5 (21)	0 (0)	5 (11)
Non-users	3 (12)	14 (73)	17 (39)
Total	24 (100)	19 (100)	43 (100)

Table 4 illustrates the descriptive distribution of participants based on their user type and demographic characteristics. A majority of long-term users (70%) were female, while only 17.6% of non-users were female. 12% of non-users had undergrad or higher education attainment whereas 5% of long-term users had undergrad or higher education attainment. We used the Kruskal-Wallis test to assess significant differences between the three groups (long-term, short-term, and non-users). The p-values of each variable show that there were no significant differences between the three groups in terms of age, sex education, or the number of chronic conditions.

Table 4.

Participant distribution based on user type.

Variables	Long term (n = 21)	Short term (n = 5)	Non-users (n = 17)	p-value
Mean (age SD)	71.57 (6.24)	66.89 (7.67)	73.76 (6.37)	0.44
Sex n (%)				0.04
Female (%)	14 (70)	5 (55)	3 (17.6)
Education n (%)				0.071
Less than high school	5 (23.81)	0	0
High-school	2 (9.52)	1 (20)	2 (11.76)
Some college/university	9 (42.86)	1 (20)	3 (17.65)
Undergrad Deg.	2 (9.52)	1 (20)	6 (35.29)
Grad/Prof Deg.	3 (14.29)	2 (40)	6 (35.29)
# of chronic disease	3.72 (2.33)	3.52 (1.72)	2.85 (1.56)	0.58

Comparison: The relationship between actual usage with the PSSUQ score

To gain insight into this pattern of usability, the authors drew a comparison of long-term users and other users (who used ePRO for less than 3 months) in terms of their mean (SD) usability score (Table 5). We combined both non-users and short-term users with the “other user” group because there were less than five participants in the non-user group for 6-month, 9-month, and 12-month data collection periods. Otherwise, we could not conduct any meaningful analysis due to the small sample size of non-users (n = less than 5).

Table 5.

Comparison of baseline and final PSSUQ score among long-term users and other users.

Month	Long term users (mean, SD) [n]	Other users (mean, SD) [n]	p-value (CI)
3	2.80 (1.38) [8]	3.04 (1.74) [10]	0.0332
3	2.80 (1.38) [8]	3.04 (1.74) [10]	(0.94–1.48)
6	3.16 (2.23) [7]	2.81 (1.73) [8]	0.2874
6	3.16 (2.23) [7]	2.81 (1.73) [8]	(−1.56–0.49)
9	3.68 (1.89) [8]	4.24 (1.74) [14]	0.6748
9	3.68 (1.89) [8]	4.24 (1.74) [14]	(−1.62–1.07)
12	2.88 (1.40) [4]	2.66 (1) [1]	^a

No statistical test was conducted because there was only one participant in “other user group.”

We have conducted the Welch’s two-sample t-test to assess the sample mean differences between long-term and “other users.” Table 5 shows that at 3 months there is a statistically significant mean difference between the two groups, with long-term users scoring lower PSSUQ score (mean = 2.80, SD = 1.38) than other users (mean = 3.04, SD = 1.74) with a p-value = .03 (95% CI: 0.94–1.48). The lower PSSUQ score at 3 months for long-term users meant that this group found the ePRO app more usable than the “other users” group and the mean difference between the two groups was statistically significant at 3 months. At 6 and 9 months, the mean difference of the two groups’ PSSUQ was non-significant. At 6 months, long-term users had a slightly higher mean score (mean = 3.16, SD = 2.23) than “other users” (mean = 2.81, SD = 1.73). Meaning at 6 months, long-term users found the ePRO app less usable than “other users” but the mean difference between the two groups was not statistically significant (p-value = .28; 95% CI: 1.56–0.49). At 9 months, the mean PSSUQ score for long-term users was lower (mean = 3.68, SD = 1.89) than “other users” (mean = 4.24, SD = 1.74). However, the mean difference between the two groups was not statistically significant (p-value = 0.6748; 95% CI: −1.62–1.07). At 12 months, only one participant who belongs to the “other user” group responded to the PSSUQ survey so we were unable to conduct any statistical test.

Figure 4 provides a visual representation of the comparison of average overall PSSUQ score to attrition rates over time. As the percentage of active participants shows a gradual decline in both groups, PSSUQ scores show a slight reduction in perceived usability as well; however, that pattern is not a consistent drop over time.

Figure 4.

Active monthly users versus average patient-reported post-study system usability questionnaire (PSSUQ) at different time points for early and late intervention groups patients.

Findings from the qualitative analysis

Analysis of memos indicates two important factors that contributed to the patients’ user experience: (1) Technical difficulty such as being logged out of the ePRO app due to inactivity, forgotten password, and inability to modify goals; (2) Provider engagement level with their patients’ goals and with the ePRO app influenced patients’ willingness to use the app. Table 6 summarizes these factors alongside sub-themes and example memo notes. Participants of late intervention groups were also less likely to troubleshoot when technical difficulties occur. Additionally, late intervention group participants faced initial training difficulties due to schedules. Secondly, when late intervention group participants experienced these technical difficulties the majority of them abandoned the app instead of communicating their difficulty with a provider or the research team. By contrast, when experiencing technical challenges early intervention group participants more often reached out to their providers to resolve the issue or to the research team. Thirdly, in early intervention group sites, primary care providers often played a dual role of being facilitators of social activities such as walking group and their primary role in the family health team such as social workers. Study participants of early intervention groups were active participants of those social activities. The providers of late intervention group sites also played these dual roles in their family health teams but the study participants were not part of it. There were different levels of enthusiasm for ePRO among the providers as documents in the memos. Based on providers’ buy-in to the app, they actively/discouraged their patients to use the app (please see Supplemental Appendix 3 for detailed thematic analysis of research memos including example scenarios).

Table 6.

Main categories and subsequent themes emerged from research memos.

Main categories	Themes	Example incidents
Technical errors experienced by patients	Onboarding experiences (i.e. scheduling issues)	Scheduling issues with the research team
Technical errors experienced by patients	Technical errors (i.e. forgotten password)	Forgotten password, a task was not saved properly so patient reached out to the provider/research team
Patient–provider interaction	Patients' interaction with the primary care team	The patient called the provider to solve ePRO related problems such as changed goals
	Part of activities facilitated by the primary care practice	The primary provider plays a dual role in facilitating social activities in the Family Health Team
	Primary care team providers' attitude toward ePRO	Care provider did not consider ePRO to be necessary for patients' conditions

Discussion

The usability score of the ePRO app remained relatively unchanged over the study period with relatively minor oscillation. The attrition rate of ePRO shows a steady decline after 3 months. While comparing the PSSUQ scores of three user groups, no obvious pattern was found, suggesting that patients’ PSSUQ score did not determine them being long-term or short-term users. Although we might expect that increased attrition could be linked to a lower usability score, in this study this was not the case. One explanation of a relatively stable PSSUQ score over time could be a self-selection issue. Patients who found the tool unusable may have stopped using the tool and but also had stopped answering the surveys. Previous usability studies of mHealth applications have found that a number of technical difficulties when using an app^13,28,43 as well as contextual factors such as providers’ engagement level, patients’ disease priorities^44,45 contribute to patient attrition.

Research memos offer insights into other contextual factors that may influence patients’ willingness to adhere to the ePRO app. The patient-provider relationship emerged as a core factor in patient engagement in the app as it impacted how and whether patients would troubleshoot an error regarding the app or any other goal-related difficulties such as changing their health goals. A strong, positive relationship with the provider may work as a resilient factor in case of the occurrence of any technical or goal-related challenges. Previous research has similarly demonstrated low attrition rates of digital interventions for patients with chronic conditions experience when health providers are more engaged in the intervention^44,46,47 However, given the potential self-selection bias and unknown degree of each factor, it is difficult to determine whether usability or context factors drove attrition rates over time.

To improve patients’ overall user experience, there is a need to focus on making apps easy to use (i.e. eliminate areas of technical difficulty) with consideration of patient context.⁴⁸ Often technical difficulties in mHealth apps are inevitable but providing ongoing training and sometimes paper-based reminders on how to log-in and use the app might be beneficial for new adopters of technology. Other mHealth studies report that older adults and new adopters of the technology have more difficulty getting used to an app interface and there can be readability issues due to screen adjustment that feels comfortable to them.^47,48

While research memos related to patient use of the data reveal some important contextual variables that may play a role in attrition, the literature suggests additional variables not captured here. Previous studies have demonstrated that acknowledging patients’ health priorities in delivering appropriate treatment plans,⁴⁹ aligning patient-provider goals from a novel app⁵⁰ and scope of personal contact with primary health providers²³ might improve patients’ engagement with a newly implemented digital innovation. These primary care-based studies flag that primary care providers play a critical role in their patients’ health-related goals. Notably, many of these factors are also related to the patient–provider relationship which was identified as a critical variable in the study presented here; potentially suggesting that therapeutic relationships may represent an underlying mechanism driving patient use of novel digital health tools, which is congruent with previous research findings.⁵¹

Strength of the study

There are two main strengths of this sub-study: (1) the analysis of patient-reported usability data and attrition of a mHealth app using a multifaceted blended analysis approach and (2) the use of real-world data to understand complex patients’ overall user experience with a goal-oriented mHealth app. The inclusion of qualitative memos suggests that the user experience of an app is nuanced and influenced by patients’ contextual circumstances. This study demonstrates the benefits of combining multiple data sources to gain further insight into app usability and its association with usage attrition.^25,26,47

This study uses real-world data from patients with complex health needs and their overall engagement with a goal-supporting mHealth app. By implementing the app in a real-world setting, we were able to uncover pragmatic factors and natural mechanisms such as the patient–provider relationship that impact patients’ user experience of adopting a mHealth app. These findings are transferable to other settings where patients and providers engage in goal development. This will guide future mHealth researchers to put patient–provider collaboration in the center of the mHealth app design.

Limitations

There are two limitations to the current study: (1) Small sample size and (2) No baseline usability score to compare patients’ initial usability score with scores after a certain exposure time. The smaller sample size has limited our ability to conduct inferential or predictive analysis. With larger data, a descriptive association between user demography and user engagement level or relationship between patients, attrition, and usability score could have been drawn. Moreover, a larger dataset would allow further mixing of quantitative and qualitative data⁵² to provide a detailed report on how patients’ characteristics such as education level may impact patients’ ability to troubleshoot technical errors related to ePRO. Lastly, we also could have benefited from conducting an immediate usability assessment just after the first use of the system as waiting 3 months may be too long to capture users’ initial perceptions of the tool and how the score changes over time.

Future direction

To go beyond evaluating usability and attrition of digital health intervention, future research will be better informed by examining contributing factors in patients’ engagement based on their preference, skills, motivation, and health priorities.^15,53 Additionally, as eHealth communities are looking at the sustainable use of mHealth intervention to ensure effective outcomes, there is a need for research to identify relational/societal factors (i.e. enhanced supervision from the provider) that contributes to patients’ usage pattern and increase patient uptake of mHealth^54,55

Drawing from the findings of this paper, we plan to conduct a qualitative narrative inquiry on patients’ experience with the ePRO app. The qualitative inquiry will help us to capture the individual-level factors that form patients’ app usage behavior and usage pattern. The qualitative exploration will be able to capture the inhibiting/facilitating factors in using a goal-management app from a complex patients’ perspective. Additionally, it has the potential to leverage patient knowledge and experience on how to address those issues. Further lived insight into the issue of abandonment and usability will create a knowledge base for future researchers to design systems that match target patients’ unmet needs.

Conclusion

The findings of this study suggest that the patient–provider relationship is an important factor in patients’ engagement with a mHealth app that supports goal-oriented care in a primary care setting. Usability questionnaires or attrition rates alone are not sufficient to evaluate a mHealth intervention's user engagement because this omits important contextual variables that may influence use. This paper also highlights the contribution of a qualitative approach as a valuable tool to capture societal/relational nuances that drive complex patients’ usage of a mHealth app.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076211045579 - Supplemental material for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach

Supplemental material, sj-docx-1-dhj-10.1177_20552076211045579 for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach by Farah Tahsin, Shawn Tracy, Edward Chau, Sarah Harvey, Mayura Loganathan, Brian McKinstry, Stewart W Mercer, Jason Nie, Tim Ramsay, Kednapa Thavorn, Ted Palen, Jasvinei Sritharan and Carolyn Steele Gray in Digital Health

Supplemental Material

sj-docx-2-dhj-10.1177_20552076211045579 - Supplemental material for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach

Supplemental material, sj-docx-2-dhj-10.1177_20552076211045579 for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach by Farah Tahsin, Shawn Tracy, Edward Chau, Sarah Harvey, Mayura Loganathan, Brian McKinstry, Stewart W Mercer, Jason Nie, Tim Ramsay, Kednapa Thavorn, Ted Palen, Jasvinei Sritharan and Carolyn Steele Gray in Digital Health

Supplemental Material

sj-docx-3-dhj-10.1177_20552076211045579 - Supplemental material for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach

Supplemental material, sj-docx-3-dhj-10.1177_20552076211045579 for Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach by Farah Tahsin, Shawn Tracy, Edward Chau, Sarah Harvey, Mayura Loganathan, Brian McKinstry, Stewart W Mercer, Jason Nie, Tim Ramsay, Kednapa Thavorn, Ted Palen, Jasvinei Sritharan and Carolyn Steele Gray in Digital Health

Footnotes

Acknowledgments

The authors thank patients and providers for their participation in the study. Additionally, we would like to Family health team administrators for their assistance and guidance in this research.

Contributorship

FT: conceptualization, data analysis, writing-first draft; ST: data analysis, writing-first draft, reviewing and editing; EC: writing-critical review and editing; BM: writing-critical review and editing; SH: writing-critical review and editing; ML: writing-critical review and editing; SM: writing-critical review and editing; JN: writing-critical review and editing; TR: writing-critical review and editing; KT: writing-critical review and editing; JS: writing-critical review and editing; CSG: conceived and designed the study, data collection, and analysis, writing-initial draft, reviewing, editing, and senior author for this project. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Conflicting interests

The employee salary of the project was funded from a research grant. CSG has received grants from the Canadian Institutes of Health Research. The funders did not have any involvement in the study's design, execution, analyses, interpretation of the data, review, or approval of the manuscript for publication.

Ethical approval

The ethics committee of the University of Toronto (REC: #33944) and Sinai health system (REB#16-0254-E) have approved this study. Study participants have provided written consent to participate in the study.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Canadian Institutes of Health Research (CIHR) (grant no. 143559).

Guarantor

FT.

Informed consent

All study participants provided written and verbal consent to participate in the study.

Trial registration

Clinicaltrials.gov Identified NCT02917954.

ORCID iD

Farah Tahsin

Supplemental material

Supplemental material for this article is available online.

References

World Health Organization. Regional office for the western Pacific. People-centred health care: a policy framework. Geneva: WHO Press, 2007.

Evans

Matheson

Buchman

, et al.

Integrating cancer care beyond the hospital and across the cancer pathway: a patient-centred approach.

Healthcare Quarterly (Toronto, Ont) 2015; 17: 28–32.

Tinetti

Naik

Dodson

. Moving from disease-centered to patient goals-directed care for patients with multiple chronic conditions: patient value-based care. JAMA Cardiol 2016; 1: 9–10.

Bickerstaffe

. Towards whole person care. London: IPPR, 2013: 1–26.

Gray

Wodchis

Upshur

, et al. Supporting goal-oriented primary health care for seniors with complex care needs using mobile technology: evaluation and implementation of the health system performance research network, bridgepoint electronic patient reported outcome tool. JMIR Res Protoc 2016; 5(2): e126.

Schaink

Kuluski

Lyons

, et al. A scoping review and thematic classification of patient complexity: offering a unifying framework. J Comorbidity 2012; 2(1): 1–9.

Marengoni

Angleman

Melis

, et al. Aging with multimorbidity: a systematic review of the literature. Ageing Res Rev 2011; 10: 430–439.

Reuben

Tinetti

. Goal-oriented patient care – an alternative health outcomes paradigm. New England J Med. Massachusetts Med Soc 2012; 366: 777–779.

Steele Gray

Miller

Kuluski

, et al. Tying eHealth tools to patient needs: exploring the use of eHealth for community-dwelling patients with complex chronic disease and disability. JMIR Res Protoc 2014; 3(4): e67.

10.

Berntsen

Høyem

Lettrem

, et al. A person-centered integrated care quality framework, based on a qualitative study of patients’ evaluation of care in light of chronic care ideals. BMC Health Serv Res 2018; 18: 479.

11.

Steele Gray

Barnsley

Gagnon

, et al. Using information communication technology in models of integrated community-based primary health care: learning from the iCOACH case studies. Implement Sci 2018; 13(1): 1–14.

12.

Wildevuur

Simonse

LWL

. Information and communication technology-enabled person-centered care for the “big five” chronic conditions: scoping review. J Med Internet Res 2015; 17(3): e77.

13.

Vaghefi

Tulu

. The continued use of mobile health apps: insights from a longitudinal study. JMIR mHealth Uhealth 2019; 7(8): e12983.

14.

Gagnon

Ngangue

Payne-Gagnon

, et al. M-Health adoption by healthcare professionals: a systematic review. J Am Med Inform Assoc 2016; 23(1): 212–220.

15.

Anderson

Burford

Emmerton

. Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS ONE 2016; 11(5): e0156164.

16.

Lewis

Ray

Liaw

Recent worldwide developments in eHealth and mHealth to more effectively manage cancer and other chronic diseases – a systematic review. Yearbook of Med Inform. Thieme Med Publish 2016; 1: 93–108.

17.

Bashi

Fatehi

Fallah

, et al. Self-management education through mHealth: review of strategies and structures. JMIR mHealth and uHealth 2018; 6(10): e10771.

18.

Mattila

Orsama

Ahtinen

, et al. Personal health technologies in employee health promotion: usage activity, usefulness, and health-related outcomes in a 1-year randomized controlled trial. J Med Internet Res 2013; 15(2): e16.

19.

Kelders

Kok

Ossebaard

, et al. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012; 14(6): e152.

20.

Kayyali

Peletidi

Ismail

, et al. Awareness and use of mHealth apps: a study from England. Pharmacy 2017; 5(4): 33.

21.

Eysenbach

. The law of attrition. J Med Internet Res 2005; 7(1): e11.

22.

Pedersen

Mansourvar

Sortsø

, et al. Predicting dropouts from an electronic health platform for lifestyle interventions: analysis of methods and predictors. J Med Internet Res 2019; 21(9): e13617.

23.

Druce

Dixon

McBeth

. Maximizing engagement in mobile health studies: lessons learned and future directions. Rheum Dis Clin North Am 2019; 45(2): 159–172.

24.

Nielsen J. Usability engineering. San Francisco: Morgan Kaufmann, 1994.

25.

Zapata

Fernández-Alemán

Idri

, et al. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015; 39(2): 1–19.

26.

Maramba

Chatterjee

Newman

. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inf 2019; 126(March): 95–104.

27.

Gray

Gravesande

Hans

, et al. Using exploratory trials to identify relevant contexts and mechanisms in complex electronic health interventions: evaluating the electronic patient-reported outcome tool. J Med Internet Res 2019; 21(2): e11950.

28.

Steele Gray

Gill

Khan

, et al. The electronic patient reported outcome tool: testing usability and feasibility of a mobile app and portal to support care for patients with complex chronic disease and disability in primary care settings. JMIR Mhealth Uhealth 2016; 4(2): e58.

29.

Steele Gray

Khan

Kuluski

, et al. Improving patient experience and primary care quality for patients With complex chronic disease using the electronic patient-reported outcomes tool: adopting qualitative methods into a user-centered design approach. JMIR Res Protoc 2016; 5(1): e28.

30.

Gocan

Mary

Laplante

, et al. Interprofessional collaboration in Ontario's Family health teams: a review of the literature. Journal of Research in Interprofessional Practice and Education 2014; 3(3): 1–19.

31.

Craig

Dieppe

Macintyre

, et al. Developing and evaluating complex interventions: the new medical research council guidance. BMJ 2008; 337: a1655.

32.

Catwell

Sheikh

. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 2009; 6(8): 1–6.

33.

Agboola

Hale

Masters

, et al. “Real-world” practical evaluation strategies: a review of telehealth evaluation. JMIR Res Protoc 2014; 3(4): e75.

34.

Lewis

. Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int J Hum Comput Interact 2002; 14: 463–488.

35.

FitzPatrick

Hess

Sudbury-Riley

, et al. A typology of patients based on decision-making styles: cross-sectional survey study. J Med Internet Res 2019; 21(11).

36.

Bradway

Pfuhl

Joakimsen

, et al. Analysing mHealth usage logs in RCTs: explaining participants’ interactions with type 2 diabetes self-management tools. PLoS ONE 2018; 13(8): e0203202.

37.

Tullis

Stetson

. A comparison of questionnaires for assessing website usability. In: Usability professional association conference, Minneapolis, Minnesota, 2004, vol. 1, pp. 1–12.

38.

Creswell

. Qualitative inquiry & research design. California: Sage Publications, Inc, 2007.

39.

Birks

Francis

. Memoing in qualitative research probing data and processes. J Res Nurs 2008; 1: 68–75.

40.

Creswell

Poth

. Qualitative inquiry and research design; choosing among five approaches, California: Sage publications, 2017.

41.

Lincoln

Guba

. Establishing trustworthiness. Naturalistic inquiry. In: naturalistic inquiry. Thousand Oaks: SAGE, 1985.

42.

Glaser

Strauss

. The discovery grounded theory: strategies for qualitative inquiry. London, England: Wiedenfeld and Nicholson, 1967.

43.

Vermeulen

Neyens

JCL

Spreeuwenberg

, et al. User-centered development and testing of a monitoring system that provides feedback regarding physical functioning to elderly people. Patient Prefer Adherence 2013; 7(May): 843–854.

44.

Schubart

Stuckey

Ganeshamoorthy

, et al. Chronic health conditions and internet behavioral interventions: a review of factors to enhance user engagement. CIN - Comput Inform Nurs 2011; 29(2): 81–92.

45.

Perski

Blandford

West

, et al. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017; 7(2): 254–267.

46.

Swinkels

ICS

Huygens

MWJ

Schoenmakers

, et al. Lessons learned from a living lab on the broad adoption of eHealth in primary health care. J Med Internet Res 2018; 20(3): e83.

47.

Matthew-Maich

Harris

Ploeg

, et al. Designing, implementing, and evaluating mobile health technologies for managing chronic conditions in older adults: a scoping review. JMIR Mhealth Uhealth 2016; 4(2): e29.

48.

Arnhold

Quade

Kirch

. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res 2014; 16(4): e104.

49.

Morris

Sanders

Kennedy

, et al. Shifting priorities in multimorbidity: a longitudinal qualitative study of patient's prioritization of multiple conditions. Chronic Illn 2011; 7(2): 147–161.

50.

Andersen

Bansler

Kensing

, et al. Aligning concerns in telecare: three concepts to guide the design of patient-centred E-health. An Int J Comput Supported Cooperative Work (CSCW) 2019; 28(1): 1039–1072.

51.

Mercer

Jani

Blane

, et al. The role of empathy in therapy and the physician-patient relationship. Forsch Komplementmed 2012; 19(5): 252–257.

52.

Johnstone

. Mixed methods, mixed methodology health services research in practice. Qual Health Res 2004; 14(2): 259–271.

53.

Walsh

Golden

Priebe

. Systematic review of patients’ participation in and experiences of technology-based monitoring of mental health symptoms in the community. BMJ Open 2016; 6(6): e008362.

54.

Greenhalgh

Wherton

Papoutsi

, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017; 19(11): e367.

55.

Townsend

Leese

Adam

, et al. eHealth, participatory medicine, and ethical care: a focus group study of patients’ and health care providers’ use of health-related internet information. J Med Internet Res 2015; 17(6): e155.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB

0.02 MB