Sage Journals: Discover world-class research

Abstract

Objective

The UPrEPU mobile app is a self-monitoring system to enable men who have sex with men to optimize their pre-exposure prophylaxis adherence for HIV prevention. The app was designed to accommodate a rather complicated event-driven dosing schedule. We aim to evaluate the usability of the UPrEPU app and its effectiveness in improving adherence monitoring.

Methods

From May to October 2020, 35 participants were enrolled for the usability study and followed up for 4 months. Blood samples for the drug concentration in the dried blood spots were obtained once during the second to fourth follow-up visits. The effectiveness of adherence monitoring was analyzed using Cohen's kappa statistic to calculate the concordance between the average number of pills taken and drug concentration in the dried blood spots.

Results

Overall retention was 91.4% (32 participants) at the end of the study. Participants used the app for a mean of 29 days and made 2565 data entries in total, with an average of 76 data entries. The average systematic usability scale score for the app was 71.5, indicating acceptable usability. Slight agreement was reached between the dried blood spots measurement and the number of pills taken and recorded in the app (weighted kappa: 0.21).

Conclusions

Our user-centered UPrEPU app demonstrated that it could accommodate both daily and event-driven dosing schedules for men who have sex with men clients with acceptable usability scores. We confirmed that complex behaviors such as different drug-dosing regimens that are contingent on sexual behaviors could be incorporated into the design of a mobile app.

Keywords

Mobile apps pre-exposure prophylaxis event-driven sexual behavior men who have sex with men user-centered design approach

Introduction

Oral pre-exposure prophylaxis (PrEP) has been regarded as a highly effective and comprehensive HIV prevention strategy. The efficacy of PrEP could reach over 90% when high adherence was kept among men who have sex with men (MSM).^1–3 The event-driven (ED) PrEP dosing regimen—which involves taking two pills of tenofovir disoproxil fumarate/emtricitabine (TDF/FTC) between 2 and 24 h before sex, followed by two pills of TDF/FTC taken 24 and 48 h after the first pill intake—was an alternative to the daily dosing regimen recommended to MSM.⁴ The ED PrEP dosing regimen among MSM has been gradually adopted in several countries, such as France,¹ the Netherlands,⁵ Belgium,⁶ and Taiwan.⁷ A pre-coital dose of PrEP should be taken at least two hours before each sexual intercourse for an ED regimen, but users may not be able to correctly estimate the exact timing of a sexual event and thus miss the pills.⁸ The post-coital dose can be easily forgotten as well. Studies have shown that 81.9% of ED PrEP users miss their pills before sex, and 4.3% miss pills after sex.⁸

The effectiveness of PrEP correlates with the level of adherence.² Current studies use two major types to estimate PrEP adherence: self-reports and objective measurement of drug concentration. Self-reporting measures vary and often include indirect measurements such as pill counts in different time frames using different methods. A short message service is an approach that facilitates self-reporting because the participants receive messages to prompt them to report frequently, thus reducing recall bias.^9,10 Drug concentrations are often measured as a more objective way to estimate adherence and for comparison.¹¹ One study that used a sample of adolescent and young MSM from the United States compared four types of adherence measurement: self-reported adherence, smart pill cap openings, TFV concentrations in hair, and TFV-DP concentrations in dried blood spots (DBS).¹² Compared to the drug concentration methods, self-reported measures overestimated the adherence; smart bottle openings underestimated adherence because the bottles were not being used consistently.¹² For evaluating adherence, however, none of these measurements is perfect for ED regimens in which the timing of the sexual event should be considered. MSM PrEP users might adopt PrEP dosing regimens based on their preferences and lifestyles.^5–7,13 Switching between two dosing regimens is not uncommon in the real world.^5–7 While ED PrEP already has a complex dosing schedule, it would be more difficult to measure the real-world ED PrEP adherence with dosing regimen switching involved. Approaches to ensure and improve PrEP adherence are crucial to maximize the benefits of PrEP.

Mobile health interventions have shown promising feasibility and acceptability in improving medication adherence and preventive behavior in various medical conditions.^14,15 Moreover, mobile phone applications have been adopted as one of the mobile health technologies to improve access to HIV prevention services and adherence for antiretroviral therapy among people who live with HIV. Given the ubiquity of smartphones, mobile apps for MSM offer distinctive opportunities for public health interventions targeting this population and empowering them to self-care and to handle their healthcare.¹⁶ Several studies have been evaluating smartphone and Internet-based interventions for HIV prevention and PrEP adherence. Most of the studies focus on MSM and aim to facilitate HIV testing or PrEP uptake by providing home delivery of prescriptions.¹⁷ Only a few other studies have been focusing on self-monitoring of PrEP adherence among MSM; however, these studies have had limitations of daily PrEP use only and have lacked a user-centered practices design.^18–20

UPrEPU is a novel mobile application designed with a user-centered approach and was developed following the multiphase optimization strategy framework (comprising a preparation phase, an optimization phase, and an evaluation phase).^21–23 The UPrEPU app incorporates concepts of a participatory design approach, recognizes the needs and the direct input of the users, and took their needs and expectations into consideration in the design process.²⁴ This app was designed to assist PrEP clients, including those who used the ED dosing regimen, to better manage PrEP use by themselves. Our app allows flexibility for MSM's dynamic dosing choices and aims to improve PrEP adherence by sending reminders based on the current dosing regimen and users’ sexual practices.

In this paper, we aim to describe the development of UPrEPU and evaluate the usability of this app. We further evaluated the effectiveness of adherence monitoring of the UPrEPU by comparing the pills taken and sexual intercourse recorded in the app with the drug concentration in DBS. We hypothesized that the UPrEPU app developed through a user-centered approach would be acceptable and easy to use. We expected moderate to high agreement between DBS measurement and the number of pills taken in the app among MSM PrEP users in Taiwan.

Methods

Development of the UPrEPU app

The design and development process of the UPrEPU app used a multidisciplinary and user-centered design approach.^25–27 Formative conceptualization qualitative interviews were conducted to understand the MSM PrEP clients’ needs. PrEP clients’ personas and journey maps were developed from the data collected in the interviews. One session of each participatory workshop was conducted with the personas and journey maps used to generate concept ideas and a low-fidelity prototype of the UPrEP app.

The participants for the participatory workshop included PrEP users, PrEP managers, researchers with expertise in human–computer interaction design or public health, and app developers. They were randomly assigned to two developing groups. A detailed description of the development process of the UPrEPU app was described in a previous report.²⁸ A low-fidelity paper prototype of UPrEPU was generated under an agreement of all participants in the workshop (Table 1).

Table 1.

Features generated from the workshops (low-fidelity paper prototype).

	Group 1	Group 2
Planned features	Chatbot^a to answer PrEP-related questions.	Tailored reminder for PrEP uptake for daily/ED PrEP users
	Information for accessing HIV testing and PrEP.	Reward system to encourage adherence to PrEP.
	Reminders	Reminders and tailored feedback messages
	Visualization of drug concentration.	PrEP diary to record their daily PrEP use
	Social support through online platforms.	Communication and social support through the private message function.

Chatbot: A chatbot or chatterbot is a software app that simulates human-like conversations with users via text messages on chat.

PrEP: pre-exposure prophylaxis.

The research team refined a high-fidelity prototype afterward. The internal testers (research team members) accessed all features of the UPrEPU app to identify any remaining glitches, bugs, or usability concerns with the heuristic evaluation method.²⁹ Then, the official UPrEPU app was developed regarding the feedback from internal testers. The UPrEPU app was developed for IOS and Android operation systems. A list of final app features and descriptions is shown in Table 2.

Table 2.

Features of UPrEPU app.

Features	Function
Customizable avatars	• Visual representations of users and their sex partners within the app to facilitate the development of an online identity.
Dashboard	The home screen where users can choose and switch their dosing regimens. Provides a visualization of the current adherence pattern. Shows the time and date of the last dose. Reminders on how many pills and when to take the next dose.
Calendar	A calendar allows users to create, edit, and delete the PrEP and sexual event diary and HIV testing records. Detailed information recorded for each sexual event, including the avatars, alias, sex position (bottom, top, or both), prevention strategies, and the mood at the moment of the sexual event. Signal on the calendar to show whether users have taken PrEP correctly regarding the users’ dosing regimens.
Data visualization of PrEP adherence and mood for sexual events	Provides a 14-day summary of PrEP adherence and mood for each sexual event with sex partners.
Information modules	Educational materials on HIV, PrEP, and other sexual health issues. Provides the nearest geographic location of HIV testing in Taiwan based on the users’ location.
Notification	Push notification text to remind app users to take PrEP, backfill their sex and PrEP record.

PrEP: pre-exposure prophylaxis.

Screenshots of the app are available in Multimedia Appendix 1. The highlighted features include visualized drug concentrations, a sex diary that allows users to create and record the details of their sexual events (including time for sex, prevention tools used in the sex, sex roles, and mood tracker), and a calendar. Participants also log their PrEP intake and choice of the dosing regimen. Based on the dosing regimen and the participants’ scheduled dates with their sexual partners—a feature that is particularly important for an ED dosing regimen—the app shows reminders for pills needed to be taken in the following days on the calendar. The resources provided on the app include HIV and PrEP-related educational multimedia materials and a global positioning system (GPS)-enabled map of HIV testing locations. Data visualization in the app shows a graph integrating the mood for each sexual event and PrEP use. Users can see the pattern of their mood with different sex partners with or without PrEP protection, which is also one of the highlighted features.

Usability, feasibility, and effectiveness of adherence monitoring of the UPrEPU app

Recruitment and enrollment

Participants were referred by physicians; PrEP navigators were recruited in the sexual health clinic from May to June 2020 and assessed for eligibility by an online screening tool. Eligibility requirements included the following: (1) Participants were men who were 20 years of age or older; (2) resided in Taiwan and able to understand, read, and speak Mandarin Chinese; (3) tested negative for HIV 3 months prior to enrollment for current PrEP users; (4) had laboratory results that indicated eligibility to initiate PrEP based on Taiwan's PrEP guidelines;³⁰ (5) currently took PrEP or were willing to initiate PrEP after enrollment; (6) reported having at least four episodes of anal intercourse with men in the previous 1 month; (7) owned an Android or Apple operating system (iOS) smartphone and were willing to download the study app; and (8) were willing to wear the Garmin wearable fitness tracker that the research team provided during the study period.

Eligible participants were asked to complete informed consent forms upon entering the study. Participants were then asked to download the UPrEPU app, and watch a short clip embedded within the app for feature demonstration. The participants were also required to answer the baseline questionnaire to finish their enrollment in the study.³¹

Participants were followed up monthly for 4 months. Each monthly visit included rapid testing for HIV antigen and antibodies and participants completed a follow-up questionnaire on their mental health scales, sexual behaviors, and PrEP use. Depression and anxiety were assessed using the Patient Health Questionnaire-9 (PHQ-9)³² and General Anxiety Disorder-7 scale (GAD-7),³³ respectively.

The System Usability Scale (SUS) for the UPrEPU app was assessed during the first follow-up visit. Blood samples for the TFV-DP and FTC-TP concentration in the DBS were obtained once during the second to fourth follow-up visits.

Outcome measurement: Usability, feasibility, and effectiveness of adherence monitoring usability

We used the SUS, a reliable and low-cost usability scale,³⁴ as the primary outcome for usability. It was a 10-item, 5-point Likert scale yielding a global view of subjective assessment of usability.³⁴ In the SUS scale, even and odd questions were scored differently. Odd questions were scored 0–4 based on the 1–5 selection. Even questions were scored 4-0 on the 1–5 selection. Scores for all 10 questions were added up for a total score between 0 and 40 points. This total was multiplied by 2.5 to generate a SUS score between 0 and 100 points.³⁵ A product with SUS scores below 50 indicated certainty that it had usability difficulties in the field; scores in the 70 s and 80 s indicated that a product was acceptable.³⁶ We evaluated SUS of UPrEPU app in the month 1 visit. Participants were further asked to clarify their reasons for providing a neutral or negative response to the following question in the SUS scale: “I would not like to use this app frequently” and “app features integration had room for improvement.”

Feasibility

The primary feasibility outcomes were app engagement including the frequency of app logins, the use of app components such as PrEP-taking, and sexual behavior reports. The frequency of app logins and the use of app components such as anal intercourse and pill-taking records were logged and extracted from the app.³⁷ For each sexual event, participants also recorded the mood at the moment of sex on the Smiley Face Likert scale (awful, not very good, okay, really good, and fantastic).³⁸

Secondary feasibility outcomes of this app included a focus on and potential improvement of the app in the future. All participants were asked an open-ended question: “What features would you like to add to the app?”

Effectiveness of adherence

Adherence was estimated by the proportion of anal sex acts that used PrEP correctly. It was calculated as the proportion of “sex-days” (i.e. the number of days that anal intercourses occurred as the denominator) for which PrEP was taken correctly (numerator). Whether PrEP was taken correctly in each sexual event was determined by the algorithm in the app. An algorithm was developed regarding the ED and daily PrEP instruments as follows:

Correct taking of ED PrEP was calculated as taking 2 pills 2–24 h prior to each sexual event, followed by 1 pill 24 h and 1 pill 48 h after the latest pill taken before sexual event. However, if at least one pill was taken 25–168 h (1–7 days) before each sexual event, taking only one pill 2–24 h before each sexual event was allowed.³⁰ Correct use of a daily dosing regimen was defined as at least 4 pills taken 168 h (7 days) before each sexual event.

The effectiveness of adherence monitoring of this app was estimated as the consistency of PrEP use between drug concentrations in the DBS sample and the pill-taking reports in the calendar component in the UPrEPU app. The DBS sample was obtained once after the first month follow-up. The procedure to measure tenofovir-diphosphate (TFV-DP) and emtricitabine-triphosphate (FTC-TP) in DBS has been described previously.³⁹ The lower limit of quantification was 25 fmol/sample for TFV-DP and 100 fmol/sample for FTC-TP. We used validated dosing cut-offs for TFV-DP in DBS to categorize participants’ weekly pill-taking on average: 7 doses/week (>1450 TFV-DP fmol/punch), 4–6 doses/week (800–1449 fmol/punch), 2–3 doses/week (400–799 fmol/punch), and < 2 doses/week (below the limit quantification 350 fmol/punch).^40,41 The pill-taking record of app users in the 6 weeks before DBS collection was estimated on a weekly average and classified in line with the categories of TFV-DP. The DBS samples were stored at − 20°C within 24 h of collection and shipped on dry ice to the Skaggs School of Pharmacy and Pharmaceutical Sciences (University of Colorado Anschutz Medical Campus, Aurora, CO, USA) for analysis.

Analysis

The demographics of participants were described in numbers with percentages. The number of pills taken, the record of sexual events, mood, and PrEP protection in the app were calculated by each individual participant. App usage was described as the total of the average of data entry and days. Descriptive statistics were used to examine the SUS results with an aggregate score. Weighted Cohen's kappa coefficient analysis was used to examine the consistency between the drug concentration in the DBS samples and the self-reported PrEP diary in the app. The difference in drug concentration in DBS was accessed by Wilcoxon rank-sum test. All analyses and visualizations were performed using R 4.0.4.

The study was approved by the Institutional Review Board of the National Cheng Kung University in Tainan City, Taiwan (A-ER-107-337).

Results

Study populations demographics

Thirty-five participants enrolled in this study from May to June 2020. The baseline characteristics of the participants are shown in Table 3. Most participants were 26–30 (16, 48.6%) years of age; 14 (39%) earned less than USD $1000 per month; most participants used iPhones (24, 70.6%); and a substantial number of participants (9, 25.7%) reported they had been switching between daily and ED PrEP prior to study participation. No participants were diagnosed with HIV or syphilis during the follow-ups.

Table 3.

Baseline characteristics of participants (n = 35).

		N (%)
Age (years)
	21–25	7 (17.1)
	26–30	16 (48.6)
	31–35	8 (22.9)
	> = 36	4 (11.4)
Monthly income range (USD)^a
	≧500	4 (10.5)
	501–1000	10 (28.5)
	1001–1500	12 (34.3)
	>1500	9 (25.7)
	Relationship status
	Single	16 (45.7)
	Fixed relationship	6 (17.2)
	Open relationship	13 (37.1)
Sexually transmitted diseases diagnosed in the past year
	Gonorrhea	4 (11.4)
	Syphilis	9 (25.7)
	Genital herpes (HSV-1, HSV-2)	1 (1)
	Genital warts on the anus	5 (14.3)
	Genital warts on the penis	1 (2.9)
	Drug use during sex in the past month
	Rush (Poppers)	4 (11.4)
	Viagra, Cialis, and other aphrodisiac drugs	4 (11.4)
	Amphetamine, methamphetamine	2 (5.7)
	Alcohol	1 (2.9)
	Cannabis	1 (2.9)
	GHB/GBL	1 (2.9)
	Post-exposure prophylaxis (PEP) use in the past 6 months	13 (37.1)
Pre-exposure prophylaxis (PrEP) dosing regimen
	Daily	13 (37.1)
	On-demand	12 (34.3)
	Switch between daily and on-demand	9 (25.7)
	Smartphone^b
	iPhone	24 (70.6)
	Android	10 (29.4)

Exchange rate: 30 New Taiwan Dollars equals 1 US dollar.

One missing.

Thirty-two participants completed the 4-month follow-ups (91.4%). Among them, 3 (8.6%) discontinued the study because of losing a device and refusing to wear the devices. Although not completing our 4-month follow-up, one person had DBS performed and examined, resulting in a sample size of 33 for DBS analysis (Figure 1).

Figure 1.

Enrolled and follow-up of study participants (n = 35).

Usability of the UPrEPU app

The results of the UPrEPU app's usability are shown in Table 4. Overall, participants found the app to be convenient to use and easy to use on their own. The average SUS score for the app was 71.5, which indicated that the UPrEPU app is acceptable and above average, with 18 participants (51.4%) who had scores of more than 70 in the SUS.

Table 4.

System Usability Scale (SUS) scores of the UPrEPU app by men who have sex with men (MSM) participants during the 4-month pilot study. Taiwan, 2020 (n = 35).

Statement	Mean^a (SD)	Absolute^b
I would like to use this app frequently.	2.6 (1.1)	2.6
The app was unnecessarily complex.	1.5 (1.0)	3.5
The app was easy to use.	3.2 (0.8)	3.2
I would need support from a technical person to be able to use this app.	0.8 (0.8)	4.2
Various functions in the app were well integrated.	2.6 (1.0)	2.6
There was too much inconsistency in this app.	1.6 (1.1)	3.4
Most people would learn to use this app very quickly.	3.3 (0.7)	3.3
The app was very cumbersome to use.	1.3 (1.1)	4.7
I felt very confident using the app.	3.3 (0.8)	3.3
I had to learn many things before I could get going with this app.	1.3 (1.1)	3.7
Calculated score		71.5 (12.4)

Scoring based on a scale from 1 = totally disagree to 5 = totally agree.

Adjust scores of negative statements, so larger numbers are associated with positive statements.

The main reasons for participants not using the app frequently included the following: infrequent sexual activities and ED PrEP-taking (n = 5, 35.7%), lack of motivations (n = 4, 28.6%), and few notifications from the app to remind users of opening the app (n = 4, 28.6%). Our participants also mentioned how the app could be improved: the data visualization for the relation between PrEP use, sex, and mood were difficult to understand (n = 6, 37.5%), and function in the calendar feature can be better integrated with other apps (n = 3, 18.8%), such as the Google calendar.

Feasibility

More than 95% (32) of the participants kept using the UPrEPU app during the study period and contributed an average of 641 data entries to the app each month (Figure 2). Participants used the app for a mean of 29 days (median: 24 days, interquartile range (IQR): 11–37 days) and made 2565 data entries in total, with an average of 76 data entries per day (median: 70, IQR: 47–106 data entries). Among the 32 participating users, data entry remained high for each follow-up (96.9%–100%).

Figure 2.

UPrEPU app engagement during the study period (n = 32).

The most used function was the PrEP intake record, with 1806 data of PrEP intake (1984 pills) and an average of 53 pills per person during the 4-month follow-up (median: 53, IQR: 34–74). PrEP was taken correctly in 378 (68.6%) sexual events with anal intercourse (ED PrEP: 254, 62.4%; Daily PrEP: 124, 86.1%). The median of correct PrEP use per person by the study protocol definition was 12 sexual events with an IQR from 4 to 20.

The second frequently used function was the scheduled dates. Participants reported 551 sexual events with anal intercourses, with an average of 17 events per person (median: 14, IQR: 8–21). Condoms were used in 123 (22.3%) events. A total of 207 (37.7%) events occurred with partners with unknown HIV status, 118 (21.5%) with partners who were HIV negative and on PrEP, 165 (30.1%) with partners who were HIV negative and not on PrEP, and 59 (10.7%) with HIV positive partners who had an undetectable viral load.

Participants recorded their mood as “really good” (264, 47.9%) and “fantastic” (181, 32.8%) in most of the sexual events (awful: 3, 0.5%; not very good: 8, 1.5%; okay: 95, 17.2%). There were 81.6% of sexual events with correct PrEP use rated at “really good” or above, and 75.3% of sexual events without correct PrEP use rated at “really good” or above.

The most wanted features for the UPrEPU app in the future was a calendar feature in the app to be integrated with their other calendar apps and keep the schedule consistent without repetitively typing content (22.9%). Some participants (8.6%) would like to have an additional note section to record their sexual events comprehensively. Given that the UPrEPU app has a lot of contexts related to sex and PrEP use, 11.4% of participants suggested that the UPrEPU app develop a social networking function to enable users to interact with each other or as an add-on function to integrate with common gay social apps, with a geosocial networking app for sexual encounters.

Effectiveness of adherence monitoring

The 33 DBS measurements from 33 participants were collected at the follow-up visits.

Overall, the median TFV-DP concentration was 821.8 (IQR: 411.9–984.8) fmol/punch, equivalent to an average of 4 doses taken per week. Twenty-three (66.7%) participants reported ED PrEP use and 10 (27.3%) reported daily PrEP use prior to the visit of DBS collection in the app. There was no significant difference in TFV-DP concentration between the ED and daily PrEP use; the medians of TFV-DP concentration were 821.8 (IQR: 439.1–961.2) and 984.6 (IQR: 763.9–1578.8), respectively (p-value: 0.145). One (3.0%) TFV-DP measurement was below the lower limit quantification (BLQ). FTC-TP concentrations were BLQ in 10 (30.3%) DBS measurements, meaning no dose intake within 48 h. Among these 10 DBS samples, 8 were from participants who reported ED PrEP use and 2 were from participants who reported daily PrEP use.

The median PrEP adherence in the app was 90%, with an IQR from 68.9% to 98.1%. Overall, there was a slight agreement between DBS measurement and the number of pills taken in the app (weighted Kappa: 0.21). According to the app data, during the 6 weeks, before the DBS measurements were collected, a median of 25 (IQR: 15–35) pills were taken. There were 22 participants (66.7%) who had inconsistency between the app data and the DBS measurements; 11 (33.3%) had higher DBS measurements than what the app data suggested; and 11 (33.3%) had higher pill numbers in the app data than DBS measurements (Table 5). Among 23 ED PrEP users, 14 (60.9%) had inconsistency between app data and DBS measurement: 8 (34.8%) had higher DBS measurement than app data and 6 (26.1%) had higher pill numbers in the app data than DBS measurement. Out of 10 daily PrEP users, 8 (80.0%) had inconsistency between app data and DBS measurement: 3 (30.0%) had higher DBS measurements than what the app data suggested and 5 (50.0%) had higher pill numbers in the app data than DBS measurements.

Table 5.

Comparing the self-report pills taken in the app and the estimated weekly average pills taken from dried blood spots (DBS).

	Self-reported pills taken in the app
Estimated weekly average pills taken from DBS		<2	2–3	4–6	7	Total
	<2	4	0	4	0	8
	2–3	3	0	5	0	8
	4–6	2	3	6	2	13
	7	0	0	3	1	4
	Total	9	3	18	3	33

Discussion

Principal findings

We presented a novel technology to assist MSM PrEP users to self-monitor their PrEP adherence and allow flexibility for the user's choice of an ED dosing regimen in real-world settings, with a high proportion of ED PrEP use and regimen switching. We assessed the usability and effectiveness of this user-centered innovative app with features of sexual event and pill diary and adherence visualization to assist MSM PrEP users. First, users reported that the UPrEPU app had an acceptable and above average usability level and was easy to use on their own. Second, regarding feasibility, participants showed high engagement and provided high numbers of sex, PrEP, and mood records in the app. Third, when compared to objective measurement of drug concentration, only slight agreement was observed between the DBS measurements and the number of pills taken and recorded in the app. Our study suggested that it is feasible for MSM to use a mobile phone app to manage their PrEP use and that the UPrEPU app is an acceptable tool in terms of technology usability; however, the potential for reporting bias using the app still exists. The app, like other technology developed to improve healthcare delivery, might be most helpful when accompanying other support, such as communicating with healthcare providers and users’ social network.

SUS being one of the most popular indexes for usability provides us with some information regarding how our app was perceived by the users. Other studies that applied an automated directly observed therapy approach in their app to improve PrEP adherence for daily MSM users reported having a median SUS score of 85 at week 4 and 80 at week 8,¹⁸ versus ours of 71.5 at month 1, a lower score but still in the acceptable and above average range. Other apps focusing on HIV prevention targeting different age ranges of MSM in the United States also have an above average SUS score,^42,43 showing an overall high acceptability across geographical regions and languages in applying technology in HIV prevention interventions, and have the potential to serve as a platform to apply artificial intelligence in the next step.⁴⁴

The app engagement remained high during the 4-month study period of our study. It is a key challenge for the majority of health-related apps since the level of engagement remains low or declines over time.^45–47 Several characteristics have been shown to enhance user engagement, including provision of personally tailored content, interactive components, social networking, and reminders.^48–51 Several common features were designed to improve adherence to PrEP, such as diary features with both a sexual activity log, medication log, and notifications.¹⁷ The application designed for PrEP adherence improved from other studies with those common features experienced reducing user retention over time.⁵² Apart from the shared common features, our app was designed to engage the users to self-monitor PrEP use, which was one characteristic to enhance retention.⁵³ We added a mood log to show the correlation and trend in mental health of the user, with data visualization of the patterns of sexual activities, PrEP use, and mood. In the UPrPEU app, the avatar features provided the users with a virtual identity that enabled users to record their sexual events in a relatively safe environment and served as a personally tailored feature to enhance engagement. A data visualization of the PrEP protection level in the app also assisted users to understand their current adherence patterns and reduce their anxiety toward sexual activities.⁵⁴

Our study found a low agreement between the self-reported pill number collected by the app and the weekly pill number estimated by the DBS measurement. There was no significant difference between the proportions of overreported and underreported groups. Both overreporting and underreporting pill numbers compared to DBS measurement happened in our sample, showing that a bias in using self-report data in an app to measure adherence can go either way. The low consistency between DBS and self-report in the app may be owing to both the incorrect self-report use in the app and noise in the DBS measurement. The underreported group may be individuals with low motivation for using the app to self-monitor their PrEP use. The main reasons for participants not using the app frequently included infrequent sexual activities and ED PrEP taking, low motivation, and not enough notifications from the app to remind the users to open the app. These reasons all led to not using our app and resulted in an underreported PrEP use compared to DBS measurement. On the other hand, overreporting individuals might be those influenced by social-desirability bias, which gives socially desirable responses instead of choosing responses that are reflective of their true state.⁵⁵

Additionally, our study sample was recruited in Taiwan and is different from the majority of study samples in studies that have assessed TFV-DP in DBS in relation to adherence to tenofovir alafenamide; such studies usually have limited race or ethnic representativeness.^40,41 Cautions are needed in terms of race/ethnicity and weight, which not only affect TFV-DP variation,^40,41,56 but also interact with the association between age and TFV-DP.⁵⁷ For example, African Americans had lower TFV-DP concentrations in DBS than White Americans which may be due to different genetic, diet, or environmental factors across races and ethnicities.⁴⁰ There were limited Asian MSM data for the estimated average of pills used by TFV-DP concentration obtained from DBS.⁵⁸ Future studies will be needed to establish the relationship between TFV-DP concentrations in DBS and adherence for Asian MSM besides samples from Thailand.^57,58

Objective measures, such as TFV-DP concentration in DBS, are generally assumed to be a more reliable measure of adherence. As demonstrated by the AMPrEP study,⁵² the concept of adherence measurement for ED PrEP use is much more complex than daily PrEP use since that study involved the timing of sexual events and the number of pills taken. Moreover, objective measurement usually requires specialized equipment and trained technicians, which would be an obstacle for monitoring PrEP in the long term. Accurate self-report or technology-assist self-monitoring of PrEP adherence should be developed and used parallel with other objective measurements.

A high percentage of sexual events were protected by PrEP in our study; however, a wide IQR suggested that there was high diversity between the participants. The protection during sexual events from ED PrEP varied in previous studies. A high proportion of correct PrEP use for high-risk sexual activities has been observed in the studies using an online diary as a data collection tool in the Netherlands and Belgium, and in the IPERGAY study in France.⁵⁹ Conversely, low adherence to ED PrEP was found among MSM and transgender women in Thailand and the United States in the HPTN 067/ADAPT study, and among MSM who sought sex partners via geosocial networking apps in China.^60–62 A high level of PrEP adherence in our study may be attributable to the flexibility of choosing PrEP dosing regimens and using an app. The possible explanations for the low adherence from the other two studies were that participants were assigned to one PrEP dosing regimen group without a cross-over in the HPTN 067/ADAPT study, and the participants had more frequent substance use in the Chinese study.

Conceptualization of PrEP adherence is shifting toward a prevention-effectiveness adherence (PEA) paradigm,⁶³ which emphasizes the consideration of other preventive measures such as condom use along with PrEP use when determining the level of protection from HIV infection. To assess PEA, comprehensive information on the timing of sexual activities, PrEP use, and the adoption of other HIV prevention strategies (e.g. condoms, serosorting, and undetectable viral load) is required. Studies have been unable to measure PEA thoroughly since such information has been hard to access without innovative electronic tools.^6,64 The UPrEPU app allows users to record their PrEP adherence patterns in the long-term and provides real-time PrEP protection levels with indications for use of PrEP for each sexual event, which enables users to better manage their risk of HIV.

Limitations

The present study included several limitations. First, we may have limited generalizability due to our strict inclusion ~~and exclusion~~ criteria. MSM recruited in this study had a relatively higher frequency of sexual behaviors compared to general MSM PrEP users in Taiwan.⁸ Highly educated people were overrepresented in this study because these people could adopt innovative health technology more easily. The feasibility and usability data may not apply to those MSM who have much less frequent sex or to those who are less tech-savvy. Second, in terms of the small sample size in our study, the only inferential statistics in this paper is weighted kappa and the kappa can be calculated for fairly small sample sizes (e.g. five participants).⁶⁵ The results in our study still need to be interpreted with caution given the relatively small sample size. Third, one major strength of the app is that it was designed to integrate ED PrEP users’ perspectives; however, routine daily PrEP users may be less motivated than ED PrEP users. Repeatedly logging daily pills may result in data entry fatigue. Fourth, we only measured DBS once in the study period. Multiple measurements may be needed to better examine the relationship between data of the number of pills used gathered from the app and the objective measurement of drug concentration.

Conclusion

This study demonstrated a user-centered designed app that accommodates both daily and ED PrEP use for MSM. We confirmed that complex behaviors such as different drug-dosing regimens that are contingent on sexual behaviors could be incorporated into the design of a mobile app. Even though adherence monitoring in the app was less effective, the app engagement remained high during the 4-month study period, showing the potential for such an app to support MSM to improve PrEP adherence regardless of the dosing regimen. While COVID-19 impeded access to PrEP services and changed the MSM's sexual lifestyles,^13,66 the UPrEPU app can not only assist PrEP users to self-manage their PrEP adherence regardless of their dosing regimens but also reduce the burden for healthcare providers by offering users personalized reminders to take PrEP based on their sexual lives. This can prevent PrEP case managers from answering repetitive questions and messages to some degree. Even though the mood at sexual events was rated positive in our app, this may not reflect the overall mental health among participants.

Future work for the UPrEPU app can incorporate more focus on addressing potential syndemic problems that include mental health, sexual health, and substance use. For example, a just-in-time adaptive intervention on the mobile app that aimed to provide interventions to improve mental health and to monitor substance use for MSM may further include the PrEP component to be a more comprehensive harm reduction tool.⁶⁷ Only a few apps have been attempted, and even fewer have evaluated the feasibility and usability of the app.⁶⁸ Future work should also include strategies to increase interest in retention and adoption of the app. Economic evaluation of mHealth interventions are crucial to explore the sustainability and cost-effectiveness of mHealth interventions.^69,70 Future studies on economic evaluation of mHealth interventions are needed to optimize such technology tools for HIV prevention.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076221102770 - Supplemental material for Usability and effectiveness of adherence monitoring of a mobile app designed to monitor and improve adherence to event-driven and daily HIV pre-exposure prophylaxis among men who have sex with men in Taiwan

Supplemental material, sj-docx-1-dhj-10.1177_20552076221102770 for Usability and effectiveness of adherence monitoring of a mobile app designed to monitor and improve adherence to event-driven and daily HIV pre-exposure prophylaxis among men who have sex with men in Taiwan by Huei-Jiuan Wu, Yi-Fang Yu, Stephane Wen-Wei Ku, Yuan-Chi Tseng, Chien-Wen Yuan, Chia-Wen Li, Po-Hsien Huang, Nai-Ying Ko, Peter L. Anderson and Carol Strong in Digital Health

Footnotes

Acknowledgments

The authors acknowledge and thank all participants involved in the design, development, and pilot study of the UPrEPU app. The MOST and the Taiwan CDC had no role in the design of the study and collection, analysis, or interpretation of data or in writing the manuscript. Part of the results were presented as a poster at the 11th IAS conference on HIV science, Berlin, Germany (abstract number: A-IAS2021-00046).

Contributorship

CS, HW, and YY conceived, designed, and performed the study. HW and YY contributed to data handling and analysis. HW and CS wrote the manuscript. HW, SK, YT, CY, CL, PH, NK, PA, and CS contributed to manuscript revision. All authors reviewed and approved the final manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The institutional review board of National Cheng Kung University Hospital approved this study (IRB #B-BR-107-076-T).

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Taiwan Ministry of Health and Welfare, Centers for Disease Control, and Taiwan Ministry of Science and Technology (grant numbers MOHW 109-CDC-C-114-000104, MOST 108-2636-B-006-004, MOST 109-2636-B-006-004, MOST 110-2636-B-006-011, and MOST 111-2636-B-006-011).

Guarantor

CS.

Informed consent

All study participants provided written and verbal informed consent to participate in this study.

Trial registration

ClinicalTrials.gov NCT04248790.

ORCID iDs

Yuan-Chi Tseng

Carol Strong

Supplemental material

Supplemental material for this article is available online.

References

Molina

Charreau

Spire

, et al. Efficacy, safety, and effect on sexual behaviour of on-demand pre-exposure prophylaxis for HIV in men who have sex with men: an observational cohort study. The Lancet HIV 2017; 4: e402–e410.

Grant

Anderson

McMahan

, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis 2014; 14: 820–829.

McCormack

Dunn

Desai

, et al. Pre-exposure prophylaxis to prevent the acquisition of HIV-1 infection (PROUD): effectiveness results from the pilot phase of a pragmatic open-label randomised trial. The Lancet 2016; 387: 53–60.

Molina

J-M

Capitant

Spire

, et al. On-demand preexposure prophylaxis in men at high risk for HIV-1 infection. N Engl J Med 2015; 373: 2237–2246.

Hoornenborg

Coyer

Achterbergh

RCA

, et al. Sexual behaviour and incidence of HIV and sexually transmitted infections among men who have sex with men using daily and event-driven pre-exposure prophylaxis in AMPrEP: 2 year results from a demonstration study. The Lancet HIV 2019; 6: e447–e455.

Reyniers

Nostlinger

Laga

, et al. Choosing between daily and event-driven pre-exposure prophylaxis: results of a Belgian PrEP demonstration project. J Acquir Immune Defic Syndr 2018; 79: 186–194.

, et al. Factors associated with preferred pre-exposure prophylaxis dosing regimen among men who have sex with men in real-world settings: a mixed-effect model analysis. AIDS Behav 2021; 25: 249–258.

Wen-Wei Ku

Chang

, et al. Imperfect adherence in real life: a prevention-effective perspective on adherence to daily and event-driven HIV pre-exposure prophylaxis among men who have sex with men - a prospective cohort study in Taiwan. J Int AIDS Soc 2021; 24: e25733.

Haberer

Baeten

Heffron

, et al. Self-reported adherence to pre-exposure prophylaxis (PrEP) and sexual behavior by text messaging: preliminary findings from the partners mobile adherence to PrEP study. In: IAPAC International Conference on HIV Treatment and Prevention Adherence 2014, pp.8–10.

10.

Haberer

. Current concepts for PrEP adherence in the PrEP revolution: from clinical trials to routine practice. Curr Opin HIV AIDS 2016; 11: 10–17.

11.

Kebaabetswe

Stirratt

McLellan-Lemal

, et al. Factors associated with adherence and concordance between measurement strategies in an HIV daily oral tenofovir/emtricitibine as pre-exposure prophylaxis (prep) clinical trial, Botswana, 2007–2010. AIDS Behav 2015; 19: 758–769.

12.

Koss

Hosek

Bacchetti

, et al. Comparison of measures of adherence to human immunodeficiency virus preexposure prophylaxis among adolescent and young men who have sex with men in the United States. Clin Infect Dis 2018; 66: 213–219.

13.

Chow

EPF

Hocking

Ong

, et al. Changing the use of HIV pre-exposure prophylaxis among men who have sex with men during the COVID-19 pandemic in Melbourne, Australia. Open Forum Infect Dis 2020; 7: ofaa275.

14.

Badawy

Barrera

Sinno

, et al. Text messaging and mobile phone apps as interventions to improve adherence in adolescents with chronic health conditions: a systematic review. JMIR Mhealth Uhealth 2017; 5: e66.

15.

Badawy

Kuhns

. Texting and mobile phone app interventions for improving adherence to preventive behavior in adolescents: a systematic review. JMIR Mhealth Uhealth 2017; 5: e50.

16.

Connelly

Faber

Rogers

, et al. Mobile applications that empower people to monitor their personal health. Elektrotech Inftech 2006; 123: 124.

17.

LaBelle

Strong

Tseng

Y-C

. Mhealth strategies to promote uptake and adherence to PrEP: a systematic review. In: Cham. Cham, Switzerland: Springer International Publishing, 2020, pp.99–113.

18.

Liu

Laborde

Coleman

, et al. DOT Diary: developing a novel mobile app using artificial intelligence and an electronic sexual diary to measure and support PrEP adherence among young men who have sex with men. AIDS Behav 2021; 25: 1001–1012.

19.

Finkenflügel

RNN

Hoornenborg

Achterbergh

RCA

, et al. A mobile application to collect daily data on preexposure prophylaxis adherence and sexual behavior among men who have sex with men: use over time and comparability with conventional data collection. Sex Transm Dis 2019; 46: 400–406.

20.

Whiteley

Mena

Craker

, et al. Creating a theoretically grounded gaming app to increase adherence to Pre-exposure prophylaxis: lessons from the development of the viral combat mobile phone game. JMIR Serious Games 2019; 7: e11861.

21.

Collins

Murphy

Strecher

. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007; 32: S112–S118.

22.

Collins

Dziak

. Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs. Psychol Methods 2009; 14: 202–224.

23.

Collins

Chakraborty

Murphy

, et al. Comparison of a phased experimental approach and a single randomized clinical trial for developing multicomponent behavioral interventions. Clin Trials 2009; 6: 5–15.

24.

Abras

Maloney-Krichmar

Preece

. User-centered design. Bainbridge, W Encyclopedia of Human-Computer Interaction Thousand Oaks: Sage Publications 2004; 37: 445–456.

25.

van Gemert-Pijnen

Nijland

van Limburg

, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011; 13: e111.

26.

Bainbridge

. Berkshire encyclopedia of human-computer interaction. Great Barrington, MA: Berkshire Publishing Group LLC, 2004.

27.

De Vito Dabbs

Myers

Mc Curry

, et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009; 27: 175–183.

28.

Hung

Y-C

. A Mobile application concept design to increase Pre-exposure prophylaxis adherence for men who have sex with men. Hsinchu: National Tsing Hua University, 2019.

29.

Nielsen

Molich

. Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems. Seattle: Association of Computing Machinery, 1990, pp.249–256.

30.

Chu

IY-H

SW-W

C-W

, et al. Taiwan guideline on oral pre-exposure prophylaxis for HIV prevention–2018 update. J Microbiol Immunol Infect 2020; 53: 1–10.

31.

Strong

Tseng

, et al. Mobile app (UPrEPU) to monitor adherence to pre-exposure prophylaxis in men who have sex with men: protocol for a user-centered approach to mobile app design and development. JMIR Res Protoc 2020; 9: e20360.

32.

Kroenke

Spitzer

Williams

. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001; 16: 606–613.

33.

Spitzer

Kroenke

Williams

, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166: 1092–1097.

34.

Bangor

Kortum

Miller

. The system usability scale (SUS): an empirical evaluation. Int J Hum Comput Interact 2008; 24: 574–594.

35.

Brooke

. SUS: A 'Quick and Dirty' Usability Scale. In: Usability Evaluation In Industry. Edited by Jordan PW, Thomas B, Weerdmeester BA, et al. London, UK: Taylor and Francis. 1996, pp.189–194.

36.

Bangor

Kortum

Miller

. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

37.

Y-F

H-J

SW-W

, et al. Condomless anal sex associated with heterogeneous profiles of HIV Pre-exposure prophylaxis use and sexual activities among men who have sex with men: a latent class analysis using sex diary data on a Mobile app. J Med Internet Res 2021; 23: e33877.

38.

Hall

Hume

Tazzyman

. Five degrees of happiness: effective Smiley Face Likert scales for evaluating with children. In: Proceedings of the 15th international conference on interaction design and children 2016, pp.311–321.

39.

Castillo-Mancilla

Zheng

Rower

, et al. Tenofovir, emtricitabine, and tenofovir diphosphate in dried blood spots for determining recent and cumulative drug exposure. AIDS Res Hum Retroviruses 2013; 29: 384–390.

40.

Anderson

Liu

Castillo-Mancilla

, et al. Intracellular tenofovir-diphosphate and emtricitabine-triphosphate in dried blood spots following directly observed therapy. Antimicrob Agents Chemother 2018; 62: e01710-17.

41.

Yager

Castillo-Mancilla

Ibrahim

, et al. Intracellular tenofovir-diphosphate and emtricitabine-triphosphate in dried blood spots following tenofovir alafenamide: the TAF-DBS study. J Acquir Immune Defic Syndr 2020; 84: 323–330.

42.

Sullivan

Driggers

Stekler

, et al. Usability and acceptability of a mobile comprehensive HIV prevention app for men who have sex with men: a pilot study. JMIR Mhealth Uhealth 2017; 5: e7199.

43.

Biello

Hill-Rorie

Valente

, et al. Development and evaluation of a mobile app designed to increase HIV testing and pre-exposure prophylaxis use among young men who have sex with men in the United States: open pilot trial. J Med Internet Res 2021; 23: e25107.

44.

Xiang

Fujimoto

, et al. Application of artificial intelligence and machine learning for HIV prevention interventions. The Lancet HIV 2022; 9: e54–e62. https://doi.org/10.1002/9781444316568.wiem02057.

45.

Torous

Nicholas

Larsen

, et al. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018; 21: 116–119.

46.

Forbes

Newton

Cantalapiedra Calvete

, et al. MEMPHIS: a smartphone app using psychological approaches for women with chronic pelvic pain presenting to gynaecology clinics: a randomised feasibility trial. BMJ Open 2020; 10: e030164.

47.

Torous

Michalak

O’Brien

. Digital health and engagement-looking behind the measures and methods. JAMA Netw Open 2020; 3: e2010918.

48.

Schneider

de Vries

Candel

, et al. Periodic email prompts to re-use an internet-delivered computer-tailored lifestyle program: influence of prompt content and timing. J Med Internet Res 2013; 15: e23.

49.

Davies

Corry

Van Itallie

, et al. Prospective associations between intervention components and website engagement in a publicly available physical activity website: the case of 10,000 steps Australia. J Med Internet Res 2012; 14: e4.

50.

Richardson

Buis

Janney

, et al. An online community improves adherence in an internet-mediated walking program. Part 1: results of a randomized controlled trial. J Med Internet Res 2010; 12: e71.

51.

Lewis

Williams

Dunsiger

, et al. User attitudes towards physical activity websites in a randomized controlled trial. Prev Med 2008; 47: 508–513.

52.

Jongen

Hoornenborg

van den Elshout

, et al. Adherence to event-driven HIV PrEP among men who have sex with men in Amsterdam, the Netherlands: analysis based on online diary data, 3-monthly questionnaires and intracellular TFV-DP. J Int AIDS Soc 2021; 24: e25708.

53.

Bort-Roig

Gilson

Puig-Ribera

, et al. Measuring and influencing physical activity with smartphone technology: a systematic review. Sports Med 2014; 44: 671–686.

54.

Huang

H-J

Strong

, et al. Unspeakable PrEP: a qualitative study of sexual communication, problematic integration, and uncertainty management among men who have sex with men in Taiwan. J Appl Commun Res 2019; 47: 611–627.

55.

Grimm

. Social desirability bias. Wiley International Encyclopedia of Marketing 2010.

56.

Ibrahim

Glidden

Grant

, et al. Letter to the editor: revisiting the pharmacological forgiveness of HIV Pre-exposure prophylaxis. AIDS Res Hum Retroviruses 2021; 37: 407–408.

57.

Peterson

Ibrahim

Anderson

, et al. A comparison of covariate selection techniques applied to pre-exposure prophylaxis (PrEP) drug concentration data in men and transgender women at risk for HIV. J Pharmacokinet Pharmacodyn 2021; 48: 655–669.

58.

Songtaweesin

Kawichai

Phanuphak

, et al. Youth-friendly services and a mobile phone application to promote adherence to pre-exposure prophylaxis among adolescent men who have sex with men and transgender women at-risk for HIV in Thailand: a randomized control trial. J Int AIDS Soc 2020; 23: e25564.

59.

Vuylsteke

Reyniers

De Baetselier

, et al. Daily and event-driven pre-exposure prophylaxis for men who have sex with men in Belgium: results of a prospective cohort measuring adherence, sexual behaviour and STI incidence. J Int AIDS Soc 2019; 22: e25407.

60.

Grant

Mannheimer

Hughes

, et al. Daily and nondaily oral preexposure prophylaxis in men and transgender women who have sex with men: the human immunodeficiency virus prevention trials network 067/ADAPT study. Clin Infect Dis 2018; 66: 1712–1721.

61.

Holtz

Chitwarakorn

Hughes

, et al. HPTN 067/ADAPT: correlates of sex-related pre-exposure prophylaxis adherence, Thai men who have sex with men, and transgender women, 2012–2013. J Acquir Immune Defic Syndr 2019; 82: e18–e26.

62.

Wang

Zhang

Chu

, et al. Risk-taking behaviors and adherence to HIV pre-exposure prophylaxis in users of geosocial networking apps: real-world, multicenter study. J Med Internet Res 2020; 22: e22388.

63.

Perski

Blandford

West

, et al. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017; 7: 254–267.

64.

Thompson

Badawy

Kuhns

. Medication adherence and technology-based interventions for adolescents with chronic health conditions: a few key considerations. J Med Internet Res 2017; 19: e8310.

65.

McHugh

. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22: 276–282.

66.

Reyniers

Rotsaert

Thunissen

, et al. Reduced sexual contacts with non-steady partners and less PrEP use among MSM in Belgium during the first weeks of the COVID-19 lockdown: results of an online survey. Sex Transm Infect 2020; 97: 414–419.

67.

Platteau

Herrijgers

de Wit

. Digital chemsex support and care: the potential of just-in-time adaptive interventions. Int J Drug Policy 2020; 85: 102927.

68.

Meiksin

Melendez-Torres

Falconer

, et al. Ehealth interventions to address sexual health, substance use, and mental health among men who have sex with men: systematic review and synthesis of process evaluations. J Med Internet Res 2021; 23: e22477.

69.

Badawy

Kuhns

. Economic evaluation of text-messaging and smartphone-based interventions to improve medication adherence in adolescents with chronic health conditions: a systematic review. JMIR Mhealth Uhealth 2016; 4: e121.

70.

Iribarren

Cato

Falzon

, et al. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PloS one 2017; 12: e0170581.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.88 MB