From hybrid to fully remote clinical trial amidst the COVID-19 pandemic: Strategies to promote recruitment,retention,and engagement in a randomized mHealth trial

Abstract

Clinical trials worldwide were disrupted when the COVID-19 pandemic began in early 2020. Most intervention trials moved to some form of remote implementation due to restrictions on in-person research activities. Although the proportion of remote trials is growing, they remain the vast minority of studies in part due to few successful examples. Our team transitioned Goals for Reaching Optimal Wellness (GROWell), an NIH-funded (R01NR017659) randomized control trial (RCT; ClinicalTrials.gov identifier NCT04449432) originally designed as a hybrid intervention, into a fully remote clinical trial. GROWell is a digital dietary intervention for people who enter pregnancy with overweight or obesity. Primary outcomes include gestational weight gain and six-month postpartum weight retention. Strategies that we have tested, refined, and deployed include: (a) use of a HIPAA-compliant, web-based participant recruitment and engagement platform; (b) use of a HIPAA-compliant digital health platform to disseminate GROWell and conduct study visits (c) interconnectivity of these two platforms for seamless recruitment, consent, enrollment, intervention delivery, follow-up, and study team blinding; (d) detailed SMS messages to address initial challenges with protocol adherence; (e) email notifications alerting the study team about missed participant surveys so they can follow-up; (f) remuneration using email gift cards with recipient choice of vendor; and (g) geotargeting social media campaigns to improve participation of Black Indigenous and People of Color Communities. These strategies have resulted in screen failure rates improving by 7%, study task adherence improving by an average of 20–30% across study visits, and study completion rates of 82%. Researchers may consider some or all of these approaches in future remote mHealth trials.

Keywords

remote clinical trials studies digital clinical trials mHealth psychology apps personalized medicine pregnancy medicine media obesity lifestyle behavior change diet digital health general

Introduction

When the COVID-19 pandemic began in early 2020, clinical trials all over the world were disrupted or delayed.^1–3 Most behavioral health intervention trials in the U.S. have had to pivot to fully remote intervention and evaluation protocols as a result of restrictions on in-person research activities.^4–6 Moving these trials into a fully remote format, however, was not without precedence. Prior to the COVID-19 pandemic, research had shown the promise and challenges with these pivots across all stages of research, including participant recruitment, intervention delivery, and data collection.⁷

One such recent innovation in remote trials is internet-mediated recruitment particularly using social media sites such as Facebook, Twitter, Instagram, and others. Studies have found that social media–based recruitment strategies can reach a massive audience relatively cheaply. As a result, these recruitment methods are rapidly gaining in popularity with those who conduct clinical evaluations.⁸ However, a recent review of the literature found that sampling representativeness can suffer when compared with traditional methods of recruitment.⁹ Although to remedy this, targeted recruitment for intervention trials and recruitment of “hard to reach” populations via social media can be especially effective.⁹ Other studies have found that there tends to be lower participation and adherence to study protocols in general with samples that are recruited online, especially if the intervention being evaluated is app-based^10,11 or utilizes passively collected data.¹² Despite these challenges, fully remote and internet-mediated recruitment strategies are likely to become even more prolific in the near future.⁸

Remote delivery of mHealth interventions also comes with its own set of strengths and challenges. For example, one of the primary strategies to increase adherence to health interventions and to study protocols is the incorporation of personalized feedback.¹³ In the case of fully remote and mobile app–based studies, this personalized feedback can plausibly be made automatic, eliminating delays and errors in feedback, reducing researcher burden, and increasing participant engagement with the study.¹⁴ However, current research is mixed on the effectiveness of this strategy. Some studies have found small positive effects on desirable data collection response behaviors,^15,16 while others have found no effects.^17–19

Finally, strategies for data collection in fully remote mHealth studies also have matured in the last 10 years. Active collection of participant data (e.g. collecting survey data within an mHealth app) is one such strategy. Studies again have shown mixed results for the effectiveness of these methods. Rangan et al. found that a real-time electronic dietary intake app showed good agreement with the established data collection methods (i.e. a 24-h diet recall survey).²⁰ In Zapata et al.'s study of an exercise-promotion app, however, researchers found that participants engaged in the promoted exercising behaviors consistently, but did not utilize the app as a companion, and in fact saw the app as a significant barrier to study adherence.²¹ Passive data collection (e.g. collecting ongoing data from a Bluetooth-enabled health device) likewise has both pros and cons. Bluetooth-enabled technology, including devices such as accelerometers, GPS tools, and remote heart rate monitors, has dramatically increased as a health behavior research tool.²² Internet-connected food scales for measuring portion sizes and food consumed have been found to be a feasible way to collect dietary information,²³ and adherence to daily weighing with a Bluetooth-enabled scale for those with heart failure has been shown to be quite high.²⁴ However, there are significant ethical issues with passively collected health data that have yet to be satisfactorily resolved.²⁵ Obtaining informed consent, data security, informational privacy, and equitable access—especially for vulnerable groups—are some of the major concerns.

Among the most challenging data collection issues that have arisen in fully remote studies has been the collection of biospecimens from participants. Some examples include the collection of blood,²⁶ saliva,²⁷ hair,²⁸ and breastmilk.²⁹ Traditionally, these data collection strategies have necessitated an in-person visit with a trained data collector, either in the participant's home or in a clinic setting.⁷ Further many of the aforementioned specimens require strict processing and storage requirements to perform required tests, such as timely centrifuging and storage on ice. It is possible for participants to appropriately self-collect biospecimens in their own home and send them to the research team,³⁰ and this has proven to be indispensable for health researchers during the current pandemic.³¹ Remote training of participants and regular reminders have been shown to be critical for adherence,^30,32 especially when reminders and training materials are culturally tailored to optimize recruitment, retention, and adherence.³³ Video- or multimedia-based training is a particularly recent innovation that has shown great promise. Allen et al., for example, found that multimedia-based training to collect dried blood spots was widely acceptable, feasible, and produced a high percentage of useable samples for daily smokers recruited on Facebook for a fully remote study.³⁴ However, unanswered questions remain about the feasibility and reliability of these methods for other populations.

Like many other researchers engaged in clinical trials,^4–6 our team had to pivot when the COVID-19 pandemic halted in-person studies. Our trial was already utilizing a hybrid approach and we were in the pre-recruitment phase when California issued the shelter-in-place order in March 2020.^35,36 Thus, we were able to transition to a fully remote clinical trial prior to enrolling our first participant. In this paper, we describe the process of making these transitions and lessons learned. We also discuss some of the challenges with this approach that we have remedied in part or in full.

Methods

Goals for Reaching Optimal Wellness (GROWell) is an NIH-funded (R01NR017659) randomized control trial (RCT; ClinicalTrials.gov identifier NCT04449432) to test a digital dietary intervention for improving diet quality during pregnancy and the first 6 months postpartum among people who enter pregnancy with overweight or obesity.³⁷ To be eligible for the study, participants must be 10–16 weeks gestation with a singleton, low-risk pregnancy, have a body mass index of 25–40 before pregnancy, living and receiving prenatal care in California, at least 12 months since a previous birth or pregnancy lasting longer than 20 weeks, non-smoking for at least 6 months, and without untreated depression or on a stable class and dose of antidepressants for at least 6 months. The study received institutional review board approval prior to study initiation and for all protocol amendments. Our primary outcomes include gestational weight gain and postpartum weight retention at 6 months post-birth. We compare the intervention to an attention control that provides pregnancy and early infancy education via weekly texts. Per our original protocol, we intended to consent in person and to conduct three in-person study visits (baseline, 36–38 weeks gestation, and six months postpartum) and two online study visits (26–28 weeks gestation and three months postpartum). With the onset of the COVID-19 restrictions on clinical research, we replaced all in-person procedures using two key technology platforms: StudyPages (Yuzu Labs Public Benefit Corporation, 2022) and Pattern Health (Durham, NC). See Figure 1 for the timeline of study activities and the effects of protocol changes. StudyPages is a HIPAA-compliant, web-based participant recruitment and engagement platform that helps academic institutions communicate about, connect with, and manage interactions with people interested in participating in research. StudyPages combine study landing pages and back-end participant management capabilities to deliver a high-touch participant interaction. Pattern Health is a digital health platform that enables clinicians and researchers to deliver and manage programs, including care plans, decisions, aids, and clinical outcomes assessments quickly and easily. StudyPages and Pattern Health use Represented State Transfer (REST) Application Programing Interfaces (APIs) to transfer data between the two systems. REST APIs enable StudyPages and PatternHealth to communicate with each other and act as a gateway between the two systems without saving client data between requests. This provides a seamless experience for the user/participant from recruitment to screening, consent, enrollment, and completion of all facets of the protocol.

Figure 1.

Protocol changes and associated improvements.

Recruitment and screening

Social media campaigns

Although the use of social media was always a component of our original recruitment plan, Facebook and Instagram ads became our primary means of recruitment when the pandemic halted in-person research. To this end, StudyPages developed a targeted outreach ad campaign to reach a defined audience based on demographics, location, and interests. Facebook and Instagram users who match this specific target audience may see the ad while scrolling and can click on the ad to be redirected to our landing page on StudyPages. These ads redirect prospective participants to our landing page on StudyPages (Yuzu Labs Public Benefit Corporation, 2021), the participant recruitment and engagement platform we employed for the study. To date, the average cost of the social media campaign per study participant is $105. Initially, we had static ads that used pictures representing the target population. However, to increase recruitment rates, we created animated gif advertisements to capture the attention of potential participants. We now use bright colors, sparkling stars and arrows, and movement to attempt to slow down the scrolling of social media users and have them click our study advertisement.³⁸ The impact of changing the study advertisements from static to animated was on average 224 more clicks per month on the advertisements and a 23% increase in consented participants.

Pre-recruitment

The enrollment and baseline window for participants is 10–16 weeks gestation. With the switch to remote, we adjusted our protocol to include the use of a Bluetooth scale for weight measurements (see below in the section on data collection), which we mailed to participants with instructions on use. In the early months of this approach, we noticed that many participants would sign up in the 16–18 week window, which ended up being too late for our team to mail the scale. Additionally, we noticed many prospective participants would fail the screen because they were less than 10 weeks gestation. To not lose these participants indefinitely, we enabled a “pre-screen” process in which prospective participants who are between 8 and 10 weeks gestation can be personally followed up when they become eligible at 10 weeks. We also remind potential participants to complete their screening procedures, as needed, to fully enroll participants by 16 weeks and get the scale to them as quickly as possible. These changes increased our eligibility rate from an average of 19% of those screened being eligible to 26% of those screened being eligible, thus reducing the screen fail rate from 81% to 74%.

Online screening

The StudyPages platform includes a landing page that describes the study, including inclusion and exclusion criteria, participation requirements, remuneration, and contact information, as well as a link to screen for eligibility. The system is automated and enables screening and enrollment 24/7 as opposed to only when clinical research coordinators (CRCs) are on the clock. After participants take the online screening survey, they receive an automated email that redirects them to sign the consent form. Landing page visitors also are able to share the study with others or take the next step towards study participation by completing a pre-screen and signing up online. When someone signs up, the team gets notified, and the contact information is stored in the study workspace. The workspace allows study teams to track and manage interested people, communicate with, engage, and retain potential and current study participants, and streamline research workflows while maintaining compliance. Study-specific features that GROWell employs include SMS text messaging, Voice Over Internet Protocol (VoIP) calling, and voicemail. In general, this process allows participants to autonomously review study materials and does not restrict study engagement to time or days when the study team is available.

Consent, enrollment, and randomization

Consent

In addition to recruitment and screening capabilities, the StudyPages platform provides a HIPAA-compliant online consent process, as well as consent to obtain medical records from the participants’ obstetric provider. After a person passes the pre-screen questionnaire, they have the opportunity to start the consent and medical release waiver process online. Also, an email is sent to allow the person to do this step at a later point in time. The completion of consent and waiver signature is tracked in the StudyPages workspace.

Enrollment and randomization

Upon completion of the consent, enrolled participants are transferred from StudyPages to Pattern Health via REST APIs. StudyPages uses the Pattern Health APIs to create a new user in the Pattern Health platform and transfers all data necessary for Pattern Health to randomize a participant. Once a user is created in Pattern Health, an email is generated inviting the user to download the Pattern Health app and continue the enrollment process.

Once the participant is appropriately linked to Pattern Health, they complete an initial demographic survey to confirm race/ethnicity and contact information, including the address to send the Bluetooth scale. Because BMI calculated using the weight from the study-provided Bluetooth scale is required for randomization, participants are randomized via the Pattern Health platform after linking their Bluetooth scale to the GROWell application and completing their baseline assessment.

Data collection

Weight

Given that gestational weight gain and postpartum weight retention are the key study outcomes, measuring weight accurately and reliably is critical. We elected to use high-quality, calibrated Bluetooth scales, which we mail to participants. The scale connects via mobile phone to the GROWell app, powered by Pattern Health. Although participants are directed to weigh themselves at each study visit, they may step on the scale when they choose (and many have), and GROWell captures these weights if their mobile phone is nearby. Although not initially part of our study design, regular weighing may impact results,³⁹ and we will assess this in our final data analysis.

Pregnancy and birth records

Given that we are recruiting throughout California, at the time of consent we have participants sign a records request for us to obtain their prenatal and delivery medical records. These data will be used to calibrate the weights we obtain using the Bluetooth scale, as well as to evaluate potential clinical mediators or moderators of outcomes.

Blinding

Our original protocol was designed as a double-blind study with different clinical research coordinators conducting initial and follow-up visits to retain blinding. With the shift to a fully remote trial, PatternHealth added support to the backend application for study personnel to remain blinded to randomization status. The PatternHealth console allows study teams to manage participation across users. In unblinded studies, the administrative team can see the participant's study arm, enrollment details, and their adherence to study tasks in addition to all study data collected. In order to support the GROWell study, PatternHealth implemented the ability to hide the study arm and all study data outside of the participants’ demographic details so that GROWell study team members remain blinded. However, it is important to keep the study team members informed of participants’ study progress for payment and follow-up purposes. To support the team, StudyPages implemented email notifications that alert the team when a participant is due for study payment for completing a milestone, or when a milestone is missed and a participant requires follow-up. These notifications enable the study team to manage participants without unblinding their data.

Adherence and retention

Given the lack of human contact in this study, we have been consistently monitoring adherence and retention and making minor adjustments to address issues that arise. To date, we have made two key changes. First, we noticed that some participants were confused about the specifics of what they were supposed to do at the start of the study, and this resulted in missed baseline surveys. To remediate this, we initiated a detailed SMS message introduction through the StudyPages workspace and a handout, both of which outline what study participation looks like. This has resulted in the majority (99%) of newly enrolled participants completing the demographics and baseline survey at the correct time.

As with any longitudinal study, completion of study visits can wane over time, and the additional lack of human connection may increase the likelihood of missed study visits and/or loss to follow-up. To improve rates of adherence and reduce attrition rates, we instituted an email reminder to the study team for each participant when the study visit is due, and when the study visit is missed. This has increased overall study task adherence by 22 percentage points to greater than 80% to date.

Consistent with most clinical research studies, GROWell employs participant remuneration that increases over time for the completion of online study visits. In the original protocol, participants were going to receive a check; however, this delays receipt and requires signed paperwork. Instead, we pay participants using an online gift card service, Tango Card, Inc., that allows participants to choose the vendor for which they would like to redeem their gift card. The service has more than 100 online vendors to choose from, and payments are released immediately when processed by the study team.

Inclusion of Black, Indigenous, and people of color communities (BIPOC)

Our recruitment goal for this study includes 50% White participants and 50% BIPOC. At approximately one-third of our sample size goal, we were recruiting approximately 65% White participants and 35% BIPOC participants. To address this gap, we initiated two key remedies. First, we are trialing the use of geotargeting with our social media ads. Specifically, we are increasing ads in zip codes that have a majority of residents who identify as BIPOC. Second, we have deployed GROWell in Spanish, given that in California, Spanish is the second most common language spoken after English, with approximately 25% of the state's population preferring Spanish as their primary language. We continue to track the effects of these adjustments on recruitment rates, and the most recent data show that we are now recruiting approximately 51% White participants, 44% BIPOC participants, and 5% who did not report.

Discussion

In the wake of the COVID-19 pandemic, we employed a number of unique strategies to conduct GROWell as a fully remotely trial. First, from recruitment through study completion, GROWell remains a primarily automated system, thereby increasing efficiency and access for potential and existing participants. Being able to have participants take an online screening survey, then receive an automated email that redirects them to sign the consent form is exponentially faster than scheduling an in-person visit and reviewing the consent form during that visit or over the phone and also relatively novel compared to many digital intervention studies that still require some human contact, even if the recruitment and/or consent are online or electronic.^40,41 Although still a relatively rare approach, we are not the first to have a fully automated and online recruitment and consent; this method has been shown to be a powerful way to access hard-to-reach populations.⁴² Second the system can be working on screening and enrollment 24/7 compared to only when clinical research coordinators (CRCs) are on the clock. CRCs can spend the time they would otherwise be recruiting and enrolling to focus instead on more time-intensive study tasks, like personalized follow-up. Third, text-message follow-up is quicker than only having the option to call or email each participant and increases reach in times when phone calls or email responses may not be feasible (e.g. participant is working or breastfeeding a baby). Fourth, a text-based welcome to the study that provides clear instructions about how to get started and outlines expectations for the study overall seems to increase participants’ engagement and willingness to complete future tasks assigned. Fifth, study staff can work from any location, thereby minimizing turnover and maintaining continuity. Sixth, participants can reside far from our medical center, thereby decentralizing this trial and expanding its reach to increase equitable access to studies for those who live far away or reside in more rural locations. Seventh, our remote trial helps to address time and travel barriers to participation by eliminating the need for participants to travel far distances to participate in the multiple study visits, which is especially difficult for those with transportation limitations and who visit the medical center infrequently. Relatedly, the reduced need to travel is a cost savings for the research team. Our hybrid approach would have required CRCs to travel to multiple university-affiliated clinics across a four-county region that spans approximately 145 square miles. Lastly, we anticipate that as we hone our online recruitment efforts and fully expand to Spanish-speaking participants, we will enroll a more diverse sample. This ultimately may lead to better overall representation in the study and more useful data that may deepen our understanding of the use of GROWell for improved dietary quality, pregnancy weight gain, and postpartum weight loss.

We also have identified some challenges that we are either in the process of remedying and/or will consider their roles in the post-implementation analysis. First, it is unclear whether participants are more or less likely to complete the online study visits when they are all remote/online. We are competing for attention during smartphone use with social media, games, and other user experiences that seek to hold their attention. That said, to date baseline and follow-up surveys are completed by participants 87% of the time on average, which is equivalent or better than anticipated for a fully remote trial.^4,43–45 Second, it is unclear the degree to which using the Bluetooth scale for weight measurements will impact our outcomes, because regular weighing may improve weight loss or weight loss maintenance.³⁹ We will need to consider this in our post hoc analyses. Third, some participants may view a fully remote trial as less personable and would prefer to interact face to face with study personnel.^46,47 However, other individuals may prefer this approach, and a qualitative exploration of those with better and/or worse adherence or those who remained versus dropped out will provide additional information. Fourth, only the most tech-savvy individuals could be enrolling in our trial, which means that we may be exacerbating the digital divide.⁴⁸ That said, the vast majority (upwards of 90%) of people of childbearing age have and regularly use mobile phones and other internet-capable devices, and the familiarity with technology is only increasing with every generation.^49,50 Therefore, this concern could be overstated and be less of an issue within 5–10 years. Fifth, while our application can be used either from cell phone tower or internet connections, technologies that require one or the other limit participation of certain subpopulations. Sixth, while mHealth is expanding exponentially because of the pandemic, data security is always a concern, and our team had to manage several layers of extra security given the large volume of the private health information we were receiving. Participants use a token and date of birth authentication method to log in to the application. In addition, users can set up a PIN, fingerprint, or face id before opening the app. All notifications to study team members communicating adherence/compliance updates provide context and a link to the users (e.g. a participant in the GROWell study has completed their 26–28-week visit), but no participant data are included to ensure that no data are transmitted via email. The issue of data security could skew our participant pool, given that some prospective participants may be wary of using digital technology for a study about their health and health behaviors. Lastly, concerns regarding ethical issues that arise from passive data collection, including ownership of data, secondary use of data that has been collected for a specific purpose, informational privacy, and equity of access should be considered.²⁵ Adherence to an ethical framework to ensure patient safety and protect patients’ rights is, therefore, an important feature of mHealth interventions and the clinical trials that evaluate them.

Conclusion

The lessons learned from the pivot of GROWell to a completely remote clinical trial in the wake of COVID-19 provide valuable information to the research community about implementing future remote clinical trials. Remote clinical trials have real potential to not only increase representation and reduce participant travel and study visit burden but also bring with them challenges in implementation and participant retention. Research teams should consider remote recruitment, intervention deployment strategies, personalized feedback, and remote data collection procedures at the study planning phase and tailor methods to best engage and meet the needs of the population of focus. When considering hybrid recruitment approaches to traditional in-person or completely remote trial procedures, personnel and cost factors also need to be considered. For example, in some cases, conducting a hybrid trial may be cost-prohibitive when considering the infrastructure needed for remote methodology and additional personnel for in-person work. Remote clinical trials, especially for mHealth studies, have the potential to expand access and improve health outcomes. The pros and cons of this approach should be weighed for all future trials to maximize the benefit to study participants and to ensure a broader representation of groups historically underrepresented in clinical research.

Footnotes

Acknowledgements

The authors would like to thank the staff of Yuzu Labs and PatternHealth for facilitating updates to the protocol using their platforms.

Conflict of interest:

KD is an employee of Yuzu Labs. JM is an employee of PatternHealth.

Contributorship:

LAS, JEP, and MW researched literature, conceived the paper, and wrote the first draft of the manuscript. All authors were involved in protocol development, gaining ethical approval, participant recruitment, intervention dissemination, and preliminary data analysis. All authors reviewed, edited, and added clarifying content to the manuscript and approved the final version.

Ethical approval:

The ethics committee of the University of California, Davis approved this study (IRB #1399548-8).

Funding:

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institutes of Health/National Institute of Nursing Research (grant number R01NR017659).

Guarantor:

LAS.

ORCID iD

Leigh Ann Simmons

References

Sathian

Asim

Banerjee

, et al. Impact of COVID-19 on clinical trials and clinical research: a systematic review. Nepal J Epidemiol 2020; 10: 878–887.

Tuttle

. Impact of the COVID-19 pandemic on clinical research. Nat Rev Nephrol 2020; 16: 562–564.

van Dorn

. COVID-19 and readjusting clinical trials. Lancet 2020; 396: 523–524.

Inverso

Abadula

Morrow

, et al. Pivoting during a pandemic: lessons learned from transitioning a multisite randomized controlled trial to a remote protocol in response to COVID-19. Transl Behav Med 2021; 11: 2187–2193.

Ramos-Gomez

Martin

Nelson

, et al. COVID-19 impact on community-based participatory randomized controlled trials—Lessons from the Oral Health Disparities in Children Consortium. Front Dent Med 2021; 2. DOI: 10.3389/fdmed.2021.671911.

Schnoll

Bernstein

Kaufman

, et al. COVID-19 challenges confronted by smoking cessation clinical trials for people living with HIV: the experience of grantees of the US national cancer institute. Nicotine Tob Res 2021; 23: 1629–1632.

Cummings

. Clinical trials without clinical sites. JAMA Intern Med 2021; 181: 680–684.

Gearhart

. Clinical trial recruitment using social media is growing. Quorum Rev 2015; 5: 1–2.

Topolovec-Vranic

Natarajan

. The use of social media in recruitment for medical research studies: a scoping review. J Med Internet Res 2016; 18: e286.

10.

Jäckle

Burton

Couper

, et al. Participation in a mobile app survey to collect expenditure data as part of a large-scale probability household panel: coverage and participation rates and biases. Surv Res Methods 2019; 13: 23–44. DOI: 10.18148/srm/2019.v1i1.7297.

11.

Kreuter

Haas

G-C

Keusch

, et al. Collecting survey and smartphone sensor data with an app: opportunities and challenges around privacy and informed consent. Soc Sci Comput Rev 2020; 38: 533–549.

12.

Wenz

Jäckle

Couper

. Willingness to use mobile technologies for data collection in a probability household panel. Surv Res Methods 2019; 13: 1–22. DOI: 10.18148/srm/2019.v1i1.7298.

13.

DiClemente

Marinilli

Singh

, et al. The role of feedback in the process of health behavior change. Am J Health Behav 2001; 25: 217–227.

14.

Rabbi

Pfammatter

Zhang

, et al. Automated personalized feedback for physical activity and dietary behavior change with mobile phones: a randomized controlled trial on adults. JMIR Mhealth Uhealth 2015; 3: e42.

15.

Balter

Fondell

, et al. Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology 2005; 16: 577–579.

16.

Balter

Fondell

Balter

. Feedback in web-based questionnaires as incentive to increase compliance in studies on lifestyle factors. Public Health Nutr 2012; 15: 982–988.

17.

Angelovska

Mavrikiou

. Can creative web survey questionnaire design improve the response quality? University of Amsterdam, AIAS Working Paper 131. 2013.

18.

Kühne

Kroh

. Personalized feedback in web surveys: does it affect Respondents’ motivation and data quality? Soc Sci Comput Rev 2018; 36: 744–755.

19.

Wenz

Jäckle

Burton

, et al. The effects of personalized feedback on participation and reporting in mobile app data collection. Soc Sci Comput Rev 2020; 0: 0894439320914261.

20.

Rangan

O’Connor

Giannelli

, et al. Electronic dietary intake assessment (e-DIA): comparison of a mobile phone digital entry app for dietary data collection with 24-hour dietary recalls. JMIR Mhealth Uhealth 2015; 3: e98.

21.

Zapata

Wang-Price

Fletcher

, et al. Factors influencing adherence to an app-based exercise program in adolescents with painful hyperkyphosis. Scoliosis Spinal Disord 2018; 13: 11.

22.

Onnela

Rauch

. Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology 2016; 41: 1691–1696.

23.

Maramis

Moulos

Ioakimidis

, et al. A smartphone application for semi-controlled collection of objective eating behavior data from multiple subjects. Comput Methods Programs Biomed 2020; 194: 105485.

24.

Haynes

Tancredi

Tong

, et al. Association of adherence to weight telemonitoring with health care use and death: a secondary analysis of a randomized clinical trial. JAMA Netw Open 2020; 3: e2010174.

25.

Maher

Senders

Hulsbergen

AFC

, et al. Passive data collection and use in healthcare: a systematic review of ethical issues. Int J Med Inform 2019; 129: 242–247.

26.

Lim

. Dried blood spots for global health diagnostics and surveillance: opportunities and challenges. Am J Trop Med Hyg 2018; 99: 256–265.

27.

Clements

. Salivary cortisol measurement in developmental research: where do we go from here? Dev Psychobiol 2013; 55: 205–220.

28.

Greff

MJE

Levine

Abuzgaia

, et al. Hair cortisol analysis: an update on methodological considerations and clinical applications. Clin Biochem 2019; 63: 1–9.

29.

Miller

Aiello

Fujita

, et al. Field and laboratory methods in human milk research. Am J Hum Biol 2013; 25: 1–11.

30.

Fernandes

Skinner

Woelfel

, et al. Implementing self-collection of biological specimens with a diverse sample. Field Methods 2013; 25. DOI: 10.1177/1525822X12453526.

31.

Bateman

. The development of patient-centric sampling as an enabling technology for clinical trials. Bioanalysis 2020; 12: 971–976.

32.

Uuskula

Kals

McNutt

. Assessing non-response to a mailed health survey including self-collection of biological material. Eur J Public Health 2011; 21: 538–542.

33.

Samayoa

Santoyo-Olsson

Escalera

, et al. Participant-centered strategies for overcoming barriers to biospecimen collection among Spanish-speaking Latina breast cancer survivors. Cancer Epidemiol Biomarkers Prev 2020; 29: 606–615.

34.

Allen

Lundeen

Murphy

, et al. Web-delivered multimedia training materials for the self-collection of dried blood spots: a formative project. JMIR Form Res 2018; 2: e11025.

35.

Eby

. Coronavirus timeline: tracking major moments of COVID-19 pandemic in San Francisco Bay area. ABC7 News. 2022. https://abc7news.com/timeline-of-coronavirus-us-covid-19-bay-area-sf/6047519/ (accessed 8 February 2022).

36.

Moreland

Herlihy

Tynan

, et al. Timing of state and territorial COVID-19 stay-at-home orders and changes in population movement—United States, March 1–May 31, 2020. MMWR Morb Mortal Wkly Rep 2020; 69: 1198–1203.

37.

Simmons

Phipps

Overstreet

, et al. Goals for reaching optimal wellness (GROWell): a clinical trial protocol of a digital dietary intervention for pregnant and postpartum people with prenatal overweight or obesity. Contemp Clin Trials 2022; 113: 106627.

38.

Bakhshi

Shamma

Kennedy

, et al. Fast, cheap, and good: why animated GIFs engage us. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose, California, USA: Association for Computing Machinery, 2016, pp. 575–586.

39.

Madigan

Daley

Lewis

, et al. Is self-weighing an effective tool for weight loss: a systematic literature review and meta-analysis. Int J Behav Nutr Phys Act 2015; 12: 104.

40.

Deegan

Fullen

Casey

M-B

, et al. Combined online interactive mindfulness and exercise programme (MOVE-online) compared with a self-management guide for adults with chronic pain: protocol for a randomised controlled feasibility trial. BMJ Open 2022; 12: e058265.

41.

McGregor

Sandhu

Bruce

, et al. Rehabilitation exercise and psycholoGical support after COVID-19 InfectioN’ (REGAIN): a structured summary of a study protocol for a randomised controlled trial. Trials 2021; 22: 8.

42.

Wozney

Turner

Rose-Davis

, et al. Facebook ads to the rescue? Recruiting a hard to reach population into an internet-based behavioral health intervention trial. Internet Interv 2019; 17: 100246.

43.

Pfammatter

Mitsos

Wang

, et al. Evaluating and improving recruitment and retention in an mHealth clinical trial: an example of iterating methods during a trial. Mhealth 2017; 3: 49.

44.

Fisher

Tuchman

Curreri

, et al. Transitioning from in-person to remote clinical research on depression and traumatic brain injury during the COVID-19 pandemic: study modifications and preliminary feasibility from a randomized controlled pilot study. JMIR Form Res 2021; 5: e28734.

45.

Watson

Mull

Heffner

, et al. Participant recruitment and retention in remote eHealth intervention trials: methods and lessons learned from a large randomized controlled trial of two web-based smoking interventions. J Med Internet Res 2018; 20: e10351.

46.

Haynes

Kamerman-Kretzmer

Khan

, et al. Telemedicine use for pediatric asthma care: a mixed methods study. J Asthma 2022: 1–10. DOI: 10.1080/02770903.2021.2019265.

47.

Haynes

Kompala

Neinstein

, et al. Disparities in telemedicine use for subspecialty diabetes care during COVID-19 shelter-in-place orders. J Diabetes Sci Technol 2021; 15: 986–992.

48.

Clare

. Telehealth and the digital divide as a social determinant of health during the COVID-19 pandemic. Netw Model Anal Health Inform Bioinform 2021; 10: 26.

49.

Perrin

Atske

. 7% of Americans don’t use the internet. Who are they? 2021. https://www.pewresearch.org/fact-tank/2021/04/02/7-of-americans-dont-use-the-internet-who-are-they/ (accessed 8 February 2022).

50.

Tsetsi

Rains

. Smartphone internet access and use: extending the digital divide and usage gap. Mob Media Commun 2017; 5: 239–255.