Testing certain and uncertain incentives on study retention in a longitudinal social media survey among young adults: An embedded recruitment trial

Abstract

Introduction

Incentives can be effective in survey research but evidence is limited on how incentive type impacts survey retention in longitudinal social media-based surveys. This study examined how certain and uncertain incentives affect study retention among US young adults recruited online and whether incentive effects vary by sociodemographic factors.

Methods

Participants were randomized in a 1:1:1 ratio to a three-arm parallel trial (n = 1615) with (1) a lottery for a $200 gift card (uncertain), (2) a cash equivalent (CE) of a $5 gift card per survey (certain); or (3) a combination of both options (combined), and were surveyed at baseline, 30 days, and 60 days. This study focused on survey retention at 30 days (among baseline completers, n = 1491) and 60 days (among 30-day completers, n = 1018). Participants were not blinded to their condition but were blinded to other conditions and researchers were blinded until data collection was complete. Logistic regressions examined survey retention as a function of incentive condition and sociodemographics, with additional analyses of interaction effects. We report average marginal effects (AMEs) with significance defined as p < 0.05.

Results

The certain CE was effective for survey retention versus the lottery at 30-day follow-up only (43.8% [lottery] vs. 77.7% [CE], AME: 0.346, p < 0.000); there were no differences between CE versus lottery at 60-day follow-up (76.1% [lottery] and 81.3% [CE], AME: 0.054, p = 0.192). The combined incentive demonstrated significantly higher retention at both follow-ups versus the lottery but no significant advantage over the CE. Incentive effectiveness showed minimal variation across sociodemographic factors.

Discussion

This study is among the few to experimentally test incentives for retention in online social-media based research. A certain CE was most effective for short-term web survey retention among young adults compared with a lottery. Findings suggest that small guaranteed rewards may better motivate study retention than uncertain larger amounts.

Keywords

Incentives retention social media studies e-cigarettes young adults behavioral economics prospect theory

Introduction

The use of appropriate incentives in longitudinal research can enhance study retention and improve researchers’ ability to interpret study findings correctly.^1–4 Understanding incentive effects on retention is especially important for studies that are delivered fully online, including those conducted on social media platforms. Online research on these platforms can suffer from recruitment and retention challenges that may be partially due to low trust and lack of a relationship with the study team.^5,6 Yet such platforms can be effective for recruiting young adults, who can be hard to reach in more traditional channels such as by mail or phone.^5,7–10 Offering and delivering on incentives in social media studies may enhance retention by building trust and increasing the perceived benefits of research participation relative to the costs.¹¹

Much of the literature on incentives concerns cross-sectional studies, which often focus on recruitment.^1,2 This research finds that response rates increase with incentive value but gains taper off as incentives rise, cash or cash equivalents (CEs) are more effective than gifts and incentives can increase response from hard-to-reach groups.^1,2,12,13 Longitudinal research indicates that incentives can be effective but questions remain on how incentive type and structure may impact survey response over time and among hard-to-reach groups.¹

Evidence finds that both certain and uncertain incentives can increase participants’ willingness to participate in a study compared with no incentives.² Here we define certain incentives as guaranteed specific rewards, such as cash or CEs, provided only upon task completion (i.e. completing a survey) whereas uncertain incentives are rewards such as lotteries offered for task completion that are of unclear probability or value. When comparing certain versus uncertain incentives, prospect theory suggests and many studies confirm that individuals are more likely to choose a certain incentive in the realm of gains.¹⁴ In other words, a certain option is preferred over an uncertain option when individuals are considering receiving something of value in return for undertaking an act desired by others, such as participating in a survey. This research suggests that a certain cash incentive or CE in response to survey completion is likely to be the most effective for survey retention.

In contrast, theory on motivating-uncertainty effect suggests an alternative pattern of response that may be more relevant for survey retention over time.¹⁵ This theory posits that uncertain incentives may increase motivation to achieve a reward more than certain incentives when participants are working toward attaining the reward (i.e. goal attainment) as opposed to initially deciding whether to pursue the reward (i.e. goal setting).^16–18 This effect is attributed to greater feelings of positive arousal and motivation in the face of uncertainty during reward pursuit whereas initial decisions of whether to pursue a reward are dominated by outcome evaluation, which elicits a more deliberative assessment of how much effort and time will be required to achieve the reward.^15,19–24 This evidence suggests that uncertain incentives may be more effective than certain incentives for incentivizing study participants to achieve a goal over time, such as continuing to participate in a research study to obtain a reward, as opposed to deciding initially whether to participate or not. This theory then posits that uncertain incentives may be more effective than certain incentives for survey retention.

Uncertain incentives such as lotteries have become more common in recent years with web-based surveys.^25–33 While lotteries typically outperform having no incentive for initial recruitment, they are generally not as effective as certain monetary payments.^2,34 The research on lotteries compared with other incentives for retention in longitudinal web-based studies is limited and somewhat mixed. A 2006 meta-analysis of web-based surveys found that incentives improved study response by 19% and retention by 26%, with similar results for both the uncertain lottery incentive and the certain cash payment when each was compared against having no incentive.²⁷ More recent research from online panel surveys has produced varying results, with some studies suggesting lotteries mainly boost initial response rather than retention,^29,30 while others indicate lotteries can enhance retention more than initial response.^29,35 A recent web-based study of young adults found no significant differences in retention between a lottery and a certain CE payment.³² Evidence also suggests that lotteries can be cost efficient^29,33,35 and, in some cases, cost effective when they provide higher prize values.^25,26

Few studies have examined lotteries combined with other incentives for online surveys. Combining certain and uncertain incentives may leverage the strengths of both approaches by addressing different aspects of motivation. For example, to incentivize study participants to complete multiple waves of a survey, a certain incentive may support the continued action that is needed by providing a stable, predictable reward that provides a realized value for the effort made. In contrast, an uncertain incentive can maintain interest over time by stimulating the positive arousal generated by an unpredictable reward. One web-based study found that a lottery combined with a gift was most effective for initial survey response versus lottery alone, gift alone or no incentive.³⁶

Survey researchers have called for more evidence on the use of incentives in longitudinal surveys,^1,2 including for previously discouraged incentive types such as the lottery.^29,33,35,37 Theory and evidence suggest that uncertain incentives such as lotteries, may be a cost-efficient option for web-based studies,^29,33,35 even if they do not match the survey completion rates of certain incentives such as guaranteed cash. Research is limited on the use of uncertain incentives or approaches that combine certain and uncertain incentives to enhance retention in longitudinal studies, particularly for short-term opt-in panels recruited online and via social media.

Social media recruitment has been effective with young adults and other groups considered hard-to-reach and retain for survey research,^5,7–9 including males, racial/ethnic minorities, those with low-income and groups that may experience stigma or have privacy concerns, such as gender minorities,³⁸ lesbian, gay, bisexual, transgender (LGBT) individuals^39,40 and those who use substances, including tobacco.^41–43 Evidence is mixed on whether the effect of different types of incentives on response or retention vary by whether a group is hard to reach. Some online studies have found that lottery incentives may be more effective for men³⁵ while other suggest that differences in effectiveness between men and women may depend on the type of prize offered.⁴⁴ Other research finds that lotteries can improve response among young people⁴⁵ and cash can improve response among low-income groups.⁴⁶ Yet many studies have found no differences in response to different incentive types based on sociodemographic factors.^13,34,47

This study seeks to expand on prior research by examining how certain and uncertain incentives as well as a combined option affects study retention among young adults who have agreed to participate in a social media-based longitudinal study. Given the mixed and inconclusive evidence on varying effects of incentives types for different social and demographic groups, we will also conduct exploratory analyses to evaluate whether incentive effects on survey retention differ by participant sociodemographics for hard-to-reach groups.

Methods

Study design and sample

This study was an experiment embedded into a primary longitudinal randomized controlled trial (RCT) examining the effects of an e-cigarette prevention campaign on young adult knowledge, attitudes, and behaviors^48,49 (host: clinicaltrials.gov NCT04867668). Participants were recruited into the primary trial and randomized to separate doses of campaign messaging exposure. The primary RCT evaluated the effectiveness and dose–response effects of varying exposure to a 60-day online e-cigarette prevention social media campaign messaging intervention delivered through Facebook and Instagram on young adults’ vaping attitudes, beliefs and behaviors. The primary and embedded trial was conducted from 29 December 2021 through 15 April 2022 and included a baseline survey and two follow-up surveys that ran approximately 30 and 60 days after baseline.

For this embedded three-arm parallel incentive trial (1:1:1), study participants were randomized to incentive arms prior to baseline (n = 1615). This study focuses on the varying incentive effects on retention at 30-day follow-up (among baseline completers, n = 1491) and 60-day follow-up (among 30-day completers, n = 1018). We followed the CONSORT reporting guidelines for embedded recruitment trials.⁵⁰ The trial protocol is available from study author WDE.

Recruitment and data collection

The Virtual Labs platform (Virtual Labs LLC, https://vlab.digital)⁵¹ was used to disseminate the campaign, recruit participants and deliver surveys. Individuals were eligible for the host and embedded recruitment trials if they were aged 18–24, used Facebook or Instagram, spoke English, and lived in the United States. Potential participants were recruited online with ads placed in their Facebook or Instagram user feeds. Upon clicking on an ad, individuals were re-directed online to a Facebook Messenger chatbot, an online messaging application and platform, where they saw a welcome message and asked permission to chat. Participants who said yes to the chat were, through the chatbot, asked a series of screening questions, viewed a study description with IRB-approved study information, and asked for informed consent. Participants provided consent by clicking on a button to proceed with the study and were randomized to an incentive condition with an explanation of the condition.

A computer-generated randomization sequence using simple randomization with no restrictions was created by the Virtual Labs team prior to study launch to assign eligible participants to one of the three incentive conditions. The sequence was embedded in the Virtual Labs study system for automatic assignment. Participants were informed of their assigned condition as it was necessary for study participation but were unaware of the other conditions. Researchers remained blinded until after data collection.

The follow-up surveys were approximately 10 minutes (median completion time for 30-day = 9.02 minutes and 60-day = 8.15 minutes) and captured demographic questions, questions on attitudes and behavior regarding smoking and vaping and, for the follow-ups, perceptions of the campaign ads. The surveys were pilot tested and revised prior to formal rollout. This study was approved as not greater than minimal risk human subjects research by the Institutional Review Board (IRB) at George Washington University (IRB NCR202837).

Survey procedure

All surveys were disseminated within Facebook Messenger using the chatbot questionnaire. Participants were required to complete all questions to move forward in the survey and be considered complete for receipt of the incentive and for follow-up. All questions provided a “Prefer not to answer” option. Participants were contacted for follow-ups via email and Facebook Messenger, with information on the study, a request to complete the next survey, a reminder of their incentive condition and a request to continue. We sent one message via Facebook Messenger prior to each follow-up notifying participants that a new survey was coming soon.

Incentives: The three incentive arms for the study were (1) a lottery ticket to win a $200 Amazon gift card with completion of all three surveys (uncertain); (2) a CE of a $5 Amazon gift card for each survey, for a total of $15 if all three surveys were completed (certain); and (3) a combination of the lottery option plus the CE option from the prior condition (combined). Participants stayed in the same incentive arm across all surveys.

We selected $5 per survey for the CE condition as this amount aligned with our budget and was within the range of comparable research. Consistent evidence for the most effective value of an incentive for retention in comparable web-based studies is lacking. Amounts for a certain incentive conditioned on survey completion, such as that provided in this study, have ranged from $2 to $75 in recent research.^{13,32,41,52–54} Similarly, we selected $200 for the lottery condition as this was within our budget, was within range of other literature on lottery prizes, and aligned with evidence suggesting that lotteries of higher amounts may be more effective.^{25,26,28,31,33}

The CE was provided at each wave, the lottery was a single payment at the end of all three surveys and, for the combined arm, the CE was provided at each wave and the lottery at the end. Prior research on lottery prizes largely finds that participation increases if the lottery prize is provided in one lump sum rather than split into several smaller lottery prizes.^26,28,31,55

We did not have an experimental arm with no incentive because the primary purpose of the project was to recruit as many individuals as possible for the campaign messaging experiment. The sample size was determined based on the primary host trial.⁴⁸

Measures

The survey retention outcomes for the 30-day and 60-day follow-up analyses were measured as completion of the last page of the respective follow-up survey versus non-completion. We used demographic and tobacco use variables as covariates. Variables included age (18–21 vs. 22–24), gender (male/female/non-binary or other identity), race/ethnicity (White, African-American, Asian, Hispanic, Other), sexual orientation (heterosexual, gay/lesbian, bisexual/other), and subjective financial situation⁵⁶ (live comfortably, meet needs with a little left, just meet basic expenses, and don’t meet basic expenses). Current cigarette and e-cigarette use was highly correlated and was collapsed into a single variable defined as past 30-day cigarette or e-cigarette use versus no past 30-day use of either product. All independent variables also included a “Prefer not to answer” category.

Analysis

The analytic samples for the 30-day and 60-day follow-up analyses included the participants who had completed the respective prior survey. Specifically, the 30-day follow-up sample included those who completed the baseline survey (n = 1491) and the 60-day follow-up sample included those who completed the 30-day survey (n = 1018). We calculated percentages for the demographics and tobacco use variables among the two samples and conducted chi-square analysis to examine whether sociodemographic and tobacco use covariates differed for those retained versus those lost to follow-up at each wave. We also calculated survey retention rates at each follow-up overall and by incentive condition, which were defined as completion of the last page of a follow-up survey among the sample who completed the prior survey.

We conducted logistic regressions to examine survey retention for each follow-up survey, with incentive condition as the main predictor and all demographic and tobacco use covariates as independent variables. We adapted the reference group for incentive condition to test differences in retention between each incentive group for each follow-up. To improve interpretability of findings, we calculated and reported average marginal effects (AMEs) for the experimental conditions and covariates instead of odds ratios.⁵⁷ The AME shows the percentage point change in the predicted probability of the outcome as a function of the independent variable, averaged across all observations in the sample. We provide coefficients, standard errors and p-values (p < 0.05).

In a second set of regressions, we added product terms between incentive condition and sociodemographic characteristics to test whether the association between each condition and retention was significantly moderated by these characteristics. We show the difference in the AME of each incentive condition on the probability of retention for each sociodemographic group (i.e. the difference between the lottery condition, the CE and the CE + lottery among men, women and non-binary individuals, etc.). To test whether the incentive condition effects varied within sociodemographic groups, we report the significance of each pairwise contrast between each group (i.e. the significance of the difference in the AME of each incentive condition comparison between males vs. females, males vs. non-binary/other, females vs. non-binary, etc.).

Results

Demographics

The sociodemographic profile of participants remained consistent across both follow-up periods (see Appendix Table A1). In both samples, participants were predominantly female (approximately 70%), and non-Hispanic White (approximately 47%), with substantial representation from Asian (20%) and Hispanic (17–18%) participants. Most participants could meet their basic expenses, with only 6% unable to do so and 28% just meeting expenses. Regarding sexual orientation, the majority were heterosexual, approximately 20% were bisexual or another orientation, and 6% gay or lesbian. Approximately one-quarter were current tobacco or e-cigarette users. There were no significant differences in covariates for those retained versus those lost to follow-up at each wave. For incentives, the lottery-only group had the lowest proportion of participants by the time of the 30-day follow-up (21%) and 60-day follow-up (14%), while the CE and CE + lottery groups had similar proportions of participants (ranging between 39 and 44%) and proportions did not change substantially between follow-ups.

Figure 1 shows the retention rate of 68.3% for 30-day follow-up (among baseline survey completers) and 82.4% for the 60-day follow-up (among 30-day follow-up completers). Retention rates are lower for each incentive condition for the 30-day follow-up versus the 60-day follow-up, although the largest difference is for the lottery condition. Retention is 43.8% for the lottery condition at the 30-day follow-up and 76.1% at the 60-day follow-up, while retention for the CE and CE + lottery groups is similarly high at the 30-day follow-up (77.7% for CE and 72.0% for CE + lottery) and the 60-day follow-up (81.3% for CE and 85.7% for CE + lottery).

Figure 1.

Incentive conditions at each survey wave and survey retention rates for the 30- and 60-day follow-ups.

Table 1 shows AMEs from logistic regression models for retention for the 30- and 60-day follow-ups, with lottery as the incentive reference category and all sociodemographic characteristics as covariates. The model fit is low for both follow-ups as seen with McFadden's pseudo-R² but is higher for the 30-day follow-up. The low model fit indicates that many other unobserved factors influence retention but this is not a problem for this study with our focus on causal effects of incentives as compared with investigating reasons for participation in general. For the 30-day follow-up, models show that relative to the lottery incentive, the CE incentive comes with a 34.6 percentage point increase in the probability of retention, and this is highly statistically significant (p = 0.000). CE versus lottery increases the probability of retention by only 5.4 percentage points for the 60-day follow-up and this is not statistically significant. For the CE + lottery compared with lottery alone, the probability of retention is 29.0 percentage points higher for the 30-day follow-up and 10.0 percentage points higher for the 60-day follow-up and is statistically significant in both models. In models for both follow-ups, the sociodemographic and tobacco use variables did not have a significant effect on the probability of retention overall. We ran the same models with the CE as the reference category to compare CE + lottery versus CE (see Appendix Table A2). With CE as the reference, the CE + lottery condition produced a probability of retention 5.6 percentage points lower than CE in the 30-day follow-up and this was statistically significant. For the 60-day follow-up, the CE + lottery came with an increase of 4.6 percentage points in the probability of retention compared with CE alone but this was not statistically significant.

Table 1.

Average marginal effects of retention for 30-day (n = 1491) and 60-day (n = 1018) follow-ups as a function of incentive condition and sociodemographics (with lottery as reference for the experimental comparison).

	30-Day follow-up (n = 1491)			60-Day follow-up (n = 1018)
	AME			AME
	β	SE	p	β	SE	p
Lottery	–	–	–	–	–	–
CE	0.346	(0.033)	(0.000)	0.054	(0.041)	(0.192)
CE + lottery	0.290	(0.033)	(0.000)	0.100	(0.040)	(0.014)
Male	–	–	–	–	–	–
Female	0.050	(0.027)	(0.068)	−0.019	(0.028)	(0.490)
Non-bin/other	−0.079	(0.071)	(0.263)	−0.050	(0.077)	(0.519)
18–21	–	–	–	–	–	–
22–24	0.013	(0.023)	(0.585)	0.015	(0.024)	(0.522)
White, NH	–	–	–	–	–	–
AA, NH	−0.049	(0.046)	(0.285)	−0.093	(0.054)	(0.086)
Hispanic	0.009	(0.032)	(0.781)	0.016	(0.031)	(0.602)
Asian, NH	−0.003	(0.032)	(0.920)	−0.025	(0.034)	(0.453)
Other, NH	−0.085	(0.061)	(0.163)	−0.008	(0.063)	(0.893)
Pref no answer	−0.026	(0.057)	(0.649)	−0.019	(0.064)	(0.769)
Heterosexual	–	–	–	–	–	–
Gay/lesbian	0.038	(0.048)	(0.422)	−0.048	(0.054)	(0.376)
Bisexual/Oth	0.046	(0.030)	(0.129)	0.004	(0.030)	(0.901)
Pref no answer	−0.021	(0.040)	(0.598)	−0.015	(0.044)	(0.726)
Live comfort	–	–	–	–	–	–
Little left	0.010	(0.029)	(0.721)	−0.024	(0.030)	(0.423)
Just meet	0.005	(0.030)	(0.856)	−0.031	(0.031)	(0.326)
Don’t meet	0.003	(0.050)	(0.960)	0.026	(0.048)	(0.579)
No curr use	–	–	–	–	–	–
Current use	−0.055	(0.028)	(0.054)	0.004	(0.029)	(0.897)
N	1,491			1018
Pseudo-R²	0.068			0.017

AME: average marginal effect; CE: cash equivalent; Non-bin/other: non-binary/other; NH: non-Hispanic; AA: African-American; Live comfort: live comfortably, Little left: meet needs with a little left over; Just meet: just meet basic expenses; Don’t meet: don’t meet basic expenses; No curr use: no current cigarette or e-cigarette use; Current use: current cigarette or e-cigarette use; Pref no answer: prefer not to answer.

Tables 2 and 3 show results from the exploratory models for 30- and 60-day follow-ups, respectively, comparing the effects of incentive condition on survey retention by sociodemographic and tobacco use variables. We included product terms of each incentive type and one characteristic at a time, which produced six models for each outcome. For the 30-day follow-up (Table 2), we find no meaningful differences in the effect of incentive condition by race/ethnicity or age. Non-binary/other individuals and males have a significantly higher increase (61.0 and 45.0 percentage point increase, respectively; both ps < 0.05) in the probability of retention for the CE compared with the lottery condition, and both were significantly different than females, who had only a 29.0 percentage point increase (p < 0.05) for CE versus lottery. No significant differences emerged between gender groups in the effect of CE + lottery on 30-day follow-up retention compared to the lottery alone. However, the effect of CE + lottery versus CE on 30-day follow-up retention was significantly reduced for non-binary/other individuals relative to males and females. It is important to note that while we find significant effects for the non-binary/other group here, these should be interpreted cautiously given small cell sizes for that group in some conditions, which may result in less reliable estimates. There are some differences in incentive effects by sexual orientation. Among gay or lesbian participants, the CE condition produces a probability of retention that is significantly higher (by 56.5 percentage points, p < 0.05) than the lottery condition, and this is significantly different from bisexuals, for whom the CE condition produces a probability of retention that is only 25.9 percentage points higher (p < 0.05) than the lottery condition. Further, among gays or lesbians, the effect of CE + lottery on the probability of 30-day follow-up retention is significantly lower (by 29.9 percentage points, p < 0.05) than in the CE condition, and this is significantly different than the effect of CE + lottery versus CE among heterosexuals (3.6 percentage points lower, NS) and those who preferred not to answer this question (4.3 percentage points lower, NS). Lastly, the CE + lottery condition produced a significantly lower probability of retention versus the CE (25.0 percentage points lower, p < 0.05) for those who did not meet basic expenses, compared with the CE + lottery condition versus CE for those who met basic expenses and those who lived comfortably. For non-current tobacco users, the effect of the CE incentive on the probability of retention was 33.0 percentage points higher (p < 0.05) than the lottery effect, and this was significantly different from current tobacco users, for whom the probability of retention for the CE was only 13.3 (NS) percentage points higher than for the lottery. In the 60-day follow-up (Table 3), we find no meaningful differences in the effect of varying incentives on the probability of retention across any of the sociodemographic groups.

Table 2.

Average predicted probabilities of 30-day survey retention by incentive condition over sociodemographic characteristics, with average marginal effects (AME) of incentive group comparisons by sociodemographic group and pairwise effect contrasts (n = 1491).

	Lott	CE	CE + L	Diff. b/t CE and Lott (AME)	Contrasts^a	Diff. b/t CE + L and Lott (AME)	Contrasts^a	Diff. b/t CE + L and CE (AME)	Contrasts^a
a-Male	0.327	0.778	0.689	0.450*	b	0.362*	–	−0.088	c
b-Female	0.474	0.770	0.751	0.296*	a, c	0.277*	–	−0.019	c
c-Non-Bin/Other	0.299	0.908	0.381	0.610*	b	0.082	–	−0.528*	a, b
a-White/NH	0.485	0.785	0.705	0.300*	–	0.220*	–	−0.080*	–
b-AA/NH	0.306	0.830	0.656	0.524*	–	0.350*	–	−0.174*	–
c-Hispanic	0.406	0.794	0.762	0.388*	–	0.356*	–	−0.033	–
d-Asian/NH	0.433	0.753	0.762	0.320*	–	0.330*	–	0.009	–
e-Other/NH	0.549	0.694	0.589	0.145	–	0.039	f	−0.106	–
f-Prefer not to answer	0.281	0.762	0.778	0.481*	–	0.497*	e	0.016	–
a-18–21 YO	0.423	0.781	0.709	0.357*	–	0.285*	–	−0.072*	–
b-22–24 YO	0.455	0.772	0.732	0.317*	–	0.277*	–	−0.040	–
a-Heterosexual	0.430	0.757	0.721	0.327*	–	0.291*	–	−0.036	b
b-Gay/lesbian	0.357	0.922	0.693	0.565*	c	0.336*	–	−0.299*	a, d
c-Bi	0.565	0.824	0.702	0.259*	b	0.137	d	−0.122*	–
d-Prefer not to answer	0.321	0.713	0.756	0.392*	–	0.435*	c	0.043	b
a-Live comf	0.412	0.760	0.739	0.348*	–	0.327*	–	−0.021	d
b-A little left	0.695	0.799	0.471	0.329*	–	0.224*	–	−0.104*	–
c-Meet expenses	0.429	0.755	0.745	0.327*	–	0.317*	–	−0.010	d
d-Don’t meet expenses	0.435	0.875	0.624	0.440*	–	0.189	–	−0.250*	a, c
a-No current use	0.425	0.783	0.755	0.358*	–	0.330*	b	−0.029	–
b-Curr use	0.486	0.759	0.619	0.272*	–	0.133	a	−0.140*	–

Lott: lottery; CE: cash equivalent; CE + L: cash equivalent + lottery; Diff.: difference; AME: average marginal effect; Non-bin/Other: non-binary/other; NH: non-Hispanic; AA: African-American; Live comfort: live comfortably; Little left: meet needs with a little left over; Just meet: just meet basic expenses; Don’t meet: don’t meet basic expenses; No curr use: no current cigarette or e-cigarette use; Current use: current cigarette or e-cigarette use.

The “contrasts” column reports incentive condition comparisons that are significantly different across levels of each sociodemographic group (second differences).

*p < 0.05.

Table 3.

Average predicted probabilities of 60-day survey retention by incentive condition over sociodemographic characteristics, with average marginal effects (AME) of incentive group comparisons by sociodemographic group and pairwise effect contrasts (n = 1018).

	Lott	CE	CE + L	Diff. b/t CE and Lott (AME)	Contrasts^a	Diff. b/t CE + L and Lott (AME)	Contrasts^a	Diff. b/t CE + L and CE (AME)	Contrasts^a
a-Male	0.794	0.797	0.904	0.003	–	0.110	–	0.107*	–
b-Female	0.757	0.824	0.838	0.068	–	0.081	–	0.041	–
c-Non-Bin/Other	0.710	0.747	0.892	0.037	–	0.182	–	0.145	–
a-White, NH	0.785	0.815	0.869	0.030	–	0.084	–	0.054	–
b-AA, NH	0.749	0.749	0.744	−0.001	–	−0.005	–	−0.005	–
c-Hispanic	0.685	0.888	0.862	0.203	–	0.177	–	−0.027	f
d-Asian, NH	0.782	0.780	0.851	−0.002	–	0.069	–	0.071	–
e-Other, NH	0.818	0.837	0.813	0.019	–	−0.004	–	−0.023	–
f-Prefer not to answer	0.587	0.687	0.958	0.101	–	0.371		0.270	c
a-18–21 YO	0.746	0.817	0.839	0.072	–	0.094	–	0.022	–
b-22–24 YO	0.779	0.807	0.877	0.028	–	0.099	–	0.070	–
a-Heterosex	0.760	0.817	0.859	0.057	–	0.099*	–	0.043	–
b-Gay/lesbian	0.548	0.849	0.751	0.301	–	0.203	–	−0.097	–
c-Bi	0.820	0.797	0.871	-0.022	–	0.052	–	0.054	–
d-Prefer not to answer	0.716	0.786	0.864	0.070	–	0.148	–	0.078	–
a-Live comf	0.741	0.819	0.893	0.078	–	0.152*	–	0.074	–
b-A little left	0.793	0.793	0.843	0.000	–	0.050	–	0.049	–
c-Meet expenses	0.738	0.808	0.831	0.071	–	0.093	–	0.023	–
d-Don’t meet expenses	0.760	0.893	0.873	0.133	–	0.133	–	−0.020	–
a-No curr use	0.775	0.803	0.860	0.028	–	0.086	–	0.057*	–
b-Current use	0.715	0.843	0.844	0.128	–	0.129	–	0.001	–

Lott: lottery; CE: cash equivalent; CE + L: cash equivalent + lottery, Diff: difference; AME: average marginal effect; Non-bin/Other: non-binary/other; NH: non-Hispanic; AA: African-American; Live comfort: live comfortably; Little left: meet needs with a little left over; Just meet: just meet basic expenses; Don’t meet: don’t meet basic expenses; No curr use: no current cigarette or e-cigarette use; Current use: current cigarette or e-cigarette use.

The “contrasts” column reports incentive condition comparisons that are significantly different across levels of each sociodemographic group (second differences).

*p < 0.05.

Discussion

This study found that an uncertain lottery incentive was significantly less effective than a small certain incentive in the form of a CE or a combined CE + lottery incentive for the first follow-up of an online social media study among young adults. Compared with the lottery alone, the CE and CE + lottery conditions had an increase of 34.6 and 29.0 percentage points, respectively, in the probability of retention for the 30-day follow-up. At the 60-day and final follow-up wave, there were minimal differences in response to the different incentive conditions. Further, the combined CE + lottery incentive did not significantly improve upon the CE option in either follow-up. We found some significant differences in incentive response by gender identity, sexual orientation, perceived financial situation, and tobacco use at the 30-day follow-up but, overall, differences in incentive effects by sociodemographics were minimal.

The high retention rate observed for the CE incentive compared with the lottery at first follow-up indicated that the pattern predicted by prospect theory held. Further, the amount of just $5 per survey at all waves is promising for social media-based samples of young adults, as this amount is lower than incentives typically used in many online studies with comparable populations, which have ranged up to $75.^{13,32,41,52–54} This suggests that modest guaranteed incentives provided at each wave may be sufficient for maintaining participant engagement in a short longitudinal study in this demographic and may be particularly important for brief online surveys. Such an approach may enhance study legitimacy, which is an important factor in social media studies where trust in the study sponsor may be lower.

The uncertain lottery performed poorly at 30-day follow-up, which may reflect how the timing and structure of a lottery incentive can impact participant retention. This poor early retention aligns with our prior study finding the lottery incentive to be least effective for recruitment.⁴⁹ Despite prior research finding that a single large lottery payout can be more effective than the same amount split up into several smaller prizes,^26,28,31,55 it may be the case that smaller lottery payouts after each survey wave might have improved both initial recruitment⁴⁹ and 30-day retention by building trust and engagement, particularly given that participants were recruited through social media rather than from established online panels where participants are generally more familiar with the sponsor.^28,55 The underperformance may also be explained by research on repetition-based decisions, which indicates that uncertainty should be resolved at each repetition for an uncertain reward to maximize motivational impact through the uncertainty resolution utility, which is the positive feeling of moving from a state of uncertainty to certainty regardless of the outcome.^58,59 Of note, by the 60-day follow-up, both lottery conditions showed strong retention comparable to the CE-only option, which may be attributed to the motivating-uncertainty effect, whereby motivation and effort increase as participants move closer to potentially receiving the final prize.¹⁸ This trend however may have also been related to individual panelists’ general inclination to participate, with those who participated in earlier waves more likely to participate in later waves regardless of the incentive.

While we did not find large differences in the effect of incentives by sociodemographics, some findings shed light on groups that are traditionally hard-to-reach for survey research and retention. Males and marginalized or low-income groups are less likely to participate in research and can also be difficult to retain.^2,60 In this study, we did not see differences in the likelihood of survey retention overall for any sociodemographic group. However, the certain CE condition in the first follow-up was particularly effective for retention of males, non-binary or other individuals, gay and lesbian individuals, non-current tobacco users, and those who struggled to meet basic expenses. Although the sample was small, the findings on the effectiveness of the CE for individuals reporting non-binary or other identities provide some of the first evidence on retention strategies in social media-based research for a group that has been understudied in survey research. Although we found no significant differences in overall survey retention by current tobacco use status, the CE option appeared less effective for tobacco users compared to non-users when contrasted with the lottery incentive. While these findings are preliminary, future studies are needed to examine larger samples from demographically diverse groups and to include a broader assessment of survey retention strategies. Further, the effect of other individual characteristics that may be more important for retention should be considered, including trust in research and institutions as well as personality traits such as agreeableness and conscientiousness.

Limitations

The sample from this study is based on a convenience sample and may not generalize to a representative population of young adults or to young adults who do not use Facebook or Instagram. Findings may also be limited to shorter web surveys similar to the median time for the follow-ups in this trial (approximately 10 minutes). The study was powered for the larger e-cigarette messaging intervention in which this incentive trial was embedded and not specifically for the incentive study. Similarly, we did not include an arm with no incentive in this study as the primary purpose of the study was to recruit participants for the larger experimental messaging trial. We did not evaluate unconditional incentives, which are a type of incentive provided regardless of participation, but further research on the effectiveness and feasibility of unconditional incentives for retention in web-based studies recruiting from social media is needed.⁴¹ Research that examines varying CE amounts over extended follow-up periods could provide insight on the most effective amount as well as whether a repeat incentive is needed at each follow-up when a recruitment incentive is provided. Given the increased use and reduced cost of lotteries versus other options in web surveys, additional research is warranted on the use of lotteries for retention in longitudinal studies with short-term online opt-in panels, including experimental comparisons of lottery options with larger single payouts versus multiple smaller payouts and lotteries versus no incentive.

Conclusions

This study is among the few to experimentally examine the effectiveness of varying incentives for retaining study participants in a social media-based study that is delivered solely online. Findings showed that a certain CE of a small amount was the most effective strategy for short-term follow-up survey retention among a sample of US young adults when compared with the lottery, and the CE effect on retention was not improved by combining the CE with the lottery prize. The perceived benefit of a small but certain incentive may outweigh the time and effort of study participation for young adults over shorter periods of follow-up when compared with the uncertainty of a lottery. For social media-based studies, a certain option may also enhance trust in study legitimacy. Although differences in incentive effectiveness by sociodemographics were minimal, our results provide exploratory evidence for the importance of a certain CE incentive for retaining males, individuals of a minority gender identity or sexual orientation, those with the lowest perceived financial status and non-current tobacco users. The exponential rise in online social media-based studies in recent years^9,61 highlights the critical need for additional research to identify evidence-based strategies for study retention in these platforms.

Footnotes

Acknowledgements

The authors gratefully acknowledge the anonymous reviewers whose insightful comments and suggestions substantially improved the quality of this manuscript.

Contributorship

JC, WDE, JB, ECH, and PM contributed to the design of the study. ST, MM, and RG were involved in protocol development, IRB and technical logistics of the study. JC, PM, and APD cleaned and/or analyzed data. JC drafted the initial manuscript. All authors contributed to revisions and final approval of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This study was reviewed and approved by the George Washington University Institutional Review Board (NCR202837).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by a National Institutes of Health/National Cancer Institute grant #R01CA253013. APD, ST, ECH, and JMK are employees of Truth Initiative Schroeder Institute. PM is the owner of Biostatistics Health.

Guarantors

JC and WDE.

ORCID iDs

Jennifer Cantrell

Megumi Ichimiya

Alexander P D’Esterre

Jeffrey Bingenheimer

Shreya Tulsiani

Elizabeth C Hair

Madeline Martin

Raquel Gerard

William Douglas Evans

Appendix

Table A1.

Sample incentive condition and demographics for the sample for the 30-day and 60-day follow-ups.

	30-Day follow-up (n = 1491) (%)	60-Day follow-up (n = 1018) (%)
Gender
Male	27	27
Female	69	70
Non-binary/other identity	4	3
Age
18–21	55	55
22–24	45	45
Race and ethnicity
White, non-Hispanic	47	47
African-American, non-Hispanic	8	7
Hispanic	17	18
Asian, non-Hispanic	19	20
Other, non-Hispanic	4	4
Prefer not to answer	5	4
Sexual orientation
Straight/heterosexual	63	63
Gay/lesbian	6	6
Bisexual/another sexual orientation	20	21
Prefer not to answer	11	10
Financial situation
Live comfortably	33	33
Meet needs with a little left	33	33
Just meet basic expenses	28	28
Don’t meet basic expenses	6	6
Cigarette/e-cigarette use
No past 30 day cigarette or e-cig use	75	76
Past 30-day cigarette or e-cig use	25	24
Incentive condition
Lottery^a	21	14
CE^b	39	44
CE + lottery^c	40	42

CE: cash equivalent; E-cig: e-cigarette.

A lottery for a $200 Amazon gift card.

An Amazon gift card for $5 for each survey.

A combination of incentive options 1 and 2.

Table A2.

Average marginal effects of retention for 30-day (n=1,491) and 60-day (n=1,018) follow-ups as a function of incentive condition and sociodemographics (with CE as the reference condition for the experimental comparison).

	30-Day follow-up (n = 1491)			60-Day follow-up (n = 1018)
	β	SE	p	β	SE	p
Lottery	−0.345	0.033	0.000	−0.053	0.041	0.192
CE	–	–	–	–	–	–
CE + lottery	−0.056	0.025	0.025	0.046	0.025	0.065
Male	–	–	–	–	–	–
Female	0.050	(0.027)	(0.068)	−0.019	(0.028)	(0.490)
Non-bin/other	−0.079	(0.071)	(0.263)	−0.050	(0.077)	(0.519)
18–21	–	–	–	–	–	–
22–24	0.013	(0.023)	(0.585)	0.015	(0.024)	(0.522)
White, NH	–	–	–	–	–	–
AA, NH	−0.049	(0.046)	(0.285)	−0.093	(0.054)	(0.086)
Hispanic	0.009	(0.032)	(0.781)	0.016	(0.031)	(0.602)
Asian, NH	−0.003	(0.032)	(0.920)	−0.025	(0.034)	(0.453)
Other, NH	−0.085	(0.061)	(0.163)	−0.008	(0.063)	(0.893)
Pref no answer	−0.026	(0.057)	(0.649)	−0.019	(0.064)	(0.769)
Heterosexual	–	–	–	–	–	–
Gay/lesbian	0.038	(0.048)	(0.422)	−0.048	(0.054)	(0.376)
Bisexual/Oth	0.046	(0.030)	(0.129)	0.004	(0.030)	(0.901)
Pref no answer	−0.021	(0.040)	(0.598)	−0.015	(0.044)	(0.726)
Live comfort	–	–	–	–	–	–
Little left	0.010	(0.029)	(0.721)	−0.024	(0.030)	(0.423)
Just meet	0.005	(0.030)	(0.856)	−0.031	(0.031)	(0.326)
Don’t meet	0.003	(0.050)	(0.960)	0.026	(0.048)	(0.579)
No curr use	–	–	–	–	–	–
Current use	−0.055	(0.028)	(0.054)	0.004	(0.029)	(0.897)
N	1,491			1018
Pseudo-R²	0.068			0.017

AME: average marginal effect; CE: cash equivalent; Non-bin/other: non-binary/other; NH: non-Hispanic; AA: African-American; Live comfort: live comfortably; Little left: meet needs with a little left over; Just meet: just meet basic expenses; Don’t meet: don’t meet basic expenses; No curr use: no current cigarette or e-cigarette use; Current use: current cigarette or e-cigarette use; Pref no answer: prefer not to answer.

References

Laurie

Lynn

. The use of respondent incentives on longitudinal surveys. Report for the Institute for Social and Economic Research, Report No. 2008-42, University of Essex, Essex, UK, December 2008.

Nicolaas

Corteen

Davies

. The use of incentives to recruit and retain hard-to-get populations in longitudinal studies, https://www.ukri.org/wp-content/uploads/2020/06/ESRC-220311-NatCen-UseOfIncentivesRecruitRetainHardToGetPopulations-200611.pdf (May 31, 2019, 3 November 2023).

Fewtrell

Kennedy

Singhal

, et al.

How much loss to follow-up is acceptable in long-term randomised trials and prospective studies?

Arch Dis Child 2008; 93: 458–461.

Göritz

. Cash lotteries as incentives in online panels. Soc Sci Comput Rev 2006; 24: 445–459.

Lane

Armin

Gordon

. Online recruitment methods for web-based and mobile health studies: a review of the literature. J Med Internet Res 2015; 17: e183.

Kosinski

Matz

Gosling

, et al. Facebook as a research tool for the social sciences: opportunities, challenges, ethical considerations, and practical guidelines. Am Psychol 2015; 70: 543–556.

Darko

Kleib

Olson

. Social media use for research participant recruitment: integrative literature review. J Med Internet Res 2022; 24: e38015.

Ford

Albritton

Dunn

, et al. Youth study recruitment using paid advertising on Instagram, Snapchat, and Facebook: cross-sectional survey study. JMIR Public Health Surveill 2019; 5: e14080.

Whitaker

Stevelink

Fear

. The use of Facebook in recruiting participants for health research purposes: a systematic review. J Med Internet Res 2017; 19: e290.

10.

King

O'Rourke

DeLongis

. Social media recruitment and online data collection: a beginner’s guide and best practices for accessing low-prevalence and hard-to-reach populations. Can Psychol 2014; 55: 240–249.

11.

Blau

. Exchange power in social life. 2nd ed. New York: Taylor & Francis, 1986, pp. 89–97.

12.

Mercer

Caporaso

Cantor

, et al. How much gets you how much? Monetary incentives and response rates in household surveys. Public Opin Q 2015; 79: 105–129.

13.

Singer

. The use and effects of incentives in surveys. Ann Am Acad Pol Soc Sci 2012; 645: 112–141.

14.

Kahneman

Tversky

. Prospect theory: an analysis of decision under risk. Econometrica 1979; 47: 263–291.

15.

Shen

Fishbach

Hsee

. The motivating-uncertainty effect: uncertainty increases resource investment in the process of reward pursuit. J Consum Res 2015; 41: 1301–1315.

16.

Bagozzi

Dholakia

. Goal setting and goal striving in consumer behavior. J Mark 1999; 63: 19–32.

17.

Gollwitzer

. Mindset theory of action phases. In: Van Lange

Kruglanksi

Higgins

(eds) Handbook theories of social psychology. London: Sage, 2012, pp. 526–545.

18.

Toure-Tillery

Fishbach

. The course of motivation. J Consum Psychol 2011; 21: 414–423.

19.

Bar-Anan

Wilson

Gilbert

. The feeling of uncertainty intensifies affective reactions. Emotion 2009; 9: 123–127.

20.

Custers

Aarts

. Positive affect as implicit motivator: on the nonconscious operation of behavioral goals. J Pers Soc Psychol 2005; 89: 129–142.

21.

Erez

Isen

. The influence of positive affect on the components of expectancy motivation. J Appl Psychol 2002; 87: 1055–1067.

22.

Fishbach

Choi

. When thinking about goals undermines goal pursuit. Organ Behav Hum Decis Process 2012; 118: 99–107.

23.

Lee

Qiu

. When uncertainty brings pleasure: the role of prospect imageability and mental imagery. J Consum Res 2009; 36: 624–633.

24.

Wilson

Centerbar

Kermer

, et al. The pleasures of uncertainty: prolonging positive moods in ways people do not anticipate. J Pers Soc Psychol 2005; 88: 5–21.

25.

Coryn

CLS

Becho

Westine

, et al. Material incentives and other potential factors associated with response rates to internet surveys of American Evaluation Association members: findings from a randomized experiment. Am J Eval 2019; 41: 277–296.

26.

Gajic

Cameron

Hurley

. The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ 2012; 13: 789–799.

27.

Göritz

. Incentives in web studies: methodological issues and a review. Int J Internet Sci 2006; 1: 58–70.

28.

Göritz

Luthe

. Lotteries and study results in market research online panels. Int J Mark Res 2013; 55: 611–626.

29.

Göritz

Wolff

. Lotteries as incentives in longitudinal web studies. Soc Sci Comput Rev 2007; 25: 99–110.

30.

Luiten

. The effect of a lottery incentive on response, representativeness and data quality in the Dutch longitudinal Labour Force Survey. In: International workshop on household survey nonresponse, Budapest, Hungary, 22–24 August 2018.

31.

Sauermann

Roach

. Increasing web survey response rates in innovation research: an experimental study of static and dynamic contact design features. Res Policy 2013; 42: 273–286.

32.

Villanti

Vallencourt

West

, et al. Recruiting and retaining youth and young adults in the policy and communication evaluation (PACE) Vermont study: randomized controlled trial of participant compensation. J Med Internet Res 2020; 22: e18446.

33.

Witte

Schaurer

Schröder

, et al. Enhancing participation in probability-based online panels: two incentive experiments and their effects on response and panel recruitment. Soc Sci Comput Rev 2022; 41: 768–789.

34.

Pforr

Blohm

Blom

, et al. Are incentive effects on response rates and nonresponse bias in large-scale, face-to-face surveys generalizable to Germany? Evidence from ten experiments. Public Opin Q 2015; 79: 740–768.

35.

Göritz

Luthe

. Effects of lotteries on response behavior in online panels. Field Methods 2013; 25: 219–237.

36.

Cobanoglu

. The effect of incentives in web surveys: application and ethical considerations. Int J Mark Res 2003; 45: 1–13.

37.

Göritz

. The long-term effect of material incentives on participation in online panels. Field Methods 2008; 20: 211–225.

38.

Salk

Thoma

Choukas-Bradley

. The gender minority youth study: overview of methods and social media recruitment of a nationwide sample of U.S. cisgender and transgender adolescents. Arch Sex Behav 2020; 49: 2601–2610.

39.

Guillory

Wiant

Farrelly

, et al. Recruiting hard-to-reach populations for survey research: using Facebook and Instagram advertisements and in-person intercept in LGBT bars and nightclubs to recruit LGBT young adults. J Med Internet Res 2018; 20: e197.

40.

Hernandez-Romieu

Sullivan

Sanchez

, et al. The comparability of men who have sex with men recruited from venue-time-space sampling and Facebook: a cohort study. JMIR Res Protoc 2014; 3: e37.

41.

Crespi

Heller

Hardesty

, et al. Exploring different incentive structures among US adults who use e-cigarettes to optimize retention in longitudinal web-based surveys: case study. J Med Internet Res 2023; 25: e49354.

42.

Ramo

Prochaska

. Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. J Med Internet Res 2012; 14: e28.

43.

Ramo

Rodriguez

Chavez

, et al. Facebook recruitment of young adult smokers for a cessation trial: methods, metrics, and lessons learned. Internet Interv 2014; 1: 58–64.

44.

Laguilles

Williams

Saunders

. Can lottery incentives boost web survey response rates? Findings from four experiments. Res High Educ 2011; 52: 537–553.

45.

Oscarsson

Arkhede

. Effects of conditional incentives on response rate, non-response bias and measurement error in a high response-rate context. Int J Public Opin Res 2020; 32: 354–368.

46.

Becker

Moser

Glauser

. Cash vs. vouchers vs. gifts in web surveys of a mature panel study: main effects in a long-term incentives experiment across three panel waves. Soc Sci Res 2019; 81: 221–234.

47.

Stern

Fordyce

Carpenter

, et al. Evaluating the data quality of a national sample of young sexual and gender minorities recruited using social media: the influence of different design formats. Soc Sci Comput Rev 2022; 40: 663–677.

48.

Evans

Bingenheimer

Cantrell

, et al. Effects of a social media intervention on vaping intentions: randomized dose–response experiment. J Med Internet Res 2024; 26: e50741.

49.

Ichimiya

Muller-Tabanera

Cantrell

, et al. Evaluation of response to incentive recruitment strategies in a social media-based survey. Digit Health 2023; 9: 20552076231178430.

50.

Madurasinghe

Eldridge

. Guidelines for reporting embedded recruitment trials. Trials 2016; 17: 27.

51.

Tulsiani

Ichimiya

Gerard

, et al. Assessing the feasibility of studying awareness of a digital health campaign on Facebook: pilot study comparing young adult subsamples. JMIR Form Res 2022; 6: e37856.

52.

Hall

Sanchez

Stephenson

, et al. Randomised controlled trial of incentives to improve online survey completion among internet-using men who have sex with men. J Epidemiol Community Health 2019; 73: 156–161.

53.

Cantrell

Bennett

Thomas

, et al. It’s getting late: improving completion rates in a hard-to-reach sample. Survey Pract 2017; 11. DOI: https://doi.org/10.29115/sp-2018-0019.

54.

Asire

. Meta-analysis of effects of incentives on RR in online surveys . Master's Thesis, University of Denver, CO, 2017.

55.

Göritz

Luthe

. How do lotteries and study results influence response behavior in online panels? Soc Sci Comput Rev 2012; 31: 371–385.

56.

Williams

Smith

Villanti

, et al. Validity of a subjective financial situation measure to assess socioeconomic status in US young adults. J Public Health Manag Pract 2017; 23: 487–495.

57.

Mize

. Best practices for estimating, interpreting, and presenting nonlinear interaction effects. Sociol Sci 2019; 6: 81–117.

58.

Ruan

Hsee

Zoe

. The teasing effect: an underappreciated benefit of creating and resolving an uncertainty. J Mark Res 2018; 55: 556–570.

59.

Shen

Hsee

Talloen

, et al. The fun and function of uncertainty: uncertain incentives reinforce repetition decisions. J Consum Res 2019; 46: 69–81.

60.

Ryan

Lopian

, et al. It's not raining men: a mixed-methods study investigating methods of improving male recruitment to health behaviour research. BMC Public Health 2019; 19: 814.

61.

Bour

Ahne

Schmitz

, et al. The use of social media for health research purposes: scoping review. J Med Internet Res 2021; 23: e25736.