Abstract
The Internet is a commonly used medium for recruiting geographically dispersed, smaller populations quickly, such as young adult men-who-have-sex-with-men (YMSM). One approach to improve reach and representativeness is to employ multiple Internet platforms to recruit this hard-to-reach population. The utility of this approach has not been studied adequately, and its impact on the study sample recruited is not yet known. Using data from a study of 18- to 24-year-old HIV-uninfected, Black, Hispanic, and White United States (US) YMSM, this investigation compared advertising and enrollment metrics and participant characteristics of those recruited across Internet platforms. Of the 2,444 participants, their median age was 22 years old; 21% were Black, 37% Hispanic, and 42% White; 90% had been tested for HIV at least once in their life; and 87% reported prior condomless anal intercourse (CAI) with another man. There were noticeable differences across platforms in the number of people accessing the study website, meeting study eligibility requirements, consenting to participate, consenting to participate per day of advertising and per click, as well as costs of advertising per consented participant. Participants recruited also varied across platform by race/ethnicity, geographic area of residence in the US, health-care insurance status, years of formal education, history of HIV testing, and CAI by partner type and sexual positioning. The investigation results indicate that the Internet platforms used for recruitment significantly impact not only enrollment but also diversity and characteristics of the sample obtained and consequently, the observations and conclusions rendered.
The Internet is a medium commonly employed for recruiting harder-to-reach, geographically dispersed, smaller populations quickly (e.g., young adult men-who-have-sex-with-men [YMSM]). Internet-based recruitment has the advantages of ubiquity of the Internet across society, low costs, presence of websites designed for specific populations (e.g., sex seeking), and ability to enroll participants in a shorter time period than some other strategies. Internet recruitment strategies are in constant flux due to continuous changes in Internet platforms over time, varying popularity of social networking sites, and dissimilar recruitment abilities and advertisement policies across platforms. In addition, methodological limitations may threaten the internal and external validity of Internet-based studies, such as the recruitment methods utilized and low participation rates. Moreover, best practices on Internet-based research methodology have yet to be established. These and other concerns can affect the conclusions of studies of significant public health importance, such as investigations on HIV/AIDS among YMSM.
For studies that aim to recruit YMSM, there is no national database or registry from which to draw a randomly selected representative sample, nor is there a perfect method of assuring external validity of the sample obtained. Internet platforms may differ by mode of delivery (web-based vs. app-based), cater to different audiences (e.g., by sociodemographic characteristics and/or interests), and have different functions in YMSM lives (e.g., socializing vs. purely sex seeking). Thus, relying on a single platform for recruitment may create unintentional selection bias and lack of representativeness in a research study. One approach to improve reach, diversity, and representativeness is to employ multiple Internet platforms for recruitment. The utility of this approach has not been studied adequately, and its impact on the composition of the study sample recruited as well as on study results and conclusions is not yet known. Knowing the variations across platforms of participants recruited is imperative in assisting researchers investigating important public health topics and choosing platforms for Internet-based recruitment, and this informs efforts to improve Internet-based research methods.
This team of researchers recently completed an Internet-based study that aimed to understand YMSM HIV testing history and sexual risk-taking behaviors (Merchant R. C., 2017) HIV-uninfected Black, Hispanic, and White YMSM (18 to 24 year-olds) from across the United States were recruited specifically for that study because they are disproportionately affected by HIV in the United States (US) and collectively form a population for whom interventions are needed to reduce HIV acquisition. The objective of the current investigation reported in this manuscript was to examine the impact of using the multiple Internet platforms chosen for the parent study to recruit these YMSM in regard to recruitment, retention, and participant characteristics. To address knowledge deficits regarding recruitment and retention patterns among YMSM recruited online, the first objective was to compare advertising and enrollment metrics (e.g., time to recruit, clicks, costs) across Internet platforms used in the parent study. The second objective was to compare demographic characteristics, HIV testing history, and HIV sexual risk-taking behaviors of participants recruited across these platforms. The third objective was to learn if variations in participant characteristics of those recruited and the homogeneity of the sample by race/ethnicity varied by Internet platform. The ultimate aim of this investigation was to assess how the choice of Internet platform affects the participant population and data obtained so that best practices on Internet-based research involving YMSM and other groups eventually can be developed to improve and inform the quality of research methodology.
Method
Study Design
This investigation was a secondary analysis of enrollment and questionnaire responses from an anonymous, Internet-based survey of Black, Hispanic, and White YMSM recruited across multiple Internet platforms between August 2014 and December 2014. Data were collected as part of a larger parent study of the HIV testing histories and opinions about HIV testing methods of 18- to 24-year-old Black, Hispanic, and White YMSM (Merchant R. C., 2017). The hospital’s institutional review board approved the study.
Participant Recruitment
A variety of general and MSM-specific social media and other Internet platforms was chosen as venues for study recruitment for the parent study based on the popularity, target audiences, cost, advertising availability, and technical capabilities of these platforms. Participants were recruited from the Internet platforms Bender, BGCLive, Facebook, Grindr, Growlr, Pinterest, and Reddit (Table 1). Recruitment strategies varied by platform capabilities, which included targeted advertisements, pop-up advertisements, banner advertisements, and postings. The recruitment strategies included brief information about the study and a link to the study website where potential participants could receive more information. Recommended techniques (Pequegnat et al., 2007) were followed to reduce fraudulent recruitment. Participants were offered a lottery for a limited number of $100 gift cards to an online store as an incentive.
Descriptions of Internet Platforms.
Note. MSM = men who have sex with men; NA = not applicable.
Study Protocol
After accessing the study website, potential participants answered questions to verify their study eligibility. Eligible participants were 18- to 24-year-old men who self-identified as Black, Hispanic, or White; communicated in English or Spanish; currently living within the 50 states or Washington, DC; ever had anal intercourse with another male; and had never received an HIV-positive test result. Participants who provided consent were asked about their demographic characteristics, HIV testing history, and sexual HIV risk-taking behaviors. Study questionnaires were derived from prior research and evaluated through cognitive-based assessments and pilot testing, as described previously (Merchant R. C., 2017). Participants completed the questionnaire sequentially and could not skip sections. However, they could respond with “don’t know” or “refuse to answer,” or drop out of the study at any time, which resulted in a small amount of missing data.
Data Analysis
For the first objective, the following metrics were summarized for each Internet platform: days of recruitment, number who saw the advertisement (or equivalent), number of clicks on the advertisement; number who accessed the study website; number and percentage agreeing to be screened for study eligibility, study eligible or ineligible, consenting to participate in the study, remaining in the study through the final question on HIV sexual risk-taking, and dropping out; and the advertising costs for paid advertising (in US dollars). Recruitment was first compared as a function of those who accessed the study website, and then as a function of those agreeing to be screened and study eligible across Internet platforms using proportions with accompanying 95% Clopper-Pearson confidence intervals (CIs). The average cost per consent by Internet platform, the average number of consents per recruitment day, the average number of consents per advertisement click, and the proportion of users who consented after accessing the study website were calculated. Retention was measured as a function of those who completed the final question in the section about HIV sexual risk-taking among those who consented to participate. For the second objective, participant demographic characteristics, HIV testing history, and sexual HIV risk-taking behaviors for those recruited were summarized using the sample mean or median along with corresponding 95% CIs or interquartile ranges (IQRs) in aggregate and by Internet platform. For the third objective, separate comparisons by racial/ethnic group were performed. Differences among those recruited across Internet platforms were assessed by comparing 95% CIs. Missing data were not imputed.
Results
Recruitment, Retention, Yield, and Costs
Of the 14,269 people who accessed the study website, 11,564 (81%) agreed to be screened for study eligibility; 3,020 (26%) of these were study eligible and 2,444 (81%) of those study eligible consented to participate (Figure 1). Figure 1 provides a comparison of recruitment as a function of those accessing the study website through each Internet platform. As shown, the number of people accessing the study website varied greatly across platforms, while the proportion accessing the study eligibility screening questions was similar, except for Pinterest being significantly lower. Agreement to be screened was similar across most platforms, although proportions were slightly lower for some platforms and significantly lower for Pinterest. Study eligibility and consent to participate in the study as a function of accessing the Internet platforms varied considerably. The most common reasons for study ineligibility across platforms were age (43.5%) and self-reported HIV infection (10.4%), although study ineligibility reasons varied substantially across platforms (Table 2). When considering recruitment yield and completion as a function of those study eligible, Reddit and Grindr had higher proportions of eligible participants; in contrast, BGCLive and Pinterest had the lowest proportion of eligible participants (Supplemental Figure 1). Reddit users had the highest completion rate. Although the frequency varied across platforms, participants indicated using multiple other MSM-centric and general social media websites, including those also recruited for the study (Supplemental Table 1).
Recruitment, eligibility, consent and completion of survey as a function of accessing study website by Internet platform.
Reasons for Ineligibility by Internet Platform.
Note. All confidence intervals are exact binomial (Copper-Pearson) confidence intervals; CI = confidence interval. DC = District of Columbia
Table 3 depicts the yield and costs of recruitment across Internet platforms as a function of advertising reach. Nearly 40 times more Grindr users were reached compared to Growlr, yet similar numbers from these two social platforms accessed the study website. Accessing the study website from advertisements ranged from 4.3% (Bendr) to 41.3% (Grindr). Consents per day of advertising and per click were highest with Grindr. For the paid advertisements, costs per consent were highest for Facebook and lowest for Growlr. Supplemental Tables 2 to 5 provide additional detail about recruitment yield and cost when available for individual platforms.
Recruitment Method, Duration, Cost, and Estimated Reach for Each Internet Platform.
Note. NA = not applicable; USD = United States dollars.
Number of consents divided by paid recruitment duration (if available) or total recruitment duration. bSee supplementary material for a more detailed breakdown. cNot necessarily unique users.
Participant Demographic Characteristics
The median age of the 2,318 participants who completed the demographic characteristics section of the questionnaire was 22 years (IQR 20–23); 21% were Black, 37% Hispanic, and 42% White. Participants predominately were from the southern US, came from a medium or large city or surrounding suburb, had a primary care provider, had health-care insurance, had either received or were in the process of obtaining a university degree, and did not live alone (Table 4). When comparing demographic characteristics of those recruited (excluding Pinterest’s two participants) across Internet platforms, Reddit and Bendr had more White participants than Grindr and Growlr; BGCLive had more Black participants than all others; Growlr had more Hispanic participants than each site except Facebook; Growlr had more of those who lived in the western US than BGCLive and Grindr; Reddit had more individuals with health-care insurance than Bendr, BGCLive, Grindr, and Growlr; and Bendr had more individuals who had not completed high school or a general equivalency degree (GED) than Grindr, Growlr, and Reddit.
Participant Demographic Characteristics by Internet Platform.
Note. All confidence intervals are exact binomial (Copper-Pearson) confidence intervals. IQR = interquartile range; U.S. = United States; GED = general education development, CI = confidence interval.
HIV Testing History
Among the 2,239 participants completing the HIV testing history questions, most had previously been tested for HIV, typically within the past 6 months and between 2 or 3 times per year (Table 5). In terms of differences in HIV testing history across platforms, Reddit participants less frequently had been ever tested for HIV than participants from BGCLive, Grindr, and Growlr; and BGCLive participants were more likely to have been tested within the past month than Reddit participants.
Participant HIV Testing History by Internet Platform.
Note. All confidence intervals are exact binomial (Copper-Pearson) confidence intervals; CI = confidence interval.
Sexual HIV Risk-Taking Behaviors
Of the 2,101 participants completing the sexual HIV risk-taking questions, the majority of participants never had condomless intercourse with a woman while most previously had condomless anal intercourse (CAI) with a man, usually within the past 6 months (Table 6). Regarding differences in HIV sexual risk-taking across platforms, fewer Reddit participants ever had CAI than BGCLive, Grindr, and Growlr participants. Reddit participants had fewer main male sexual partners than Growlr, Grindr, and BGCLive participants. Supplemental Table 6 provides additional details about CAI HIV risk-taking behaviors by sexual positioning.
Participant Sexual HIV Risk-Taking Behavior History by Internet Platform.
Note. CAI = condomless anal intercourse; CI = confidence interval; NA = not applicable.
Differences in Participant Characteristics by Race/Ethnicity Across Internet Platforms
Supplemental Tables 7 to 15 portray comparisons of participant demographic characteristics, HIV testing history, and HIV sexual risk-taking across Internet platforms. For Black YMSM, there were more participants from the western US for Growlr than BGCLive, and more non-blood donation testing for BGCLive than Grindr. For Hispanic YMSM, there were more participants from the southern US for Grindr than Growlr, yet more western US participants from Growlr than Grindr. For White YMSM, there were more northeastern US participants from Growlr than Grindr, yet more midwestern US participants from Growlr than Grindr; more college or graduate students or graduates from Grindr than Growlr; and more who had condomless intercourse with women from Growlr than Grindr and Bendr.
Discussion
This investigation provides several useful insights into Internet-based research, particularly for HIV-related studies among YMSM. The findings first demonstrate that choice of the Internet platform for recruitment impacts practical aspects of the conduct of a study, notably, how many people are reached and from where, how long it takes to recruit, and the costs and effort of recruiting. These aspects undoubtedly are a function of platform popularity, novelty, viewership, coverage and reach, accessibility, and advertising (including format, quality, type, and when and for how long advertisements are displayed). Also probably important are the reasons the platform is accessed, which affects whether or not an advertisement for research is considered a nuisance (e.g., accessing the website to find an immediately available sexual partner vs. browsing for potential dates) and time devoted to exploring the platform, which likely affects consent for participation, study completion, and perhaps veracity of responses. The lesson for researchers considering which platforms to use is to explore the capabilities and features of platforms and how they might impact their recruiting efforts. Unfortunately, however, platform capabilities and features and their relationship to study data obtained might not be available due to a number of reasons, including proprietary restrictions, the platform not providing applicable metrics, the relevant information never having been collected, similar studies not having been performed, and the nature and topic of the study being conducted and the consequent data obtained (e.g., survey on HIV vs. substance abuse). In the meantime, metrics that can be made available from Internet-based investigations should be included in published research so that future researchers can make better-informed decisions.
Perhaps a more concerning finding of this investigation is how platform choice affects the study sample obtained. Variations in sample obtained across platforms could have an important impact on observed data and subsequent conclusions. For example, one might conclude that the prevalence of HIV testing among Black, Hispanic, and White YMSM is exceptionally high if the parent study only had sampled participants from BGCLive (94.5% ever tested); or one might believe the prevalence is much lower if participants had been recruited solely from Reddit (79.4% ever tested) or that CAI was less frequent if sampling only was from Bendr and not from Grindr or Growlr. Given the relationship of platform to the characteristics observed, one might argue that sampling from multiple and different types of platforms yields a more diverse sample of Black, Hispanic, and White HIV-negative YMSM in the United States. The diverse sample obtained might better reflect the true spectrum of HIV testing history and sexual risk-taking. However, representativeness of the underlying sample cannot be claimed with any level of certainty, given that there is no “central registry” of US YMSM against which to compare the sample obtained for the parent study, the low recruitment yield achieved from platforms that have large memberships (e.g., Pinterest), and the limits of advertising time and reach of some platforms (i.e., missed vital samples due to when and where advertising occurred). Further, it cannot be claimed that those recruited from a given Internet platform are truly representative of all those who use that website. Additionally, the nature of the study, incentives offered, length of the survey or intervention, participant interest, trust of the study sponsor (e.g., government vs. academic vs. community organization), and other factors influence study enrollment. It is also feasible that some YMSM might use several of the social media platforms, which could bias sampling. It is probable that most other Internet-based YMSM studies have the same limitations. Future research into methods that might achieve better representativeness of this population would assist in improving the validity of studies that answer important public health questions.
There are numerous published HIV-related studies involving MSM recruited through Internet platforms. Relatively few provided assessments of recruitment across Internet platforms. In 2010, the European MSM Internet Survey (EMIS) recruited participants across Europe through non-Internet sources, MSM-centric organizations, and multiple Internet platforms (Weatherburn et al., 2013). Of three pan-European MSM-focused websites, recruitment was greatest for PlanetRomeo (103,000 men recruited, 25 languages), much less for Manhunt/Manhunt Cares (12,000 men, 6 languages) and Gaydar (11,000 men), and recruitment varied within country by website. Of those who viewed the first survey question, 31.5% dropped out of the study. The study authors also noted that eligibility for inclusion in the study varied across the country of residence among those who accessed the study site, although they did not report this by Internet platform. Thériault et al. recruited MSM through advertising in South Australia in 2009 through the Internet (banner advertisement on sponsor website, Gaydar banner advertisement and chat rooms, Facebook advertising and posts) and non-Internet sources (gay newspaper advertisements; cards distributed and posters displayed at sex venues, a bar, a clinic, and an HIV/AIDS support service). These researchers observed that 95% of those who completed the first page of the survey completed the entire study (although there were missing data); 70% of the 243 participants came from the Gaydar banner advertisement web link and 6.3% through the website chat room, and Facebook had a click-through rate of 0.06% and yielded only 18 enrollments. Although only using a single Internet platform for recruitment (MySpace.com), using a banner advertisement sent to ≥18-year-old men who self-identified as gay, bisexual, or unsure on their profile, Sullivan et al. observed a lower click-through rate among Black (0.36%) and Hispanic (0.35%) than Whites, and higher click-through rates for those with more years of formal education and who self-identified as gay or bisexual (Sullivan, 2011). Of 9005 participants, 69% completed the 30-minute survey, and completion was greater among White (77%) than Hispanic (71%) and Black (66%) participants. These investigations concur with the findings from this current investigation of recruitment variations by Internet platform and race/ethnicity.
Of MSM-focused published investigations that compared participant characteristics (demographic characteristics, HIV testing history, sexual risk-taking behaviors, or other aspects) by recruitment source, several used single (Grov, 2012; Grov & Crow, 2012; Grov, Rendina, & Parsons, 2014; Hernandez-Romieu et al., 2014; Hospers, Kok, Harterink, & de Zwart, 2005; Mor & Dan, 2012; Saxton, Dickson, & Hughes, 2013) instead of multiple (Bolding, Davis, Hart, Sherr, & Elford, 2005; Elford, Bolding, Davis, Sherr, & Hart, 2004a, 2004b; Fernandez-Davila, Lupianez-Villanueva, & Zaragoza Lorca, 2012; Fernandez-Davila & Zaragoza Lorca, 2009; Leung, Poon, & Lee, 2015; Parsons, Vial, Starks, & Golub, 2013; Sanchez, Sineath, Kahle, Tregear, & Sullivan, 2015; Sanchez, Smith, Denson, Dinenno, & Lansky, 2012; Tsui & Lau, 2010; van den Boom et al., 2015; Vial, Starks, & Parsons, 2014; Zhang, Bi, Lv, Zhang, & Hiller, 2008) platforms for recruitment, unlike this current investigation. These studies often compared participants recruited via the Internet to those recruited in person at MSM-associated venues (e.g., bars, bathhouses, special events) or through other “off-line” methods. Studies using multiple platforms typically compiled participants into a single Internet group and did not compare participants recruited across individual platforms as done in this current investigation. For example, investigators from Hunter College contrasted adult MSM recruited via multiple MSM-targeted websites (e.g., gay, squirt, blackgaychat) and non-MSM-specific (e.g., Facebook) websites to those recruited using field-based (bars and clubs, bookstores, coffee shops, and street fairs) strategies in New York City (Parsons et al., 2013; Vial et al., 2014). Internet-recruited participants from all sources were more often older, White, HIV infected, and reported more frequent drug use and higher sexual risk-taking behaviors than field site-recruited participants; however, those recruited from MSM “dating/hookup” websites tended to be older, while those recruited from Facebook were more likely to use stimulant drugs. In 2010, van Dem Boom et al. compared MSM participants from sex and non-sex venues in The Netherlands to those recruited from six dating websites and two social network websites (van den Boom et al., 2015). Social network-recruited participants tended to be younger, had completed fewer years of formal education, and held different views about condom use than those from the dating websites. In the American Men’s Internet Survey, Sanchez et al. compared MSM recruited by four types of Internet platforms, which they termed gay social networking (2 websites), gay general interest (3 websites), general social networking (1 website), and geospatial social networking (1 website) Internet platforms (Sanchez et al., 2015). Participants recruited from the geospatial social networking website were less likely to be White and ≤40 years old, yet more likely to live in the southern US, in an urban area, and be HIV infected. In a meta-analysis of 14 studies comparing CAI prevalence of MSM recruited “online” versus “off-line,” Yang et al. observed widely discrepant CAI prevalence values for “online”-recruited MSM from 9.8% to 59.9%; they attributed these variations to individual study sample size, Internet recruiting platforms, global region, and definitions of CAI (Yang, Zhang, Dong, Jin, & Han, 2014). Although not comparing individual platforms, these aggregated data studies nevertheless support the finding that choice of Internet platform can greatly affect the collected data and conclusions drawn about MSM from the observations.
Limitations
This investigation had several limitations. The study population was not a random sample, and the quality of the sample cannot be verified. The results for each Internet platform cannot be generalized to the entire population or of that platform’s user base. Because Internet platforms are changing constantly with respect to advertising, popularity, user base, and other features, it is possible that future studies will find different results than those reported here. The inclusion of other Internet platforms, use of different advertising approaches, and other study procedures also might give dissimilar outcomes. While measures were taken to reduce fraud, it cannot be guaranteed that participants answered the survey questionnaire truthfully or accurately, or only once. Because the parent study was limited by the sample size obtained, larger sample sizes might have demonstrated significant differences among platforms that this current investigation could not detect. The use of 95% CIs to compare platforms can also reduce the ability to detect differences, since a common variance is not calculated between or among platforms. However, presenting pairwise comparisons between platforms would have been difficult to interpret due to the large number of platforms involved and because of greater chances for Type I errors from multiple comparisons. Although there was a small amount of missing data, missing values were not imputed. Missing data might have caused small errors in estimations and comparisons, but the impact should be minimal.
Conclusions
In conclusion, the results of this investigation make evident how Internet platform choice affects recruitment and participant diversity, which, in turn, impacts observations and conclusions. Researchers should carefully consider during the planning stage the recruitment needs for their investigation with respect to the available data about Internet platforms. Researchers also should make available detailed metrics about their Internet recruitment data as explicitly as possible to guide future researchers as to which Internet platform best suits their needs. Creating cost-effective, efficient, and standardized procedures for recruiting YMSM through Internet platforms only can be done through such transparency.
Footnotes
Acknowledgements
The authors gratefully recognize the assistance of Ms. Sarah Marks in the production of this manuscript and Mr. Ian Donaghy in the execution of this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a grant from the National Institute of Nursing Research (R21 NR023869). ClinicalTrials.gov Identifier: NCT02369627