Abstract
Objective
The objective was to recruit 800 participants for a research study examining online health information exposure and engagement. Removing duplicate and/or fraudulent responses emerged as a major challenge during data collection.
Methods
Most recruitment took place online through U.S. national volunteer registries. Participants were required to share Facebook timeline data and take an online survey. We describe initial indicators of fraud and our process to create a prevention and exclusion protocol. We added three automatic prevention methods: CAPTCHA, blocking users with Virtual Private Network (VPN) and non-U.S. IP addresses, and requiring at least three years of Facebook timeline history. Additionally, we manually reviewed all responses, checking payment information, demographics, and IP addresses.
Results
During the two months that the VPN/IP block was live, 7% of entries were blocked for using a VPN or having an IP address outside the U.S. 7% of consented participants who shared Facebook data were automatically excluded based on having insufficient timeline post history. Two months after implementing the VPN check, fraudulent enrollments stopped. Ultimately, 9.6% of enrollments were excluded based on suspected fraud.
Conclusion
Other than the Facebook post history check the VPN/IP block had the largest impact on preventing misrepresentative and/or fraudulent entries. We share three strategies for online recruitment: create and test a protocol for preventing and excluding fraud before data collection starts; review participants before payment; and add a VPN block to the beginning of your survey.
Keywords
Introduction
Executing appropriate recruitment remains a serious challenge for researchers. Today’s digital era (e.g., online forums, social media, websites) has drastically changed the landscape and increases the potential to expand outreach to individuals across demographics and geographies: as of 2024, 96% of American adults use the internet 1 and 91% own a smartphone. 2 While some groups (based on age, income, and race) are less likely to use the internet and/or own a smartphone,1,2 usage represents a broad span of Americans and thoughtful online recruitment methods may still reach more diverse participants than offline methods.3,4
The reach of online social networks can also help health-related researchers connect with diverse communities. 5 Online recruitment methods can include outreach through volunteer registries, survey panels administered through companies like Prolific or Qualtrics, and direct advertising through social media. In addition to being more finely targeted to reach populations of interest, online methods are promoted as requiring less funding and human time. 6
As research has increasingly been conducted online, misrepresentation and fraud have emerged as a serious concern, threatening data quality, validity, and accuracy of results. The literature categorizes misrepresentation and fraud in a few ways, including instances where participants submit multiple surveys accidentally or take steps to hide their identity in an attempt to submit multiple responses for additional research incentives. 7 Fraudulent responses may be individuals who take the survey multiple times, “bots” or software programmed to automatically complete surveys rapidly, or a combination of bot software and human intervention.8–10 Studies have reported fraudulent (or suspected fraudulent) response rates between 3% and 94% 11 ; the wide variation in methods and frequency demonstrates the need for further information to understand this issue.
We describe our experience recruiting for a large, online study requiring participants to share their Facebook post history and complete surveys. Despite precautions prior to data collection, we encountered high rates of suspected fraud, requiring us to pivot in an attempt to identify, prevent, and exclude fraudulent responses. This effort required time and resources beyond our original assumptions. Our experience exposes tensions between data quality, data collection ethics, and emerging technology. We share this experience and lessons learned to counter assumptions about online recruitment with the hope that future researchers can better anticipate and prevent misrepresentative and/or fraudulent responses.
Study objective
The objective was to recruit a large national sample for a research study examining online health information exposure and engagement, an issue affecting communities across the U.S. Our goal was to develop a participant-centered study recruitment process and lower barriers to participation.
A key feature of this study was to create and deploy a study infrastructure designed to be completed quickly and independently by participants entirely online. By creating a simple screening, consent, social media data sharing, and survey design, we aimed to maximize ease of participation and minimize identifiable information collected with a fair incentive paid electronically to participants.
Methods
Between May 2023 and May 2025, this study aimed to recruit 800 Black and White Americans living in urban and rural communities. The purpose of the study was to investigate individual perceptions towards online health information and misinformation through longitudinal surveys and Facebook timeline posts, with a subset of the sample also sharing Google and YouTube search history data. If participants were eligible, shared sufficient Facebook data, and completed the baseline survey, they would earn $25 through Venmo, PayPal, or a Visa gift card paid through ClinCard.
Participants were recruited mostly online, with the majority of outreach sent through online volunteer databases (e.g., ResearchMatch); online ads posted to social media (Facebook, Pinterest) were used later in the recruitment process as well as in-person methods such as posting flyers in rural health clinics. During the period examined in this article, recruitment occurred through ResearchMatch and an institutional (university health system) registry, both established online volunteer databases designed for broad online research recruitment. Importantly, initially most volunteers in the institutional database were patients in the university health system who wanted to participate in university-run research studies.
The study eligibility criteria included: participants must be 18 or over, Black or White, live in a rural or urban area, and have an active Facebook account. This initial screening survey was hosted on Qualtrics. Individuals who met these criteria and consented to participate would share their contact and payment information on Qualtrics and then run their Facebook data through an app we built to quickly assess that they had sufficient Facebook data. This app immediately checked the number of words posted to ensure participants had enough words for language analysis, which was a core outcome of the study. At this stage, our data collection pipeline checked for duplicates by comparing Facebook IDs, email addresses and phone numbers provided for contact, and email addresses or phone numbers provided for payment through Venmo or PayPal. Any duplicate response was removed. If the participant had sufficient data, they would be redirected to Qualtrics again to complete the baseline survey and share Google/YouTube data.
Initial indicators of potential misrepresentation and/or fraud
Quickly after beginning online recruitment through volunteer registries, a major challenge emerged: identifying unique, genuine enrollments from participants who met eligibility criteria and removing duplicate and/or fraudulent responses. Within three months, the team identified an inflow of potentially misrepresentative or fraudulent enrollments. The first clue was a cluster of incoming emails to the study inbox requesting immediate payment. All emails were sent within a one-week span, had similar text patterns, including grammatical errors, and similar email address formatting. Due to this additional contact outside the survey, we suspected potential misrepresentation or fraud rather than simply low quality data. Following this, we began investigating individual enrollments to identify potentially misrepresentative or fraudulent enrollments and prevent these entries going forward.
Review process to identify potential misrepresentation and/or fraud
First, we implemented a series of manual checks to identify potentially misrepresentative and/or fraudulent entries (see Figure 1) [insert Figure 1]. Two team members worked closely together to review responses. All potentially misrepresentative and/or fraudulent entries were then reviewed and discussed with one of two co-investigators. This collaborative manual review and pattern identification process allowed the team to note two major flags for potentially misrepresentative and/or fraudulent entries: Facebook post history and possible virtual private network (VPN) use. VPNs provide a secure and encrypted connection between a user’s device and a remote server operated by the VPN provider. This masks the user’s true IP address and can make it appear as though they are browsing from a different location, thereby enhancing privacy and anonymity online; it can also make it appear as though different IP addresses, and therefore different unique individuals, are submitting responses. Manual review process to identify patterns in flagged entries.
From Facebook post history, it was clear that multiple entries flagged as suspected fraud were using Facebook accounts with a post history of 1-2 days prior to enrollment. These accounts also contained long entries that appeared to be pasted from Wikipedia every couple minutes to meet preexisting word count requirements (500 words or more were required). Additionally, regardless of self-reported zip code, almost all flagged entries’ IP addresses were from New York City or Los Angeles. This, in combination with many entries having IP addresses outside the U.S., led the team to suspect VPN use.
Verification process: Final protocol
Following these manual checks, we implemented three additional automatic checks to prevent potentially fraudulent entries. 1. 2. 3.
These automatic checks were used in combination with an abbreviated, ongoing manual review to check the following. • • •
Two team members reviewed flagged responses; any suspected fraudulent responses were removed following discussion with an investigator.
Trial and error: Inconclusive indicators of fraud
Finalizing a protocol for participant verification required trial and error as several initial verification checks were too generic and risked excluding too many valid entries. We addressed this by using several indicators as flags, but not relying on them exclusively for removal. For example, we identified multiple responses with different ages reported in the survey and volunteer database. As age was not an eligibility criterion for this study (beyond being over 18), we made a decision not to exclude based on this inconsistency if it was the only flag. Similarly, another pattern identified was a difference between names reported in the survey vs. volunteer database. However, as using different names is common practice, this was also not used as an exclusion criterion on its own.
Some patterns identified within misrepresentative and/or fraudulent entries were also very common practices for valid entries; below are examples of patterns that almost all misrepresentative or fraudulent entries followed, but are also very common. • All suspected misrepresentative/fraudulent enrollments signed up with email addresses that followed the format “First Name Last Name Number” - but this is commonly used by many people, so it became a review criterion, but not necessarily an exclusion criterion. • All suspected misrepresentative/fraudulent enrollments selected PayPal for receiving their incentive. While we chose to use this as an indicator for review, we did not automatically exclude participants for selecting it or remove PayPal as a payment method as it is broadly used: 47% of valid participants selected it. • All suspected misrepresentative/fraudulent enrollments provided different contact and PayPal email addresses - but as many people have multiple email addresses associated with different accounts for different purposes, we used it as a criterion for review, but not necessarily exclusion. • Many suspected misrepresentative/fraudulent enrollments had differences in self-reported zip code and IP address location, with many reporting a variety of zip code locations but IP address location was almost always New York City or Los Angeles. However, traveling or moving are common reasons for a mismatch between these two fields, so this was not used to exclude participants on its own.
We faced significant challenges in identifying true indicators of fraudulent entries and adjusting the protocol to remove fraudulent entries without removing too many valid entries. When we first began addressing the issue of fraudulent enrollments, we likely removed valid participants. While inconsistencies in personal information between volunteer databases and survey entries may point to duplicate or misrepresentative enrollments, individuals may also choose to hide personal information to protect their privacy: for example, reporting a slightly different zip code or date of birth (age). It can be helpful to distinguish between criteria for review and exclusion.
Moreover, as we began excluding responses, suspicious response patterns changed, indicating the need for continuous review. This dynamic of changing patterns, coupled with subtle indicators of fraudulent responses, raised the likelihood of overexclusion and removal of legitimate participants. To counter this, when we excluded responses, we sent an email notifying the participant and allowing them to get in touch with us directly. We did not receive any credible responses, although we did receive a few emails requesting immediate payment, sometimes to other payment methods not offered by the study, and all containing the same text and email address formatting patterns.
Results
At baseline, there were four study activities to assess eligibility, consent, and collect data: 1) an initial screening survey to assess participant-level eligibility; 2) a consent form; 3) a Facebook data share with automatic eligibility checks to ensure participants had sufficient post history; 4) a baseline survey. To be considered “enrolled,” participants had to complete all four activities.
Six months passed between study launch and implementation of a finalized, successful fraud prevention protocol (illustrated in Figure 2). As described above, our fraud prevention protocol developed over four months. Study timeline showing fraud detection and prevention measures over six months.
Recruitment progressed rapidly during this period (see Figure 3 for detailed recruitment metrics). Results indicate that the automatic prevention protocols were successful. During the two months that the VPN/IP block was live in this period, 7% of entries (102 out of 1,415 total entries) were blocked for using a VPN or having an IP address outside the U.S. Individual outcomes at all stages of the study enrollment process over the six months fraud prevention protocols were implemented.
Additionally, 7% (42 out of 577 total participants during this time period) of consented participants who shared Facebook data were automatically excluded following implementation of the date check. 122 participants were ineligible based on word count (must be ≥ 500); during our manual reviews, we found that these participants were not consistently flagged for being misrepresentative or fraudulent. Low word count could be an indicator of simply posting to Facebook less often, which made it a less reliable prevention tool.
Finally, 55 participants (or 9.6% of enrollments) were flagged for review post-enrollment and excluded based on suspected misrepresentation/fraud, with the majority (43) of these participants enrolling pre-VPN check.
Discussion
Research in online recruitment and fraud prevention is emerging, with teams employing different strategies and reporting varying results, making it difficult to recommend an effective protocol. 11 Our experience underscores this finding, highlighting the urgency of identifying effective fraud prevention and exclusion protocols to ensure reliable data quality. Researchers are working to address this issue by evaluating and comparing different methods 8 and creating tools such as conceptual frameworks 7 and decision trees 13 to approach online data collection with intention. Future research must evaluate prevention and exclusion methods as well as report fraudulent response rates and protocols to increase transparency. 7
Additionally, studies report higher rates of fraudulent responses as compensation amounts increase, 11 and higher rates with direct compensation versus lottery-style compensation12,13 or donations to a charitable organization. 12 This pattern suggests incentives as a motivation for submitting fraudulent responses. Researchers have identified fraudulent responses originating from outside the country or area of interest, 11 noting that study compensation may incentivize fraud from countries where the exchange rate may make fraud especially worthwhile.9,14 Understanding motivations and structural inequalities leading to fraudulent responses may help prevent it.
Importantly, while online recruitment methods are promising, both in terms of cost as well as reaching specific communities, our experience and the literature highlight that ensuring data quality is time-consuming and requires adequate planning and resources. 15 Researchers must be aware of the risks involved in online data collection and be prepared to dedicate time and resources to continuous monitoring, which is required to prevent and exclude fraudulent responses. Throughout recruitment for our study, finding the balance of sensitivity and specificity for flagging and excluding potentially fraudulent entries remained a challenge. Our goal was to keep true participants enrolled and build robust methods for preventing and excluding fraud. Without collecting extensive, increasingly sensitive individual-level data (e.g., social security number) or requiring manual screening points, differentiating between valid and fraudulent or misrepresentative enrollments required significant attention.
Recommendations for research teams using online recruitment methods
Based on our experience, we recommend three strategies for research teams recruiting participants online.
Online recruitment literature separates fraud protection protocols into pre-data collection (prevention) and post-data collection (exclusion) methods.7,8 Studies suggest that a multi-step approach is necessary, especially as automation becomes more sophisticated and human-bot collaboration more common.8,10 A study evaluating the effectiveness of 22 prevention and detection tests concluded that protocols must incorporate multiple, complementary methods for identifying and preventing fraud as individual tests were insufficient. 8 Fraudulent responses were found to originate from both bots and humans, indicating sophisticated, hybrid approaches for online survey-taking and underscoring the need for multi-method prevention and detection protocols.11,18–20
Researchers have found that domain knowledge tests are precise methods for excluding fraudulent responses,8,21 although efficacy may be diminishing as technology evolves. 17 For example, in a survey of U.S. beekeepers, asking participants about their beekeeping practices quickly eliminated responses unlikely to be true. 22 Other prevention and exclusion methods have been documented, evaluated, and prioritized.8–11,17,23,24 Feasibility and fit will depend on the study; we recommend reviewing the literature and testing multiple prevention and exclusion methods before launching data collection.
Cross-referencing enrollments across survey items and/or different platforms can serve as a convenient tool for verifying enrollments: for example, checking for consistent responses to demographic questions.21,27 Be aware of what information is collected in outreach platforms, the study instruments, and anywhere else that can be used to additionally verify participants. Other studies have implemented checks ranging from comparing self-reported personal information against national databases 28 to brief phone conversations to set up interviews 17 or verify suspected fraudulent responses as legitimate. 29 The correct fit will depend on your study protocol and resources available. Regardless of method, we recommend checking responses before payment to limit incentive distribution to fraudulent responses.
Limitations
Due to the emerging nature of fraudulent responses, the study did not include an adequate fraud detection, prevention, or exclusion protocol prior to starting data collection, which may affect internal validity. While an automated duplicate check was used from the beginning, preventing responses with duplicate email addresses, phone numbers, and/or Facebook IDs, this proved to be insufficient shortly after beginning data collection. This resulted in a process of trial and error to identify, prevent, and exclude fraudulent responses during data collection. Further, it is likely that a small number of valid respondents were excluded, an issue other teams have faced as well. 31 Future studies could consider piloting fraud prevention and exclusion protocols, 14 which may help ensure valid respondents are included.
Conclusion
Soon after beginning data collection, we suspected fraudulent responses and began a process to identify, prevent, and exclude fraud beyond the duplicate protection methods already in place. The first sign was a sudden appearance of multiple emails containing similar text and grammatical errors, all sent from addresses with the format FirstNameLastNameNumber. These indicators have also been observed by other teams.14,17,21 Notably, at this time all online recruitment occurred through research volunteer registries (ResearchMatch and an institution-specific registry).
While eligibility for our study was broad, participants were required to share Facebook post history, which provided an external indicator for legitimacy and uniqueness. We leveraged this to remove duplicates and fake accounts by requiring at least 500 words and a post history starting on or before January 1, 2020. Additionally, we implemented automatic prevention protocols: CAPTCHA, VPN and non-U.S. IP address block, and removal of duplicate email addresses, phone numbers, and/or Facebook IDs. Finally, we manually reviewed all responses to flag and exclude suspected fraudulent responses, checking payment information, demographics, and IP addresses.
Overall, other than the Facebook post history requirement, we found that the VPN check had the largest impact on preventing misrepresentative and/or fraudulent entries. This step blocked 340 total entries to the study, representing 9% of all survey entries since the VPN check was introduced. At the end of this six-month period, and two months after implementing the VPN check, misrepresentative and/or fraudulent enrollments completely stopped.
In conclusion, as reported above, we ultimately removed 9.6% of completed enrollments (eligible, consented, shared social media data, and completed the baseline survey). Planning for prevention and verification from the beginning will ensure consistency as well as efficiency in study activities that include verification or review. Unfortunately, given the observations of evolving evasion techniques throughout the lifetime of the study, adjustments during data collection may be necessary. Ultimately, we expect both online research and fraudulent enrollment rates to increase, especially as technology (e.g., large language models) improves. Future work should continue to report fraudulent response rates, evaluate methods to prevent and exclude suspected fraud, and explore structural dynamics and motivations driving fraud.
Footnotes
Acknowledgements
We acknowledge contributions and support from Haitao Huang.
Ethical considerations
This study received ethical approval from the University of Pennsylvania IRB (approval #852013) on October 25, 2022.
Consent to participate
The IRB determined that this research involved minimal risk and approved a waiver of written documentation of informed consent. All participants electronically indicated informed consent prior to enrollment in the study.
Author contributions
All authors wrote this article and approved the final manuscript.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by NIMHD [R01MD018340] and Penn Global Engagement and Research Fund. The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Data sharing is not applicable to this article as the manuscript focuses on challenges encountered during recruitment and data collection processes for a study that contains PHI. Details on anonymized features of the data can be obtained from the corresponding author.