Using Reddit as a Source for Recruiting Participants for In-Depth and Phenomenological Research

Abstract

The website Reddit has been noted as a fertile space for researchers due to its large user base and collection of distinctive communities. Reddit communities may be encouraging areas for specialized participant recruitment, though there is currently a lack of literature integrating Reddit into qualitative research participant identification and recruitment. This paper examines unique challenges and opportunities associated with using Reddit for sampling in-depth, qualitative research designs. Using a case example of a qualitative study which recruited from Reddit, the paper offers insight into what social scientists may expect to experience if attempting this form of recruitment. Reddit can potentially offer contact with hard-to-reach populations, and using it for recruitment may broaden the quality of qualitative data if sourced in ways which are ethical and conform to the tolerated standards of the specific communities being sampled.

Introduction

Adequate participant recruitment is a perpetual challenge within qualitative research methods such as focus groups or in-depth interviews. Eligible participants may not wish to travel to the research site, may not see the value of the research, or may feel that their participation requires too much time and effort (Patel et al., 2003). University-led studies tend to recruit from the sample most available to them—i.e., university students—though these participants represent a small minority of the global population (Henrich et al., 2010b). As a result, a large proportion of research knowledge generated comes from populations that are Western, educated, wealthy, and industrialized (Arnett, 2008). Despite the fact that such characteristics are not present in most people across the world, authors persist in their claims of the generalizability of results using these samples (Henrich et al., 2010a).

In recent years, online recruitment has become an increasingly common alternative to traditional participant recruitment. Online recruitment and data collection have been empirically supported to provide samples and results that are comparably valid to research conducted offline (Birnbaum, 2004; Evans & Mathur, 2005; Wilson & Dewaele, 2010; Woo et al., 2015). Online recruitment allows researchers to easily reach participants (Buhrmester et al., 2011), easily collect data (Shapiro et al., 2013), and gain access to varied population pools (Rouse, 2015). Global Internet usage has continued to grow in recent years (Silver, 2019), which allows for recruitment in larger areas of the world or targeted to geographic areas that are far away from the research teams. In response, scholars have begun investigating and evaluating various Internet platforms for their potential as sources of data-rich participants for research activities (Buhrmester et al., 2011; Lane et al., 2015).

Social networking platforms and sites have therefore become the focus of emerging literature in online recruitment. For example, Facebook has an incredibly large and diverse user base, with nearly 3 billion site visits per month (Dixon, 2022). When compared to traditional mass media recruitment, paid Facebook research advertisements generate more user impressions for less money (Whitaker et al., 2017). Facebook has also been praised for its ability to recruit from hard-to-reach populations (Sikkens et al., 2017; Whitaker et al., 2017). Social networking sites such as Facebook also allow researchers to seek out their participant pool directly, build trust and rapport necessary for engaging in research activities, and to gate-keep their own personal information for both safety and rigor purposes (Sikkens et al., 2017).

Online recruitment and data collection are not without their challenges. For example, studies using one of the most popular online recruitment sources, Amazon Mechanical Turk (MTurk), have repeatedly needed to discard data for being unsuitable or unusable (Chandler et al., 2014). Further, since MTurk users represent a very specific sector of Internet users, it may not be a viable option for recruiting members of specialized populations (Samuels & Zucco, 2013). More popular, broader-reaching platforms such as Facebook or Twitter present their own obstacles. In order to reach large numbers of users, research teams must know how to make connections through potential participants’ social networks or otherwise pay money to engage in targeted advertising (Johnson et al., 2014; Kosinski et al., 2015; Whitaker et al., 2017). Like offline research, online research recruitment generally benefits from promising some sort of compensation for participations; however, such compensation can lead to poor data quality or attempts to please the researchers (Gadiraju et al., 2015; Lane et al., 2015), and efforts to mitigate these circumstances have been criticized for their relative ineffectiveness (Chandler et al., 2014). Depending on the participant populations being sought and the study goals, recruiting from social media sites such as Facebook may not generate representative samples (Whitaker et al., 2017). Researchers are therefore in continual need of inexpensive, reliable, honest sources for online recruitment which offer the greatest eligible participant pool and highest-quality data.

One platform with potential to meet these criteria is Reddit. Reddit is a message board website where users can post text, media, or external links, as well as interact with one another through comments and direct messages. A hallmark feature of Reddit is its voting system, whereby users can give posts (or comments) a positive or negative vote, with the resulting scores determining the level of visibility the content has for other users (Massanari, 2017). The site is divided into a multitude of communities separated by topic area, called “subreddits”, and an individual user’s main landing page is a collection of the highest-voted posts from the various subreddits they are subscribed to. Reddit is an extremely popular website, currently ranked 19^th globally in popularity (2022) with tens of millions of unique users logging in and engaging with content daily (Reddit, 2020). Due to its large user base and influence over broader online culture, Reddit calls itself “the front page of the Internet”. Individual Reddit communities are also highly specialized, with tens of thousands discrete subreddits (Shatz, 2017).

Empirical research has therefore begun to use Reddit as a source for participant recruitment (Jamnik & Lane, 2017; Richard et al., 2021; Shatz, 2017). Reddit-based data are high-quality and have demonstrated equivalent psychometric validity and when compared to prior in-person research (Jamnik & Lane, 2017). As Reddit recruitment or data analysis has become more common, multiple studies have described the benefits and challenges associated with recruitment from Reddit, noting that Reddit’s broad demographics and highly specific subreddits offer the ability to communicate with members of distinct communities or affinities quickly, easily, and in sufficient numbers (Amaya et al., 2021; Proferes et al., 2021; Shatz, 2017). However, these studies primarily describe or employ quantitative, survey-based methods, or content analyses on previously-existing user content generated and collected with minimal participant interactions on the part of the research teams. One recent study described the viability of Reddit as a location to recruit participants for qualitative online focus groups, though these focus groups were created and supported asynchronously using Reddit’s message board features (Richard et al., 2021).

Given the growing popularity in using Reddit to access research participant populations, there is a need to systematize the various ways it can be used for research purposes (Amaya et al., 2021; Proferes et al., 2021). Currently, literature is only beginning to emerge detailing the use of Reddit as a primary source to recruit participants for synchronous, in-depth qualitative interviews conducted off-site. Through the example of a social work research project which drew participants from a specialized Reddit community, this paper describes the processes, challenges, and lessons learned from using Reddit for participant recruitment. In doing so, this discussion seeks to guide and empower future researchers to pursue Reddit as an opportunity to access high-quality data in niche populations.

Methods

The research study was an exploratory phenomenological study into the experiences of current or former intimate partners of men who were involved with The Red Pill (TRP). TRP is a community largely organized on Reddit and which advocates the use of systematic manipulation as a seduction technique (Bratich & Banet-Weiser, 2019; pk_atheist, 2012). TRP is primarily intended for heterosexual men trying to seduce heterosexual women, and its systems advocate a type of hypermasculinity which prioritizes being an ‘alpha’ at all times (Ging, 2019). The name “The Red Pill” comes from a sequence in the film The Matrix where swallowing a red pill allows the protagonist to become aware of the unpleasant truths of the world (The Wachowskis, 1999); TRP philosophy maintains that the ugly truth men need to realize in this world is that feminism gives women a monopoly of power over who is able to engage in romantic and sexual relationships (Ging, 2019). TRP claims that its tactics will specifically empower adherents to circumvent the roadblocks to sexual and romantic success presented by feminism, which it refers to as women’s “sexual strategy” (pk_atheist, 2012).

The two subreddits chosen for recruitment were The Blue Pill and Ex-Red Pill. The Blue Pill is a community where users post topics and discourse from TRP, discussing and satirizing the elements deemed most ridiculous (Reddit, 2013). Ex-Red Pill is a subreddit where users who had been part of, or adjacent to, TRP process their exit from that community and its worldview (Reddit, 2014). These subreddits were selected in accordance with the steps outlined in Shatz (2017), which state that a chosen subreddit must have adequate fit and reach. Both The Blue Pill and Ex-Red Pill were determined to have adequate fit because they require community members to have familiarity with TRP and potentially even have been (or known) participants. Both subreddits were of a size believed to be large enough to find enough participants for a small qualitative study without being so large that the research team would be overwhelmed by responses. Recruitment occurred in two periods, between March-April 2019 and November-December 2020. The initial period recruited solely from The Blue Pill, while the second period added Ex-Red Pill in order to increase user reach and engagement.

The research team began the recruitment process by creating a Reddit username separate from any team members’ personal accounts and only for use during research activities. Using this account, the team posted a thread on The Blue Pill to determine if there was sufficient community interest and eligibility to support recruitment. This was viewed by the research team as a digital form of outreach, similar to methods employed for assessing feasibility of recruiting participants from hard-to-reach populations in-person (Alvarez et al., 2006; Stewart et al., 2020). Enough positive responses were observed in the interest thread to suggest that recruitment could, in fact, be sustained via The Blue Pill. The outreach thread was subsequently deleted and the research team directly messaged the community moderators, explaining the project’s aims and procedures while requesting permission to use their subreddit for recruitment. Once full moderator permission was obtained, the research team again published a thread containing the official recruitment information and protocols.

Users were instructed to directly message the research team’s user account if they believed themselves to be eligible and were interested in participating. Direct message conversations were used to answer questions, assuage users’ concerns around safety or anonymity, arrange schedules for the in-depth interviews, and exchange non-Reddit contact information to be used when conducting the interviews. The research team established a Skype account specific to research activities which allowed for participants’ choice of traditional phone or internet-enabled voice calls. Using this Skype information, team members conducted one-time, in-depth, semi-structured interviews with participants.

Findings

A total of 6 interviews were conducted with eligible participants recruited from The Blue Pill and Ex-Red Pill. Participants all identified themselves as heterosexual women who were ex-partners of Red Pill men and who had become users of the included subreddits as a result of their experiences with their former partners. Four participants were located within the United States, while two were located in Europe. In addition, one interview was scheduled and had begun when the participant identified himself as a heterosexual man from the Dominican Republic who had been friends with, but never in a romantic relationship with, a Red Pill man. The interview was completed but his data were excluded from the final analysis, as they did not meet research goals. However, this experience is elaborated in greater detail below as part of the discussion regarding navigation of website features.

Although an adequate number of participants were recruited for the study, the recruitment experience also yielded a range of opportunities and challenges. This information can be beneficial to future studies attempting to recruit from Reddit, particularly as related to internal community response, the Institutional Review Board approval process, navigating site features, and external responses to recruitment.

Internal Community Response

The internal community response to the participant recruitment efforts was positive. In addition to allowing permission to post recruitment information, community moderators offered to “sticky” the post, or have it temporarily placed immovably at the top of the main page in order to maximize its visibility. Community members interacted with the post, offering comments and words of encouragement. Several users stated that they were not eligible for participation, but supported the research goals and were interested in reading any publications which resulted from the project. One user identified herself as an author on a published quantitative research article which had collected data from similar online men’s groups, and shared her article with the study team. Participants expressed considerable eagerness to share their stories while coordinating their interview schedules, and several even sent follow-up messages to the research team’s account in order to offer supplementary information. Users also demonstrated a protectiveness toward their community, requesting more specificity about research activities and team credentials, as well as offering advice to one another toward maintaining personal safety and anonymity during the data collection process.

There were additional users who contacted the research team inaccurately claiming their eligibility due to miscommunications or misunderstandings surrounding eligibility requirements. These users were generally men who had been Red Pillers themselves (though never romantically involved with other Red Pillers) and had either misread the recruitment information or otherwise had a desire to assist with the project anyway. The team communicated to these users that they did meet the conditions for participation, but that they were welcome to join any potential future studies which targeted current or formerly active members of TRP. All of the ineligible users responded positively toward this proposition.

It should also be noted that there is a subreddit adjacent to TRP called Red Pill Women that the team did not interact with. The research team considered recruiting from this subreddit to provide alternative perspectives, but ultimately chose against it for two primary reasons. First, Red Pill Women is not a community for women who date Red Pill men, but rather a community where women discuss and advocate adapting elements of TRP tactics for their own benefit (pearlsandstilettos, 2018). As such, it is possible that women in that community have dated or been intimately involved with Red Pill men, but it is not a foregone conclusion. Second, although interviews with women who were positive toward TRP tactics would have offered a broader range of data, there was caution from the team about internal community reactions within the Red Pill Women subreddit. As discussed further below, the team received slightly aggressive reactions from people who were supportive of TRP during the recruitment periods. Given that Red Pill Women is a pro-Red Pill community, the research team was concerned that it would invite too much antagonism from users if the study were to attempt to recruit from this space.

Despite these minor challenges, enough participants were successfully recruited across the two periods to satisfy the study’s aims. Overall, recruitment was feasible and well-tolerated by the chosen subreddits.

IRB Approval

In order to ensure the study was conducted professionally and within the bounds of all legal and university-led ethical standards, the research team obtained Institutional Review Board approval. However, several challenges were experienced which seemed unique to the Reddit-based application of in-depth participant recruitment standards. The first challenge was regarding clarity between the research team and the IRB regarding the difference between feasibility determination within the community and formal participant recruitment. Ultimately an agreement was reached whereby all formal recruitment was required to have its own unique and separate communications and language.

Another challenge occurred when several participants requested to perform their interview via Skype internet-enabled audio. By adding this provision to the ethics protocol, the research team was subsequently required to add language to the informed consent oral script which suggested the possibility that an external third party could intercept the Skype call and unmask the anonymity of the participant. Though this situation is hypothetically possible, it would be highly improbable due to the anonymity of the Reddit usernames and potential pseudonyms used by the participants, as well as the overwhelming unlikelihood that a third party would have the ability to capture and triangulate the communication by knowing the date, time, and associated Skype accounts for the interview. On the other hand, the participants in the project had been in relationships with men who were known to use coercion and manipulation, and hearing such language had the potential to trigger fears or residual traumas by insinuating that their ex-partners were attempting to intercede on the interviews or that the research institution could potentially mishandle data security. Fortunately, none of the participants rescinded their involvement or declined to consent to the interview.

Navigating Site Features

Team members were familiar with the primary features of Reddit prior to recruitment, but the research activities still demonstrated a need to pay close attention to site functionality. Specifically, between the first and second recruitment periods, Reddit added an “instant messaging” feature in addition to their existing direct messaging feature. The icons and notifications for the two messaging features are entirely different, and direct messages take the users to a separate landing page whereas the instant messaging opens a chat dialog box which appears on whichever screen the user currently occupies. As a result of being unfamiliar with this change, the research team nearly missed several participant communications which were sent via instant message rather than traditional direct message. In fact, several of the instant messages were weeks-old by the time the team saw them, and the team had to hope that their long response time did not deter those users from participating.

In addition, the instant messaging feature created some uncertainty regarding identity. Reddit’s logo is a cartoon alien without gendered characteristics (Reddit, 2021), but the instant messaging allows users to display a customized avatar of the logo with additional defining features. One user requested to participate over instant message, and their avatar featured the Reddit alien logo sporting long hair and a necklace. However, when the Skype Internet call began, the participant immediately identified himself as a man. Although men were not excluded from the study, the researcher conducting the interview became concerned that this was some type of hostile setup. Instead, this participant indicated that while he had never been in a romantic relationship with any Red Pillers, he wished to discuss a friend who was very involved with TRP. Although this did not entirely meet eligibility requirements, the interview was completed in full. Moving forward from this participant, the research team chose to reaffirm eligibility status prior to conducting interviews.

External Responses

While internal community response from The Blue Pill and Ex-Red Pill were positive, the research team did receive a number of messages displaying some form of hostility from individuals who identified themselves as personally involved with TRP. Several of these users questioned the value of the project and accused the research team of being “unscientific”. These users seemed unfamiliar with purposive sampling, as some also asserted that by sampling only current or former partners of Red Pill men that the study was biased. Another user claimed the study lacked rigor and validity because it did not include a control or comparison group comprised of women who had not been in relationships with Red Pillers. Multiple comments and messages stated that the study should directly interview Red Pillers instead of their romantic partners. Messages from TRP users often expressed concern that the research study would portray their ideology in a negative light by only highlighting participants who had negative experiences with Red Pill men. One such user messaged the research team claiming his girlfriend was highly supportive of being in a TRP relationship and would be happy to offer her perspective as a contradictory case analysis. When the research team responded that she would have to write them directly in order to participate, neither that user nor his girlfriend made any further contact.

One user in particular expressed his doubts about the nature of the study repeatedly and at great lengths. This user initially messaged the team stating that their study was redundant because academic literature already existed examining relationship experiences and outcomes with individuals who suffer from BPD, NPD, and other personality disorders. Though the user did not identify any particular studies or authors, he nonetheless alleged that findings related to TRP would be highly similar to those within personality disorder literature. The research team politely responded their disagreement, noting that no such study on TRP had been performed before, and questioned this user’s suggestion that TRP behaviors mimic symptoms of personality disorders. The user replied with lengthy messages expanding on his beliefs about the nature of the research study, masculinity, and TRP practices. The user positioned himself as a community member knowledgeable about TRP, offering claims about what women find attractive in men (such as their jawline), and stating that TRP exists because TRP members believe female arousal is inherently anti-male. This user also indicated his opinions on the research process, asserting that the team’s study would fail rigor due to self-selection bias by the participants while reiterating that the project would not produce worthwhile knowledge. The user seemed to believe he was an authority on these topics, and he offered to answer any additional questions the research team might have. Members of the research team felt uncomfortable with this user’s attempts to insert himself into the research process, and disengaged from corresponding with him.

Discussion

Previous research has indicated that recruitment from Reddit for academic research is feasible and offers high-quality data so long as certain steps and guidelines are followed (Amaya et al., 2021; Shatz, 2017). However, there have been few studies describing the application of these steps for the recruitment of participants in synchronous, in-depth qualitative research. The case example presented here demonstrates that recruiting individual qualitative participants from specialized Reddit communities is feasible and acceptable. The lessons observed in this process can be applied to making future research recruitment operate smoothly and accessibly.

The experiences of recruiting from Reddit also broadly echo prior research into recruitment from social media and networking sites such as Facebook, with certain key differences. The message-board communication and specialized community nature of Reddit means that researchers do not need to seek out participants or directly initiate communication. Once contact is made, however, the process of using direct messages to build rapport and coordinate research activities is quite similar to recruiting specialized populations from Facebook (Sikkens et al., 2017). Reddit recruitment is a free alternative to recruiting through targeted advertising, although without the outcome metrics available through advertising on sites like Facebook it can be difficult to know if recruitment is having the necessary level of reach or interaction (Shatz, 2017; Whitaker et al., 2017). Much like recruitment from Facebook, it is also important that researchers using Reddit create user accounts which are neutral to potential participants and/or are otherwise unconnected to any personal accounts they may have (Sikkens et al., 2017).

Overall community response in this study indicates the potential for using Reddit for recruiting active qualitative participants in future studies. However, not all subreddit communities can sustain participant recruitment. In his guidelines for Reddit recruitment, Shatz (2017) states that “the more subscribers a subreddit has, the better” (p. 544). For qualitative recruitment, this may not be as true. While a chosen subreddit must have adequate fit and reach, a larger user base may lead to a larger amount of necessary work for the study team to parse replies, screen potential participants, and navigate encounters with hostile outsiders such as experienced during this project. Targeting a more popular or larger subreddit could encourage these events to occur more frequently, as there is a higher likelihood of posts being viewed by outside users who are not subscribed to those subreddits. Reddit’s voting features may expose non-subscribed users to this content on their feeds if it gains popularity, and users also have the ability to share or “cross-post” the content in other communities. In addition, subreddits may not wish to participate in recruitment due to fear of repercussions, desire for anonymity, antagonism toward the idea of being studied, or any other reason. As such, we highly recommend for future studies to familiarize themselves with any existing guidelines a subreddit has regarding research, and to always request permission from moderators/administrators of a chosen subreddit prior to posting any calls for participation.

The Blue Pill and Ex Red Pill communities tolerated recruitment in their space very well. Users did exhibit a natural protectiveness, which was to be expected given the subject area. Men associated with communities related to TRP have been observed to harass, stalk, or even commit violence against women they believe to have wronged them (Bratich & Banet-Weiser, 2019; Ging, 2019). Despite this, the community seemed encouraged by the notion that the academic community would pay attention to their experiences as valuable knowledge, and users appeared to regard this as outweighing any inherent risks associated with participations. Community moderators and administrators were efficient, polite, and supportive of the process, which facilitated the research team’s ability to reach this unique participant pool. Nevertheless, researchers should maintain readiness and willingness to share information such as university credentials with their potential participants, but should do so in ways that protect their own safety and privacy. For example, the researcher may consider offering their IRB reference ID or a general institutional contact point (such as a study-specific email) if they do not feel comfortable offering their own personally identifiable information.

Researchers should practice flexibility within Reddit communities chosen for recruitment, understanding that the way that information (such as eligibility) has been conveyed may be misunderstood, glossed over, or may necessitate additional effort to connect with interested site users. It cannot be presumed that all users will be inherently familiar with the nuances of research procedures or will intuit what a research team is intending to accomplish. This project demonstrates the benefits of continuous dialogue with community participants, as well as clarifications and modifications when necessary. In the case of the male participant whose friend was a Red Piller, the research team had never considered verifying gender or romantic relationship status after users had self-identified their eligibility, and the team had incorrectly presumed that this user had fully understood eligibility requirements before making initial contact. Likewise, the team nearly missed connecting with users expressing interest due to site-wide messaging feature changes. Future research projects should therefore consider collecting baseline information before moving into formal data collection, as well as confirming that they have explored all possible avenues for participants to make contact before moving on from that community.

This research also demonstrates the necessity of ongoing collaborations with university IRBs in order to better navigate participant recruitment from Reddit and similar websites. Online participant recruitment, particularly via social media or platforms such as Reddit, is still a new phenomenon. It is clear that IRBs have been working tirelessly to ensure ethical research practices and the safety of participants while supporting the value of using the Internet to access small or difficult-to-reach populations. However, informed consent language suggesting the threat of external hacking or data theft may undermine participants’ trust in the efforts made by research teams to preserve their confidentiality, thereby damaging rapport. As demonstrated by this research, these types of consent scripts do not necessarily account for the personal histories of study participants nor the trauma which could be triggered by hearing these consequences. Consent language such as required in this study may also reflect poorly on research teams, as neglecting to acknowledge the high improbability of having data stolen and personally identified could lead participants to believe that the research team is not technologically literate or is potentially mishandling data safety protocols. We therefore propose that researchers and IRBs engage in ongoing conversations and partnerships which can satisfy the safety and ethics of human participant research while also preserving the integrity and trustworthiness of the specific research activities.

Despite the clear benefits to using Reddit for recruiting qualitative or in-depth research participants, it is also possible that bad-faith actors from outside of the groups being studied may try to intercede and disrupt the research process. Though this can often occur with in-person participant recruitment, affordances of technology such as anonymity and ease of contact may allow users to feel more empowered to be antagonistic than they would offline (Suler, 2004). During the recruitment process for this project, self-identified Red Pill users repeatedly sent messages to the research team’s Reddit account rebuffing the rigor and validity of the research being carried out. This research team considered these types of messages to be a collateral nuisance and varied in response based on how threatening they perceived a given message to be. Researchers recruiting from Reddit should recognize the likelihood of these types of messages and likewise act accordingly. However, it must be reiterated that members of Reddit and other similar online message board communities have been observed to mobilize harassment and threat campaigns, as well as in rare cases commit deadly acts of violence, against people they perceived as enemies (Bratich & Banet-Weiser, 2019). It is therefore of the utmost importance that future researchers recognize this risk and take appropriate steps to protect themselves and their participants’ personally identifiable data, as well as having communication and safety plans in place with administration should any types of these consequences occur.

Conclusions

Beyond recruiting participants for survey-driven or content analysis research, Reddit offers fertile ground for recruiting niche or specialized populations for in-depth qualitative data collection. Though it can offer researchers free and rapid access to large amounts of unique data, researchers should maintain respect and deference to community organization and etiquette when entering their space. The barriers and facilitators described as part of this research may help guide future researchers to recruit Reddit-based participants safely and ethically while yielding high-quality, trustworthy results.