Sage Journals: Discover world-class research

Abstract

Although psychological research often relies on convenience samples, the most informative participants may be individuals who are reluctant to engage because of vulnerability, mistrust of the research process, and/or disagreement with a study’s goals. Although this concern is particularly urgent for certain research questions, it is relevant for all researchers because relying solely on easy-to-reach participants limits a study’s validity and generalizability and may substantially hollow out or even render unanswerable some research questions. We review a number of challenges for conducting research with “reluctant but informative” participants and strategies to meet and overcome these challenges. We argue that engaging reluctant participants requires attention at every phase of the research process, including study design and planning, participant recruitment and testing, data analysis and interpretation, and reporting and broader impacts. We also illustrate with our own recent experiences conducting research on children’s gender concepts in rural, conservative U.S. communities that expressed skepticism about the value of this research. Although every study is different, notable across all projects are the need to respect both individual participants and their communities and to balance competing desiderata. Finally, we discuss the importance of transparency around sample composition and constraints to generalizability and the potential utility of collaborative research both across research institutions and between researchers and participant communities.

Keywords

reluctant participants hard-to-reach populations hidden populations vulnerable populations sampling

In psychology, as in all science, some questions offer easier answers than others. There are questions for which accessing the most informative participants may present logistical difficulties, from geographic distance to language barriers to sensory, cognitive, or physical differences that present unique challenges to participation. There are also important scientific questions for which the most informative participants may be the least interested in participating.

In this article, we are concerned with the latter situation: conducting research for which recruitment of appropriate participants presents a challenge because of their reluctance to take part. Studies may seek to recruit participants who are disinclined to engage in scientific research in general or in research focused on a particular topic. Potential participants may have a historical basis for discomfort with being subjects of research because of prior severe and troubling examples of exploitation or pathologization (e.g., the Tuskegee Syphilis Study), as is the case for African Americans (Robinson & Trochim, 2007; Shavers-Hornaday et al., 1997) and other racial or ethnic minority groups (Rowley & Camacho, 2015) and transgender populations (Tebbe & Budge, 2016; Vincent, 2018). They may have concerns about stigma, vulnerability, or negative consequences of participation because of issues such as mental health, substance use, or criminal liability (Mirick, 2016; Oruche et al., 2012; Young et al., 2020). Participant groups of interest may also mistrust researcher goals, intentions, or perspectives related to the research question itself (e.g., in investigations of extreme political views and other radicalized beliefs, Sazak, 2019; Sikkens et al., 2017; vaccine hesitancy, Hilário et al., 2023; Marshall et al., 2002; Wiley et al., 2021; gender diversity, Fine et al., 2024; Gross et al., 2024). These issues are not mutually exclusive; potential participants may experience reluctance from several sources at once.

Although engaging reluctant participants is a more urgent concern for some research questions than others, we maintain that it is an issue for all researchers to carefully consider. Excluding people from participation because engaging them poses difficulties is problematic in principle because doing so limits the validity and generalizability of psychological research, likely in systematic ways (i.e., undersampling on the basis of race, ethnicity, education, or political beliefs). This issue is a broad problem that has received substantial attention in recent years (Henrich et al., 2010; Kidd & Garcia, 2022; Nielsen et al., 2017; Roberts et al., 2020; Roberts & Mortenson, 2023). At times, excluding hard-to-reach populations may substantially hollow out the value of a research question (e.g., studying responses to messaging about climate change in individuals who are already receptive to scientific expertise) or even render it unanswerable (e.g., conducting an educational program designed for children in poverty by studying a sample of middle-class children). Furthermore, the issues for which researchers seek reluctant participants are not niche questions given that they include some of the most consequential and timely issues of the day. More positively, taking care to engage reluctant participants, although challenging, permits a more robust, informative, and socially accountable science.

Framing the Issue

There are two points regarding the scope of the issue that provide a framework for understanding the challenges, strategies, and lessons learned.

First, many researchers have grappled with recruitment challenges as a methodological issue that can be addressed with attention to sampling strategies or techniques (e.g., Hilário et al., 2023; Patel et al., 2003; Shaghaghi et al., 2011). Although outreach is certainly important, we view it as just one component of a broader set of considerations that involve every phase of the research process. Accordingly, we address how engaging reluctant participants raises challenges regarding study design, data analysis and interpretation, and postproject impacts—in addition to recruitment per se (see Table 1). This broader framing of the problem raises practical, scientific, and ethical considerations.

Table 1.

Challenges and Strategies in Research With Reluctant Participants as a Function of Study Phase

Study phase	Tasks and considerations	Challenges	Strategies
Study design and planning	Identifying population of interest Understanding population of interest Defining the sample Ethical considerations and constraints	Unexamined researcher assumptions Unintended harm to participants Resource limitations Potential for fraud/scams	Participatory research approach Integrating community priorities Balancing eligibility criteria with feasibility; flexibly revisiting criteria Prescreeners and individualized access to study materials
Participant recruitment and testing	Reaching participants Engaging participants Research methods	Identifying potential participants Outreach to participants Participant mistrust Sampling bias Resource limitations	Learning about participant community Careful attention to participant-facing language Communicating respect for participants and their contributions Targeted recruitment approaches Snowball sampling Respondent-driven sampling Web-based respondent-driven sampling
Data analysis and interpretation	Data analysis plan (and preregistration, if included) Considering generalizability Revisiting researcher assumptions	Preregistration vs. flexibility Unrepresentative samples Researcher interpretation vs. community experience	Preregistration with potential for change Compare target population with final sample; discuss limitations to generalizability Member checking or participant consultation
Reporting and broader impacts	Follow-up with community Downstream consequences for trust in researchers/research process, future participation	Balancing community priorities with full/accurate reporting Unintentional harm to community	Share results with community, open discussion Acknowledge community input clearly and openly Address community priorities

A second overarching issue is who is a reluctant participant in research and why. The primary message here is that the reasons (and potential consequences) are highly varied and need to be understood from the participant’s point of view. Much of the literature focuses on working with populations that are characterized as “hard to reach,” “hidden,” or “vulnerable” (e.g., Ellard-Gray et al., 2015; Hilário et al., 2023; Rockliffe et al., 2018; Shaghaghi et al., 2011)—classifications that focus on the perspective of the researcher more than the participant. To take the participant’s perspective, one must consider the reasons behind these classifications. For example, populations may be “hard to reach” not only because they are physically remote but also because of their social status (e.g., individuals who are unhoused or social elites; Ellard-Gray et al., 2015; Shaghaghi et al., 2011). “Hidden” participants are individuals who cannot easily be tracked because no clear record exists of the characteristics that define them as the population of interest—but for variable reasons (e.g., LGBTQ+ people because insufficient resources are devoted to identifying their needs, women who experience abuse but may choose not to report it out of concern for the consequences, radicalized groups who mistrust anyone deemed an outsider; Ellard-Gray et al., 2015; Sazak, 2019). “Vulnerable” participants are individuals who may perceive (and experience) heightened risk from research participation—but again, the reasons can vary, including potential discrimination, stigma, or legal consequences (e.g., undocumented immigrants, intravenous drug users; Ellard-Gray et al., 2015; Young et al., 2020). Moreover, communities may resent “helicopter” research, in which researchers land in their community only long enough to gather data, offering nothing in return (Rowley & Camacho, 2015).

A further concern is that these classifications (e.g., hard to reach, hidden) may be characterized as features of the individual (Patel et al., 2003; Shaghaghi et al., 2011) rather than acknowledging the complex interaction of structural forces at play (e.g., social positions, life experiences, and systems of power). When framed from participants’ perspectives, some documented reasons for reluctance to engage with research include experience of stigma, suspicion of research, experiences of vulnerability or disenfranchisement, and history of research abuses (see Table 2). Although these sources of reluctance may be associated with the listed groups, they do not necessarily apply to all group members (e.g., not every member of a racial-minority group necessarily experiences themselves as vulnerable or disenfranchised).

Table 2.

Some Sources of Participant Reluctance

Source of reluctance	Examples of participant groups
Experience of stigma	Survivors of domestic abuse (Ellard-Gray et al., 2015) Parents of children with autism or mental-health problems (Oruche et al., 2012; Shaia et al., 2020) Parents who choose not to vaccinate their children (Wiley et al., 2021) Families involved in child-protective services (Mirick, 2016) LGBTQ individuals (Ellard-Gray et al., 2015) People who use drugs (Young et al., 2020)
Suspicion of research	Individuals with politically extreme views (Sazak, 2019; Sikkens et al., 2017) Anti-vaccination individuals (Hilário et al., 2023; Wiley et al., 2021)
Vulnerable or disenfranchised	Racial minority and Indigenous populations (García et al., 2017; Rowley & Camacho, 2015; Shavers-Hornaday et al., 1997) Populations with low socioeconomic status (Rowley & Camacho, 2015) Undocumented immigrants (Ellard-Gray et al., 2015)
History of research abuses	African American populations (Rowley & Camacho, 2015; Shavers-Hornaday et al., 1997) Indigenous populations (Rowley & Camacho, 2015) Immigrant communities (Ellard-Gray et al., 2015) Transgender and gender-diverse populations (Tebbe & Budge, 2016; Vincent, 2018)

In the present article, we review the literature on challenges and strategies for research with “reluctant but informative” participants, informed in part by our own recent experiences. From 2022 to 2024, we conducted a study aimed at reducing gender essentialism and prejudice against gender nonconformity with children ages 6 to 10 living in rural, conservative communities in the United States (Gross et al., 2024). We were especially interested in these children given prior research indicating higher levels of gender essentialism and prejudice against gender nonconformity in these communities (Fine et al., 2024). But the same reasons these communities were of interest also meant that many families were reluctant to participate and even suspicious of or hostile toward the project. Consequently, it took us 2 years to recruit our sample (compared with fewer than 2 months to recruit the same number of children in this age range for other research conducted in our lab), and throughout the process, we were required to step back, reconsider, and pivot numerous times. We are immensely grateful to all families who expressed interest in the study, especially families who ultimately chose to participate. At the same time, we also have learned important lessons—including from individuals who chose not to participate.

The rest of the article has four main sections, each focused on one of the four phases of the research process listed in Table 1: (a) study design and planning, (b) recruitment and data collection, (c) data analysis and interpretation, and (d) reporting and broader impacts. For each, we discuss challenges and strategies for meeting these challenges, illustrating with our own case study. We bring together a range of thoughtful considerations and suggestions provided in the literature within and beyond psychology to discuss common themes in participant reluctance, recruitment challenges and strategies, and our own missed opportunities and lessons learned. Finally, we conclude with some general lessons and directions for the future.

Study Design and Planning

In the study-design phase, important tasks and considerations include identifying and gaining familiarity with potentially informative participant populations, which will aid in defining sampling parameters. There are also important ethical considerations at this stage, including awareness of community interests and concerns.

Identifying the population of interest

In the early phase of a research project, researchers need to consider how to balance two interests that may at times conflict: identifying participants who would be most informative in answering the research question and what is feasible in terms of recruitment and outreach. For example, if one is interested in a rural sample, theoretically, the ideal participant would be one living in a county with a Rural-Urban Continuum Code score of 9, but as of 2022, these accounted for only 1.4% of the population, by definition (Purdue Center for Regional Development, 2024). There are many different criteria a team could set to identify any population of interest, and small differences may have a big impact on recruitment. To the extent possible, researchers should maintain flexibility and be willing to revisit specific criteria as they reevaluate the balance between accessing an informative sample and one that is large enough to allow for meaningful analysis.

Understanding the population of interest

Understanding the population of interest includes consideration of a participant’s context, values, and beliefs and is of central importance for two key reasons. First, from the standpoint of a researcher’s ethical responsibilities, understanding the population of interest is necessary to achieve the ethical goals of treating participants with respect, enhancing the benefits that participants can receive from participating in the research, and minimizing potential risks or harms. But second, from the standpoint of achieving one’s research goals, this is the most effective means of identifying why a participant may be reluctant and thereby beginning to work to address their concerns. It also is the basis for a more valid and meaningful program of research (Rowley & Camacho, 2015).

Strategies for understanding the population of interest include conducting focus groups or pilot studies with members of the participant community (Lewis et al., 2020), working with community partners or advisory boards to help build rapport and gain initial insights in participants’ experiences, or including community members on the research team (Ellard-Gray et al., 2015; Rowley & Camacho, 2015). For some projects, it may be feasible to adopt a participatory or community-based research design, involving collaborative discussion of the project with target communities from its inception (Levac et al., 2019; Tebbe & Budge, 2016). Participatory approaches may be particularly helpful in situations in which potential participants are vulnerable or have a historical basis for mistrust of research; these approaches include meeting with community members to listen to their concerns and priorities, involving some members on research teams, and ensuring that the research in some way “gives back” to the community of interest (Levac et al., 2019; Rowley & Camacho, 2015; Tebbe & Budge, 2016; Vincent, 2018). Participatory research requires some degree of access to the community; researchers may begin by identifying and contacting key stakeholders and introducing the research project and the team involved, including at least some information about researcher backgrounds and identities (Tebbe & Budge, 2016; Vincent, 2018). Resource intensiveness is a major limitation to the participatory approaches discussed above, and thus, they do not lend themselves to every project and are more commonly adopted for qualitative investigations than quantitative investigations (Levac et al., 2019). In the absence of a full participatory design, holding meetings with members of participant populations (or attending public town halls, school board meetings, or similar events) can allow researchers to gain insight into community priorities and potentially integrate these into the project, adding value for participants and strengthening the scientific work.

Defining the sample

The design phase may also include significant challenges related to defining the sample population, particularly for hidden participants (Ellard-Gray et al., 2015). In addition to considering how eligibility criteria will be set, in studies that allow for participant self-identification, researchers will likely need to incorporate strategies for identifying and excluding fraudulent participants (a problem that is particularly common in research conducted online; Lawlor et al., 2021).

Familiarity with the population and carefully considered sampling parameters can aid in selecting respectful and appropriate language for recruitment and other study materials (and avoiding serious pitfalls). It is important to avoid stigmatizing language in recruitment materials; doing so requires awareness of how stigma is experienced in a given community (Ellard-Gray et al., 2015). As one example, for work in LGBTQ+ populations, familiarity with differing terminology can help in recruiting the desired sample (Ellard-Gray et al., 2015). Whereas some older transgender adults prefer the term “transexual,” younger trans people may find this outdated or even offensive (Tebbe & Budge, 2016). Likewise, recruitment targeting “homosexuals” may sound overly clinical or even offensive, and advertising to “gay or lesbian men and women” is likely to draw a different sample than a more gender-neutral and less academic-sounding advertisement to “queer folx.” Likewise, Sikkens et al. (2017) found that participants were more open to participation when asked to discuss their “strong ideals,” whereas the word “radicalization” could be a barrier. Even when the terminology used is not overtly stigmatizing or offensive, using wording that is overly clinical, dated, or otherwise “out of step” with the community can indicate that a research team is not sufficiently well informed, heightening issues of mistrust.

Ethical considerations and constraints

A major ethical consideration in the design phase of a project is accounting for participant needs in a responsible manner that minimizes harm to both the individual participants and the community or communities to which they belong (e.g., exploitation or contributing to deficit approaches or disparaging stereotypes; Cheng et al., 2021; Rowley & Camacho, 2015). Researchers’ scientific responsibilities are also important (e.g., to minimize confounding factors that may undermine the validity of the findings and to report all results free from bias even if doing so may be uncomfortable for participants).

An example in which these ethical goals may come into conflict is projects involving either outright deception (see Nicks et al., 1997) or failure to disclose the true aim of a study. These practices may be especially common in the context of reluctant participants when the topics under study are controversial. For example, researchers who are interested in participants’ views on one topic (e.g., vaccination, Horne et al., 2015) may devise a survey including a host of other topics (e.g., abortion, euthanasia) to mask their primary focus. Debriefing participants following deception or incomplete information is standard practice in such cases (Greene & Murphy, 2023; McShane et al., 2015; Nicks et al., 1997), although care needs to be taken that the information is clear, memorable, and actually read by participants.

Our case study

In recruiting for participants in our own project, we had to iterate over time to satisfy two criteria that were somewhat opposing: on the one hand, restricting our sample to our theoretically based “ideal” (children living in rural, politically conservative, predominantly White communities) and on the other hand, broadening our sample to feasibly and realistically permit data collection within a promised time frame. On multiple occasions, when data collection stalled, we successively broadened the range of communities that we included, primarily by loosening the inclusion criteria, from definitively “rural” communities to those that are “nonmetropolitan.” The slow pace of data collection meant that we had to ask for a year-long extension from the journal at which we had an approved Phase 1 Registered Report (and they were graciously accommodating). We recruited participants based on characteristics of the communities in which they lived rather than characteristics of the individuals themselves. As a result, the families in our final sample were on the whole less conservative, more educated, and higher income than the communities in which they lived, likely because they were more comfortable with the focus of our study or with participating in psychological research more generally.

We grappled with the ethical issue of how much information to provide to potential participants regarding the nature of our study. On the one hand, we had an ethical obligation to let parents know that the study was focused on how children think about and evaluate boys and girls. We told prospective families that we were trying to learn about how children think about individuals who do not conform to gender stereotypes and how they use these beliefs. We also shared details of the study procedure with those parents who requested more information about the study itself (admittedly a small number). On the other hand, our recruitment materials and consent form deliberately did not include the word “gender” because we wished to avoid triggering a negative response based on terminology. Nor did we discuss the concept of “essentialism” or our goal of increasing children’s tolerance for gender-diverse individuals.

Participant Recruitment

Recruitment challenges have been a primary focus of research on reluctant participants (e.g., Hilário et al., 2023; Patel et al., 2003; Shaghaghi et al., 2011) and include both engaging participants (helping them overcome sources of reluctance, suspicion, or distrust) and reaching them (identifying individuals who match the desired characteristics). A variety of solutions have been developed and discussed, each with a different set of benefits and issues to consider.

As a basic first step, providing clear and complete information about the study aims and value of participation in language that is approachable to participant communities and being available to respond to questions from potential participants can help increase interest and reduce reluctance because of both mistrust and disinterest. For example, in research with immigrant groups, ensuring that study information is available in the language(s) spoken by community members, including study-team members who are fluent in such languages and can answer questions, and checking that the reading level of materials will be available to participants with less formal education may help reduce reluctance and uncertainty (Ellard-Gray et al., 2015; Rowley & Camacho, 2015). In situations in which participants may be particularly vulnerable (e.g., victims of sexual assault), keeping language in recruitment materials less specific—for instance, discussing unwanted or distressing dating experiences or focusing on resilience—can attract desired participants who can then choose to self-identify later in the research process (Ellard-Gray et al., 2015). In all situations and especially in face-to-face interactions, maintaining respect for participants, including their unique experiences and the time and effort involved in contributing to the research process, is of particular importance.

Snowball sampling (in which initial participants identify others, such as friends or family members) can generate large samples in potentially hard-to-reach populations but may produce bias because of differences in individuals with the largest social networks (Johnston & Sabin, 2010; Shaghaghi et al., 2011). Respondent-driven sampling (RDS), in which researchers purposively identify “seed” participants who are each provided with a limited number of recruitment “coupons” to expand recruitment, has been suggested as a less biased option but is resource-intensive, requiring printing and tracking of individual coupons and recruitment (Johnston & Sabin, 2010). A related method, web-based RDS (webRDS), is identical to traditional RDS except for the provision of free and more easily tracked e-coupons instead of physical printouts (Young et al., 2020).

Young et al. (2020) reported on the efficacy of several targeted and in-person approaches for recruiting in a study of people who use drugs. These approaches included hosting cookouts and walking through neighborhoods with high rates of reported drug overdoses and both traditional RDS and webRDS. They found that simply walking through neighborhoods and posting flyers was the most time-intensive and the least effective strategy and that although the cookouts got the most participants “in the door,” relatively few of these actually led to valid data. By contrast, referrals through webRDS had a high rate of initial response and a much higher rate of conversion to usable data, leading to a higher percentage of includable participants, and was more effective than traditional RDS (Young et al., 2020). However, it is unknown how broadly these relative costs and benefits would extend to other research projects. The strategy of tracking the efficacy of different methods is an excellent way to gauge relative success, allowing research to iterate as needed over the course of data collection.

Online surveys offer a different avenue to reach a wide variety of potential participants, and anonymized responses can allow participation with less risk for vulnerable populations. They also come with a heightened risk of fraud, from both bots and human respondents who are not actually eligible for the study (Chandler & Paolacci, 2017; Lawlor et al., 2021). Prescreening surveys, single-use links, “check” questions, and use of CAPTCHA technology can help in reducing these issues, but particularly when financial incentives are involved, scam participants can be persistent. Researchers should consider vulnerabilities in their survey and how final data reflect expectations (e.g., if more than 90% of respondents to a survey exploring the lives of people with disabilities report working 40 or more hours a week, there is a mismatch between the expected and obtained data; Lawlor et al., 2021). Each additional effort to protect a survey against fraud has some trade-offs for either researchers or participants, and moreover, online research will not be effective for populations with limited internet access.

For some studies and populations, it may be necessary to employ more resource-intensive approaches, including targeted and time-and-location sampling strategies, which involve conducting in-person recruitment at specific locations where potential participants gather, such as when recruiting individuals with radicalized political beliefs (Sazak, 2019; Shaghaghi et al., 2011). When target participants are individuals who tend to use specific services (e.g., HIV-positive populations’ attendance in clinics that treat sexually transmitted infections, families involved in child welfare system’s connection to mental-health and substance-abuse programs), researchers can partner directly with the facilities that serve these groups (Mirick, 2016; Shaghaghi et al., 2011). This facility-based sampling has the benefit of connecting researchers with populations that are often hidden and hard to reach but has drawbacks related to gatekeeping by agencies and biasing the sample toward participants who are well engaged with services (Mirick, 2016; Shaghaghi et al., 2011). Another strategy that has been discussed is focused recruitment of specific, individual participants who are identified as eligible through publicly available information (e.g., Facebook profiles; Sikkens et al., 2017). This approach is not widely used because of a combination of resource intensiveness and ethical concerns.

Researching without recruitment

There are at least two avenues for research that do not involve specifically recruiting potentially reluctant participants, instead drawing on anonymized analysis of large data sets. One strategy involves secondary analysis of large-scale studies, such as nationally representative surveys. For example, the Behavioral Risk Factor Surveillance System, conducted by the Centers for Disease Control and Prevention (Centers for Disease Control and Prevention, 2024), allows states to opt in to collecting data on gender identity. Although only a subset of states includes these items and transgender respondents make up only a small fraction of the overall data set, they nonetheless number in the thousands—thus allowing researchers to study health and well-being among transgender populations without having to engage in large-scale recruitment (e.g., Conron et al., 2012; Ferrucci et al., 2021; Narayan et al., 2017).

A second option is to examine information that does not come directly from participants or survey respondents at all, including research analyzing publicly available text or video content (e.g., scraping Twitter posts, analyzing YouTube videos; Grimmer & Stewart, 2013; Miller et al., 2020; Nicolas et al., 2021). These strategies have been useful in seeking to understand the attitudes of populations least likely to choose to participate in any research (e.g., individuals who hold broadly anti-science views; Erviti et al., 2020; Rao et al., 2021). As technological options continue to advance, machine-learning systems are an increasingly useful tool for investigating particularly hard-to-research issues, such as terrorism (Atran et al., 2017) and other similarly challenging social phenomena.

Our case study

In retrospect, we did not give sufficient care and attention to planning participant recruitment in our own project, and we could have engaged in this process more thoughtfully and effectively. One missed opportunity is that we did not engage with the relevant community of interest ahead of time. Doing so could have allowed us to explain the purpose of the research in a direct, face-to-face manner and alerted us to potential concerns raised by community members; ultimately, it may have garnered greater “buy-in” from skeptical families. We also did not include a member of these communities on the leadership of the research team (although some of our undergraduate research assistants were from eligible communities), and thus, we were operating based on our own assumptions or stereotypes about what is important to these families. As recruitment went on, we tried out a range of different strategies (e.g., postings on a university-run recruitment website, Facebook ads, ads in local newspapers, flyers in coffee shops and libraries) but did not track the routes by which participants learned of our study. Had we done so, we could have determined which strategies were most (or least) successful and used that information accordingly.

Data Analysis and Interpretation

Important considerations arise when analyzing and interpreting the data from reluctant populations. Although much of this process takes place after the hard work of data collection is complete, the first such step may occur during initial planning in the form of preregistration. Preregistering one’s study is both valuable and increasingly common (thanks to registries such as OSF and AsPredicted). At the same time, researchers need to be mindful of unanticipated complications that may arise (Nosek et al., 2019), especially those that stem from working with reluctant participants. For studies with many anticipated “unknown unknowns,” researchers can provide broader preregistrations, which allow for more exploratory research and serendipitous discovery while still providing transparency regarding initial plans and hypotheses (Hardwicke & Wagenmakers, 2023). When deviating from preregistered recruitment (as Nosek et al., 2019, noted, “Preregistration is a plan, not a prison,” p. 817), transparency is critical. Although the registered-report format (in which studies are reviewed before data collection) may further limit flexibility, it has not been shown to be less friendly to unexpected findings, and in-principle acceptance ensures that a second round of difficult recruitment will not be required for publication (Briker & Gerpott, 2024; Higgs & Gelman, 2021).

When considering the analytic plan for a study involving understudied populations, it is also important to reflect on assumptions of what constitutes a “neutral” comparison, how to interpret group differences, and whether a comparison group is even needed or appropriate (Roberts & Mortenson, 2023; Sampson, 1993). As one example, the achievement gap between Black and White students in the United States is predicated on measurements that do not value Black children’s particular strengths (e.g., oral-narrative skills; Gardner-Neblett et al., 2023). Examining a participant group on its own merits and exploring the specific and intersectional experiences of group members can provide a more accurate (and more nuanced) picture than a simple comparison, especially one based on unexamined criteria or assumptions that one group’s experiences are normative (Cole, 2009). For qualitative methods, engaging in member checking has the benefit of both sharing findings with participants and ensuring that researcher analyses and interpretations are not out of step with the experience before submitting results for publication (Tebbe & Budge, 2016).

Even after data analysis is complete, interpreting one’s data can be complicated when conducting research with reluctant-participant populations. One major issue is that study samples may be unrepresentative of the broader target groups of interest. For example, in populations experiencing poverty, individuals who participate may have more stable home environments or more extensive educational background than individuals who do not. Likewise, for participants with ideological reasons for their reluctance (e.g., vaccine hesitancy, strongly held religious or other values, or examination of prejudices), individuals who hold more moderate views may be more willing to enroll in a study. Researchers can gather demographic information not only on the final sample but also on the larger target community to identify differences and openly address limitations to generalizability. A similar issue is that given challenges in recruitment, samples may be smaller than desired—a limitation that needs to be noted (Ellard-Gray et al., 2015; Rowley & Camacho, 2015; Sullivan & Cain, 2004).

Our case study

As noted earlier, we conducted this study as a Registered Report; this was altogether a highly positive experience. Doing so meant that we obtained valuable feedback from reviewers during the design phase of the study (before data collection). It also had the benefit of providing a clear roadmap for our data analysis and did not prevent us from conducting and reporting on exploratory analyses (explicitly flagged as such). As noted earlier, we did find that the families in our final sample were less conservative, more educated, and higher income than the communities in which they lived and noted this limitation in our write-up.

Reporting and Broader Impacts

The final phase we discuss involves sharing one’s work: with scientists, with participants and their community, and with the public at large. At minimum, researchers have a responsibility to report findings in ways that are respectful of the participants and their community, appropriately convey participants’ perspectives and context (and are mindful of the researchers’ own biases), and avoid portrayals that are sensationalized, simplistic, or overly generalizing (DeJesus et al., 2019; Simons et al., 2017; Sullivan & Cain, 2004). Failing to do so risks harming not only participants and their community (the most pressing concern) but also the scientific enterprise by undermining participants’ trust in the scientific community and willingness to participate in future research (Rowley & Camacho, 2015). It is also important to acknowledge participants for their contributions to the research; in qualitative work that draws heavily on specific experiences, researchers may even highlight individual participants, should they choose to be identified (Ellard-Gray et al., 2015; Sullivan & Cain, 2004).

Researchers have a responsibility to report findings to participants in ways that are accessible and have the potential to improve their experience while being transparent about what can and cannot be guaranteed (e.g., for a study of domestic violence, this could entail ensuring that a final report will be presented to the domestic-violence agency in which recruitment occurred while clarifying that the team does not have a say in how the agency acts—or does not—on this report; Ellard-Gray et al., 2015). Researchers can bring results back to those places where participants were initially contacted—whether this means posting on websites or social media groups accessed for online recruitment or sharing results with support groups or other meetings—and as possible, endeavor to incorporate results into educational resources or trainings that will benefit participant communities (Tebbe & Budge, 2016).

Our case study

We took no special steps in reporting beyond best practices available to all researchers. In addition to doing our best to write up our report clearly and completely, we thanked the families who joined our study in the acknowledgments. We also asked participating families if they would like to receive an update on the outcomes of the study—everyone who expressed interest was provided a copy of the abstract after publication.

Discussion

In this article, we have argued that “reluctant” participants are essential to psychological research: Certain research questions cannot be fully answered without the participation of such individuals, but more broadly, all research would benefit from making serious efforts to expand sampling beyond individuals who are easiest to reach or most willing to engage. Understandably, recruitment has received the most attention in the literature, but we have argued that meaningfully broadening participation requires attention at each phase of the research process—study design, participant recruitment and testing, data analysis and interpretation, and reporting results—with learning from and about participant communities early in research as an especially important step. To address challenges, researchers will face competing demands and must make difficult decisions that at best will satisfice.

To provide a concrete illustration, we return to the example of our own research with a reluctant population, investigating gender essentialism and prejudice against gender nonconformity in a sample of children from rural, conservative communities in the United States (Gross et al., 2024). In this work, we encountered challenges related to eligibility, recruitment strategy, language choice and use of social media, and generalizing the results. We initially set more stringent criteria for what “counted” as a rural community than in prior work given the goal of focusing on individuals with especially high rates of gender essentialism. Although this approach had the benefit of clarity, we found that it substantially reduced the number of eligible families, leading us to create a more flexible eligibility metric—a compromise that resulted in a sample that was less representative of the ideal (because it was less conservative and less rural). During recruitment, we were transparent about the goals of the study but took care to use language that we hoped would be less politically charged. Nonetheless, our advertisements received many negative comments (e.g., “Sounds like a brainwashing set up” and “Is this about introducing our children to the trans lifestyle?”). Testing over Zoom had the advantage of accessing participants who lived too far away to test in person but meant an increase in fraudulent sign-ups, in some cases, with adults posing as 10-year-old children. Over time, we introduced new strategies for targeting eligible communities, including posting flyers at local libraries and other community spaces and setting up tables at cafés and farmers’ markets within driving distance. We regret not tracking how participants learned of our study, which would have helped us determine which efforts were most effective. Overall, our sense was that studying children’s concepts of gender in a rural, conservative community has become more difficult over time, corresponding to increases in U.S. legislation and scrutiny around gender diversity and gender identity in young people (Ronan, 2021; Trans Legislation Tracker, 2023). More generally, shifting attitudes may require researchers to be nimble and flexible in their approaches.

Every study is different, and the challenges and solutions involved will vary accordingly. Two general lessons, however, transcend these differences. First is the importance of respect for both individual participants and their communities, including the need to understand concerns and historical contexts to minimize discomfort and maximize positive impact. Second is the need to balance competing desiderata. There is a push and pull to research with participant groups that may be reluctant to engage: between the theoretical best match of participant characteristics to study question and resource constraints, between the goal of deeply understanding communities and time constraints, between priorities of the community and priorities of the researchers, between convenience sampling and bias, between reporting negative results and protecting vulnerable groups, and so forth.

We believe that these tensions are inherent to the research process and consequently, that it is important (and most constructive) for researchers to acknowledge them openly and consider how they may have implications for the generality of their research findings (Simons et al., 2017). Ideally, the challenges will provoke researchers to generate new and increasingly productive solutions. Looking to the future, collaborative research teams may be a productive strategy for engaging diverse and potentially hard-to-reach populations—including collaborations between researchers and community members (Ellard-Gray et al., 2015; Rowley & Camacho, 2015) and collaborations across labs and institutions (Byers-Heinlein et al., 2020). Greater transparency in describing samples can help bring clarity to participant groups (DeJesus et al., 2019) and undermine problematic assumptions (e.g., that certain groups are neutral and can represent the larger population; Roberts & Mortenson, 2023). Thoughtful consideration of these issues holds promise for strengthening the value, reach, and transparency of psychological science.

Footnotes

Transparency

Action Editor: David A. Sbarra

Editor: David A. Sbarra

Author Contributions

E. B. Gross: Conceptualization; Writing – original draft.

Susan A. Gelman: Conceptualization; Writing – review & editing.

ORCID iD

E. B. Gross

References

Atran

Axelrod

Davis

Fischhoff

(2017). Challenges in researching terrorism from the field. Science, 355(6323), 352–354. https://doi.org/10.1126/science.aaj2037

Briker

Gerpott

F. H.

(2024). Publishing Registered Reports in management and applied psychology: Common beliefs and best practices. Organizational Research Methods, 27(4), 588–620. https://doi.org/10.1177/10944281231210309

Byers-Heinlein

Bergmann

Davies

Frank

M. C.

Hamlin

J. K.

Kline

Kominsky

J. F.

Kosie

J. E.

Lew-Williams

Liu

Mastroberardino

Singh

Waddell

C. P. G.

Zettersten

Soderstrom

(2020). Building a collaborative psychological science: Lessons learned from ManyBabies 1. Canadian Psychology/Psychologie Canadienne, 61(4), 349–363. https://doi.org/10.1037/cap0000216

Centers for Disease Control and Prevention. (2024). Behavioral risk factor surveillance survey. https://www.cdc.gov/brfss/index.html

Chandler

J. J.

Paolacci

(2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508. https://doi.org/10.1177/1948550617698203

Cheng

L. S. P.

Burgess

Vernooij

Solís-Barroso

McDermott

Namboodiripad

(2021). The problematic concept of native speaker in psycholinguistics: Replacing vague and harmful terminology with inclusive and accurate measures. Frontiers in Psychology, 12, Article 715843. https://doi.org/10.3389/fpsyg.2021.715843

Cole

E. R.

(2009). Intersectionality and research in psychology. American Psychologist, 64(3), 170–180. https://doi.org/10.1037/a0014564

Conron

K. J.

Scott

Stowell

G. S.

Landers

S. J.

(2012). Transgender health in Massachusetts: Results from a household probability sample of adults. American Journal of Public Health, 102(1), 118–122. https://doi.org/10.2105/AJPH.2011.300315

DeJesus

J. M.

Callanan

M. A.

Solis

Gelman

S. A.

(2019). Generic language in scientific communication. Proceedings of the National Academy of Sciences, 116(37), 18370–18377. https://doi.org/10.1073/pnas.1817706116

10.

Ellard-Gray

Jeffrey

N. K.

Choubak

Crann

S. E.

(2015). Finding the hidden participant: Solutions for recruiting hidden, hard-to-reach, and vulnerable populations. International Journal of Qualitative Methods, 14(5). https://doi.org/10.1177/1609406915621420

11.

Erviti

M. C.

Codina

León

(2020). Pro-science, anti-science and neutral science in online videos on climate change, vaccines and nanotechnology. Media and Communication, 8(2), 329–338. https://doi.org/10.17645/mac.v8i2.2937

12.

Ferrucci

K. A.

Walubita

Beccia

A. L.

Ding

E. Y.

Jesdale

B. M.

Lapane

K. L.

Streed

C. G. J.

(2021). Health care satisfaction in relation to gender identity: Behavioral Risk Factor Surveillance Survey, 20 states (2014–2018). Medical Care, 59(4), 312. https://doi.org/10.1097/MLR.0000000000001508

13.

Fine

R. D.

Olson

K. R.

Gülgöz

Horton

Gelman

S. A.

(2024). Gender essentialism predicts prejudice against gender nonconformity in two cultural contexts. Social Development, 33(1), Article e12720. https://doi.org/10.1111/sode.12720

14.

García

A. A.

Zuñiga

J. A.

Lagon

(2017). A personal touch: The most important strategy for recruiting Latino research participants. Journal of Transcultural Nursing, 28(4), 342–347. https://doi.org/10.1177/1043659616644958

15.

Gardner-Neblett

Iruka

I. U.

Humphries

(2023). Dismantling the Black–White achievement gap paradigm: Why and how we need to focus instead on systemic change. Journal of Education, 203(2), 433–441. https://doi.org/10.1177/00220574211031958

16.

Greene

C. M.

Murphy

(2023). Debriefing works: Successful retraction of misinformation following a fake news study. PLOS ONE, 18(1), Article e0280295. https://doi.org/10.1371/journal.pone.0280295

17.

Grimmer

Stewart

B. M.

(2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267–297. https://doi.org/10.1093/pan/mps028

18.

Gross

E. B.

Fine

R. D.

Gülgöz

Olson

K. R.

Gelman

S. A.

(2024). Children’s gender essentialism and prejudice: Testing causal links via an experimental manipulation. Developmental Science, 27(5), Article e13532. https://doi.org/10.1111/desc.13532

19.

Hardwicke

T. E.

Wagenmakers

E.-J.

(2023). Reducing bias, increasing transparency and calibrating confidence with preregistration. Nature Human Behaviour, 7(1), 15–26. https://doi.org/10.1038/s41562-022-01497-2

20.

Henrich

Heine

S. J.

Norenzayan

(2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 111–135. https://doi.org/10.1017/S0140525X10000725

21.

Higgs

M. D.

Gelman

(2021). Research on registered report research. Nature Human Behaviour, 5(8), 978–979. https://doi.org/10.1038/s41562-021-01148-y

22.

Hilário

A. P.

Scavarda

Numerato

Mendonça

Cardano

Marhankova

Gariglio

Vuolanto

Anderson

Auvinen

Bracke

Douglass

Hobson-West

Lermytte

Polak

Rudek

(2023). Recruiting a hard-to-reach, hidden and vulnerable population: The methodological and practical pitfalls of researching vaccine-hesitant parents. Qualitative Health Research, 33(13), 1189–1202. https://doi.org/10.1177/10497323231196439

23.

Horne

Powell

Hummel

J. E.

Holyoak

K. J.

(2015). Countering antivaccination attitudes. Proceedings of the National Academy of Sciences, 112(33), 10321–10324. https://doi.org/10.1073/pnas.1504019112

24.

Johnston

L. G.

Sabin

(2010). Sampling hard-to-reach populations with respondent driven sampling. Methodological Innovations Online, 5(2), 38–48. https://doi.org/10.4256/mio.2010.0017

25.

Kidd

Garcia

(2022). How diverse is child language acquisition research? First Language, 42(6), 703–735. https://doi.org/10.1177/01427237211066405

26.

Lawlor

Thomas

Guhin

A. T.

Kenyon

Lerner

M. D.

Drahota

(2021). Suspicious and fraudulent online survey participation: Introducing the REAL framework. Methodological Innovations, 14(3). https://doi.org/10.1177/20597991211050467

27.

Levac

Ronis

Cowper-Smith

Vaccarino

(2019). A scoping review: The utility of participatory research approaches in psychology. Journal of Community Psychology, 47(8), 1865–1892. https://doi.org/10.1002/jcop.22231

28.

Lewis

N. A.

Bravo

Naiman

Pearson

A. R.

Romero-Canyas

Schuldt

J. P.

Song

(2020). Using qualitative approaches to improve quantitative inferences in environmental psychology. MethodsX, 7, Article 100943. https://doi.org/10.1016/j.mex.2020.100943

29.

Marshall

Swerissen

Biuso

(2002). Community recruitment of non-immunisers and partial immunisers: Problems encountered. Health Promotion Journal of Australia, 13(2), 62–64. https://doi.org/10.3316/informit.456365399859989

30.

McShane

K. E.

Davey

C. J.

Rouse

Usher

A. M.

Sullivan

(2015). Beyond ethical obligation to research dissemination: Conceptualizing debriefing as a form of knowledge transfer. Canadian Psychology/Psychologie Canadienne, 56(1), 80–87. https://doi.org/10.1037/a0035473

31.

Miller

Linder

Mebane

W. R.

(2020). Active learning approaches for labeling text: Review and assessment of the performance of active learning approaches. Political Analysis, 28(4), 532–551. https://doi.org/10.1017/pan.2020.4

32.

Mirick

R. G.

(2016). Challenges in recruiting parents to participate in child welfare research: Implications for study design and research practice. Child & Family Social Work, 21(4), 484–491. https://doi.org/10.1111/cfs.12165

33.

Narayan

Lebron-Zapata

Morris

(2017). Breast cancer screening in transgender patients: Findings from the 2014 BRFSS survey. Breast Cancer Research and Treatment, 166(3), 875–879. https://doi.org/10.1007/s10549-017-4461-8

34.

Nicks

S. D.

Korn

J. H.

Mainieri

(1997). The rise and fall of deception in social psychology and personality research, 1921 to 1994. Ethics & Behavior, 7(1), 69–77. https://doi.org/10.1207/s15327019eb0701_6

35.

Nicolas

Bai

Fiske

S. T.

(2021). Comprehensive stereotype content dictionaries using a semi-automated method. European Journal of Social Psychology, 51(1), 178–196. https://doi.org/10.1002/ejsp.2724

36.

Nielsen

Haun

Kärtner

Legare

C. H.

(2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. https://doi.org/10.1016/j.jecp.2017.04.017

37.

Nosek

B. A.

Beck

E. D.

Campbell

Flake

J. K.

Hardwicke

T. E.

Mellor

D. T.

Van ’T Veer

A. E.

Vazire

(2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815–818. https://doi.org/10.1016/j.tics.2019.07.009

38.

Oruche

U. M.

Gerkensmeyer

J. E.

Austin

J. K.

Perkins

S. M.

Scott

Lindsey

L. M.

Mullins

(2012). Recruitment strategies for caregivers of children with mental health problems. Clinical Nurse Specialist, 26(4), 216–221. https://doi.org/10.1097/NUR.0b013e31825ae9fb

39.

Patel

M. X.

Doku

Tennakoon

(2003). Challenges in recruitment of research participants. Advances in Psychiatric Treatment, 9(3), 229–238. https://doi.org/10.1192/apt.9.3.229

40.

Purdue Center for Regional Development. (2024). The Rural Urban Continuum Code: What changed? Purdue Center for Regional Development. https://pcrd.purdue.edu/the-rural-urban-continuum-code-what-changed/

41.

Rao

Morstatter

Chen

Burghardt

Ferrara

Lerman

(2021). Political partisanship and antiscience attitudes in online discussions about COVID-19: Twitter content analysis. Journal of Medical Internet Research, 23(6), e26692. https://doi.org/10.2196/26692

42.

Roberts

S. O.

Bareket-Shavit

Dollins

F. A.

Goldie

P. D.

Mortenson

(2020). Racial inequality in psychological research: Trends of the past and recommendations for the future. Perspectives on Psychological Science, 15(6), 1295–1309. https://doi.org/10.1177/1745691620927709

43.

Roberts

S. O.

Mortenson

(2023). Challenging the White = neutral framework in psychology. Perspectives on Psychological Science, 18(3), 597–606. https://doi.org/10.1177/17456916221077117

44.

Robinson

J. M.

Trochim

W. M. K.

(2007). An examination of community members’, researchers’ and health professionals’ perceptions of barriers to minority participation in medical research: An application of concept mapping. Ethnicity & Health, 12(5), 521–539. https://doi.org/10.1080/13557850701616987

45.

Rockliffe

Chorley

A. J.

Marlow

L. A. V.

Forster

A. S.

(2018). It’s hard to reach the “hard-to-reach”: The challenges of recruiting people who do not access preventative healthcare services into interview studies. International Journal of Qualitative Studies on Health and Well-Being, 13(1), Article 1479582. https://doi.org/10.1080/17482631.2018.1479582.

46.

Ronan

(2021). 2021 officially becomes worst year in recent history for LGBTQ state legislative attacks as unprecedented number of states enact record-shattering number of anti-LGBTQ measures into law [Press release]. Human Rights Campaign. https://www.hrc.org/press-releases/2021-officially-becomes-worst-year-in-recent-history-for-lgbtq-state-legislative-attacks-as-unprecedented-number-of-states-enact-record-shattering-number-of-anti-lgbtq-measures-into-law

47.

Rowley

S. J.

Camacho

T. C.

(2015). Increasing diversity in cognitive developmental research: Issues and solutions. Journal of Cognition and Development, 16(5), 683–692. https://doi.org/10.1080/15248372.2014.976224

48.

Sampson

E. E.

(1993). Identity politics: Challenges to psychology’s understanding. American Psychologist, 48(12), 1219–1230.

49.

Sazak

O. F.

(2019). Recruiting strategies in studies about radicalization and extremism. Beyond the Horizon International Strategic Studies Group. https://behorizon.org/recruiting-strategies-in-studies-about-radicalization-and-extremism/

50.

Shaghaghi

Bhopal

R. S.

Sheikh

(2011). Approaches to recruiting ‘hard-to-reach’ populations into research: A review of the literature. Health Promotion Perspectives, 1(2), 86–94. https://doi.org/10.5681/hpp.2011.009

51.

Shaia

W. E.

Nichols

H. M.

Dababnah

Campion

Garbarino

(2020). Brief report: Participation of Black and African-American families in autism research. Journal of Autism and Developmental Disorders, 50(5), 1841–1846. https://doi.org/10.1007/s10803-019-03926-0

52.

Shavers-Hornaday

V. L.

Lynch

C. F.

Burmeister

L. F.

Torner

J. C.

(1997). Why are African Americans under-represented in medical research studies? Impediments to participation. Ethnicity & Health, 2(1–2), 31–45. https://doi.org/10.1080/13557858.1997.9961813

53.

Sikkens

van San

Sieckelinck

Boeije

de Winter

(2017). Participant recruitment through social media: Lessons learned from a qualitative radicalization study using Facebook. Field Methods, 29(2), 130–139. https://doi.org/10.1177/1525822X16663146

54.

Simons

D. J.

Shoda

Lindsay

D. S.

(2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 2(6), 1123–1128.

55.

Sullivan

C. M.

Cain

(2004). Ethical and safety considerations when obtaining information from or about battered women for research purposes. Journal of Interpersonal Violence, 19(5), 603–618. https://doi.org/10.1177/0886260504263249

56.

Tebbe

E. A.

Budge

S. L.

(2016). Research with trans communities: Applying a process-oriented approach to methodological considerations and research recommendations. The Counseling Psychologist, 44(7), 996–1024. https://doi.org/10.1177/0011000015609045

57.

Trans Legislation Tracker. (2023). 2023 Passed anti-trans bills. https://translegislation.com/bills/2023/passed

58.

Vincent

B. W.

(2018). Studying trans: Recommendations for ethical recruitment and collaboration with transgender participants in academic research. Psychology & Sexuality, 9(2), 102–116. https://doi.org/10.1080/19419899.2018.1434558

59.

Wiley

K. E.

Leask

Attwell

Helps

Barclay

Ward

P. R.

Carter

S. M.

(2021). Stigmatized for standing up for my child: A qualitative study of non-vaccinating parents in Australia. SSM - Population Health, 16, Article 100926. https://doi.org/10.1016/j.ssmph.2021.100926

60.

Young

A. M.

Ballard

A. M.

Cooper

H. L. F.

(2020). Novel recruitment methods for research among young adults in rural areas who use opioids: Cookouts, coupons, and community-based staff. Public Health Reports, 135(6), 746–755.

Research With the Reluctant: Challenges and Strategies

Abstract

Keywords

Framing the Issue

Study Design and Planning

Identifying the population of interest

Understanding the population of interest

Defining the sample

Ethical considerations and constraints

Our case study

Participant Recruitment

Researching without recruitment

Our case study

Data Analysis and Interpretation

Our case study

Reporting and Broader Impacts

Our case study

Discussion

Footnotes

Transparency

ORCID iD

References