Sage Journals: Discover world-class research

Abstract

This paper examines the effect of having a third-party scientific expert present in stakeholder interviews. The study was conducted as part of a larger project on stakeholder engagement for natural resource management in the Verde Valley region of Arizona. We employed an experimental design, conducting stakeholder interviews both with and without an identified scientific expert present. Our sample consisted of 12 pairs of interviewees (24 total participants) who we matched based on their occupation, sex, and spatial proximity. For each pair, the scientific expert was present as a third party in one interview and absent in the other. We used a word-based coding strategy to code all interview responses for three known areas of sensitivity among the study population (risk, gatekeeping, and competence). We then performed both quantitative and qualitative analyses to compare responses across the two interview groups. We found that the presence of a scientific expert did not have a statistically significant effect on the mention of sensitive topics among stakeholders. However, our qualitative results show that the presence of a scientific expert had subtle influences on the ways that stakeholders discussed sensitive topics, particularly in placing emphasis on their own credibility and knowledge. Our findings indicate that researchers may be able to pursue collaborative, interdisciplinary research designs with multiple researchers present during interviews without concerns of strongly influencing data elicitation on sensitive topics. However, researchers should be cognizant of the subtle ways in which the presence of a third-party expert may influence the credibility claims and knowledge assertions made by respondents when a third-party expert is present during stakeholder interviews.

Keywords

response effects stakeholder research stakeholder interviewing community-engaged research collaborative research

Introduction

Stakeholder research—research that assesses the views of people who have key interests in a natural resource system (Grimble & Wellard, 1997)—has become a popular approach to understanding contemporary problems and deriving novel, community-based solutions. Given the complexity of contemporary problems, stakeholder research is often carried out by collaborative teams of researchers from diverse disciplinary backgrounds. This means that stakeholder interviews may be increasingly conducted with more than one researcher present. Yet, the presence of a third-party in an interview has the potential to influence interviewee responses (Z. N. Mneimneh et al., 2018). A key concern for the growing field of stakeholder methodology, then, is whether or not the presence of a third-party scientific expert may affect the validity of stakeholder interview data.

The tendency of interviewees to adjust their responses when a third-party is present is a common response effect known as the “third-party-present-effect” (Bernard, 2011). Third-parties are by no means unique to stakeholder interviewing. Their presence is often unavoidable in a variety of research contexts—whether the third party is an interviewee’s family member, a caregiver, or simply a bystander in a public area (Boeiji, 2004; Gfroerer, 1985; Reuband, 1992; Smith, 1997). Methodological research has thus sought to systematically understand when and how the presence of a third party may affect the scope, the validity, or the reliability of interview data. This research has found that the extent of a third-party’s influence is largely determined by two factors: (a) the sensitivity of the questions being asked (Barnett, 1998; Boeije, 2004; Diop et al., 2015; Mneimneh et al., 2015), and (b) the power differentials between parties involved (Cronk et al. 2009; Taietz, 1962; Tourangeau & Yan, 2007; Zipp & Toth, 2002). To date, however, no research has examined the potential effect of a third-party scientific expert. This is of particular concern because scientific experts may be viewed as elite or powerful individuals, based on their expertise and their potential to influence resource allocation. Furthermore, stakeholder research often asks interviewees about sensitive topics like political power, inequities, resource allocation, and expertise (Lukasiewicz & Baldwin, 2017; Wutich et al., 2010). Both these factors may cause stakeholders to augment their response in ways that affect the validity of interview data.

To address this gap in the methodological literature, we examine the effect of having a scientific expert present during stakeholder interviews conducted as part of a large-scale collaborative, interdisciplinary natural resource management project. In particular, we explore if the presence of a third-party scientific expert affects stakeholders’ discussions of sensitive topics—those that are threatening or have potential costs to the interviewee or other research participants (Lee & Renzetti, 1990). Understanding the extent and nature of the third-party effect in stakeholder interviews can help to improve the interview process among stakeholders, which is crucial for collaborative community-engaged research.

What Is Known About Third-Party Interview Effects?

Previous research on the presence of third parties during face-to-face interviews demonstrates that third-party presence has the potential to influence interviewees’ responses (Aquilino, 1993; Barnett, 1998; Boeije, 2004; Casterline & Chidambaram, 1984; Cronk, Gerkey, and Irons 2009; Z. M. Mneimneh et al., 2015; Taietz, 1962; Zipp & Toth, 2002). Much of this research has focused on the third-party presence of a spouse during an interview. For example, Aquilino (1993) found that respondents gave more positive responses to questions about the importance of marriage in the presence of their spouses. Boeije’s (2004) research on couples facing multiple sclerosis found that a spousal presence during interviews prevented interviewees talking about sensitive issues pertaining to themselves, their spouse, or their relationship. After spouses left the room, however, Boeije’s interviewees spoke openly and critically about their partners and their relationship. Similarly, in their research on women’s contraceptive knowledge, Casterline and Chidambaram (1984) found that a husband’s presence hindered women’s responses about their contraceptive knowledge.

Other research indicates that third parties have little to no effect on interviewees’ responses (Lau et al., 2017; Pollner & Adams, 1994, 1997; Quetulio-Navarra et al., 2015; Smith, 1997). Pollner and Adams’ (1994) research on mental health disorders found little difference in how interviewees reported symptoms or performed on tests of cognitive impairment in the presence of spouses or children. In subsequent research, Pollner and Adams (1997) investigated emotional support provided by spouses and household divisions of labor, again finding that spousal presence did not significantly influence respondents’ answers (Pollner & Adams, 1997). Smith (1997) examined a wide range of responses given in the General Social Survey and concluded that the third-party effect, “is rare and mostly small,” finding that the only significant third-party effect was that respondents were less-likely to indicate approval of pre-marital sex in the presence of an older child (over the age of 6).

Given these mixed results, researchers have sought to better understand the specific conditions that may factor into the extent of the third-party effect on interview responses. Several studies indicate that both the nature of topics being discussed and the role and/or relationship of the third party to the interviewee determine whether or not an interviewee adjusts their response. Topics that are sensitive and pose a potential threat to an interviewee appear to be more subject to the third-party effect (Barnett, 1998; Boeije, 2004; Diop et al., 2015; Milewski & Otto, 2017; Z. M. Mneimneh et al., 2015). For example, based on data from 22,070 interviews from the World Mental Health Survey Initiative, Z. M. Mneimneh and colleagues (2015) found that respondents curtailed their reporting of sensitive information in the presence of a third-party, but they did not curtail their reporting of neutral information.

The role and/or relationship of the third party to the interviewee also appears to be a significant factor in the extent of the third-party effect (Cronk, Gerkey, and Irons 2009; Herrera et al., 2017; Müller, 2019; Taietz, 1962; Tourangeau & Yan, 2007; Zipp & Toth, 2002). For example, Tourangeau and Yan’s (2007) meta-analysis concluded that the presence of parents significantly reduced children’s reporting of socially undesirable information. Additionally, Zipp and Toth (2002) found that husbands and wives reported greater agreement on attitudinal and behavioral beliefs in each other’s presence, but the source of the increased agreement was different for men and women. Women were more likely to agree with their husbands (i.e., when their husbands were third parties), and husbands were less likely to agree with their wives (i.e., when their wives were third parties). Zipp and Toth argue that this difference likely stems from an imbalance in marital power and commitment to gender roles. Such research indicates that the third-party effect is highly context specific and that both the sensitivity of the topics discussed and the power-dynamics between the third party and the interviewee are significant factors in the extent of the effect.

The contextual nature of the third-party effect and the mixed results of previous research call for further studies with strong methodological designs to better understand the extent of this effect in broad array of interview situations (Smith, 1997; Zipp & Toth, 2002). Given that previous third-party effect research has overwhelmingly focused on the presence of household members—and concludes that the power dynamics between the third party and interviewee are a key factor in how the interviewee may adjust responses—there is an unmet need to broaden research on the third-party effect, especially to examine its potential impacts in the context of other types of power differentials between the third-party and the interviewee. As persons who hold positions of prestige and the power to influence resource allocations, but are not members of the interviewee’s household, scientific experts may have a unique effect on the ways that interviewees report sensitive information.

Study Background

This study is part of a larger USDA-NIFA funded project called, “Water for Agriculture,” which aims to understand the processes and outcomes of stakeholder engagement in agricultural water management in five communities across Arizona, Nebraska, and Pennsylvania in the United States. To help fulfill the project’s aim of transdisciplinary collaboration, the Arizona team’s biophysical scientist was present for baseline semi-structured interviews with the stakeholders conducted by a social scientist. The biophysical scientist’s participation in interviews provided him with firsthand knowledge about stakeholders’ interests and concerns, so he may proactively think about potential community-driven research. Stakeholders of the Arizona site include farmers, ranchers, conservationists, consultants, water managers, and government officials from local, county, state, and federal agencies who live or work in the study site (Arizona’s Verde Valley). We were concerned, however, that the presence of a biophysical scientist could affect interview responses, as literature shows with the presence of other third parties. Very little previous research examines third-party effects of a topical expert on interview responses, so we built an experiment into the research design of the baseline interviews in the Arizona study site.

Research Methods

Study Design & Rationale

To assess the third-party effect of a biophysical scientist in stakeholder interviews, we employed an experimental design. Twelve pairs of interviewees were matched on key characteristics; each interviewee (n = 24) was interviewed once. One matched interviewee (n = 12) had the scientific expert present during the interview; the other matched interviewee (n = 12) had the scientific expert absent during the interview. This means we had two sets of interviews: half of the interviewees with the scientific expert present; half of the interviewees with scientific expert absent.

We matched interviewees to make it more likely that any differences in matched pairs’ responses were due to the expert’s presence. To ensure that stakeholder interviewees across matched pairs were likely to be similar—in terms of knowledge sets and perspectives on natural resource management in Arizona’s Verde Valley—we matched interviewees three criteria: occupation (e.g., farmers or ranchers or government officials), gender, and geographical location (i.e., lived/worked within 25 miles of one another) (see Table 1). This helped to ensure that any differences we noted across the two sets of interviews were due to the expert’s presence and not due to differences in stakeholders’ occupations, sex, or geographical location. All stakeholder interview participants were White and between the ages of 26 and 76.

Table 1.

Characteristics of interview pairs. The research sample included 12 matched pairs of interviewees (24 individual participants). Across each pair, the scientific expert was absent in one interview and present in the other interview. Interviewee pairs were matched based on the interviewees’ occupation, geographic proximity (residing or working within 25 miles of each other) and gender.

Interviewees w/Expert Absent(Participant ID)	Interviewees w/Expert Present(Participant ID)	Interviewee Occupation	Interviewee Geographic Proximity (located w/in 25 miles from one another in the Verde Valley)	Interviewee Gender
AZ-40	AZ-38	Government Official	Y	M
AZ-5	AZ-7	Government Official	Y	M
AZ-35	AZ-31	Government Official	Y	M
AZ 39	AZ 37	Government Official	Y	M
AZ-20	AZ-29	Government Official	Y	F
AZ-13	AZ-17	Agricultural Producer	Y	M
AZ-2	AZ-34	Agricultural Producer	Y	M
AZ-28	AZ-32	Agricultural Producer	Y	F
AZ-36	AZ-8	Agricultural Producer	Y	M
AZ-33	AZ-16	Conservationist	Y	F
AZ-12	AZ-9	Conservationist	Y	M
AZ-11	AZ-41	Conservationist	Y	M

Data Collection

The semi-structured interviews were conducted in person between March 2018 and May 2019. The study received ethical approval under IRB# STUDY00007549 at the Institutional Review Board Office of Arizona State University. All stakeholder interviewees were informed about the study purpose prior to the interview, and all gave their verbal informed consent to participate and have their responses used in scientific publications. Interviews lasted from one to two-and-a-half hours. We asked participants about the primary water/agriculture issues in the region (e.g., irrigation, water quality, water rights); the role of science and expertise with those issues; key stakeholders and collaborations; previous efforts in stakeholder engagement; and future opportunities for stakeholder engagement to address those issues.

One social scientist, Julia C. Bausch, conducted all the interviews for the Arizona site, and one biophysical scientist, Clinton Williams, served as the biophysical science expert. The interview protocol was identical for both groups. In the third-party present interviews, the biophysical scientist introduced himself at the outset to establish his professional expertise and role on the project. He truthfully described himself as a scientist with the USDA Agricultural Research Service (USDA-ARS), with expertise in soil chemistry and soil physics, and member of the research project’s biophysical science team. For further evidence of his expertise and professional credentials, he gave the participant his business card, and wore his USDA-ARS identification badge. While the social scientist conducted the interview, the biophysical expert observed silently and took notes. If a participant spoke to the biophysical expert during the interview, he would give a short response, and suggest following up on any questions or details at the end of the interview. After the social scientist had asked all the questions in the semi-structured protocol, she would turn to the biophysical expert and ask him if he had any comments or questions.

Data Analysis

The interviews were audio-recorded and transcribed verbatim. To determine potential differences in the ways that stakeholders discussed sensitive topics with the expert present versus the expert absent, we performed both quantitative and qualitative word-based analyses. We conducted quantitative word-based analysis to statistically determine if there was a significant difference in the mention of sensitive topics across interview groups (those in which the scientific expert was present vs. those in which the scientific expert was absent). And we conducted qualitative word-based analysis to determine how stakeholders might talk about sensitive topics differently during interviews in which a scientific expert was present vs. during interviews in which a scientific expert was absent. Our analysis steps are outlined below.

Selection and coding of key terms

Before analyzing our transcripts, we outlined three known areas of sensitivity among water stakeholders in Arizona: risk, gatekeeping, and competence (Wutich et al., 2010). Risk pertains to discussions of decision-making around water and the impacts and trade-offs that result from those decisions (Wutich et al., 2010). Discussions of the risks around water decision-making are sensitive for interview participants because they may reveal socially unacceptable opinions about such decisions (Wutich et al., 2010). Gatekeeping pertains to discussions of powerbrokers and boundary management around water (Wutich et al., 2010). Discussions of gatekeeping around water are sensitive for interview participants because they may reveal their own political alliances or oppositions (Wutich et al., 2010). Competence pertains to discussions of knowledge and expertise around water issues (Wutich et al., 2010). Discussions of competence around water issues are sensitive for interview participants because they may reveal deficiencies in their own knowledge or expertise thus exposing them to status loss (Wutich et al., 2010).

Based on these theoretically predetermined areas of sensitivity, two members of our team with extensive ethnographic knowledge of water issues in the Verde Valley, Sarah Porter and Julia C. Bausch, then identified a broad array of words (approximately 15−20 words for each category) that represent each of these sensitive areas. Words that represent issues of risk around water in the Verde Valley included “groundwater,” “economy,” and “growth.” Words that represent issues of gatekeeping included “Forest Service,” “SRP” (Salt River Project—the primary water provider in central Arizona), and “fed*” (a word root set to capture the slang term “feds” or the formal term “federal” in reference to the federal government). Words that represent issues of competence included “science,” “research,” and “monitor.” We used a qualitative data management software program (VERBI GmbH MAXQDA) to search for and code all interview transcripts for the theoretically-determined words at the level of the sentence.¹ We then systematically cleaned the data set to delete any coded segments that did not accurately capture a sensitive topic (e.g., description of the interviewee’s background such as college major, discussion of a location outside of the Verde Valley). Finally, we selected the most frequently used words in each category—six words per theoretical category for a total of 18 words—to proceed with our analysis (see Table 2).

Table 2.

Analysis of Most Frequently Used Words. The six most frequently chosen words representing each thematic area of sensitivity. Numbers in columns indicate how many interviewees with the scientific expert absent (“Absent” Column) and how many interviewees with the scientific expert present (“Present” column) mentioned the word representing a thematic area of sensitivity (RISK, GATEKEEPING, or COMPETENCE). Counts were conducted for the mention of the word in the interview (Simply dichotomized); the mention of the word above the mean number of mentions (Dichotomized at mean break-point); the mention of the word above the median number of mentions (Dichotomized at median break-point). *Asterisk indicates a word root used to capture related words. ^†Prescott is a community to the northwest of the verde valley. ^‡Adjudication is the legal process through which water rights are determined in arizona.

		Present in the interview (simply dichotomized)		Present at or above MEAN			Present at or above the MEDIAN
THEME	WORD	Expert Absent(n = 12)	Expert Present(n = 12)	Mean	Expert Absent(n = 12)	Expert Present(n = 12)	Median	Expert Absent(n = 12)	Expert Present(n = 12)
RISK	Groundwater	9	7	3.1	2	3	2	7	7
	Economy	5	5	0.7	5	5	0	5	5
	Growth	7	8	1.7	3	5	1	7	8
	Rights	10	11	7.5	5	4	4.5	8	4
	Financial	5	3	0.6	5	3	0	5	3
	Adjudication^‡	7	10	3.2	3	4	1.5	6	6
	Total mentions of words representing RISK	50	58		23	24		38	33
GATEKEEPING	Forest Service	7	4	2.5	4	2	0	7	4
	SRP	11	9	3.7	3	6	2	6	7
	Salt River Project	5	4	0.6	5	4	0	5	4
	Prescott^†	7	7	3.6	3	7	1	7	7
	Nation	7	6	1.3	4	3	1	7	6
	Fed* (e.g. Federal)	7	8	1.7	2	7	1	7	8
	Total mentions of words representing GATEKEEPING	56	50		21	29		39	36
COMPETENCE	Stud* (e.g. Studies)	8	9	3.5	3	5	1	8	9
	Science	6	6	3.0	0	5	0.5	6	6
	Research	7	6	1.7	4	4	1	7	6
	Monitor	4	6	1.2	2	7	0	4	6
	Educate	6	5	1.3	5	4	0	6	5
	Data	3	6	2.7	2	3	0	3	6
	Total mentions of words representing COMPETENCE	46	50		16	28		34	38

Determine statistical differences in the mention of sensitive topics

Our coding produced a word-by-respondent matrix containing counts of how many times each interviewee mentioned each one of the 18 sensitive topics. Because we were interested in whether or not respondents mentioned a sensitive topic (not how many times they discussed a sensitive topic), we dichotomized the data to turn the counts into nominal (present or absent) variables. We performed this dichotomization in three ways: First, we simply dichotomized the counts based on whether or not the respondent mentioned the topic in the interview. Second, we dichotomized based on the mean (i.e., we coded the topic for each respondent only if it appeared at or above than the mean number of mentions across all interviewees). Third, we dichotomized based on the median (i.e., we coded the topic for each respondent only if it appeared at or above the median number of mentions across all interviewees) (See Table 2).

We then performed a series of one-way ANOVA tests with bootstrapping to statistically compare the mention of sensitive topics between interviews with the scientific expert present versus interviews with the scientific expert absent. Tests were performed for each area of sensitivity (risk, gatekeeping, competence) separately. To account for the small number of interviews, we bootstrapped at the 95% confidence level with 500 re-samples. Bootstrapping is a common approach to ensure the normality assumption for ANOVA tests is met (Pek et al., 2018) and to address the small sample size through the use of resampling to better estimate the distribution of the population. We performed this analysis three times for each area of sensitivity: for the simply dichotomized counts, for word counts with a mean break-point, and for word counts with a median break-point.

Determine qualitative differences in the discussion of sensitive topics

Second, we examined the data for differences in how interviewees spoke about sensitive topics in the presence of a scientific expert. To assess potential qualitative differences, we performed an inductive thematic analysis of all coded segments. We used the constant comparative method (Boeije, 2002) to identify key themes within each sensitive topic category (risk, gatekeeping, and competence) and assess how those themes varied across the two groups of interviews (those with the expert present and those with the expert absent).

Results

Our quantitative analysis showed no statistically significant difference in the mention of sensitive topics between the two interview groups. However, our qualitative results showed thematic differences in how interviewees talked about sensitive topics when the scientific expert was present. We outline our results below.

Risk

When discussing topics that pertain to risk, we found no significant difference in the mention of risk between interviewees with the scientific expert present vs. interviewees with the scientific expert absent at the p < .05 level. This was the case for the simply dichotomized counts [F(1, 10) = .013, p = 0.913], the mean break-point counts [F(1,10) = .065, p = 0.804], and the median break-point counts [F(1,10) = .0839, p = 0.381].

Indeed, our qualitative results showed that both groups of interview participants frequently discussed risks around securing water rights. Typical responses across all interviews included statements like, “I personally don’t want to see surface water rights lost from the Verde Valley” (AZ 40—expert absent). And, “the Verde River Conservation is a slippery slope to taking away their water rights” (AZ 12—expert absent). And, “people have this water rights thing…I mean, the river could dry up and they would still never give up their water rights” (AZ 7—expert present).

However, two major themes pertaining to risk appeared in the interviews in which the scientific expert was present that were not salient in interviews in which the scientific expert was absent: (1) the benefits and impacts of agriculture for the economy and (2) challenges to water management that stem from other people’s lack of awareness and knowledge about water rights and water systems. When the scientific expert was present, respondents spoke readily about the benefits of agriculture for the economy with statements such as, “But that is what’s right now kind of driving our economy, is the wine industry” (AZ 7—expert present). And, “The cattle growers are very good for our economy…I don’t know what we’d do if we didn’t have these guys up there really kind of making sure there’s water” (AZ 37—expert present). While it is difficult to draw definitive conclusions, this may be because the scientific expert’s affiliation with the USDA, together with the project’s interest in supporting agriculture (apparent in the name “Water for Agriculture”) resulted in agriculture becoming a more explicit focus of participants’ responses. It also possibly reflects participants’ efforts to appease the expert (and interviewer) by articulating a position favorable to agriculture.

Additionally, with the scientific expert present respondents spoke extensively about other people’s lack of awareness and knowledge around water with statements such as, “I think a lack of education with certain things is a big factor within the Verde Valley in general—as far as historic education, what people’s independent rights are with the water, and things of that nature” (AZ 16—expert present). And, “the public is aware because water is always a conversation, but it seems like it’s more of a conversation to be used as a scare tactic by anti-growth people than about people trying to make you just more aware of the resource” (AZ 31—expert present). These and similar quotes may indicate that participants were deflecting lack of knowledge from themselves to others in the presence of an expert, and/or perceived that a scientist would be sympathetic to calls for more education.

In interviews in which the scientific expert was absent, however, respondents did not discuss the benefits of agriculture or the challenges of water management due to lack of awareness. Rather, when the scientific expert was absent, respondents widely discussed the risks of running out of water. These discussions often referred to decreased water availability due to increased population growth. For example, when the scientific expert was absent, respondents said, “as more wells go in, they can interfere with existing [water] levels, drop the water tables so people have to deepen their wells, which costs us money” (AZ 33—expert absent). And, “the biggest threat to water is growth and people coming in” (AZ 39—expert absent). And, “Cause that’s what we’re doing. We’re mining groundwater” (AZ 20—expert absent). This may be because participants felt more confident asserting their views of cause and effect as fact in the absence of the scientific expert.

Gatekeeping

When looking at sensitive topics of gatekeeping, we also found no statistically significant difference in the mention of gatekeeping between the two groups at the p < .05 level. Again, this was the case for the simply dichotomized counts [F(1,10) = .738, p = 0.411], the mean break-point counts [F(1,10) = 1.882, p = 0.200], and the median break-point counts [F(1,10) = .429, p = 0.527].

Our qualitative results showed that respondents in both groups spoke readily about issues of gatekeeping, often mentioning the role that gatekeepers played in controlling water rights in the Verde Valley. Common statements about issues of gatekeeping and water rights included, “Now the people that have been around a long time lived through times when their claims were threatened…by SRP or somebody else, usually SRP ‘cause they’re the big guy on the block” (AZ 12—expert absent). And, “So, here we have Hauser Farming, the biggest irrigators in the Verde Valley. Yavapai Apache Nation second-biggest. Arizona State Parks has water claims, so does Freeport so they kind of control the water” (AZ 41—expert absent).

However, while both groups of respondents mentioned issues of gatekeeping, our qualitative results show clear thematic differences in how participants talk about gatekeepers in the presence of a scientific expert. With the scientific expert present, respondents focused heavily and solely on the negative aspects of gatekeepers. For example, when the scientific expert was present, respondents commonly complained about SRP (the primary public water utility in central Arizona). Comments about SRP included, “SRP came in and started taking out some of the trees, you know, in the river bottom to help the flow, which was fine, except they got greedy…and people didn’t want the banks torn up that much” (AZ 34—expert present). And, “it’s all based on the fight, ongoing fight, that everybody in northern Arizona has with SRP over what is and isn’t appropriate flows of the very end usable water in the Verde” (AZ 31—expert present). And, “SRP is their, is the arch nemesis” (AZ 37—expert present). With the expert present, respondents did not mention positive aspects of SRP or other gatekeepers. Here again it is difficult to draw definitive conclusions: it may be that these participants viewed the expert as a potential ally for their concerns, or saw the expert as a proxy for, or reminder of, gatekeepers. More neutrally, they may have been acknowledging the political nature of the water conflict, much of which scientific evidence cannot resolve.

However, when the scientific expert was absent, respondents either solely mentioned positive aspects of gatekeepers (such as giving examples of gatekeepers engaging in the community, collaborating effectively, being open to finding consensus of values and being supportive of other organizations and individuals), or respondents spoke about the positive and negative aspects of gatekeepers in balance. Solely positive comments about gatekeepers included statements like, “SRP has their own great conservation projects. And the Forest Service…does too” (AZ 11—expert absent). And, “It was all public, everybody was there. Friends of the Prescott Forest, everybody was there. And as soon as we started talking amongst each other about ‘oh, you are interested in this too, so are we’” (AZ 28—expert absent). Comments that focused on positive and negative aspects of gatekeepers included statements like, “And so well, SRP, when we get a little closer, will help us do their severance and transfer and find where we can move stuff,” while later in the interview mentioning, “I find it horribly constricting for us in particular because it’s the Forest Service” (AZ 5—expert absent). Another respondent stated, “…and SRP…you’re working hand in hand with them,” while commenting later in the interview that, “it appears that Salt River Project wants to control that also, and the flow of the river; it almost looks like they want to be the water czar for the state” (AZ 39—expert absent). These and similar quotes reflecting a more balanced perspective may be further evidence that the expert was seen as a proxy, or (negative) reminder of gatekeepers and existing conflicts over water rights and water management.

Competence

Finally, we also found no statistically significant difference in the mention of sensitive topics pertaining to competence between the two interview groups at the p < .05 level. Again, this was the case for the simply dichotomized counts [F(1,10) = .500, p = 0.496], the mean break-point counts [F(1,10) = 4.865, p = 0.052], and the median break-point counts [F(1,10) = .500, p = 0.496].

Our qualitative results, however, again showed subtle differences in how the two groups spoke about competence as a sensitive issue. In interviews in which the scientific expert was present, interview participants spoke at length about and emphasized their own competence. For example, a local government official told us, “If I ever had my druthers, and I could make this the most amazing research project that people from all over the world would come here to find out about it” (AZ 7—expert present). Another participant, an education and outreach coordinator, told us, “If they tell me, for instance they try to shut me down by saying, 'Well, that’s not peer-reviewed science.' I just say, 'Excuse me, it is peer-reviewed science.' I’m at the end of my career enough that I can just dig in my heels like that” (AZ 9—expert present). Another participant who was a government official stated, “We are doing some pretty marvelous things with our wastewater treatment plant” (AZ 38—expert present).

On the other hand, in the interviews in which the scientific expert was absent, respondents did not emphasize their own competence or knowledge, but rather criticized the incompetence of others. Their statements often expressed exasperation that other stakeholders were uneducated and uninformed on important issues with comments such as, “smaller growers just need to get onboard and educate themselves” (AZ 13—expert absent), and “elected officials really need to be much more aware” (AZ 40—expert absent), and “how can we educate a population of four million people who probably came from a place that is green” (AZ 28—expert absent). In this case, we believe the presence of the scientific expert may have encouraged interview participants to tailor and adjust their responses for the purpose of making themselves appear knowledgeable and informed.

Discussion

Our analysis was designed to examine the effect of having a third-party scientific expert present during stakeholder interviews. Specifically, we sought to examine if and how the presence of a scientific expert affected stakeholders’ discussions of sensitive topics. We found that the presence of the scientific expert did not have a statistically significant effect on the mention of sensitive topics during interviews. Both interview groups readily discussed known sensitive topics among Arizona water stakeholders (issues pertaining to risk, gatekeeping, and competence).

However, our qualitative analysis shows thematic differences in how stakeholders discussed sensitive topics when the scientific expert was present in the interview. Respondents often framed the same sensitive topics differently when an expert was present, for example, offering more positional perspectives on gatekeepers. Our qualitative findings indicate that the interviewees may have felt a need to perform or establish their credibility with the third-party scientific expert present. They did this in two critical ways: by re-emphasizing their knowledge and by diminishing the knowledge of others who might participate in a stakeholder process. It may be important to consider power in this context. Scientific expertise in itself represents a form of knowledge power, and the potential influence of that expertise on the actual allocation decisions being made. In this case, stakeholders may have perceived that their responses, given in the presence of the scientific expert, could impact the way environmental problems are defined, the availability of funds for research and problem-solving, and the position of the federal government (through the USDA Agricultural Research Service affiliation) in the Verde Valley. Our results indicate a need for researchers to attend to the ways in which the presence of a third-party scientific expert could alter respondent’s credibility claims and knowledge assertation, and subtly influence their interview responses (and, possibly, data quality). These findings lend further evidence that power differentials can play an important role in the nature and extent of the third-party effect (Taietz, 1962; Tourangeau & Yan, 2007; Zipp & Toth, 2002), and applies it to the context of stakeholder interviews.

Our findings must be interpreted in view of the limitations to our research. First, although our experimental design paired stakeholders based on their occupation, gender, and location—in ways that made it more likely they share similar views on sensitive resource management topics—we acknowledge that the intersections of the topic being discussed, the expertise of the third-party expert, and the background or role of the stakeholder likely all play a role in how and to what extent sensitive issues are discussed during interviews. Further research is necessary to investigate the intersections of these specific variables and the effects that they may have on stakeholders’ discussion of sensitive topics in the presence of a scientific expert. Second, while our sample size (12 matched pairs) exceeds the recommended minimum size to detect themes in qualitative data (Guest et al., 2006), it is nonetheless still very small for statistical comparisons. A larger sample of matched pairs may reveal more discernable and predictable patterns in the ways that sensitive topics are discussed in interviews in which the scientific expert is present versus interviews in which the scientific expert is absent. And finally, while our word-based coding technique was rigorous and systematic, we recognize that it may not have captured all the relevant nuances around sensitive topics for this sample. A different coding strategy could have captured subtle dynamics or sensitive topics that were only mentioned obliquely in stakeholder interviews. Nonetheless, given the deep regional and ethnographic expertise of our research team (which enables us to confidently predict known areas of sensitivity) and the results of our qualitative findings, we cautiously conclude that there is no clear evidence of a strong, predictable third-party effect for scientific experts in stakeholder interviews, although there are indications of qualitative differences that researchers should consider when planning stakeholder interviews and engagement processes.

Conclusion

Our research found that there was no significant third-party effect in the sensitive topics stakeholders raise in interviews with a scientific expert present, though there were subtle qualitative differences in how stakeholders conveyed their credibility and knowledge. This finding is important for stakeholder research because it indicates that researchers may be able to pursue collaborative, interdisciplinary research designs with multiple researchers present without concerns of strongly biasing responses in stakeholder interviews. However, researchers should be aware that the third-party presence of a scientific expert may subtly influence how stakeholders discuss sensitive topics, and should closely monitor interview dynamics. These findings make a valuable contribution to the broader methodological literature on the third-party-present response effect, adding further evidence to what we know about the nature and extent of this response effect. Future research will determine if our findings are relevant to a wider array of stakeholder studies, and if larger-n research will reveal the presence of small but statistically significant third-party effects in stakeholder interviews.

Footnotes

Acknowledgements

We would like to acknowledge the invaluable contributions of our research participants and to thank them for sharing their time and insights.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Agriculture and Food Research Initiative (AFRI) Water for Agriculture grant no. 2017-68007-26584/project accession no. 1013079 from the USDA National Institute of Food and Agriculture, and the National Science Foundation under grant no. NSF-SBE 2017491.

ORCID iD

Melissa Beresford

J. Leah Jones

Kathryn J. Brasier

Note

References

Aquilino

W. S.

(1993). Effects of spouse presence during the interview on survey responses concerning marriage. Public Opinion Quarterly, 57(3), 358–376.

Barnett

(1998). Sensitive questions and response effects: An evaluation. Journal of Managerial Psychology, 13(1/2), 63–76.

Bernard

H. R.

(2011). Interviewing I: Unstructured and semistructured. In Research methods in anthropology: Qualitative and quantitative approaches (5th ed., pp. 156–186). Rowman & Littlefield.

Bernard

H. R.

Wutich

Ryan

G. W.

(2016). Analyzing qualitative data: Systematic approaches. Sage.

Boeije

(2002). A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Quality and Quantity, 36(4), 391–409.

Boeije

H. R.

(2004). And then there were three: Self-presentational styles and the presence of the partner as a third person in the interview. Field Methods, 16(1), 3–22.

Casterline

J. B.

Chidambaram

V. C.

(1984). The presence of others during the interview and the reporting of contraceptive knowledge and use. Survey analysis for the guidance of family programs.

Cronk

Gerkey

Irons

(2009). Interviews as experiments: using audience effects to examine social relationships. Field Methods, 21(4), 331–346.

Diop

K. T.

Traugott

(2015). Third-party presence effect with propensity score matching. Journal of Survey Statistics and Methodology, 3(2), 193–215.

10.

Gfroerer

(1985). Influence of Privacy on Self-Reported Drug Use by Youths. In Rouse

Kozel

Richards

(Eds.), Self-report methods of estimating drug use: Meeting current challenges to validity. National Institute on Drug Abuse Research Monograph (Vol. 57, pp. 22–30). https://archives.drugabuse.gov/sites/default/files/monograph57.pdf

11.

Grimble

Wellard

(1997). Stakeholder methodologies in natural resource management. Agricultural Systems, 55(2), 173–193. https://doi.org/10.1016/S0308-521X(97)00006-1

12.

Guest

Bunce

Johnson

(2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82.

13.

Herrera

A. V.

Benjet

Méndez

Casanova

Medina-Mora

M. E.

(2017). How mental health interviews conducted alone, in the presence of an adult, a child or both affects adolescents’ reporting of psychological symptoms and risky behaviors. Journal of Youth and Adolescence, 46(2), 417–428.

14.

Lau

C. Q.

Baker

Fiore

Greene

Lieskovsky

Matu

Peytcheva

(2017). Bystanders, noise, and distractions in face-to-face surveys in Africa and Latin America. International Journal of Social Research Methodology, 20(5), 469–483.

15.

Lee

R. M.

Renzetti

C. M.

(1990). The problems of researching sensitive topics. American Behavioral Scientist, 33(5), 510–528.

16.

Lukasiewicz

Baldwin

(2017). Voice, power, and history: Ensuring social justice for all stakeholders in water decision-making. Local Environment, 22(9), 1042–1060.

17.

Milewski

Otto

(2017). The presence of a third person in face-to-face interviews with immigrant descendants: Patterns, determinants, and effects. In Bolzman

Bernardi

LeGoff

J. M

(Eds.), Situating children of migrants across borders and origins (pp. 77–96). Springer.

18.

Mneimneh

Z. M.

Tourangeau

Pennell

B. E.

Heeringa

S. G.

Elliott

M. R.

(2015). Cultural variations in the effect of interview privacy and the need for social conformity on reporting sensitive information. Journal of Official Statistics, 31(4), 673–697.

19.

Mneimneh

Z. N.

Elliott

M. R.

Tourangeau

Heeringa

S. G.

(2018). Cultural and interviewer effects on interview privacy: Individualism and national wealth. Cross-Cultural Research, 52(5), 496–523.

20.

Müller

(2019). Capturing multiple perspectives in a multi-actor survey: The impact of parental presence during child interviews on reporting discrepancies. Survey Research Methods, 13(2), 137–151.

21.

Pek

Wong

(2018). How to address non-normality: A taxonomy of approaches, reviewed and illustrated. Frontiers in Psychology, 9, 2104.

22.

Pollner

Adams

R. E.

(1994). The interpersonal context of mental health interviews. Journal of Health and Social Behavior, 35(3), 283–290.

23.

Pollner

Adams

R. E.

(1997). The effect of spouse presence on appraisals of emotional support and household strain. Public Opinion Quarterly, 61(4), 615–626.

24.

Quetulio-Navarra

van der Vaart

Niehof

(2015). Can third-party help improve data quality in research interviews? A natural experiment in a hard-to-study population. Field Methods, 27(4), 426–440.

25.

Reuband

K. H.

(1992). On third persons in the interview situation and their impact on responses. International Journal of Public Opinion Research, 4(3), 269–274.

26.

Smith

T. W.

(1997). The impact of the presence of others on a respondent’s answers to questions. International Journal of Public Opinion Research, 9(1), 33–47.

27.

Taietz

(1962). Conflicting group norms and the “third” person in the interview. American Journal of Sociology, 68(1), 97–104.

28.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859.

29.

Wutich

Lant

White

D. D.

Larson

K. L.

Gartin

(2010). Comparing focus group and individual responses on sensitive topics: a study of water decision makers in a desert city. Field Methods, 22(1), 88–110.

30.

Zipp

J. F.

Toth

(2002). She said, he said, they said: The impact of spousal presence in survey research. Public Opinion Quarterly, 66(2), 177–208.

Third-Party Effects in Stakeholder Interviews

Abstract

Keywords

Introduction

What Is Known About Third-Party Interview Effects?

Study Background

Research Methods

Study Design & Rationale

Data Collection

Data Analysis

Results

Risk

Gatekeeping

Competence

Discussion

Conclusion

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

ORCID iD

Note

References