Gatekeeping's Influence on Equitable Evaluation Practice

Abstract

The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information, or resources. Gatekeepers limit both the participation of key community members and, therefore, the exchange of relevant information related to the evaluation—a process called gatekeeping. Little research attention has been placed on studying gatekeeping, resulting in a dearth of knowledge about the influence of gatekeeping on stakeholder-engaged evaluations and social-structural dynamics that potentially perpetuate gatekeeping practices. In this article, we propose a gatekeeping influence theory grounded in the findings from 14 interviews. With a constructed theory of gatekeeping, we document the emergent social-structural and relational dynamics involved in stakeholder-engaged evaluation, with a focus on evaluations that include community partners and members.

The American Evaluation Association (AEA) Guiding Principles for Evaluators (AEA Guiding Principles, 2018) and the evaluation field's standards for conducting program evaluation (Yarbrough, Shulha, & Caruthers, 2004), along with several evaluation approaches such as empowerment evaluation (Fetterman & Wandersman, 2007) and transformative evaluation (Mertens, 2017), urge evaluators to include stakeholders in evaluation practices from inception through dissemination (Fetterman et al., 2014). These evaluative principles, standards, and approaches prioritize stakeholder involvement to ensure that the evaluation process is informed by community worldviews and local knowledge (Shoemaker & Riccio, 2016). The predominant discourse on stakeholder involvement in evaluation stems from the discovery of potentially deleterious effects of evaluators limiting or excluding important perspectives that inform evaluation projects (Khanlou & Peter, 2005). Our experiences with evaluators suggest that gatekeepers and gatekeeping practices influence the extent to which evaluators can involve stakeholders; thereby, potentially compromising the integrity of information collected during an evaluation and biasing judgments made about a program's desired impacts (Goodman & Sanders Thompson, 2017).

Despite advances in theory and methods for stakeholder involvement in the evaluation process, little research attention has been placed on the agents who moderate stakeholder participation, resulting in a dearth of knowledge about the role that gatekeeping plays in stakeholder involvement approaches such as those found in participatory evaluation or empowerment evaluation (Cousins & Chouinard, 2012). Understanding the role of gatekeeping in evaluation can expand our understanding of stakeholder involvement dynamics and achieving equity in evaluation practice.

Background

Defining Gatekeepers and Gatekeeping

At the end of World War II, where much of Europe and Asia had been reduced to ruins, American society became much more affluent. Public policies like the GI Bill of Rights provided money for veterans to attend college, to purchase homes, and to buy farms, and influenced the proliferation of forming families and having children in unprecedented numbers. But not all Americans had the chance to participate equally in these opportunities and in the growing economic prosperity. It was within this growing disparity that the term gatekeeping emerged. In 1944, social psychologist Kurt Lewin studied how families select food items to explore the decision-making process (Shoemaker & Riccio, 2016). Lewin compared the act of deciding which food to select to gates, where the binary of open and closing a gate referred to the selection or rejection of food items (Brown, 1979). In this sense, the grocery shopper was a gatekeeper of the food the family eventually consumed, selecting which food items were “in” and which were “out,” thereby deciding the nutrition of the basic social unit.

Stakeholder-engaged research in the past two decades has given gatekeeping an array of meanings. In research on evaluation scholarship that reports on stakeholder involvement, researchers have focused on gatekeeping as a process by which individuals decide who has access to socially excluded people in research, facilitating, or inhibiting researchers’ and evaluators’ access to the research phenomenon under study and to the evaluators in that setting (Emmel et al., 2007; Kawulich, 2011; Wanat, 2008). Researchers who focus on gatekeepers as intermediaries generally view access, or the lack of, as a function of power differentials, trust, and social capital (Doll et al., 2012; Emmel et al., 2007; Kawulich, 2011). The term gatekeeper has been used to describe community advisory boards (Kaiser et al., 2017), key informants and stakeholders (McKenna & Main, 2013), community health advisors (Story et al., 2010), interpreters (Edwards, 2013), sentries (McGregor et al., 2017), and insiders (Schatz et al., 2015). Considering these meanings and terms, we began our research defining gatekeeping as the process by which decisions are made about who gains access to people, information, materials, and/or goods. Gatekeepers, then, are those who decide who or what is “in” and who or what is “out.”

Research on gatekeeping in evaluation is scarce, and topics that come close to examining gatekeeping practices in evaluation tend to examine organizational power (Kim & Cervero, 2007), power and role sharing (Cartland et al., 2008), and stakeholder engagement (Gilliam et al., 2002). From these topics alone, gatekeeping seems to be experienced as and is created by power differentials in the process of stakeholder engagement and inclusion during an evaluation project. Widening the scope to literature on gatekeeping in research, we find that researchers focus on similar aspects of gatekeeping such as access to minority communities (Lund et al., 2016; McAreavey & Das, 2013), university-community partnerships (Suarez-Balcazar et al., 2006), and trust (Emmel et al., 2007). There is also an emphasis on gatekeeping dynamics and relationships in research on access to palliative care (Kars et al., 2016). Thus, the combination of common elements of gatekeeping, such as lack of access to stakeholders and power imbalances, is a commonly referred to but seldom researched relational phenomenon.

The purpose of this study was to generate a theory, grounded in data, that explains (1) why and how evaluators experience gatekeeping; (2) how gatekeeping impacts evaluators, and their respective projects; and (3) the various processes and strategies evaluators deploy to resolve their main concerns regarding the impact gatekeeping has on evaluation practice.

Process and Procedures

Design

Grounded theory methodology, as originally developed by Glaser and Strauss (Glaser & Strauss, 2017) and separately refined by Glaser (Glaser, 2002) and Charmaz (Charmaz, 2014; Charmaz & Belgrave, 2015), informed the plan of research for this study. The grounded theory process consists of five basic components: theoretical sensitivity, theoretical sampling, coding, theoretical memoing, and sorting. These five components were integrated by the constant comparison method of data analysis. The goal of the research was to understand evaluators’ experiences of gatekeeping in evaluation. Once the main themes emerged from the data, researchers developed a theory, grounded in data, about how and why evaluators experience gatekeeping.

Recruitment and Sample

Two rounds of recruitment were used. The first round utilized authors’ professional network of evaluators in Wisconsin. These evaluators were recruited through email and phone and utilized for both pilot and official interviews. The second round of recruitment began after the authors’ professional networks were exhausted and consisted of emailing both Topical Interest Groups (i.e., research on evaluation, advocacy and policy change, collaborative, participatory, and empowerment evaluation, health evaluation, theories of evaluation, and use and influence of evaluation) and regional offices of the AEA.

An important tenet of grounded theory is the idea that researchers should not assume the relevance of identity data, including race, age, gender, sexual orientation, class, and ability. Thus, the relevance of these variables was not assumed. Specifically, 14 evaluators were interviewed with 30% identifying as male and 70% identifying as female. All evaluators identified as evaluators who conduct evaluations with varying levels of stakeholder involvement. Additionally, we left the definition of stakeholder open for evaluators to identify themselves. Evaluators used “stakeholder” to describe an array of individuals in their evaluation projects, with most describing various mid-level managers (i.e., those who decide which programs, practices, and tools to adopt; deliberating ways to improve existing services; shaping the conditions for implementation; and making resource allocation decisions), and community members or those who participate in programs or practices being evaluated.

Data Collection and Analysis

Theoretical sampling, the process of data collection that allows for the generation of theory, was the primary sample method used in this study. When using theoretical sampling, the researcher simultaneously collects, codes, and analyzes data and uses this ongoing process to determine what data to collect next and where to find them. In line with theoretical sampling, evaluator selection was informed by the analysis and coding of the interviews. Semistructured, adjusted conversational interviewing was utilized. Interview times ranged from 30 minutes to approximately three hours, with an average of approximately one hour and 10 minutes. Adjusted conversational interviewing was utilized because it is regarded as the most effective grounded theory approach to interviewing (Rensen et al., 2017). This occurred in three phases.

In the first phase, we began piloting our interview questions with local evaluators who categorized their approach to evaluation as stakeholder-engaged, a term used to capture the spectrum of community member involvement in evaluation projects ranging from little involvement (e.g., input on instrument development) to major involvement (e.g., co-leading the evaluation). In Phase 2, we used an initial interview protocol that consisted of broad questions about each evaluator's experience in research and evaluation projects with a focus on dynamics of engagement, participation, and stakeholder involvement. Through line-by-line coding, capturing emerging concepts, and constant comparison, categories emerged and informed a second phase of theoretical sampling as well as a refined interview protocol. In Phase 3, we continued theoretical sampling but, this time, with a focus on (1) identified themes that emerged in Phase 2 such as issues of power, temporality, and trust; and (2) evaluators who did and did not categorize their evaluation projects as stakeholder-engaged, seeking to refute initial findings. As additional interviews occurred, categories were reconceptualized and the properties that inform each category were identified. Selective coding was used after interview 12, when core concepts emerged, and the data were saturated across categories and across their properties. Two additional interviews were conducted for verification purposes. Phase 3 ended when we reached category saturation.

Data sources for this study included 14 evaluator interviews, notes taken when interviewing evaluators and gathering feedback from students and experts, and field notes from graduate students who conducted evaluator interviews and assisted with a literature review. The lead author, along with two graduate students and one undergraduate scholar, collected, transcribed, and coded all data in Dedoose Version 8.0.35. To ensure intercoder reliability, the lead author created a training session or “test” based on the coding of a single interview. This works by selecting the codes to be included in the “test,” the selection of previously coded/rated excerpts to comprise the test, and then specifying a name and description for the test. The lead author chose to include all codes for research assistants to use for coding a single interview. Research assistants were prompted to access the test and to apply all codes to the set of excerpts making up the test. During the test, research assistants were blind to the work that was done by the lead author or other research assistants. Upon completion, Dedoose reported a Cohen's Kappa of .61 otherwise known as substantial inter-rater agreement. As we proceeded, each interview was coded twice by research assistants and reviewed by the first author. To aid in the process of generating categories, the team utilized Microsoft Excel to track quotes across categories and Miro (2020), an online collaborative white board, to map categories to concepts and properties. The last author, an experienced evaluator and researcher, oversaw the project and provided guidance and advice throughout the study. She also assisted with conceptualizing the broader themes that were emerging from data analysis.

Ethical Considerations

The University of Wisconsin-Madison Institutional Review Board reviewed and approved the research protocol on June 21, 2018 prior to the scheduling of evaluator interviews. Evaluators were given the opportunity to review field notes and the final substantive theory prior to the conclusion of the study. They were also assured that field notes would not include identifying information.

Findings

Overview of Gatekeeping Influence Theory

We termed the theory emerging from this process as Gatekeeping Influence Theory (GIT). GIT offers a working definition of gatekeeping and a conceptual identity for gatekeepers. As seen in Table 1, GIT describes drivers that influence the conduct of evaluation; identifies the strategies that evaluators use to navigate the social-structural conditions of evaluation projects; and the gatekeeping disruptions that problematize equitable evaluation practice. Within each theme are categories and properties of categories that emerged across interviews such as power differential, navigating contextual dimensions such as politics and culture, temporality (e.g., when the evaluator enters the evaluation process; whether the evaluator is given enough time to foster equitable relationships with stakeholders), and positionality, or where the evaluator is positioned within the evaluation project, among other organizations and stakeholders.

Table 1.

Summary of Themes and Categories in the Gatekeeping Influence Theory.

Themes	Categories
Equitable Evaluation Practice: evaluators describe their aim for evaluation is an equitable process of involvement, valuation, and discovery	Experiences in Evaluation: refers to the practical contact with and observation of events in evaluation that support evaluators’ desire to conduct equitable evaluation practices
	Expanding the Scope of Evaluation: refers to evaluators’ desire to include an array of stakeholders as well as broaden the scope of the field
Gatekeeping Disruptions: small, often contentious moments in evaluation projects that, when left unaddressed, flourish into cultures of distrust, and power imbalance	Access to People, Knowledge, and Data: refers to a lack of access to important relationships and resources that limit equitable evaluation practice
	Political Dynamics of Stakeholder Involvement: refers to evaluators’ preoccupation with group dynamics such as decision making and power relations
	Social-Structural Context: refers to the patterned social arrangements (e.g., socio-economic status, institutionalized racism) in society that are both emergent from and determinant of the actions of evaluators
	Disruptive Effects: refers to the various interruptions evaluators experience to achieving equitable evaluation practice
Evaluator Ability to Engage Stakeholders: evaluators’ attempt to include an array of stakeholders to circumnavigate gatekeeping disruptions	Stakeholders’ and Evaluators’ Attitudes and Beliefs: evaluators’ attitudes and beliefs of and about themselves, stakeholders, and evaluation are predisposing conditions that drive which strategies evaluators use to navigate evaluations
	Navigating Group Membership: refers to evaluators’ ability to navigate their and others’ out-group and in-group identification in evaluation projects
	Normative Beliefs and Subjective Norms: evaluators’ beliefs about others’ expectations
Strategies for Equitable Evaluation Practice: evaluator approaches to achieving stakeholder empowerment	Developing an Equitable Approach: refers to the intentional deployment of specific evaluation approaches or methods to foster equity
	Flexibility and Redefining the Evaluator Role: refers to evaluators’ shifting their role in evaluation projects
	Relationship Building: refers to evaluators fostering trusting, respectful relationships with themselves and stakeholders
	Creating and Engaging Intermediaries: refers to evaluators attempt to create groups (e.g., community advisory board) to protect a specific population of people
Enabling Conditions: the situations that must occur to allow for equity in evaluation to take place	Levels and Histories of Stakeholder Involvement: refers to the importance of understanding of and engaging in potential histories of collaboration with stakeholders and community members
	Leadership and Shared Stewardship: refers to the dialogue, co-learning and co-creation, and shared governance required for enabling more equitable conditions in evaluation
	Power (Im)Balance: refers to existing sociopolitical relationships among evaluators and stakeholders that limit equitable decision-making and ability to achieve program goals
	Self-Collective Awareness: refers to the required interconnected awareness of both self and others during the evaluation process
	Capacity: refers to resources (e.g., time, funding, personnel) and community readiness that support engagement in equitable evaluation

Equitable Evaluation Practice

Evaluators described fears, hopes, and dreams when conceptualizing the purpose of their evaluation projects. Evaluators tended to describe their aim for the evaluation as a process of involvement, valuation, and discovery with one evaluator describing evaluation as “a process that I aim to be inclusive and also a way to discover and value more about what community partners do.” When defining equitable evaluation practice, evaluators generally refrained from using terms that might fall under capitalistic framings; words such as product, profit, and incorporate were not used. Instead, they used phrases such as “co-create” and “jointly-owned.” They embraced references to evaluation as a means of valuing, distributing, and owned or regulated by all stakeholders in the evaluation. The theme of equitable evaluation practice emerged out of several categories as a goal of evaluation practice. These categories include experiences in evaluation and expanding the scope of evaluation.

Experiences in Evaluation

Experiences in evaluation refer to the practical contact with and observation of events in evaluation that support evaluators’ desire to conduct equitable evaluation practices. The experiences in evaluation category emerged from the following properties: maximizing the benefits and reducing the risks of evaluation (mitigating bias and potential power imbalances); experiencing conflict and setbacks; and unintended consequences (reinforcing existing inequities). As evaluators described their experiences in evaluation projects, they recounted the need to maximize the benefits and reduce the risks of their projects by engaging stakeholders in an “empowering and emancipatory” process. This process was put into place in their evaluations to mitigate bias and potential power imbalances, as indicated by one evaluator: “We use engagement and participation to empower people partly because, as a methodologist, we want to mitigate bias and, and an evaluator, deal with aspects of power.” Additionally, evaluators discussed their experiences of stakeholder conflict and unintended consequences such as reinforcing existing inequities. As evaluators described their experiences in evaluation, they cycled through a feedback loop of noticing risks, wanting to reduce risks and maximize the benefits of evaluation, and then wanting to increase the equity of the evaluation process.

Expanding the Scope and Purpose of Evaluation

This category includes the properties: stakeholder engagement, definitions of evaluation, and politics of evaluation. When evaluators described their evaluation projects, they often identified a desire to expand the scope of which evaluation projects typically include as well as expand the scope of the definition of evaluation to normalize evaluators who act as advocates, champions, technical assistants, and proponents of specific members of the community and political causes. These evaluators desired to expand the scope of evaluation's audience as well as its normative definition, as one evaluator described, “The role of the evaluator, whatever that ends up being by the time the evaluation project unfolds, generally shifts and, generally, it's hopeful that means we still get to be as inclusive as we’d like.” Evaluators described a desire to reframe evaluation away from a rigid process by which valuations are made about programs and toward thinking about evaluation as nested in efforts for social, economic, and health equity. One evaluator said, “…it's mostly about making value assessments and judging programs, so what I do is more like a common space to create change.” This suggests not only a desire for expanding the definition, but a specific political orientation to evaluation, too.

Gatekeeping Disruptions

As evaluators revealed their interactions with other members of their evaluation projects, they also revealed the frequent disruptions in the evaluation process: moments of interpersonal conflict resulting from organizational hierarchies; extended periods of being unable to access information needed to answer evaluation questions; running into stakeholders or other evaluators who control access to other stakeholders and/or information; or, more rarely, the protection of community groups by establishing community advisory boards. These disruptions represent small, often contentious moments in evaluation projects that, when left unaddressed, flourish into cultures of distrust and power imbalance. For instance, discussing a hostile and conflict heavy work environment one evaluator narrated how organizational culture and personal grievances can come together to create a disruptive environment:

I was running into issues with a colleague who was setting themselves up as the gatekeeper in this particular example. Which led to a lot of control of the information that was received and then just controlled a lot of the process of the evaluation…I think the gatekeeping aspect has been really tough in that they don’t understand how that has impacted evaluation and moving it forward and negatively impacted an inclusive climate, an inclusive process, and collaborative process…So, my organization is tough in general, it is very political and—it's all about politics and it's about maneuvering and it's about power.

Gatekeeping disruptions are the fractious interactions that result in problematic relationships and work cultures when evaluators attempt more equitable stakeholder involvement. Gatekeeping disruptions thwart evaluators from practicing equity in their projects, indicating that there exist relational dynamics that need to be addressed. The gatekeeping disruptions theme emerged from the following categories: barriers to access to people, knowledge, and data; politics of stakeholder involvement; social-structural contexts; perceptions of self and others; and disruptive effects.

Access to People, Knowledge, and Data

Evaluators reflected on instances in which they could not access specific stakeholders, knowledge, and/or data vital to informing the evaluation. Evaluators described these instances as barriers to following through on evaluation activities and potential disruptions to equitable evaluation practice. Access to people, knowledge, and/or data emerged as a key category consisting of the following properties: lack of access; barriers to access; and systemic barriers to conducting equitable evaluation practice. Barriers (e.g., time, funding, individuals who intentionally limit access to people or materials) are described as instances where an evaluator is unable to accomplish their tasks (or responsibilities or activities), especially due to gatekeeping. Barriers were the roadblocks that prevented access to people, knowledge, and data.

Lack of access consisted of evaluators experiencing an inability to access stakeholders (e.g., decision-makers, community members); knowledge (e.g., existing evaluation reports from community-based organizations); and/or data (e.g., medical records). Evaluators described their lack of access as a byproduct of barriers that occur iteratively throughout a project, “sometimes barriers just pop up, you have to deal with it.” These barriers to access generally related to a lack of resources such as time or funding, a lack of stakeholder buy-in, or poor evaluation design that did not include stakeholder involvement principles. Finally, evaluators described systemic barriers to conducting equitable evaluation practice. For example, evaluators described confronting gender norms and racism in stakeholder meetings that either assumed the normative role of the evaluator or prevented the evaluator from building the necessary relationships needed to facilitate more equitable forms of stakeholder involvement. In an example of gender bias, an evaluator admitted that simply by virtue of her gender she likely “changed how people reacted to the project.”

Political Dynamics of Stakeholder Involvement

Each evaluator described experiencing contentiousness during collaboration and participation. Evaluators were preoccupied with the politics of decision making in groups and other forms of power relations between individuals, such as the distribution of resources, regardless of the stakeholder involvement approach chosen. Political dynamics of stakeholder involvement is a category that emerged out of evaluators’ identification of the relational dynamics involved in stakeholder involvement practices. These dynamics include the following properties: conflict, consensus, and power sharing.

In the conflict property, evaluators encountered instances of disagreement often in the form of protracted and unspoken misunderstandings. This type of conflict did not always result in ruined relationships. Instead, sometimes conflict created important moments of reflection on how collaboration is or is not proceeding effectively (i.e., creating opportunities for all members of the evaluation to voice their ideas or concerns). Evaluators often qualified their explanation of conflict by suggesting that conflict is an inherent part of evaluation, or when (1) the evaluation is used to achieve a form of social justice or (2) when the evaluator evaluates an intervention designed to change unwanted or unjust social activities. Finally, the property power sharing emerged out of discussions about why stakeholder involvement was important and how involvement could function to redistribute power among stakeholders in an evaluation. This property emerged as evaluators described the need to and difficulty of addressing social hierarchies within evaluations consisting of an array of stakeholders.

Social-Structural Context

Social-structural context, defined as the patterned social arrangements (e.g., socio-economic status, geography, social environment) in evaluations that are both emergent from and a determinant of the action of evaluators, emerged as a category with the following properties: institutionalized politics and racism; histories of collaboration. Evaluators described running into issues of institutionalized politics and racism, or the practices of social and political institutions that overtly or covertly discriminate based on political viewpoint or race. One evaluator described dealing with institutional conflict as follows:

the case is sometimes—especially if you address issues around race and structural racism—it's not an easy to convey, sometimes, to stakeholders, especially when these stakeholders are sitting in organizations—when they're public organizations or even non-profits that they want to do the right thing, but they're maybe not seeing that they're racist. And I think those are tough pieces.

Evaluators described that preconceived notions were held by many stakeholders and were rooted in histories of collaborating with researchers and evaluators from the university. Thus, the role of the evaluator was often called into question: “My role changes based on the contract, which includes stakeholders who sometimes expect you do be a social justice advocate and sometimes they don’t and they just want you to uphold the status quo.”

Disruptive Effects

As evaluators described their confrontation with both gatekeepers and cultures of gatekeeping, they also detailed the disruptive effects these had on the evaluator, the stakeholders, and the evaluation. The category of disruptive effects emerged from the following properties: emotionality, reinforcement of social-structural inequities, influencing evaluation deliverables; and evaluation complexity. In the face of relational tension, evaluators often described feeling annoyed, confused, and frustrated, as indicated by increased emotionality during interviews, particularly that of anger. As one evaluator said, people in evaluation “expect you to get it all done. And this is all the money and time you get. It's bullshit. I mean, you can’t do it. There's no integrity to it.” This emotionality was mainly the result of either failed attempts at resolving perceived issues or not knowing how to resolve them. Evaluators also noticed that relational tension reinforced existing inequities and power differentials within the evaluation, among stakeholders. As a few evalautors noted, within the context of the intervention or program the evaluation was based on: “So really, just paying attention to the culture of that community. But people with bad history can mess that up.” Evaluators noted that these tensions influenced evaluation deliverables by limiting the type and number of stakeholders available to the evaluation and therefore the credibility of the results. Finally, gatekeeping disruptions created issues of complexity. The complexity described was a confluence of (1) a multiplicity of levels of disruptions involved, occurring between individuals, groups of individuals, and organizations; (2) a diversity of actors with different backgrounds and cultures; and (3) interactions between evaluator and stakeholders that went unnoticed or unaddressed. This complexity led to evaluators developing strategies to achieve more equitable stakeholder involvement in their projects, described below.

Evaluator Ability to Engage Stakeholders

GIT posits that evaluators who primarily work to achieve equity in their evaluations are thwarted by specific disruptions to the evaluation process, limiting their ability to engage stakeholders and to create the enabling conditions (e.g., trust and respect) that lead to more equitable relationships. The evaluator's ability to engage stakeholders emerged as a discrete theme foundational to achieving more equitable evaluation practices but realized only through complex interaction between the categories: stakeholders’ and evaluators’ attitudes and beliefs; navigating group membership, or the social strategies evaluators perform to circumnavigate gatekeeping disruptions. Achieving more equtiable evaluation practices were also associated with an evaluator's belief about (1) how they should act based on others' perceptions of their behavior (i.e., normative beliefs) and (2) the total accumulation of normative beliefs that create a perceived social pressure to engage in a specific behavior (i.e., subjective norms).

Stakeholders’ and Evaluators’ Attitudes and Beliefs

GIT posits that evaluators’ attitudes and beliefs of and about themselves, others, and evaluation are predisposing conditions that drive the types of strategies evaluators use to navigate the conditions of evaluation; the evaluator's ability to engage stakeholders; and the conditions that enable equitable evaluation practice. This category is defined as an evaluator's acceptance and point of view about themselves, other evaluators and stakeholders, and the field of evaluation that facilitates or limits their ability to engage in equitable relationships. The essence of this category is captured by one evaluator's belief about gatekeeping:

So I think gatekeeping is incredibly more complicated than when I started, and I think that there are more people talking about it and I think everybody recognizes it. I think where the rubber hits the road, everybody kinds of backs away from that conversation and says, ‘That's somebody else's problem to fix. That's not me.’ So evaluators in philanthropy are coming together talking about this very issue, but nobody wants to change the way they do the work to make it better.

Navigating Group Membership

Evaluators often run into instances where they are simultaneously an outsider to (out-group) and a part of (in-group) stakeholder groups. Originally defined by Henri Tajfel (2010), an out-group consists of an individual that does not identify as a member of a social group whereas an in-group is a social group in which a person identifies as being a member. Navigating group membership emerged from four properties: culture, social ties, evaluator identity, and positionality. The culture property manifested as differences between social behavior and norms, diverging across discrete geographies, institutions, socio-economic groups, histories, and ways of being (e.g., white, academic evaluator evaluating an indigenous community organizing group). Differences among culture often coincided with challenges of group membership, which contributed to the circumstances in which access to stakeholders or information was controlled. The social tie property emerged as evaluators described different types of relationships with stakeholders. These interpersonal ties—or interpersonal relationships among evaluators and other stakeholders that vary in their level of trust, influence, and transparency—contributed to how evaluators perceived their group membership, with low levels of each attribute fostering perceptions of increased discrimination, favoritism, and group polarization.

Evaluators described that both their role as an evaluator and their personal identity (e.g., race, sexual orientation) mediated their social ties and, in turn, their perceptions of group membership. The category of group membership emerged as an aspect of why, for what purpose, and which some evaluators or stakeholders may control access to people or information. For example, some evaluators described that their out-group status, which varied from project to project, sometimes made it more difficult or altogether prevented their ability to gain access to people and information needed to inform the evaluation, as community or organizational mid-level managers often denied their requests for access. In mitigating these issues some evaluators discussed their connection to in-group members who helped address these issues, “I have a 900-plus friends list of Native Americans when I need to reach out to them. And so, what it's done is that—I think one of the things it probably created, it really fostered a strong relationship between tribal colleges, myself, tribal communities.”

Closely related to evaluator identity emerged the positionality property, or the way in which stakeholders’ and evaluators’ attitudes and beliefs were shaped, biased, and influenced by the roles or identities they assume when navigating relationships and evaluation processes. Though not always related to attitudes and believes, this property emerged from evaluators’ differing attitudes and beliefs about evaluation based on the relationship between the evaluator and the sociopolitical context of the evaluation that includes community members, community organizations, decision-makers, and intermediaries.

Normative Beliefs and Subjective Norms

Normative beliefs are “individuals’ beliefs about the extent to which other people who are important to them think they should or should not perform particular behaviors” (Trafimow & Fishbein, 1995, p.?). And subjective norms are the societal accumulation of normative beliefs that then have a reciprocal effect on intention and behavior. This category emerged from the following properties: idealizing inclusivity, moral imperative (social and institutional pressure), feasibility, and context. Evaluators tended to believe that stakeholders should be included, knowing that the AEA prioritizes the equitable inclusion of stakeholders in evaluation practice. This property revealed that evaluators perceive inclusion as a moral imperative of the field, noting that peer and institutional pressure influenced their perception of the type of inclusion that was valued. Evaluators acknowledged that the idealization of this perceived moral imperative often led to confusion about how, when, and where to access stakeholders for participation in evaluation, which stakeholders to include, and whether their evaluation projects were sufficiently empowering. Evaluators described their perception of stakeholder involvement as a conflicting duality of desiring to ensure equitable inclusion and being unable to do so due to limited resources (e.g., time, money, capacity). For example, one evaluator lamented, “We couldn’t really access or do that. Part of it was cultural so it would have taken a significant amount of time and resources.”

Strategies for Equitable Evaluation Practice

GIT posits that evaluators, when their aims are to engage in equitable evaluation practices with stakeholders, develop strategies to overcome gatekeeping. GIT proposes that these strategies are heavily influenced by stakeholder and evaluator beliefs and attitudes regarding evaluation and evaluators. This theme consists of the following categories: developing and defining an equitable approach; flexibility and redefining the evaluator role; relationship building; and creating and engaging intermediaries (e.g., mid-level managers).

Developing and Defining an Equitable Approach

Evaluators developed and deployed specific evaluation approaches in their projects as one strategy to fostering equity. This category emerged from the following properties: choosing an evaluation approach, defining equity, iterative stakeholder analysis, and stakeholder power sharing. First, evaluators chose evaluation approaches (inclusive of frameworks and principles), such as culturally responsive evaluation and transformative evaluation, to center community voices and foster equitable relationships among stakeholders. Evaluators described that the tenants of these approaches, as well as the principles of community-based participatory evaluation, were strategically deployed with the hope that the approaches themselves would generate equity in evaluation: “We generally use participatory approaches to create the right kinds of relationships…equitable relationships.” However, evaluators described one caveat to this process: the presence and level of equity generated in an evaluation project depended on how the evaluator spent time building relationships. That is, the evaluator's achievement of more equitable evaluation practices was mediated by building trusting and respectful relationships.

Relatedly, another property that emerged was around how equity means something different to each stakeholder and each stakeholder group. One evaluator described the necessity to build consensus, often at multiple time points in the evaluation project, around what equity meant to each stakeholder group and how and for whom equity could be achieved: “In my experience I’ve always had to work with everyone to define what equity means because it changes every time I start a new project with new people and in a new place.” Evaluators noted that conducting stakeholder analysis, especially in cases where evaluation projects were longer than a year, aided in maintaining a current understanding of which stakeholder voices needed to be empowered and who in the evaluation had the most power. Finally, evaluators often described having to relinquish control of the developments and choices outlined above. That is, to infuse empowerment and emancipatory processes into the evaluation project, evaluators described a process of abdicating their role as the “evaluator” to both dispossess themselves of the normative “evaluator” role and provide space for stakeholders to take control of the evaluation process, deciding what equity for their community means and what the goals of the evaluation should be: “Definitions are important and I think more distributed power structures and redefined roles are necessary.”

Flexibility and Redefining the Evaluator Role

Evaluators described having to shift their role in evaluation projects as a strategy of creating more equitable evaluation practices: “A big part of my job is helping stakeholders take control and learn from each other.” Flexibility and redefining the evaluator role consists of the following properties: evaluator, facilitator, observer, capacity builder, and advocate. Each of these properties has considerable overlap as well as the ability for the evaluator to occupy more than one role at once. Evaluators described taking the role of facilitator when the evaluator needed to enhance discussion among stakeholders to advance evaluation activities, overcome barriers, and create a more collaborative environment: “I transitioned to being a facilitator…I facilitated the development of the process and framework, and then worked with stakeholders to administer surveys and create discussion.” Evaluators described becoming more of an observer, including having to listen more than speak, reflect more than make decisions. In all cases, the evaluators described having to advocate on behalf of specific stakeholder groups whether to funders, oversight committees, or key policymakers and decision-makers: “I brought the funders and tribal colleges closer…the National Institute of Health has never had such a project to interact with diverse groups.”

Relationship Building

Relationship building emerged as a category marked by an array of interpersonal and intrapersonal properties. These properties include trust, respect, language used, rapport, and reflexivity. Evaluators described each of these properties as essential to the work of building equitable relationships. Trust was most often cited as the key relational property needed in conducting equitable evaluation practices and consisted of listening and developing reliability:

At the beginning [evaluators] are very advisory. And we were learning about the program. As we understood it better, we could make better recommendations. As they learn to trust us, they were more ready to accept our advice. It was a stronger partnership with trust.

Trust was diminished when the evaluator controlled access to people or information: “Administrators were so controlling it eventually got to a point where they got overly involved and essentially dissolved the partnership because a lack of trust and willingness to work collaboratively around shared interest.” Experiencing open and honest communication was a driver of trust, and therefore a driver of what circumstances generate issues of control over people and information. Trust, respect, and the language used usually led evaluators into discussing building rapport, or simply long-term relationship building based on mutual attentiveness, positivity, and coordination, or the feeling of being “in sync” with one another: “In evaluation relationships don’t have over night. And so, you’re building relationships, new relationships, with others in the community…it's about attentiveness and coordination.” These properties coincided with the emergence of the reflexivity property, or the process of attending systematically to the context of knowledge construction, especially to the effect of the evaluator, at every step of the evaluation process:

I come at it from a feminist evaluative perspective, meaning that I think about how I influence evaluation. One of the ways that I’m cognizant about how I influence evaluation is that I come from a university and I have, at my fingertips, a log of resources that communities don’t.

Creating and Engaging Intermediaries

Creating and engaging intermediaries emerged as a category rooted in the more contemporary strategy of some evaluators creating or encountering community advisory boards or other individuals and groups assembled or chosen to protect a specific population of people. This category emerged out of the properties of protection and ethics. Evaluators described needing alternative protocols that ground the assessment of risk to specific communities in the achievement of equity and other community-based participatory evaluation principles for the purposes of protecting and honoring community partner knowledge and culture:

you can’t just go waltzing into any tribal community and expect folks to just sit down with you. I mean you need to know who are. Sometimes I have talked to the tribal chairman or tribal council to talk about what I was doing. So, it's kind of the people that know the other stakeholders in the community that can assist you in moving forward with the evaluation processes.

In some cases, this involved shifting away from traditional questions such as “describe exactly how the research will be carried out” and toward potential new questions such as “how will the community be involved in the research and at what levels?” In many cases, evaluators helped build a community advisory board consisting of representatives of the general public who meet with representatives of the evaluation project to relay information between the two groups. In other cases, the evaluator became the intermediary responsible for both honoring the ethics of evaluation practices and protecting community members from potentially harmful evaluation practices.

Enabling Conditions

Enabling conditions is defined as the situations that must occur to allow for equity in evaluation to take place. Enabling conditions emerged as the research team reviewed notes, memos, and codes that described contextual and circumstantial elements surrounding evaluators’ ability to engage stakeholders on the path to promoting equity in evaluation. In addition to influencing evaluators’ ability to engage stakeholders and encompassing the drivers of stakeholders’ and evaluators’ beliefs and evaluators’ strategies, this theme emerged from the following categories: levels and histories of stakeholder involvement; leadership and shared stewardship; power (im)balance; self-collective awareness; and capacity. GIT posits that these categories represented the conditions under which evaluators found themselves more easily ensuring equitable evaluation practices such as sharing power and integrating awareness of histories of collaboration into relationship building.

Levels and Histories of Stakeholder Involvement

Evaluators frequently discussed the varying levels of stakeholder involvement that facilitated equitable evaluation practices, as well as the importance of understanding of and engaging in potential histories of collaboration with stakeholders and community members. This category emerged from the following properties: information-sharing and consensus building, and history of collaboration.

Evaluators described that the enabling conditions of equitable evaluations relied on information sharing across stakeholders and across the lifespan of the evaluation. However, evaluators often noted that before information exchange could happen, the various histories of stakeholders needed to be both solicited and addressed. That is, the level of exchange of any resource relied on trust and safety building between the evaluator and stakeholders, especially in cases where stakeholders have experienced a history of harmful collaborations with researchers and/or evaluators. This was certainly the case when evaluators were working with minority populations: “Yeah, especially given, kind of, the historical research in certain communities of color. We’re wanting to be really respectful of that history, and that's why we ultimately don’t push evaluation.”

Leadership and Shared Stewardship

Evaluators discussed leadership and stewardship as a behavioral category giving form to the conditions enabling more equitable evaluation practices. This category consisted of the following properties: dialogue, co-learning and co-creation, and shared governance. The dialogue property emerged out of evaluators reflecting on the importance of transparent communication in the process of determining roles and tasks. Evaluators also discussed dialogue on a spectrum of listening to directing and knowing where to exist on the spectrum in any given dialogue as essential to effective leadership in evaluation: “…listening more than directing. [The evaluation] is way more equal and co-constructed.” Both the co-learning and co-creation and the shared governance properties emerged out of evaluators’ reflection on the process of sharing power and communicating in a way that invites instead of coerces. Evaluators described sharing their ownership over the evaluative process to create a participatory decision-making process as well as put the potential of the evaluation project to be empowering in the hands of those who will operationalize the results of the project. By the same token, one evaluator mentioned that the lack of shared governance,

meant there was no distribution of power. There should have been a structure in place to identify what everyone's roles were and the expectations for each group and the decision-making process in regard to how the evaluation would be conducted, how the groups would work together. No leadership for how to overcome conflict, which was something that was regularly seen within the partnership because of the lack of structure and leadership.

Power (Im)Balance

The first round of interviews of evaluators, which included five semistrutured interviews, did not reveal any properties related to power. In the authors’ practice, however, one of the central issues involved in community-based evaluation is the presence, persistence, and effects of power differentials created by evaluators or preexisting in evaluation projects; one evaluator captured this well,

I think power can corrupt. And I think the negative gatekeeper in this instance, they’ve gained more power over the last several years. And it's somewhat corrupting, but I also think it's their personality. And I think that they probably value less respect for other people. So, I think it comes down to values, too. And I think they had a lot to gain from being a gatekeeper.

Thus, the second round of interviews focused on tensions evaluators encountered and why they encountered them. The category of power (im)balance emerged out these discussions along with the following properties: exposing oppressive contexts, structures, and language, oppressive versus emancipatory, combating hierarchies, and community resilience and strength.

Most evaluators described engaging stakeholders in a process of open communication and deliberative democracy to expose historically and existing co-occurring oppressive contexts (e.g., socially oppressive relationships, policies), structures (e.g., organizational hierarchies), and language (e.g., abstruse or inaccessible academic language). Several evaluators explained that outlining these oppressive contexts in the beginning of an evaluation created space to engage stakeholders on how to combat these elements during or with evaluation: “We really pay attention to power structures and some of the tenants found in culturally responsive evaluation to combat oppressive structures.” Evaluators noted that this process was accompanied with discussions and actions around creating evaluation practices that were emancipatory rather than blindly reinforcing existing or potential oppressive evaluation practices. These actions usually accompanied discussions around how to combat hierarchies that exist within institutions such as the university or government agencies. By challenging inequities inherent in institutions’ hierarchies, evaluators were better able to create conditions in which stakeholders could imagine more empowering structures: “This particular side of the project is actually doing a pretty good job with evening out hierarchy that is typically present in evaluations…I guess we are able to talk about inequity and it allows us to combat stuff.”

Self-Collective Awareness

As evaluators described their experience of conducting evaluation, they often reflected on the need for both self and collective (e.g., partnerships of stakeholders) awareness. This category emerged as evaluators discussed power dynamics and structural inequities in evaluation projects and refers to the following properties: consciousness knowledge, self, and other. These three properties are intimately tied to one another. Evaluators reflected on situations where being more conscious of their character, feelings, motives, and desires as an evaluator and for the evaluation would have aided in creating more equitable evaluation practices. Evaluators also described needing to develop conscious knowledge around the historical, structural, racial, and economic inequities that exist among and within specific populations and contexts to avoid reproducing inequities within the evaluation: “I educated myself. I looked at socio-historical-ecological maps of people in Wisconsin. So I read every single report on the topic from 1890 to 19070 about black folks in [name omitted] because how do we effectively evaluate without context?”

Capacity

When evaluators described their ability to facilitate or partake in the conditions in which equitable evaluation can take place, they also mentioned capacity for both the evaluator and stakeholders to engage one another in such a process. The capacity category, or the resources an evaluator or stakeholder has to produce something, consists of the following properties: time, funding, personnel, and community readiness. Evaluators often described their inability to include stakeholders in the evaluation due to a lack of one or more resources such as the lack of funding and personnel allotted to evaluate a community organization's program: “And sometimes, I think that's why the evaluations that I’ve done haven’t’ been as participatory as I would like because we’ve just run out of time.” Thus, the capacity category emerged as an enabling condition out of evaluator descriptions of constraints on creating equitable evaluation practices.

In summary, the GIT offers a working definition of gatekeeping and a conceptual identity for gatekeepers. The GIT describes themes that influence the conduct of evaluation; identifies the strategies that evaluators use to navigate the social-structural conditions of evaluation projects; and the gatekeeping disruptions that problematize equitable evaluation practice. Summarized in Table 1, these themes are interrelated, influencing each aspect and each stage of an evaluation project.

Discussion

Positioning Gatekeeping Influence Theory in the Literature

The purpose of this section is to assess the ways in which the hypotheses and theoretical concepts that emerged from the grounded theory process supports or challenges existing literature. GIT proposes a contextualized and multidisciplinary understanding of relational dynamics in evaluation that is not easily categorized into any of the approaches found in evaluation theories or approaches. GIT brings together social, psychological, and decolonial approaches to relationships in evaluation, addressing several gaps in our knowledge about gatekeeping in evaluation, and informing several existing approaches, theories, and models used in evaluation. In order of influence, GIT builds significantly on the community-based participatory research (CBPR) conceptual model, participatory evaluation, transformative and empowerment evaluation, feminist evaluation, critical race theory, and the concept and practice of community development.

The GIT is best supported by, and lends to most support to, the contextual and partnership dynamics dimensions of the CBPR conceptual model. The CBPR conceptual model grew out of the examination of the promoters and barriers of CBPR partnerships by scholars at the University of New Mexico Center for Participatory Research and at the University of Washington Indigenous Wellness Research Institute. The CBPR conceptual model focuses on the influence of contextual factors on community-academic group dynamics and how these dynamics go on to influence research and intervention designs. This influence on research and intervention designs then has several interrelated outcomes on the achievement of systems and capacity change, with the CBPR conceptual model focusing on changes in health disparities and social justice (Belone et al., 2016; Wallerstein et al., 2008).

The fundamental differences between GIT and the CBPR conceptual model is GIT's inductive approach to understanding the contextual and relational factors involved in evaluation partnerships; GIT's focus on gatekeeping disruptions; and GIT's focus on power differentials. Thus, GIT builds on the CBPR conceptual model by looking through an evaluative lens to understand the potential dynamics and effects of disruptive relational processes generally created by those who control access to people or information. With the focus on these gatekeeping disruptions, the GIT also questions the purpose of evaluation (and potentially research) within the university. Moreover, the GIT goes a step further than the CBPR conceptual model and attempts to initialize how various partnership and relationship dynamics are co-constructed through joint-action and various social-structural contexts (e.g., collaborative histories) that impact relational dynamics such as trust and power differentials.

GIT builds from and adds to participatory evaluation (Cousins & Chouinard, 2012). Traditionally, evaluation has been driven by the postpositivist paradigm that places empirical method and rigor over a concern for the population studied (Greenhalgh & Russell, 2010; Parker, 2004). This view of evaluation has often worked against the type of participatory process that attends to critical social context issues that affect the program and issue being studied. Traditional evaluation methods leave much to be desired when the voices of the individuals being studied are excluded from the process. Participatory approaches to evaluation (Cousins & Chouinard, 2012; Cousins & Whitmore, 1998) calls for program evaluators to assume the role of the knowledgeable insider. Here, the evaluator and the evaluators can better examine and expose the intended and unintended consequences and benefits of the programs. An evaluator, who is a member of the group, or has familiarity and trust with that group, is in a better position to ask the questions to illuminate the complexity of the issue under investigation (Lund et al., 2016). GIT challenges the assumption that there exist clear distinctions between the evaluator as independent outsider and knowledgeable insider, calling for evaluators to consider positionality and social-structural contexts. Additionally, GIT problematizes evaluators who are simply “members of the group,” adding factors to consider such as membership dynamics, evaluator reflexivity, and power, factors that add dimension to participatory evaluation's principle of forming trusting partnerships.

Transformative evaluation's axiological assumption is that evaluation must be designed so that it promotes social justice and human rights (Mertens, 2017). The GIT departs from transformative evaluation in two important ways. One, the GIT problematizes both the evaluator who and evaluation that extends from the university and other institutions that are wellsprings of hierarchy and ideological dogma. Two, the GIT suggests that evaluators who take on the role of the social justice advocate are prone to believing that only others are withholding access to people and information and neglecting to take into account collaborative histories that have occurred long before their evaluation project.

While the transformative paradigm asserts itself as “a meta-physical umbrella that brings together philosophical strands associated with feminism, critical theory, Indigenous and post-colonial theories” (Mertens, 2017, p. 1), GIT suggests that there are a great number of relational tensions, and strategies to counteract these tensions, that arise precisely because there is a confluence of philosophical and ideological clashes when groups of stakeholders collaborate. Take Feminist evaluation, for example. Feminist evaluations seek to uncover how gender bias “…is manifest in the major institutions in society … Feminism examines the intersection of gender, race, class, and sexuality in the context of power” (Mertens, 2017, p. 154; Podems, 2010). Thus, a feminist evaluator seeks to improve programs, processes, practices, and/or policies by seeking to uncover the ways in which power imbalances are a function of systemic gender bias, race, and class. In practice, feminist evaluation emphasizes participatory, empowering, and social justice agendas (Howton et al., 2011; Patton, 2010). In addition to examining power imbalances, GIT proposes that the process of improving programs, process, and practices must also consider the evaluand's social-structural context, including prior research and evaluation histories, the evaluator's identity and background, and new and ongoing relationships that create the enabling conditions for equitable evaluation practice. One similarity between feminist evaluation and GIT is the value they place on multiple ways of knowing; this includes an emphasis on researcher reflexivity as a form of self-knowledge. However, feminist evaluation is more specific about how this is done, seeking alternative ways to knowing in programs, policies, and practices apart from explanations grounded in men's reality. GIT, on the other hand, suggests co-creation and joint decision making to seek alternative ways of knowing apart from colonialist knowledge production. In any case, the prioritization of multiple ways of knowing translates into the need for inclusive evaluation practice. Thus, both feminist evaluation and GIT suggest diversifying key stakeholder inclusion to empower stakeholders with different perspectives and identities. Further, Black feminist evaluation (Collins, 1986; Haley, 2019) is slightly more in alignment with GIT since it places more emphasis on interlocking systems of oppression and examines the types of political economy that are manifest in differences in culture and group membership as expressed as “insider” versus “outsider.” While Black feminism captures the unique standpoint that the “outsider within” status can create (Hooks, 2000), which aligns well with GIT, GIT adds that this status is subject to relational dynamics in community-based evaluation projects that complicate insider and outsider demarcations; from the evaluator perspective doing inclusive work, membership status is fluid and often unclear.

In another example that complicates transformative evaluation's meta-physical umbrella, critical race theory (CRT) and GIT align and depart in several ways. CRT provides the basis for an analytical model that exposes how racism functions in America to oppress racially/ethnically diverse students, particularly African-Americans, to diminish its effects and achieve equality (Choet al., 2013; Mertens, 2016; Newell & Kratochwill, 2007). Thus, critical race evaluation, combined with elements of participatory evaluation, examines the social contexts of racism in the broader society where minority groups that are the subjects of the evaluation have to be full evaluators throughout the process so evaluators can gain insights into not only racial oppression from the evaluators’ perspectives but its many intersections that affect evaluation outcomes, too (Reynolds, 2015). Though not specifically centering the Black experience, GIT builds from this perspective in several ways: (1) making the perspectives of socially marginalized group the central axis around which discourse on a topic revolves by building trusting, transparent relationships with community members; (2) awareness of personal biases through evaluator reflexivity and building Self-Collective awareness; (3) examining histories of collaboration that have reinforced negative understandings of race and race relations based on historical, contextual, political, or other social considerations (Grahamet al., 2011).

The GIT extends prior work by Shulha et al. (2016), who call for more research on the nuances of evaluator/stakeholder interdependence in collaborative approaches to evaluation. These evaluators outlined the evidence-based principles to guide collaborative approaches to evaluation that include developing a shared understanding of the program, promoting appropriate processes, monitoring and responding to the resource availability, monitoring evaluation progress and quality, promoting evaluative thinking, following through to realize use, clarifying motivation for collaboration, and fostering meaningful relationships. The GIT advances their principles of fostering meaningful relationships and promoting appropriate participatory processes in evaluation by providing added social, political, and structural interpersonal dynamics that contribute to the success or failure of evaluation collaborations. Moreover, combining the principles with the strategies for equitable evaluation practice outlined within the GIT may provide additional guidance for collaborative approaches to evaluation. For example, the capacity category under the enabling conditions theme within the GIT aligns well with the monitor and responds to resource availability principle, extending this principle by considering whether and to what extent stakeholders are ready for collaborative evaluation.

Finally, GIT supports the role of community development in evaluation. Community development is a process where community members come together to take collective action and generate solutions to common problems. The GIT proposes community development as a strategy in evaluation that enables the conditions for more equitable practice. However, again, the GIT questions the role of the evaluator and evaluation in the process of community development activities. When evaluation is viewed as the professionalization of community development in the form of empowerment evaluation, the GIT suggests that an array of social-structural, cultural, and relational dynamics must be taken into account. Thus, the GIT ultimately proposes that understanding the contextual and relational factors involved in disruptive collaborations may yield more equitable evaluation practices. For this reason, we must redefine what we mean by gatekeeping and gatekeepers in context of the GIT and the relevant literature.

(Re)Defining Gatekeeping and Gatekeepers

When we set out to understand what was meant when we heard evaluators, researchers, and others use the term gatekeeper, we ran into definitions that identified specific agents as those who controlled the access to other people and/or information within an evaluation project. We listened to presentations by nonprofit leaders and philanthropists who identified gatekeepers as foundations or nonprofits that mediated resource allocation and, in some cases, overemphasized resource protection to the detriment of fulfilling the organization's mission. In these cases, gatekeepers function from a mindset of “no” (e.g., looking for reasons not to fund an organization); lack community engagement; and think in terms of “us” versus “them” when granting resources. Our original definition of a gatekeeper would broadly fit this description; however, the GIT and relevant literature suggests a more nuanced process involved in creating gatekeepers, which we define as gatekeeping.

GIT proposes that gatekeeping is a psycho-social-cultural construct that reflects a process by which gatekeepers, either as an individual (e.g., program manager) or a group of individuals (e.g., foundations, nonprofits, universities, etc.), emerge from contentious intepersonal and intrapersonal dynamics as key decision-makers through which people, knowledge, and data are filtered. In evaluation, the GIT proposes that the effect of gatekeeping is dual. One, that it mediates how well evaluators and their evaluations can attain equity in their projects. Two, that the products of the evaluation are not as reliable or representative. Importantly, the social-structural contexts that evaluators find themselves in have a large impact on their ability to engage stakeholders.

Limitations

One potential limitation to this study is the sample size. We rely on insights from just 14 interviews. A larger number of interviews might have led to a greater diversity of insights.

Another potential limitation, as with any grounded theory study, is the introduction of sampling bias and how the sample may bias grounded theory results. Following theoretical sampling procedures, we interviewed individuals who identified as evaluators. After several rounds of analysis, we noticed that (1) the data indicated that our interview protocol did not include any guiding questions; that (2) evaluators had primarily identified as oriented toward conducting inclusive evaluation projects, including an array of stakeholders; and (3) subsequent sampling needed to offer both confirmation and potential refutation of prior findings. In light of these weaknesses, in the second round of sampling we chose to interview evaluators who did and did not categorize their evaluations as stakeholder-engaged. Reasoning for interviewing evaluators who categorized their work as stakeholder-engaged was driven by data indicating a higher incidence of gatekeeping practices as well as to interview those who align their work with AEA's guiding principles for equitable evaluation practice. Reasoning for interviewing evaluators who did not categorize their work as stakeholder-engaged was to seek counterfactual evidence refuting established categories from prior themes. Thus, our final sample included evaluators who experienced an array of gatekeeping instances irreducible to how they categorized their approach to conducting evaluation. Overall, we believe our theoretical sampling method was strengthened by allowing for diverse evaluator perspectives to contribute to theory construction.

Conclusion

This article presents the empirical foundation for a new approach to understanding the influence of gatekeeping in evaluation and its impact on the achievement of more equitable evaluation practice. The GIT offers a more nuanced perspective of evaluation practices that are aimed at equity and social justice, highlighting how important relational dynamics and evaluation contexts are in building lasting partnerships. Future research is necessary to fully develop the GIT conceptually. We selected evaluators who conduct evaluations using an array of approaches to be included in the grounded theory processes. However, we argue that the GIT would greatly benefit from a complex systems perspective, wherein each stakeholder who was a part of an evaluation was interviewed so that a map of the GIT could be drawn, and the influence of gatekeeping could be better tracked. We are currently reengaging evaluators from this study, asking them to reconvene stakeholders from their past or current evaluation projects, so that we can build systems maps of the GIT. Future research is also necessary to test the practice application of the GIT. Although expanding this work may include recruitment challenges due to the negative connotation associated with the concept of gatekeeping, we must expand our inquiry to include others’ perspectives. This may include the perspectives of those who participate in evaluation projects rather than those who facilitate them.