Sage Journals: Discover world-class research

Abstract

Evaluation policy has been identified as an important means of shaping and influencing organizational evaluation practice, yet, to date, little empirical research has been conducted to deepen our understanding of this relationship. The purpose of this study was to illuminate evaluation policy’s role in leveraging organizational capacity to do and use evaluation. We interviewed 18 published evaluation scholars and practitioners from North America and Europe about this topic. A thematic analysis of findings underscores the importance of context, policy attributes, enablers, and organizational benefits. Based on the findings, we developed an ecological conceptual framework to guide thinking about the role of evaluation policy in capacity building. We discuss these findings in terms of practical implications of understanding context, redressing the imbalance between learning and accountability purposes of evaluation, and organizational leadership, and we conclude with some implications for research.

Keywords

evaluation capacity building evaluation policy organizational evaluation capacity

Introduction

Evaluation scholars who are concerned with building evaluation culture and practices in organizations have been adding to a growing literature on evaluation capacity building (ECB). Organizational ECB is commonly defined as “the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine” (Stockdill et al., 2002, p. 14). Historically, organizational evaluation capacity has been thought of as the competencies and structures required to conduct high-quality evaluation studies (i.e., the capacity to do evaluation; Preskill & Boyle, 2008; Sanders, 2003). More recently, this definition has been expanded to include organizational capacity to use evaluation or the organization’s ability to integrate evaluation into its decision-making processes (Bourgeois & Cousins, 2008; Bourgeois et al., 2013; Cousins & Bourgeois, 2014).

The enhancement of organizational evaluation capacity is essential to the production of quality evaluations that enable organizations to meet their accountability requirements and enhance organizational learning and decision making (e.g., Bourgeois et al., 2013; Boyle et al., 1999; Cousins et al., 2004; Sanders, 2003; Stockdill et al., 2002). In this study, we are interested in the extent to which and how evaluation policy is implicated in building and maintaining organizational evaluation capacity.

Evaluation policy remains an obscure and underdeveloped research area (Dillman & Christie, 2017; Trochim, 2009). It has been defined as “any rule or principle that a group or organization uses to guide its decisions and actions when doing evaluation” (Trochim, 2009, p. 16). Based on this definition, evaluation policy is inherently collective and could be written and explicit or unwritten and implicit. Despite recent, growing interest, there is a particular paucity of research examining the impact of evaluation policy on organizational evaluation culture and practice (Mark et al., 2009).

The purpose of this exploratory study is to investigate the extent to which and how evaluation policy has the potential to improve organizational capacity to implement evaluations and use them. Does evaluation policy have the potential to leverage such capacity? If so, under what conditions or circumstances?

Conceptual Background

The Nature and Consequences of Evaluation Policy

Evaluation policies exist within public, private, and nonprofit sectors. They are created and applied in specific political, economic, and cultural contexts, whether at organizational, national, or global levels. Trochim (2009) categorized their content into specifications about evaluation purposes, participation, capacity building, management, process and method, role, use, and meta-evaluation or quality assurance. In general, evaluation policy is intended to govern the way evaluation is conducted and used. It carries an organization’s intentions to be accountable to stakeholders, to support organizational learning, and to use evidence for decision making (Datta, 2009).

Evaluation policy has the potential to influence decision making in a range of ways (Chelimsky, 2009; Datta, 2009; Mark et al., 2009; Trochim, 2009). For example, consider the following policy attributes and potential influences:

When to evaluate outcomes: If done early, could help determine whether the program would have the potential for desired impact;

How to evaluate: Could ensure that appropriate methods are used to answer identified questions (e.g., improvement, impact);

Evaluator locus—choosing internal or external evaluation professionals: Could enhance evaluation relevance to information needs and whether and/or how evaluation results are used;

Links to funder demands: Could increase program and organizational accountability and ensure efficient use of resources;

Use enhancement strategies—user identification, administrative response to results, “demonstrated use” reports: Could inform program decisions about continuance, improvement, termination, and the like.

Policy thus sets an important frame of reference for how funders and evaluators define, implement, report, and use evaluations. For instance, in politically contentious organizational contexts, evaluation policy can help address challenges to evaluation credibility or accuracy (Chelimsky, 2009). In hyper-political contexts, policies related to evaluation use and its links to decision making can help organizations to meet international transparency and accountability demands (Stern, 2009). In challenging economic conditions, policies requiring evaluation to include “reconsideration study” can help decision makers understand potential consequences for goal achievement in the hypothetical event of a significant intervention budget cut (Leeuw, 2009). These examples show how evaluation policy is influenced by and can influence contextual conditions.

Evaluation Policy as Part of the Evaluation System

The role and consequences of evaluation policy can be examined using a systems lens. According to Liverani and Lundgren (2007), the term evaluation system refers to “the procedural, institutional and policy arrangements shaping the evaluation function and its relationship to its internal and external environment” (p. 241). Implicated are evaluation independence, its resources, and cultural attitudes toward it. Using a systems lens highlights influences on evaluation demand and its use, including the dissemination and integration of results. Dahler-Larsen (2006) argued that an evaluation system acts as “a continuous stream of information” (p. 1) that plays a role in the production of evidence to support decision making. To a great extent, system functioning depends on evaluation policy, organizational design, and staffing structures that allow the flow of information. According to Dillman and Christie (2017), policies constructed from a systems perspective, rather than isolated rules and guidelines, help organizations achieve desired outcomes, increase evaluation use, and support organizational learning.

As Leeuw and Furubo (2008) asserted, an evaluation system must represent an organization’s internal consensus about the meaning of evaluation, the purpose it serves, and the type of knowledge it produces. To that end, internal responsibility for evaluation, as opposed to outsourcing, may act to enhance evaluation capacity and promote institutionalization. We now turn to a review of what we know about organizational evaluation capacity and how to enhance it.

Organizational ECB

Evaluation scholars have developed a number of conceptualizations of the meaning and the practices that constitute ECB. For example, Taylor-Powell and Boyd (2008) introduced a three-component framework that focuses on structuring strategies used in ECB. Professional development, resources and supports, and organizational environment are the three main dimensions of ECB practice. Along the same lines, Preskill and Boyle (2008) developed an ECB model that outlines a range of factors (e.g., evaluation knowledge, skills, attitudes) that may influence the development and impact of ECB processes on evaluation practice. Such elements may be intentionally integrated into organizational evaluation policy. In doing so, the focus would be on enhancing organizational capacity to practice evaluation or engage with it in an oversight function. Yet, as Cousins et al. (2004) asserted, “the integration of evaluation into the culture of organizations…has as much to do with the consequences of evaluation as it does the development of skills and knowledge of evaluation logic and methods” (p. 101). As such, “ECB is about developing the capacity to do evaluation, as well as the capacity to use it” (Levin-Rozalis et al., 2009, p. 194).

Organizational Capacity for Evaluation

Cousins et al. (2008), Nacarrella et al. (2007), and Nielsen et al. (2011) lamented that while much had been written about the role of ECB, scholars had directed considerably less attention toward the concept of evaluation capacity itself. Whereas organizational evaluation capacity manifests as visible, enacted evaluation practices and processes, ECB is the process by which an organization develops its understanding and ability to undertake such. As has been the case with scholarship on ECB, an emphasis on understanding evaluation use as an integral part of organizational evaluation capacity has been missing (Cousins & Bourgeois, 2014).

With this gap in mind, Bourgeois and Cousins (2008, 2013) developed and validated a comprehensive conceptual framework of the organizational dimensions of evaluation capacity. The framework consists of six subdimensions: three associated with the organizational capacity to do evaluation (human resources, organizational resources, and evaluation planning and activities), and three associated with the organizational capacity to use evaluation (evaluation literacy, organizational decision making, learning benefits). This evidence-based framework has been used for research and practice in governmental and other public-sector organizations (Bourgeois & Cousins, 2008; Bourgeois et al., 2013, 2016).

While evidence is beginning to accrue as to how organizational evaluation capacity can be enhanced, evaluation policy’s role in this regard is not well understood. This study sought to explore the role of evaluation policy in building (or impeding) organizational capacity to do and use evaluation through one-on-one interviews with knowledgeable evaluation scholars and practitioners. Next, we briefly describe the study participants and the research method.

Method

The study relied exclusively on in-depth interviews with participants with high degrees of knowledge about evaluation policy and/or ECB. To follow is a description of the strategies, we used for sampling, data collection/processing, and analysis.

Study Participants

We selected the participants on a reputational basis. Each participant had published books and/or peer-reviewed journal articles on evaluation policy and/or ECB or, to a lesser extent, had significant professional experience in at least one of those areas. We identified candidates from the latter group through consultation with special interest group leadership, evaluation conference presentation descriptions, and collegial nomination. Including knowledgeable practitioners enabled deeper first-hand interpretations of both the role of evaluation policy in building organizational evaluation capacity and the multiple and varied dimensions at play within an organizational context.

Of 22 invited evaluation scholars and practitioners from North America and Europe, 18 agreed to participate in a remote interview. While the majority of the participants were university professors, some were working as evaluation consultants and/or with government or private organizations. The participants had between 15 and 30 years of experience working in evaluation in regional, national, and international contexts. All participants resided in the global West, but some worked in international development.

Interview Methods

Relying on teleconference/telephone technology, the lead author interviewed all 18 participants once they formally agreed to the terms of a letter of informed consent. The sessions were audio-recorded with permission and lasted 60–90 min.

The interviews were semi-structured. To mitigate interview bias, we asked participants for their definition of evaluation policy and organizational capacity for evaluation and refrained from introducing them to our conceptualizations, as described above. Listed in Box A are sample questions from the interview protocol.

Box A

Sample Interview Questions

So, first of all, I’d like to ask you about your own definition of:

Evaluation policy

Organizational capacity to do evaluation

Organizational capacity to use evaluation

What impact do you think evaluation policy has on organization capacity for evaluation?

How does evaluation policy influence evaluation practice?

How does evaluation policy influence evaluation use?

In your opinion, what evidence should organizations collect on the impact of the evaluation policy on evaluation practice and use?

In your opinion, what factors contribute to benefitting from evaluation policy in building/enhancing organizational capacity for evaluation? (Probe: resources; organizational support; organizational culture; economic, social and political context)

Repeat for “impede benefitting”

Analysis

The lead author transcribed the audio recordings verbatim into individual Word files, which were then integrated into NVivo (Version 12) for analysis.¹ The analytic strategy we employed was to use the previously described conceptual aspects of evaluation policy and dimensions of organizational evaluation capacity as the basis for an initial list of “start codes” while remaining open to identifying additional emergent codes.

The work was carried out in two stages. During the first stage, the lead author read through each transcript, flagging specific pieces of information to which she applied start and emergent codes. She then identified patterns and connections between specific codes and subsequently developed emergent themes based on the implicated concepts. The themes included nuanced elaborations of the concepts and their relationship to other variables. This process was iterative. The second stage involved the classification of the themes according to their focus on the relationship between evaluation policy and organizational capacity for evaluation. Both authors collaborated throughout this stage in order to “support, strengthen, modify, or disconfirm the findings” (Saldana, 2009, p. 229). Joint analysis included interrogation and resolution of coding discrepancies, theme identification, and relationship verification. As a measure of data quality assurance, we include rich, thick verbatim descriptions of participants’ accounts in order to support our interpretation of the findings. We now turn to a thematic summary of the findings.

Findings

In this section, we begin with an analysis of participants’ understandings of the main constructs of interest: evaluation policy and organizational evaluation capacity. We then turn to a thematic analysis of the variables that moderate the relationship between policy and capacity and close with the presentation of an integrative conceptualization of the relationship.

Defining Evaluation Policy

There was some variation in participant responses to the question of how they define evaluation policy. Table 1 shows the results of a content analysis of the answers given by the interviewees.

Table 1.

Participants’ Definitions of Evaluation Policy.

Theme	Coded Chunks^a	Illustrative Interviewee Remarks ^b
Guidelines	12	I think evaluation policy is the underpinning guidelines that direct the activities of evaluators within a particular organization within a particular context. (IP12) I think policy is generally a statement that includes purposes or goals at some level of specificity the procedures for reaching these goals or specifications and then details about how this is to be accomplished. (IP4)
Strategy framework	12	At the organization level, evaluation policy is a statement of the organization evaluation value. In some organizations it is very high level…it is a framework of guiding evaluation within the org but it can be very high level or it can be very specific. (IP11) The policy should also show or explain how the values of the organization are woven into the way evaluation is conceived and to the way in which it’s practiced. (IP9) [It is] a strategy tool and practice itself to guide and direct how the ways in which evaluation is used, how it’s conceived, how it’s looked at, how it’s conceptualized within whichever of those levels we are talking about. And how that’s used to guide and direct evaluation practices within whatever setting were talking about. (IP3)
Codification	8	formal legal, regulatory or administrative directive that defines the scope of evaluation work or various characteristics of evaluation (IP14) It is a set of goals and rules…maybe in some countries laws, but that’s not often the case. (IP7) Informal would be just something that people are used to. A new program officer is hired at a foundation and they just kind of see or get the idea that large projects have a certain kind of evaluation and they start doing the same thing. (IP14)
Miscellaneous characteristics	3	Collective:…a set of values, a set of commitments that they want everyone in the organization to honor as a regard to evaluation (IP10) …continuously reviewed by a number of stakeholders of the organization (IP5) encourage the learning component rather than the accountability component and those [policies] are more effective in terms of building the organizational capacity than those that say the policy is here to judge whether something is good or bad…(IP16)
No consensus	3	there is a limited amount written about evaluation policy I’m not sure that there is a complete consensus what constitutes evaluation policy. (IP14) Yikes what are they talking about” because it’s not always clear what it means…(IP2)

^a Segments of texts coded at the level of ideas. The count represents the number of chunks, not necessarily the number of participants. However, in general, ideas or chunks were spread across participants.

^b Interviewee identifier in parentheses.

Participants often talked about evaluation policy in terms of a set of rules or guidelines or as a strategic framework within the organization. In referencing guidelines, participants focused on descriptive statements about how evaluation is to be conducted within the organization, including roles and responsibilities. Such statements are intended to guide evaluator practices. Participants also talked about evaluation policy at the level of organizational strategy; it was not so much about when and how to do evaluation but what evaluation is expected to accomplish within the organization. There were sometimes references to organizational values embedded within the evaluation policy. When conceived as a strategic framework, evaluation policy could be either quite high level or very specific.

Several participants also referred to the explicit nature of evaluation policy or the extent to which it was codified. Of the eight instances of such references, all but one referred only to explicit statements or formal legal, regulatory, or organizational directives. Our own definition, provided above, encompasses the notion that policy can be implicit or hidden. Yet only one participant inferred that evaluation policy can be implicit or informal and that new members of the organization develop an understanding of evaluation expectations over time. In a few instances, miscellaneous characteristics of evaluation policy were referenced, such as its collective nature, it being subject to ongoing review and revision, and the relative emphasis on learning, as opposed to accountability, and attendant implications for ECB. Finally, there were three references to a lack of clarity or consensus about how evaluation policy is defined.

Defining Organizational Evaluation Capacity

To capture the meaning of organizational evaluation capacity, we asked participants to define the capacity to do evaluation and the capacity to use it. Our analysis of the responses shows widespread agreement among the participants that the two aspects are interactive and inform one another. Further, their comments align to a substantial degree with our own conceptualization as described above (see also, Bourgeois & Cousins, 2008, 2013). Appearing in Table 2 are the results of our content analysis along with illustrative quotations.

Table 2.

Participant Definitions of Organizational Evaluation Capacity.

Theme	Coded Chunks^a	Illustrative Interviewee Remarks ^b
Capacity to do evaluation
Organizational culture	15	Organizational culture is very important to support the evaluation function…I think the organizational culture comes first because nothing will happen if the organization doesn’t believe in evaluation. (IP1) You need the right kind of organizational culture. I wouldn’t even say an evaluation culture but sort of a learning-oriented culture centered upon improvement where staff and managers are rewarded, not for meeting objectives but for asking difficult questions, for experimenting or trying different things even if they don’t work. That’s really what it is. (IP17)
Infrastructure/system	13	I think the degree of hierarchy and formalization of tasks within the organization will have an important impact over the success of the evaluation. (IP2) To conduct evaluation you need a system, you need resources, you need big support from management. That’s really important. (IP11) There should be a set of standards, there should be processes, there should be procedures. (IP15) A specific system, and structure and resources. Those are very important things to be in place to conduct evaluations. (IP14) I believe a commitment to the broad idea of evaluation as an integral part of how an organization or a system evolves or learns has to be there. If you don’t have that broad level commitment, then I think all is lost. (IP3)
Resources	11	Organization capacity is based on its resources…without resources how are you going to do that? (IP6) Sufficient budget could be the key to successful evaluations or the reason for the failure of evaluations. (IP17) Allocating resources to evaluation is important to conduct and use evaluation as well as developing the capacity of the staff to do training. (IP1)
Leadership	8	Unless evaluation and capacity development are supported by senior managers, they won’t be prioritized throughout the organization. (IP18)
Competencies/literacy	4	You need skilled evaluators who can work with people to develop their evaluation capacity. You need evaluators who have high communication skills to raise the importance of evaluation beyond the evaluation unit. (IP4) I think people need to understand evaluation and why it is important. They need to be educated. (IP13)
Capacity to use evaluation
Leadership/decision making	16	I think evaluation use mostly depends on how managers make their decisions and where they get the information they need. Do they get it from evaluation? (IP18) A lot of evaluation use has to do with management. (IP7) they use it for decision making, they encourage reflection, dialogue and findings, they share their learning internally and externally and the leader does model and reinforce and reward people for learning from data. (IP9) Does top management live the talk? Do they really talk of evaluation, can they give positive examples of how they use evaluation for their decisions, how they reinforce and reward people who have done evaluation and been subject to evaluation? Do they really take it seriously themselves? (IP14)
Relevance	11	People will use evaluation or they won’t use it when it’s in their interest. The question is how can you persuade them that it’s in their interest? (IP8) If you can’t show people that evaluation can benefit them and benefit the organization, then why did you ask them to do it? It doesn’t matter if you have a policy or not. (IP9) Improving awareness, helping people understand. People have to understand what we mean by evaluation…we have to use the same language and the same definitions. (IP17)
Learning	7	Developing the ability of an organization to engage in an evaluative process, which means ask the right questions, use the right methods, collect the right data and analyze it and make sense of it. (IP3) I think in terms of a specific organization, it means we are able to learn how to improve and how to change what we are doing in a particular area. (IP6)
Engagement	3	I think the more individuals that are involved in a process, the greater the likelihood that they’ll find the process and some of its critical results credible, meaningful, and so on and so forth. (IP5) You really need to engage people in the evaluation process and make them really believe in it. (IP1)

^b Interviewee identifier in parentheses.

The majority of the participants referred to the capacity to do evaluation in terms of organizational culture, infrastructure, and resources. Participants focused on a learning-oriented culture that accepts experimentation for development and growth. They also highlighted the importance of a supportive system to set the stage for the evaluation function activities. Participants tended to commingle thoughts about ECB with organizational evaluation capacity in their definitions. For example, the availability of resources was identified as essential when thinking about ECB. Participants also noted that leadership support is necessary to establish an organizational environment for ongoing learning and improvement. To a lesser extent, evaluation literacy and competencies were identified as critical to understanding and promoting ECB activities.

Almost all of the interview participants referred to the organizational capacity to use evaluation in terms of management’s application of evaluation to support decision-making processes. There were also several references to the relevance of evaluation and the benefits it brings to organization members. Seven references were made to the learning that takes place during and after the evaluation processes as having significant implications for use. Finally, three participants mentioned the engagement of stakeholders in the evaluation process in their definition of the capacity to use evaluation.

It is interesting to note that many of the participants’ definitions of evaluation policy and capacity touched on variables that moderate the relationship between the two. That policy was framed in strategic terms and that capacity for evaluation use was grounded in considerations of organizational culture, ensuing benefits, and learning are cases in point. Table 3 shows the themes that emerged about this topic. We take up these themes, organized by higher order categories—context, attributes, enablers, benefits—in turn.

Table 3.

Concepts Moderating the Relationship Between Evaluation Policy and Organizational Evaluation Capacity.

Category	Theme
External and internal context	Context
	Organizational culture
	Evaluation system
Policy attributes	Policy content
Policy attributes	Perceived purpose: Learning vs. accountability
Enablers	Organizational leadership
	Evaluator role/skill
	Stakeholder engagement
Organizational benefit	Integration and use

Context

Contextual factors and conditions affect the development, implementation, and outcomes of policies and programs because they shape the way in which evaluation is approached, practiced, and used. For example, some participants shared the view that evaluation policy is likely to have a major impact on evaluation practice and capacity building in developing country contexts. This is because the needs and demands for evaluation focus on supporting the development of evaluation capacity and strengthening related systems of management, learning, and accountability. As one of the participants suggested:

If you want to develop evaluation capacity in a sub-Saharan African country, you better have an evaluation policy that tells people there what evaluation is and what they need to know and what they are supposed to do. They need something to refer to; it will guide them through the process. (IP8)²

On the other hand, in the global West (i.e., developed countries), evaluation policy is mostly perceived as another level of bureaucracy and an additional requirement that actors need to fulfill. This is especially the case in political contexts where there is a high demand for evaluation for accountability purposes, as is implied by the following comment: “In Europe, we have evaluations, we have performance audits, we have oversight, we have supervision, we have inspections…we are evaluated to death” (IP2). It seems likely that in contexts where evaluation is mainly used for performance measurement and performance accountability, evaluation policy becomes the dominant accountability instrument. This can hinder learning, improvement, and, ironically, performance:

If policy is not well thought out, it can damage practice. In England, strict policy that focused mainly on accountability was a disaster because it was driving teachers out of the profession. (IP1)

Some participants also pointed to the influence of the historical and economic aspects of the evaluation policy context and the ways in which they affect the practice of evaluation and people’s perceptions of evaluation. For example, some argued that the European Structural Funds were a major driver for spreading the practice of evaluation throughout Europe in the 1980s. They indicated that the introduction of evaluation into many European countries was a result of the requirements of the structural fund regulations to assess value for money, requirements that continue to influence how people think about evaluation today. In such contexts, evaluation is associated with a high level of accountability, with the main motivation being the fulfillment of external requirements accompanied by little or no interest in learning in organizations.

Many of the contextual variables referenced were external to organizations, but their influence on internal variables is often quite evident. As a case in point, significant external accountability demands influence and shape internal organizational perceptions about evaluation being exclusively accountability oriented.

Organizational Culture

Evaluation policy will have little to no effect on capacity building if the organization lacks a culture that supports making evaluation strategic and useful, which in many cases is established at the management level. Many participants argued that when organizations are committed to evaluation, having limited resources is not an obstacle for evaluation; instead, it is a matter of management being strategic with the available resources and more intentional about learning and evaluation activities. According to most participants, to benefit from a well-thought out evaluation policy to create or support a dynamic environment, organizations must have a culture that encourages learning, risk-taking, and dialogue to improve their performance. Many participants also shared the view that the type of organization and the stability of its management and governance structure affect its culture and influence both the role of policy in promoting evaluation capacity and the feasibility and sustainability of capacity-building efforts.

Evaluation System

A supportive evaluation system within the organization is essential for setting the stage for evaluation policy fostering ECB activities within the organization. As one participant put it, “Evaluation policy can light a flame, but the flames are only going to catch it if there’s dry timber and so a certain infrastructure has to be in place to see an impact” (IP16).

An essential aspect of the organizational infrastructure is the degree of coordination within the system to ensure access to evaluation resources and information, which is affected by the alignment of work processes and data sharing between units and departments. For example, some interviewees indicated that responsibility for evaluation data collection and use could implicate different units within the organization. Without proper coordination, the potential for evaluation policy to leverage ECB activities remains unrealized. This is particularly the case with the ongoing collection of performance measurement data by program staff and its consequent suitability for evaluation purposes.

Another important moderator is the availability of appropriate resources to implement the evaluation policy and to turn its strategies into actions that can accomplish its objectives and, ultimately, achieve the desired impact on organizational capacity. This includes supporting staff participation in capacity building activities and allocating sufficient time to integrate the policy into the organizational system. Some participants noted that resources are needed in order to bring organization members on board. They must have enough time to implement any evaluation activities that they are not currently performing.

Policy Content and Type

Participants suggested that whether evaluation policy is descriptive or prescriptive, policy content, particularly requirements, significantly impacts evaluation practice and capacity. Some interviewees were of the view that descriptive evaluation policies are more favorable for guiding evaluation practice because they provide general guidance for the evaluation function without restricting practice choices. Prescriptive policies, on the other hand, include extensive details and instructions that effectively constrain evaluation practice and reduce the flexibility needed to learn from it. As one participant explained,

Evaluation policy at the federal government is considered to be very prescriptive, and it results in evaluations that are so focused to meet the requirements of the policy that they set aside the actual informational needs of stakeholders. So evaluators really struggle because the policy is so explicit about the fundamental issues and the key evaluation questions to be asked. They are so focused on those that they are unable to match [them] to the actual needs of stakeholders. So they produce evaluations, and they check the boxes and are able to say yes, we did it. (IP16)

Further, participants thought that overly detailed evaluation policies not only restrain capacity and are difficult to maintain but are also deleterious to the quality of evaluation and thus may potentially do more harm than good. As one of the participants pointed out,

The EU is prescribing its evaluation policy and the rats pop out of the earth; not bureaucrats but rats. Folks that have no idea what they are doing, but they use the right words because the evaluation policy is prescribed and articulated. These guys are copying the words and putting them out to tender, and then they get it done. This is really a perverse effect of evaluation policies, and I see it happen all the time. (IP2)

Nonetheless, many participants agreed that effective evaluation policy is both a source of information and a tool for highlighting potential areas of concern and offering guidance without restricting evaluation practice. When there are restrictions on evaluation practice, capacity can be limited. Thus, recognizing the effect of these contextual factors is critical for understanding the dynamics of the role of evaluation policy in building evaluation capacity and identifying the challenges involved in the process.

Perceived Purpose: Learning Versus Accountability

Organization members’ perceived purpose of the evaluation function is critical for the role of evaluation policy in fostering evaluation use. As a participant noted: “Evaluation is actually something that means something when you’re tasked with either carrying out evaluations or contracting them out or managing them” (IP17). A common understanding of the meaning of evaluation within the organization and the familiarity of organization members with evaluation principles moderate the extent to which evaluation policy will impact use. Another participant explained:

There are different ways of viewing evaluation, and then there are different ways of understanding what policy is. As a strategy tool to guide and direct how the ways in which evaluation is used, it depends on how it’s conceived, how it’s looked at, how it’s conceptualized within the organization and how that’s used to guide and direct evaluation practices. (IP5)

Organization members tend to confuse or conflate evaluation and audit as a single concept or function within the organization with deleterious effects. Such confusion undermines support for and use of evaluation as a mechanism for learning and improvement and, ultimately, limits the role of evaluation policy. While the purpose of evaluation is to understand and assess the merit and worth of programs and to identify the best alternatives to inform decision making and generate improvement, the purpose of auditing is to assess the degree of compliance of a program with predetermined standards or objectives. When evaluation is done in an audit-like fashion, its meaning, value, and purpose diminish; the knowledge that the evaluation eventually produces is likely to be either useless or irrelevant for capacity building activities. For many participants, the primary focus on accountability as the major purpose of evaluation has significant, potentially negative, implications for evaluation, especially in terms of its learning benefits and use. Nevertheless, a number of participants acknowledge that accountability is an integral part of evaluation and that evaluation policy should reflect this reality. Yet, for evaluation policy to foster evaluation use, it should make accountability meaningful in a way that reflects the true value of evaluation.

Organizational Leadership

Several interviewees underscored the importance of organizational leadership to the evaluation policy–capacity building relationship. They suggest that leaders must support evaluation and communicate its importance to improve the organization’s effectiveness to both staff and partners. Such practice can make a significant difference in establishing an organizational environment for ongoing learning and improvement in which organization members consistently want to perform at their best. A participant explained as follows: “Unless evaluation and capacity development are supported by senior managers, they won’t be prioritized throughout the organization” (IP18). The right blend of leader pressure and support can breathe life into evaluation policy and move it from a static document to actions that boost opportunities for learning and capacity building.

Evaluator Role and Skill

The role evaluators play within the organization and their evaluation knowledge and skills influence the effectiveness of evaluation policy and enable or hinder its role in building organizational capacity. Essential technical skills are the identification of evaluation issues, the use of appropriate data collection methods, and the generation of evidence-based recommendations. Softer interpersonal skills such as building client trust, communicating evaluation messages with clarity and transparency, and being responsive to stakeholders’ information needs were mentioned as salient. A few participants also shared that evaluators have an important role in planning for and facilitating learning from the evaluation. Evaluators play a key role in educating organization members and improving their knowledge and skills regarding evaluation. As a participant noted: “Evaluators are the best people to talk about the policy. They are the passionate people to talk about it. They are able to answer questions and explain things, so they are best placed to do that” (IP18).

Another participant further argued that evaluators could act like coaches to guide organization members to achieve the goals of the evaluation policy and support them in building their capacity and knowledge about evaluation. Indeed,

When the organizations have a policy or guidelines, I think the evaluator needs to take on the role of a coach, coaching people about evaluation to keep them moving along. The last thing you want is this whole document to sit on the shelf because people don’t know what to do with it. (IP5)

Evaluators should also work with the organizational leadership to promote the value of evaluation as a management tool. One interviewee put it this way:

The directors of the organizations are not trained in evaluation, and they don’t have the skills, so they do the best they can to be compliant, but they don’t have the background or training like evaluators. They are not evaluators. Evaluators need to work with them to get there. (IP7)

The evaluator’s communication skills are critical for promoting evaluation and using the policy to foster evaluation capacity. One of the participants commented that “people need to be pulled into evaluation more than pushed to evaluation. You can push them to evaluation through mandates and policies, but I don’t think that’s going to get you much more than sort of ‘check-off-the-box’ answers” (IP1).

Communication between evaluators and other organization members is an important factor that affects not only whether they are supportive of the evaluation but also their perception of the value of the evaluation and its use. Thus, evaluators with the right skill set, and perhaps especially soft skills, play a critical role in helping organization members to understand the value of evaluation and develop their knowledge to address the long-term change the policy aims to achieve.

Stakeholder Engagement

Through our analyses, we identified the engagement of policy stakeholders in the policy-making process and/or the evaluation process and the learning that takes place during and after such processes as enabling moderating variables. A large majority of the participants emphasized the importance of ensuring the responsiveness of evaluation policy by actively involving those who will be affected by it.

Stakeholder engagement can enhance voluntary compliance with the policy for two reasons. First, stakeholders may adjust to changes more easily because they are made with opportunities for input to allow them to anticipate and overcome challenges quickly. Second, stakeholder engagement engenders a sense of legitimacy and shared ownership that motivates them to abide by the policy.

Some participants also affirmed that the evaluation policy should set clear objectives for stakeholders’ engagement in the evaluation process in order to strengthen the relevance and use of evaluation findings. They explained how evaluation policy could leverage participation in evaluation activities that will eventually increase their evaluation capacity. One of the participants indicated that if evaluation policy relies on a participatory approach and promotes the inclusion of all relevant actors in the evaluation process, more attention will be paid to marginalized, disadvantaged, or less powerful groups: “Participation is crucial. It is everything. If you don’t have the agreement of the people who are going to use it, who are going to endure the change, I don’t think it’s worth doing it.” Thus, evaluators can use the evaluation policy to take on a purposeful role in stimulating evaluation capacity use by engaging stakeholders, communicating with them throughout the evaluation process, and identifying teachable moments when participants can learn and apply various evaluation skills.

Integration and Use

We learned from participants that the impact of evaluation policy on the organizational capacity to use evaluation is moderated by its integration into decision-making processes and by how the purpose of the evaluation function is conceptualized within the organization. The ability of evaluation policy to positively foster evaluation use depends on the extent to which the organization’s management relies on evaluative information and uses it for decision making. If management does not demonstrate to staff how they can integrate evaluation and use it to make decisions, take action, and learn, evaluation policy’s potential to enhance evaluation use remains unrealized. This claim is illustrated nicely by the following comment:

It’s not just the policy that is going to make that difference in itself. It’s one way, but we’re talking about evaluation use. If it is in the policy but you never hear about it from top management, then other managers are not going to take it seriously themselves. (IP14)

Some participants also pointed out that when there is an evaluation policy at the government level, evaluation may be thought of as a burdensome “fill-in-the-form” and “check-the-box” exercise. As such, it will have no impact on building organizational capacity because it is not integrated or used for strategic decision making. The mentality around evaluation use in most government organizations is more summative or accountability-focused than formative. It is perceived as a control mechanism rather than as a learning and improvement process. In such cases, the role of evaluation policy in building evaluation capacity is compromised; it is most likely perceived as a mere tool for enforcing control.

Some participants also explained that the symbolic use of evaluation as a strategy for retrospectively justifying decisions already made within an organization might be prevalent. Such practice not only limits the role of evaluation policy in fostering evaluation capacity but also reflects the persistence of an unfavorable gap between evaluation practice, on the one hand, and the use of evaluation as a mechanism for learning and improvement, on the other. In this sense, many participants expressed concern that evaluation and evaluation policy are used as tools to preserve the status quo rather than to explore and challenge assumptions. Alternatively, some participants noted that when evaluation information is clearly integrated and connected to decision-making processes, the evaluation policy can signal to staff and the organization what the evaluation is going to be used for and on what values the evaluation practice is based.

Summary and Integrative Conceptualization

The above thematic analysis shows a range of variables that interview participants discussed as moderators evaluation policy’s influence on organizational evaluation capacity. Described were variables associated with external and internal organizational contextual conditions, attributes, and perceptions of the policies themselves, enabling factors associated with policy implementation, and reciprocal effects of organizational benefits from policy. Several of these variables touch multiple levels—individual, organizational, and extra-organizational—and tend to interact across these levels. With this in mind, we developed a three-level ecological framework (see Figure 1) that illuminates the multiple interconnected variables/conditions that moderate the relationship between evaluation policy and organizational capacity to do and use evaluation.

Figure 1.

Ecological framework of moderating variables that explain the relationship between evaluation policy and organizational capacity for evaluation.

This framework comprises the following: the macro level (which focuses on the social and political context), the meso level (which focuses on the organization level), and the micro level (which focuses on organization stakeholders). The framework represents an emergent integration of our response to the research questions posed for the study. Our thinking in developing it was influenced not only by participants’ responses but also by Bronfenbrenner’s well-known ecological systems theory (1979). This theory is well suited to understanding evaluation capacity development because it focuses on providing a systematic approach to explaining social development and emphasizes each level’s importance for the development process. The theory recognizes the influence of context, community, and culture on the process of development. Rather than ignoring these fundamental variables, the ecological system theory explains the interrelationship among multiple variables of the development process and acknowledges their interdependence, a perspective that helps to promote a holistic understanding.

As shown in Figure 1, moderating variables operate at one of three levels. At the macro level, our analyses revealed that the complexity of organizational context and characteristics of the evaluation policy itself were operative. At the meso and micro levels, moderating variables were associated with the organizational capacity to do evaluation and to use it, respectively. Having a supportive evaluation system and required resources helps to foster capacity to implement and/or oversee evaluation studies at the organizational level. To the extent that the organizational culture is conducive to evaluation and leaders were inclined to promote it, capacity building would be enhanced. These variables influence and are influenced by evaluator roles and skills and their relations with other organizational members at the stakeholder level.

Organizational capacity to use evaluation, on the other hand, is enhanced as evaluation becomes more integrated into the decision-making processes and the purposes of evaluation extend well beyond meeting accountability demands. Specifically, embracing a learning orientation with regard to evaluation use is likely to be a powerful means for organizations to develop capacity. At the stakeholder level, stakeholder engagement and learning from evaluation processes and results are salient elements of building the capacity for use. We now turn to a discussion of these findings and their implications moving forward.

Discussion

In this section, we discuss three significant themes informed by the findings of this research pertaining to evaluation policy’s role in developing organizational capacity for evaluation: (1) evaluation context, (2) evaluation purposes (accountability vs. learning), and (3) leadership. These three themes interconnect with, and arise from, those emerging from the interview findings. We conclude by considering some limitations of our study and the implications for ongoing research.

Evaluation Context

Over the course of this empirical study, the undeniable role of context and its influence on the relationship between evaluation policy and organizational capacity for evaluation became increasingly evident. The sociopolitical, cultural, and economic contexts of an organization are critical and pervasively influence evaluation practice. Context is also the most salient, influential consideration shaping the ECB process. Many scholars have argued that understanding an organization’s context can generate insights that are important for explaining how the interaction between hierarchies, systems, structures, and people influences evaluation processes and use within organizations (Chelimsky, 2012; Coldwell, 2019; Vo, 2013; Vo & Christie, 2015).

Organizations considering the development and installation of evaluation policy would benefit from a thorough contextual analysis. First, such analysis can identify where gaps exist in the organizational capacity to implement evaluation, including in human resources, organizational, and administrative systems. Second, it can help illuminate why these gaps exist in relation to sociopolitical, economic, and cultural factors, as well as specific statutory and regulatory systems that constrain or enable capacity-building efforts. Evaluation policies based on the premise that capacity building is essentially a matter of “replicating best practices” regardless of organizational context have little chance of being effective.

Evaluation Purpose: Accountability and Learning

Study participants almost uniformly insisted that evaluation policies are mostly symbolic and accountability-driven and that the requirements of evaluation policies add to the existing tension between the accountability-oriented needs of funders and the learning needs of other evaluation stakeholders (e.g., program developers, managers, and organization staff). They argue that a prevailing emphasis on accountability can have considerable adverse effects on the evaluation function precisely because it undermines the learning value of evaluation. Organizations that are only producing evaluations to satisfy the funders’ needs and leverage ongoing funding unwittingly increase the incentive for “strategic misrepresentation” of evaluation findings (Lehtonen, 2014). As such, evaluation becomes a time-consuming, menial task likely to be irrelevant for internal use within the organization (Ebrahim, 2005).

This tension between accountability and learning as competing justifications for evaluation has been a subject of interest for a number of scholars in recent years because of its significant impact on evaluation practice (Chouinard, 2013; Feinstein, 2012; Guijt, 2010). Some evaluation theorists assert that “accountability-based” and “learning-based” evaluations are qualitatively different (Armytage, 2011; Lennie & Tacchi, 2014). The question for many is how to reconcile the imbalance of these two forces, especially in the face of seemingly ever-increasing demands for accountability (Armytage, 2011; Lennie & Tacchi, 2014; Regeer et al., 2016). This is not an either/or question. It is a question of redressing the imbalance and highlighting the potential power of learning-oriented evaluation; to a substantial degree, the enhancement of organizational capacity for evaluation depends on it. By implication, there is a role for evaluation policy in reconciling this tension.

Organization Leaders’ Evaluation Capacity

We learned about the indispensable role of leadership in fostering the capacity to do and use evaluation. Leadership must be seen as supporting and promoting evaluation in order for policy to leverage organizational capacity for evaluation. Leaders can do this by: (i) understanding how strategy and evaluation are interconnected (or should be interconnected); (ii) ensuring adequate resources for evaluation; (iii) being active consumers of evaluation information; and (iv) using evaluation as a means for ongoing organizational learning (Preskill, 2014). Leaders should place a high value on evaluation by consistently seeking evaluative information for planning, implementation, and decision-making processes—in other words, leaders have to walk the talk (Norman, 2002).

Within our ecological framework, leadership not only moderates the role of evaluation policy, but it serves to link the capacity to do evaluation and the capacity to use, particularly at the organizational level. Leadership emerges as a variable critical for supporting or leveraging the capacity to plan and implement evaluation effectively, on the one hand, while being responsible for integrating evaluation into decision-making processes, on the other. Leaders (especially senior organizational leaders) have the power and the influence to support the ongoing development of organizational capacity for evaluation.

As such, the implications for leadership development are significant. Yet, as is demonstrated by the findings of Labin et al. (2012) regarding ECB, “leadership was the least frequently targeted organizational factor and the least frequently reported organizational outcome” (p. 321). At the same time, as Preskill (2014) noted, we have not “paid enough attention to the role senior leaders play in organizations, and how they influence, shape, and sustain an evaluation and learning culture” (p. 117). Our research underscores the need to delve more deeply into leadership as a moderating variable than is presently the case.

Limitations of the Study

The context within which this research was undertaken entails three noteworthy limitations on the interpretation of its findings. First, the design of the study leaves the findings and interpretations open to researcher bias. To address this, we kept systematic records of all data collection sources and efforts. Further, our analyses were informed by previous scholarship, and the conclusions are linked to extant contributions.

A second limitation is that our findings are somewhat evaluation-centric. Those who participated in the interviews were evaluation scholars and practitioners who have made significant contributions to the advancement of the evaluation field. The study could have benefitted from a more deliberate inclusion of evaluation users, such as organization managers and senior decision makers, to ensure the integration of multiple points of view. Nevertheless, we can say with confidence that the findings of this study represent points of view of participants from many different regions and backgrounds, which touches on a third limitation.

Our data emerged from interviews with professionals located in Canada, the United States, and Europe. While we have highlighted the importance of context, a comparison of, for example, North American versus European perspectives was beyond the scope of the present study, and our data do not provide latitude to support such comparative analyses. We note that several participants, although located in North America or Europe, work in international development. In the literature, there is some evidence to show that global Western approaches to evaluation are pervasive, not only in the West but also in international development contexts (Chouinard & Hopson, 2016). Still, it would be beneficial for ongoing inquiry to look more closely at organizational context and its implications for evaluation.

Implications for Research

This exploratory research breaks new ground in the ECB domain of inquiry. As a field, we have developed a reasonably deep understanding of what constitutes effective ECB. Our research takes the next step with this knowledge by situating it within the context of evaluation policy influence. We hope that the evidence-based ecological framework we developed (see Figure 1) will serve to guide ongoing inquiry in this area. Important research questions—not the least of which is the role of context in moderating the influence of policy on organizational evaluation capacity—might arise from the constructs identified in Figure 1 and the suggested relationships among them.

Our understanding of evaluation policy could be enhanced by more conceptually informed investigations. Thus far, much of the research on evaluation policy has focused on aspects of evaluation practice that these policies might be used to govern. Findings from the present research suggest that the field may benefit from continued probing into how specific aspects of evaluation practice interact with various features of evaluation policy and how such interaction influences ECB. For instance, how does evaluation policy affect evaluation design? How does it affect ECB activities? How does it affect stakeholders’ level of engagement?

Another implication for research is methodological. We used qualitative interviews as the primary means of data collection in this study. Future research might benefit from using mixed and/or quantitative methods to test the variables in the ecological framework, thereby providing a different level of analysis. For example, questionnaire survey methods could yield interesting findings in terms of weighting the importance of variables, understanding interrelations among them, and providing insights into the existence of other variables. Particularly rewarding might be studies featuring stakeholder perceptions about these variables or even comparisons with those of evaluators. Alternatively, case studies of varying organization types would provide a deeper understanding of what evaluation policy means in practice.

In conclusion, the findings presented here serve as an important first step in broadening our knowledge of the impact of evaluation policy on evaluation practice. If we agree that ECB has an explicit goal of developing organizational capacity for evaluation, it is important that we expand and deepen our knowledge about ECB processes by situating them within a broader context of evaluation policy.

Footnotes

Authors’ Note

This study was part of the doctoral dissertation of the first author completed at the University of Ottawa. The authors are grateful to thesis committee members for their input.

Acknowledgments

The authors thank the Saudi Cultural Bureau and the University of Ottawa for support provided during the research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Hind Al Hudib

Notes

References

Armytage

(2011). Evaluating aid: An adolescent domain of practice. Evaluation, 17(3), 261–276. https://doi.org/10.1177/1356389011410518

Bourgeois

Cousins

J. B.

(2008). Informing evaluation capacity building through profiling organizational capacity for evaluation: An empirical examination of four Canadian federal government organizations. Canadian Journal of Program Evaluation, 23(3), 127–146.

Bourgeois

Cousins

J. B.

(2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299–319. https://doi.org/10.1177/1098214013477235

Bourgeois

Hotte

Simmons

Osseni

(2016). Measuring evaluation capacity in Ontario public health units. Canadian Journal of Program Evaluation, 31(2), 165–183. https://doi.org/10.3138/cjpe.306

Bourgeois

Toews

Whynot

Lamarche

M. K.

(2013). Measuring organizational evaluation capacity in the Canadian Federal Government. Canadian Journal of Program Evaluation, 28(2), 1–19.

Boyle

Lemaire

Rist

R. C.

(1999). Introduction: Building evaluation capacity. In Boyle

Lemaire

(Eds.), Building effective evaluation capacity: Lessons from practice (pp. 1–19). Transaction.

Chelimsky

(2009). Integrating evaluation units into the political environment of government: The role of evaluation policy. New Directions for Evaluation, 123, 51–66. https://doi.org/10.1002/ev.305

Chelimsky

(2012). Valuing, evaluation methods, and the politicization of the evaluation process. New Directions for Evaluation, 133, 77–83. https://doi.org/10.1002/ev.20008

Chouinard

(2013). The case for participatory evaluation in an era of accountability. American Journal of Evaluation, 34, 237–253. https://doi.org/10.1177/1098214013478142

10.

Chouinard

Hopson

(2016). A critical exploration of International Development Evaluation. Canadian Journal of Program Evaluation, 30(3), 248–276. https://doi.org/10.3138/cjpe.30.3.02

11.

Coldwell

(2019). Reconsidering context: Six underlying features of context to improve learning from evaluation. Evaluation, 25(1), 99–117. https://doi.org/10.1177/1356389018803234

12.

Cousins

J. B.

Bourgeois

(2014). Cross-case analysis and implications for research, theory, and practice. New Directions for Evaluation, 141, 107–125. https://doi.org/10.1002/ev.20078

13.

Cousins

J. B.

Elliott

Amo

Bourgeois

Goh

Lahey

(2008). Organizational capacity to do and use evaluation: Results of a pan-Canadian survey of evaluators. Canadian Journal of Program Evaluation, 23(3), 1–35.

14.

Cousins

J. B.

Goh

S. C.

Clark

Lee

L. E.

(2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.

15.

Dahler-Larsen

(2006). Organizing knowledge: Evidence and the construction of evaluative information systems. In Rist

R. C.

Stame

(Eds.), From studies to streams: Managing evaluative systems (pp. 65–80). Transaction.

16.

Datta

L.-e.

(2009). Golden is the sand: Memory and hope in evaluation policy and evaluation practice. New Directions for Evaluation, 133, 33–50. https://doi.org/10.1002/ev.304

17.

Dillman

Christie

(2017). Evaluation policy in a non-profit foundation: A case study exploration of the Robert Wood Johnson foundation. American Journal of Evaluation, 31(1), 60–79. https://doi.org/10.1177/1098214016642864

18.

Ebrahim

(2005). Accountability myopia: Losing sight of organizational learning. Nonprofit and Voluntary Sector Quarterly, 34(1), 56–87. https://doi.org/10.1177/0899764004269430

19.

Feinstein

O. N.

(2012). Evaluation as a learning tool. New Directions for Evaluation, 35, 103–112. https://doi.org/10.1002/ev.20022

20.

Guijt

(2010). Accountability and learning: Exploding the myth of incompatibility between accountability and learning. In Ubels

Acquaye-Baddo

N. A.

Fowler

(Eds.), Capacity development in practice (pp. 277–291). Earthscan.

21.

Labin

Duffy

J. L.

Meyers

D. C.

Wandersman

Lesesne

C. A.

(2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338. https://doi.org/10.1177/1098214011434608

22.

Leeuw

F. L.

(2009). Evaluation policy in the Netherlands. New Directions for Evaluation, 123, 87–102. https://doi.org/10.1002/ev.307

23.

Leeuw

F. L.

Furbow

(2008). Evaluation systems: What are they and why study them? Evaluation, 14(2), 157–169. https://doi.org/10.1177/1356389007087537

24.

Lehtonen

(2014). Evaluating megaprojects: From the “iron triangle” to network mapping. Evaluation, 20(3), 278–295. https://doi.org/10.1177/1356389014539868

25.

Lennie

Tacchi

(2014). Bridging the divide between upward accountability and learning-based approaches to development evaluation: Strategies for an enabling environment. Evaluation Journal of Australasia, 14(1), 12–36.

26.

Levin-Rozalis

Rosenstein

Cousins

(2009). A precarious balance: Educational evaluation capacity building in a globalized society. In Ryan

K. E.

Cousins

J. B.

(Eds.), The Sage international handbook of educational evaluation (pp. 191–212). Sage.

27.

Liverani

Lundgren

(2007). Evaluation systems in development aid agencies: An analysis of DAC peer reviews 1996–2004. Evaluation, 13(2), 241–256. https://doi.org/10.1177/1356389007075226

28.

Mark

M. M.

Cooksy

L. J.

Trochim

W. M. K.

(2009). Evaluation policy: An introduction and overview. New Directions for Evaluation, 123, 3–11. https://doi.org/10.1002/ev.302

29.

Nacarrella

Pirkis

Kohn

Morley

Burgess

Blashki

(2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30(3), 231–236. https://doi.org/10.1016/j.evalprogplan.2007.05.001

30.

Nielsen

S. B.

Lemire

Skov

(2011). Measuring evaluation capacity: Results and implications of a Danish study. American Journal of Evaluation, 32(3), 324–344. https://doi.org/10.1177/1098214010396075

31.

Norman

(2002). Managing through measurement or meaning? Lessons from experience with New Zealand’s public sector performance management systems. International Review of Administrative Sciences, 68, 619–628.

32.

Preskill

(2014). Now for the hard stuff: Next steps in ECB research and practice. American Journal of Evaluation, 35(1), 116–119. https://doi.org/10.1177/1098214013499439

33.

Preskill

Boyle

(2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214013499439

34.

Regeer

B. J.

Wildt-Liesveld

R. D.

Mierlo

B. V.

Bunders

J. F.

(2016). Exploring ways to reconcile accountability and learning in the evaluation of niche experiments. Evaluation, 22(1), 6–28. https://doi.org/10.1177/1356389015623659

35.

Saldana

(2009). The coding manual for qualitative researchers. Sage.

36.

Sanders

J. R.

(2003). Mainstreaming evaluation. New Directions for Evaluation, 99, 3–6. Jossey Bass

37.

Stern

(2009). Evaluation poolicy in the European Union and its institutions. In Trochim

W.M.K.

Mark

M. M.

Cooksy

L. J.

(Eds.) Evaluation policy and evaluation practice. New Directions for Evaluation, 123, 67–85. doi: 10.1002/ev306

38.

Stockdill

S. H.

Baizerman

Compton

(2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 93, 7–25.

39.

Taylor-Powell

Boyd

H. H.

(2008). Evaluation capacity building in complex organizations. New Directions for Evaluation, 120, 55–69. Wiley Periodicals. https://doi.org/10.1002/ev.276

40.

Trochim

W. M. K.

(2009). Evaluation policy and evaluation practice. New Directions for Evaluation, 123, 13–32. https://doi.org/10.1002/ev.303

41.

A. T.

(2013). Visualizing context through theory deconstruction: A content analysis of three bodies of evaluation theory literature. Evaluation and Program Planning, 38, 44–52. https://doi.org/10.1016/j.evalprogplan.2012.03.013

42.

A. T.

Christie

C. A.

(2015). Advancing research on evaluation through the study of context. New Directions for Evaluation, 148, 43–55. https://doi.org/10.1002/ev.20156

Understanding Evaluation Policy and Organizational Capacity for Evaluation: An Interview Study

Abstract

Keywords

Introduction

Conceptual Background

The Nature and Consequences of Evaluation Policy

Evaluation Policy as Part of the Evaluation System

Organizational ECB

Organizational Capacity for Evaluation

Method

Study Participants

Interview Methods

Analysis

Findings

Defining Evaluation Policy

Defining Organizational Evaluation Capacity

Context

Organizational Culture

Evaluation System

Policy Content and Type

Perceived Purpose: Learning Versus Accountability

Organizational Leadership

Evaluator Role and Skill

Stakeholder Engagement

Integration and Use

Summary and Integrative Conceptualization

Discussion

Evaluation Context

Evaluation Purpose: Accountability and Learning

Organization Leaders’ Evaluation Capacity

Limitations of the Study

Implications for Research

Footnotes

Authors’ Note

Acknowledgments

Declaration of Conflicting Interests

Funding

ORCID iD

Notes

References