Sage Journals: Discover world-class research

Abstract

The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession. This article investigates how the terms are defined and understood by different institutions and communities. It systematically investigates representative sets of definitions, analyzing them to identify 16 distinct defining elements. This framework is then used to compare definitions and assess their usefulness and limitations. Based on this assessment, the article proposes a remedy in three parts: applying good definition practice in future definition updates, differentiating causal perspectives and using appropriate causal language, and employing meaningful qualifiers when using the terms outcome and impact. The article draws on definitions used in international development, but its findings also apply to domestic public sector policies and interventions.

Keywords

evaluation definitions development interventions outcomes impacts causality

Introduction

The terms “outcome” and “impact” are ubiquitous in development evaluation discourse and in program documents of development interventions. However, there are many competing definitions and uses for these terms that cannot easily be reconciled. This leads to confusion, undermines efforts to improve learning and accountability, and may negatively affect the effectiveness of public programs and policies. In international development, the most widely used and internationally accepted definitions are those of the Organization for Economic Co-operation and Development (OECD, 2002) in which outcomes are defined as short- and medium-term changes and impacts as longer term changes resulting from a development intervention. We show in this article that these influential definitions, and many other definitions by different organizations and communities, are ambiguous and lack the conceptual clarity and precision needed for understanding, planning, and evaluating the complex development interventions, programs, and policies of our time.

We demonstrate that a common definitional weakness is the use of an “intervention perspective” and related language that suggest that outcomes and impacts can be fully attributed to an intervention. Because, as we show later, this assumption is unrealistic for most development efforts, this definitional weakness can drive unrealistic expectations and reduce the effectiveness of programs attempting to cope with them. This is particularly worrisome in view of heightened attention to impact evaluation and result-based management (RBM) intended to increase the effectiveness of aid through learning, improved decision-making, and strengthened accountability (Mayne, 2007; Vähämäki, Schmidt & Molander, 2011).

While there is widespread agreement in the evaluation community about the unsatisfactory state of affairs regarding outcome and impact terminology, there is only isolated systematic analysis of the problem (e.g., Economic Cooperation Group [ECG], 2012; Hearn & Buffardi, 2016; see Supplemental Material, available with the online version of this article) and no comprehensive approach, based on such analysis, toward solving or improving the situation. The most promising recent attempt seems to be the 2011 United Nations Development Group (UNDG, 2011) publication of a set of harmonized definitions intended to be used across the entire United Nations (UN) system. However, those definitions do not live up to the good practices and recommendations developed in this article and have not been adopted widely across the international development community. A year later, the ECG, representing the evaluation units of the major multilateral development banks and other international financial institutions, issued its own set of definitions (ECG, 2012), much shorter,¹ with a minimal consensus regarding the term impact (as discussed later) and important differences in meaning from UNDG’s, for example, regarding the causal perspectives adopted. Clearly, the unsatisfactory state of affairs regarding outcome and impact terminology has not yet been—but urgently needs to be—resolved. Because achieving international consensus on a new set of sounder definitions may be challenging, advice is needed to mitigate problems associated with current definitions.

The findings and recommendations of this article are not restricted to international development. The OECD and similar definitions are used by public sector planners and evaluators, domestic and international, and around the globe. To do planning, evaluation, and RBM well and to enable improved understanding of how public programs and policies make a difference, we need conceptual clarity about the nature of and the relationships between the results associated with them.

After this introduction, we briefly introduce our terminology in the second section. In the third section, we systematically review the influential OECD definitions and those of several other international development organizations and then summarize key observations from the analysis of the overall set in the fourth section. In the fifth section, we provide recommendations to address identified shortcomings, with brief concluding remarks in the sixth section. All definitions referred to in this article are provided in the Supplemental Material.

Terminology

Before characterizing outcomes and impacts, we need to clarify our own working terms and concepts.

“Intervention” describes a deliberate involvement in a process or system intended to influence events and/or consequences. The term may refer to single activities but often refers to sets of activities organized within a project, program, or instrument. We use the term “development intervention” or simply “intervention” to refer to the activities of a project, program, or instrument in the field of international development.

“Change” refers to any event or variation in the state of affairs. Change may happen at any point in time or place and may or may not be causally related to an intervention.

We use the terms “contribution” and “attribution” to ascribe a causal relationship between an intervention and a change. In most cases, the intervention is only one of many factors with causal influence on the change. Hence, we use expressions such as “the intervention has contributed to the change” or “the change can be attributed to the intervention” interchangeably and without implying that the intervention was sufficient to cause all the change. This is consistent with other common uses of the term attribution. For example, International Initiative for Impact Evaluation (3ie) defines attribution as the extent to which an observed change results from an intervention, clearly implying that the change could be only partially caused by the intervention.

In contrast, if we wish to express an exclusive causal relationship in which there are no other contributing causal factors, we say that the change can be “fully attributed” to the intervention. Full attribution is a theoretical simplification of a more complicated reality in which observed change depends on many contributing factors beyond an intervention, program, or policy.

We understand a “causal perspective” as the viewpoint from which interventions and changes are observed or described. We differentiate between two principal causal perspectives that exist independent of how causality is defined or measured:

In the intervention perspective, changes produced by an intervention are described. As illustrated in Figure 1, this perspective sees the intervention at the origin of the causal system and is interested in the results and effects it causes. The intervention perspective is reflected in log frames, results chains, and impact pathways that start with intervention activities and then explain successive layers of results. The intervention perspective does not assume that the intervention causes all of the change. Rather, it is only interested in the portion of change caused by the intervention. The same holds when several interventions are described. However, language used in this perspective expresses ownership of results by the intervention and can easily be interpreted to imply full attribution (Table 1).

In the “system perspective,” the viewpoint shifts from the intervention to the changes of interest. Starting from those changes, it traces back to different contributing causes (Figure 2). Some contributing causes may be related to the intervention(s) while others may not. In the system perspective, the existence and degree of influence the intervention has on changes remains of central interest, but full or even partial attribution are usually not implied, not even to a proportion of observed change. In other words, one or a set of interventions contribute to observed change but are usually not understood to represent necessary or sufficient causes for it. System perspective language is nonproprietary and more generally applicable but less intuitive when describing strong causal relationships (Table 1).

Figure 1.

Intervention perspective.

Table 1.

Terminology for Different Causal Perspectives in Outcome and Impact Definitions.

	Intervention Perspective	System Perspective
Domain	Strong causal linkages between the intervention and the effect: the intervention is dominant and largely sufficient for producing the effect.	Weak causal linkages between the intervention and the effect: the intervention is one (distant) contributing cause among many.
Suggested terminology	– Full attribution– Dominant cause– Results and effects of an intervention– Results and effects produced by an intervention– Results and effects (fully) attributed to an intervention	– Contribution– Contributing causes, causal package– Change to which the intervention contributed– Change that the intervention influenced– Contributing causes unrelated to the intervention

Figure 2.

System perspective.

Parsing Internationally Accepted Definitions

This section reviews and analyses several sets of widely used definitions of outcome and impact (see Supplemental Material) and identifies and discusses the defining elements these definitions employ. Overall, we have identified 16 distinct elements across all definitions that can be grouped into five categories (Table 2).

Table 2.

Defining Elements in Definitions of Outcome and Impact.

Group	Defining Element	Explanation	Values
Causality related	Causal perspective	Causal perspective (see terminology and Figures 1 and 2)	Intervention perspective, system perspective
Causality related	Causal ordering	The result being defined happens because of earlier results and/or contributes to subsequent results	Yes, no (for each pair of results)
Time related	Temporal ordering	The result being defined happens before or after other results	Yes, no (for each pair of results)
	Time to results	Indication of how long it takes until the result being defined materializes	Long, medium, short
	Longevity of results	Indication of how long the result being defined lasts	Long, medium, and short
Illustrative	Past/future applicability	Definition applies to past and/or future results	Ex ante, ex post, and both
	Beneficiality of results	Definition applies to beneficial and/or nonbeneficial results	Positive, negative, and both
	Intentionality of results	Definition applies to intended and/or unintended results	Intentional, unintentional, and both
	Directness of results	Definition applies to directly reached (primary) or indirectly reached (secondary) results	Direct (primary), indirect (secondary), and both
Results related	Type of results	Description of the functional type of change results (e.g., behavior change)	For example, change of awareness or skill, behavior, or state
	Results level	Using the concept of “levels” to differentiate between results, sometimes without further specification, sometimes referring to the degree of aggregation or reach. Often used together with target group description or relation to purpose.	High(er/est), low(er/est)
	Target group for result	Description of who or what is affected or experiences the result (e.g., rural people)	Description of target group for that result
	Relation to purpose	Using the goal hierarchy of the institution responsible for the intervention to describe the result	Purpose (raison d’être), goal, objective, and target
Observation related	Observation based	Whether the result is defined through observation (instead of also in the absence of observation)	Yes, no
	Observation method	Whether the result is defined by perceived/measured change instead by simply referring to changes (without indication of measurement)	Description
	Observation time	Whether the result is defined by change at the time of measurement instead of simply referring to change (without indication on if and when it is measured)	Yes, no

OECD Definitions

The most widely used terms for the results of development interventions are those presented in the “Glossary of Key Terms in Evaluation and Results Based Management” of the Development Assistance Committee (DAC) Working Party on Aid Evaluation of the OECD (2002), shown in Table 3.

Table 3.

Organization for Economic Co-operation and Development (OECD) Development Assistance Committee Definitions for Outcome and Impact.

Term	OECD Definitions
Outcome	The likely or achieved short-term and medium-term effects of an intervention’s outputs.
Impact	Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, and intended or unintended.

The OECD (2002) defines both outcomes and impacts as “effects” which are in turn defined as “intended or unintended changes due directly or indirectly to an intervention”. The term change itself is left undefined.

Both definitions imply cause–effect relationships and take an intervention perspective that looks for results caused by the intervention. As discussed above, we refer to this defining element as causal perspective.

All other defining elements in these definitions are qualifiers intended to specify the nature of the effects and, ostensibly, to differentiate the two concepts.

One important qualifier for differentiating between outcomes and impacts could be the defining element of causal ordering. For example, outcomes could be defined as causes of impacts and, reciprocally, impacts as effects of outcomes. However, neither the outcome nor the impact definition of the OECD makes any explicit statement as to how outcomes and impacts are related causally. Taking the definitions literally, it is left open whether outcomes are required as causal precursors of impacts. The situation is different between outputs and outcomes. The OECD defines outcomes as effects of an intervention’s outputs, so the causal ordering is clearly established. This causal ordering between outputs and outcomes is mirrored in the definition of outputs. From context, it appears that causal ordering between outcomes and impacts is assumed, that is, that each impact can be traced back to one or more outcomes, just as each outcome can be traced back to outputs. The inverse is not necessarily true; some outputs may not produce outcomes, and some outcomes may not cause impacts.

Time is the main qualifier used to differentiate between outcomes and impacts in the OECD definitions. Outcomes represent “short- or medium-term effects” and impacts “long-term effects.” Two observations can be made here.

First, it is unclear whether the definitions refer to the time to onset, to the longevity of effects, or to a combination of both. In the first case, outcomes and impacts would be strictly time ordered with respect to their onset, that is, to when they first occur. If this understanding is correct, the reference points are inconsistent between definitions. For outcomes, the time between production of outputs and outcomes is relevant; but for impacts, the time between the intervention and the onset of impacts counts. When interpreting time qualifiers as time to onset, we describe the defining element as time to results. Alternatively, if we interpret time qualifiers as longevity of effects, outcomes are defined by having short- or medium-term life spans while impacts persist for the long term, somewhat akin to the sustainability criterion. In other words, outcomes would refer to passing changes caused by the intervention, whereas impacts would refer to lasting changes. We refer to this defining element as longevity of results. A literal interpretation in this sense would mean that outcomes could occur before or after impacts.

Second, both definitions refer to time but not to absolute timescales. Hence, the terms “short,” “medium,” and “long” are open to individual interpretation. However, relative ordering can be safely inferred. If we understand the definitions to imply time to result (first case above), outcomes start to occur before impacts. In terms of longevity of results (second case), the duration of impacts is longer than that of outcomes. In both cases, we speak of relative temporal ordering, that is, the comparison of absolute or the use of relative time measures to differentiate between outcomes and impacts. As causes occur before effects in time, the causal ordering discussed above also implies temporal ordering.

The OECD definitions employ four additional defining elements to characterize outcomes and impacts more precisely but still fail to differentiate them as all possible options are allowed for each qualifier.

The OECD definition of outcome makes explicit that it can be applied ex ante, to describe likely future effects, or ex post, to refer to effects that have been achieved. We refer to this defining element as past and future applicability.

The impact definition reminds that the effect may be positive or negative, which illuminates the definition but does not render it more precise. We refer to this element as beneficiality of results, which can take the values “positive” and/or “negative.” No statement on beneficiality of results is made for outcomes, but negative outcomes are not excluded.

The impact definition notes explicitly that effects may be intended or unintended. The outcome definition refers to effects, which are also intended or unintended. Hence, these qualifiers are purely illustrative in both cases. We refer to this element as intentionality of results.

The impact definition refers to primary and secondary effects produced by a development intervention. The definition of the term effects itself describes changes due directly or indirectly to an intervention. We refer to this element as directness of results, taking the values direct or indirect. The impact definition also specifies that the effects may be produced “directly or indirectly.” As there is no other meaning apparent, we assume that this to be redundant and that direct and primary (and indirect and secondary) can be used interchangeably. The definition of outcome does not explicitly address the directness of results but also builds on the term effects, which describes changes as being due directly or indirectly to an intervention.

Overall, outcomes and impacts as defined in the OECD DAC glossary are changes that share many attributes: Both are caused by interventions, apply ex ante and ex post, and can be beneficial or detrimental, intended or unintended, and direct or indirect. The only differentiating elements are that impacts operate on longer timescales than outcomes, though it is not clear whether time to results or longevity of results is referred to. Importantly, causal ordering between outcomes and impacts is not explicit. Clearly, this is not a good basis for unambiguously defining or for distinguishing such important concepts.

The OECD definitions have a central role because many development organizations such as multilateral development banks, UN agencies, bilateral donors and their agencies, and research for development (R4D) organizations adopted them or used them as a starting point for developing their own adapted definitions. The German Federal Ministry for Economic Cooperation and Development (BMZ), for example, adopted (translated) OECD DAC definitions with annotations regarding type and level of results, target groups, and relationship to intervention purposes (BMZ, 2006, p. 9). These represent additional defining elements that are introduced below.

Other Definitions and Additional Defining Elements

Many organizations and groups have developed their own sets of definitions (see Supplemental Material). Similarities and differences along all defining elements (see Table 2) are indicated in Tables 4 and 5 for outcomes and impacts, respectively. Below, we highlight defining elements in several important, internationally accepted alternatives to the OECD definitions.

Table 4.

Use of Defining Elements by Outcome Definition.

Group	Defining Element	OECD	CIDA	ECG	OM	IFAD	UNDG	UNAIDS	USAID	3ie	NONIE	CGIAR	CIFOR
Causality related	Causal perspective	X	X	X	X	X	X	X	X	X	Item used but not defined	X	X
Causality related	Causal ordering	X	X	X			X					X	X
Time related	Temporal ordering	X	X				X	X	X			X
	Time to results	?	X					?	X			?
	Longevity of results	?	?					?				?
Illustrative	Past/future applicability	X
	Beneficiality of results
	Intentionality of results	X							X			X
	Directness of results	X	X		X
Results related	Type of result				X		X	X					X
	Results level		X	X		X
	Target group		X		X
	Relation to purpose		X			X	X
Measurement related	Observation based									X
	Observation method									X
	Observation time									X

Note. CIDA = Canadian International Development Agency; ECG = Economic Cooperation Group; IFAD = International Fund for Agricultural Development; UNDG = United Nations Development Group; NONIE = Network of Networks on Impact Evaluation; OM = Outcome Mapping; CGIAR = Consultative Group on International Agricultural Research; CIFOR = Centre for International Forestry Research; OECD = Organization for Economic Co-operation and Development.

Table 5.

Use of Defining Elements by Impact Definition.

Group	Defining Element	OECD	CIDA	ECG	OM	IFAD	UNDG	UNAIDS	USAID	3ie	NONIE	CGIAR	CIFOR
Causality related	Causal perspective	X	N/A	X	N/A	X	X	X	X	X	X	X	X
Causality related	Causal ordering	X									X	X
Time related	Temporal ordering	X						X	X		X	X
	Time to results	?					?	?	X		?	?
	Longevity of results	?					?	?			?	?
Illustrative	Past/future applicability												X
	Beneficiality of results	X				X	X		X		X	X
	Intentionality of results	X				X	X		X		X	X	X
	Directness of results	X					X				X	X	X
Results related	Type of result						X	X				X	X
	Results level			X		X		X	X
	Target group					X	X
	Relation to purpose					X	X	X
Observation related	Observation based					X				X
	Observation method					X				X
	Observation time					X				X

Canadian International Development Agency (CIDA)

The definitions by the CIDA (2008), now Global Affairs Canada, take a fundamentally different approach. Impact is not defined at all, but, instead, three levels of outcomes are distinguished and defined.

The CIDA definitions employ several of the defining elements discussed above but also add four defining elements not found in the OECD definitions:

First, each outcome level is defined in part by the nature of the change engendered: Immediate outcomes relate to increases in awareness and skills among beneficiaries, intermediate outcomes are manifest as changes of behavior or practice among beneficiaries, and ultimate outcomes take the form of a sustainable change of state among beneficiaries. We refer to this as functional type of result.

Second, each definition specifies “beneficiaries” as the unit of analysis. We refer to this as target group for results.

Third, the definitions specify that ultimate outcomes “usually represents the raison d’être of an organization, policy, program, or initiative,” that is, the relationship of the result to the intervention’s (or organization’s) purpose. We refer to this as relationship to purpose.

Finally, all definitions make use of the term “level” which, without further specification, remains rather vague. We refer to this defining element as results level.

We find CIDA’s use of functional result types to differentiate between immediate outcomes (e.g., changes in awareness and skills) and ultimate outcomes (change of state) to be particularly useful as a differentiating element.

CIDA’s definitions also exhibit several weaknesses. For example, they restrict all three levels of outcomes to changes among beneficiaries and suggest strict ordinal relationships among the different levels of outcomes, which may not hold for interventions in complex systems with feedback loops, time lags, and different rates of change. In practical terms, it may be difficult to distinguish outcomes sequentially. And the qualification that a change of state must be “sustainable” adds another definitional challenge; it is usually not possible to be certain about whether such changes will be sustained or not.

Similar to CIDA, other groups and approaches also focus their definitions on outcomes and either consider impact as part of outcome or don’t define and use the term at all:

Evaluation Cooperation Group (ECG): The ECG, representing the evaluation units of major multilateral development banks and other international financial institutions established in 1996, issued its own set of definitions (ECG, 2012). ECG’s definition of outcome builds on the concept of a “results chain,” takes an intervention perspective, and uses causal ordering to differentiate between outcomes and outputs. ECG defines impacts as higher level outcomes and remarks that the term is avoided in its public sector guidance. ECG’s definitions are minimalistic in the sense that they do not use other defining elements.

Outcome mapping (OM) likewise eschews any reference to impact and focuses on defining outcome, with three defining elements: target group for results; functional results types such as changes in behavior, relationships, activities or actions; and directness of results (Earl, Carden, & Smutylo, 2001). Notably, in contrast to other definitions, directness of results is not used for illustration but is an essential part of the definition. This reflects the focus of OM on changes in “boundary partners.” Also, in contrast to other definitions, no causal perspective or ordering is explicit, but it is understood in the approach that project activities contribute to outcomes. OM also offers a useful conceptual framework which explicitly recognizes that a project or program has progressively declining influence, from a sphere of control (i.e., within the project itself) through a sphere of influence (beyond the project boundary) to a sphere of concern, where the project has no direct influence but contributes to triggering and/or influencing a chain of events leading to observed change.

International Fund for Agricultural Development (IFAD)

The IFAD focuses on the term impact, and outcome is understood as part of impact (Gujit & Woodhill, 2002). The IFAD definition of impact employs several of the defining elements introduced earlier. However, in contrast to the intervention perspective predominant in other definitions, it takes a system perspective by starting with observed changes and then identifying the project as one among several contributing factors. The definition uses beneficiality and intentionality of results in an illustrative fashion, similar to OECD’s definition. It also sharply restricts the target group for results to rural people, employs result-level language, and uses relation to purpose as a defining element. IFAD also introduces a new category of defining elements, clearly stating that impacts are changes perceived by intended beneficiaries. That is, the IFAD definition is observation based; it focuses on results that are subjectively perceived (observation method) and observed at the time of the evaluation (observation time). These attributes can be understood in the context of participatory and democratic approaches to evaluation.

Definitions used in the impact evaluation community

The definitions promoted by the 3ie stand in stark contrast to other definitions that understand outcomes and impacts as sequential changes; 3ie sees them at the same causal or temporal distance from the intervention, with outcomes being the variable(s) and impact the attributable change in that variable (3ie, 2012). This is also reflected in 3ie’s (2012) definition of “impact evaluation” as “A study of the attribution of changes in the outcome to the intervention. […].” The 3ie definitions of outcome and impact therefore have potential to cause confusion including within the impact evaluation community itself. Seeing outcome as a measurement variable rather than an intermediate result largely departs from colloquial understanding of the term and may therefore be perceived as counterintuitive and confusing by professionals not using 3ie’s set of definitions.

Moreover, within the impact evaluation community, the 3ie definitions apply mainly to experimental and quasi-experimental designs based on explicit counterfactual thinking. As such, outcomes and impacts as understood by 3ie are observation based, linked to certain observation methods and to the time of observation. Other approaches to impact evaluation such as contribution analysis, process tracing, qualitative comparative analysis, and theory-based evaluation designs (e.g., Stern, Stame, Mayne, Forss, & Befani, 2012) do not necessarily employ explicit counterfactual logic for causal inference and do not introduce observation-based definitions.

In contrast, the Network of Networks on Impact Evaluation explicitly retains the OECD DAC definition of impact and also, by inference, of outcome (Leeuw & Vaessen, 2009). However, issues with using OECD DAC definitions in the context of impact evaluation have been highlighted (e.g., Caspari & Barbu, 2008).

Definitions used in the R4D community

Conducting research with the explicit aim of contributing to development impact inevitably involves long and complex causal chains ultimately linking research activities to development results on the ground. In R4D, results language needs to cover both the research and the development domain, and the terms outcome and impact have their own meaning in each. In R4D systems, other factors quickly become the dominant causal drivers (e.g., Mayne & Stern, 2013), and hence adopting a system perspective is important. We reviewed two sets of definitions, both drawn from the large international agricultural R4D consortium CGIAR (previously stood for Consultative Group on International Agricultural Research; acronym now used as name only):

CGIAR IEA: The CGIAR Independent Evaluation Arrangement (CGIAR IEA, 2015) definitions are closely based on OECD definitions and UNDG’s proposed harmonized terms for RBM. Compared to the OECD definition, the definition for outcome makes only illustrative changes. It mentions explicitly that effects may be intended or unintended (intentionality of results) but does not state that they may be positive or negative (beneficiality of results). While the definition of outcome does not make direct reference to research, the definition of impact does. It explicitly refers to research and, in contrast to the OECD definition, assumes a system perspective (causal perspective) by stressing contribution. This seems to recognize that full attribution of development impacts to research activities is usually impossible. The same insight is not reflected in the definition of outcome, for which similar attribution issues exist because of the long and complex causal chains linking research activities to development results. The definition of impact also lists a wide range of type of results very similar to those employed in UNDG (2011) proposed harmonized terms. Importantly in the R4D context, the CGIAR IEA glossary also defines “research outcomes,” that is, outcomes in the research domain, shifting the emphasis from the time taken to achieve a result to the mechanism and actors involved. This important addition is however not built upon in—or put into context with—the definitions of outcome and impact.

Centre for International Forestry Research (CIFOR): The CIFOR, one of CGIAR’s 15 international research centers, has adopted a different set of definitions. CIFOR’s definitions build on the functional type definitions provided by OM and CIDA, distinguishing outcome and impact by the type of results they describe. Adapted to R4D results, CIFOR understands outcomes as changes in behavior and institutions resulting from changes in knowledge, attitudes, skills, and relationships. Impact is qualitatively different, defined as “change in flow” or a “change in state,” referring to parameters such as income, poverty status, carbon flows, or forest condition.²

Both sets of definitions adapt OECD, UNDG, and other definitions to the R4D context and focus on outcomes and impacts in the development domain “to which research has contributed” through “a chain of events.” The CGIAR IEA also defines “research outcomes” as the “effects from research outputs applied by intermediary users, for instance, by national partners or international research or development organizations,” which we find useful (see below). The CIFOR definitions are useful as they functionally specify the types of development changes that are expected, within a system that aims to contribute to improved livelihoods and natural resource conditions. This kind of framing is needed for research that deliberately engages in social processes (Belcher, Rasmussen, Kemshaw, & Zornes, 2016). Still, there is a large proportion of academic research, and indeed of research done within CGIAR, that uses traditional academic pathways where the main outputs are improved methods, theory, and empirical understanding, which are then employed in further work to advance knowledge. Results language needs to accommodate these kinds of outcomes within the research domain as well. We believe that a useful way forward is to distinguish academic/scientific impact pathways, where the main intended users are other researchers and the main intended outcomes are improved knowledge and improved science, from research-for-development impact pathways, which must be assessed in terms of contributions to social, economic, and/or environmental change. Outcomes and impacts for each should be defined according to their purpose. Nevertheless, even in traditional academic pathways, there is a need for explicit theoretical explanation of how research outcomes and impacts relate to the development outcomes and impacts they are expected to contribute to.

Summary Observations

We identified a total of 16 defining elements in the definitions we reviewed. They can be sorted into five groups: two elements relate to causality, three relate to time, four elements are used mainly for illustration, another four relate to the nature of results, and three are observation based (Table 2).

Two elements are relative and describe causal and temporal ordering. They position the result being defined vis-à-vis other results. Thirteen elements are absolute in the sense that they add description to the result being defined without referring to other results. One element, the “levels” of results, is used both in a relative and in an absolute sense.

The application of these defining elements in the various definitions is summarized in Tables 4 and 5 for outcome and impact definitions, respectively.

The following observations relate to which defining elements are used in definitions of outcomes and impacts:

Many organizations and groups use different defining elements in their outcome and impact definitions and not all sets of definitions cover both outcome and impact. Some organizations and groups use many of the 16 elements in their definitions, whereas others use very few.

Causal perspective is the most-used defining element for both definitions, and the main common denominator across all impact definition. All definitions of outcome and impact express a causal perspective. Causal and temporal ordering, time to results, and intentionality of results are also frequently used (7 of 12), followed by longevity and directness of results (6 of the 12). All other elements appear in less than half of all definitions.

The elements time to results and longevity of results are particularly ambiguous. Most usage of time to results and all usage of longevity of results remain vague about the meaning.

The use of defining elements varies considerably across definitions of the same term by different organizations and groups. For outcome, only 3 of the 16 elements are used in more than half of the definitions (4 of the 10 for impact).

Clearly, the 12 sets of definitions reviewed by us show more differences than commonalities in their choice of what defining elements are used. Lacking a common denominator, practitioners are hard pressed to even pinpoint the dimensions in which definitions may or may not differ. Ambiguity at this level is fundamental, and followers of different definitions may even lack a common language to express commonalities and variations. Remembering that the terms in question are widely applied, this is reason for grave concern regarding the effectiveness with which the (often very different) concepts associated with outcome and impact are communicated.

Several additional observations relate to how defining elements are used:

5. The intervention perspective is predominant in definitions of outcome (e.g., “produced by the intervention”; OECD, 2002). In contrast, half of the impact definitions take a system perspective and express this as “changes […] to which the program has contributed” (Gujit & Woodhill, 2002, p. A-6) or “long-term effects resulting from a chain of events to which research has contributed” (CGIAR IEA, 2015, p. 21).

6. The four defining elements that we have termed illustrative are indeed used to that purpose in most definitions of outcome and all definitions of impact. They point out dimensions along which the results being defined can take all possible values. It seems possible that the illustrations used by the OECD definitions (“likely and achieved,” “positive and negative,” “primary and secondary,” “directly or indirectly,” and “intended and unintended”) have inspired several other definitions. However, two of the nine outcome definitions use these elements in a more definitive way to reduce scope:

The OM definition of outcome excludes indirect effects (“changes in […] of […] with whom a program works directly”)³ and

US Agency for International Development (USAID) refer to “more immediate and intended effects.”

7. Several of the sets of definitions fail to differentiate outcomes and impacts. The influential OECD definitions are among the worst in this respect. In contrast, the USAID definitions make the most explicit distinction, using the same structure for each definition and differentiating by time to results.

8. Finally, we note that language suggesting full attribution is explicit in two sets of definitions (CIDA, 2008; USAID, 2009) and implicit in many through ownership language such as “effects of an intervention’s outputs” or “effects produced by a development intervention” in the OECD definitions.

Our analysis of how defining elements are used reveals further reasons for concern.

For most definitions, the use of the term “outcome” implies an intervention perspective, but, in our experience, the term is also broadly applied to changes lacking strong causal linkages to the interventions, program, or policy at hand. This is a serious problem because it encourages an unrealistically oversimplified world view in which the owners of interventions, programs, and policies alone plan, deliver, and have responsibility for all associated change.

Some organizations and institutions chose not to define the term “impact” at all. In our view, this represents a step back—rather than forward and rather than avoiding confusion, it opens the door to unguided individual interpretations without proper reference.

In comparison to these issues, the use of illustrative defining elements in many definitions is relatively harmless but unnecessary.

Toward Coherence

The previous sections have surfaced several serious weaknesses in the various definitions of outcome and impact. In this section, we suggest a remedy in three complementary parts: applying good definition practice, getting causality right, and using additional meaningful qualifiers. Because achieving international consensus on a new set of sound definitions may be challenging, we hope this advice can also mitigate some of the problems associated with current definitions and encourage users to provide precise and accurate definitions of their own terms.

Apply Good Definition Practice

We suggest the following good practice when developing or updating definitions for outcome and impact. In the absence of commonly agreed definitions, authors should also apply this good practice and explicitly define the terms they use in their communications.

Only use only defining elements that add clarity

Some commonly used defining elements are particularly vague, leading to more rather than less confusion. For example, the “levels” of results can be understood along different dimensions, like reach, time, or hierarchy. It does not add clarity to a definition if used without specification. Other elements must be used with care as they appear to have clear meaning at first but which is lost upon closer examination. For example, a direct (or primary) result of an intervention is probably intended to mean that there is no additional causal step between the intervention and the result. However, a more detailed examination of almost any process will reveal additional intermediate causal steps, effectively rendering a direct result indirect. This remark applies to many defining elements and we suggest to carefully balance the pros and cons associated with their use.

We also consider the two measurement-related defining elements to be fundamentally flawed without dismissing the importance of user perceptions when assessing results. However, taking IFAD’s definition of impact literally would lead to the counterintuitive conclusion that there exists no impact if rural people and their partners do not perceive it at the time of the evaluation, even if changes can be demonstrated by other means or will occur later. Similarly, the United Nations Development Programme's (UNDP) definition would imply that there is no impact if effects in the targeted population groups are not identifiable, again confounding change with measured change. We consider it more intuitive and more useful to define outcome and impact as concepts rather than as subjective or time-dependent measurement results.

In contrast, we find that describing functional types of results in definitions holds much potential. In this case, the type of change is characterized concretely, which greatly enhances specificity. This can be done by developing a theory of change with key expected/intended outcomes, including reasonable end-of-program outcomes, that is, outcomes that could reasonably be expected to occur within the time and resource constraints of the intervention under consideration (Belcher, Suryadarma, & Halimanjaya, 2017). On the downside, being too specific about results types, such as “change in HIV infection, AIDS-related morbidity and mortality” (UNAIDS, 2009, p. 3), may reduce the cross-sectoral applicability of definitions.

Do not use defining elements for which all values are possible

It does not sharpen definitions to point out that results can be likely and achieved, positive and negative, primary and secondary, direct or indirect, or intended and unintended. These illustrations are helpful to remind users to consider a range of possibilities and to avoid narrow interpretation of a definition. However, such reminders add no definitional value and would be better placed in accompanying comments rather than in the definitions themselves.

Clearly differentiate outcomes and impacts from each other and clearly define them in relation to other terms

Definitions of the terms output, outcome, and impact are likely to be most useful if they are mutually exclusive and collectively exhaustive, that is, covering all types of changes related to a development intervention without overlap.

The relations between these terms should therefore be made explicit. We suggest to explicitly mention boundary terms in the definitions and to use causal and temporal ordering to describe their relationships, acknowledging that causal relations may not be explicit or sufficient. Hence, outcomes should be defined in relation to both outputs and impacts, and impact definitions should relate to outcomes (and, of course, outputs should relate to outcomes as well).

In addition, an overarching framework illustrating these relations would be useful. An example for this is the monitoring framework of the Food and Agriculture Organization (FAO, 2016) that carefully summarizes conditional causal linkages in a systems perspective:

if particular FAO products/services are completed as planned, then the Output will be delivered; if the Outputs are delivered and the assumptions hold true, then that should lead to the desired Outcome; if the Outcomes are achieved, then the conditions are in place to result in the Objective’s development impact. (p. 42)

Get Causality Right

We have shown that the causal perspective taken is the single most frequently used defining element. We also consider this element to be useful for conceptual differentiation between outcome and impact, something we suggest future definitions should follow.

We consider an intervention perspective (Figure 1) and intervention language (Table 1, first column) to be useful for describing changes that have strong causal links with the intervention, that is, for which the intervention is the principal and dominant cause. In this domain, simplified cause–effect thinking can be applied, overlooking the complexities of the underlying causal system. In this usage, expressions such as “the effects produced by the intervention,” “the results of an intervention,” and “the change can be fully attributed to the intervention” have clear meaning.

However, most development interventions can only be grasped from a system perspective (e.g., Mayne & Stern, 2013). Beyond the output level, the intervention usually ceases to be the principal and dominant cause for change and other contributing causes and conditions begin to exert superior influence. In other words, the intervention is not sufficient to produce the result. In this domain, intervention-perspective language becomes ambiguous or erroneous. Referring to “the effects produced by the intervention,” “the results of an intervention,” or (fully) attributing changes to an intervention can easily be misunderstood to imply that the intervention is the principal and dominant cause of change, which it is not. Hence, a system perspective (Figure 2) and system language (Table 1, second column) should be used when defining or describing effects further removed from the intervention.

We therefore suggest that:

Impact should always be defined from a system perspective, with appropriate language (Table 1, right column). We consider this an important and urgently needed clarification that will help to communicate the impossibility of fully attributing changes to distant and weakly linked causes.

Definitions of outcome should clearly include effects across the entire crossover region between dominant and weak causal linkages. Outputs, as defined by the OECD (see Note 3), clearly fall into the domain of strong causal linkages and can be adequately described with intervention perspective language while impacts, as suggested above, clearly fall into the domain of weak causal linkages. Outcomes lie in-between. While it is tempting to simply suggest that the term outcome be restricted to a strong causal linkage domain to clearly differentiate it from impact, we believe that such definitions would be overly restrictive vis-à-vis current usage and understanding. For example, the three CIDA definitions for immediate, intermediate, and ultimate outcomes clearly cover both domains. We consider CIDA’s differentiation of outcome useful, but it could be further improved by making the domain crossover more explicit in content and language used.

We also note that use of perspectives and language is not symmetric between outcomes and impacts: While intervention language is ambiguous in a weak causal linkage domain, system language is always correct, even if it may require some getting used to.

Use Meaningful Qualifiers

Considering definitions currently in use, we also suggest improving their practical application; we suggest to use the terms together with meaningful qualifiers and explanations to improve clarity and specificity and to mitigate some of the problems associated with existing definitions.

Several of the defining elements reviewed in this article offer useful language for that purpose (Table 2). Descriptions of the functional types of changes, and of the target groups to which they apply, are very useful, adding clarity and specificity.

In addition, the central importance of causal perspectives and their implications described in the previous section applies to qualifiers for outcome and impact. It is critically importance to clearly specify what types of claims are implied: Is the intervention considered to be the principal and dominant cause for changes, or is it considered one among several contributing causes? As we have pointed out, full attribution and ownership can be claimed only in the first case.

Using qualifiers systematically can also lead to the emergence of more precise, nested definitions as in the definition of immediate, intermediate, and ultimate outcomes by CIDA.

In addition to usage in definitions themselves, we consider result types to be useful qualifiers when using the terms.

Conclusions

We have reviewed several common and influential sets of definitions of outcome and impact and identified 16 defining elements they used. Our assessment shows that most definitions suffer from weak definition practice. Several defining elements have practical use and are employed effectively in some definitions. Other elements are ambiguous in themselves and are used in ways that add little or no clarity. Overall, with most current definitions, it remains unclear what outcomes and impacts are, how they relate to other types of changes, and how they are different from each other.

Our analysis also shows that most definitions use an intervention perspective—and language—that implies linear, sequential, direct, and determinate causal links which are not well-suited to describe changes occurring in the social and ecological systems that international development targets. While useful in some cases, this perspective and the associated language need to be modified to accommodate relationships in more complex systems.

Overall, current outcome and impact terminology reflects substantial ambiguity, internal inconsistency, and conceptual confusion. Different organizations and communities relate different ideas and concepts with the same terms, and the same is true even within single organizations. In our view, this situation is not only a nuisance for policy makers, program staff, and evaluators but also affects human well-being negatively through contributing to unrealistic and ineffective programming and reduced learning from evaluation.

Based on our findings, we have pointed out several ways forward. We recommend applying good definition practice, provide guidance on correct usage of causal perspectives and related terminology, and suggest using the terms together with a set of meaningful qualifiers.

Beyond this, we also suggest to revisit the influential set of OECD definitions. We have demonstrated in systematic detail that the OECD definitions of outcome and impact are not adequate and expect a strong diffusion of improved definitional practice once this core set of definitions has been overhauled. Updated OECD definitions should reflect our recommendations by removing superfluous and ambiguous content and adopting relevant and adequate causal language. It should be noted that definitions of outcome and impact cannot be updated in isolation, as they are interrelated to several other terms. Hence, we also suggest to update several OECD definitions (e.g., output, effect, result, attribution, and counterfactual) and to introduce required additional terms (e.g., contribution, influence, cause, condition, necessary, and sufficient).

Supplemental Material

AJE765698_Supplemental_Material - Outcomes and Impacts of Development Interventions: Toward Conceptual Clarity

AJE765698_Supplemental_Material for Outcomes and Impacts of Development Interventions: Toward Conceptual Clarity by Brian Belcher, and Markus Palenberg in American Journal of Evaluation

Footnotes

Acknowledgments

We thank Stefano D’Errico, Vincent Gitz, John Mayne, Ray Rist, Terry Smutylo, and Michaela Zintl for their helpful comments on drafting this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work of the first author was undertaken, in part, thanks to funding from the Canada Research Chairs program.

Notes

Supplemental Material

The Supplemental Material is available with the online version of this article at .

References

Belcher

Rasmussen

Kemshaw

Zornes

(2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25, 1–17.

Belcher

Suryadarma

Halimanjaya

(2017). Evaluating policy-relevant research: Lessons from a series of theory-based outcomes assessments. Palgrave Commmunications, 3, 1–16.

Caspari

Barbu

(2008). Wirkungsevaluierungen: Zum Stand der internationalen Diskussion und dessen Relevanz für Evaluierungen der deutschen Entwicklungszusammenarbeit (Evaluation Working Papers). Bonn, Germany: Bundesminis-terium für wirtschaftliche Zusammenarbeit und Entwicklung.

Canadian International Development Agency. (2008). Results-based management policy statement. Retrieved from http://international.gc.ca/world-monde/funding-financement/policy_statement_2008-enonce_principe_2008.aspx?lang=eng

CGIAR IEA. (2015). CGIAR standards for independent external evaluation. Retrieved from http://iea.cgiar.org/wp-content/uploads/2016/10/Standards.pdf

Earl

Carden

Smutylo

(2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, Canada: International Development Research Centre.

Economic Cooperation Group. (2012). ECG big book on evaluation good practice standards. Retrieved from https://www.ecgnet.org/document/ecg-big-book-good-practice-standards

Food and Agriculture Organization. (2016). Reviewed strategic framework and outline of the medium term plan 2018–21. Rome, Italy: Author.

The German Federal Ministry for Economic Cooperation and Development. (2006). Evaluierungskriterien für die deutsche bilaterale Entwicklungszusammenarbeit [Evaluation criteria for German bilateral development cooperation]. Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung. Retrieved from https://www.bmz.de/de/zentrales_downloadarchiv/erfolg_und_kontrolle/evaluierungskriterien.pdf

10.

Gujit

Woodhill

(2002). A guide for project M&E. Rome, Italy: Office of Evaluation Studies, IFAD.

11.

Hearn

Buffardi

A. L.

(2016). What is impact? London, England: Methods Lab ODI.

12.

International Initiative for Impact Evaluation. (2012). Impact evaluation glossary (Report No. 7). New Delhi, India: Author.

13.

Leeuw

Vaessen

(2009). Impact evaluations and development: NONIE guidance on impact evaluation. Washington, DC: NONIE.

14.

Mayne

(2007). Best practices in results-based management: A review of experience (A Report for the United Nations Secretariat). New York, NY: UN Secretariat.

15.

Mayne

Stern

(2013). Impact evaluation of natural resource management research programs: A broader view (ACIAR Impact Assessment Series Report No. 84). Canberra, Australia: Australian Centre for International Agricultural Research.

16.

Organization for Economic Co-operation and Development. (2002). Glossary of key terms in evaluation and results-based management. Retrieved from https://www.oecd.org/dac/evaluation/2754804.pdf

17.

Stern

Stame

Mayne

Forss

Befani

(2012). Broadening the range of designs and methods for impact evaluation. London, England: DFID.

18.

UNAIDS. (2009). Glossary: Monitoring and evaluation terms, working draft. Retrieved from http://www.unaids.org/sites/default/files/sub_landing/files/11_ME_Glossary_FinalWorkingDraft.pdf

19.

United Nations Development Group. (2011). Results-based management handbook: Harmonizing RBM concepts and approaches for improved development results at country level. Retrieved from https://undg.org/wp-content/uploads/2016/10/UNDG-RBM-Handbook-2012.pdf

20.

US Agency for International Development. (2009). Glossary of evaluation terms. Retrieved from http://pdf.usaid.gov/pdf_docs/Pnado820.pdf

21.

Vähämäki

Schmidt

Molander

(2011). Review: Results based management in development cooperation. Stockholm, Sweden: Riksbankens Jubileumsfond.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.57 MB