Abstract
Evaluation should be part of the sustainable transition, but the transformational power of evaluation is limited by a short-term focus. This article explores long-term issues, according to evaluation practitioners, when evaluating the consequences of an initiative. A survey (n = 282) followed by discussions with 12 evaluators investigates what is long term in evaluations of impacts and outcomes, and why it is included or not. The results identify 2 to 5 years after the implementation of an initiative as the most common long-term timeframe. The main challenges to including long-term perspectives are missing data, attribution difficulties, and the lack of resources. The main motivations are discovering particular causalities, incorporating contextual elements, and answering the demands of other actors involved in the evaluation. These findings have implications for the transformational power of evaluation: despite challenges and a long-term timeframe that is rather short compared to sustainable development issues, practitioners are positive toward long-term integration.
Introduction
“We are trapped in an age of short-term thinking.”
This is how Krznaric (2020) starts the summary of his thought-provoking book named The Good Ancestor: How to Think Long Term in a Short-Term World. In light of various crises such as climate change, water scarcity, and biodiversity losses, there is tension between the urgency to act now and the long-term impacts of our actions. To make informed decisions in the present, it is important to know what works, what does not, and what the consequences of specific strategies are, both positive and negative, both intended and unintended, and both in the short term and in the long term. Evaluation can create the knowledge base needed to act now for both the present and the future, and this study investigates long-term issues in evaluations of impacts and outcomes.
Evaluation usually involves three main components: it is (1) a value judgment of an initiative (2) based on scientific inquiry that (3) aims at producing knowledge that can be used by decision-makers (Lemire et al., 2020). The act of evaluating a situation, accurate or potential, is carried out every day by everyone without being aware of it (King, 2013), but evaluation has also been institutionalized as a professional activity in the United States from the 1960s (see Alkin (2013b) for the main theorists’ influences and roots), followed in Europe about a decade later (Mertens and Wilson, 2018).
Since evaluation is closely related to politics and societal development (Schwandt, 2015), one could expect that the importance of sustainable development in our current societies is a concern incorporated in evaluation practices while judging the merit and worth of an initiative—especially the worth dimension This is a judgment about the value outside the initiative itself, for the global society, while merit is about judging the initiative based on its own objectives (Schwandt, 2015). In recent years, many publications have offered insights into the role of evaluation in a society that aims to develop sustainably (Van den Berg et al., 2019). The original OECD-DAC evaluation criteria, 1 a referenced framework for evaluation, include a sustainability criterion; however, it has been criticized for only focusing on the continuity of the initiative after the end of the funding (Martinuzzi and Meyer, 2016; Patton, 2019b) and ignoring its impacts on sustainable development. Some evaluation practitioners have highlighted that evaluation should support transitions to sustainability (Julnes, 2019b), and hold a primordial role in promoting sustainable development (e.g. Patton, 2019a, 2021), and Rowe (2019) even states that “incorporating sustainability into evaluation is no longer a choice or moral issue but an imperative” (p. 32).
There is no consensus definition of sustainable development (Martinuzzi and Meyer, 2016) and the aims of the concept evolve over time (Julnes, 2019a), yet Martinuzzi and Meyer (2016) identify three main aspects relevant for evaluation: targets, territories, and time. This last one represents intergenerational integration, therefore the well-being of living entities in the long term. It resonates with the Brundtland definition: meeting the needs of current generations without compromising the ability of future generations to meet their own (World Commission on Environment and Development, 1987). To reach such a target, society has to be transformed. Discussions about the transformational power of evaluation are many, and in this regard, several aspects of “traditional” evaluation have been criticized, such as a silo focus and a project-box focus (Chaplowe and Hejnowicz, 2021; Patton, 2021; Rowe, 2019). One other important aspect that explains evaluation’s struggle to take advantage of its transformational power is the focus on the short term (Chaplowe and Hejnowicz, 2021). This is problematic for several reasons, for instance because short-term successes do not guarantee long-term results (Feinstein, 2019), and an apparent absence of change in the short term can still contribute to reaching a tipping point later (Uitto et al., 2019). Since change processes are various (Forss, 2021) and non-linear (Julnes, 2019b; Magro and Van den Berg, 2019; Uitto et al., 2019), long-term perspectives should be integrated in addition to short-term ones. As Julnes (2019b) points out, expanding the timeframe is required to broaden the scope of evaluations. Thinking long term is also one of the recommendations of Brousselle and McDavid (2020) to improve the contribution of evaluation to social betterment in the anthropocentric era.
Nevertheless, long-term perspectives in evaluation are challenging. It can be complicated to include the interests of future generations in evaluation since their needs and interests change over time and are not necessarily similar to the ones of the current generations (Julnes, 2019b). In addition, human and natural systems have different temporal realities that differ in scope and scales (Rowe, 2019; Uitto, 2019), and “the overriding focus of evaluation is on human systems, whereas its competence to cover natural systems is limited” (Rowe and Uitto, 2023: 125). Moreover, even if long-term perspectives can be included in both retrospective (backward-looking) evaluation and prospective (forward-looking) evaluation (Forss et al., 2021), considering sustainability in prospective evaluation requires questions that need different treatments and analyses than those developed for retrospective investigations which are much more common in evaluation practices (Chelimsky, 2019).
Regardless of the difficulties, evaluating for the public interest requires broadening the scope of evaluation and not only focusing on short-term self-interests through efficiency, as is traditionally done in evaluation, but also on the long-term common good (Julnes, 2019b). The timeframe when looking at the consequences of an initiative can influence the judgment of this initiative a great deal (Hildén, 2009), especially when it comes to its impacts on sustainable development. Despite the challenges, long-term perspectives are possible to integrate and are very useful for various reasons (Forss et al., 2021): to help detect the aggregated effects of policy change, uncover demi-regularities, show multiple and interconnected causal patterns, detect contextual factors, uncover impacts that take more time to appear, and reveal path-dependency. One way to include longer perspectives in evaluation can be to encourage the participation of young people in the evaluation processes (Montrosse-Moorhead et al., 2019). Another can be to learn from different worldviews (Julnes, 2019b). For instance, indigenous evaluation might be more prone to the integration of longer temporality through the seven-generation principles (Julnes, 2019a) and the integration during the evaluation process of considerations for the Rights of Nature which promote a greater focus on long-term resilience (Gallagher and Ofir, 2021).
Various elements can influence the consideration and understanding of the long term since the concept of long term is context dependent. Forss et al. (2021) argue that there is no agreement regarding what is the minimum timeframe required to speak about the long term in evaluation, and they offer a subjective minimum of 10 years. By contrast, in climate change policy, 10-year strategies are considered as short-term targets in comparison to long-term goals, which address climate trends that can be observed over centuries (Hasselmann et al., 2003). Transition management usually requires an approach of at least 25 years (Foxon et al., 2009). In long-term socio-ecological research, which combines human and natural systems, the timeframe can easily be over 50 years, with some studies spanning more than 250 years (Singh et al., 2013).
Since it is unclear what long term can and should be, but it is clear that its integration is needed to enable the transformational power of evaluation to support sustainable development, it is important to explore if the long term is considered in evaluation practices, specifically from the point of view of evaluation practitioners, and if yes, how and why, and if no, why not. Research on evaluation, which is research leading to knowledge about evaluation and not from evaluation, is important to “provide an empirical basis for improving practice and enhancing [. . .] understanding of the types of evaluation most likely to move us toward social betterment” (Henry and Mark, 2003: 70). To put it succinctly, the present study aims at investigating, from the point of view of evaluators, how and why (or why not) long term is considered in evaluations of impacts and outcomes. The evaluation of impacts and outcomes is here defined as the evaluation of the consequences of an initiative. This general research question implies different sub-questions, such as the following: What is considered as long term in evaluations of impacts and outcomes? What are the main reasons that motivate the inclusion of the long term in evaluations of impacts and outcomes? What are the main difficulties encountered when attempting to include the long term in evaluations of impacts and outcomes?
The next section presents the methods of the study. Then come the results, followed by a section discussing them in light of the transformational power of evaluation and suggesting ideas for further research. At the end of the discussion, a conclusion summarizes the main findings of the study.
Methods
The main data collection consisted of a survey addressed to evaluation practitioners all over the world. The survey was constructed with the help of four evaluators who suggested improvements at an early stage, and a test study was conducted before the final survey was launched. About 180 national and international evaluation societies (listed on the IOCE’s VOPE repository) 2 and international evaluation networks were offered to share the survey with their members; 94 of them accepted, and 72 explicitly confirmed sending the invitation. The means for distributing the survey were chosen by the evaluation societies: mailing lists, newsletters, websites, and social media. In addition, some active evaluation practitioners agreed to share the invitation with their networks. It is impossible to know how many evaluation practitioners were potentially aware of the existence of the survey because many practitioners are part of several networks, and some evaluation societies group individual and/or institutional members, as well as members related to evaluation who are not evaluation practitioners themselves. The data collection took place from January 1, 2022 until March 20, 2022. In total, 282 participants responded.
The survey contained general questions about the evaluation experience of the practitioners, such as the number of years of experience and the type of evaluation (backward- or forward-looking) that they have most often worked with, as well as the timeframe they usually consider and the methodologies they usually apply. Then came a main section about the long term in evaluations of impacts and outcomes, with questions such as what is long term, in theory and in practice, what are the main reasons to include it, in theory and in practice, and what are the main challenges encountered, as well as what types of impacts are considered when including long-term perspectives. A third part presented statements about evaluations of impacts and outcomes, the long term, and sustainable development, and asked if the respondents agree or disagree with them. Finally, general background questions about the respondents were asked.
The survey’s structure is described in Supplemental Appendix 1. Since it is an exploratory work, the data are analyzed with descriptive statistics. This gives an indication of the practice of evaluation in general, even though the conclusions cannot be generalized to the total population of evaluation practitioners. All answers have been included, which means that the number of respondents for each question can vary slightly since the latter questions were adapted to the answers given earlier in the questionnaire, and not all questions were compulsory to answer to be able to proceed to the rest of the survey.
Some limitations should be mentioned. The evaluation field is so pluralistic that some survey questions may have been interpreted slightly differently by different respondents, and some questions have been signaled by a few respondents as unclear to them. In addition, several evaluation societies have signaled survey fatigue among their members. Moreover, the survey was offered in English and French, with the purpose of reaching out to as many potential respondents as possible given the resource limitations, and this may have induced some translation issues. This also explains a smaller representation of countries where neither English nor French is a common language. Finally, this exploratory study prioritizes the general understanding of the topic rather than splitting the answers depending on the characteristics of the respondents, such as gender, country of origin, or years of experience. Further research could investigate these aspects.
Afterward, the preliminary results of the survey were presented and discussed with 12 evaluation practitioners who have themselves completed the survey to enlighten and clarify the findings. These individuals, five women and seven men, aged between 45 and 65 years old, are situated in most continents in the world (Africa, Americas, Asia, Europe, and Oceania) and have experience in the evaluation field ranging from 3 to 30 years. The discussions were held online through Zoom. The insights of these evaluators are incorporated in the results presented in the next section.
Results
This part is divided into four sections. The first one describes the characteristics of the survey respondents, the second presents what is understood as long term in evaluation, the third elaborates on the motivations to include the long term in evaluation, and the fourth describes the challenges involved.
Note that some concepts were explicitly defined in the survey (see Table 1). However, the term “evaluation” itself was on purpose not defined to avoid theoretical debates about its definition and keep the survey open to all evaluation practitioners. Also, “long term” was not defined since one of the purposes of the survey was to investigate what is considered long term in evaluation practice.
Terminology explicitly presented in the survey.
Characteristics of the survey respondents
The first part of this section presents personal characteristics of the respondents, such as their age, gender, and country of origin, and the second part focuses on characteristics related to evaluation practices, in general, and to long term and sustainability, in particular.
Most of the respondents are between 30 and 75 years old, with the apogee around 40–45 years old. Concerning gender, slightly more men (almost 52%) than women (45%) answered; one person responded other gender, and eight respondents did not wish to provide this information. The geographical origin is spread over the world (see Figure 1 for details). Given an over-representation of practitioners from North America and Europe, the results have a probable bias toward a more “Western” understanding and experience of long term in evaluation.
Geographical origin of the survey participants. The central pie chart gives the global picture per continent, associated with the percentage of the respondents. The smaller pie charts show the share of sub-regions, labeled with the number of respondents for each of them.
Most respondents consider themselves as professional evaluators (more than 92%). More than 95 percent of all the participants declare having more than 2 years of experience in evaluation practice and 81 percent more than 5 years. The respondents conduct both external and internal evaluations and the conditions of their work vary a great deal, but almost 50 percent are independent evaluators, either exclusively or among other duties. Almost 90 percent have experience in evaluation of impacts and outcomes, and they conduct both backward- and forward-looking evaluations of impacts and outcomes; however, very few respondents conduct mostly forward-looking evaluations (less than 4%).
The respondents are open to long-term perspectives, with more than 87 percent who are willing to include long term in their evaluations 3 and more than 85 percent who agree that “in general, long term should be more often included in evaluations of impacts and outcomes” (4% disagree). Interestingly, very few respondents (3.6%) declare having never included long-term perspectives in their evaluations. In practice, 13.1 percent of the survey participants declare that they always include long-term perspectives in their evaluations of impacts and outcomes, 30.3 percent often do it, and 36.7 percent sometimes. Therefore, around 80 percent of the respondents include long-term considerations in their evaluations at least sometimes (3.6% never do, 16.3% rarely). A third of the respondents estimate that between 10 percent and 30 percent of their evaluations take long term into consideration, almost another third estimate this amount between 31 percent and 70 percent, 30 percent estimate it to strictly more than 70 percent of their evaluations, and fewer than 5 percent to strictly less than 10 percent of their evaluations. 4 Concerning the training and education of the respondents to include the long term in their evaluations, almost 40 percent have not received any, about 27 percent did receive specific training, and slightly more than 33 percent declared that it was part of their general education in evaluation.
The integration of long term into evaluation is not automatically connected to sustainable development concerns (and was not presented as such in the survey), yet the respondents are mostly open for evaluation and evaluators to play a role in sustainable development (see Supplemental Appendix 2). For instance, more than 90 percent of the respondents agree that evaluators have a role to play in the quest for sustainable development (2.2% disagree) and almost 70 percent agree that sustainable development cannot happen without the engagement of evaluators (12% disagree).
What is long term in evaluations of impacts and outcomes and how is it included?
The survey inquired about timeframes in terms of a number of years to understand what is considered as long term in evaluations of impacts and outcomes (see Figures 2 and 3 and Supplemental Appendix 3). In theory, the minimum period to speak about the long term in evaluations of impacts and outcomes is at least 4 years after the implementation of the initiative (median value, lower quartile 3 years, upper quartile 5 years). In practice, when the respondents do include the long term in their evaluations, they consider a range of 2 to 5 years after the implementation of the initiative (median values). This timeframe is similar for both backward- and forward-looking evaluations of impacts and outcomes. In forward-looking evaluations, the maximum timeframe that is considered possible and realistic to evaluate is up to 5 years after the implementation of the initiative (median value, lower quartile 3.5 years, upper quartile 10 years). It is possible to see in Figure 2 a few standout answers that consider the long term to be closer to 100 years, or even 250 years after the implementation of the initiative, in one case.
Box and whisker plot representing the meaning of long term in evaluations of impacts and outcomes. The corresponding questions in the survey are the following:
Zoom-in of Figure 2.
Note that a couple of respondents mentioned in the comments that they were having difficulties answering the questions related to how many years is long term, notably because the answer is highly dependent on context and on what is evaluated. This point was highlighted during the discussions with the practitioners afterward as well, and it was also pointed out that a timeframe of 2 to 5 years can correspond to what commissioners are ready to finance, given that it is the usual timeframe for international aid or development programs and projects. One practitioner warns about a recurring issue in evaluation which is to confuse the timing of a project with the timing of change. In addition, another practitioner pointed out that the long term in evaluation is not only a question of years represented in a data set, but also the issue of how the evaluation is used and how long the evaluation process will affect ways of thinking among the stakeholders.
It is not possible to identify a clear privileged approach for including the long term in evaluation from the answers to the survey. Deductive (starting from a scientific theory and then collecting data to check if it matches the theory), inductive (collecting data first and then extracting a model or a theory from it), and abductive (making probable hypotheses to test them and adapt them) approaches are used. (Note that there are similar results when the participants are asked about approaches in evaluations of impacts and outcomes in general, without specifically including the long term.) The respondents, however, promote mixed methodological tools for including the long term since more than 75 percent declare using them in practice.
Finally, it is unclear if the demand for long-term perspectives in evaluation has changed over time. Some say it has increased, and some say that it has not changed. Very few respondents state that it has decreased.
Reasons to consider the long term
Several reasons can be identified as motivations to include the long term in evaluations of impacts and outcomes, as shown in Figure 4. Among the reasons that were selected by more than half of the respondents of the survey, two main categories can be identified. One groups motivations to uncover particular causalities (“to uncover impacts that take long to appear” and “to show multiple and interconnected causal patterns”) and the other groups reasons that are associated with the wishes of other actors involved in the evaluation (“the commissioner of the evaluation asks for it” and “it is mentioned as important by the stakeholders”).
The main motivations to take long term into consideration in evaluations of impacts and outcomes.
Discovering particular causal patterns is consistent with what the inclusion of long-term perspectives has helped the respondents to reveal in practice. These are contextual factors and impacts that take a long time to appear for more than half of the respondents, as well as multiple and interconnected causal patterns, aggregated effects, and impacts that are only short term and do not imply long-term changes for more than a third of the respondents (Figure 5). The respondents highlighted in their comments the need for long-term considerations to incorporate contextual factors and that some changes take time.
What integrating long-term perspectives in evaluations of impacts and outcomes brings in practice.
Concerning other actors implicated in the evaluation process, the comments in the survey confirm that the evaluator’s motivations are not the only ones at stake. Other actors involved in the evaluation can push for including the long term, for instance if the commissioners realize the benefits of long-term perspectives, or if the stakeholders ask for the long term to be included or at least acknowledged in the evaluation. More generally, external factors influence the inclusion of the long term in evaluation. Needs related to policy, investment, and decision-making can encourage long-term considerations. The taxpayers might also want to know what policy works or not in the long term. According to the discussion with one practitioner, society at large can put pressure to include the long term when evaluating what does or does not work. It can also be that including a long-term perspective is mandatory in the evaluation of large public investment projects.
The types of long-term impacts usually included by the respondents in their evaluations of impacts and outcomes are most often both intended and unintended (almost 82%), but if it is only one of those, it is usually the intended ones (more than 17%; fewer than 1% look at unintended impacts only). The types of impacts are also most often both positive and negative (almost 90%), but if it is only one of those, it is usually the positive ones (more than 10%, fewer than 0.5% look at negative ones only). The types of impacts are also most often both direct and indirect (more than 84%), but if it is only one of those, it is usually the direct ones (14.5%, only 1.3% look at the indirect ones only). Finally, more than 55 percent of the respondents investigate the impacts on both human and natural systems, but if it is only one of them, it is usually on human systems only (more than 44%; fewer than 0.5% look at the long-term impacts on natural systems only).
According to the responses to the survey, the evaluations of impacts and outcomes seem more prone to include long-term perspectives depending on the focus area of the initiative (the most cited examples are, in order, education, environmental issues, health care, economic development, social or community changes, and behavioral changes). Some respondents commented that long-term perspectives are important as soon as the problem the initiative is trying to solve is itself long term. The type of evaluand also influences whether an evaluation is more prone to include long-term aspects (especially public policies, long-term programs, organizations, and systems rather than projects), as well as the type of commissioner (governmental agencies and international organizations such as United Nations agencies or international non-governmental organizations rather than private companies). By contrast, the geographical area does not seem to have large implications on the likeliness that the long term is included. The respondents (23.5%) who do state that some evaluations of impacts and outcomes are more prone to include the long term due to the geographical area where the initiative is implemented mention either countries with robust data availability and countries with a stable situation or on the contrary the most marginalized and vulnerable places.
Importantly, almost 65 percent of the respondents agree with the fact that the inclusion of the long term in evaluations of impacts and outcomes increases the use of evaluation (fewer than 8% disagree). However, only slightly more than 50 percent agree that the longer the timeframe in the past, the more reliable the results in backward-looking evaluation of impacts and outcomes (23% disagree).
The respondents also mention other reasons to include the long term that were not listed in the question presented in Figure 4: first, to show a general evolution of the territory, rather than just the results of a specific initiative, and because it is critical in assessing system and transformational changes, as well as behavioral changes. These reasons were mentioned both in the survey comments and in the discussions with the practitioners. The environmental and ecological context, including climate change, encourages practitioners to evaluate long-term impacts. Development in general, and sustainable development in particular, needs long-term perspectives. The issue of the sustainability of an initiative is also underlined by the respondents in terms of the need to identify leverage points to make the initiative sustainable and to investigate whether the changes will last after the intervention is completed, for instance through the institutionalization of change or long-term behavioral changes. As a practitioner pointed out during the discussions, there is also a desire to unpack what could happen in the future.
In addition, several respondents state that it should be a norm, a standard, to include the long term in evaluation. Others point out that looking at long-term issues is actually the purpose of impact evaluations. This point also came back during the discussions with the practitioners.
Second, the timing of the evaluation must be compatible with the inclusion of long-term impacts. On the one hand, when evaluations take place just after the implementation of the initiative, it is harder to include the long term compared to when the evaluation takes place later. On the other hand, the timing can be related to the purpose of the evaluation, and an evaluation contributing to an ongoing development program is more likely to include long term than after-the-fact accountability. Other examples are summative evaluations and evaluations comparing the long-term effects of alternative programs.
In general, the survey respondents are positive toward the theoretical idea to include the long term in evaluations of impacts and outcomes (together with shorter term perspectives, not replacing them); however, they encounter several difficulties that explain its limited inclusion in practice.
Difficulties encountered in including the long term
Figure 6 presents the main difficulties encountered by evaluators in including the long term in evaluations of impacts and outcomes. It is important to underline that several respondents have commented that those are only challenges and can be overcome, some even arguing that it is the role of evaluators to do so. Several practitioners also insisted on this during the discussions.
The main challenges encountered in including long term in evaluations of impacts and outcomes.
Clearly, the main difficulty standing out in Figure 6 is that the necessary data to include long-term perspectives are not available. According to the evaluators who discussed the preliminary results of the survey, recurrent reasons for that problem involve a combination of data not being collected at the time of the implementation of the initiative, memories fading away, and key stakeholders being impossible to find. Survey respondents commented on the fact that long-term perspectives often turn out to be addressed theoretically in practice because of a lack of information.
Then there are problems with the attribution of the impacts (“I cannot isolate the impacts of the object of evaluation from the impacts of other interventions” and “I cannot establish a causal relationship between the evaluand and the impacts”; Figure 6). This is linked to the reliability of the results, since more than 60 percent of the respondents agree that the longer the timeframe extends into the future, the less reliable the conclusions for forward-looking evaluations of impacts and outcomes (fewer than 18% disagree). Several respondents also commented in the survey about the issue of credibility of predicting outcomes and the need for relevant and robust methodologies. Despite the fact that very few respondents selected methodology as one of the main difficulties in Figure 6, more than 63 percent of the respondents declared having been at least once in a situation in which they encountered methodological issues that they could not overcome (Table 2). Besides this, since some of the main struggles are related to the question of attribution of the impacts in the long run (Figure 6), they are related to methodological challenges. In the comments of the survey, respondents mentioned the lack of foresight analysis, challenges in obtaining a large enough sample, the fact that long-term investigations are often based more on the subjective views of experts and stakeholders and less on rigid methods, and the idea that it can be very difficult even to assess short-term outcomes, so it would make no sense to try for long-term ones. One practitioner mentioned during a discussion that often the problem is not that the methodologies are not available, but rather how rigorously the causal relationship between the initiative and the impacts can be established. More than a third of the respondents also selected the problem of too many uncertainties as a main challenge (Figure 6). Related to the latter, more than 20 percent of the respondents declared having several times been in a situation where they started to include long-term perspectives but there were so many uncertainties that they changed their mind and focused on short-term consequences only (Table 2; but 41% have never encountered this situation).
Answers to the question “Have you ever been in the following situations during some evaluations of impacts and outcomes?”.
Another very important challenge is the issue of limited financial resources (Figure 6). Table 2 shows that the problem of limited resources is indeed an obstacle to the integration of the long term in evaluation: more than 60 percent of the respondents declared having been at least once in a situation where they wanted to include the long term but they did not have time for it, and more than 71 percent did not have the financial resources for it (40% have been “many times” or “most of the time” in this situation). That aspect was also highlighted in the comments of the survey and during the discussions with the practitioners, with some potential explanations such as a lack of interest of the commissioner and sometimes a lack of awareness about the benefits of long-term perspectives in evaluation. According to the respondents’ comments, even if they are willing to include the long term, limited time and limited money are among the main drawbacks in practice.
More than a third of the respondents also selected as a main difficulty the fact that the other actors of evaluation (commissioners and stakeholders) do not express any interest in long-term issues (Figure 6). Slightly more than 20 percent declared having been several times in a situation where the commissioner refused to include long-term perspectives after having discussed it with the evaluator (Table 2; but almost 45% have never encountered this situation). Some comments in the survey give details about the limited interest of some other actors of evaluation, which can lead to a lack of available resources. The lack of interest of the commissioner, an issue that was often raised during discussions with the practitioners, can come from several sources such as myopia in the short term, notably because those who run the programs are only responsible for the immediate and near-future outcomes, and it is not the mission of the organization to consider the long term. Also, organizations can feel that it is not their responsibility to mitigate grand challenges such as climate change, which tend to be highlighted by long-term aspects. Besides this, there were some comments in the survey on the lack of culture for forward-looking policies. Another reason for the commissioner to avoid long-term perspectives is the fear of failure. A lack of interest of the stakeholders can come from a high turnover, and sometimes because they assume it is not possible to include the long term in evaluation. Also, some beneficiaries of the initiative live in a precarious situation that does not enable them to think long term. In general, the limited interest of the other actors of the evaluation can come from rapid changes in the local or societal context. The interest in long-term impacts fades over time because society moves on rapidly to focus on other issues. The world is changing so quickly that some respondents explain that long-term views might be irrelevant. For instance, long-term impacts are affected by rapid changes in technologies or sudden events such as COVID-19. Also, investigating long-term impacts can be irrelevant if the time limit for decision-making has passed. In addition to a lack of interest, there can also be a conflict of interest between the various actors involved in the evaluation and the initiative implementation.
Moreover, it is worth noting that it is most often not up to the evaluator to decide on the timeframe of the evaluation: more than 46 percent of the respondents declare that it is the choice of the commissioner, almost 40 percent declare that it is a joint decision, and only 8 percent declare that it is the choice of the evaluator (and 6.4% of someone else). The respondents mention in the comments that sometimes the involved stakeholders can influence the decision on the timeframe, while other times the timeframe depends on legal frames and regulations. When the timeframe is decided by the commissioner, it can depend on the funding organization and/or the implementing one.
Other challenges mentioned in the comments by the survey’s participants are related to the initiative itself, with a limited period of funding and no anticipation for impact evaluation. Sometimes, the scale of the intervention is too small to identify long-term impacts. Someone states that when the initiative is already incoherent and unrealistic, it is irrelevant for the evaluation to look at the impacts over time. Another respondent points out that long-term inclusion is challenging when there are no clear goals for the initiative or when the implementers have a weak understanding of them. The evaluation contract can also be a difficulty, for instance when the terms of references do not enable impact evaluation or when the purpose of the evaluation is not adapted to long-term investigations. One given example is when the evaluation work is intended for process use.
Finally, some findings from the survey present ambiguous situations that are difficult to judge if they should be considered as challenging or not because the results are slightly contradictory. For instance, very few respondents declared that including long term in their evaluations in practice did not bring anything different than short-term perspectives (Figure 5), while at the same time almost a quarter declared having several times been in a situation where they included long term but it did not result in more interesting findings than the conclusions from short-term perspectives only would have (Table 2). In the same line of contradictory results, while more than 47 percent of the respondents declared having never been in a situation where including long-term perspectives had made the conclusions of the evaluation so complex that it was hard to communicate them clearly, almost 30 percent declared having at least several times encountered this issue (Table 2). At the same time, this problem was also selected by much fewer than a quarter of the respondents as a main difficulty (Figure 6), while the complexity of long-term impacts was also mentioned in the comments as a challenge.
Discussion: Long term in evaluations of impacts and outcomes within complex sustainable transformations
The literature presented in the Introduction section justifies the importance of including the long term in evaluations of impacts and outcomes for two main reasons: first, change is not linear, and second, it is necessary to realize the transformational power of evaluation. The first two parts in this section discuss these two aspects in light of the results presented earlier, and the following parts reflect on power issues and societal development and evaluation, which are both important drivers of the inclusion of long term in evaluations of impacts and outcomes.
Which methodology for non-linear change?
Concerning the non-linearity of change, the answers to the survey about motivations to include long term highlighted that, in practice, most applications of long-term perspectives result in detecting non-linear changes. This non-linearity of change can render difficult the choice of a timeframe to integrate in an evaluation, as shown by Hildén (2009) when evaluating environmental policies. Non-linear changes take place in complex situations (Barnes et al., 2003) and therefore evaluations that integrate long-term perspectives must be ready to also address complexity. Evaluating complexity brings its fair share of motivations and challenges too (Forss et al., 2011); among these are methodological issues. This is reflected by the survey’s respondents who point out difficulties related to the attribution of the impacts to a specific initiative, such as the isolation of the impacts and the causal relationship between the evaluand and the impacts. There are several strategies to address causal attributions in impact evaluation (Rogers, 2014), and as mentioned in the Results section, the survey respondents tend to use mixed methods for including the long term in evaluation. The evaluation field also discusses the interest to evaluate the contribution of an initiative rather than attribution (Mayne, 2012), especially when including long-term perspectives (Mayne, 2021) and when evaluating large-scale transformational processes (Junge et al., 2020). This idea is in line with the result showing that integrating long-term perspectives helps evaluators to detect contextual factors. Further studies could elaborate recommendations on methodological tools to integrate the long term in evaluation and test those frameworks with evaluation practitioners.
Together with the methodological issues comes the problem of the availability of data. Indeed, collecting data a long time after the implementation of an initiative means facing difficulties such as fading memories, since “the amount of measurement error is greater the longer the recall period and the more detailed the information requested” (Creevey, 2008: 2). The problem of the lack of data is linked with other evaluation issues: for instance, the general evaluative thinking in organizations, so that collection of data becomes systematic and is already conducted during the implementation of the initiative (Baker and Bruner, 2012); the role of the evaluators in overcoming challenges, as was pointed out in the survey and during discussions with the practitioners; and the resources at the disposal of the evaluation team, such as time and money to collect the missing data.
What timeframe for a sustainable transformation?
In terms of the transformational power of evaluation, as presented in the section about the characteristics of the respondents, most of them are positive toward the role of evaluation for sustainable development. But even if the inclusion of the long term in evaluation is welcomed by practitioners and can provide evaluation with a greater transformational power (Chaplowe and Hejnowicz, 2021), clear challenges appear in practice, one of which relates to what is considered as long term. Indeed, the timeframe given by evaluators of around 4 years after the implementation of an initiative, in both theory and practice, questions the transformational power that evaluation can have over society. Actually, most grand challenges of our time, such as climate change, would require longer timeframes. For instance, the Intergovernmental Panel on Climate Change (IPCC, 2021) establishes climate change scenarios toward 2100, calling 2040 near term. One could ask if, in the end, the long term considered in the practice of evaluations of impacts and outcomes is not rather short term compared to societal challenges. When considering the definition of sustainable development given in the Brundtland report (World Commission on Environment and Development, 1987), future generations should be included, which can hardly be done over 4 years. The chosen timeframe can affect the conclusions of evaluation drastically (Hildén, 2009), and limiting long-term timeframes to a few years after the implementation of an initiative can result in an emphasis on costs at the expense of a consideration of benefits. For instance, climate mitigation often presents a situation in which the economic costs of the strategies unfold quicker than the positive effects, therefore having net costs in the short and medium terms (Deichmann and Zhang, 2013).
The limited timeframe that seems to be the most common in evaluation practices when including long-term perspectives might reflect, as mentioned in the Results section, the most common type of evaluand, namely projects. According to the respondents, other types of evaluand, such as public policies, organizations, and systems, would be more prone to integrating long-term perspectives. Yet, it is also important to underline that, as the results show, a few practitioners consider a longer timeframe than average, which can go up to 250 years. No specific personal or professional characteristic of those respondents distinguished them from the others in the survey. Further research could investigate who they are, what kinds of evaluation they conduct, and how the challenges encountered by evaluators in this type of timeframe differ from the ones related to a more limited timeframe.
Who has the power to integrate the long term in evaluation?
In general, whatever is understood by the long term and the motivations for including it, some of the main challenges pointed out by the practitioners concern financial resources and the willingness of other actors, such as the commissioners and the stakeholders, to integrate the long term. Some respondents comment that it should be a norm to include long-term perspectives when evaluating the consequences of an initiative. While the timeframe that is included in an evaluation should be an active choice, it can be contentious (Hildén, 2009). As the results show, evaluators rarely decide by themselves the timeframe included in the evaluation. Therefore, evaluation practitioners do not have the full power to change evaluation practices toward longer term evaluations, and it is a profound change in the overall evaluation field that needs to take place to modify practices. These findings are in line with the need for a transformation of the whole evaluation field toward sustainability-focus evaluation (Rowe and Uitto, 2023). In general, “evaluators are induced (or seduced) to share their clients’ perspectives, while giving lip service to the public good: they cannot afford to bite the hands that feed them” (Picciotto, 2020: 51). Indeed, the lack of independence forces evaluators to reflect the visions and judge the initiatives based on mainstream neoliberalism, which focuses most of the time on short-term measurable outcomes, and therefore hinders the potential of evaluation to contribute to the public good (Mathison, 2018).
Further research needs to investigate the point of view of evaluation commissioners on the integration of long-term perspectives in evaluation. Since evaluation is a process embedded in society in general, integrating complexity through long-term insights also requires flexibility within the policy-making process that commissioners depend on (Cox and Barbrook-Johnson, 2021), which could be investigated as well. In addition, further research should also investigate what institutional model for evaluation would be best suited to integrate longer term perspectives.
What relationships between evaluation and societal sustainable development?
Not only is evaluation embedded in society, it also influences it. Studies about the role and influence of evaluation in society suggest general purposes of evaluation such as social betterment (Mark and Henry, 2004) and change (Segerholm, 2020). As Alkin (2013a) points out, “circumstances shape receptiveness to new ideas” (p. 284) in evaluation. More and more literature and research show an increasing interest in the role of evaluation specifically for promoting sustainable development (e.g. Brousselle and McDavid, 2021; Larsson, 2021; Rowe, 2019), with for instance ethical discussions about the purpose of monitoring and evaluation systems in the socio-ecological transformation of society (see Gates et al., 2023) and recommendations for adapting evaluation response to current environmental challenges (e.g. Brousselle and McDavid, 2020; Davidson et al., 2023). While the field is still lagging behind compared to the urgency to integrate sustainability considerations systematically in evaluation (Rowe and Uitto, 2023), these debates and theoretical elaborations can encourage modifications of evaluation practices. The focus on sustainable development today triggers that change, as illustrated by the new definition of sustainability criteria by the OECD (2023a), which now includes evaluation of the various needs to sustain net benefit over time with considerations for the medium and longer term. The OECD (2023b) presented “effective results frameworks for sustainable development” that group concrete suggestions for evaluation practitioners and justify the importance of integrating sustainable development in results frameworks by referring to the need for a long-term vision, among other elements. The findings of the research presented here show that practitioners acknowledge several benefits of including the long term in evaluation, and are most often motivated to do so if they can overcome the challenges. Accordingly, there are opportunities to get out of a short-term-only focus and realize the transformational power of evaluation toward sustainable development.
Concluding remarks
This study investigates the notion of the long term in evaluations of impacts and outcomes, from the point of view of evaluation practitioners. It is an exploratory work that has the ambition to promote broader discussions about this topic within the evaluation community to help evaluation play a key role in the transformation of our society toward more sustainability. In light of the discussion, the results show an ambivalence with several challenges counterbalanced by several motivations to include long-term perspectives when evaluating the consequences of an initiative.
Many challenges limit the transformational power of evaluation and need to be addressed further. The greatest one is that the period considered long term in practice (2 to 5 years) is out of phase with long-term impacts that affect society (e.g. climate change scenarios consider effects over 80 years ahead). But at the same time the context is changing so quickly and is so complex that some respondents question the relevance of including such an extended timeframe, notably because of a lack of robust methodologies in forward-looking evaluation and limited causal attribution in backward-looking evaluation. In addition, the evaluation team rarely chooses the timeframe of the evaluation by itself: other actors take part in the decision (e.g. commissioners and stakeholders).
Yet, the results underline some opportunities for the transformational power of evaluation, with most respondents willing to integrate long-term perspectives, justified by several motivations. Evaluators want to discover particular causal patterns (e.g. effects that take a long time to appear or that fade over time), to include contextual factors, and to respond to the demands of other actors interested in the long term. Some focus areas of the evaluated initiatives also prompt long-term considerations (e.g. educational, environmental, or health issues). Practitioners clearly claim that the challenges to incorporate long-term perspectives in their evaluations can be overcome, especially if commissioners are interested in long-term consequences as they provide the resources needed to gather missing data, which is the main obstacle to overcome. Relevant methodologies and frameworks help with the issue of the credibility of the evaluation findings.
Therefore, in spite of the difficulties, opportunities exist for evaluation to contribute to a sustainable transformation through the integration of long-term perspectives when evaluating the consequences of an initiative. The evaluation community at large should open a dialogue about these matters, informed by this study, which broadens our understanding of the issues encountered by practitioners.
Supplemental Material
sj-pdf-1-evi-10.1177_13563890241258202 – Supplemental material for The transformational power of evaluation: Exploring long-term perspectives in evaluations of impacts and outcomes from the standpoint of practitioners
Supplemental material, sj-pdf-1-evi-10.1177_13563890241258202 for The transformational power of evaluation: Exploring long-term perspectives in evaluations of impacts and outcomes from the standpoint of practitioners by Mathilde de Goër de Herve in Evaluation
Footnotes
Acknowledgements
The author would like to warmly thank every participant in this study: all the respondents who contributed to the survey and each person who took the time to share it in their networks, as well as the evaluation practitioners who agreed to be interviewed. Thanks also to the evaluation societies and networks who have responded positively to the dissemination of the survey. In addition, the author would also like to thank the evaluation practitioners who participated in the test study, as well as the four people who agreed to give feedback on earlier versions of the survey. Thanks also to the reviewers, before and during the publication process, who advised on earlier versions of this article to improve it.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
