Abstract
Starting with the Australian Evaluation Society International Conference 2024 theme of ‘wayfinding’, the article reflects on how evaluators navigate time as well as space, with particular reference to how they incorporate qualitative approaches to ‘futures thinking’ into their evaluative practice. The article takes a reflexive case study approach, drawing on a simple model of evaluative practice to review the role of futures thinking in evaluation of doctoral research projects, impact investments, social assistance, and development studies as an academic field. The article focuses particularly on the role of ‘backcasting’, as well as deliberative processes and the causal mapping of narrative claims linking future possibilities to current thinking and activities.
Introduction
The aes2024 conference theme of wayfinding has inspired me to revisit how we navigate through time as well as space, and how we meet the challenge to be relevant to the accelerating pace of global change. My starting point is the division of labour between evaluators, futurologists, and those who commission their services. If, as professional evaluators, we are to be more incisive, forward-looking, and useful, then perhaps we need to revisit how we relate to these other two groups. How do we evaluate policy and practice in ways that better anticipate decisions required to address looming challenges?
But first, do we and should we as a community of evaluators take this on? Figure 1 comes from a friend’s sat-nav, taken a few weeks ago while she was driving through Birmingham in the UK. Every conceivable route she could take showed either a traffic jam or roadworks. ‘How useful was that?!’ she commented. Sat-nav image depicting predicted congestion and road works.
Nassim Taleb, the author of ‘Black Swans’ (Taleb, 2008), warns against trying too hard to predict the future, adding that there’s much we can do when we focus instead on what we don’t know. However, if we aspire as evaluators to inform and influence important public decisions, then we cannot avoid peering into the future. In his powerful recent book on ‘navigating the global poly-crisis’, Michael Albert (2024) argues that ‘thinking usefully about the future of the planet is too important to be left to states, corporations, and technologists. The difficulty of the task is an argument for investing more in tackling it, not less’ (p. xix). I agree, subject to being realistic about how accurately we can do so. This caution leads to a particular interest in reflecting on open-ended and qualitative approaches to thinking about the future, as a normative or developmental as well scientific endeavour – a way of opening-up possibilities, as well as analysing them.
I’m not alone in arguing that evaluators need to engage more with the future. The last few years have seen repeated calls for the transformation of evaluative practice, particularly in the face of climate change. And a recent special edition of New Directions in Evaluation (Volume 2024, Issue 182) makes a sustained case for building a stronger bridge between evaluation and foresight communities of practice. Dart and Gates (2024) remind us that theories of change, at the heart of much of what we do, are not just concerned with explaining and exploring causal processes, but also with drawing on our imagination to peer into the future. And the article by Davies (2024) offers one avenue for doing so through reflections on use to date of ParEvo, his brilliant tool for integrative participatory scenario building. I’m inspired by these examples and hope this talk can add something to promoting more forward-looking and useful evaluative practice.
This article takes a case study approach, reflecting on four situations where I’ve been trying to be more forward-looking in my own thinking. To facilitate generalisation from these experiences, I frame them within a simple model of development practice (Figure 4, below). I also limit the scope of what I say by focusing particularly on the use of causal analysis to backcast from anticipated future outcomes (Robinson, 2003). By focusing on anticipatory backcasting, I neglect other issues bridging evaluation and futures thinking, including the evaluation of anticipatory interventions, such as in the field of disaster risk financing, and evaluation of the use of foresight tools like ParEvo. To explain this more clearly, Figure 2 contrasts four different ways in which we do causal analysis through time (arrows indicating causal links from activities to outcomes, rather than the direction of analysis). Forms of causal analysis through time.
In the language of causal mapping (Powell et al., 2024), the first two are examples of forward chaining. Both start by identifying causal drivers (written in bold), differing according to whether they start with causes in the past or in the present. The second two are examples of backward chaining that start the analysis with outcomes (again in bold) and work backwards (against the direction of the arrows) to consider possible causal factors. Anticipatory backcasting is the most forward-looking of the four, and that’s what I’m going to focus on.
Framing evaluation around an anticipated outcome or a single imagined future scenario may seem novel but is not uncommon. Official international development cooperation is framed by the Sustainable Development Goals for 2030, particularly in the UN system, and another leading example is adoption of net zero targets for decarbonising organisations and whole countries as a response to climate change. This builds on a long tradition of backcasting from demand forecasts to plan investment in sustainable energy systems (Vergragt & Quist, 2011). Meanwhile, military planners chase moving targets for rearmament to match the build-up of similarly motivated adversaries, and political scientists explore what might constitute a sufficient package of conditions for achieving a lasting peace. And the ‘turnpike theorem’ in economics has long served as a reminder that the most direct path to a distant destination is not always the fastest.
Fixing on one future outcome, and assessing prospects for achieving it, can help break debate out of a rut, and even inspire social movements – to ‘ban the bomb’, to ‘achieve universal basic income’ or to ‘get promoted to the Premier league’. Of course, there is also a risk that simplistic aspirational target-setting can backfire – the challenge being to identify an outcome that is aspirational, but neither an impossibility nor a distraction from more important long-term priorities.
To further illustrate what I mean by backcasting, consider the familiar but remarkable case of Google Map. This speaks to the ‘wayfinding’ theme of the conference by illustrating how our spatial recognition capability simultaneously tells us a story about time. The process of working back from an anticipated outcome can be drawn on a map (Figure 3, left) and also in analytical time as a causal map (Figure 3, right), which is also as a simple theory of change. Backcasting with Google Maps.
The causal outcome at the top left of the Google Map picture happens to be Porthmadog in Wales, where I spent my summer holiday. My reason for choosing this example is that between leaving home in Bath and crossing the Severn Estuary, Google Maps switched between several different ‘optimal’ routes for me, the change being triggered by real-time information about changes in traffic congestion. Some of these alternatives, I suspect, were endogenously generated by other people who were navigating, like me, with Google Maps. Note my preference for avoiding a route through Birmingham, even though the projected journey time was less. Google Maps provides a far more sophisticated navigational aid than most policy makers can draw upon to guide them – a geographical map, a causal map, a theory of change, and an interactive simulation model all rolled into one! It illustrates backcasting because we generally use it by asking it to achieve a specific future outcome.
Explicit causal maps, including theories of change, use space on the page as an extended metaphor. The arrows between boxes represent vaguely defined causal connections between events, but we tend to read them as if they were fixed and unambiguous – like roads between towns. This is clearly not the case. I think causal mapping is key to being more transparent about how we think about complex processes of change, and to enabling us to build and discuss theories of change more rigorously than we do in everyday speech. But I also think that we are a long way from doing this well, particularly when constructing theories about what could happen in the future, as well as what we think might explain what happened in the past. This article mainly explores how we already incorporate future thinking into evaluative practice, but it also suggests how we might be able to do this better in the future, particularly through anticipatory backcasting. I think more rigorous causal mapping can also help, but it is beyond the scope of the article to develop this point much further. Powell et al. (2024) goes into its intricacies in more depth, drawing directly upon experience of developing software - Causal Map - to facilitate more rigorous coding, mapping and visualisation of causal claims embedded in natural language.
A model of evaluative practice
Events and empirical information about events flow forward from past to present, but our current actions and ideas are also informed by expectations, imagination, and foresight about the future. Figure 4 provides a simple causal model of evaluative practice that sets out these flows, distinguishing between four core tasks causally linked to each other by solid black arrows. Together, these can be thought of as creating a space for deliberation outside of the complex daily flow of events. Dotted lines represent flows of data and ideas informing these tasks, both flowing forward in time – when referring to the past – and backwards through time, when anticipating the future. A model of evaluative practice.
Problem framing, the first task, may be triggered by a formal commissioning process, and linked directly or indirectly to future decision making (1). It is also informed by knowledge of the past, including prior evidence and theory (2), expectations of the future, such as a specific theory of change (3), and formal or informal evaluability assessment (4). I first drew this diagram as an attempt to think more clearly about how causal analysis (including causal mapping) relates to the wider evaluative judgements we make of value, merit, and worth. But these two tasks are hard to disentangle, so the diagram puts them into one box. Traditional backward-looking evaluation draws primarily on historical data (5), whereas central to this article is the suggestion that we can go further in incorporating foresight thinking into evaluative practice too (6). This may include use of specific techniques, such as ParEvo. It also includes sensemaking interactions (friendly or otherwise) between evaluators, commissioners, and others involved in future decision making (7).
Applying the model to the case of wayfinding using Google Map.
Case studies
Doctoral research
The remainder of the article uses this model of evaluative practice to reflect on how foresight thinking (particularly backcasting) has influenced my own work. Much of my time is currently spent working with doctoral students, so I start with the challenge of evaluating ideas they formulate for their research thesis. This is inspired partly by the work of Catherine Manathunga on the images students and supervisors in Australia construct of their doctoral journeys (Manathunga et al., 2021). Nobody starts a doctorate without some vision of the outcome and what the journey might be like, but my experience is that this thinking can be surprisingly vague. Figure 5 illustrates with a selection of images that can help. Visualising doctoral research.
The first picture (a) is a standard timeline showing formal steps for the professional doctorate in policy research and practice at Bath which I help to run. It anticipates an outcome (graduation) and focuses on signposts with causality implicit in the assumption that each is a necessary condition for progression to the next. A project management approach to doing a thesis is important, but the second picture (b) – drawn by a student – is a reminder that the journey is far messier than this, both chronologically and emotionally. Not only are there different possible routes up the ‘thesis mountain’ (c), but the mountain itself keeps changing! A useful backcasting exercise to assist students with thesis planning challenges them to identify six necessary and interconnected choices they must make to determine what their final thesis looks like: the field of study; the research community or peer group it is primarily intended for; the specific research question addressed; the broad analytical framework; specifics of the research methodology; and the thesis chapter structure. These may partly be tackled sequentially, but they are not fixed staging points along a timeline or up a mountain. Rather they form and develop concurrently at variable rates like the parts of an embryo (d), a visual metaphor that invites very different ways of thinking about the thesis and how to nurture it.
The final visualisation (e) is a causal map constructed from the coded causal claims embedded in typescripts of interviews with students about drivers and obstacles to their learning as experienced over the previous six months. This is an example of a conventional backward-looking evaluation. To be more forward-looking, such maps can also be constructed through a visioning exercise in which students are first asked to identify what they ultimately want to achieve through their doctoral research, then to identify causal factors likely to help and hinder their progress towards these anticipated outcomes.
Impact investment
My second case moves from the individual to the organisational level. Following seminal Rockefeller Foundation sponsored conferences in 2007 and 2008 in Bellagio, Italy, the impact investment community set itself the goal of transforming the financial industry by mainstreaming an approach to investment that goes beyond risk and return by incorporating credible prospective and retrospective assessment of wider environmental and social impact. This is captured by the following aspirational statement by the Global Impact Investment Network (2018, p. 4): ‘We believe that the concept of “externalities” will be relegated to history, with finance theory accounting for risk, return and impact equally well. Ultimately, financial markets will be central in supporting solutions to critical threats facing the world’.
GIIN did not place a date on this anticipated outcome, but it nevertheless focused attention on the methodological difficulties and weak incentives to evaluating the wider impact of financial investments credibly. Conversely, failure to address them has exposed the industry to damning criticisms of ‘impact wash’ – i.e. that claims to achieve wider impact are self-delusional, tokenistic, and/or a deliberate smokescreen for business-as-usual.
It is beyond the scope of this article to chart in detail how impact investors have sought to address this goal, but it is relevant to highlight the responses of two leading standard setters for the industry, summarised by Figure 6 and recast in Table 2. These are deeply forward-looking in the sense that they seek to institutionalise the processes by which future outcomes – financial, social, and environmental – affect current investment decisions. These may currently fall short of the standards of rigorous ex post social cost benefit analysis espoused by academic economists, but they are also more appreciative of the need for timeliness and of the importance of transparent deliberation. Impact investment benchmarks. Impact investment as evaluative practice.
Poverty elimination in Malawi
The next case moves up to the level of national policy, and draws on my involvement with mixed method impact evaluation studies of two pilot cash ‘plus’ transfer programmes in Malawi during the last year – one implemented by Concern International, and the other by Save the Children (Copestake, 2024). As these proceeded, I began to question the implicit assumption that these pilots might eventually be passed on to the government and/or scaled up across the country. One way to address the scaling-up question head-on is through an anticipatory evaluation question, that starts by asking: how big would a cash transfer programme have to be in Malawi to eliminate poverty? The answer then becomes a reference point for considering financial, administrative, and political explanations for why more sustained poverty reduction has not been possible – and might remain so.
A simplistic estimate of the annual cost of eliminating poverty in Malawi.
The future of development studies
The last case I will consider is global, and closer to my academic home. The Centre for Development Studies at the University of Bath turns fifty this year, and this has prompted some reflection. Is fifty years long enough? Does the Centre at least need a radical makeover? Development studies as an academic field constantly questions its own identity and purpose (Copestake, 2015). Rather than being defined geographically (e.g. focusing on poor countries) it can indeed be better defined as a field for exploring globally diverse views about what the idea of development means – and for whom – as an ideal, as an actual historical process, and as a field of public action. To illustrate (and at the risk of huge simplification) contrast ‘modernisation’ and ‘decolonial’ views of development. Modernisation suggests an ideal of shared material prosperity to be realised through scientific and technological progress, subject to rational policy adjustments to promote social inclusion and the diffusion of ideas, opportunities, wealth, and power. Decolonisation, in contrast, aspires to human emancipation by overcoming deeply embedded structures of inequality, discrimination and injustice. It is fundamentally not about policy reform, but about political struggle between oppressed and their privileged oppressors.
Given its historical association with Western-dominated ideas, interests, and activities there is a strong current of opinion that development studies itself needs to be decolonised. Unequal opportunities and resources to pursue an academic career continue to have a significant influence over which ideas gain influence and which don’t, as revealed by the geographical distribution of contributors to leading development journals, for example. But in addition to engaging in the struggle to decolonise our minds and institutions, there is also a case to be made for imaginary prefiguration of what a post-colonial or fully decolonised version of development studies might look like. This is another example of the potential value of anticipatory backcasting evaluative practice. While the question of how to decolonise practice and thinking predominates (starting with the here and now) there is scope for discussion of what development studies might eventually come to look like.
A useful list of normative principles for post-colonial development studies would need to be broad enough to accommodate a viable academic community, but not so broad and anodyne as to exclude nothing – a tension that echoes that between identifying with an academic discipline or a more open academic field. The list below, for example, perhaps errs towards being too broad: 1. to embrace plurality in conceptualisation of development; 2. to embrace plurality in research methods; 3. to employ research processes that avoid harm; 4. to seek socially just and emancipatory research outcomes; 5. to avoid reducing opportunities for autonomous research.
The potential benefit to taking such a prefigurative approach depends not only on reaching agreement about a final list (as an anticipated outcome) but also on the deliberative process of seeking consensus over it. The practical challenge then shifts to what determines equitable and inclusive participation in the deliberative, consensus-building process. One potential benefit of the anticipatory element is to alter how inclusivity is defined: by strengthening the case for greater inclusion of younger people, for example, as well as others currently at the margins of deliberation but should be more central to it.
Conclusions
It is important to acknowledge that futures thinking is already integral to evaluative practice. The model of evaluative practice I have employed (see Figure 4) makes this clear by distinguishing between five different routes by which evaluative practice already routinely draws upon information about the future, both at the stage of problem framing and during the interplay between causal analysis and evaluative judgement. One way in which evaluators can become more forward-thinking is to recognise how anticipation already infuses evaluative practice, as it does the social sciences (Poli, 2014).
Another approach is to be bolder in making imaginative or abductive leaps to anticipated outcomes at the stage of problem framing. This entails challenging institutional inertia or path dependence by going beyond the theory of change specified in the terms of reference of a commissioning organisation. It also entails involving more stakeholders in the outcome framing process through participatory and deliberative (‘second generation’) approaches to backcasting (Robinson, 2003; Vergragt & Quist, 2011). There are risks involved, and it is likely to work best when there is a consensus in favour of the need for transformative change. But at the very least there is scope for continued experimentation and evaluation of such approaches.
Identifying and committing to a long-term anticipated outcome also offers an additional point of departure for analysis of complex causal processes. Backcasting approaches can complement those that work forward from current or ‘business-as-usual’ realities. Causal mapping (i.e. collecting, coding and visualising the causal arguments of different stakeholders in a deliberative process) can contribute to doing this more transparently and systematically. One indicator of this contribution is how it stimulates analysis across multiple time horizons, and enriches debate over messy medium range causal processes, critical junctures and potential tipping points.
Dystopian evaluative practice as an anticipated outcome.
Footnotes
Authors note
This article is an edited version of a keynote address delivered to the Australian Evaluation Society International Evaluation Conference in Melbourne on 18 September 2024. I am grateful to the AES for the invitation, and for the financial support that enabled me to participate in the conference. I am also grateful to the 25 anonymous participants who submitted questions after the talk, as well as to Maya Cordeiro, Rick Davis, Shaun Finnetty, John Guenther, and Fiona Remnant for comments on earlier drafts.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
