Abstract
Introduction
With the global COVID-19 pandemic continuing to impact our lives, we live in unprecedented times. This book explores how the pandemic impacted evaluation practices, architectures, and values and how evaluation practice and knowledge contributed (or not) to the formulation and adjustment of government policy responses and socio-economic recovery planning. An impressive group of expert authors provide a comprehensive synthesis of insights based on the experiences from four high income countries (Australia, Canada, and the UK) as well as international perspectives from the United Nations.
The use of case studies and real-world examples as well as practical insights to improve evaluation practice will be valuable to evaluators and organisations that undertake, commission, or interpret and use evaluation work. There are lessons learned and calls for an ongoing transformation of evaluation practice, a more collaborative approach where many different actors contribute insights from multiple view points and where human-rights approaches are always considered.
Description
This book of 213 pages is organised into nine chapters which are ‘topped-and-tailed’ by a comprehensive Introduction and Afterword. The book was written in the second year of the COVID-19 pandemic while the crisis was ongoing. This provides a fresh and immediate view of the reality of conducting evaluations during a global crisis. The Introduction is presented in two sections with the first providing an overview based on a meta-analysis of evaluation literature published during the first year of the pandemic and how this could inform future evaluation practice. The second section addresses systems thinking and complexity when undertaking evaluation during turbulent times whilst considering human rights–based approaches. The Afterword, which summarises key lessons into five insightful yet succinct observations, provides a most helpful summary.
Chapter 1 explores the role of evaluation in clarifying interventions, assumptions, and knowledge to inform policymaking decisions and how this differs during crises. The author then discusses the interface between ‘decision-makers and knowledge producers’ and potential reasons for the underutilisation of knowledge during crises.
Chapter 2 uses examples of evaluation innovations from Canada that were sparked by the pandemic. Broad improvements in evaluation practice are discussed, recognising that evaluation can play a key role to proactively support decision-making, to leverage system synergies and collaborations, and to define and assess key performance measures continually over time.
Chapter 3, entitled The Unbearable Lightness of Rights, echoes Milan Kundera’s novel, The Unbearable Lightness of Being (1984), and suggests that there is an imperative to deliberately consider and strengthen human rights–based approaches in evaluation practice, not only during crises but as a matter of routine, making human rights–based approaches substantial rather than ‘light’.
Chapter 4 provides a comparative analysis of national organisations in two countries (Canada and the United Kingdom) that sought to produce knowledge to support accountability under crisis conditions. Importantly, the chapter explores the value of evaluation work to support learning to inform decisions and preparedness for responses to future crises.
Chapter 5 focuses on the impact of the pandemic on the progress towards the 17 Sustainable Development Goals (SDGs) and the assessment of progress towards achieving SDGs, underpinned by the monitoring, evaluation, and learning framework. The chapter provides important lessons for evaluators on how to adjust their practice to be more nimble and to balance their advisory role whilst maintaining ethical and professional standards.
Chapter 6 reflects on the learnings from COVID-19 socio-economic response plans from the UN and at country level. The need for, and benefits of, greater collaborative work and the inclusion and involvement of many actors in addition to evaluators is discussed. The pandemic was the impetus for this reconfiguration, and the value of this more collaborative evaluation approach is recommended in future evaluation work.
Chapter 7 provides a detailed case study of the UK’s response to the pandemic, specifically the impact of ‘lockdowns’, their timing, effectiveness, and efficiency in curbing the spread of COVID-19. The chapter draws on implementation science frameworks citing ‘implementation drift’, political influences, the effectiveness of public communication, and ambiguities of changing recommendations and laws as significant factors influencing the impact of lockdowns and other responses.
Chapter 8 discusses the Australian responses to COVID-19 and the impacts of significant and rapid power shifts to governments. Taking a human rights lens, the tension between implementing new laws and recommendations and balancing this with fairness and proportionality of response is explored. Importantly, the limited utility of evaluation and evaluative information to support decision-making is highlighted. This chapter will be of great interest to the EJA readership.
Chapter 9 explores the importance of evaluation during turbulent times whilst taking into consideration the challenging and changing contexts and systems perspectives to inform transformative change through evaluation, whilst at the same time transforming evaluation practice. Elegantly summarised in the first sentence of the conclusion: ‘…evaluating transformation requires transforming evaluation’.
The Afterword provides an insightful synthesis of the book summarised into five succinct Observations:
Observation 1: the re-emergence of the importance, reach and speed of government responses in a crisis as opposed to reliance on market forces.
Observation 2: exposure of pre-existing deep structural flaws and limited societal responsibility for vulnerable populations such as the elderly, children, and socio-economically disadvantaged and marginalised groups.
Observation 3: the vulnerability of national health systems to cope with crises and the significant inequities in access to healthcare.
Observation 4: significant societal costs and costs in terms of human lives attributed to misinformation is a significant area for evaluation work to inform strategies to avert the impact of misinformation in the future.
Observation 5: turbulence caused by the COVID-19 crisis and the tensions between economic gain and public health are likely to continue.
Assessment
This book is a great resource for all evaluators and organisations that conduct or commission evaluations. It also provides valuable lessons for governments, decision-makers, policymakers, and pandemic response planners. Human rights approaches should be routinely applied in evaluations to ensure that an equity lens applied to evaluation approaches, methods, and interpretation of evaluation results.
If you read nothing else, please read the Introduction (23 pages) and the Afterword (4 pages). These excellent sections provide informative, comprehensive yet succinct summaries and very neatly top and tail the book. If read first, they will undoubtedly spark enthusiasm to delve deeper for greater detail in the intervening chapters. As the book was written before the pandemic was over, the authors are rightly cautious about making ‘broad pronouncements’. Furthermore, this book is limited to perspectives and experiences predominantly from high income countries (Canada, the UK, Sweden, and Australia), although the perspectives from the United Nations provide a broader international view. I particularly enjoyed Chapter 3, Chapter 9, and as an Australian academic and evaluator, Chapter 8 was a priority read for me.
After reading this book, it seems that evaluators should move their practice to more collaborative and inclusive approaches, rather than working in isolation. This could enrich their evaluation work by bringing perspectives from organisations, policymakers, end-users and other stakeholders, to provide more nuanced understandings from different viewpoints. Furthermore, evaluators must increasingly consider the impacts of communication strategies, including the role of popular media, social media, and the potential role of misinformation to influence policy, programs, and the interpretation of program outcomes. Evaluators and policymakers alike must work more flexibly and be increasingly nimble in their approaches and methods to support ongoing evaluations that embed multiple feedback loops at multiple times rather than the more traditional models of evaluating at the end of a trial period – there were no such luxuries during the pandemic where decisions needed to be made swiftly based on the best information available at the time. The limited inclusion of evaluation to support ongoing decision-making is aptly illustrated by the Australian case study (Chapter 8). Finally, evaluators must be proactive in developing a deep understanding of the social, cultural, and systemic contexts within which they work and make adjustments to their methods and interpretations accordingly.
The observations on evaluation practices and approaches that emerged from the necessity to adjust to the pandemic crisis have broad relevance and should be considered in all evaluation practice, not only during times of global crisis. Importantly, this book provides insights that can be applied to prepare evaluators and organisations for the inevitable future crises.
