Abstract
As editor of A Research Agenda for Evaluation, Peter Dahler-Larsen boldly opens the book by critiquing distinctions of evaluation as a logical process, a semi-professional field, a socio-political practice, and as evaluation research. The discussion points to the necessity of a ‘skeptical turn’ (p. 4), shining a light on the modernist and rational assumptions underpinning much of evaluation practice. From within this contextual framing arises the critical agenda for the future of evaluation research and the purpose of the book, being ‘to offer a fresh perspective on a new research agenda for evaluation, while taking complications and reflexivities onboard’ (p. 4).
Rather than presenting answers, a series of questions are provided to promote critical thinking and point toward areas of importance for this new research agenda. These areas include: • New social configurations of evaluation • Involvement and participation in evaluation • Evaluation data • Consequences of evaluation • Concepts and perspectives on evaluation • New practices of evaluation • Evaluation and democracy
Within the final chapter of the book, Dahler-Larsen returns to question ‘What does the future of evaluation look like?’ (p. 225). Here, the dual nature of the evaluation researcher is presented as someone who does evaluation, but also research on evaluation itself. This duality fosters reflexivity, enabling the future of evaluation to include acknowledgement of its contestability, subjectivity, and lack of neutrality. Dahler-Larsen postulates that ‘rather than being merely the further promotion of even more evaluation, research on evaluation may be a form of reflexivity that helps evaluation find its place in society – a place that is never going to be without controversy’ (p. 227–8).
The 11 chapters between Dahler-Larsen’s bookends evidence the reflexive work of evaluators where perspectives of not only ‘evaluation people’ but also ‘real people’ (p. 227) are included. The chapters cultivate diverse views on the research agenda questions posed at the outset. While I will give a summary of each of these 11 chapters, the value for the reader, as Dahler-Larsen suggests, is in reading the chapters themselves, to observe the tone and voice within and also the ‘the music arising from their interconnections, even their counterpoints’ (p. 9).
Whilst brief in length, Chapter 2 gives voice to Dr Fileberto Reynaldo Lopez from the Pascua Yaqui Nation via an interview conducted at an American Evaluation Association conference with Dahler-Larsen. Here, Lopez discusses what evaluation means to him as an American Indian, sharing insights on the modern Western conception of evaluation, from the outside. Clearly, to Lopez, ‘evaluation’ is a foreign concept, yet is artfully reconsidered away from traditional discipline related conceptions, towards a foundation within community and respect; respect for re-evaluating one’s self-presentation, how one carries integrity and their love for others. Highlighted within this chapter is the importance of acting locally and the potential to make authentic, positive social change when one does.
In Chapter 3, Jaakko Kauko and Mika K. T. Pajunen introduce the reader to the concept of the ‘thickening modern’ (p. 21), referencing modernism. The authors’ aim is to ‘identify items for a research agenda that recognize non-rationalistic epistemologies and then to ponder what this means for evaluation research’ (p. 22).
In Chapter 4, Estelle Raimondo claims to contribute to literature on evaluation systems in international organisations that identify positive and negative effects on the institutionalisation of evaluation. Here, the author furthers Dahler-Larsen’s commentary on the ‘skeptical turn’ by clarifying that it ‘means that evaluators should have the same level of scrutiny on their own practice and the systems in which they operate as they do when assessing the merit and worth of policies and programs’ (p. 44). Specifically, this chapter examines the institutionalisation of evaluation practices in the World Bank’s self-evaluation system. It does this to identify potential enablers and barriers to operating a ‘skeptical turn’ in practice.
Chapter 5 is centred around international organisations and the enactment of ‘non-knowledge’ in global governance (p. 63). Non-knowledge being the strategic making of (for example) uncertainty, ignorance or error (p. 63). Author Sotiria Grek proposes that ‘political sociology can become a productive tool for evaluation research in order to explain the simultaneous making of knowledge and non-knowledge’ (p. 65). The author shows, through case studies, how metrics play an important role in transnational governance.
Chapter 6, written by Lehn M. Benjamin, sets out a research agenda to move evaluation past its anchoring in ‘the program’, where beneficiaries are mere recipients, and toward enabling a fuller picture of beneficiaries as organisational actors in evaluation theory and practice. After recounting the history of the U.S. evaluation field and considering evaluation across the U.S. nonprofit sector, Benjamin offers five starting points to what evaluation research could do to move evaluation beyond the program, to the benefit of the participants. These include, for example, examining participant’s authority and influence within their settings, and how this affects the work of staff.
Chapter 7, written by Maria Ørskov Akselvoll and Peter Dahler-Larsen further develops some issues introduced in Chapter 1, and particularly considers the difference between ‘evaluation people’ and ‘real people’. The authors contend that in the evaluation society, ‘evaluation has become a pervasive phenomenon, and the lives of real people are affected by the world created for them by the evaluation people’ (p. 106). This distinction raises an empirical possibility toward analysing its impact upon social relations and inequality.
In Chapter 8, Jill Anne Chouinard maps the ecology of knowledge in collaborative practice to offer a critical perspective on a new research agenda for participatory approaches to evaluation. This significant chapter prompts the importance of questioning ‘…who can speak? What truths are created? Whose voices speak the truth? What is lost or occluded in this truth? And most important, who decides such questions?’ (p. 140). Chouinard challenges evaluators to look beyond technique and expertise, to see the limitations and restrictions of the current evaluation field and democratic possibilities that lie beyond. Here, a research agenda for rethinking epistemology is presented. In doing so, a research agenda for participative evaluation that questions who gets a voice in the construction of evaluative knowledge arises.
Chapter 9, written by Emily St. Denny, applies feminist evaluation principles to policy evaluation challenges and presents possibilities for future research. Denny ‘sets out to initiate a conversation about the value and possibility of developing a feminist policy evaluation approach’ (p. 147). To move this forward, the chapter concludes with a theory of action for feminist policy evaluation which promotes methodological pluralism, inclusive dialogue, and mobilising social change.
In Chapter 10, Ismael Ráfols and Andy Stirling explore ‘how quantification can be developed and embedded in evaluation so that it offers “plural and conditional” perspectives both to the evaluator and associated wider debate’ (p. 167). In doing so, the authors explore how indicators (e.g. evaluation indicators and performance indicators) might be used, not as rationalist tools of New Public Management (for example) but as emancipatory tools to open up, broaden out and pluralise evaluation.
In Chapter 11, Bénédicte Vidaillet critiques workplace evaluations and performance assessments, and, through drawing on the thinking of Lacan, questions the subconscious need that our desire to be evaluated at work fulfils. The purpose of this is two-fold, to understand the forms that modern-day evaluation at work takes and to postulate that the evaluation we apparently all desire, is indeed harmful. The author then sets out a series of hypotheses as to why we desire to be involved in something (workplace evaluation) that harms us.
In Chapter 12, Felicitas Hesselmann and Cornelia Schendzielorz explore how evaluations are shaped in language. Specifically, the authors explore how evaluations are constituted and transformed by talking about them, and they do this through analysing an exemplary case. The authors contend that ‘we, as researchers studying evaluations, publishing about evaluations, and discussing evaluations, also partake in reconfiguration of the evaluations we study, willingly or not’ (p. 221). The chapter prompts the reader to reflect on the position from which we speak.
As a whole, this book provides those interested in evaluation research much to reflect upon. Whilst individual chapters will resonate more with some than others, overall, the book surpasses its goal of providing a research agenda for evaluation. As an evaluation researcher interested in challenging dominant conceptions of evaluation and questioning ownership of evaluative rights to voice, I found much to spur my thinking within these pages. And, given the breadth of topics covered, I find it hard to envision other readers not being positively challenged in some way by the thoughtful voices herein.
ORCID iD
Kylie Kingston https://orcid.org/0000-0002-8859-758X