Abstract
While evaluation is an established discipline, methodologies and approaches vary considerably depending on context, commissioner requirements, evaluator expertise and the culture of evaluation organisations. The field is continually developing with innovations and adaptations emerge, often in the pursuit of improved quality or impact. The articles in this first issue of the Evaluation Journal of Australasia for 2024, represent some of the cutting edge methodologies and innovations that are evident in evaluation practice around the globe.
Based on a keynote address to the Australian Evaluation Society Conference in 2023, the special feature by Andrew Leigh reports on new developments initiated by the Australian Government through the newly formed Australian Centre for Evaluation. A key feature of its agenda will be to conduct evaluations using Randomised Control Trials. While Leigh acknowledges that RCTs are not always feasible or practical, he argues that this approach avoids self-selection bias and the inherent problems associated with observational studies. In my reading of evaluation literature, RCTs (and quantitative studies more generally) are largely missing. Leigh argues that the Centre will support ‘high quality’ evaluations. While I don’t think it is his intention, the assertion suggests that evaluations that are not based on RCTs or presumably some other quantitative methodology, are not ‘high quality’. While I see a role for RCTs in evaluation practice, I cannot agree with this conclusion. I have been involved with and read hundreds of papers that report on what any reasonable assessment would be described as ‘high quality’ qualitative and mixed methods studies. Nevertheless, I look forward to reading about what the Centre produces in the future.
Sandra Ayoo, Meghan Leeming and Stacy Huff also raise the issue of ‘high-quality’ evaluation in their article about program evaluation standards through the United Nations Evaluation Quality Assessment. The one finding that struck me in this article was the lack of correlation between assessed quality of evaluations examined, and the management response to those evaluations. This may suggest that the impact of an evaluation report is not determined by how good the methodology is, how reliable the data are, or how strong the conclusions are, but on something else. The author suggest that “rejected recommendations were often related to budget or administrative concerns or inadequate evaluator understanding of the intervention and its contexts” (p. 35). I suspect it goes beyond this, to how credible or trustworthy the evaluator is perceived to be (Guenther & Falk, 2021). My point is—at least partially supported by Ayoo and colleagues—that ‘quality’ is in the eye of the beholder, not in quality standards. That is, no matter how ‘good’ the evaluation is, recommendations for change will not be adopted without a corresponding perception that the case for change is warranted.
Sandra Opoku and Genevieve Heard’s article discusses evaluation innovation in the context of a developmental evaluation of a men’s behaviour change program during COVID-19. This required a rapid shift to online delivery. Perhaps surprisingly, the authors do not point to limitations created by the pandemic, but rather to the benefits of developmental evaluation in adapting the program to the changed circumstances created by the pandemic. Indeed the authors indicate that participants felt that the program met its objectives despite the adaptations to online delivery. While we might not be able to say from this article that the evaluation was responsible for program delivery change, we can say it is a good example of evaluation supporting innovation and adaption.
The article from Ralph Renger, Jessica Renger, Melanie Pescud, Richard Van Eck, Marc Basson and Jirina Renger focuses on evaluations of complex interventions. There is extensive literature on evaluation and complexity with some focus on systems thinking (for example Roche et al., 2021; Stack et al., 2018) but this article perhaps for the first time raises the prospect of an alternative to outcomes measurement towards measurement of what the authors describe as ‘emergent properties’. While there is undoubtedly more to be done to theorise this positioning of evaluating complex interventions, the article offers a novel alternative—or perhaps a novel adaptation—of traditional outcomes assessments in complex program environments.
Anthea Rutter’s evaluator perspective is from Nan Wehipeihana, who has been a member of the AES since the 1980s. In 2021 she was made a Fellow of the Association. Nan suggested that her involvement in evaluation was driven, at least in part, by a need to support high quality delivery program delivery to Māori. She also reflects on equity and the environment as two key social issues that require attention, and in relation to environmental concerns and evaluation, I agree with her assessment that this is a “work in progress” (p. 74). Given the imminent risks associated with climate change, there is a pressing need for independent evaluation of policy and programs in this space.
Rounding out this issue, Yvonne Zurynski’s book review of Policy Evaluation in the Era of COVID-19, edited By Pearl Eliadis, Indran A. Naidoo and Ray C. Rist, reflects on a series of international case studies. In terms of the role of evaluation in a COVID-19 crisis, the lessons learned point to the need for nimble, responsive evaluations to inform timely decision making, to highlight ethical and social justice issues, and to provide accountability. The book ends with a series of observations, which are primarily concerned with the challenges for government policy in a crisis environment. What surprises me somewhat is a lack of consideration of evaluation’s role to inform innovative policy responses. I note that there is mention of innovative, adaptive and flexible evaluation practice (particularly in Chapter 2), and this is certainly a recurring theme in articles published in this journal (for example D’Onise & Pontifex, 2023; Olson et al., 2021), including the article by Opoku and Heard in this issue.
The collection of articles in this issue could be seen as an eclectic and random mix of reports of methodologies, approaches, theories and results. However, I prefer to look at them as a representation of the inherent diversity and interdisciplinarity of the field: from RCTs, to meta-evaluations, and developmental evaluation. What is also clear is that evaluators represented in these articles are concerned about strengthening the quality and utility of evaluation to inform policy and improve program practices. Creative thinking is also represented in these articles, leading to innovations in approaches, theory, methods and outcomes from evaluation.