Abstract
An interview with Anthea Rutter
Members who have made a long-term and wide-ranging contribution to the Australian Evaluation Society (AES) and the field of evaluation are recognised through a special category of membership called ‘Fellow of the AES’.
Nan was made a Fellow in 2021. She has been in the evaluation profession since the late 1980s. For more than 20 years Nan has run an independent evaluation company in Wellington, Aotearoa New Zealand. Nan builds evaluation teams on a project-by-project basis. She brings together evaluators with a broad mix of skills and experience tailored to meet the evaluation, government, community or tribal context. Nan comes to evaluation with business, retail management, market research and social policy experience. She decided to set up an evaluation company because of the highly variable quality of evaluations she was managing and because she wanted to make a difference for Māori. Nan has been a member of the AES since the late 80s. As well as being co-opted onto the AES Board she was part of the Wellington Chapter and has been involved in the Aotearoa New Zealand Evaluation Association conferences and running workshops for the AES.
Tell me what brought you into the field of evaluation?
I fell into evaluation! I was working for a research company, and they also had a social research division – evaluation was very new, and I was really interested in it. I liked the idea that publicly funded programs and services should be scrutinised to make sure they were doing a good job and applauded the idea of accountability – to government, the taxpayer but mostly to the service users or program participants. I also wanted to find out how to support programs and agencies to deliver high quality services and programs to Māori.
In terms of your evaluation expertise what do you regard as your main areas of interest?
I think I’m probably best known for Indigenous evaluation (Wehipeihana, 2013) and Kaupapa Māori evaluation (Cram et al., 2015), developmental evaluation (Patton et al., 2015), evaluative rubrics (King et al., 2013) and developing culturally grounded evaluation frameworks (McKegg et al., 2012).
Looking back over your career, what have been the major challenges to your practice?
I think there have been many challenges. Two main challenges come to mind. Firstly, advocating for the validity and legitimacy of Kaupapa Māori and Indigenous knowledge systems has been a priority (Wehipeihana & McKegg, 2018). In the last decade particularly we’ve seen the visibility and position of Indigenous people and cultures come to the fore nationally and internationally, and this has positively influenced our practice as evaluators.
I think the other challenge that we all face is being a reflective practitioner (Wehipeihana et al., 2014). How do we keep learning and how do we incorporate this learning into our practice? Evaluation conferences, evaluation and research journals, and evaluation networks or communities of practices have been valuable for my learning. Also, my Kinnect Group 1 colleagues have been an important forum to grapple with new ideas and issues affecting evaluation and our practice.
What have been the highlights of your evaluation career?
I guess a highlight for me was my 2013 AES keynote. I shared a reflective model, now called shifting the locus of power, that helps people to reflect on their positioning in evaluation, and in different contexts and settings such as disability, education, health and with indigenous peoples. The ripple effects are still being felt and its influencing thinking in education, health and international development.
Being one of the early adopters of developmental evaluation has also been a highlight. I’ve presented on developmental evaluation, run developmental evaluation workshops across the globe and was a co-editor of the text Developmental evaluation exemplars: Principles in practice with Michael Quinn Patton and Kate McKegg. Michael was a generous and supportive co-editor and working alongside C. Deb Laughton from Guilford publishing was a valuable learning experience.
What have been the major influence(s) which have helped to define your evaluation practice?
My major influence has come from my parents. They were highly involved in iwi (tribal) activities and active contributors to school and community initiatives. I remember having foster kids coming to stay with us, embodying manaakitanga or an ethic of care my parents lived by. My parents modelled the responsibility we had to work for the good of our community and iwi. My mother was involved in the Catholic Church, so I was exposed in my late teens to structural analysis and liberation theology (a religious movement which applies religious faith by aiding the poor and oppressed through involvement in political and civic affairs).
John Owen, Patricia Rogers and Sue Funnell were part of my initial evaluation grounding as were the works of Michael Scriven, Michael Patton, Carole Weiss and Eleanor Chelimsky. In Aotearoa New Zealand people such as Linda Smith, Fiona Cram and Helen Moewaka Barnes advocating for Kaupapa Māori evaluation. I had the privilege of meeting and getting to know Jennifer Greene, Ernie House, Donna Mertens, Karen Kirkhart, Rodney Hopson and Stafford Hood. Their thinking was particularly impactful, in part because their thinking reflected and affirmed my own values and what I thought was important in evaluation.
When Jane Davidson returned home to Aotearoa New Zealand in 2004 she shared her approach to valuing with the New Zealand evaluation community. Jane communicated the complex ideas in Scriven’s logic of valuing in a ‘nuts and bolts’ down to earth way (Davidson, 2005). She provided well-reasoned and practical steps to integrate values more explicitly and effectively into our evaluation practice through the use of rubrics. This spawned a ‘rubric revolution’ in Aotearoa New Zealand and transformed my own practice (Davidson et al., 2011).
My Māori evaluation colleagues Kataraina Pipi, Louise Were, Debbie Goodwin and Laurie Porima have been supportive and kept me anchored to Māori and tribal ways of knowing and being.
How has the field of evaluation changed during your career?
In my time, we’ve seen more explicit values-anchored approaches to evaluation such as empowerment evaluation, transformative evaluation and culturally responsive evaluation. And we’ve extended our view of evaluation as a technocratic craft to evaluation serving the public good. There has been a more intentional response to issues such as social justice, equity and racism.
Also in the last decade, we’ve seen the emergence of systems thinking and complexity informed methods and approaches such as developmental evaluation. So, the scope of our evaluation practice sees us grappling with local, national and global ‘wicked problems’ challenges such as fresh water, food sovereignty, climate change and pandemics.
What are the main skills or competencies that evaluators need to have or develop to keep pace with emerging trends in evaluation practice?
I don’t think you get into or stay in evaluation unless you have an enquiring mind. So critical thinking or evaluative thinking as both a disposition and skillset is important.
My practice is more intentionally mixed methods, so evaluators need to have both quantitative and qualitative skills. Also, good facilitation skills are needed both for data collection and for engaging with communities and with our clients. A deep methodological toolbox and strong facilitation skills are needed.
Our clients are typically time poor. A key skill is communicating our findings and evaluative judgements simply and clearly. This includes different reporting formats such as single page info-graphic reports, summary reports, as well as full end of project evaluation reports.
Decision makers are often under time pressure when having to make program decisions either part way through the program or towards the end. They need information on the run, so progressive insights (e.g. oral presentations or written reports) are needed to inform decision-making. This is different from ‘the old’ mid and end of project reporting. Rapid insight cycles (Wehipeihana et al., 2022) was the methodology we developed to respond the need for fast feedback.
Increasingly evaluations need to provide economic, financial, and cost modelling information. We don’t have to have or be expert in all those areas ourselves, but we need to build teams and work with others – not always evaluators, to include these elements in the design and conduct of the evaluation.
What do you see as the main social issues/problems that evaluators ought to be thinking as about and seeking to resolve in the next decade?
Equity and the environment. Equity is about contributing to better world, not just for Māori but for all peoples and communities. The economic divide between the rich and poor is widening. I think it’s important to make a start. While planning is important, I think it’s important to get started and not wait for the perfect plan. The world is changing so quickly, that plans can rapidly become outdated. There is an environmental imperative – climate change and degradation threaten the environmental legacy we leave for future generations. Achieving the deliberate inclusion of environmental or climate change in non-climate or environment specific evaluations is still a work in progress.
Has your involvement in the AES been beneficial to you, and in what ways?
Membership in AES has been very beneficial. It has been an important learning community and evaluation community of practice. My early formal evaluation education was through AES conference and branch workshops as well as week-long courses in Aotearoa New Zealand. John Owen, Patricia Rogers and Sue Funnell were key educators. The branch workshops and networking events were important support and community building activities, particularly in the late 80s and early 90s when evaluation was new to the New Zealand public sector.
How can the society best position itself to still be relevant in the future?
I think it’s challenging. In Aotearoa New Zealand we have a high demand for evaluation. However, there are other competing fields such as co-design, data analytics and data science which carry out evaluation like activities, for example, assess performance or effectiveness without the grounding or theory that is core to evaluation. I think AES needs to help members stay focused on our core function and purposes – making evidence-based evaluative judgements and evaluation as a learning mechanism.
Perhaps collaborating with similar professional bodies, being engaged in state and national conversations, planning and strategy around evaluation – and supporting member involvement and contributions to these forums. More difficult I think is for AES to be more visible and finding ways which serve the profession and the world we want to create.
What advice would you give to an emerging evaluator?
Our theories and methods particularly our methods of valuing are a core attribute of our practice. This is what we utilise to make our evaluation judgements. However, they have to be sound so hone your craft and do the basics. You also need to read broadly, not just evaluation as it is important to understand the world and the issues, so whether it is social justice, climate change, innovation – it is important to be informed. In the past I’ve been part of various networking groups which meet monthly or bi-monthly: each person picks a topic, a chapter, a paper and a key question or questions guide the reading and reflection. Often it is something that I would not been aware of or picked up and read. Also, it is important to build a network of people that you can have conversations; about specific evaluation topics or more broadly. Read widely in the field and outside of it. Go to workshops. Be challenged in your thinking!