Evaluation that supports social, ecological, and governance systems change and transformation raises ethical questions about what and whose worldviews do and should ground evaluative processes. This article illustrates one approach to ethical analysis within a case study of the first phase of an initiative to co-create a monitoring, evaluation, and learning system. The case drew on the principles of Blue Marble Evaluation in partnership with local staff and Indigenous leaders of the Amazon Sacred Headwaters Initiative. The approach uses critical and relational systems thinking to examine the sources of motivation, power, knowledge, and legitimacy that influence and should influence an evaluation system. Results reframe typical early phase evaluation process work from a contractual agreement to a co-created ethical space that engenders the legitimacy of the evaluation process. Contributions include a conceptual framework and process for ethical analysis that could be adapted by others.
AdlerPSHeckscherCMcCarthyJE, et al. (2015) The mutations of professional responsibility: Toward collaborative community. In: MitchellDEReamRK (eds) Professional Responsibility: The Fundamental Issue in Education and Healthcare Reform. Berlin: Springer, pp. 309–326.
2.
AhenakewCR (2017) Mapping and complicating conversations about indigenous education. Diaspora, Indigenous and Minority Education11(2): 80–91.
3.
Amazon Sacred Headwaters Initiative (ASHI) (2021) Bioregional plan 2020: Ecological transitions for the Amazon Sacred Headwaters of Ecuador and Peru. Available at: https://cuencasagradas.org/bioregional_plan/
4.
BartlettCMarshallMMarshallA (2012) Two-eyed seeing and other lessons learned within a co-learning journey of bringing together indigenous and mainstream knowledges and ways of knowing. Journal of Environmental Studies and Sciences2(4): 331–40.
5.
BowmanN (2020) Nation-to-nation in evaluation: Utilizing an indigenous evaluation model to frame systems and government evaluations. New Directions for Evaluation2020(166): 101–18.
6.
BustamanteMVidueiraPBakerL (2021) Systems thinking and complexity science–informed evaluation frameworks: Assessment of the economics of ecosystems and biodiversity for agriculture and food. New Directions for Evaluation2021(170): 81–100.
7.
CabreraDCabreraLMidgleyG (2022, forthcoming) The four waves of systems thinking. In: Cabrera, et al. (eds) Routledge Handbook of Systems Thinking. London: Routledge.
8.
ChurchmanCW (1979) The Systems Approach and Its Enemies. London: Basic Books.
9.
CramF (2018) Conclusion: Lessons about indigenous evaluation. New Directions for Evaluation2018(159): 121–33.
10.
CrutzenPStoermerE (2013) The “Anthropocene.” In: RobinLSörlinSWardeP (eds) The Future of Nature: Documents of Global Change. New Haven, CT: Yale University Press, 479–90.
11.
Dean-CoffeyJ (2018) What’s race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation39(4): 527–42.
12.
EarlSCardenFSmutyloT (2001) Outcome Mapping: Building Learning and Reflection into Development Programs. Ottawa, ON, Canada: International Development Research Centre.
13.
ErmineW (2007) The ethical space of engagement. Indigenous Law Journal6(1): 193–203.
14.
ErmineWSinclairRJeffreyB (2004) The ethics of research involving indigenous peoples. Report of the Indigenous Peoples’ Health Research Centre to the Interagency Advisory Panel on Research Ethics. Available at: http://gfbr.global/wp-content/uploads/2015/09/GFBR9.pdf
15.
EscobarA (2014) Sentipensar con la tierra. Nuevas Lecturas Sobre Desarrollo, Territorio Y Diferencia. Medellín: Ediciones UNAULA.
FazeyISchäpkeNCanigliaG, et al. (2018) Ten essentials for action-oriented and second order energy transitions, transformations and climate change research. Energy Research & Social Science40: 54–70.
18.
FisherCP (2022) Culture and dreaming: A story of co-creation. International Journal of Applied Psychoanalytic Studies19: 230–40.
19.
FlyvbjergB (2001) Making Social Science Matter. Cambridge: Cambridge University Press.
20.
FolkeCPolaskySRockströmJ, et al. (2021) Our future in the Anthropocene biosphere. Ambio50(4): 834–69.
21.
GatesEFWaltonMVidueiraP, et al. (2021) Introducing systems- and complexity-informed evaluation. In: GatesEFWaltonMVidueiraP (eds), Systems and Complexity-Informed Evaluation: Insights from Practice. New Directions for Evaluation, 2021, 13–25.
22.
GatesEFFils-AimeF (2021) System change evaluation: Insights from The Rippel Foundation and its ReThink Health initiative. In: GatesEFWaltonMVidueiraP (eds), Systems and Complexity-Informed Evaluation: Insights from Practice. New Directions for Evaluation, 2021, 125–138.
23.
GatesEF (2018) Towards valuing with critical systems heuristics. American Journal of Evaluation39(2): 201–220.
24.
GoodchildM (2021) Relational systems thinking: That’s how change is going to come, from our earth mother. Journal of Awareness-based Systems Change1(1): 75–103.
25.
HallME (2020) Blest be the tie that binds. New Directions for Evaluation2020(166): 13–22.
26.
HargreavesM (2018) Leveraging Systemic Change: Evaluating What Works. Chicago, IL: NORC at the University of Chicago.
27.
HepiMFooteJAhuriri-DriscollA, et al. (2021) Enhancing cross-cultural evaluation practice through kaupapa Māori evaluation and boundary critique: Insights from Aotearoa New Zealand. New Directions for Evaluation2021(170): 51–65.
28.
HopsonRKKirkhartKEBledsoeK (2011) Decolonizing evaluation in a developing world: Implications and cautions for equity-focused evaluation. In: SegoneM (ed.) Evaluation for Equitable Development Results. New York: UNICEF, 59–82.
29.
HummelbrunnerR (2015) Learning, systems concepts and values in evaluation: Proposal for an exploratory framework to improve coherence. IDS Bulletin46(1): 17–29.
30.
Intergovernmental Panel on Climate Change (IPCC) (2022). Summary for policymakers. In: PörtnerH-ORobertsDCPoloczanskaES (eds) Climate Change 2022: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge: Cambridge University Press. Available at: https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_SummaryForPolicymakers.pdf
31.
JacksonMC (2019) Critical Systems Thinking and the Management of Complexity: Responsible Leadership for a Complex World. London: John Wiley & Sons.
32.
LanderE (2000) Ciencias Sociales: Saberes Coloniales Y Eurocéntricos. Buenos Aires, Argentina: Clacso.
33.
Levin-RozalisM (2014) A purpose-driven action: The ethical aspect and social responsibility of evaluation. New Directions for Evaluation146: 19–32.
34.
LevinsonM (2016) Normative case studies as both source and method for action-guiding theory. In: Keynote paper, philosophy of education society of Great Britain annual conference, Oxford, 1–3 April.
35.
McKeggKWehipeihanaN (2016) Chapter 14: Developmental evaluation in synthesis: Practitioners’ perspectives. In: PattonMQMcKeggKWehipeihanaN (eds) Developmental Evaluation Exemplars: Principles in Practice. New York: Guilford Press, pp. 271–288.
36.
MidgleyG (1997) Dealing with coercion: Critical systems heuristics and beyond. Systems Practice10(1): 37–57.
37.
MorrisM (ed.) (2008) Evaluation Ethics for Best Practice: Cases and Commentaries. New York: Guilford Press.
38.
NobreCABormaLDS (2009) “Tipping points” for the Amazon forest. Current Opinion in Environmental Sustainability1(1): 28–36.
39.
OfirZRuggD (2021) International developments in evaluation: Transforming evaluation for times of global transformation. American Journal of Evaluation42(1): 47–52.
40.
OlsenSBPageGOchoaE (2009) The analysis of governance responses to ecosystem change: A handbook for assembling a baseline. LOICZ reports & studies no. 34. Geesthacht: GKSS Research Center.
41.
OlssonPGalazVBoonstraWJ (2014) Sustainability transformations: A resilience perspective. Ecology and Society19(4): 26269651.
42.
OlssonPMooreM-LWestleyFR, et al. (2017) The concept of the Anthropocene as a game-changer: A new context for social innovation and transformations to sustainability. Ecology and Society22(2): 26270142.
PageGGWiseRMLindenfeldL, et al. (2016) Co-designing transformation research: Lessons learned from research on deliberate practices for transformation. Current Opinion in Environmental Sustainability20: 86–92. https://doi.org/10.1016/j.cosust.2016.09.0
PageGWiseRMLindenfeldL, et al. (2016) Co-designing transformation research: Lessons learned from research on deliberate practices for transformation. Current Opinion in Environmental Sustainability20: 86–92. https://doi.org/10.1016/j.cosust.2016.09.0
47.
PalomoILocatelliBOteroI, et al. (2021) Assessing nature-based solutions for transformative change. One Earth4: 730–41.
48.
PanikkarR (1996) Religión, Filosofía y Cultura. Ilu. Revista De Ciencias De Las Religiones: 1: 125–48.
49.
ParsonsBDhillonLKeeneM (eds) (2020) Visionary Evaluation for a Sustainable, Equitable Future. A Volume in Evaluation and Society. Charlotte, NC: Information Age Publishing, Inc.
50.
PattonMQ (2020) Blue Marble Evaluation: Premises and Principles. New York: Guilford Press.
51.
PattonMQ (2021) Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation42(1): 53–89.
52.
PicciottoR (2020) Towards a complexity framework for transformative evaluation. Journal of Multidisciplinary Evaluation16(35): 54–76.
53.
ReynoldsM (2014) Equity-focused developmental evaluation using critical systems thinking. Evaluation20(1): 75–95.
54.
ReynoldsMGatesEFHummelbrunnerR, et al. (2016) Towards systemic evaluation. Systems Research and Behavioral Science33(5): 662–73.
55.
RoordaMGullicksonAM (2019) Developing evaluation criteria using an ethical lens. Evaluation Journal of Australasia19(4): 179–94.
56.
RoweA (2019) Sustainability-ready evaluation: A call to action. New Directions for Evaluation2019(162): 29–48.
57.
SantosB (2010) Descolonizar el Saber, Reinventar el poder. Montevideo, Uruguay: Ediciones Trilce.
58.
SantosBDS (2011) Epistemologías del sur. Utopía Y Praxis Latinoamericana16(54): 17–39.
59.
SasakamooseJBellegardeTSutherlandW, et al. (2017) Miyo-pimātisiwin developing indigenous cultural responsiveness theory (ICRT): Improving indigenous health and well-being. The International Indigenous Policy Journal8(4): 1.
60.
SayerA (2011) Why Things Matter to People: Social Science, Values and Ethical Life. Cambridge: Cambridge University Press.
61.
Schmidt-AbbeyBReynoldsMIsonR (2020) Towards systemic evaluation in turbulent times—Second order practice shift. Evaluation26(2): 205–26.
62.
SchwandtTA (2018) Evaluative thinking as a collaborative social practice: The case of boundary judgment making. New Directions for Evaluation2018(158): 125–37.
SchwandtTAGatesEF (2016) What can evaluation do? An agenda for evaluation in service of an equitable society. In: DonaldsonSIPicciottoR (eds), Evaluation for an equitable society. Charlotte, NC: Information Age.
65.
SchwandtTAGatesEF (2018) Case study methodology. In: DenzinNKLincolnYS (eds), Handbook of Qualitative Research5th ed. (pp. 341–358). Thousand Oaks, CA: SAGE.
66.
SchwandtTAGatesEF (2021) Evaluating and Valuing in Social Research. New York, NY: Guilford Press.
67.
ScrivenM (2016) The last frontier of evaluation: Ethics. In: DonaldsonSIPicciottoR (eds) Evaluation for an Equitable Society. Charlotte, NC: Information Age Publishing, Inc., pp. 11–48.
68.
ThacherD (2006) The normative case study. American Journal of Sociology111: 1631–76.
ThomasVGMadisonARockcliffeF, et al. (2018) Racism, social programming, and evaluation: Where do we go from here?American Journal of Evaluation39(4): 514–26.
71.
UittoJI (2021) Surviving the Anthropocene: How evaluation can contribute to knowledge and better policymaking. Evaluation27(4): 436–52.
72.
UlrichW (1983) Critical Heuristics of Social Planning: A New Approach to Practical Philosophy. Bern: Swiss National Science Foundation.
73.
UlrichW (2000) Reflective practice in the civil society: The contribution of critically systemic thinking. Reflective Practice1: 247–68.
UlrichWReynoldsM (2010) Critical systems heuristics. In: ReynoldsMHolwellS (eds) Systems Approaches to Managing Change: A Practical Guide. London: Springer, 243–92.
76.
van den BergRMargoCSalinas MulderS (eds) (2019) Evaluation for Transformational Change: Opportunities and Challenges for the Sustainable Development Goals. Exeter: IDEAS.
77.
VisserM (2017) Learning and unlearning: A conceptual note. The Learning Organization24(1): 49–57.
78.
WalshC (2005) Interculturalidad, conocimientos y decolonialidad. Signo Y Pensamiento24(46): 39–50.
79.
WestSSchillC (2022) Negotiating the ethical-political dimensions of research methods: A key competency in mixed methods, inter- and transdisciplinary, and co-production research. Humanities and Social Science Communications9(294): 1–11.
80.
WestleyFOlssonPFolkeC, et al. (2011) Tipping toward sustainability: Emerging pathways of transformation. AMBIO40(7): 762.
81.
WilliamsADickmanJSmurthwaiteR (2021) Advancing evaluation and learning on transformational change: Lessons from the climate investment funds’ transformational change learning partnership. American Journal of Evaluation42(1): 90–109.
82.
WilliamsB (2015) Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS Bulletin46: 7–16.
83.
WilliamsBHummelbrunnerR (2010) Systems Concepts in Action: A Practitioner’s Toolkit. Stanford, CA: Stanford University Press.
84.
YunkaportaT (2020) Sand Talk: How Indigenous Thinking Can Save the World, 1st edn. New York: HarperOne, an imprint of HarperCollins Publications.
85.
ZazuetaAELeTTBahramalianN (2021) Development trajectories and complex systems–informed theories of change. American Journal of Evaluation42(1): 110–29.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
