Evaluation Criteria for Evaluating Transformation: Implications for the Coronavirus Pandemic and the Global Climate Emergency

Abstract

Fundamental systems transformations are needed to address the global emergency brought on by climate change and related global trends, including the COVID-19 pandemic, which, together, pose existential threats to the future of humanity. Transformation has become the clarion call on the global stage. Evaluating transformation requires criteria. The revised Organization for Economic Cooperation and Development/Development Assistance Committee criteria are adequate for business as usual summative and accountability evaluations but are inadequate for addressing major systems transformations. Six criteria for evaluating transformations are offered, discussed, and illustrated by applying them to the pandemic and the Global Alliance for the Future of Food. The suggested criteria illustrate possibilities. The criteria for judging any intervention should be developed in the context of and aligned with the purpose of a specific evaluation and information needs of primary intended users. This article concludes that the greatest danger for evaluators in times of turbulence is not the turbulence—it is to act with yesterday’s criteria.

Keywords

criteria transformation systems change complexity global

Get full access to this article

View all access options for this article.

References

American Evaluation Association. (2018). Guiding principles for evaluators. https://www.eval.org/p/cm/ld/fid=51

Azzam

(2018). The application of the logic of evaluation to the real world: Understanding the four-step logic of evaluation process. Psychology Today. https://www.psychologytoday.com/us/blog/evaluation-social-betterment/201810/the-application-the-logic-evaluation-the-real-world

Bamberger

Mabry

(2020). Realworld evaluation (3rd ed.). Sage.

Bamberger

Segone

(2011). How to design and manage equity-focused evaluations. UNICEF.

Bamberger

Vaessen

Raimondo

(2016). Dealing with complexity in development evaluation: A practical approach. Sage.

Barry

J. M.

(2020, May 1). Politics and science. The Week, p. 17.

Better Evaluation. (2020). Adapting evaluation in the time of COVID-19. https://www.betterevaluation.org/en/blog/adapting-evaluation-time-covid-19-part-1-manage

Bitar

(2020, May 1). If not now, when? Evaluators as agents of change in the time of a global pandemic. May 1 Blog. http://khalil-bitar.com/index.php/blog/

Bowman

N. R.

Francis

C. D.

Tyndall

(2015). Culturally responsive indigenous evaluation a practical approach for evaluating indigenous projects in tribal reservation contexts. In Hood

Hopson

Frierson

(Eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice (pp. 335–359). Information Age.

10.

Capra

(2004). The hidden connections: Integrating the biological, cognitive, and social dimensions of life into a science of sustainability. Anchor.

11.

Capra

(2017, February 5). The science of connection. Uplift Blog. https://upliftconnect.com/science-of-connection/

12.

Cartier

K. M. S.

(2020, April 24). Eight lessons from COVID-19 to guide our climate response. EOS (science news) blog. https://eos.org/articles/eight-lessons-from-covid-19-to-guide-our-climate-response

13.

CDC (Centers for Disease Control and Prevention) (2020) Flattening the coronavirus curve. New York Times, March 27. https://www.nytimes.com/article/flatten-curve-coronavirus.html

14.

Chaplowe

(2020, May 4). Evaluation in the face of the climate emergency: Observations and synthesis. Blue Marble Evaluation Blog.

15.

Chelsky

Kelly

(2020, April 1). Bowling in the dark: Monitoring and evaluation during COVID-19 (coronavirus): Lessons from past experience can help creatively and responsibly adapt M&E practices. Independent Evaluation Group Blog. https://ieg.worldbankgroup.org/blog/mande-covid19

16.

Chouinard

J. A.

Cram

(2019). Culturally responsive approaches to evaluation: Empirical implications for theory and practice. Sage.

17.

Chouinard

Hopson

(2016). Decolonizing international development evaluation. Canadian Journal of Program Evaluation, 30(3, December).

18.

Christensen

C. M.

Raynor

M. E.

McDonald

(2015, September). What is disruptive innovation? Harvard Business Review.

19.

Climate and Traditional Knowledges Workgroup (CTKW). (2014). Guidelines for considering traditional knowledges in climate change initiatives. http://climatetkw.wordpress.com/

20.

Coffman

(2018). Equitable evaluation applies to all evaluation. Center for Evaluation Innovation. https://www.evaluationinnovation.org/insight/equitable-evaluation-applies-to-all-evaluation/

21.

Conway

(2019, February 11). Put climate in every nonprofit mission. Nonprofit Quarterly Blog.

22.

DAC Network on Development Evaluation. (2018). OECD DAC evaluation criteria: Summary of consultation responses. https://ieg.worldbankgroup.org/sites/default/files/Data/DAC-Criteria/ConsultationReport_EvaluationCriteria.pdf

23.

Davidson

E. J.

(2005). Evaluation methodology basics: the nuts and bolts of sound evaluation. Sage.

24.

Dean-Coffey

(2018). What’s race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation, 39(4), 527–542.

25.

Dean-Coffey

(2019). Evaluation in an equitable society: one person’s perspective. Evaluation and Program Planning, 75, 89–90.

26.

Dean-Coffey

(2020). How a seasoned consultant navigates social-justice-oriented evaluation. AEA365 blog, January 22. https://aea365.org/blog/how-a-seasoned-consultant-navigates-social-justice-oriented-evaluation-by-jara-dean-coffey/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+aea365+%28AEA365%29

27.

Donaldson

S. I.

Picciotto

(Eds.). (2016). Evaluation for an equitable society. Information Age.

28.

Duhigg

(2020, May 4). Seattle’s leaders let scientists take the lead. New York’s did not . The New Yorker, pp. 16–22.

29.

The Economics of Ecosystems and Biodiversity. (2018a). Measuring what matters in agriculture and food systems: A synthesis of the results and recommendations of TEEB for Agriculture and Food’s Scientific and Economic Foundations report. UN Environment.

30.

The Economics of Ecosystems and Biodiversity. (2018b). TEEB for agriculture & food: Scientific and economic foundations. UN Environment. http://teebweb.org/agrifood/home/scientific-and-economic-foundations-report/

31.

EEI (Equitable Evaluation Initiative) (2020). Equitable evaluation inititaive. https://www.equitableeval.org/

32.

Ehrenfeld

J. R.

(2005). “Eco-efficiency: Philosophy, theory and tools.” Journal of Industrial Ecology, 9(4), 6–8. https://doi.org/10.1162/108819805775248070

33.

Elkington

(2018, June 25). 25 years ago i coined the phrase “triple bottom line.” Here’s why it’s time to rethink it. Harvard Business Review.

34.

Ellis

E. C.

(2018). Anthropocene. Oxford University Press.

35.

Emerson

(2019). What’s in store for corporate diversity, equity, and inclusion in 2019? Paradigm blog, January 30. https://medium.com/inclusion-insights/what-in-store-for-corporate-diversity-equity-and-inclusion-in-2019-d3c1292dd920

36.

Euber

(2020, May 1). Unanticipated crisis lessons: Nonprofit’s pivot to online increases participation. NPQ Blog. https://nonprofitquarterly.org/unanticipated-crisis-lessons-nonprofits-pivot-to-online-increases-participation/?utm_source=NPQ+Newsletters&utm_campaign=95879bfe53-EMAIL_CAMPAIGN_2018_01_11_COPY_01&utm_medium=email&utm_term=0_94063a1d17-95879bfe53-12318013&mc_cid=95879bfe53&mc_eid=707ec28f50

37.

Eval4action (2020). The decade of evaluaton for action. https://www.eval4action.org/

38.

Fazey

(2020). Keynotes on transformation and sustainability. https://ioanfazey.com/keynotes/

39.

Fazey

Schäpke

Caniglia

Patterson

Hultman

van Mierlo

Säwe

Wiek

Wittmayer

Aldunce

Waer

H. A.

Battacharya

Bradbury

Carmen

Colvin

Cvitanovic

D’Souza

Gopel

Goldstein

Hämäläinen

(2018). Ten essentials for action-oriented and second order energy transitions, transformations and climate change research. Energy Research and Social Science, 40, 54–70.

40.

Feinstein

(2019). Dynamic evaluation. In van den Berg

R. B.

Magro

Mulder

S. S.

(Eds.), Evaluation for transformational change: Opportunities and challenges for the sustainable development goals (pp. 17–31). IDEAS. https://ideas-global.org/wp-content/uploads//2019/11/2019-11-05-Final_IDEAS_EvaluationForTransformationalChange.pdf

41.

Feinstein

(2020, April 28). Dynamic evaluation. Evaluation for Development Blog. https://zendaofir.com/dynamic-evaluation-for-societal-transformation/

42.

Food and Agriculture Organization. (2017). Sustainability pathways. http://www.fao.org/nr/sustainability/full-cost-accounting/e

43.

Ford Foundation (2018) Diversity, equity, inclusion. https://www.fordfoundation.org/about/people/diversity-equity-and-inclusion/

44.

Forss

Marra

Schwartz

(Eds.). (2011). Evaluating the complex: Attribution, contribution, and beyond (Comparative Policy Evaluation Series, vol. 18). Transaction.

45.

Fortune. (2020, May). The coronavirus economy, how the pandemic is reshaping every aspect of business. Fortune Magazine.

46.

Furubo

J.-E.

Rist

R. C.

Speer

(Eds.). (2013). Evaluation and turbulent times: Reflections on a discipline in disarray. Transaction.

47.

Global Alliance for the Future of Food (2020). The Global Alliance formally adopts a theory of transformation. https://futureoffood.org/the-global-alliance-makes-history-with-formal-adoption-of-a-theory-of-transformation/

48.

Global Environment Facility (2018). Strengthening monitoring and evaluation of climate change adaptation. https://www.thegef.org/publications/strengthening-monitoring-and-evaluation-climate-change-adaptation

49.

Gunderson

L. H.

Holling

C. S.

(2002). Panarchy: understanding transformations in human and natural systems. Island Press.

50.

Guterres

(2019, December). UN Secretary-General remarks at opening ceremony of UN Climate Change Conference COP25.

51.

Guterres

(2020, April 28). Collaborative leadership is critical to fight both coronavirus and climate change. TIME. https://time.com/5828535/un-chief-coronavirus-climate-change-collaboration/

52.

Hayes

D. J.

(2020). Flattening the global warming curve. NYU Energy & Environmental Impact Center.

53.

Heider

(2017). Rethinking evaluation. The World Bank Independent Evaluation Group. http://ieg.worldbankgroup.org/sites/default/files/Data/RethinkingEvaluation.pdf

54.

Hepburn

O’Callaghan

Stern

Stiglitz

Zenghelis

(2020). Will COVID-19 fiscal recovery packages accelerate or retard progress on climate change? Smith School Working Paper 20-02. https://www.smithschool.ox.ac.uk/publications/wpapers/workingpaper20-02.pdf

55.

Hernandez-Licona

(2019). Foreword. In van den Berg

R. B.

Magro

, & Mulder

S. S.

(Eds.), Evaluation for transformational change: Opportunities and challenges for the sustainable development goals (pp. x–xii). IDEAS, Exeter. https://ideas-global.org/wp-content/uploads//2019/11/2019-11-05-Final_IDEAS_EvaluationForTransformationalChange.pdf

56.

Hodgson

(2019, November 28). Transformation needs fresh perspectives [Blog post]. https://www.transformationsforum.net/transformation-needs-fresh-perspectives/

57.

Hodgson

(2020). Systems thinking for a turbulent world: A search for new perspectives. Routledge.

58.

Hood

Hopson

Frierson

(Eds.). (2015). Continuing the journey to reposition culture and cultural context in evaluation theory and practice. Information Age.

59.

Hussain

Vause

(2018). TEEB for agriculture & food: Background and objectives. In Hussain

Vause

(Eds.), TEEB for agriculture & food: Scientific and economic foundations (pp. 3–5). UN/FAO.

60.

Independent Evaluation Group. (2016). Supporting transformational change for poverty reduction and shared prosperity—Lessons from the World Bank experience. World Bank.

61.

Independent Evaluation Group. (2018). Evaluation of GEF support for transformational change. Global Environmental Facility. file:///C:/Users/Owner/Documents/Transformation/DAC%20criteria/transformational-engagement-2017-brief.pdf

62.

Independent Evaluation Group. (2020a). Evaluative resources and lessons to inform the COVID-19 response. The World Bank: IEG. http://ieg.worldbankgroup.org/topic/covid-19-coronavirus-response?utm_source=Independent+Evaluation+Group+Contacts&utm_campaign=fee9c85338-EMAIL_CAMPAIGN_2020_05_04_05_44_COPY_03&utm_medium=email&utm_term=0_fc6e7f2a32-fee9c85338-113584913

63.

Independent Evaluation Group. (2020b, April 1). Lessons from past experience can help creatively and responsibly adapt M&E practices. Independent Evaluation Group Blog. https://ieg.worldbankgroup.org/blog/mande-covid19

64.

The Investment Integration Project. (2020, May 12). Who is doing well, who isn’t: Evaluating the performance of asset managers against systemic social and environmental progress. Online Panel Discussion.

65.

International Development Evaluation Association. (2019). Prague declaration on evaluation for transformational change. International Development Evaluation Association, adopted on Friday, October 4. https://ideas-global.org/wp-content/uploads//2019/10/Prague-Declaration-4-October-2019.pdf

66.

Johnson

A. F.

(2019). (On the absence of) whiteness. AEA365 blog, June 25. https://aea365.org/blog/a-look-at-language-week-on-the-absence-of-whiteness-by-a-rafael-johnson/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+aea365+%28AEA365%29

67.

Joint Committee on Standards for Educational Evaluation. (2010). The program evaluation standards. http://www.jcsee.org/program-evaluation-standards

68.

Julnes

(Ed.), (2019a). Evaluating sustainability: Evaluative support for managing processes in the public interest. New Directions for Evaluation, 162 (Summer).

69.

Julnes

(2019b). Supporting transitions to sustainability: Evaluation for managing processes in the public interest. In Julnes

(Ed.), New Directions for Evaluation, 162, 119–154.

70.

Karalisi

(2020). Planning and evaluation during educational disruption: Lessons learned from covid-19 pandemic for treatment of emergencies in education. European Journal of Education Studies, 7(4), https://oapub.org/edu/index.php/ejes/article/viewFile/3047/5685

71.

Kennedy-Chouane

(2020). Evaluation criteria: What has changed, why, and why does it matter? UNICEF/EVALSDGS webinar, January 16. https://www.youtube.com/watch?v=r3jzS16D-QM&feature=youtu.be

72.

Kleissner

(2020, January 16). Impact investing for systems change. Impact Entrepreneur’s Luminarias Series.

73.

Kolbert

(2020, January 13). Don’t wait. The New Yorker, pp. 13–14.

74.

Lundgren

(2019). OECD evaluation norms and criteria. https://www.mofa.go.jp/mofaj/gaiko/oda/files/000447961.pdf

75.

Lydenberg

Burckart

(2020). Assessing system-level investments: A guide for asset owners. The Investment Integration Project. https://www.tiiproject.com/wp-content/uploads/2020/04/Assessing-System-Level-Investments_FINAL_04-21-2020.pdf

76.

McKegg

(2019). White privilege and the decolonization work needed in evaluation to support indigenous sovereignty and self-determination. Canadian Journal of Program Evaluation, 34(2, Fall), 357–367. https://doi.org/10.3138/cjpe.67978

77.

McKnight Foundation (2019). Diversity, Equity, Inclusion. https://www.mcknight.org/news-ideas/mcknight-foundation-statement-diversity-equity-inclusion/

78.

Mertens

D. M.

(1999). Inclusive evaluation: Implications of transformative theory for evaluation. American Journal of Evaluation, 20(1), 1–14.

79.

Mertens

D. M.

(2009). Transformative research and evaluation. Guilford

80.

Mertens

D. M.

Wilson

A. T.

(2018). Program evaluation theory and practice (2nd ed.). Guilford.

81.

Miller

R. L.

Campbell

(2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27(3), 296–319.

82.

Moore

G. F.

Evans

R. E.

Hawkins

Littlecott

Melendez-Torres

G. J.

Bonell

Murphy

(2019). From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation. Evaluation, 25(1), 23–45. https://doi.org/10.1177/1356389018803219

83.

Mora

Spirandelli

Franklin

E. C.

Lynham

Kantar

M. B.

Miles

Smith

C. Z.

Freel

Moy

Louis

L. V.

Barba

E. W.

Bettinger

Frazier

A. G.

Colburn

I. X. J. F.

Hanasaki

Hawkins

Hirabayashi

Knorr

Little

C. M.

Emanuel

(2018). Broad threat to humanity from cumulative climate hazards intensified by greenhouse gas emissions. Nature Climate Change, 8, 1062–1071.

84.

Mukherjee

(2020, May 4). After the storm: The pandemic has revealed dire flaws in American medicine. The New Yorker, 24–31.

85.

Müller

Sukhdev

(2018). Preface. In The economics of ecosystems and biodiversity (TEEB). (2018). Measuring what matters in agriculture and food systems: A synthesis of the results and recommendations of TEEB for Agriculture and Food’s Scientific and Economic Foundations Report (pp. 3–5). UN Environment.

86.

Natsios

(2010). The clash of the counter-bureaucracy and development. Center for Global Development. https://www.cgdev.org/publication/clash-counter-bureaucracy-and-development

87.

NY Times . (2020, April 14). Black Americans face alarming rates of coronavirus infection. https://www.nytimes.com/2020/04/07/us/coronavirus-race.html

88.

Oakes

L. E.

Olson

S. H.

Watson

(2020, April 22). Preventing pandemics, global warming and environmental degradation all at once. CNN. https://www.cnn.com/2020/04/22/opinions/pandemics-global-warming-linked/index.html

89.

Ofir

(2018a, February 15). Updating the DAC evaluation criteria, part 5. Non-negotiable criteria. Evaluation for Development Blog. http://zendaofircom/updating-dac-evaluation-criteria-part-5/

90.

Ofir

(2018b, June 8). Made in Africa evaluation, part 2: “Africa-rooted” evaluation. Evaluation for Development Blog.

91.

Ofir

(2018c, July 31). Evaluation for Development Blog. http://zendaofircom/zendas-top-ten-tips-for-yees-2/

92.

Ofir

(2018d, August 28). The NICE framework, part 1. Where culture and evaluation meet. Evaluation for Development Blog.

93.

Ofir

(2018e, October 13). Updating the DAC criteria. Evaluation for Development Blog. http://zendaofircom/dac-criteria-part-11/

94.

Ofir

(2020, March 29–April 20). Transforming evaluations and COVID-19. Evaluation for Development [4 blog posts]. https://zendaofir.com/transforming-evaluations-and-covid-19-part-4-accelerating-change-in-practice/

95.

Organization for Economic Cooperation and Development. (2013). The DAC network on development evaluation: 30 years of strengthening learning in development. OECD DAC Network on Development Evaluation.

96.

Organization for Economic Cooperation and Development. (2020). Evaluation criteria frequently asked questions. https://www.oecd.org/dac/evaluation/eval-criteria-faq.htm

97.

Organization for Economic Cooperation and DevelopmentSecretariat. (2002). Indicators to measure decoupling of environmental pressure from economic growth. Sustainable development. http://www.docstoc.com/docs/84838188/oecd_decoupling

98.

Organization for Economic Cooperation and Development/Development Assistance Committee Network on Development Evaluation. (2019a). Better criteria for better evaluation: Revised evaluation criteria definitions and principles for use. Author. www.oecd.org/dac/evaluation

99.

Organization for Economic Cooperation and Development/Development Assistance Committee Network on Development Evaluation (2019b). Evaluation criteria. https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

100.

Oxfam (2020). World’s billionaires have more wealth than 4.6 billion people. https://www.oxfam.org/en/press-releases/worlds-billionaires-have-more-wealth-46-billion-people

101.

Ozeki

Coryn

C. L. S.

Schröter

D. C.

(2019, October). Evaluation logic in practice: Findings from two empirical investigations of American Evaluation Association members. Evaluation and Program Planning, 76. https://www.sciencedirect.com/science/article/abs/pii/S0149718919301429

102.

Parsons

Dhillon

Keene

(2020). Visionary evaluation for a sustainable, equitable future. Information Age.

103.

Patton

M. Q.

(2008). Utilization-focused evaluation. Sage.

104.

Patton

M. Q.

(2011). Developmental evaluation: applying complexity concepts to enhance innovation and use. Guilford.

105.

Patton

M. Q.

(2015). Essentials of utilization-focused evaluation. Sage.

106.

Patton

M. Q.

(2016). What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term Commitment, integrity, and sensitivity in implementing evaluation models. American Journal of Evaluation, 37(2), 250–265.

107.

Patton

M. Q.

(2018). Principles-focused evaluation: The GUIDE. Sage.

108.

Patton

M. Q.

(2019a, December 2). Climate emergency from a Blue Marble Evaluation perspective. Blue Marble Evaluation Blog. https://bluemarbleeval.org/latest/climate-emergency-blue-marble-evaluation-perspective

109.

Patton

M. Q.

(2019b, December). Expanding futuring foresight through evaluative thinking. World Futures Review, 11(4), 296–307.

110.

Patton

M. Q.

(2019c). Transformation to global sustainability: Implications for evaluation and evaluators. New Directions for Evaluation, 162(Summer), 103–118.

111.

Patton

M. Q.

(2020a). Blue marble evaluation: Premises and principles. Guilford.

112.

Patton

M. Q.

(2020b, March 23). Evaluation implications of the coronavirus global health pandemic emergency. Blue Marble Evaluation Blog. https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency

113.

Patton

M. Q.

(2020c). Launch of Eval4Action. https://www.youtube.com/watch?v=SjsktJJjQss&feature=youtu.be

114.

Picciotto

(2009). Evaluating climate change and development. In van den Berg

R. D.

Feinstein

(Eds.), Evaluating climate change, and development (pp. 19–46). Transaction.

115.

Picciotto

(2013). The logic of development effectiveness: Is it time for the broader evaluation community to take notice? Evaluation, 19(2), 155–170.

116.

Picciotto

(2020a). From disenchantment to renewal. Evaluation, 26(1), 49–60.

117.

Picciotto

(2020b, January 23). The DAC criteria, personal email correspondence.

118.

Ramalingam

Singh

N. S.

Mahieu

Blanchet

(2020). Responding to covid-19. Guidance for humanitarian agencies. ALNAP.

119.

Richardson

(2018). Foreword. In Hussain

Vause

(Eds.), TEEB for agriculture & food: Background and objectives (In TEEB for Agriculture & Food: Scientific and Economic Foundations) (p. 6). UN Environment.

120.

Rockström

Sukhdev

(2016). How food connects all the SDGs. Stockholm Resilience Center. https://www.stockholmresilience.org/research/research-news/2016-06-14-how-food-connects-all-the-sdgs.html

121.

Rodin

(2014). The resilience dividend: Being strong in a world where things go wrong. The Rockefeller Foundation/Public Affairs.

122.

Rowe

(2019). Sustainability-ready evaluation: A call to action. New Directions for Evaluation, 162(Summer), 29–48.

123.

Rugg

Buehler

Renaud

Gilliam

Heitgerd

Westover

Wright-Deaguero

Bartholow

Swanso

(1999). Evaluating HIV prevention: A framework for national, state and local levels. American Journal of Evaluation, 20(1), 35–56. https://doi.org/10.1177/109821409902000104

124.

Rugg

(2015, March 13). Key role of evaluation in new development agenda. UN News. https://news.un.org/en/story/2015/03/493362

125.

Rugg

(2016). The role of evaluation at the UN and in the new sustainable development goals: Towards the future we want. Global Policy, 7(3), 426–430. https://doi.org/10.1111/1758-5899.12346

126.

Schwab

(2017). We need a new narrative for globalization. World Economic Forum, 3/17. https://www.weforum.org/agenda/2017/03/klaus-schwab-new-narrative-for-globalization/

127.

Schwandt

(2018, January 9). DAC evaluation criteria. Evaluation for Development Blog comments. http://zendaofircom/updating-dac-evaluation-criteria-part-5/

128.

Schwandt

(2019). Post-normal evaluation? Evaluation, 25(3), 317–329

129.

Scriven

(2016). The last frontier of evaluation: Ethics. In Donaldson

S. I.

Picciotto

(Eds.). Evaluation for an equitable society (pp. 11–48). Information Age.

130.

Segone

Tateossian

(2017). No one left behind. In Bamberegr

Segone

Tateossian

(Eds.) Evaluation for Agenda 2030: Providing evidence on progress and sustainability (pp. 23–34). https://ideas-global.org/wp-content/uploads/2017/12/Chapter-2.pdf

131.

Shanker

(2019a). Definitional tension: the construction of race in and through evaluation. Ph.D. Dissertation, University of Minnesota, Minneapolis.

132.

Shanker

(2019b). Tidying evaluation of “diversity” & “culture.” AEA365 blog, June 24. https://aea365.org/blog/a-look-at-language-week-tidying-evaluation-of-diversity-culture-by-vidhya-shanker/

133.

Slaper

T. F.

Hall

T. J.

(2011). The triple bottom line: What is it and how does it work? Indiana Business Review, 86(1). https://www.ibrc.indiana.edu/ibr/2011/spring/article2.html

134.

Smith

L. T.

(1999). Decolonizing methodologies: Research and indigenous peoples. Zed.

135.

Stame

(2018). Strengthening the ethical expertise of evaluators. Evaluation, 24(4), 438–451.

136.

Steel

(2018). This is the ‘last generation' that can save nature. CNN News. https://www.cnn.com/2018/10/29/health/wwf-wildlife-population-report-intl/index.html

137.

Suarez

(2020, May 5). Sustainment: Moving beyond sustainability. NPQ Blog. https://nonprofitquarterly.org/sustainment-moving-beyond-sustainability/?utm_source=NPQ+Newsletters&utm_campaign=8ad3ae2f2d-EMAIL_CAMPAIGN_2018_01_11_COPY_01&utm_medium=email&utm_term=0_94063a1d17-8ad3ae2f2d-12318013&mc_cid=8ad3ae2f2d&mc_eid=707ec28f50

138.

Systems in Evaluation. (2018). Principles for effective use of systems thinking in evaluation. Systems in Evaluation Topical Interest Group, American Evaluation Association. https://www.systemsinevaluation.com/wp-content/uploads/2018/10/SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf

139.

Taleb

N. N.

(2018). Skin in the game: Hidden asymmetries in daily life. Random House.

140.

TCC Group (2019). Equity and evaluation: Models of how equity can and does impact evaluation. https://www.equitableeval.org/blog-main/2019/4/17/equity-and-evaluation-models-of-how-equity-can-and-does-impact-evaluation

141.

Tolley

L. M.

(2020, May 3). Learning, evaluating, and staying strong together. AEA365 Blog.

142.

Uitto

J. I.

(2019). Sustainable development evaluation: understanding the nexus of natural and human systems. New Directions for Evaluation, 162, 49–68.

143.

United Nations Environment Program. (2019). The United Nations emissions gap report (released November 26). https://www.unenvironment.org/resources/emissions-gap-report-2019

144.

van den Berg

R. D.

Feinstein

(Eds.). (2009). Evaluating climate change, and development. Transaction Publishers.

145.

van den

Berg

R. B.

Magro

Mulder

S. S.

(Eds.). (2019). Evaluation for transformational change: Opportunities and challenges for the sustainable development goals. IDEAS. https://ideas-global.org/wp-content/uploads//2019/11/2019-11-05-Final_IDEAS_EvaluationForTransformationalChange.pdf

146.

Vataja

Parkkonen

(2019, November 12). Evaluation as a reinforcer of democratic ideals: What is the role and purpose of evaluation in our post-normal era? Helsinki: The Finnish Innovation Fund Sitra. https://www.sitra.fi/en/blogs/evaluation-as-a-reinforcer-of-democratic-ideals/

147.

Vidueira

(2020, April 23). Aftermath of the 50th anniversary of the blue marble (earth) day. Blue Marble Evaluation Blog. https://bluemarbleeval.org/latest/aftermath-50th-anniversary-blue-marble-earth-day

148.

Westley

F. R.

Tjornbo

Schultz

Olsson

Folke

Crona

Bodin

Ö.

(2013). A theory of transformative agency in linked social-ecological systems. Ecology and Society, 18(3), 27. http://dx.doi.org/10.5751/ES-05072-180327

149.

Westley

F. R.

Zimmerman

Patton

M. Q.

(2006). Getting to maybe: how the world is changed. Random House Canada.

150.

Williams

(2018, January 7). DAC evaluation criteria. Evaluation for Development Blog comments. http://zendaofircom/updating-dac-evaluation-criteria-part-5/