Sage Journals: Discover world-class research

Abstract

Amid calls for evaluations to advance equity, there are ongoing debates, varied guidance, and limited empirical research on how evaluators practically attend to equity in their work. This article identifies ethical questions—about the right thing to do when there are multiple options—that arise when evaluators attend to equity and factors that influence how evaluators interpret and address these questions. Our identification of questions and factors comes from a qualitative analysis of semi-structured individual interviews with 21 evaluators based in New England, U.S., predominantly White women representing multiple practice areas. Ethical questions address evaluators’ expertise, negotiation with funders, methodological norms, use of data about disparities, and influence of evaluators’ worldviews in cross-cultural settings. The ways evaluators navigate these questions are influenced by their workplace, professional roles, interpersonal relationships, power differences, and available time. Findings pose implications for evaluator training and future directions for research on equity and ethics.

Keywords

ethics equity research on evaluation evaluation practice professional guidelines

Get full access to this article

View all access options for this article.

References

American Evaluation Association (AEA) (2018). American Evaluation Association: Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles

Anderson

M. A.

Mastri

(2021). Culturally responsive and equitable evaluation for federal evaluation staff. Mathematica. www.mathematica.org/publications/culturally-responsive-and-equitable-evaluation-for-federal-evaluation-staff

Andrews

Parekh

Peckoo

(2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Child Trends. https://cms.childtrends.org/wp-content/uploads/2019/09/RacialEthnicEquityPerspective_ChildTrends_October2019.pdf

Annie E. Casey Foundation (2014). Race equity and inclusion action guide. Embracing racial equity: 7 steps to advance and embed race equity and inclusion within your organization. Issue Lab. https://search.issuelab.org/resource/race-equity-and-inclusion-action-guide-embracing-equity-7-steps-to-advance-and-embed-race-equity-and-inclusion-within-your-organization

Bamberger

Segone

(2011). How to design and manage equity-focused evaluations. UNICEF. https://agora.unicef.org/course/info.php?id=1238

Bowleg

(2008). When Black + lesbian + woman ≠ Black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles: A Journal of Research, 59(5-6), 312–325. https://doi.org/10.1007/s11199-008-9400-z

Boyce

A. S.

Reid

Avent

Adetogun

Moller

J. R.

Singletary

B. H.

(2022). Social justice as ontology: The intersection of Black evaluators’ identities, roles, and practice. American Journal of Evaluation, 44(3), 528–548. https://doi.org/10.1177/10982140221108664

Brown

M. J.

(2020). Science and moral imagination: A new ideal for values in science. University of Pittsburgh Press.

Carter

C. E.

Hossain

(2020). How to apply a diversity, equity, & inclusion lens to strengthen evidence building in the social sector. Project Evident. https://toolkit.projectevident.org/p/how-to-apply-a-diversity-equity-inclusion-lens-to-strengthen-evidence-building-in-the-social-sector

10.

Center for Evaluation Innovation, Institute for Foundation and Donor Learning, Dorothy A Johnson Center for Philanthropy, and Luminare Group (2017). The Equitable Evaluation Framework™ framing paper. Equitable Evaluation Initiative. https://www.equitableeval.org/_files/ugd/21786c_04631f8cfdd44cb4a10a4097bf85acd5.pdf

11.

Cerna

Condliffe

Wilson

(2021). Guiding questions for supporting culturally responsive evaluation practices and an equity-based perspectives. MDRC. https://www.mdrc.org/work/publications/guiding-questions-supporting-culturally-responsive-evaluation-practices-and

12.

Chicago Beyond (2018). Why am I always being researched? https://chicagobeyond.org/researchequity/

13.

Chouinard

J. A.

Cousins

J. B.

(2021). Developing an ethical rationale for collaborative approaches to evaluation. Evaluation, 27(3), 364–381. https://doi.org/10.1177/1356389020978501

14.

Coleman

S. E.

Byrd

Scaccia

J. P.

Stout

Schall

Callender

, … Mann

(2017). Engaging community residents with lived experience. Institute for Healthcare Improvement. https://app.ihi.org/Marketing/program_documents/P2PH/Engaging-Community-Residents-with-Lived-Experience_7.10.17.pdf

15.

Cooksy

L. J.

(2007). Ethical challenges: Should we call the whole thing off? American Journal of Evaluation, 28(1), 76–78. https://doi.org/10.1177/1098214006298063

16.

Cooksy

L. J.

(2009). Evaluators’ reflections on the ethical implications of their early experiences. American Journal of Evaluation, 30(4), 572–574. https://doi.org/10.1177/1098214009349794

17.

Dahler-Larsen

(2021). Chapter 1: Introduction to a research agenda for evaluation: Inspirational themes. In: Dahler-Larsen

(Ed.), A research agenda for evaluation (pp. 1–14). Edward Elgar Publishing Limited. https://doi.org/10.4337/9781839101083.00006

18.

Dean-Coffey

Coné

(2023). The equitable evaluation framework: May 2023 expansion. https://www.equitableeval.org/framework

19.

Desautels

Jacob

(2012). The ethical sensitivity of evaluators: A qualitative study using a vignette design. Evaluation, 18(4), 437–450. https://doi.org/10.1177/1356389012461192

20.

EvalIndigenous (2021). 10 questions Indigenous communities should ask evaluators. The Indigenous Network for Evaluation. https://www.evalforward.org/sites/default/files/document/2023-05/10%20Questions%20Indigenous%20communities%20should%20ask%20evaluators.pdf

21.

Farrow

Morrison

(2019). Placing equity concerns at the center of knowledge development. Center for the Study of Social Policy. https://cssp.org/wp-content/uploads/2019/05/Putting-Equity-at-the-Center-of-Knowledge-Development.pdf

22.

Gates

E. F.

Madres

Hall

J. N.

Benitez Alvarez

(2022a). It takes an ecosystem: Socioecological factors influencing equity-oriented evaluation in New England, U.S., 2021. Evaluation and Program Planning, 92, https://doi.org/10.1016/j.evalprogplan.2022.102068

23.

Gates

E.F.

Williamson

Madres

Hall

J.N.

Benitez Alvarez

(2022b). Equity “on the sideline”: A mixed methods study of New England evaluation practice in 2020. Journal of Multidisciplinary Evaluation, 18(42), 3–33. https://doi.org/10.56645/jmde.v18i42.715

24.

Goodyear

(2012). Resources to support ethical practice in evaluation: An interview with the director of the National Center for Research and Professional Ethics. American Journal of Evaluation, 33(3), 459–461. https://doi.org/10.1177/1098214012453493

25.

Greene

J. C.

(1997). Evaluation as advocacy. Evaluation Practice, 18(1), 25–35. https://doi.org/10.1177/109821409701800103

26.

Gullickson

A. M.

King

J. A.

LaVelle

J. M.

Clinton

J. M.

(2019). The current state of evaluator education: A situation analysis and call to action. Evaluation and Program Planning, 75, 20–30. https://doi.org/10.1016/j.evalprogplan.2019.02.012

27.

Hall

M. E.

(2018). Evaluation’s race problem in the United States: A call to action for the profession and the American Journal of Evaluation. American Journal of Evaluation, 39(4), 569–583. https://doi.org/10.1177/1098214018792624

28.

Hall

J. N.

(2019). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485

29.

Hopson

(2003). Overview of multicultural and culturally competent program evaluation issues, challenges and opportunities. The California Endowment.

30.

Hopson

R. K.

Kirkhart

K. E.

Bledsoe

K. L.

(2012). Decolonizing evaluation in a developing world: Implications and cautions for equity-focused evaluations. In Segone

(Ed.), Evaluation for equitable development results (pp. 59–83). UNICEF.

31.

Hurteau

Gagnon

(2022). Contribution of practical wisdom to resolving ethical issues. Evaluation, 28(2), 182–191. https://doi.org/10.1177/13563890221086306

32.

Leone

Stame

Tagle

(2016). Exploring ethical issues and conditions for institutionalizing evaluation in the public sector. Evaluation, 22(2), 149–167. https://doi.org/10.1177/1356389016640626

33.

Levin-Rozalis

(2015). A purpose-driven action: The ethical aspect and social responsibility of evaluation. New Directions for Evaluation, 146, 19–32. https://doi.org/10.1002/ev.20117

34.

Espiritu

(2021). Evaluation is so white: Systemic wrongs reinforced by common practices and how to start righting them. Funder & Evaluator Affinity Network. https://search.issuelab.org/resource/evaluation-is-so-white-systemic-wrongs-reinforced-by-common-practices-and-how-to-start-righting-them-2.html

35.

Maxwell

(2013). Qualitative research design: An interactive approach (3rd ed.). Sage.

36.

Mendez

Taniuchi

(2020). Expanding the Bench™ week: Culturally responsive and equitable evaluation: What is it and why is it important? American Evaluation Association. https://aea365.org/blog/expanding-the-bench-week-culturally-responsive-and-equitable-evaluation-what-is-it-and-why-is-it-important-by-karla-mendez-alina-taniuchi

37.

Morris

(ed.). (2008). Evaluation ethics for best practice: Cases and commentaries. Guilford Press.

38.

Nelson

A. H.

Jenkins

Zanti

Katz

Berkowitz

Burnett

Culhane

(2020). A toolkit for centering racial equity throughout data integration. Actionable Intelligence for Social Policy, University of Pennsylvania. https://aisp.upenn.edu/wp-content/uploads/2022/07/AISP-Toolkit_5.27.20.pdf

39.

Nowell

L. S.

Norris

J. M.

White

D. E.

Moules

N. J.

(2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16(1). https://doi.org/10.1177/1609406917733847

40.

O’Connor

(2023). Procedural and participatory ethics: Community-based evaluation in practice. Evaluation Journal of Australasia, 23(2), 91–100. https://doi.org/10.1177/1035719X231166206

41.

Onyura

Main

Barned

Wong

T. D.

Chandran

, … Hamza

D. M.

(2023). The “what” and “why” of (un)ethical evaluation practice: A meta-narrative review and ethical awareness framework. Canadian Journal of Program Evaluation, 38(2), 265–312. https://doi.org/10.3138/cjpe-2023-0023

42.

Patton

M. Q.

(2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). Sage.

43.

Public Policy Associates, Inc. (2015). Considerations for conducting evaluation using a culturally responsive and racial equity lens. https://mphi.org/wp-content/uploads/2022/05/Considerations-for-Conducting-Evaluation-Using-a-Culturally-Responsive-and-Racial-Equity-Lens.pdf

44.

Reid

A. M.

Boyce

A. S.

Adetogun

Moller

J. R.

Avent

(2020). If not us, then who? Evaluators of color and social change. New Directions for Evaluation, 166, 23–36. https://doi.org/10.1002/ev.20407

45.

Rog

D. J.

(2012). When background becomes foreground: Toward context-sensitive evaluation practice. New Directions for Evaluation, 135, 25–40. https://doi.org/10.1002/ev.20025

46.

Roorda

Gullickson

A. M.

(2019). Developing evaluation criteria using an ethical lens. Evaluation Journal of Australasia, 19(4), 179–194. https://doi.org/10.1177/1035719X19891991

47.

Rosga

Stanton

Chomsky

Shin

(2022). Data ethics guidebook: Cultivating an ethical mindset in research & evaluation. Informing Change. https://informingchange.com/wp-content/uploads/2022/11/FINAL-data-ethics-guidebook.pdf

48.

Rossi

P. H.

Lipsey

M. W.

Henry

G. T.

(2019). Evaluation: A systematic approach (8th ed.). Sage.

49.

Schwandt

T. A.

(2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Stanford University Press.

50.

Schwandt

T. A.

(2018). Acting together in determining value: A professional ethical responsibility of evaluators. Evaluation, 24(3), 306–317. https://doi.org/10.1177/1356389018781362

51.

Schwandt

T. A.

Gates

E. F.

(2021). Evaluating and Valuing in Social Research. Guilford Press.

52.

Scriven

(2016). The last frontier of evaluation: Ethics. In Donaldson

S. I.

Picciotto

(Eds.), Evaluation for an equitable society (pp. 11–48). Information Age Publishing, Inc.

53.

Shanker

Maikuri

(2020). “They are intertwined” harm and the M&E cycle. University of the Witwatersrand Johannesburg. https://wiredspace.wits.ac.za/server/api/core/bitstreams/54ba8484-08c3-4572-a9b5-fbf513810308/content

54.

Smith

N. L.

(2002). An analysis of ethical challenges in evaluation. American Journal of Evaluation, 23(2), 199–206. https://doi.org/10.1177/109821400202300208

55.

Stame

(2018). Strengthening the ethical expertise of evaluators. Evaluation, 24(4), 438–451. https://doi.org/10.1177/1356389018804942

56.

Stern

Guckenburg

Persson

Petrosino

(2019). Reflections on applying principles of equitable evaluation. WestEd. https://www.wested.org/wp-content/uploads/2019/07/resource-reflections-on-applying-principles-of-equitable-evaluation.pdf

57.

Thomas

V. G.

Campbell

P. B.

(2020). Evaluation in today’s world: Respecting diversity, improving quality, and promoting usability. Sage.

58.

Thomas

V. G.

Madison

(2010). Integration of social justice into the teaching of evaluation. American Journal of Evaluation, 31(4), 570–583. https://doi.org/10.1177/1098214010368426

59.

van den Berg

Hawkins

Stame

(eds.). (2021). A framework for approaching ethics in evaluation. In Ethics for evaluation: Beyond “doing no harm” to “tackling bad” and “doing good” (pp. 3–16). Routledge.

60.

Wehipeihana

McKegg

(2018). Values and culture in evaluative thinking: Insights from Aotearoa New Zealand. New Directions for Evaluation, 2018(158), 93–107. https://doi.org/10.1002/ev.20320

61.

Wiggins

M. E.

Sileo

(2020). What’s the role of equity in evaluation policy? The Forum for Youth Investment. https://forumfyi.org/wp-content/uploads/2020/02/Equity-in-Evaluation-Policy-Winter-2020.pdf

62.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). The program evaluation standards: A guide for evaluators and evaluation users. Sage.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

Equity-Related Ethical Questions in Evaluation Practice: Examples From Interviews With Evaluators in New England,U.S.

Abstract

Keywords

Get full access to this article

References

Supplementary Material