“A Lot of It Really Does Come Down to Values”: An Empirical Study of the Values Advanced by Seasoned Evaluators

Abstract

This study challenges persistent misrepresentations of evaluation as a value-neutral inquiry process by presenting an empirical study that deepens understanding of evaluators’ values and how they “show up” in evaluation practice. Through semistructured interviews and inductive analysis, we examined the values advanced by a sample of eight experienced evaluators. We surfaced and examined 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies these evaluators conducted and guided how the evaluators positioned their work. Our findings provide empirical evidence about the role of values in evaluation practice and can support evaluators in reflecting on their own values and enacting their professional and ethical responsibilities to identify and articulate their values in the context of evaluation practice.

Keywords

values research on evaluation evaluation practice values engagement

Get full access to this article

View all access options for this article.

References

American Evaluation Association. (2018). Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles

American Evaluation Association. (n.d.). Values of the American Evaluation Association. https://www.eval.org/About/About-AEA

Benson

Cribb

Barber

. (2009). Understanding pharmacists’ values: A qualitative study of ideals and dilemmas in UK pharmacy practice. Social Science and Medicine, 68, 2223–2230. https://doi.org/10.1016/j.socscimed.2009.03.012

Caldwell

L. D.

Bledsoe

K. L

. (2019). Can social justice live in a house of structural racism? A question for the field of evaluation. American Journal of Evaluation, 40(1), 6–18. https://doi.org/10.1177/1098214018815772

Campbell

D. T

. (1969). Reforms as experiments. American Psychologist, 24, 409–429. https://doi.org/10.1037/h0027982

Chadwick

(1998). Professional ethics. In Crane

(Ed.), Routledge encyclopedia of philosophy (pp. 733–737). Abingdon, UK: Routledge.

Corbin

Strauss

(2015). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage.

Coryn

C. L. S.

Wilson

L. N.

Westine

C. D.

Hobson

K. A.

Ozeki

Fiekowsky

E. L.

,. . . Schröter

D. C.

(2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347. https://doi.org/10.1177/1098214016688556

Davidson

E. J.

(2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.

10.

Fournier

D. M

. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 1995(68), 15–32. https://doi.org/10.1002/ev.1017

11.

Fournier

D. M.

(2005). Evaluation. In Mathison

(Ed.), Encyclopedia of evaluation (pp. 139–140). Thousand Oaks, CA: Sage.

12.

Gioia

D. A.

Corley

K. G.

Hamilton

A. L

. (2013). Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational Research Methods, 16(1), 15–31. https://doi.org/10.1177/1094428112452151

13.

Greene

J. C

. (1997). Evaluation as advocacy. Evaluation Practice, 18(1), 25–35. https://doi.org/10.1177/109821409701800103

14.

Greene

J. C

. (2002). With a splash of soda, please: Towards active engagement with difference. Evaluation, 8(2), 259–266. https://doi.org/10.1177/1358902002008002522

15.

Greene

J. C.

(2006). Evaluation, democracy, and social change. In Shaw

Greene

J. C.

Mark

M. M.

(Eds.), The Sage Handbook of Evaluation (pp. 118–140). London: Sage.

16.

Greene

J. C.

(2012). Values-engaged evaluations. In Segone

(Ed.), Evaluation for equitable development results (pp. 192–206). New York: UNICEF.

17.

Greene

J. C.

Boyce

A. S.

Ahn

(2011). A values-engaged, educative approach for evaluating education programs: A guidebook for practice. Urbana, IL: University of Illinois at Urbana-Champaign.

18.

Greene

J. C.

Millett

R. A.

Hopson

R. K.

(2004). Evaluation as a democratizing practice. In Braverman

M. T.

Constantine

N. A.

Slater

J. K.

(Eds.), Foundations and evaluation: Contexts and practices for effective philanthropy (pp. 96–118). San Francisco: Jossey-Bass.

19.

Gullickson

A. M.

Hannum

K. M

. (2019). Making values explicit in evaluation practice. Evaluation Journal of Australasia, 19(4), 162–178. https://doi.org/10.1177/1035719X19893892

20.

Hall

M. E

. (2018). Evaluation’s race problem in the U.S.: A call to action for the profession and the American Journal of Evaluation. American Journal of Evaluation, 39(4), 569–583. https://doi.org/10.1177/1098214018792624

21.

Harman

Azzam

. (2018). Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards. Evaluation and Program Planning, 71, 68–82. https://doi.org/10.1016/j.evalprogplan.2018.08.004

22.

Hassell

Gullickson

A. M.

Boyce

A. S.

Hannum

(2020). Editorial: What do we mean when we talk about values? Evaluation Journal of Australasia, 20(2), 63–67. https://doi.org/10.1177/1035719X20931250

23.

House

E. R.

(2015). Evaluating: Values, biases, and practical wisdom. Charlotte, NC: Information Age.

24.

House

E. R.

Howe

K. R.

(1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.

25.

Howe

K. R.

(2003). Closing methodological divides: Toward democratic educational research. Boston: Kluwer.

26.

Jiménez-López

F. R.

Roales-Nieto

J. G.

Vallejo Seco

Preciado

. (2016). Values in nursing students and professionals: An exploratory comparative study. Nursing Ethics, 23(1), 79–91. https://doi.org/10.1177/0969733014557135

27.

Kushner

(2000). Personalizing evaluation. London: Sage.

28.

LaVelle

J. M.

Stephenson

C. L.

Donaldson

S. I.

Hackett

J. D

. (2022). Findings from an empirical exploration of evaluators’ values. American Journal of Evaluation. Advanced online publication: https://doi.org/10.1177/10982140211046537

29.

Lee

Y. K.

Low

W. Y.

C. J

. (2013). Exploring patient values in medical decision making: A qualitative study. PLoS One, 8(11), e80051. https://doi.org/10.1371/journal.pone.0080051

30.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Newbury Park, CA: Sage.

31.

McBride

Casillas

LoPiccolo

. (2020). Inciting social change through evaluation. New Directions for Evaluation, 166, 119–127. https://doi.org/10.1002/ev.20405

32.

Miles

M. B.

Huberman

A. M.

Saldaña

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.

33.

Ofir

Rugg

. (2021). American Journal of Evaluation section on international developments in evaluation: Transforming evaluation for times of global transformation. American Journal of Evaluation, 42(1), 47–52. https://doi.org/10.1177/1098214020979070

34.

Ozeki

Coryn

C. L. S.

Schröter

D. C

. (2019). Evaluation logic in practice: Findings from two empirical investigations of American Evaluation Association members. Evaluation and Program Planning, 76, 101681. https://doi.org/10.1016/j.evalprogplan.2019.101681

35.

Patton

M. Q.

(2015). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

36.

Patton

M. Q

. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200. https://doi.org/10.1177/1098214018763121

37.

Patton

M. Q

. (2021). Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation, 42(1), 53–89. https://doi.org/10.1177/1098214020933689

38.

Picciotto

. (2019). Is evaluation obsolete in a post-truth world? Evaluation and Program Planning, 73, 88–96. https://doi.org/10.1016/j.evalprogplan.2018.12.006

39.

Putnam

(2002). The collapse of the fact/value dichotomy. Cambridge, MA: Harvard University Press.

40.

Ravari

Bazargan-Hejazi

Ebadi

Mirzaei

Oshvandi

. (2012). Work values and job satisfaction: A qualitative study of Iranian nurses. Nursing Ethics, 20(4), 448–458. https://doi.org/10.1177/0969733012458606

41.

Roorda

Gullickson

A. M

. (2019). Developing evaluation criteria using an ethical lens. Evaluation Journal of Australasia, 19(4), 179–194. https://doi.org/10.1177/1035719X19891991

42.

Ros

Schwartz

S. H.

Surkiss

. (1999). Basic individual values, work values, and the meaning of work. Applied Psychology: An International Review, 48(1), 49–71. https://doi.org/10.1111/j.1464-0597.1999.tb00048.x

43.

Sagiv

Roccas

Cieciuch

Schwartz

S. H

. (2017). Personal values in human life. Nature Human Behavior, 1, 630–639. https://doi.org/10.1038/s41562-017-0185-3

44.

Saldaña

(2016). The coding manual for qualitative researchers. London, UK: Sage.

45.

Schwandt

T. A

. (1997). The landscape of values in evaluation: Charter terrain and unexplored territory. New Directions for Evaluation, 76, 25–39. https://doi.org/10.1002/ev.1085

46.

Schwandt

T. A.

(2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Palo Alto, CA: Stanford University Press.

47.

Schwandt

T. A.

Gates

E. F.

(2021). Evaluating and valuing in social research. New York, NY: Guilford.

48.

Schwartz

S. H

. (1999). A theory of cultural values and some implications for work. Applied Psychology: An International Review, 48(1), 23–47. https://doi.org/10.1111/j.1464-0597.1999.tb00047.x

49.

Schwartz

S. H.

Caprara

G. V.

Vecchione

. (2010). Basic personal values, core political values, and voting: A longitudinal analysis. Political Psychology, 31(3), 421–452. https://doi.org/10.1111/j.1467-9221.2010.00764.x

50.

Schwartz

S. H.

Cieciuch

Vecchione

Davidov

Fischer

Beierlein

,. . . Konty

(2012). Refining the theory of basic individual values. Journal of Personality and Social Psychology, 103(4), 663–688. https://doi.org/10.1037/a0029393

51.

Scriven

. (2007, June 6–7). The logic of evaluation. Paper presented at the Ontario Society for the Study of Argumentation, Windsor, Ontario.

52.

Scriven

(1991). Evaluation thesaurus (4th ed.). Newbury Park CA: Sage.

53.

Shadish

W. R

. (1994). Need-based evaluation theory: What do you need to know to do good evaluation? Evaluation Practice, 15(3), 347–358. https://doi.org/10.1177/109821409401500315

54.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

55.

Smith

N. L

. (1980). Sources of values influencing educational evaluation. Studies in Educational Evaluation, 6(2), 101–118. https://doi.org/10.1016/0191-491X(80)90013-9

56.

Teasdale

R. M

. (2021). Evaluative criteria: An integrated model of domains and sources. American Journal of Evaluation, 42(3), 354–376. https://doi.org/10.1177/1098214020955226

57.

Teasdale

R. M

. (2022). How do you define success? Evaluative criteria for informal STEM education. Visitor Studies 25(2), 163–184. https://doi.org/10.1080/10645578.2022.2056397

58.

Thomas

V. G.

Campbell

P. B.

(2021). Evaluation in today’s world: Respecting diversity, improving quality, and promoting usability. Thousand Oaks, CA: Sage.

59.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.