What do tools do for evaluators? An affordances framework sheds light on the multiple ways that evaluators interact with tools. “Affordances” are ways that tools enable and constrain action, but the ways that tools do so depend on underlying social conditions. I apply this framework to the case of the Organization for Economic Cooperation and Development Assistance Committee criteria, which are widely used in international development evaluation. Through interviews and document analysis, I find that the DAC criteria offer technical and social affordances—they enable and constrain ways that evaluators navigate evaluation’s technical tasks, but they also enable and constrain ways that evaluators interact with others and draw symbolic boundaries around evaluation expertise. I also find these affordances are shaped by social conditions, like an evaluator’s perceptions or position in the organization. These findings suggest the importance of assessing evaluation tools’ multiple dimensions in their social contexts to understand tools’ role in evaluation work.
AskinNMauskapfM (2017) What makes popular culture popular? Product features and optimal differentiation in music. American Sociological Review82(5): 5910–44.
2.
BlankG (2007) Critics, Ratings, and Society: The Sociology of Reviews. Lanham, MD: Rowman & Littlefield.
3.
ChongPK (2020) Inside the Critics’ Circle: Book Reviewing in Uncertain Times. Princeton, NJ: Princeton University Press.
4.
DavisJLChouinardJB (2016) Theorizing affordances: From request to refuse. Bulletin of Science, Technology, and Society36(4): 241–8.
5.
Engle MerryS (2016) The Seductions of Quantification: Measuring Human Rights, Gender Violence, and Sex Trafficking. Chicago, IL: University of Chicago Press.
6.
EspelandWSauderM (2016) Engines of Anxiety: Academic Rankings, Reputation, and Accountability. New York: Russell Sage Foundation.
7.
EspelandWStevensM (1998) Commensuration as a social process. Annual Review of Sociology24: 313–43.
8.
EverittJG (2013) Inhabitants moving in: Prospective sense-making and the reproduction of inhabited institutions in teacher education. Symbolic Interaction36(2): 177–96.
9.
EyalG (2013) For a sociology of expertise: The social origins of the autism epidemic. American Journal of Sociology118(4): 863–907.
10.
HorvathA (2023) Organizational supererogation and the transformation of nonprofit accountability. American Journal of Sociology128(4): 1031–76.
11.
JensenTSandströmJHelinS (2009) Corporate codes of ethics and the bending of moral space. Organization16(4): 529–54.
12.
JerolmackCKhanS (2014) Talk is cheap: Ethnography and the attitudinal fallacy. Sociological Methods & Research43(2): 178–209.
13.
KrauseM (2014) The Good Project: Humanitarian Relief NGOs and the Fragmentation of Reason. Chicago, IL: University of Chicago Press.
14.
LamontMMolnárV (2002) The study of boundaries in the social sciences. Annual Review of Sociology28: 167–95.
15.
LatourB (1992) Where are the missing masses? The sociology of a few mundane artifacts. In: BijkerWELawJ (eds) Shaping Technology/Building Society: Studies in Sociotechnical Change. Cambridge, MA: MIT Press, 225–58.
16.
MasonS (2023) Finding a safe zone in the highlands: Exploring evaluator competencies in the world of AI. New Directions for Evaluation178–179: 11–22.
17.
Montrosse-MoorheadB (2023) Evaluation criteria for artificial intelligence. New Directions for Evaluation178–179: 123–34.
OfirZ (2017) The DAC criteria (part 1-11). Evaluation for Transformation: The Big Picture. Available at: https://zendaofir.com/blog/
21.
PattonMQ (2021) Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation42(1): 53–89.
22.
PollockND’AdderioL (2012) Give me a two-by-two matrix and I will create the market: Rankings, graphic visualisations and sociomateriality. Accounting, Organizations and Society37: 565–86.
23.
PorterTM (1995) Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton, NJ: Princeton University Press.
24.
RahmanHA (2021) The invisible cage: Workers’ reactivity to opaque algorithmic evaluations. Administrative Science Quarterly66(4): 945–988.
25.
RaimondoE (2018) The power and dysfunctions of evaluation systems in international organizations. Evaluation24(1): 26–41.
26.
ReyesV (2020) Ethnographic toolkit: Strategic positionality and researchers’ visible and invisible tools in field research. Ethnography21(2): 220–40.
27.
SchmidtRC (2007) Scaffolds for social meaning. Ecological Psychology19(2): 137–51.
28.
SchwandtT (2015) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Stanford, CA: Stanford University Press.
29.
SeabrookeL (2014) Epistemic arbitrage: Transnational professional knowledge in action. Journal of Professions and Organization1(1): 49–64.
30.
SharkeyAKovácsBHsuG (2023) Expert critics, rankings, and review aggregators: The changing nature of intermediation and the rise of markets with multiple intermediaries. Academy of Management Annals17(1): 25.
31.
SmallML (2009) “How many cases do I need?” On science and the logic of case selection in field-based research. Ethnography10(1): 5–38.
32.
SpringerE (2021) Caught between winning repeat business and learning: Reactivity to output indicators in international development. World Development144: 105506.
33.
StarSLGriesemerJR (1989) Institutional ecology, “translations” and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science19: 387–420.
34.
TavoryI (2020) Interviews and inference: Making sense of interview data in qualitative research. Qualitative Sociology43: 449–65.
35.
TimmermansSBergM (1997) Standardization in action: Achieving local universality through medical protocols. Social Studies of Science27: 273–305.
36.
TimmermansSTavoryI (2022) Data Analysis in Qualitative Research: Theorizing with Abductive Analysis. Chicago, IL: University of Chicago Press.
