Abstract
In this reflexive piece, I consider these questions: How can evaluators reframe validity as a stance and agenda rooted in restoration and healing? How can evaluation contribute more fully to transformative, restorative, and abolitionist signposts such as reparation, reclamation, and renewal? I first review the restorative validity agenda, showing how evaluators can (re)orient their theories and practices toward relationships, justice, and liberation. Drawing on an example of a transformative participatory evaluation in Guatemala, I demonstrate how evaluation can be used to repair (evaluative) harms, renew relationships, and restore and reclaim the humanity of evaluator and evaluated alike. I conclude with a discussion of how our evaluative activities shifted: moving beyond the utilitarianism in punitive, colonial, and carceral logics; understanding how to consider and apply redistributive and reparative justice; and promoting opportunities for evaluator and evaluated to restore and reclaim their cultures-histories-identities-lands.
Keywords
How can we (re)imagine a healing and restorative space for evaluation? Based on the American Evaluation Association's Guiding Principles for Evaluators, evaluators should respect people by honoring dignity and acknowledging cultural influences and perspectives, while working toward a common good and an equitable society (King & Stevahn, 2020). However, our methodological practices and academic socialization are often passively informed through a cultural hegemony (Gramsci, 1971) that values capitalistic and colonial tendencies toward (infinite) productivity and growth, individualism (over the common good), and (epistemic) standardization; in turn, (re)producing the harms we may seek to counter (Dazzo, 2023; Shanker & Hopson, 2023; Shanker & Maikuri, 2020). In response, justice-oriented evaluators attempt to balance rigor and relevance (Mertens & Wilson, 2019). They not only strive to systematically determine the merit, worth, or value of a program, policy, or process (Scriven, 1991), but that their work can hold merit, worth, or value for the communities involved in/subjected to evaluation. Justice-oriented evaluators seek to create significance for/with/alongside communities, and for themselves as mindful and empathic human beings who seek to serve (Symonette, 2015). This conscious theorization and practice of evaluation is done with “Housian notions of justice” (Hopson, 2014, p. 83). These Housian notions direct evaluators to consider how the validity of our theories and practices are comprised of truth (i.e., “the attainment of arguments soundly made”), beauty (i.e., “the attainment of coherence well wrought”), and justice (i.e., “the attainment of politics fairly done”); that is, “if an evaluation is untrue, or incoherent, or unjust, it is invalid” (House, 2014, p. 10).
In this article, I add to the conversation by addressing notions of democracy and justice in evaluation theory and practice (Greene, 2006; Mertens & Wilson, 2019). House (1980) called on evaluators to understand and assess grand theories of justice “as signposts telling us where we are and where we want to go” (p. 136). Hopson (2014) responded by adding signposts directing us toward liberatory and decolonizing aims. I continue this call-and-response, answering Hopson's (2014) challenge: “there is a clear need for more developments recognizing how justice signposts influence evaluation” for the purpose of “building branches of justice and advancing an important and core attribute for the field” (p. 84). I ask: What would it mean for evaluation to contribute more fully to restorative and abolitionist signposts such as reparation, reclamation, and renewal? I offer my response in this paper, the fourth in a series documenting participatory research and evaluation activities in Guatemala. The previous three include:
A theoretical grounding of the restorative validity agenda (Dazzo, 2024), which demonstrates how knowledge across three disciplines (critical theories and pedagogy; research methodology and evaluation theory; and peace, rights, and justice studies) were referenced to explore how researchers and evaluators could reorient their theories and practice toward relationships, justice, and liberation. A co-authored visual ethnographic case study documenting how our collective employed culturally sustaining and participatory approaches (Dazzo et al., 2023). A narrative inquiry using counter-storytelling where co-researchers speak back to the axiological challenges they face when attempting to direct inquiry away from technocratic obligations (Dazzo, 2023).
This work focuses on the evaluative portions of our collective's work. I expand on the notion of restorative validity through the concepts of reparation, reclamation, and renewal.
To apply restorative validity, evaluators use theories to enact liberatory practices (i.e., praxis) to restore how they view their individual and collective humanity, including their own relationships to cultures-histories-lands-identities. 1 This is a call to consider how we (re)humanize the self (evaluator), the other (evaluated 2 ), and the process itself (evaluation). “Whose ways of being, doing, engaging matter and are thus privileged? Who authorizes, decides, and how?” (Symonette, 2015, p. 125). Whose notions of validity are being attained? I then situate this work through the case of a transformative participatory evaluation (T-PE; Cousins & Whitmore, 1998). This was conducted alongside forensic anthropologists from the Forensic Anthropology Foundation of Guatemala (FAFG) and Kaqchikel Maya community members in Guatemala. We focused on two areas: (1) evaluating the significance of a community memorial site, which commemorates the lives of the forcibly disappeared during the country's 36-year armed conflict, and (2) understanding Indigenous values and priorities for justice, memorialization, and genocide education (Dazzo et al., 2023). We chose the memorial as a site because it foregrounds each part of FAFG's forensic anthropology process: family members assisted FAFG in identifying possible mass graves (at the site), FAFG and community members exhumed the remains of those forcibly disappeared, FAFG matched DNA from bone fragments with living family members’ samples, and remains were returned to loved ones so they could hold funeral ceremonies.
With the restorative validity agenda, I continue to challenge where we are and where we want to go as a discipline (Hopson, 2014; House, 1980). I do this by offering these signposts: reparation, reclamation, and renewal. I suggest the following directions to move evaluators, as individuals and a community, toward action:
Moving beyond the utilitarianism behind punitive and carceral logics in evaluative theories and practices. Understanding how evaluators can draw on restorative justice theories to repair past injustices and to reimagine new futures alongside communities. Promoting opportunities to restore and reclaim cultures-histories-lands-identities through evaluation theory and practice.
Through the work in Guatemala, I review how I used several reflexive questions within the restorative validity agenda, which can be found in earlier publications (Dazzo, 2024). I adapted Zehr's (2015a, 2015b) original questions for restorative justice practitioners to explore the values behind our collective's work. This is just one example of how evaluators can interrogate their paradigms and methodologies, moving beyond doing no harm to enacting practices that do good (van den Berg et al., 2021).
The Restorative Validity Agenda
Within evaluation theory, I frame restorative validity through culturally responsive evaluation (Hood et al., 2015) and the transformative paradigm (Mertens & Wilson, 2019). As House (2014) argued, “broadening the validity concept doesn’t replace other conceptions, but rather focuses on the arguments that all must depend on” (p. 10). For me, it became important to note how I interpreted validity, in my everyday capacity, not simply through academia. Etymologically, validity originates from the Latin term, validus: “strong,” “healthy.” It was derived from valeō: “I am strong,” “I am healthy,” “I am of worth.” I argue if evaluators are to be attuned to restorative validity, they must question whether their praxis contributes to the validity/strength/health/worth of evaluator and evaluated, and the evaluation process itself (Dazzo, 2024). Speaking through the “I” in valeō forces evaluators to think about their personal responsibilities, as they move to evaluate alongside communities, rather than conducting evaluations on/about them. It critically requires evaluators to ask:
When was the last time an evaluator and those being evaluated truly felt strong, healthy, or worthwhile as an outcome of an evaluation project? If evaluation does (or, has done) harm, what are evaluators doing to right those wrongs? How will the evaluator work toward reparation, reclamation, and renewal? How are oppressive superstructures recreated if evaluators use their expert/objective conclusions to replace the merit/worth/significance of communities’ knowing/being?
In the next section, I review the literature and components within the restorative validity agenda (Dazzo, 2023, 2024; Dazzo et al., 2023), outlining several key elements relevant to a critical and restorative praxis of evaluation.
Movement: From Evaluator-as-Oppressor to Liberatory Logics
Through socialization, evaluators are often taught to focus on problems (e.g., research problems, problem statements). Restorative validity requires questioning the oppressive superstructures within evaluation and society to understand what it would take to no longer see the world, and what or who is evaluated, as problems. It acknowledges the validity of individuals’ knowing (epistemology)/being (ontology); in turn, restoring the histories, identities, and memories (Martín-Baró, 1994) of evaluator and evaluated alike. This must occur as evaluators (re)consider the ethicality and rationality behind their methodological decision-making and judgments (Gates, 2018). It requires asking how evaluation has fortified the oppressive superstructures observed in society, including social control, surveillance, and punitive logics. Do evaluators enact evaluation in service of restoration, or through the (neo)colonial imagery of knowledge production so it simply becomes another “efficient means of social and cultural control” (Quijano, 2007, p. 169)?
Through restorative validity, it becomes the evaluator's responsibility to see themselves as oppressor as we work within systems that may hide the humanity of others through metrics of constant productivity. The evaluator is responsible for restoring the voices, identities, and values that have been lost through scientism and positivism. This includes the loss of the evaluator's freedoms, which includes the freedom to practice their known methods (i.e., familiar and familial ways of knowing/being) that have been subsumed by academic and professional training.
Toward Restorative Validity: Reparation, Reclamation, and Renewal
As illustrated in Figure 1, the restorative validity agenda requires evaluators to (re)orient theory and practice toward relationships, justice, and liberation (Dazzo, 2024).
Conceptual framework: orientations toward restorative validity.
Figure 1 expands on these orientations by focusing on how the concepts of reparation, reclamation, and renewal are at the nexus of these categories. The purpose is not to simply evaluate a project, but for evaluators and communities to co-construct restorative and reparative forms of inquiry that humanize evaluator and evaluated alike, and the evaluation process itself.
Signpost 1: Reclamation—At the nexus of relationships and justice
In this first signpost, there is a call to move beyond the utilitarianism behind punitive and carceral logics in evaluation, redirecting evaluative logics toward healing and interdependence. Our collective's work in Guatemala became a “project in humility” (F. Peccerelli, personal communication, July 19, 2019). Within qualitative inquiry, this is addressed through an ethic of humility (Limes-Taylor Henderson & Esposito, 2017), where the researcher assesses self against a larger environment, specifically one in which they may know very little. This lesson holds in evaluation. Releasing oneself from “the façade of authority” allows one to become learner, unsettling expertise and “academia's role of perpetuating Western settler colonial ways of knowing” (Limes-Taylor Henderson & Esposito, 2017, p. 12). Bridging relationships and justice, there is an opportunity to create space for healing and interdependence in evaluation. As hooks (1999) notes: “Rarely, if ever, are any of us healed in isolation. Healing is an act of communion” (p. 215). This space establishes evaluation as moving closer to a community of care, just as those within “professions of systemic compassion” (e.g., health care, conflict resolution) must “recognize that a certain population group is situated within their circle of moral concern” (Rothbart & Allen, 2019, p. 374). When evaluators engage in care ethics, the “commitment to democratic values of equity, fairness, and epistemic justice is realized” (Simmons & Greene, 2018, p. 91). These principles are further represented as Cram et al. (2023) observe how various Indigenous evaluation approaches must contribute to “Indigenous sovereignty, vitality, and prosperity” (p. 55). As an example, Cram et al. (2018) draw on Smith's (1997) Kaupapa Māori as a theoretical framework, showing how Kaupapa Māori evaluation ethics must be attuned to relationality (whakawhanaungatanga, meaning connections), reciprocity (manaaki ki te tangata, meaning share, host, and be generous), and humility (kia māhaki, meaning be humble).
Signpost 2: Reparation—At the nexus of justice and liberation
House (1980) noted that naming dominant theories of justice (e.g., utilitarianism, pluralism, justice-as-fairness) is not “entirely satisfactory as a basis for evaluation” (p. 134) as the evaluator's “theory operates implicitly and subtly” (p. 136). There is value though in understanding how recognition and redistribution hold various remedies for (evaluative and societal) injustices. Fraser's (2005) proposed redistributive and recognitive remedies provide evaluators with a heuristic to question how methodology can be better informed. For instance, Indigenous oral traditions have been devalued when compared to Western research (Smith, 2012), and many have been asked to share knowledge without benefit while evaluators continue to profit through contracts and tenure. Recognitive remedies require practices that (re)value disrespected ways of knowing through the Indigenization of method, or by citing Indigenous knowledge outside academia. Redistributive remedies also question the economic sphere, such as compensation, redistribution of funds, and rematriation of land. As evaluators remedy the redistributive and recognitive injustices in our practices, they can associate these questions by understanding that decolonization is not a metaphor (Tuck & Yang, 2012). Much like Giri (2011), who speaks of the dialectic tension between compassion and confrontation, Tuck and Yang (2012) raise issue with the notion of reconciliation when it does not attempt to “unsettle innocence” and understand what is “incommensurable between decolonizing projects and other social justice projects” (p. 4). Decolonization requires understanding complicity and how one is implicated in the colonial project, regardless of race, immigration status, and one's social justice project (e.g., abolition, historical third world decolonization). Through an ethic of incommensurability, evaluators must consider the tension in one's innocence and complicity, and how they reinforce settler colonialism. Solidarity is not addressed through words but redressed through action (Fraser, 2005). By assessing Freire's (2012) Pedagogy of the Oppressed, Tuck and Yang (2012) refer to a “settler fantasy of mutuality based on sympathy and suffering” (p. 20) and the existence of a third party to the oppressed–oppressor relationship: the enlightened human. This recognition pushes one to understand reconciliation and solidarity are not about “rescuing settler normalcy” (Tuck & Yang, 2012, p. 35). Evaluators must consider their status as (potential) oppressor and complicate reconciliation and solidarity.
Reconciliation is difficult to achieve because it requires a “brutal confrontation with oneself and a painful recognition of one's own moral degradation” (Bhargava, 2012, p. 369). This theory distances reconciliation as solely a social or political improvement among parties in conflict. It must be interpersonal. In evaluation theory, connections among the political, social, and interpersonal are seen within Made in Africa Evaluation (MAE), as nine principles are defined: relationality, responsibility, reverence, reciprocity, respectful representation, reflexivity, responsivity, rights and regulations, and decolonization (Chilisa & Mertens, 2021). In MAE, there is a connection between justice and liberation. This nexus presents an opportunity to explore how evaluators must hold themselves accountable for their actions and those within the discipline, as they seek to redress (evaluative and societal) injustice.
By connecting reconciliation with accountability, what one decides to do is manifested through the emancipation of moral thought and transformation of social behavior (Bhargava, 2012). This idea evokes Verdeja's (2009) maximalist interpretation: reconciliation as transformation. This requires repentance and forgiveness, moral repair, restorative justice, healing, and social renewal. Transformative reconciliation can also be utilized to demonstrate how certain protocols could be used to alleviate the societal wrongs (re)produced in evaluation. For instance, as evaluators, how do we account for the fact that many of the scientific management practices we use to assess value and determine worth are rooted in the detailed record-keeping practices of plantation slavery (Rosenthal, 2018)? If evaluation is to truly move toward liberatory logics grounded in relationality, evaluators must move toward love and empathy, rather than measuring unchecked productivity and growth.
Signpost 3: Renewal—At the nexus of liberation and relationships
Basing evaluation on the logic of love requires an attempt to personalize evaluation (Kushner, 2000), rather than the punitive or capitalistic logics evaluators have so often been socialized to accept as objective. As Kushner (2000) notes: The alternative is to document people's lives and to use these as contexts in which to read the significance and the meaning of the program—to invert the relationship between program and person… Not, as I will argue, to continue to scrutinize the powerless, but to attempt to see power structures through their eyes and against their criteria—that is, to subject power structures to their judgement. (p. 33)
The meeting point: Restoration
Because individuals can fluidly move from evaluated to evaluator (depending on their relationship to a project or policy), it is vital to understand how co-researchers within a collective can work together to restore the humanity and dignity of all those involved in an evaluation. For this reason, I began applying ideas from restorative justice (Zehr, 2015a, 2015b) and reparation (Verdeja, 2009) to research and evaluation. This included asking who has been hurt and how evaluators can repair (past) harms. As defined by Zehr (2015b), “restorative justice requires, at minimum, that we address victims’ harms and needs, hold offenders accountable to put right those harms, and involve victims, offenders and communities in this process” (p. 25). Like Zehr (2015b), I simply provide a series of reflexive questions for evaluators to consider as they explore what works in their contexts. Table 1 includes questions adapted from Zehr's (2015a) work on restorative justice, and my reframing for research and evaluation. The full list of questions (Dazzo, 2024) requires the evaluator to explore (1) who has been harmed by research, evaluation, and societal structures; (2) whose obligations are these and who has a role; and (3) what response/engagement is appropriate. Table 1 focuses on engagement, after harms, needs, and obligations are addressed.
Guiding Questions for Restorative Validity.
Source: Adapted from Dazzo (2024).
The questions from Table 1 must be considered relationally, alongside communities in “fellowship and solidarity” (Freire, 2012, p. 85). The evaluator cannot be outside this process. Enacting restorative validity requires the evaluator to question their involvement: How have I been harmed by research and evaluation? How have research and evaluation structures affected my knowledge systems, my participation, my inclusion? How have I enacted harm through evaluation? This occurs throughout the process: evaluation question formation, data collection and analysis, and re/distribution of knowledge. From a procedural or methodological standpoint, humanization makes it possible to acknowledge and learn how communities have their own forms of inquiry and methodological decision-making, which they conduct in their everyday (Martín-Baró, 1994). This reflexivity pushes back against the notion that researchers/evaluators must export their methods or their ways of asking questions to build communities’ capacities.
Signposts and Turns
During our collective's work on restorative validity, I sought to further understand how evaluation, as theory and practice, could restore the history, identities, and memories of evaluator and evaluated alike. Here, I demonstrate the directions we took through a reflexive praxis. As Bowman (2021) recommends, I “use praxis and experiential knowledge… to balance and speak back to academy where theory and method are learned in an artificial environment” (p. 321). Methodologically, I use narrative inquiry, drawing on the power of/in space, place, and time (Kim, 2016), while rooting this analysis in the Maya cultural elements of my Kaqchikel co-researchers: “land, community, and an attachment to place” (Lovell, 1988, p. 27).
Situating Myself: Power, Privilege, Positionality
I start this reflexive analysis by situating myself. How does my approach require me to reflexively interrogate my social and political location (Harding, 1992)? As researcher and evaluator, I came to this work with power and privilege. At the time, I was working at a human rights funding agency and was invited to Guatemala through my relationship with FAFG. As a citizen of the United States, and at the time, an employee within the federal government, it required me to interrogate the neo-imperialist history of my home country and my employer, and how the United States’ clandestine political actions were linked to the atrocities that occurred on Kaqchikel land. How would Kaqchikel and Guatemalan community members view me: As colleague or colonizer (Lamar, 2024)? It required me to ask myself: As researcher/evaluator, how am I oppressor (Dazzo, 2024)? On the other hand, as a child of immigrants, born and raised in a rural area with familial ties to farming, I was comfortable being in agricultural areas such as those around the memorial. I understood the importance of land. In my youth, I was involved in educational evaluations as evaluated, not evaluator, and understood how certain terms (e.g., first-generation, rural, low-income) led me to feel less than/othered by those evaluating my academic prospects. This reflexivity led me to identify and push against the researcher–researched binary while not falling prey to the settler-colonial “fantasy of mutuality” (Tuck & Yang, 2012, p. 16). I was not there as an enlightened third party, distanced from the site, ready to empower others. The oppressions I felt were not those that co-researchers experienced.
Situating the Site
The site of our work was the San Juan Comalapa Memorial for Victims of Enforced Disappearance (Dazzo et al., 2023), which commemorates the lives of those forcibly disappeared during the country's internal armed conflict (1960–1996). The primary team included me, Carmencita Cúmez (Kaqchikel community leader), Erica Henderson (FAFG staff), and Fredy Peccerelli (FAFG Executive Director). The broader collective included 12 nonprofit staff from the FAFG and 40 Kaqchikel Maya community members who were all involved in agenda setting, data collection, analysis, and constructing recommendation. 3
Situating the Evaluation
Why did this evaluation begin? In 2018, Fredy from FAFG contacted me, as we had worked closely together. He was perplexed to receive two evaluation reports—one from a funder, one from an independent evaluator—providing starkly different evaluative judgments. The independent evaluator lauded the organization for their community-informed approach, without much evidence. One of their funders, however, countered with findings from a survey they administered to Indigenous Maya community members who had participated in the forensic anthropology process. Within this survey, the primary question was: “Are you satisfied with the forensic anthropology process?” Many respondents noted dissatisfaction, setting off questions by the funder and FAFG. In the next section, I narrate our experience of framing our evaluative activities through the guiding questions within the restorative validity agenda.
Moving Beyond the Utilitarianism Behind Punitive and Carceral Logics in Evaluative Theories and Practices
First, the funder's survey was poorly designed. The process and purpose amounted more to surveillance than evaluative judgment. It is not surprising to guess why many respondents were not satisfied. The probability of exhuming human remains, some of which have been buried since 1960 in Guatemala, and identifying a match through DNA sampling, is approximately 14% (F. Peccerelli, personal communication, July 17, 2019). The survey item was not designed to consider Maya culture. For the Maya, families must find the remains of their lost loved ones so they may properly bury them, allowing the living to retain communication with their ancestors (Dazzo et al., 2023; Peccerelli & Henderson, 2021). Second, the report by the external evaluator, although positive, was not rigorous. The overly positive findings were written more like marketing statements for FAFG, rather than evaluative judgments to determine the value of their services. One could attribute the overly negative findings to the misapplication of performance measures (Perrin, 2018) while the overly affirmative findings as evaluation misuse (Patton, 2001). The weak evidence resulted in nonuse due to misevaluation (Patton, 2005). However, it is possible to move beyond use and understand how evaluation can create a bidirectional influence (Kirkhart, 2005). This case of misevaluation led FAFG to reconsider their organizational culture, especially how they were not abiding by their principles. Our collective began discussing why the organization's community-engaged and culturally responsive approach to forensic anthropology was not being valued within their evaluative processes. These conversations led to new processes where we incorporated critical participatory inquiry (Call-Cummings et al., 2023; Fine et al., 2012) and T-PE (Cousins & Whitmore, 1998).
We established early we would not set the objectives of any evaluative activities without Kaqchikel partners present. After several co-learning sessions where FAFG staff taught me about the experiences and knowledge systems they drew from as forensic anthropologists and geneticists, I spoke about the values behind community-engaged research and evaluation. For FAFG scientists, these co-learning sessions were useful for unlearning the colonial and racist vestiges of the scientific method (Fanon, 2014; Smith, 2012). These sessions helped me to learn about the Kaqchikel community's history and their onto-epistemological commitments, so we could approach our evaluation through Maya culture. Although we briefly discussed the FAFG's donor-mandated performance measures, this was done to differentiate between compliance mechanisms and FAFG's accountability to Maya communities. Rather than overemphasizing audit culture and its punitive effects (Owczarzak et al., 2016), we focused our inquiry through the positive freedom 4 of what could be (i.e., sites of possibility; Greene, 1988) and where we wanted to go (i.e., signposts; Hopson, 2014; House, 1980).
To free ourselves from the punitive and carceral logics associated with measurement (Rosenthal, 2018), we rooted our evaluative activities in dialogue, pushing against the utilitarianism in methodological training. We began with community consultations where nonprofit staff and Kaqchikel community members discussed their priorities and evaluative questions. These questions included the significance of (not) finding a family member through the forensic anthropological process; the extent to which the memorial (could) serve(d) as a form of collective memory; the community's hopes for the memorial; and alignment between definitions of justice among funders, FAFG, and the community. Being at the memorial meant existing in relationship with one another as co-researchers and human beings, respecting Maya cultural elements, and respecting the land and those who had been clandestinely buried there by paramilitary forces. We learned as we sat and listened during storytelling sessions founded in Kaqchikel oral tradition, while sharing meals, and exchanging personal stories. Walking or go-along interviews were used to learn how the natural and built environment of/on the land evoked community members’ memories of the disappeared and their hopes for the memorial. We engaged in interactive data collection activities, such as Ripples of Change and dot voting, to plot and prioritize hopes, not solely for the purpose of generating data but to promote democratic deliberation (Dazzo et al., 2023). These relational practices were meant to push against the surveillance and control predicated by performance measures, which underscore a neoliberal approach that prioritizes market thinking, economic efficiency, and productivity (Picciotto, 2019) over relationships, justice, and liberation.
These punitive and carceral logics can inhibit full participation. Evaluators must question the assumption that communities feel comfortable participating alongside them, or if communities see them as oppressors. If participatory practitioners do not follow critical theories closely, dialectically noting the tensions within participation (fear | power), they may assume participation is a given, rather than questioning the inherent power differentials that reside in evaluation. It is possible to see how dialogue and participation moved toward transformative aims within our T-PE as we documented this over the course of two years. As observed from Carmencita's words in Figure 2, while participation may evoke fear, it can also manifest power if attending to the connections among relationships, justice, and liberation. As Christens (2019) notes, power can liberate, neutralize, or suppress. To intentionally design research and evaluation as community-driven change, it is imperative to understand the institutional, situational, and systemic dimensions of power (Gaventa, 1980; Lukes, 1974).
Dialectic notation: tensions of/within participation—Fear | Power.
Understanding How Evaluators can Draw on Restorative Justice Theories to Repair Past Injustices and to Reimagine New Futures Alongside Communities
It was clear FAFG and the Kaqchikel were harmed by injustices stemming from the punitive and carceral logics of evaluation. What some evaluators regard as methodological decisions or numerical objectivity, however, can amount to the silencing of identities (e.g., cultural assimilation; TallBear, 2020) and destruction of knowledges (e.g., epistemicide; de Sousa Santos, 2008) that diverge from majoritarian paradigms. For the Kaqchikel, harms enacted by the state during the armed conflict, and by researchers/evaluators after, demonstrate how cultural hegemony not only erases through genocide, but epistemicide. To counter this, I continuously drew on the reflexive questions within the restorative validity agenda (Dazzo, 2024; see Table 1). As evaluator, I acknowledged and examined the tensions between the obligations held by various actors. These obligations included those I, the evaluator, held for myself (e.g., professional/academic institutions, methodology), those of the community (e.g., social group), and those of the institutions funding an evaluation (Dazzo, 2023).
As Carmencita mentioned, the space to participate evoked fear, not just the emancipatory aims we sought through T-PE. For FAFG staff members, such as Fredy and Erica, while they were trained in the physical and social sciences and employed a culturally responsive worldview in their forensic services, they still saw evaluation as an exercise distanced from the humanity they practiced in their day-to-day work. For myself, the harms done through research and evaluation came through my experiences of being observed during my own schooling, defined by my deficits by educational researchers and evaluators. The forms of validity evaluators are taught often promote efficiencies that silence; subjugating, rather than sustaining the diversity of knowledge, culture, and history within humanity.
Through the work of Maxine Greene (1988), Stanfield (2006) saw the restorative justice functions of qualitative inquiry as a space where researchers could elucidate their axiological commitments and free themselves of domineering paradigms. Freeing ourselves from the utilitarianism of methodology meant shedding the idea that our practices would be beholden to European and Euro-settler culture; in turn, disrupting “evaluation's colonial roots and ongoing disciplining function” (Shanker & Hopson, 2023, p. 36). This did not signify throwing out methodology completely, but to think communally. This act relates to a positive freedom to collectively reject our “preoccupations of appearing scientific enough” (Stanfield, 2006, p. 723), moving toward forms of validity founded in humanistic integrity, healing, and liberation. The goal is for researcher and researched to “mutually discover their humanity and the deeply rooted injuries, hurts, phobias and prejudices that have amputated parts of their selves” (Stanfield, 2006, p. 726). As evaluators, the power in this restorative function, if critically self-reflexive, is the necessity to question how these injuries, hurts, phobias, and prejudices are at the center of and have been (re)produced because of research and evaluation. In moving these discussions from qualitative research to evaluation, Stanfield's (2006) work extends how evaluators understand the shortcomings of theories and practices, while imagining new forms of liberatory and anticolonial praxes. This counters the negative freedom of distance, of objectivity, of isolation. It is a positive freedom to “join with others and transform” (Greene, 1988, p. 73).
To think with restorative and reparative justice necessitates asking what these theories can do for evaluators who hope to work beyond the confines of our discipline, as we seek to effect positive societal change. As Kaqchikel community members collaborated on our evaluation agenda, questions, and methods, it was necessary to compensate them as co-researchers/co-evaluators. Compensatory practices did not follow capitalistic systems of individual payment. Discussions with Kaqchikel leaders led us to consider how compensation could honor collectivist Maya traditions. Compensation recognized the community's knowledge and labor. Integrating Kaqchikel knowledge, culture, and histories within our practices was a form of cultural recognition and redistribution. When evaluators consider reparation, it is necessary to think how these must be done toward transformative aims, rather than the transactional give-and-take tendencies we see in superficial calls for reciprocity.
Compensation for co-evaluators was funded through various means, including the redistribution of personal funds, salaries, grants, and fellowships. For me, it was important to not only compensate Kaqchikel co-researchers through grant and fellowship funding, but to redistribute some of my personal means from this work, such as my salary, just as the community had done when they hosted me on their land and cooked for me. Because these actions were coordinated through a collectivist tradition, we prioritized the reparative effects of our work. Furthermore, FAFG and the Kaqchikel community used narrative evidence from our evaluation to write grant proposals to justify the community's recommendations (E. Henderson, J. Helmuth, & F. Peccerelli, personal communication, March 25, 2025). These included:
Enhanced security to reduce vandalism and unauthorized use of the memorial site so the Kaqchikel could feel safe visiting their ancestors. Associated findings: Community members expressed how trespassers were accessing and vandalizing the site, and how this inhibited their ability to honor the land and their loved ones. The community also expressed their desire for visitors to understand the sanctity of the site. Beautifying ceremonial areas for culturally sustaining funerary practices. This included the planting of flowers to further draw on the community's idea that the memorial is a living space. Associated findings: The memorial was being treated more like a cemetery, rather than a space showcasing Maya culture, which sees life and death as circuitous and intertwined. This balance is evidenced on the murals painted on the nimajay. Obtaining an additional plot of land to expand the site. Associated findings: This was based on the community's land-back initiatives and their interests of culturally rematriating sacred land to communal ownership. Creating playground areas for children to improve intergenerational use. Associated findings: As the search for the disappeared continues, the Kaqchikel wanted to ensure children could find a place for themselves while still learning about the genocide.
Recommendations for improved Kaqchikel use were enacted while still welcoming others to visit. The Kaqchikel and FAFG explored how the memorial could serve as a form of collective memory to educate others about the genocide and the sacred land at Chiq’a’l/Comalapa. In reports to their funders, FAFG included donor-mandated performance measures, but spent more time presenting the narratives we generated. This provided a more culturally sustaining frame of what justice could look like from the Kaqchikel community's viewpoint, rather than Western-based priorities for punitive justice and how memorial sites should be built.
Promoting Opportunities to Restore and Reclaim Cultures-Histories-Land-Identities Through Evaluation Theory and Practice
Guided by our restorative validity agenda, the recognition and acknowledgment of cultures-histories-lands-identities within our collective became vital to our evaluative work. By honoring Kaqchikel oral tradition as a form of sacred data collection and by foregrounding our analyses through the Maya Cosmovision and the Kaqchikel cultural elements of land, community, and attachment to place (Dazzo et al., 2023), we recognized the expertise of Kaqchikel co-evaluators. As Honneth (1995) discussed, there are often issues of nonrecognition behind the power dynamics of our relationships. Those with more power (evaluators) may ignore the agency of those with less (the evaluated). By problematizing limited definitions of validity, evaluators can counter the nonrecognition of community members’ knowledge (Honneth, 2005). In our case, it meant understanding how the social practice of methodology can reify community members as object/other, rather than persons/co-evaluators with full agency.
It was very sacred work for us
Returning to my conversations with Carmencita, she noted the sacredness of our methodological decisions and evaluative judgments as we reflected on our collective work: For me, this was very important, that the participation of each one depends on us, the women and the young people. The ideas we contributed were very important. That no, it was not, let's say, a job like any other. But, for us, this job was very important, it was very valuable, because it was very sacred work for us, because in everything we did it is also part, shall we say, of our culture. It is part, let's say, of the history, of our town. (C. Cúmez, personal communication, November 30, 2021)
Cu’ nun havi spierenza nun havi spierienza
To be culturally responsive while restoring the humanity of those within an evaluation process does not simply mean to speak of how it was restorative for others. In my case, how did this work restore my own humanity: my own relationships to culture-history-land-identity? I begin by reflecting on a Sicilian proverb: Cu’ nun havi spierenza, nun havi spierienza/Those who have no experience, have no science. For me, writing this in Sicilian is intentional, as it is a language I have often been forced to forget by replacing it with formal Italian. Reflecting on my time visiting the Kaqchikel community and memorial site, I recalled the ways of knowing/being I forgot due to my academic and professional life. What familial teachings had I forgotten? How was I taught to be/know in my every day? The reflexive questions within the restorative validity agenda (Dazzo, 2024) required me to rethink how I would conduct research and evaluation in the everyday. As I was taught in my own rural upbringing, I walked on Kaqchikel land, picked up dirt, and observed it as community members narrated their creation story. With the Kaqchikel, I learned about the nimajay, the memorial, their loved ones, and their land by taking a stroll/fare una passeggiata. Just as we recognized oral tradition as a Kaqchikel known method, this strolling/passeggiata was one of my known methods. This act caused me to reflect on my own culture, and why it was important for me to walk/move as a form of observation. With the passeggiata, there is no end goal; we are not trying to arrive somewhere. This practice is simply about engaging in reflection and dialogue in community. I hold this ritual and social practice to be sacred as it is grounded in my culture-history-land-identity. As we took a stroll around the site, I learned as Kaqchikel co-researchers referenced the land to recall memories.
While I did not intentionally seek for this work to become sacred, our collective's work did breathe life into it for me. I examined how my familial teachings and my rural connections continue to be sacred, further restoring my relationship to my culture-history-land-identity. I can also attest to how this evaluation will continue to remain sacred for me. Several years after our evaluation activities, I visited the site with my family. I observed my children move seamlessly between the new playground, nimajay, exposed dirt (where remains were exhumed), and nichos, simultaneously learning and playing as children do. According to our Kaqchikel guide, the expanded plot, enhanced security, beautification, and playground have served their purposes as visitors, young and old, now frequent the memorial to learn about the genocide.
Conclusion
If attuned to restorative validity, evaluators should ask if their work contributes to the strength, health, and worth of evaluator and evaluated, and the evaluation process itself. The emphasis on relationality, justice, and liberation requires evaluators to co-create what (restorative) validity means with community members. It necessitates a negotiation of how the evaluative work will be rigorous and relevant. Within our collective, we observed and experienced several transformative aims from our critical participatory work: community interest to engage as co-evaluators was strong; co-evaluators from nonprofit organizations reimagined how evaluation could be healthy, rather than extractive; and co-evaluators from the Kaqchikel community noted how we increased the worth of the evaluation by prioritizing Maya culture. It is through this work that I call on evaluators to consider these signposts—reparation, reclamation, and renewal—as they explore how their methodologies and paradigms can restore the cultures-histories-lands-identities of evaluator and evaluated alike. This is of paramount importance as hegemonic structures erode community participation and the public spheres of democracy, justice, and policymaking. Evaluators must continue exploring how evaluation can become a public science. Our primary clients must include communities, not those who seek to privatize evidence to further justify (neo)colonial and hyper-capitalistic objectives that primarily benefit those with power.
Footnotes
Acknowledgments
This article would not have been possible without the knowledge, labor, and care of co-researchers from the Fundación de Antropología Forense de Guatemala and the Coordinadora Nacional de Viudas de Guatemala.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Portions of this research were funded by a doctoral fellowship granted by George Mason University. CONAVIGUA received compensation for the knowledge, time, and labor shared by their Indigenous Kaqchikel members. Funding from the Robert Wood Johnson Foundation was used to spend time researching theories and practices related to reparation, reclamation, and renewal.
