Re-purposing evaluation to learn about social justice: Reconfiguring epistemological politics through the regulative ideal of ‘participatory parity’

Abstract

The article aims to re-purpose evaluation to learn about social justice by anchoring evaluation in normative dimensions. This article demonstrates the ways in which evaluation with an establishment orientation can limit the scope for dialogue and neglect narratives that contest the status quo. It explains how a more participatory approach that engages with the standpoints of marginalised participants can enhance the potential to learn about social justice. An ethical commitment to social justice does not mean a rejection of rigour in evidence-based evaluation. Relating Fraser’s critical theory of participatory parity to the regulative ideal of evaluation creates a foundation to systematically foreground explanations about how an intervention has delivered social justice.

Introduction

All evaluation approaches are in some way pedagogical, although there is variation in what they teach and convey (Patton, 2017). Evaluation has the potential to produce knowledge to learn about social justice. But reflecting on 40 years of conducting evaluations, Mathison (2018) concludes that evaluation has not fulfilled its potential to contribute to transformative social change. She identifies three reasons for this: first, the practice of evaluation is constrained by dominant ideologies; second, that evaluation is not independent and has become a service provided to those with power; and third, that evaluation is a practice that follows pre-determined boundaries and so produces knowledge to support learning within the frames of the status quo.

Evaluation is assembled from a broad diversity of disciplines and has its own concepts, toolkits and epistemic communities (Picciotto, 2017); yet as Kuhn’s (1962) work on scientific paradigms shows, consensus in scientific research stabilises what counts as evidence in different modes of inquiry. Evaluation that is considered to be more robust by policy-makers and grant-making organisations tends to be concerned with ‘what is’ and not ‘what ought to be’ (Stame, 2018: 440). This article will engage with critical questions about what purpose this serves and for whom.

The article aims to re-purpose evaluation to be able to better learn about social justice by anchoring evaluation theory and practice in social justice values and ideals. An engagement with normative commitments to social justice does not mean a rejection of rigour in evidence-based evaluation; nor does it mean abandonment of the fundamental aim to produce causal explanations. An ethical commitment to social justice requires an alternative mode of social scientific inquiry for evaluation. Bevir and Blakely (2017) identify a ‘philosophical conflict’ between naturalist and anti-naturalist philosophies of social science research. The contested role of normative foundations in evaluation can be framed through this philosophical prism. Naturalism arises from a belief that the social and natural worlds are similar enough that they should be studied in the same way and that an ‘objective’ process of social science can be achieved (Bevir and Kedar, 2008). In contrast, anti-naturalism is rooted in the position that human agency makes social science incompatible with the methodologies of natural science and so demands a critical engagement with normative foundations as a key dimension of social research (Bevir and Blakely, 2016: 31).

This article will argue that an engagement with normative foundations can broaden the potential for evaluation to learn about social justice. Re-purposing evaluation requires a reconfigured epistemological politics. Epistemological politics is the idea that certain types of knowledge and knowers are privileged over others. In Decolonizing Methodologies, Smith (2013) asks researchers to develop a ‘more critical understanding of the underlying assumptions, motivations and values which inform research practices’ (p. 20). As Collins (2019: 127) explains, social injustice can be reproduced through the application of apparently objective rules, and the substance of these rules can serve to perpetuate social injustice.

The implications of a reconfigured epistemological politics of evaluation are explored through this article, and it will be explained how a more critical participatory approach can enhance the capacity to learn about social justice. Fraser’s (Fraser and Honneth, 2003) critical theory of participatory parity is related to the regulative ideal of evaluation to reconfigure the epistemological politics of evaluation to be more inclusive of participants and sensitive to injustice. Participatory parity as a regulative ideal deliberately distributes the power to shape the judgements from an evaluation and ensures that the knowledge of people who experience injustice is valued. By doing so, concerns about social justice are foregrounded. This article aims to reconfigure the epistemological foundations for a re-purposed evaluation that is empirically informed, normatively defensible and practically orientated to assess how an intervention has delivered social justice. Through relating the regulative ideal of participatory parity to evaluation theory and practice, it is possible to realise Stame’s (2018) ambition that ‘an evaluation should be able to perform two tasks: document discrimination and injustice; and highlight autonomous ways of finding solutions [to social problems] through empowerment, creativity and the mobilisation of hidden resources’ (p. 448).

Picciotto (2015) demands that the time has come for the field of evaluation to adopt a more activist outlook: activist evaluators can challenge the status quo, promote new social policy approaches, and make the political dynamics behind the policies and programmes that they assess more visible. The approach advanced through this article contributes towards this movement and will be most useful for activist-scholars who can use evaluation to support learning about social justice.

An establishment orientation to evaluation that limits the scope for learning

Weiss (1993) argued that political influences mean that evaluation predominantly has an ‘establishment orientation’, which tends to produce learning within the frames of the status quo. Weiss (1993: 101) identifies three political influences that result in an establishment orientation: first, the agency that is responsible for designing the intervention tends to commission the evaluation and define its terms; second, the evaluator’s boundaries of investigation are pre-determined by the politics that shaped the original design of an intervention; and third, that evaluation reports are primarily produced for policy-makers rather than the public. Establishment-oriented forms of evaluation are based on questions of ‘what is’ and neglect more critical questions of ‘what ought to be’. The primary purpose of establishment-oriented evaluation is to produce knowledge that can be used to support technocratic policy-making. Mathison (2018) argues that ‘while it is entirely appropriate for evaluation practitioners to work in this way, it blunts our attention to big questions about why this intervention, why this strategy, why these people and not others?’ (p. 117).

Fischer (1993) argues that the discursive practices of dominant modes of knowledge production, such as establishment-oriented evaluation, should not be seen as value-free constructions of knowledge as they reflect values consistent with the status quo. Technocratic forms of policy-making are rooted in naturalist philosophies of science that reject an engagement with competing narrative frames (Bevir and Blakely, 2016). Establishment-oriented evaluation is rooted in the naturalist philosophy that aims to produce an ‘objective’ social science, constructed as a depoliticised practice that at the very least aims to have objectivity as ‘a regulative ideal’ (Smith, 1990). The idea of objectivity as a regulative ideal is ‘about keeping the inner workings of science – its inferences, its data – free from value judgments’ (Scriven, 2007: 6). Through an ambition to conduct ‘value-free’ social science, establishment-oriented evaluators proceed from the naturalist basis that the social world is a neutral object, upon which they can apply their theories as instruments of ‘discovery, description, classification, and explanation’ (Bevir and Kedar, 2008: 508).

But if we think about research as a purely technical process, then we ignore the significance of power relations in shaping research itself (Collins, 2019: 143). Critical debates in the field of evaluation are fundamentally about epistemological politics – ostensibly about who has the power to shape the knowledge that is produced (Schwandt, 2007). The epistemological frames that are selected by evaluators serve to construct evidence in particular ways (Learmonth and Harding, 2006). Evidence that is produced through establishment-oriented evaluation reflects discourses that are rooted in expert and technocratic objectivism (Vanderplaat, 1995: 83). Narratives that might contest dominant discourses through making inequalities visible are often neglected as they are outside the frames of investigation.

The regulative ideal of objectivity aims to reduce the role of interpretive values in evaluation (Smith, 1990). There is limited scope for dialogue with participants that might contest the ways in which an intervention has originally been framed, shift the boundaries of investigation, or contest dominant discourses. A rejection of the constitutive role of normative foundations leads to a privileging of the position of evaluators. Evaluators are positioned as objective scientists who are uniquely placed to provide explanations of an intervention through a value-free assessment of empirical data (Bevir and Kedar, 2008: 507). Through an establishment orientation, the evaluator is valorised as the ultimate arbiter, making judgements against criteria that have often been determined by the policy-makers who designed the intervention. There is limited space in the process for the participants to act as subjective agents who can construct their own explanations to disrupt the original framing of an intervention.

The original frames of an investigation remain undisturbed through establishment-oriented evaluation, with too few opportunities for participants to shape the contours of the study. For instance, Randomised Controlled Trials (RCTs) aim to ensure that the research ‘objects’ who have received a particular intervention are kept ‘blind’ to the process of evaluation. Closed forms of research are designed by the evaluators and delivered through standardised surveys. These surveys produce data to test whether the intervention has met its original objectives. The data produced through the surveys can then be statistically analysed. A closed approach is taken to maintain the scientific discipline of the inquiry. The capacity to produce knowledge through an establishment-oriented evaluation is tightly controlled following the regulative ideal of objectivity. The possibilities for co-constructing judgements about an intervention with participants are limited.

RCTs can add diversity through delivering the studies in different locations, with different populations, and in different settings. However, evaluators conducting experiments still tend to demand a standardisation of the instruments used to test whether the original policy objectives have been met. While recognising the inevitable trade-offs, Katz et al. (2011) argue that the tensions between participatory research methods and RCTs are not insurmountable. Katz et al.’s suggested method of bridging the divide is the ‘Multisite Translational Community Trial’. Like conventional RCTs, they insist that all sites must have standard intervention components, outcome measures, data capture methods and analytical approaches. Where Katz et al.’s approach differs from conventional RCTs is that the means of delivering the intervention and reaching the population of interest can be shaped by the community. Katz el al.’s approach still prefers ‘blinding’. It remains based on standardised outcome measures and analytical approaches rooted in closed forms of research. While Katz et al. think creatively about how to reconfigure the epistemological approach to experimental evaluation, the dialogic scope of their approach remains fundamentally limited. Participants can shape how the intervention is delivered, but they cannot contribute their interpretations to shape the frames of investigation used to arrive at evaluative judgements, nor can they influence the final explanations.

Uncritically accepting the original framing of an intervention to make evaluative judgements is a normative choice (Berk and Rossi, 1976: 339). Establishment-oriented evaluations use pre-determined criteria for assessment to evidence outcome judgements and tend not to provide a value-critique of the original aims of the intervention (Taylor, 2006). The possibilities for critical assessment of the values that are internal and associated with the intervention remain constrained (Gates, 2018: 210). The external values that shape the construction of policy ‘problems’ and ‘solutions’ are considered outside the boundaries of investigation for an evaluation (Silver and Crossley, 2019). The idea that the primary purpose of evaluation is to find out whether an intervention delivers the originally designed effect(s) is rarely contested (Stame, 2018: 439).

Following a naturalist approach to science, establishment-oriented evaluators privilege qualities of control, measurement and precision – qualities which are seen as more robust ways through which to produce credible evidence. Establishment-oriented evaluation has become ‘incorporated into managerialist philosophies and embedded in bureaucratic and administrative practices and standardised procedures for monitoring and performance assessment’ (Schwandt, 2017: 548). The standardisation of competencies required from evaluators has meant that establishment-oriented evaluation has too often become a mechanistic assessment into whether an intervention’s original objectives have been achieved (Dahler-Larsen, 2012, cited in Julnes and Bustelo, 2017: 542) – and so neglect the scope for learning from more critical explanations that surface alternative approaches and make structural inequalities more visible.

Evaluation has become increasingly technocratic and based upon methodological determinism that can result in a ‘tool kit approach’ to evaluation (Schwandt, 2017: 549). Through this process, technocratic knowledge is produced that can be used instrumentally within the frames of existing social policy (Knorr, 1977, cited in Daviter, 2015: 493). Technocratic knowledge produced through establishment-oriented evaluation is intended to contribute directly towards engineering changes to social ‘problems’ that have been defined by policy elites (Head, 2013). As noted by Cox (1996, cited in Lingard, 2013), the problem-solving model ‘accepts the world as it finds it, including existing power relations, inequalities, and oppressions, and thus accepts the status quo as the framework for action’ (p. 120).

Transformation away from the status quo is required in a world that is characterised by increasing social and economic inequality, far-right nationalism, and ecological catastrophe.

We must ask the following question: how can evaluation produce robust knowledge that can contribute towards a more socially just world?

A reluctance to engage with normative foundations through evaluation leads to a closure of possibilities for learning from competing narratives that can challenge the status quo. Harding (1991: 10–11) demonstrates that ‘science is politics by other means’, explaining how scientific knowledge is always ‘socially situated’, and that knowledge production can never be ‘impartial, disinterested, or value neutral’. Assessing pre-determined criteria should not preclude an openness to critical engagement with ‘the unexpected, hidden rationalities and emergent or new values’ (Stame, 2018: 447). By restricting inquiry within investigative frames that have been pre-determined (and which usually reflect the values of the status quo), the scope for producing knowledge about social justice beyond these frames is limited.

Drawing on Foucault’s notion of disciplinary power, Chouinard (2013) argues that the adherence of establishment-oriented evaluation to dominant discourses and standards can be conceptualised as ‘technologies of control’. She identifies how technocratic notions of accountability have become separated from the democratic principle of pluralism and the existence of competing normative frames. By limiting the scope for multiple explanations, establishment-oriented evaluation is rarely used to develop new political imaginations that consider alternative approaches to political economy or different types of social protection (Reddy, 2012: 62).

Vanderplaat (1995: 86) argues that the establishment orientation of evaluation produces knowledge that limits the types of questions that can be explored; she explains how establishment-oriented evaluators position themselves as ‘disinterested and apolitical informants’, and as such they do not address questions of ‘what ought to be’, rejecting the possibilities of producing knowledge for social justice beyond dominant discourses and frames of social policy. An alternative approach is needed that opens up evaluation processes to more systematically engage with narratives that contest unequal relations of power. The following section engages with ideas that have challenged the establishment orientation of evaluation and builds on these as the foundations for a new form of counter-establishment evaluation that deliberately foregrounds an ethical commitment to social justice.

A counter-establishment orientation to learn about social justice

As critical theorists of evaluation have concluded, evaluation is not value-free, and so normative foundations should be directly engaged with (Guba and Lincoln, 1989; Mertens and Wilson, 2012; Schwandt, 2007; Weiss, 1979). Evaluation involves making a value-judgement through determining the merit or worth of an intervention (Scriven, 1991). Judgements on the value of merit are based on intrinsic concerns, which are informed through an investigation into whether an intervention has been delivered effectively (Alkin and King, 2016: 570). Judgements on the value of worth, on the contrary, ultimately involve making ethical decisions. Making judgements based on normative assessment is therefore an inevitable dimension of evaluative inquiry, and so all evaluators should make their normative foundations explicit (Greene, 1997).

Making judgements in evaluation through normative assessment remains deeply controversial. Greene (1997: 26) explains that ‘the notion of evaluation as advocacy still rings as exquisite heresy to many ears’. Becker (1967) maintained that it is not possible to do research without normative dimensions. The question for Becker is not whether we should take sides, since we inevitably will, but rather whose side we are on. Warren and Garthwaite (2015) explain that Becker was ultimately asking ‘whether social research and the individuals engaged in it were part of the established order or could contribute to the emerging counterculture’ (p. 225). Becker (1967) recognised how research with a counter-establishment orientation is often dismissed by people in positions of authority or those who have a high status that has been achieved through producing establishment-oriented research:

. . . we provoke the charge of bias . . . by refusing to give credence and deference to an established status order, in which knowledge of truth and the right to be heard are not equally distributed . . . By refusing to accept the hierarchy of credibility, we express disrespect for the entire established order. We compound our sin and further provoke charges of bias by not giving immediate attention and ‘equal time’ to the . . . explanations of official authority. (pp. 241–242)

Becker demonstrated that accusations of bias against counter-establishment research can often be unfounded and disproportionate. Exploring ‘what ought to be’ should not preclude a robust investigation into ‘what is’. Evaluation with a counter-establishment orientation can still systematically investigate the design, implementation and effectiveness of an intervention, while also producing knowledge rooted in an ethical commitment to produce knowledge to learn about social justice. Greene (1997) argues that bringing normative commitments into evaluation does not compromise the process, so long as the evaluator does not take a ‘partisan stance toward a particular program’ (pp. 28–29). A counter-establishment evaluation must still determine the merit of an intervention. Evaluators with an ethical commitment to social justice still need to systematically find out how effective an intervention has been and the extent to which it has delivered outcomes that promote social justice.

Normative dimensions are not segregated from empirical research through an anti-naturalist philosophy of science, and competing narratives are used to generate more robust explanations. Instead of a single ‘official’ scientific account, multiple narratives can better reflect the realities of the social world (Bevir and Blakely, 2016: 42). Anti-naturalist researchers do not privilege a particular explanation and all accounts can be contested (Bevir and Blakely, 2016: 41). Engaging with different narratives through dialogue is an important dimension of social inquiry as a way to explain contingency, human agency and situated practices that help researchers make sense of the world and provide valid explanations based on evidence (Bevir and Blakely, 2017; Bevir and Kedar, 2008).

Causation does not need to follow the science of prediction that is dominant in establishment-oriented forms of evaluation (Silver and Crossley, 2019). The causal logic for anti-naturalist evaluation is one of explanation. Instead of aiming to find predictive laws, counter-establishment evaluation can focus upon generating theories to produce causal explanations. The basis of explaining a causal relationship changes from an ‘effects-of-causes-stance’ (a what works question) which is typical for establishment evaluations, to ‘a causes-of-effects-stance’ (why does it work or did it not work?) (Pattyn et al., 2017: 7). Counter-establishment evaluators working from an anti-naturalist foundation can co-construct causal explanations as partners in dialogue with participants (Bevir and Blakely, 2017: 14).

A counter-establishment evaluation rooted in an anti-naturalist philosophy of science does not view research methods as a set of ‘mechanical procedures’ that can be applied to discover an objective truth, but rather as a rigorous means through which to understand an intervention (Sanderson, 2000). Pluralism, devolution of expertise and dialogue with participants are central to anti-naturalist social science (Bevir and Blakely, 2017: 13), representing an ‘epistemic commitment to dialogical engagement’ (Collins, 2019: 144). The practice of evaluation should be seen as a ‘relational public craft’ (Schwandt, 2017: 551), through which the evaluator’s role is to enable ‘assisted sense-making’ (Mark et al., 2000, cited in Dahler-Larsen, 2013: 156). A counter-establishment form of evaluation can develop this relational public craft by cultivating the sociological imagination, defined by Mills (1959) as

. . . the capacity to shift from one perspective to another. It is this imagination . . . that sets off the social scientist from the . . . technician. Adequate technicians can be trained in a few years. The sociological imagination can also be cultivated . . . Yet there is an unexpected quality about it, perhaps because its essence is the combination of ideas that no one expected were combinable . . . There is a playfulness of mind back of such combining as well as a truly fierce drive to make sense of the world . . . Perhaps [the technician] is too well trained, too precisely trained. Since one can be trained only in what is already known, training sometimes incapacitates one from learning new ways . . . (p. 211)

Mills articulates a vision of research as a craft, which is open to multiple perspectives and embraces the unpredictable. Developing the sociological imagination opens up the potential to connect everyday problems with public issues. Counter-establishment evaluation as a craft of ‘assisted sense-making’ involves working alongside participants in the search for explanations of an intervention that might not have been predicted. Narratives of explanation that might sit outside pre-determined boundaries can be critically engaged with. As the scope of evaluation is not restricted within pre-determined boundaries of investigation, there are more possibilities to generate learning about social justice beyond dominant discourses and frames of social policy.

Promoting the inclusion of research participants to be able to co-construct explanations through dialogue does not dismiss the knowledge of policy-makers or evaluators. Rather, it is a demand to abandon the idea of a ‘privileged position’ in determining the final evaluative judgements about an intervention (Vanderplaat, 1995: 91). Multiple ways of seeing and knowing can enable an epistemic advantage to enrich an evaluation study (Greene and Caracelli, 1997). As Fine (2011) reasons, ‘knowledge of injustice is widely distributed even if legitimacy is not’ (p. 14). Feminist standpoint epistemologies identify a particular promise for learning about social justice from the views of marginalised groups. These theories are based on the conviction that people who have direct experience of struggling against social injustice on a daily basis produce unique insights. Forms of daily resistance against injustice have often secured better conditions and delivered social justice more effectively than formal political institutions (Harding, 1991: 130). A commitment to social justice demands systematic engagement with the standpoints of participants who experience social injustice. When aiming to develop an evaluation approach to learn about social justice, it is therefore crucial to value the experiential knowledge of marginalised communities.

The traditions of participatory evaluation are based on the inclusion of marginalised communities to promote social justice. Participatory evaluation is based on collaboration between evaluators and community members (Chouinard and Cousins, 2015: 6). Participatory evaluation has been put forward as an approach that can overcome the problems with technocratic evaluation (Smits and Champagne, 2008). Participants have control at each stage of the evaluation process (Cousins and Chouinard, 2012; Cousins and Whitmore, 1998). There are three main dimensions of participatory evaluation, along which the depth and range of participation lies: first, the degrees of control over technical decisions of the evaluation processes and conduct; second, the selection of the types of stakeholders who participate; and third, the participation that happens through different stages of the research process (Cousins and Whitmore, 1998: 9). Participants can shape the ways that an investigation unfolds through an evaluation.

There are two models of participatory evaluation (Cousins and Whitmore, 1998: 6). The first is practical participatory evaluation, which is more instrumental and aims to support programme decision-making. The second model is transformative participatory evaluation, which aims to empower marginalised communities through a process of radical education. The transformative model of participatory evaluation is based on the theories of Paolo Freire and the work of the Highlander Research and Education Center (Gaventa, 1988). Freire (1970) emphasises the importance of dialogue to develop critical analysis of social action through reflection on practice, supporting participants to situate their everyday problems in social, historical and political contexts. Patton (2017: 69) identifies four ways that evaluation based on radical pedagogy can support learning about social justice: first, the definition of a problem that is being evaluated can be reconsidered; second, dominant assumptions can be exposed through experiential knowledge; third, alternative explanations about an intervention can be put forward; and finally, strategic points of social change can be identified through discussion.

More epistemologically open forms of evaluation that engages with participants’ narratives can generate new explanations and provide critical insights into social justice. Engaging with narratives does not mean we abandon systematic social research. Haraway (1988) identifies a danger of romanticising the standpoints of the most marginalised. Privileging particular standpoints can leave them unaccountable, critically unexamined and at risk of relativism. Instead of romanticising the standpoints of marginalised communities, Haraway (1988: 584) argues that scientists should aim for ‘passionate detachment’ and seek to locate these standpoints within broader frames of knowledge.

Haraway’s idea of situated knowledges is intended as a scientific approach against both subjective relativism and objective universalism. Haraway makes the case that subjective narratives must be accountable to the objective conditions in which they are situated. One way to make standpoints more accountable is to engage in dialogue. While it is important to remain sensitive about how to proceed with these discussions (Patrick, 2020: 17), dialogue between evaluators and participants can produce knowledge that is made more accountable by situating particular experiences of injustice in critical theories of social science.

The judgement of quality in research should take account of the epistemological foundations from which it is built (Fossey et al., 2002). The rigour of an evaluation can be ensured by validity measures and the inclusion of different cases through empirical investigation. Lincoln and Guba (1985: 314) describe member checking as ‘the most crucial technique for establishing credibility’. Member-checking distributes validity procedures between researcher and participants (Creswell and Miller, 2000). To achieve a valid foundation for evaluation, different experiences of an intervention must be critically examined. Heterogeneity therefore becomes a source of knowledge and validity.

Critical participatory research that engages with narratives and foregrounds social justice can strengthen the construct validity of an evaluation by enabling the people most adversely affected by social policy to re-shape the frames of inquiry (Sandwick et al., 2018: 479–480). Changing Minds (Fine et al., 2001) was a participatory evaluation conducted by women in prison and from outside. The evaluation was instigated after federal funding had been cut when President Clinton made incarcerated people ineligible for Pell grants in the 1990s. This policy resulted in a reduction in college programmes in prison from 350 programmes in 1994 to just 8 in 1995. The aim of the evaluation was to ‘generate science that would influence state and national policy on college and prison’ (Fine, 2019: 85) A research collective was created that included women ‘inside’ at Bedford Hills Correctional Facility and women from ‘outside’ at City University New York. These women had different insights rooted in their diverse experiences:

Some were involved in grassroots advocacy; some were mothers; some spoke Spanish as their first language; some were raised in poverty, some with wealth; some had confronted serious health issues; some held multiple degrees, some had just begun college while in prison. (Sandwick et al., 2018: 480)

The women who were in prison made the case that the evaluation should investigate the ways in which the college at Bedford Hills affected many other interactions – for instance, discipline, children’s education and interactions on cell blocks and in the yard. Sandwick et al. (2018) report that through an enhanced understanding of the construct ‘college in prison’, the research collective expanded the investigative frames of the evaluation (p. 480). The frames of investigation were broadened to include evidence not only about reductions in recidivism (and associated cost savings), but also changes in prison culture and the effects on the families of the women in prison. By including the critical insights of the women in prison, the frames of investigation were broadened and the validity of the evaluation was strengthened. The evaluation documented the causes of structural injustice and evidenced the impacts from a solution that was at hand. The researchers produced a robust critique of the political economy of racialised capitalism and patriarchy, and identified the impacts of an intervention to disrupt these inequalities.

Evaluation creates a rare opportunity for marginalised groups to influence public deliberation (Vestman and Conner, 2013). Advocating the position that evaluation should meaningfully engage with the standpoints of people who are the most marginalised does not mean other viewpoints are excluded (Greene, 1997). To do so would invite legitimate questions concerning the validity of an evaluation. More democratic processes of evaluation can avoid accusations of bias through an alternative conception of objectivity.

In order to reconfigure the regulative ideal of objectivity to broaden the scope for dialogue, democratically oriented approaches to evaluation have advanced the regulative ideal of democratic pluralism (Greene, 1997). Democratic pluralism as a regulative ideal is designed to make sure that an evaluation provides a balanced and complete view. Marginalised participants are actively sought out and supported to engage in an evaluation, but their explanations of an intervention do not produce an exclusive evidence-base for judgement. Triangulation of data through a mixed-methods approach that includes a full range of perspectives from all stakeholders can help achieve a valid evaluation (Mertens, 1999). Through a more democratic regulative ideal for evaluation, the absolute authority of the evaluator to make judgements about a programme becomes more distributed and balanced. The judgement of an intervention is shaped through a collective process of explanation. These judgements are fair and unbiased, not because they are impartial, but because they are multi-interested (Greene, 1997: 33).

A regulative ideal of democratic pluralism can provide an alternative conceptualisation of rigour for evaluation, while enhancing the potential to learn about social justice. As explanations of an intervention are co-constructed with participants, evidence and outcomes can be contested (Arensman and Van Wessel, 2018). As different narratives are potentially valid, an evaluation must draw on evidence to construct an explanation and demonstrate why it is preferable to others. Normative foundations can provide guidance through which to inform how this might be done. Critical theorists of evaluation do not share a consensus about the particular social and political values that should influence evaluation (Gates, 2018: 209). The following section sets out a new regulative ideal for evaluation based on an ethical commitment to social justice by bringing Fraser’s critical theory of participatory parity together with the regulative ideal of democratic pluralism. Participatory parity as a regulative ideal can contribute to evaluation theory and practice by providing a common foundation for systematic research into social justice.

‘Participatory parity’ as a regulative ideal for evaluation

All evaluations are ‘bounded’ through the selection of certain criteria of performance. These boundaries define the frames of investigation that will take place. Boundaries are established according to what evidence is considered necessary to judge an intervention. Choices must be made to determine the boundaries that frame the investigation. The boundaries for establishment-oriented forms of evaluation are often pre-determined by commissioners and followed by evaluators (Picciotto and Donaldson, 2016: 72). But neither commissioners nor evaluators have an ‘in-principle advantage’ in determining the criteria for judgement (Ulrich, 2000: 8). Instead, we can collectively create boundaries through deliberation between competing narrative claims about facts and values, in which critical decisions on the value of worth of an intervention are not made by the evaluator alone, but rather through a participatory process that the evaluator facilitates (Schwandt, 2018: 133).

Ulrich and Reynolds (2010) present boundary critique as a ‘participatory process of unfolding and questioning boundary judgements rather than as an expert-driven process of boundary setting’. Co-constructing the boundaries of an evaluation with participants can situate the criteria for assessment in how an intervention has created outcomes for participants in the context of their particular experiences of injustice. Through the collective reconstruction of boundaries to frame an investigation, evaluation becomes a ‘collective undertaking rooted in social interaction and negotiation’ that is situated in the local contexts in which an intervention takes place (Schwandt, 2018: 131).

The regulative ideal of objectivity results in the boundaries for judgement reflecting the ways in which an intervention has been originally framed – which privileges the often pre-determined analytical explanations of the agency responsible for designing the intervention, rather than the people who have experienced it. Narratives grounded in experience that might contest the original framing of an intervention or highlight structural inequalities can be neglected within establishment-oriented evaluations. As a result, the potential to document social injustice and learn about alternative approaches that can deliver social justice is limited.

Deliberative democratic evaluation is based on the regulative ideal of pluralism and is designed to ensure that evaluation provides a balanced and complete view, as the perspectives from all stakeholders are included. The evaluator must reflect the results of deliberation through analysing competing values as well as facts, representing all stakeholders’ views and ensuring that the necessary steps of deliberation have been taken to arrive at findings (House and Howe, 2000). It is through ensuring fair access to deliberation to reach evaluative conclusions that deliberative democratic evaluation advances a normative foundation of justice (House and Howe, 2000). Within the model of deliberative democratic evaluation, the final judgements about value on the merit or worth of an intervention are made by the evaluator. The justification of the evaluator’s position as final judge is rooted in ideas of procedural justice, which demands that all voices have a fair hearing and are involved, but that the perspective of less powerful voices are not necessarily advocated for (Vestman and Conner, 2013). Such a purely procedural view of pluralism limits normative concerns to inclusion in the process of evaluation. But critical deliberative democratic scholars have demonstrated that unequal relations of power shape what is considered to be a better form of argument, and so inclusion in debates is not sufficient to promote equality (Benhabib, 1992; Sanders, 1997). Participatory parity as a regulative ideal provides the foundations to be able to push the normative dimensions of democratic evaluation further by systematically distributing power to participants who experience injustice to shape the boundaries of investigation and foreground concerns about social justice.

Relating Fraser’s norm of participatory parity to evaluation as a regulative ideal deepens the normative foundations of deliberative democratic approaches. Participatory parity brings together moral philosophy, social theory and critical analysis of the political economy to provide a democratic epistemology that is normatively and practically defensible (Fraser and Honneth, 2003: 4). Relating participatory parity to the regulative ideal of evaluation grounds the process in a theoretically robust foundation of social justice.

Participatory parity is a two-dimensional normative theory of justice that considers both intersubjective and objective conditions that undermine equal participation in society. Fraser (Fraser and Honneth, 2003) draws together a two-dimensional theory of justice that accommodates defensible claims for recognition and redistribution. Fraser identifies intersubjective ‘misrecognition’ as cultural and symbolic injustice that is rooted in social patterns of disrespect including stigmatisation in public cultural representations and daily life; cultural domination where patterns of communication associated with another culture are constructed as alien or hostile; and non-recognition in which people are rendered invisible through representative, communicative and interpretative practices. Objective conditions of injustice are identified as being rooted in the systemic social and economic inequalities (re)produced by the dominant political economy (Fraser and Honneth, 2003: 7). While misrecognition and distributive injustices are separate, both are ‘primary and co-original’ (Fraser and Honneth, 2003: 18). Neither claims hold more weight than the other, and people can experience injustice based on an intersection of both. For instance, redressing gender injustice requires an attendance to both the unequal cultural status of women and the economic dimensions of inequality that effect women. The relationships and intersections between injustices are situated within their particular conditions, which must be empirically studied (Collins, 2019: 40). If participatory parity is achieved, all people have equal access to participate in social life as objective and intersubjective conditions of injustice have been addressed.

Participatory parity creates a normative standard through which to make judgements about how an intervention has responded to objective and intersubjective conditions of injustice (or the ways in which it has failed to). By using the normative standard of participatory parity, it is possible to evaluate the extent to which an intervention has fostered participatory parity or undermined it. In order to be able to make such a judgement, evaluators must engage in dialogue with participants to determine how social injustice affects their capacity to participate fully in social life. The experiential knowledge of participants should be contextualised in relation to their objective location in the social world. As such, evaluators should have knowledge of critical theories to be able to situate the everyday struggles of participants in broader systems of injustice.

Boundaries of investigation can be co-constructed with the participants through a critical engagement to broaden the frames of investigation. Once the particular problems of participants have been identified, it is then possible to collectively evaluate the extent to which an intervention has provided a remedy to these forms of injustice. Co-constructing the boundaries of an evaluation with participants can situate the criteria for assessment in particular contexts of social injustice. The capacity for evaluation to learn about social justice is enhanced, and it is rooted in a theoretical foundation that can ensure different dimensions of injustice have been meaningfully engaged with.

Participatory parity can guide deliberation in evaluation to make inequalities visible and assess social justice. Using participatory parity as a regulative ideal enables a more critical participatory epistemology, which enhances the sensitivity of evaluation to particular experiences of injustice. Based on an assessment of how objective and subjective conditions for justice have been fostered, narratives that can explain how participatory parity has been achieved (or not) can be reasonably considered to be more defensible when assessing competing claims. A more radically inclusive interpretation of pluralism can be achieved – pushing the traditions of democratic evaluation forward.

A normative commitment to social justice must remain ‘critical and open-ended’ (Collins, 2019: 284). Participatory parity as a regulative ideal does not mean other viewpoints are excluded. Conclusions are not foreclosed, nor any groups excluded from the process of explanation. No given view is indefeasible, and no explanation is infallible (Fraser and Honneth, 2003: 44). To do so would legitimately invite questions concerning the validity of an evaluation.

Through a counter-establishment, evaluation judgements are collectively determined through dialogue in the evaluation process. As such, participatory parity can be used as a regulative ideal to assess the relative justice claims of competing narratives. As Fraser (Fraser and Honneth, 2003: 42) explains, there is not an ‘algorithmic metric or method’ to determine how participatory parity has been fostered; rather, through discussion and argumentation, ‘conflicting judgements are sifted and rival interpretations are weighed’ against the normative standard of participatory parity. The validity of the evaluation process is strengthened as a result of using participatory parity as a regulative ideal. Using participatory parity as a regulative ideal for evaluation provides guidelines through which to re-draw the boundaries of investigation so that narratives usually marginalised are instead promoted, and concerns for social justice become foregrounded when judgements are made. Through a regulative ideal of participatory parity, an intervention is put through a robust form of questioning concerning the extent to which it has addressed the participants’ particular experiences of social injustice. Following feminist standpoint theories that highlight the value of experiential knowledge to contribute understanding about social (in)justice, assessments produced through an evaluation using participatory parity as a regulative ideal can enhance learning about social justice. Participatory parity as a regulative ideal enables evaluators to ensure that this process is transparent, valid and systematic.

To illustrate how the regulative ideal of participatory parity can work in practice, we can look at an evaluation of an intervention designed to address food insecurity among vulnerable groups; food insecurity was defined as not always having the economic, social and physical resources to ensure a sufficient supply of nutritionally appropriate food (Purdam and Silver, 2020). The intervention was based around a series of cooking classes for people living in food insecurity. The intervention had received local government funding based on the idea that through improved cooking, people would eat healthier food and would therefore be more ‘resilient’ to food insecurity – predicated on a theory of behavioural change. The evaluation had initially been designed to evidence outcomes in terms of improved cooking skills. Following an establishment orientation to evaluation, these initial frames would determine the boundaries of judgement. While the evaluation did investigate these lines of inquiry, it also remained open to alternative explanations.

Through an embedded approach to evaluation grounded in dialogue with participants, the research identified that the primary cause of participants not eating healthily was due to material conditions such as low budgets or lack of cooking facilities, rather than a deficit of cooking skills (as the intervention had been originally framed). The boundaries of judgement were shifted by narratives that unfolded through the evaluation and contested the original framing. These narratives were situated within broader frames of knowledge. Participatory parity as a regulative ideal provided a systematic basis for determining the merits of each claim. Critical theories of political economy were drawn upon that located the root causes of food poverty in welfare reforms (objective injustice), together with research (Garthwaite, 2016) that situates the aim to improve cooking skills of people living in food insecurity as a response rooted in stigmatisation (subjective injustice). Through a more open approach to evaluation, the importance of providing material support for people (through the direct provision of healthy food ingredients and material items, such as ovens) was identified as a key outcome to deliver social justice in regard to food insecurity. In addition, the ways in which the intervention provided non-judgemental support for participants in response to stigmatisation were included. As these explanations reflected measures to address participatory parity, they were prioritised as being more pertinent than the original framing of increasing cooking skills. Using the regulative ideal of participatory parity, these claims held more weight than explanations that privileged the initial anticipated causal relationship between improved cooking skills and healthy eating.

The boundaries of judgement for the evaluation were shifted to consider how the intervention was able to respond to objective and subjective inequalities. Evidence was still presented on the original objectives, but new evidence was needed to assess how the intervention was responding to the participants’ particular experiences of injustice. As a result, the evaluation process demanded further engagement in the field to collect data on how the intervention was providing material support for participants, and the extent to which the intervention was non-judgemental and did not cause stigma. New explanations were produced which documented a broader set of outcomes that responded to the particular injustices experienced by participants.

The evaluation documented discrimination and injustice by evidencing causes of social injustice through welfare reforms and stigmatisation of people living in food insecurity; it was able to highlight local solutions to social problems through the mobilisation of community resources and creation of a supportive and non-stigmatising space for participants. Participatory parity provided a robust mechanism through which to interrogate the original framing of the intervention and guide further investigation into how the intervention was responding to participants’ particular experiences of injustice. The evaluation surfaced knowledge about an alternative approach to addressing food insecurity and produced evidence about the structural causes of injustice that require remedy beyond the scope of the intervention. Through using participatory parity as a regulative ideal, the purpose of evaluation can be broadened to contribute towards what Weis and Fine (2012) identify as a ‘dual lens of critical bifocality’, which documents the structural causes of social injustice and highlights resistance to these inequalities.

Conclusion

This article has argued that an engagement with normative foundations broadens the potential for evaluation to produce knowledge about social justice. Relating Fraser’s critical theory of participatory parity to the regulative ideal of evaluation provides a framework to ensure that problems of social inequality are foregrounded in evaluation – within which critical theories can be drawn on to situate these within broader structures of social injustice. Through a reconfigured epistemological politics that is more radically pluralist, evaluation can comprehensively document particular experiences of injustice and evidence the ways in which an intervention has responded to these (as well as the structural conditions that require further changes beyond the intervention site).

The approach advanced through this article to re-purpose evaluation to learn about social justice is not accompanied by a toolkit of methods or techniques – an extensive range of methods already exist that are consistent with the counter-establishment demand to surface multiple narratives. Nor does it mean we abandon methods associated with a naturalist philosophy of research as the pluralist dimensions of counter-establishment evaluation can be strengthened by treating these methods as one source of knowledge to be assessed within broader contexts and narrative positions (Byrne, 2002). Rather, it is a call to reflect on how we approach research in a more open and ethical way. In the Art of Listening, Back (2007) invites us to ‘pay attention to the fragments, the voices, and the stories that are otherwise passed over or ignored [and to] admit these voices and pay them the courtesy of serious attention’. Participatory parity provides a guide for a reconfigured epistemological politics of evaluation to enable us to fulfil this courtesy. By doing so, we can bring in new frames of understanding to enhance the capacity to learn about social justice. Exploring ‘what ought to be’ should not preclude a robust investigation into ‘what is’. Participatory parity as a regulative ideal can enhance rigour in evidence-based evaluation while anchoring the process in an ethical commitment to social justice.