Sage Journals: Discover world-class research

Abstract

The growing interest in systems-change initiatives sits alongside increasing pressure to demonstrate value for money (VFM), which is challenging for emergent, interconnected and often intangible work. A new way to assess the VFM of systems-change work involves considering the value of changing system conditions and the creation of potential value for future systems transformation. This innovation combines the Water of Systems Change framework with the Cycles of Value Creation to create five types of value for systems-change work: inherent, potential, applied, realised and transformative. The VFM of systems-change efforts is then best understood, negotiated and judged by different stakeholders through democratic deliberation. A practical case study of WorkWell (a systems-change initiative to improve mental well-being in Victoria, Australia) is used to demonstrate the use of the approach and offer reflections on its rigour.

Keywords

deliberative democratic evaluation rubrics systems-change evaluation value for money valuing

Introduction

Governments and other actors are increasingly interested in addressing the deep-rooted, complex and interconnected issues facing society – the idea of creating systems change or ‘shifting the conditions that are holding the problem in place’ (Kania et al., 2018: 2). At the same time, systems-change efforts sit alongside ever-increasing pressure to demonstrate the value for money (VFM) of government spending (where VFM is understood as the good use of resources to create changes that people consider valuable (Gargani and King, 2024)) – creating a burgeoning demand to assess the VFM of systems-change initiatives.

However, this is no easy feat. The challenges of evaluating systems-change work are well articulated in the literature. These initiatives operate in dynamic and unpredictable systems where change is nonlinear, often emergent in nature, and driven by many actors, with a long time to see impact (Gear et al., 2025). Assessing the VFM of systems-change work is particularly tricky. Established methods of evaluating VFM such as Cost Benefit Analysis require the evaluator to know, or be able to predict, likely outcomes of an intervention to be able to place a monetary value on these outcomes (King, 2017). However, in systems change, outcomes are ‘dependent on the deeper dynamics of a whole system . . . It is not possible to predict [outcomes], because we cannot know how things will change with each iteration, and what choices will open up to us’ (Burns, 2014: 5).

In this paper, I respond to the challenge of assessing the VFM of systems-change initiatives by presenting a framework, anchored in democratic deliberation, for the different types of value creation. I posit that the complexity of systems-change work is best understood, negotiated and judged by multiple stakeholders through dialogue and deliberation. However, in order to navigate this complexity with diverse stakeholders, we need to ‘simplify and structure it [in order] to draw meaningful conclusions’ (Bouyousfi and Ouedraogo, 2025: 7). The framework achieves this goal by conceptualising five types of value created through systems-changes work: inherent, potential, applied, realised and transformative value. I then explore how I practically applied this framework to the case study of WorkWell and offer reflections on the rigour of the approach for other professional evaluators attempting to conduct a VFM assessment of a systems-change programme or other such complex initiative.

The shift towards deliberative valuing

Apgar et al. (2024: 101) noted recently that values are making a ‘comeback in evaluation’. That is, there has been an increased focus on the role that values play in evaluation, and how to make underlying values more explicit and more transparent. As Gullickson and Hannum (2019: 163) explain,

Decisions about what gets evaluated, the focus and nature of evaluation, what evidence is (or is not) deemed credible, and who makes the narrative and judgement about merit, worth and significance are all rooted in an axiological stance which, by definition, is not and cannot be objective.

Where there are competing values in an evaluation, King et al. (2013: 12) contend that the evaluator should ‘intentionally surface these values and [apply] them in a transparent, systematic way that results in valid and credible judgements’.

One way in which an evaluation can surface values and reach judgements is through deliberation. Deliberation is the process of reasoning to reach conclusions (Abelson et al., 2003) – and, when done as a group, this can prove a powerful way to negotiate competing values and perspectives in an evaluation (House and Howe, 2003). There has been renewed interest in deliberative processes in evaluation, particularly in the area of evaluating systems change (Apgar et al., 2024). In this context, I understand a system to be the ‘interconnected set of actors, elements, relationships, rules, and resources that jointly produce and sustain a particular outcome’ (USAID, 2024: 3).

It is worth considering here how Schmidt-Abbey et al. (2020) differentiate between two types of systems-based evaluations. The first, which they call ‘systematic-oriented evaluation’, is underpinned by a positivist epistemology and considers the system as a real entity that can be modelled and systematically studied. The second, ‘systemic evaluation’, is underpinned by a constructivist worldview where:

Systems are brought forth, or distinguished by, practitioners interested to engage with a situation and understand or change it . . . The role of a system is to be used as an epistemological rather than ontological device, that is, as a way of knowing about a situation of concern, including an evaluand. (Schmidt-Abbey et al., 2020: 211)

I ascribe to the latter type of evaluation (which is also why I prefer the aforementioned USAID definition of a system, because it defines the system around the outcome that we are trying to change). Of course, this also means that ‘everyone will see the systems they are part of differently and will hold different perspectives on the boundaries of the system and the nature of the relationships within it’ (Burns, 2014: 5). Therefore, the evaluator must navigate the multiple perspectives through which a system may be understood, as well as the judgements that are made about what is in and out of the system of interest (Schmidt-Abbey et al., 2020). This is before we can even come to valuing and judging efforts to change a system! Hence, Gates and Fils-Aime argue that we need to:

Reshape evaluation from rendering discrete assessments of performance to facilitating ongoing evaluative processes and deliberation among those involved and affected about the value of what they’re up to and what should be done next. Systems change is a long game, in which conflicting perspectives and shifting evidence are likely. Evaluative processes should provide ways to continuously codevelop value. (Gates and Fils-Aime, 2022: 7)

The turn towards deliberative processes in evaluating systems change mirrors a similar shift in assessing VFM, where there have been ‘healthy and long overdue debates [forcing] practitioners to make explicit the reasons why we think we offer value and the values we use to assess it’ (Shutt, 2015: 74). There has been a greater appreciation that VFM is a subjective and relative concept, despite traditional methods like CBA that purport to be objective – and that such methods struggle to handle non-linear, intangible and non-quantifiable outcomes (Peterson and Skolits, 2020; Shutt, 2015).

King (2019) pioneered a new approach to VFM, called Value for Investment (VFI), that co-develops what VFM looks like for a particular intervention and uses evaluative reasoning to reach judgements around the extent to which it offers VFM. Peterson (2023) explained that the VFI approach lends itself to constructivism as its philosophical home because it recognises that value is constructed by stakeholders; that there are a plurality of values that will shape constructions; and that values can change over time. Peterson (2023) argued that the VFI approach was, therefore, suited to participatory approaches to understand more fully what stakeholders value and to ensure that the evaluation was oriented towards learning and action.

However, when evaluating the VFM of systems-change efforts, I propose that we need to go beyond participation and move towards collective deliberation in order to reach legitimate judgements about the value of systems-change efforts. Systems in and of themselves, and efforts to change systems, are simply too complex for one person to able to make sense of and judge (USAID, 2024). Rather, the different views and perspectives of stakeholders are essential – but negotiating the competing values of said stakeholders is challenging (Apgar et al., 2024; Burns, 2014). Indeed, Shiell et al. (2008: 1283) argue that ‘more collective, deliberative methods of eliciting social value are needed’ for economic evaluations of interventions in complex systems.

I suggest that deliberative democratic evaluation provides a useful North Star here. Deliberative democratic evaluation emphasises the process by which people discuss, deliberate and reach a shared judgement (House and Howe, 2003).¹ Deliberative democratic approaches tend to involve deeper engagement and a focus on consensus building, with particular attention given to ensuring the inclusion of all relevant stakeholders (Alunni-Menichini et al., 2023; House and Howe, 2003; Rey and Fortin, 2023). The pioneers of deliberative democratic evaluation, House and Howe (2003: 80), noted that ‘no one has formulated better ways of reconciling conflicting perspectives, values and interests than through democratic processes, imperfect though such processes might be’ – and this still rings true some 20 years later.

The three essential principles of deliberative democratic evaluation are inclusion, dialogue and deliberation (House and Howe, 2003). Developing rubrics allows these tenets of inclusion, dialogue and deliberation to be upheld, as Gates et al. (2024) recently demonstrated. A rubric sets out criteria (aspects that are determined to be important or valued by stakeholders) and standards (levels of performance against these criteria) in order to reach transparent, evaluative conclusions (Dickinson and Adams, 2017). Apgar et al. (2024) have argued that rubrics are useful for determining what success looks like in systems-change work where change (and, therefore, evidence) cannot be predefined. When applied to VFM assessments, Gargani and King (2024: 65) articulate that ‘from a democratic perspective, a rubric supports power sharing to co-create context-specific definitions of value for money with stakeholders and to make evaluative judgements systematically and transparently’. Therefore, taking a rubric-based approach to evaluating the VFM of systems-change efforts seems an appropriate way to grapple with the complexity of systems-change work and to negotiate the competing values of stakeholders.

Conceptualising the value for money of systems-change efforts

Having determined that applying a deliberative democratic approach to assessing the VFM of a systems-change initiative was an appropriate method, I faced a challenge. How might one conceptualise an initiative’s contribution to systems change that would enable such an assessment? VFM evaluations usually occur at certain points (i.e. the end of a funding period), where the primary achievement may have been creating potential for systems change. As Burns explains,

Change can be highly discontinuous – nothing appears to be happening and then suddenly radical change is evoked as a tipping point is reached. . . On the surface, problems may seem entrenched and resist change. Under the surface, attitudes may be changing, innovations may be garnering support, and suddenly there is a ‘phase change’ signalling radical change. (Burns, 2014: 6)

If we only focus on evidencing and valuing changes on the surface, we risk misconstruing the true value of an initiative. As Gargani and King (2024: 53) note, ‘the value stakeholders place on [initiatives] . . . depends in part on which impacts evaluators choose to describe and how well they describe them’. Therefore, we need to be able to ‘unpack and articulate the mechanisms that a program can use to create value’ (King, 2021: 6).

In order to address this challenge, I developed a schema to help conceptualise the relationship between systems-change initiatives and creating value.² The schema takes the commonly used ‘Water of Systems Change’ conditions (Kania et al., 2018) and aligns them with the ‘Cycles of Value Creation’ (Wenger et al., 2011).³ The Water of Systems Change (Kania et al., 2018) is a framework that argues that systems change occurs when the conditions holding the problem in place are shifted. There are six conditions, moving from explicit to implicit. While Kania et al. (2018) posit that shifts in the system are more likely to be sustained when there are changes across all three levels, they argue that it is mental models (underlying beliefs and assumptions) that need to be changed in order for there to be transformative change in a system.

The Cycles of Value Creation is a framework developed by Wenger et al. (2011) as a way of considering the types of value created by communities of practice (in a social learning context). They identify five types of value cycles: immediate, potential, applied, realised and reframing value. These two frameworks can be overlaid to create three types of value creation for systems-change efforts (see Figure 1).⁴

Figure 1.

The six conditions of systems change (diagram from Kania et al., 2018) with the associated value creation cycles (Wenger et al., 2011).

I then added the other two types of value creation from Wenger et al. (2011) (immediate and realised value) to create the five types of value creation in systems-change efforts, as shown in Figure 2.⁵

Figure 2.

Proposed schema: types of value creation for systems change.

The types of value created through systems change work are, therefore, as follows:

Inherent value: Systems-change activities

This type considers the inherent value of systems-change activities, which can produce value in and of themselves and is often more immediately evident (Wenger et al., 2011). For example, some systems-change work begins with pilots or prototyping, which may provide some localised impact. Other systems-change work may involve training advocates, which may bring some immediate benefits to participants. Or it may be as simple as understanding more about yourself as you begin to understand your role in the system (Renaisi, 2024).

2. Potential value: Enablers for systems change

This type considers the potential value created by building and strengthening relationships and connections, or by changing power dynamics in the system. As Wenger et al. (2011: 23) explain, ‘not all value produced by [our activities] is immediately realised . . . [they] can produce “capital” whose value lies in its potential to be realised later’.

In systems change, this ‘loading’ of potential can be a long and slow process until a tipping point is reached (Burns, 2014). Creating potential value could also include other enablers for systems change that have been identified in the literature, such as supporting the emergence of a learning culture (Francis-Auton et al., 2024), creating a shared agenda for change and building trust (Renaisi, 2024).

3. Applied value: The system starts to shift

Applied value entails ‘identifying the ways practice has changed in the process of leveraging [potential value]’ (Wenger et al., 2011: 25). In systems-change work, there may be changes in policies, practices and/or resource flows when potential value is applied in practice.

4. Realised value: Systems outcomes

Systems outcomes are the ultimate aim of systems-change efforts – to create a difference for people in the system. While one would expect that changes in policies, practices and/or resource flows will result in systems-change outcomes, this is not guaranteed. If the changes in policies, practices and resource flows do create changes for people in the system, we would consider this as realised value.

5. Transformative value: Systems transformed

Popular systems-change frameworks (such as the Water of Systems Change (Kania et al., 2018) and the Four Keys (Leadbeater and Winhall, 2020)) emphasise the importance of changing ‘implicit’ system conditions – critically, the mental models/underlying philosophies (deeply held beliefs and assumptions) that underpin the purpose of a system. Shifting the underlying philosophy and purpose of the system creates transformative change – which we would consider as transformative value. The fundamental changing of the system then necessitates re-strategising systems-change work, and, therefore, reframing the value of said work.

Why are types of value creation for systems change useful?

The types of value creation can be used at different (or multiple) stages of the VFM assessment, as shown in Table 1.

Table 1.

Application of the value types through a VFI assessment.

Stage of VFI assessment (King, 2023)	Application of the value types
Understand the programme	Use the types of value to articulate a theory of value creation. This could be done before, during or after implementation to understand how the initiative proposes/d to create different types of value.
Develop criteria and standards	Use the value types as the basis for criteria, developing standards for each value type.
Gather and collate evidence against the criteria	Synthesise evidence against the value types (i.e., evidence whether and how the initiative has created immediate value, applied value etc.).
Facilitate a judgement using the rubric	Where the rubric and/or the evidence has been structured using the value types, the judgements will consider the different types of value created through the initiative.

Using the types of value creation allows a ‘theory of value creation’ to be developed – that is, we can articulate the ways in which the initiative has (or will) create sufficient value to justify resource use (i.e. offer VFM) (King, 2021). The theory of value creation developed in this paper is based on changing system conditions. Considering system conditions moves us from attribution to contribution because we can acknowledge that we are not the only ones changing said conditions. It is well accepted in systems-change evaluation that the focus must be on contribution, as attribution is inappropriate and impossible when many interconnected factors interact and contribute to system changes (Patton, 2012). Much, too, has been written on the importance of involving key stakeholders in constructing and articulating plausible contribution claims in order to have a more nuanced understanding of the intervention in context (Bouyousfi and Ouedraogo, 2025; Junge et al., 2020; Patton, 2012). The schema aids a focus on contribution by allowing contributions to changes in system conditions to be valued (rather than only attributable, tangible benefits).

Moving from attribution to contribution can also encourage us to lift our gaze beyond the single programme or initiative that is being evaluated to the other actors creating potential (or other) value. Gates and Fils-Aime argue that it is important to move beyond evaluating a single intervention or initiative to value and evaluate what is changing in the broader system:

Evaluation expands from a conception of using values and evidence to render judgments of the value of some bounded intervention to a process of developing value within ongoing system change efforts. The latter conception still brings evidence and values to bear, but conceives of the nature of evaluating as part of learning and adapting processes. Shifting the object(s) of evaluation from a bounded intervention to system change requires focusing on processes for generating these changes and conditions that sustain these changes. (Gates and Fils-Aime, 2021: 134)

Using the types of value creation allows us to apply King et al.’s (2023) approach to investigating VFM more appropriately (i.e., developing criteria and standards to make a transparent judgement on the VFM of an initiative). This is because the schema provides a way for different stakeholders to conceptualise and co-develop the value (and, therefore, criteria and standards for the VFM) of their systems-change efforts through dialogue and deliberation. Stakeholders’ perceptions of value and the potential value that is being created through systems-change work is likely to shift as the context evolves and systems-change activities adapt. Therefore, we can also expect that the VFM criteria and standards are likely to change as we co-develop and question the value of our systems-change work.

In practice, I also found that the types of value creation helped negotiate the competing values that drive an evaluation. This is what Schwandt and Gates (2021) refer to as ‘valuing schemes’: how the values of the institutions that commission and conduct evaluation can inform the purpose and approach of an evaluation. The inherent value of activities are often the easiest changes to measure and include but offers the least insight into changing systems. Realised value is what funders are often most interested in, but it is often unrealistic to demonstrate realised value within a funding period, or to attribute it to the intervention of interest. Therefore, we need to be able to articulate and include the potential value created through systems-change efforts. A theory of value creation can ‘support evaluation of processes and interim value that are assumed, with solid rationale, to contribute to long-term value . . . [A] judgement can still be made about how well a program is working toward meeting its value proposition’ (King, 2021: 7).

In summary, the schema encourages us to consider the following when evaluating the VFM of systems-change work:

Codeveloping and framing value (and subsequently developing criteria and standards) as it relates to changing system conditions;

Shifting from attribution to contribution;

Thinking beyond the evaluand in question to the wider system;

Expecting that value will change as systems-change work progresses;

Negotiating the competing values driving the evaluation.

How did this work in practice? WorkWell case study

I now turn to my experience undertaking a VFM assessment of WorkWell as a form of ‘second-order research’ (where the evaluation practitioner reflects on the ‘situations they themselves are part of rather than being distanced observers’ (Schmidt-Abbey et al., 2020: 208)).

WorkWell was a 6-year, $50 million programme delivered by WorkSafe (Victoria’s workplace health and safety regulator, and workplace injury insurer). With around 3.7 million people employed in Victoria at the start of the programme (Australian Bureau of Statistics, 2017), WorkWell was an ambitious initiative to reduce the emerging burden of mental injury claims. Workers with a mental injury experience poorer recovery and return to work outcomes than workers with a physical injury, requiring support for longer periods. The programme took a multi-pronged approach, providing employers with access to resources and funding opportunities aimed at building the capacity, confidence and commitment of workplace leaders to drive organisation-wide culture change and prevent mental injury among Victorian workers.

Clear Horizon was commissioned to support a final evaluation of WorkWell over five months in 2023, including undertaking a VFM assessment. The purpose of the VFM assessment was to answer the evaluation question ‘to what extent is WorkWell likely to achieve outcomes and impacts commensurate with expectations for the level of resources invested?’, as well as to identify learnings and improvements for the next phase of the programme. Primary stakeholders were WorkWell itself, the Victorian Government (as the funder), and the funded projects.

The raison d’être for WorkWell was to reduce mental injury claims (which would have had a clear monetary benefit), yet these continued to increase through WorkWell’s initial funding period. However, the programme unexpectedly contributed to a paradigm shift in workplace mental health from a focus on building the personal resilience of individual workers to employers taking a primary prevention approach to address psychosocial work factors that impact mental health at their source (Clear Horizon, 2023b). The VFM assessment needed to appropriately account for a complex, systems-change effort like WorkWell, where the outcomes were emergent and could not be fully attributed to WorkWell in particular.

We used King et al.’s (2023) VFI approach outlined earlier, broadly following four steps that I will discuss in turn, noting that we used the ‘types of value creation’ in steps three and four:

Understanding the programme;

Developing criteria and standards as a rubric;

Collating evidence against the criteria;

Facilitating a judgement using the rubric.

Understanding the programme: What did we think WorkWell would do?

Before a VFM assessment can be undertaken, stakeholders must first deliberate and reach a shared understanding of (a) the system that the intervention is trying to influence, including creating boundaries for the sake of the evaluation; and (b) the types of value that the initiative will create or has created. WorkWell developed a Theory of Change at the beginning of the programme with key stakeholders to articulate the changes they thought they would contribute to.⁶ However, WorkWell had emergent and unexpected outcomes that were not necessarily reflected in the Theory of Change – our understanding of the programme was actually further developed and refined through undertaking the final evaluation and VFM assessment.

Developing the criteria and standards: What would ‘good’ look like?

As the final evaluation began, Clear Horizon facilitated a workshop with key stakeholders (the WorkWell programme team, senior WorkSafe staff and a funded project) to collaboratively develop definitions of VFM using criteria (aspects of VFM that were important and meaningful to WorkWell) and standards (defined levels of performance against the criteria).

The standards were structured around expectations (below, meets or exceeds expectations relative to the investment). WorkWell’s contribution to systems change (criterion 3.4 in supplementary material 1) was only one part of a wider VFM assessment, which also considered aspects such as economy (good governance of expenditure) and efficiency (including WorkWell’s ability to learn and adapt, and collaborate with others – prerequisites for systems-change work).

The standards for criterion 3.4 were defined by WorkWell’s ability to evidence contribution to systems change. Stakeholders wanted to reflect the idea that in a complex system, it is impossible to distinguish all of the effects of a particular contribution, and we can only assess what we can identify and evidence (Gargani and King, 2024). That is, the further the assessment moves into complexity, the less we can be certain that we have been able to identify and evidence all the (actual and potential) contributions of a particular intervention – and the more we are reliant on the level of evidence as an imperfect proxy for the level of performance. While combining the degree of contribution and strength of evidence worked pragmatically in this case, others may prefer to use two separate rubrics.

Collating evidence: What did WorkWell actually do?

As part of the final programme evaluation, Clear Horizon synthesised evidence of WorkWell’s contribution to systems change. Clear Horizon facilitated a sensemaking workshop with WorkWell staff to interrogate and articulate the types of value that had been created and how. This was an important step to deliberate and reach a shared understanding of what WorkWell had contributed to, as it is not easy to untangle the contribution of the initiative at hand from other actors in the system. The types of value created through WorkWell’s systems-change activities are summarised below.

Summary of the types of value created through WorkWell (Clear Horizon, 2023b)

Inherent value: WorkWell created a new network of leaders to champion their primary prevention approach. They engaged 30,000 leaders, creating some immediate value in terms of connection and community for these leaders. Several funded projects also demonstrated quantifiable improvements to worker mental well-being and higher productivity.

Potential value: WorkWell catalysed partnerships across industries, including bringing together stakeholders who had not traditionally worked together on prevention. Partnerships were described as highly valuable for funding recipients, project partners, and participating workplaces. Analysis showed that partnerships tended to move from lower levels of maturity (networking and coordinating) to the higher levels of cooperating and collaborating. Changing relationships and connections is a condition of systems change identified by Kania et al. (2018), creating potential for these actors to align and coordinate their work. WorkWell also shifted underlying beliefs and assumptions about how regulators should work through its deliberate and collaborative approach to partnering in the programme, creating the potential for improved ways of working together to solve problems in the future.

Applied value: Employer practices and policies changed amongst participating workplaces, with some industry-wide changes to policy (such as for frontline workers). The Victorian Government also decided to continue funding for WorkWell as a business-as-usual function of WorkSafe, allowing for continued work in the primary prevention of mental injury in the workplace.

Transformative value: WorkWell contributed to a shift in mental models towards primary prevention of mental injury in the workplace – from a focus on building the personal resilience of individual workers to employers taking an intentionally preventative approach to address work-related factors that impact mental health at their source. It is now widely accepted in the Victorian context that the prevention of mental injuries requires proactive identification, assessment and control of the work-related factors that impact mental health (Victorian Government, 2024).

Panel assessment: What was the value for money of WorkWell?

WorkWell appointed a panel comprising members of WorkWell’s academic Expert Reference Group, senior WorkSafe manager, WorkWell’s director, and representatives from government, a union and a funded project. The panel membership was designed to include a range of different stakeholders with contrasting perspectives, in line with the principles of deliberative democratic evaluation. The panel reviewed the evidence ahead of a facilitated workshop to deliberate and decide the final ratings. The workshop proved an important space for panel members to dialogically deliberate and reach a collective decision on the VFM of WorkWell. The workshop also allowed learnings to be elicited by the group that could be applied to the next phase of WorkWell, and, indeed, to any other government programmes considering a similar approach to prevention.

The panel judged that WorkWell had exceeded expectations for contributing to wider systems-level changes (three of the eight members rated ‘meets expectations’ and five ‘exceeds expectations’) (Clear and Horizon, 2023a). The panel determined that WorkWell’s contribution to shifting mental models around a primary prevention approach had been a significant contribution towards systems change. The panel also agreed that WorkWell had created potential value for future systems change by shifting the underlying beliefs and assumptions about how regulators should work and partner with industries, paving the way for new and improved relationships in the future. The panel also noted that the endorsement of WorkWell to receive continued funding (applied value) was a significant contribution to systems-level change as it will ‘keep the momentum going’ (Clear and Horizon, 2023a: 8).

Reflections on inclusive rigour

The WorkWell case study was an early foray into applying VFM to systems-change initiatives and affords the opportunity to critically reflect on the rigorousness of the approach. Apgar et al. (2024) argue that evaluating complex systems requires new conceptions of rigour. They present three interconnected domains for inclusive rigour: effective methodological bricolage, meaningful participation and inclusivity, and utilisation and impact.

Achieving effective methodological bricolage

Bricolage is the mixing of different parts of relevant evaluation methods and tools to enable reasoning and credibility – and is particularly appropriate for contending with the challenge of evaluating systems change (Aston and Apgar, 2022). The approach presented in this paper (which combines artefacts from the systems-change and social learning literatures, theory-based evaluation, rubrics and deliberative democratic evaluation) enables evaluative reasoning through deliberation and dialogue. However, the approach relies on credible evidence for justifiable contribution claims that stakeholders can then value and judge. While specific contribution methods are outside the scope of this paper (and certainly much has been written in the contribution analysis literature), if the evidence for the contribution claims is not of sufficient quality, the risk arises that the VFM assessment itself will not be sufficiently credible or could even be misleading.

Importantly, Patton (2012) notes that when evaluating contribution in complex settings, rigorous thinking is more important than rigorous methods. While the quality of the evidence is, of course, important, the more crucial aspect is engaging multiple perspectives in systems thinking, to which I will now turn.

Ensuring meaningful inclusion: The role of the evaluator in supporting systems thinking and participation

The literature is clear that honouring and including diverse perspectives in systems-change evaluation is critical (USAID, 2024) – and indeed, inclusion is a key tenet of deliberative democratic evaluation (House and Howe, 2003) and of Apgar et al.’s (2024) inclusive rigour framework. However, this does raise the challenge for evaluators in balancing the technocratic nature of the assessment with a participatory process. We observed in this case study that some members were more prepared to ‘think’ at a systems level (where systems thinking is both a mindset and a set of skills to engage with systems (USAID, 2024)).

Some of the principles of emergent learning (‘a set of principles and practices to help people across a system think, learn and adapt together’; Darling and Eenigenburg, 2023: 1) may be useful to explore further here. One relevant principle is holding expertise in equal measure which ‘refers to seeking and using multiple sources of knowledge and perspectives to deepen understanding of a topic’ (Bell and Anderson, 2023: 101). The process should not only involve those with technical expertise but those who have lived experience of inequity in the system. It is easy to see how the position and role of the professional evaluator becomes paramount in facilitating this. Indeed, when undertaking deliberative democratic evaluation, the evaluator must structure the process to ensure effective and inclusive dialogue and deliberation (Alunni-Menichini et al., 2023), positioning themselves as ‘reflexive facilitators rather than objective judgers’ (Apgar et al., 2024: 101) – reflecting Peterson’s (2023) argument that VFI is a constructivist approach where the evaluator co-constructs knowledge and judgements.

Importantly, the ability to hold expertise in equal measure is ‘dependent upon [the facilitator’s] awareness of power, privilege and positionality’ which requires ‘humility, curiosity and respect for the various voices and types of knowledge brought to the table to inform . . . decision-making processes’ (Bell and Anderson, 2023: 102, 107). Interestingly, USAID (2024: 7) also notes that an important systems practice is to inquire with humility and curiosity:

We can never fully understand a system; systems are always changing, and our perspectives will always be biased and limited . . . We should be humble about our own knowledge and expertise, and genuinely curious about and respectful of the knowledge and expertise of others within the system . . . stepping away from the role of ‘expert’ in favour of other roles such as facilitator, convenor, advisor or collaborator.

The other relevant principle is making thinking visible, which ‘clarifies and communicates how we uniquely understand an issue at hand’ (Ochtera and Gaspari, 2023: 23). This principle helps to ‘create space that encourages participation, acknowledges a diversity of experiences and helps make it acceptable to hold more than one truth’ (Ochtera and Gaspari, 2023: 24). This space is partly achieved through dialogue, as espoused in deliberative democratic evaluation (House and Howe, 2003). It is also achieved through making evaluative criteria explicit, such as through a rubric (Gates et al., 2024).

Supporting utilisation: The importance of expanding the types of value created

Apgar et al. (2024) identify the domain of utilisation as the place where we strive to navigate competing stakeholder needs for evidence and learning. The oft-discussed struggle is between the valuing schemes of the commissioner versus those ‘on the ground’, particularly as commissioners hold different (and usually higher) forms of power to influence the evaluation’s focus, methods and what evidence is seen as credible. Indeed, the push for VFM has largely been driven by governments anxious to demonstrate accountability to taxpayers, and funder-driven VFM can reflect ‘market-based values that do not resonate with the social value systems’ of those engaged in systems-change initiatives (Shutt, 2015: 68).

As previously noted, the proposed framework for assessing the VFM of systems change was helpful in negotiating competing value schemes. In this case, we had limited evidence to suggest changes for realised value – what funders have traditionally been most interested in. A more traditional VFM assessment would have homed in on realised value (in this case, a decrease in mental injury claims) – of which we were not (yet) able to demonstrate any positive impact. Even if statewide claims had decreased, it would have been a long bow to attribute this to WorkWell when other factors would have, of course, influenced this trend.

However, we were able to articulate and, therefore, judge other types of value creation, particularly the potential value of changing the relationship between WorkWell as a regulator and Victorian industries, and the transformative value of shifting mental models around preventing mental injury in the workplace. These changes were not conceived of at the start of the WorkWell programme; WorkWell underwent a shift in strategy two years into the funding period to focus on primary prevention (Clear and Horizon, 2023b). The impact of WorkWell’s efforts was also heavily influenced by the COVID-19 pandemic, where workplace mental health during lockdowns became more visible on the public agenda. WorkWell’s unexpected outcomes demonstrates the importance of being able to identify and include emergent contributions and outcomes – and expand what we value to include creating the potential for future systems change.

However, there is certainly the risk that even with the best intentions and political nous, evaluators may be unable to successfully navigate the competing demands driving the evaluation (Aston and Apgar, 2022), and the VFM assessment may be largely based around commissioner values.

Deliberative democratic approaches require pragmatism

It would be understandable to have read thus far and think ‘this is all well and good . . . if you have the time and budget!’ And I agree! Inclusive and democratic processes tend to be time intensive. In the case study of WorkWell, we did not engage all the funded projects nor the intended beneficiaries of these projects. Doing so may have enhanced our process and findings but was not possible under the time and budget constraints. House and Howe (2003: 90) recognise this:

any study is a compromise, of course. We can never accomplish all that we want. We can never fully implement idealised principles such as those we have expounded. We need to be grounded in the real world even as we maintain democratic aspirations.

Similarly, Apgar et al. (2024: 419) concur that ‘perhaps the core competency that underpins quality . . . is being able to balance principles with pragmatism’.

By attempting to apply – at least in part – the three principles of deliberative democratic evaluation, evaluators ‘can democratise any evaluation project’ (Rey and Fortin, 2023: 136) – including VFM. We aimed to include a diversity of perspectives and to facilitate a collective deliberation that would reach more legitimate and useful judgements than if we had not tried.

Conclusion

The demand to judge the VFM of systems-change efforts is only going to get louder as public spending becomes increasingly constrained. This pressure poses challenges for the evaluation profession as it is notoriously difficult to evaluate systems-change interventions, let alone apply a VFM assessment, and work in this area is nascent. Through exploring the case study of WorkWell, this paper has made two propositions for evaluating the VFM of systems-change efforts.

First, we need to move beyond participation towards deliberative democratic evaluation in order to make more legitimate and meaningful judgements. This is because the messiness and complexity of systems-change work is best understood – and therefore valued – by including the views of different stakeholders in an inclusive and transparent manner, with clear evaluative reasoning achieved through dialogue and deliberation.

Second, in order to support a deliberative democratic VFM assessment – as well as do more justice to the intervention at hand – we need to be able to articulate, and, therefore, include the different types of value that are created through systems-change work. While inherent value may be the most straightforward to evidence, and realised value is what funders are often most interested in, at the time of the evaluation, most initiatives are likely to have predominately created potential value. In some cases, systems-change efforts may have contributed to transformative value (shifting the underlying beliefs, assumptions and purpose of a system) – perhaps the hardest to evidence but the most important for lasting systems change.

A rubric-based approach displays promise, particularly as it allows the three tenets of deliberative democratic evaluation (inclusion, dialogue and deliberation) to be upheld when negotiating competing values and grappling with the complexity of systems change. A rubric-based approach also allows stakeholders to consider all types of value, including potential value. However, work in this area is nascent, and there is still much to wrestle with – not least, how to conduct more democratic evaluation in an inherently technocratic area such as VFM and systems change. Nevertheless, this rubric-based approach grounded in democratic deliberation offers a North Star bearing for navigating waters that are still to be charted.

Supplemental Material

sj-docx-1-evi-10.1177_13563890251386814 – Supplemental material for Democratic deliberation as a North Star: Showcasing a framework to assess the value for money of systems-change efforts

Supplemental material, sj-docx-1-evi-10.1177_13563890251386814 for Democratic deliberation as a North Star: Showcasing a framework to assess the value for money of systems-change efforts by Heidi Peterson in Evaluation

Footnotes

Acknowledgements

The author would like to thank Nicholas Crooks for being willing to test, embrace and further develop this methodology at WorkWell, and for his feedback on the paper. The author would also like to thank Professor James Copestake, Dr Julian King, Dr Jess Dart, Kate McKegg and Gillian Asquith for their valuable feedback on the paper.

Author’s Note

The work reported in this article was conducted while I was an employee of Clear Horizon Consulting.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author received no financial support for the research, authorship and/or publication of this article.

Ethical approval

Ethical approval to report this case study was obtained from the University of Bath Data and Digital Science Research Ethics Committee (reference: 2494-9249). Written informed consent was obtained from a WorkWell representative for the case study to be used in this article.

ORCID iD

Heidi Peterson

Supplemental material

Supplemental material for this article is available online.

Notes

Heidi Peterson is a Lead Principal Consultant at Clear Horizon, currently undertaking a professional doctorate in participatory approaches to Value for Money.

References

Abelson

Forest

P-G

Eyles

, et al. (2003) Deliberations about deliberative methods: Issues in the design and evaluation of public participation processes. Social Science & Medicine 57(2): 239–51.

Alunni-Menichini

Bertrand

Brousselle

(2023) Combining the world café and the deliberative democratic evaluation: A win-win strategy. Journal of Participatory Research Methods 4(1): 13–23.

Apgar

Aston

Snijder

, et al. (2024) Raising the bar: Improving how to assess evidence quality in evaluating systems-change efforts. The Foundation Review 16(2): 11.

Apgar

Bradburn

Rohrbach

, et al. (2024) Rethinking rigour to embrace complexity in peacebuilding evaluation. Evaluation 30(3): 408–33.

Aston

Apgar

(2022) The art and craft of bricolage in evaluation. Institute of Development Studies, 14 October. Available at: https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17709 (accessed 13 July 2025).

Australian Bureau of Statistics (2017) Labour Force, Australia (September 2017 (6202.0). Canberra, ACT, Australia: Australia Bureau of Statistics. Available at: https://www.ausstats.abs.gov.au/ausstats/meisubs.nsf/0/F69C835E876B088ECA2581BD000FCEB3/$File/62020_sep_2017.pdf

Bell

Anderson

(2023) Holding expertise in equal measure. In: Darling

Eenigenburg

(eds) Guide to the Principles of Emergent Learning. The Emergent Learning Community Project, pp. 101–14.

Bouyousfi

Ouedraogo

(2025) Streamlining complex intervention evaluation through participatory systems mapping and contribution analysis: A comprehensive framework for actionable complexity evaluation. American Journal of Evaluation 46: 6–21.

Burns

(2014) Systemic action research: Changing system dynamics to support sustainable change. Action Research 12(1): 3–18.

10.

Clear Horizon (2023a) Value for money assessment report. Available at: https://content-v2.api.worksafe.vic.gov.au/sites/default/files/2023-09/WW-value-for-money-assessment-report-2023.pdf (accessed 22 February 2025).

11.

Clear Horizon (2023b) WorkWell’s journey and achievements: A summary. Available at: https://content-v2.api.worksafe.vic.gov.au/sites/default/files/2023-09/WW-summative-report-2023-2023-09.pdf (accessed 22 February 2025).

12.

Darling

Eenigenburg

(eds) (2023) Guide to the principles of emergent learning. The Emergent Learning Community Project. Available at: https://emergentlearning.org/guide-to-the-principles-of-emergent-learning/

13.

Dickinson

Adams

(2017) Values in evaluation: The use of rubrics. Evaluation and Program Planning 65: 113–6.

14.

Francis-Auton

Long

Sarkies

, et al. (2024) Four system enablers of large-system transformation in health care: A mixed methods realist evaluation. The Milbank Quarterly 102(1): 183–11.

15.

Gargani

King

(2024) Principles and methods to advance value for money. Evaluation 30(1): 50–68.

16.

Gates

Fils-Aime

(2022) Five Tips for Evaluation of System Change. Washington, DC: American Evaluation Association. Available at: https://aea365.org/blog/systems-and-complexity-informed-evaluation-week-five-tips-for-evaluation-of-system-change-by-emily-gates-and-francisca-fils-aime/ (accessed 18 April 2025).

17.

Gates

Fils-Aime

(2021) System change evaluation: Insights from The Rippel Foundation and its ReThink Health initiative. New Directions for Evaluation 2021(170): 125–38.

18.

Gates

Teasdale

Shim

, et al. (2024) Whose and what values? Advancing and illustrating explicit specification of evaluative criteria in education. Studies in Educational Evaluation 81: 101335.

19.

Gear

Koziol-McLain

Eppel

, et al. (2025) Capturing the emergence of change in complex systems: The ‘atawhai’ study in Aotearoa, New Zealand. Evaluation 31(2): 163–84.

20.

Gullickson

Hannum

(2019) Making values explicit in evaluation practice. Evaluation Journal of Australasia 19(4): 162–78.

21.

Gunderson

Holling

(2002) Panarchy: Understanding Transformations in Human and Natural Systems. Washington DC: Island Press.

22.

House

Howe

(2003) Deliberative democratic evaluation. In: Kellaghan

Stufflebeam

(eds) International Handbook of Educational Evaluation. Dordrecht: Springer, pp. 79–100.

23.

Howley

Campbell

Cowley

, et al. (2022) Pathways and structures: Evaluating systems changes in an NSF includes alliance. American Journal of Evaluation 43(4): 632–46.

24.

Junge

Cullen

Iacopini

(2020) Using contribution analysis to evaluate large-scale, transformation change processes. Evaluation 26(2): 227–45.

25.

Kania

Kramer

Senge

(2018) The water of systems change. FSG. Available at: https://www.fsg.org/wp-content/uploads/2021/08/The-Water-of-Systems-Change_rc.pdf

26.

King

(2017) Using economic methods evaluatively. American Journal of Evaluation 38(1): 101–13.

27.

King

(2019) Evaluation and Value for Money: Development of an Approach Using Explicit Evaluative Reasoning. Melbourne, VIC, Australia: The University of Melbourne. Available at: https://rest.mars-prod.its.unimelb.edu.au/server/api/core/bitstreams/e519211c-8758-5e2a-a470-a639a35bec1a/content (accessed 18 April 2025).

28.

King

(2021) Expanding theory-based evaluation: Incorporating value creation in a theory of change. Evaluation and Program Planning 89: 101963.

29.

King

McKegg

Oakden

, et al. (2013) Rubrics: A method for surfacing Values and improving the credibility of evaluation. Journal of Multidisciplinary Evaluation 9(21): 11–20.

30.

King

Wate

Namukasa

, et al. (2023) Assessing Value for Money: The Oxford Policy Management Approach, 2nd edn. Oxford: Oxford Policy Management. Available at: https://www.julianking.co.nz/wp-content/uploads/2023/12/opm-vfm-approach-2.pdf (accessed 18 April 2025).

31.

Leadbeater

Winhall

(2020) Building Better Systems: A Green Paper on System Innovation. Copenhagen: The Rockwool Foundation. Available at: https://static1.squarespace.com/static/632b07749e5eec1fde3510bd/t/63610361fe3ff372e1d9b33e/1667302288381/Building%2BBetter%2BSystems%2Bby%2Bthe%2BROCKWOOL%2BFoundation.pdf (accessed 23 February 2025).

32.

Ochtera

Gaspari

Brittney

(2023) Making thinking visible. In: Darling

Eenigenburg

(eds) Guide to the Principles of Emergent Learning. The Emergent Learning Community Project, pp. 23–35.

33.

Patton

(2012) A utilization-focused approach to contribution analysis. Evaluation 18(3): 364–77.

34.

Peterson

Skolits

(2020) Value for money: A utilization-focused approach to extending the foundation and contribution of economic evaluation. Evaluation and Program Planning 80: 101799.

35.

Peterson

(2023) Cost–benefit analysis (CBA) or the highway? An alternative road to investigating the value for money of international development research. The European Journal of Development Research 35(2): 260.

36.

Renaisi (2024) Our Guide to Systems Change. London: Renaisi. Available at: https://renaisi.com/wp-content/uploads/2025/01/Renaisi_Guide-to-Systems-Change_final.pdf (accessed 17 April 2025).

37.

Rey

Fortin

(2023) Participatory evaluation, deliberation and democracy. In: Varone

Jacob

Bundi

(eds) Handbook of Public Policy Evaluation. Political Science and Public Policy 2023. Cheltenham: Edward Elgar Publishing Limited, pp. 132–53. Available at: https://www.elgaronline.com/edcollchap/book/9781800884892/book-part-9781800884892-17.xml (accessed 22 February 2025).

38.

Schmidt-Abbey

Reynolds

Ison

(2020) Towards systemic evaluation in turbulent times: Second-order practice shift. Evaluation 26(2): 205–26.

39.

Schwandt

Gates

(2021) Evaluating and Valuing in Social Research. New York: The Guilford Press.

40.

Shiell

Hawe

Gold

(2008) Complex interventions or complex systems? Implications for health economic evaluation. British Medical Journal 336(7656): 1281–3.

41.

Shutt

(2015) The politics and practice of value for money. In: Eyben

Guijt

Roche

, et al. (eds) The Politics of Evidence and Results in International Development: Playing the Game to Change the Rules? Rugby: Practical Action, pp. 57–78.

42.

USAID (2024) Local Systems Position Paper. Washington, DC: USAID. Available at: http://alanhudson.info/wp-content/uploads/2025/02/Local-Systems-Position-Paper.pdf

43.

Victorian Government (2024) The Victorian mentally healthy workplaces framework. Victorian Government. Available at: https://www.vic.gov.au/sites/default/files/2023-12/Mentally-Healthy-Workplace-Framework_0.pdf (accessed 17 April 2025).

44.

Wenger

Trayner

De Laat

(2011) Promoting and Assessing Value Creation in Communities and Networks: A Conceptual Framework. Heerlen: Open Universiteit, Ruud de Moor Centrum. Available at: https://research.manchester.ac.uk/en/publications/promoting-and-assessing-value-creation-in-communities-and-network (accessed 22 February 2025).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB