Abstract
This article focuses on the evaluation of the Children’s Community programme, addressing the question: how can evaluators effectively address complex place-based system change initiatives, particularly those working across differing contexts? After considering alternatives, including what we describe as ‘first-generation’ linear, logic model-based approaches and second-generation Theory of Change approaches, we discuss the development of ‘third-generation’ systems-change evaluations and articulate a novel comparative maturity matrix approach, developed for the evaluation of Children’s Communities with potential wider applicability for evaluation of complex systems change interventions. The article discusses the evaluation approach in depth and concludes with reflection on what learning points may be derived for use in subsequent evaluations of place-based and complex systems change initiatives.
Introduction
This article discusses a programme evaluation carried out between 2017 and 2020 in partnership with Save the Children UK and three Children’s Communities in Wallsend, Pembury and Smallshaw Hurst. It addresses the question: how can evaluators effectively address complex place-based system change initiatives, particularly those working across differing contexts? The article begins with an outline of the Children’s Communities programme before briefly reviewing approaches to evaluating local system change initiatives, drawing on Best and Holmes’ (2010) three generations framing. In so doing, the article considers Theory of Change (ToC), introduced to the evaluation community by the Aspen Institute in the early 1990s as a method of evaluating complex change programmes (Connell and Kubisch, 1998). ToC approaches have been widely used in the field, yet in practice they appear to have limitations in relation to dealing with context, causal theory and their use to guide implementation (Breuer et al., 2015). Furthermore, they are limited in supporting evaluation of programmes with multiple sites that aim to make comparisons between sites, and in helping evaluators and programme leaders to understand how system change over time can be tracked to assess progress towards programme outcomes.
In the body of the article, we discuss an approach taken that was co-produced with the evaluation funder (Save the Children UK) and the Children’s Community teams in the three areas to address these issues. The approach extends and supplements an initial ToC model by introducing a comparative system approach, maturity modelling, which is suitable for multi-site local system change initiatives. We go on to discuss the implementation of the maturity model through operationalising an analytical framework, which articulated progress against dimensions of systems change in the areas studied.
The article concludes with a reflection on benefits and challenges to the approach and a discussion of learning points, which may be derived for use in subsequent evaluations of complex systems interventions which are looking across multiple sites and contexts.
The Children’s Communities programme
The Children’s Communities programme is a local system change initiative, working in and with three differing communities in England. There are multiple definitions of systems change. We draw on the definition of Abercrombie et al. (2015), which defines systems change as ‘intentional process designed to alter the status quo by shifting the function or structure of an identified system with purposeful interventions’. The charity Save the Children UK launched the Children’s Community model in three areas in England: Wallsend in North Tyneside, Collyhurst in Greater Manchester and Pembury in Hackney. Following developmental work (Dyson et al., 2013), Collyhurst left the Save the Children programme and Smallshaw Hurst in Tameside became a Children’s Community in 2017.
Children’s Communities are loosely based on the Harlem Children’s Zone in the United States. The Children’s Community model adopted by Save the Children UK reflects the importance of place (in both its physical and social manifestations) as an influential context in children and young people’s lives. The approach recognises the need to respond to local issues and contexts and to harness local assets and capabilities to build solutions (Dyson et al., 2013).
Although the three Children’s Communities share a common set of principles there are key differences in terms of their size, social, economic and institutional landscapes, and the origin of the Children’s Community partnership. These differences are important because they shape the individual Children’s Community Theories of Change (discussed below), and present challenges for evaluators aiming to articulate change processes across different spatial, cultural and temporal contexts. Some features of the Children’s Communities are outlined in Table 1. 1
Children’s Communities: key features.
Percentage of population aged <25 years. 4
Percentage of population aged 16–24 years with a Black or Minority ethnic background.
The Children’s Community as a local systems change approach
A Children’s Community is conceptualised as a local system. The aim of the Children’s Community is to catalyse change in the local system to improve services, build community capacity and strengthen collaborative working. The ultimate objective of the Children’s Community is to improve outcomes for children and young people living in the area, over an anticipated lifespan of 10 years or longer.
Local system change approaches designed to meet durable challenges with long-term sets of interventions, focusing on sequenced change processes, have a long history. Some of the earliest were the comprehensive community initiatives in the United States. Kubisch et al. (2001: 83) identify key features of the comprehensive community initiatives, which are common to local systems change approaches and are outlined in Box 1.
Features of comprehensive community initiatives.
Evaluating local system change approaches
In this article, we frame the development of approaches to evaluate local system change by drawing on Best and Holmes’ (2010) perspective, which identified three generations of approaches to understanding system change in the health sector. While the example presented here is a simplification, the framing has wider applicability to other fields and is helpful for our purposes in articulating a generalised conceptualisation of three ‘phases’ of evaluation practice. The first generation is a linear model, suggesting a focus on understanding the impacts of system-based activity as a largely uni-directional process from providing inputs to outcomes. In the evaluation field, this encompasses approaches drawing on relatively simple logic models, and utilising methods such as quasi-experimental designs. Second-generation approaches move beyond this to focus on understanding relationships and networks. We ally this to ToC approaches, which allow for more iterative and circuitous processes of change – but can underplay the complexity of local systems. Third-generation approaches actively engage with systems thinking to understand change. In this article, we discuss an approach which draws on both second- and third-generation models, by linking a ToC approach to the development of a systems informed maturity model.
First-generation approaches: Linear models
Place and area-based initiatives have been described as ‘experiments’ in determining what will happen in specific places when investment (in people, resources and infrastructure) takes place over a defined period of time (Lawless, 2012). ‘First-generation’ evaluations of area-based initiatives can be characterised as those that have assumed linear processes of change in the relationships between programme inputs, outputs and outcomes. They have traditionally fallen into two broad groups: those that have focused primarily on audit and ‘process outcomes’ (Lawless, 2007; Thomson, 2008); and those that have sought to use methodologies such as randomised controlled trials and quasi-experimental designs to ‘isolate’ the impacts of area-based initiatives through longitudinal assessment of change in ‘treatment’ areas compared with similar areas where investment has not been made. The most comprehensive UK example of the latter was the evaluation of ‘New Deal for Communities’ (Batty et al., 2010), which adopted a longitudinal household survey methodology, combined with case study research and secondary data analysis, to assess change in 39 New Deal for Communities areas against that in statistically comparable areas over a 10-year period.
The New Deal for Communities evaluation represented a ground-breaking step forward in area-based initiative (ABI) evaluation and was consistent with government guidance at the time (HM Treasury, 2003; Office of the Deputy Prime Minister, 2004). There are nevertheless limitations in this approach, particularly in relation to the degree to which such evaluations can isolate the drivers of change in social environments where a range of complex factors are at play. New Deal for Communities areas (or indeed other places) cannot be treated as ‘policy-on’ environments, which can be compared with ‘policy-off’ contexts (Beatty et al., 2015). Identified outcome change in these areas will reflect not only the impacts of investment, but also a range of other contextual factors including previous patterns of investment (and disinvestment), local policy and organisational landscapes, cultural dimensions, population characteristics and dynamics and so on. Crucially, ‘closed’ programme evaluation designs also struggle to account for the importance of people and their relationships as both drivers of, and barriers to, outcome change and as strategic actors in the harnessing of resources for collective effort (Sullivan et al., 2006). Rhodes et al. (2005), reflecting on the evaluation of the Single Regeneration Budget (a programme of area-based interventions implemented in England in the 1990s), highlighted that evidence on the ‘success’ of area-based initiatives generated through first-generation evaluations remained elusive. In part because they lacked sufficient attention to the theories of change underlying programme interventions, resulting in an associated dearth of evaluation theory to frame and operationalise evaluations, which assessed measurable outcomes, and the contexts and processes through which these had been achieved (see Pawson and Tilley, 1997).
Second-generation approaches: ToC
Identifying these problems, theory-based approaches emerged as an alternative. Drawing on the work of Weiss (1997) and Chen’s (1990) parallel theory-driven evaluation, Connell et al. (1995) and Connell and Kubisch (1998) describe how ToC developed in parallel with the development of comprehensive community initiatives, a form of area-based initiative developed in the United States to ‘both meet the need to estimate these initiatives’ effects on interim and longer-term outcomes and the need for information on how the interventions produce those outcomes’ (Connell and Kubisch, 1998: 1).
Weiss and Connell (1995) who first introduced the term (at least in its uncapitalised form) describe ToC as a theory of how and why an initiative works. While Connell and Kubisch (1998: 1) define the approach as one to uncover via a ‘systematic and cumulative study of the links between activities, outcomes and contexts of the initiative’, they identify three benefits of a ToC approach: to sharpen planning and implementation, to facilitate data collection and the evaluation process and to support claims of causal attribution.
ToC approaches often use visual representations and the same kind of terminology, and are sometimes even described, as programme logic models. However, we, along with others, see them as having different emphases (Funnell and Rogers, 2011); discussed further below. Drawing on Connell and Kubisch (1998), this approach aims to follow the process as laid out in Box 2.
Theory of Change approach.
Adapted from Connell and Kubisch (1998).
The ToC approach focuses on change processes. Typically, they begin with a focus on end point outcomes and then consider the current position. It is then that the activities and change programme(s) that might be used to achieve these changes come into play: these should be considered in relation to their (initially theoretical, subsequently realised) ability to help meet these aims. The intermediate outcomes that would be expected to occur if the programme was on track are then considered in the context of the initiative. If the activities do not meet the aims set, then they need to be modified or abandoned and replaced, to enable the medium- and longer-term goals of the initiative to be met: it is the goals that take precedence.
This differs from other theory-based evaluation perspectives. For example, Breuer et al. (2015: 13) suggest logic models are ‘conceptually similar to ToC but are usually presented in a linear form with boxes for inputs, activities, outputs and outcomes with little explanation of the causal pathways linking them’. Logic models can be designed to deal with this and other shortcomings (Coldwell and Maxwell, 2018). We distinguish the two differently, by describing the use of ToCs as an example of an evaluation approach suitable to an open, system-focused initiative. This is because the ToC is focused on a system or organisational change process (such as a comprehensive community initiative). In contrast, a logic-model based approach is more suitable for a closed, programme-focused initiative. The (programme) Logic Model approach aims to lay out how a particular activity or initiative is theorised to work to achieve a set of goals or aims. The focus here is on understanding how, whether and why the activity works to achieve its theorised aims. Here, the evaluator is interested in the efficacy of a specific programme, such as a particular approach to improving children’s reading. The key difference of the focus is on dealing with the complexity of open system-focused initiatives, a point we return to in the next sub-section.
In the UK context, Mason and Barnes (2007) note ToC approaches are often developed after a programme has begun. This brings to the fore one key element of the ToC approach: the continuing review and development of the ToC to take into account shifting contextual circumstances. Dyson and Todd (2010: 125), discussing the use of a ToC approach applied to the ‘extended schools’ programme in England, argue that this dialogue and restatement is part of the value of the ToC approach – ‘the dialogical nature of theory of change evaluations means that actors can receive continual feedback from evaluators on the coherence of their theories’.
Therefore, the ToC approach involves active work from the evaluator, which can be problematic (an issue also identified by Connell and Kubisch, 1998): Dyson and Todd (2010: 127) note their work ‘did not feel much like the “surfacing” of a theory of change’ – instead it involved ‘helping leaders articulate some broad assumptions which guided their work in general terms’. However, a strength of ToC was that it enabled local staff to identify additional benefits which traditional ‘area-level’ approaches can miss. Dyson and Todd (2010: 129) state that ‘we found ourselves able to identify outcomes and beneficiaries that might have slipped below the radar of more conventional evaluation designs’, an issue of relevance as we discuss in the main body of this article.
Despite these key benefits, the only substantial review of ToCs to date, conducted in relation to public health research (Breuer et al., 2015), found some significant issues in relation to the reporting of ToC use at least in this field. First, context was often missing, and where mentioned ‘there was little description of how context affected the interpretation of the evaluation’ (p. 6). Furthermore, it was rare that ToCs explicitly developed causal explanations (Breuer et al., 2015: 12; see also Barnes et al., 2003); and they often failed to include service users in their design (p. 13). Perhaps most surprisingly, especially given the points above, the review found few uses that actually described the use of the ToC in practice (p. 13): ‘we had expected that more papers would use ToC during the implementation phases to assess progress towards outcomes as well as modify implementation where necessary’.
In the current evaluation, a further limitation was identified, which is that ToC approaches focus on a specific change context, typically a bounded geographical area. This is not surprising, given its origins in evaluating change in relation to comprehensive community initiatives. While the approach is a helpful adaptable frame for understanding change in a single area it is not designed to support comparison between areas. And this is especially in cases, like the Children’s Communities initiative, where different geographical areas have broadly similar aims but are at very different stages of development. For this, an additional approach is needed as we go on to discuss.
Third-generation systems-based approaches: drawing on systems thinking
Lai and Lin (2017: 2) define system theory as ‘a macro-level theory that can be used to understand biological, physical, and social systems’, drawing out a set of ‘tenets’ summarised as follows:
Elements of systems, and organisation within systems, are structured in a hierarchical and interdependent way (either tightly or loosely);
Feedback mechanisms operate to change the system;
These processes operate as whole rather than a reductive way leading to ‘emergence’ (i.e. creation via a complex, non-linear set of processes) of outcomes;
This can create stability in the system; or may lead to instability.
This approach relates to understandings of complexity, which is of interest to evaluators addressing complex change processes in local areas and across policy agendas (Hummelbrunner, 2011; see also HM Treasury, 2020 Handling Complexity in Policy Evaluation). For example, Walton (2016) suggests complexity includes the core features of non-linearity, emergence, adaptation and uncertainty which align closely with Lai and Lin’s (2017) analysis. These features all apply to complex systems using Rogers’ (2008: 32) application of Glouberman and Zimmerman’s (2002) distinction between the simple (which Rogers interprets in evaluation terms as assuming single linear paths from inputs to outcomes), complicated (evaluations with several causal strands) and complex (recursive and emergent causal pathways).
In our role as evaluators of complex area-based interventions, it is therefore crucial that we understand and bear in mind these features. A specific issue in relation to ToC approaches is how this might relate to visual representations. Although not specifically engaging with ToC, Rogers (2008) discusses possible visualisations in her review of how complexity can be addressed in relation to programme theory, including circular pathways and annotation of models. In previous work (Coldwell, 2019: 108), we have built on Rogers’ work to suggest multiple pathways ‘using a set of interlocking models at different system levels’ and this approach was of relevance to the Children’s Community evaluation.
While such approaches are helpful in relation to description, they can fall short in relation to supporting action. This introduces an important feature in system approaches to complexity: the role of the system leader. Boylan (2016) discusses three meanings of system leadership. First is leadership across and within the system, acting as a leader outside one’s own organisation. Second, system leadership is described as ‘a form of leadership practice or orientation’ (Boylan, 2016: 61) which Boylan (2018) categorises in a later paper as adaptive leadership, a style with an orientation towards and understanding of complexity in the local system. Third, system leadership can be conceived as not merely leading within a system, but actually leading system change. Importantly, Boylan (2016: 61) notes, common to all three is ‘the importance of a sense of moral purpose’.
Synthesising ToC approaches with maturity models
ToC can be helpful to support system leaders in their systems change endeavours since it allows for constant renewal and reconsideration as time goes on. However, on its own it is less useful in supporting leaders in understanding progress, especially in comparative terms. The system leadership field has become well developed in social policy fields, and it is here that maturity models started to be introduced. For example, a maturity model is described by The Early Intervention Foundation, the What Works centre for early intervention to prevent negative outcomes for disadvantaged children and young people. They provide a maturity framework in the context of early intervention as a tool to support local areas to take a system-wide approach to improving outcomes for children and families. It is a self-assessment tool to support local partners to understand the position on early childhood intervention [in this example], identify areas for improvement, and work together to deliver positive change. (EIF, 2021)
Maturity models have also been used across other social policy contexts including by the Welsh government to look at safeguarding, and they can be found in sectors from Further Education to compulsory schooling to examine school governance. Their value to support action is exemplified by Hargreaves’ (2014) focus on using a maturity model to enable school leaders in creating and maintaining a ’self-improving system’ of schools by becoming ‘system leaders’ in the sense discussed above. In each case, they allow comparison against a set of broad standards to understand maturity against them.
The Children’s Community evaluation faced the challenge of making formative assessments of complex and emergent change processes across different social and spatial contexts then using this evidence to provide system leaders with evidence to support improved programme implementation. We achieved this through synthesising a ToC approach with a maturity model.
In the examples of the models developed in education contexts as outlined above, the content of the maturity model is invariably developed from a range of sources: literature and evidence review; practitioner consultation; and other means. The approach taken in the Children’s Communities evaluation differed in two ways. First, it came from an evaluation (rather than system leadership) frame, using a systematic approach to data gathering to build the model. Second, the model built out from a ToC approach to articulate the core aspects of systems infrastructure that were common across the three different contexts and then gained consensus about the appearance of these infrastructure at different levels of maturity – supplemented with evidence sources – to build the maturity matrix. An articulation of this process is the focus of the next section of this article.
Evaluating the Children’s Communities initiative
The discussion above highlighted how both ToC and maturity models have been used in specific practice contexts to understand both complexity and the role of system leadership in system change initiatives. In this section we apply this model to the evaluation of the Children’s Communities, setting out the development of the combined ToC and maturity matrix approach
The initial ToC approach
A theory-based approach was adopted to evaluate the Children’s Communities Initiative, and from the outset we considered complexity and systems approaches. This reflected the inherent complexity of the intervention and limitations in applying an experimental evaluation design, as discussed above. The evaluation team first supported the three Children’s Communities to develop Theories of Change for their overall programmes and in specific theme areas (each Children’s Community identified three or four priority themes for action). These initial visual presentations varied in presentation, as some had been produced as part of earlier work with the University of Manchester team, and others emerged via the work with our team. Reflecting the systems thinking approach, which address the interactions between different (hierarchical) elements of the system, the Communities were encouraged to consider change and interactions at different system levels. Specifically, the area strategic level (conceptualised as area-based governance in the form of a programme board or similar with responsibility for strategic oversight across the Children’s Community); the operational management level (defined here as the those working in organisations and services with responsibility for resource allocation and utilisation in the Children’s Community areas) and services and programmes delivery level (practitioners and workers delivering services in Children’s Community areas). This encouraged consideration of dynamic, interactive, systemic change. Figure 1 visually depicts a top-level change hypothesis developed in the early evaluation stages with the funder.
How the work of the Children’s Communities is hypothesised to influence long-term outcomes for children and young people.
Second, as the initiative and its evaluation developed, the evaluation team aimed to observe Walton’s (2016) features of complexity playing out and encouraged the Communities to expect and plan (as far as possible) for adaptation and change. This may be both non-linear (for instance, many different activities or developments happening simultaneously) and sporadic (such as long periods of little apparent progress followed by tipping points).
The local Theories of Change were then used in the evaluation to inform a programme-wide evaluation framework that was based on assumptions around the causal process underlying the programme. This framework aimed to provide an understanding, and a guide to analyse the Children’s Communities programme by specifying the goals and measurable objectives; defining relationships between inputs, activities, outputs, outcomes and impacts that can be tested and hypothesised through the evaluation; and clarifying the relationship with external factors that may mediate, constrain or assist the programme.
An evaluation framework aimed to specify the categories of information that would be collated and analysed to understand and demonstrate what difference the Children’s Communities had made. Broadly speaking this reflected both system change processes (as the mechanisms for change) well as beneficiary and area-level outcomes (the potential for systems change to lead to improved outcomes for children and young people living in the Children’s Communities).
Many standard measures and tools were available to understand and measure beneficiary and area-level outcomes, which included educational attainment, childhood wellbeing and health measures. However, the evaluation faced a challenge in how to assess systems change and how to inform the development of the systems approach across the Children’s Communities which were at various stages in the development and implementation of their programmes. The development of a maturity model, drawing on those utilised in other social policy evaluations as outlined above, was identified as a solution to these challenges.
As noted above, maturity models provide an ordered series of levels of effectiveness (Wendler, 2012). They assume that components of the system will pass through given levels in the sequence of maturity as they become more effective. That said maturity models, like any model, are a simplification of what otherwise would be a more complex process of change. Developing a successful maturity model requires agreement on the key aspects that are required within an overall change process to improve programme effectiveness and deliver desired outcomes. They also work best when a clear and logical sequencing of change can be set out. Working with a maturity model begins with a maturity-level assessment to determine the current level of progress. Once the assessment has determined the current level, the description of the level above can be used to prioritise what needs to be done next to improve. This prioritisation is a benefit of using a maturity model: it informs both the evaluation and the future delivery of the programme.
Having identified the appropriateness of a maturity model approach the next section sets out the development of a maturity model for the Children’s Communities Initiative.
Development of a Children’s Community maturity model
The development of a maturity model for the Children’s Communities emerged out of recognition that although the ToC informed the evaluation framework in relation to the programme’s aims, objectives and mechanisms of change, the Children’s Communities had different origins and two were further ahead as established partnerships with shared aims (see Table 1). Consequently, the evaluation team sought an evaluation design to incorporate the implications of this in relation to the development of system-wide approaches to improving outcomes for children and young people. Working with the funder and the project teams in the three Children’s Community areas the evaluators co-produced an initial change model for the Children’s Community programme, utilising the concept of a maturity model as discussed in the previous section. This early model, shown in Figure 2, identified three distinct, but potentially overlapping phases to reflect the expected progress of systems change processes.
Initial maturity model for Children’s Community programme.
First, in the building phase, the Children’s Community partnership is established as a partnership between Children’s Community teams (funded by Save the Children UK in Smallshaw Hurst and Wallsend, and by Hackney Council and Peabody in Pembury) and local agencies working to improve outcomes for children and young people. The partnership undertakes early work to frame the approach – identifying issues, working out the focus, gathering evidence, creating a governance structure and aligning community priorities with the services and organisations engaged in it (as well as the core Children’s Community principles). Some early, partnership-building activities and programmes may take place during this phase. Second, the Children’s Community will gradually move into the development phase, when a coherent set of activities is implemented, and monitored in relation to changes they aim to engender in the short and medium term at different system levels. Third, in the mature embedding phase, the Children’s Community becomes more self-sustaining (although there may be a continued need for resources for system stewardship) and focuses on longer term change, continually monitoring and amending both the activities undertaken and the partnership in relation to governance and engagement of partners and stakeholders (indicated by the arrows).
Developing an analytical framework
Having identified three levels of maturity within the system change process it was necessary to build an analytical framework through which to assess processes of change within the Children’s Communities. Again, this was co-produced with Save the Children UK and the Children’s Community teams to ensure multiple viewpoints on the key aspects of the change processes involved were captured. Adopting co-production helped ground the maturity framework in the ToC. It meant articulating the work of the Communities at various stages of development as they progress towards becoming sustaining systems which improve the lives of all children and young people in a local area.
The resulting analytical framework encompassed three overarching system-level categories: strategic direction, operational management and services and programmes. Strategic direction broadly refers to the planning and governance within the systems. Operational management encapsulates the operation and management of the Children’s Community and services. Finally, services and programmes focus on the delivery of services and experience of service users. An overarching category of ‘people’ was also included, to reflect the importance of people and relationships as central to systems change (Lowe et al., 2021).
Within these categories, 11 infrastructure aspects (Table 2) were identified which relate to things that can be seen (and evaluated) within the system to assess and promote maturity. These aspects were informed by a wide range of evidence. It included the experience of Save the Children UK and Children’s Community teams in implementing systems change and separate work commissioned by Save the Children UK to develop a framework for local systems change (this applied to Children’s Communities and other local systems change programmes supported by the funder). They were also based on early evidence gathering through interviews, focus groups and observations in the first year of the evaluation and by review of relevant academic and grey literature undertaken by the evaluation team. Each infrastructure aspect had an associated set of features. For example, ‘governance’ was identified as a key element of infrastructure to promote the strategic direction of systems focused on improving outcomes for young people, children and families. Governance was identified as a body to provide formal oversight and coordination within the system, bringing together key partners with responsibility for strategic oversight. It was identified as being important to agree recommendations, provide feedback, set direction and/or strategy, provide co-ordination and timelines, promote and facilitate networking and ensure success stories and learning are shared. The 11 infrastructure aspects that were identified were not mutually exclusive across the system-level categories, with four aspects featuring within more than one category.
Children’s Communities analytical framework: system-level categories.
Under each infrastructure aspect, in line with the maturity model discussed above, a three-stage categorisation of maturity (‘building’, ‘developing’, ‘sustaining’) was set out to capture the processes through which a newly formed Children’s Community becomes an established agent of local change. The associated assumption is that moving through the maturity phases – from ‘building’ to ‘sustaining’ – will induce positive longer-term outcomes for families, children and young people in the area. The categorisation provided qualitative descriptions of what is expected to be seen at the three levels of maturity. These descriptions articulate a base to assess the level of maturity of the system’s infrastructure, as well as providing guidance on where the Children’s Communities should focus to attain higher levels of maturity.
For evaluation purposes, the maturity model acts as an evaluation rubric (Martens, 2018). Developed initially for assessment of pupil progress in education contexts, a rubric is described by Better Evaluation as a framework that sets out both criteria and standards for different levels of performance and describes what performance would look like at each level. 5 In programme evaluation, rubrics can be used to support evaluators to make systematic evaluative judgements, based on explicit reasoning. In this evaluation the maturity model, which is rooted in a qualitative methodology, is a mechanism for making explicit evaluative assessments about the progress of Children’s Communities working in different contexts towards an agreed model of systems change.
Figure 3 illustrates the analytical framework for one infrastructure aspect: governance. For the purposes of brevity, we have included only one infrastructure aspect in the main body of this article. The full analytical framework is available in Supplementary Figure S1.
Extract from analytical framework on governance.
Using an analytical framework: The example of governance
Data were gathered using a range of tools: interviews and focus groups with Children’s Community Teams and representatives of partner organisations, observations of meetings of Children’s Community Boards and Management Groups, analysis of Board reports and Children’s Community documentation and interviews with residents. These data were then analysed for evidence against each of the relevant infrastructure criteria and consequently to assess progress against the maturity categories. The assessment could also be used to identify recommendations for system leaders to improve programme implementation. For example, at the end of the second year, the Wallsend Children’s Community was judged to be placed in the ‘developing’ category against governance, with two developmental goals identified to move forward:
Working to meaningfully engage the local community in setting the direction for the Children’s Community (see also operational management section).
Creating space in governance group meetings for higher level, purposeful strategic discussion.
In the final year, we were able to ascertain development against these goals, as indicated in the extract from the evaluation report for Wallsend Children’s Community in Box 3.
Extract of final evaluation report for Wallsend Children’s Community – Focus on governance and strategic direction.
This enabled the evaluation team to make an informed judgement that Wallsend Children’s Community had progressed from the Developing stage to the Sustaining stage over the course of the final evaluation period. Similar processes, and data sources, were used for the other two Children’s Communities – and across all elements of the analytical framework. This allowed us to present, in very condensed form, a graphical representation change across the 3 years, as indicated in Figure 4.
Summary of progress against the analytical framework.
Discussion
This article began by identifying the challenges for evaluators grappling with the complexity of area-based system change initiatives, given their adaptive, multi-layered, emergent and inter-related features. The article addresses the question: how can evaluators effectively address complex place-based system change initiatives, particularly those working across differing contexts? Based on our experience of conducting an evaluation of the Children’s Community initiative in England, we articulate a response which straddled evaluation paradigms. This developed a theory-based approach which embraced complexity as well as articulating an understanding of the dynamics and processes shaping outcomes within an area. In doing so, it constructed an evidence-based account of both how, and why, local systems change happens, against which progress can be evaluated and which provides a basis for recommendations for programme implementation.
A ToC was adopted to promote a systematic and cumulative study of the causal links between inputs, activities, outcomes and contexts of the Children’s Communities initiative. To support the evaluation of the ToC a maturity matrix approach was developed to measure and evaluate systems change: in this case the progress of the local system towards the display of system features and behaviours likely to lead to improved outcomes for children and young people. To further operationalise this, systems change analytical frame was established to set out the systems infrastructure that would be assessed for levels of maturity. This analytical frame identified key systems infrastructure – common aspects across local systems – and described their characteristics at different levels of maturity. The development of a local (place-based) systems maturity model was a key aspect of our approach, helping us as evaluators to make evidence informed assessments of change over time, and to compare different geographical areas while also acknowledging that each individual aspect of systems infrastructure will play out in each context.
In the remainder of this section, we distil the key features of the approach, to allow other evaluators to adopt or – more likely – adapt such an approach for other system change interventions. We also reflect on the benefits and limitations of such an approach.
The development of our approach was messy and iterative; indeed, this may be a typical feature of evaluating adaptive system change initiatives (Gates, 2017); nevertheless, there was a broadly sequential process we used, which we represent schematically in Figure 4. First, working with stakeholders in each local change programme, we developed and agreed a set of ToC representations of their overall approach to improving outcomes for children and young people in their communities. An alternative to ToC could be used of course. It was then necessary to gather initial data to both help test out the ToC and make a first rough judgement of progress. At this stage, we then worked with Save the Children UK and the three Children’s Communities to build an understanding of their local systems and the common infrastructure upon which the systems were underpinned. This included governance, culture, strategy/planning, people and workforce and services.
A developmental maturity model emerged from this work alongside an analytical framework which described the broad appearance of each aspect of infrastructure at three levels of maturity: building, developing and sustaining. This allowed us to frame sampling and data gathering in the second stage of data collection and make the first approximate judgements against the analysis framework. This phase also allowed for some refinements to the framework. The third phase of the evaluation gathered further data, allowing for a (for our purposes) final evaluation against the framework, allowing us to examine progress made, and to make comparative judgements across the three communities as identified earlier. As indicated in Figure 5, this process could then be repeated as part of an evaluation cycle.
Schematic representation of development process for the evaluation model.
Such an approach is useful in that it is flexible and adaptable, recognising the dynamic nature of local system change initiatives while providing a frame for comparative judgements to be made, of progress both within and between related areas. However, we have presented our model as developmental and there are, of course, caveats:
First, the approach is only as good as the framework used: the early evidence gathering in each Children’s Community area, working collaboratively with the funder and the Children’s Community teams and the use of wider literature gave us confidence in the frame, and it is essential to see this as an iterative process, with the frame developing as it is used.
Second, it should be used in tandem with other outcome measures. The maturity model provided an evaluative ‘bridge’ from system change to outcomes for children and families. The link between systems change approaches and improved outcomes for target populations remains something of a ‘leap of faith’, however. Earlier in the article, we noted that there is an assumption that moving through the maturity phases – from ‘building’ to ‘sustaining’ – is assumed to have positive effects on longer-term outcomes for families, children and young people. However, it was not possible within the relatively short timescale of this evaluation to robustly test that assumption. The shortcomings of area-based initiatives in addressing the broader structural constraints experienced in disadvantaged communities are well documented (Crisp et al., 2014, 2015); and while there was ample evidence captured through the maturity model of changes in system behaviours, the model requires further testing in conjunction with additional outcomes data to generate confidence that these local approaches will lead to the desired improvements for communities experiencing deprivation. Even then, we would caution against claims of causality.
Third, we are well aware of the dangers of imposing structured, comparative models on largely unstructured, emergent and adaptive processes. Therefore, the judgements made on progress, for example, need to be treated as indicative and used as a source of evidence among many others – and should be clearly represented as such. Having an external evaluation reference group, in this case, helped provide challenge and rein in any tendencies to overclaim, for example.
Finally, while in this evaluation we have outlined a largely positive utilisation of a maturity model as a tool for supporting system leaders to confidently implement programme change it is important to note also that we did not explore the impact of the approach as a motivator or catalyst for system leaders. In line with Hargreaves (2014) our conceptualisation applies a maturity model as a tool for assessment of progress, but the role of external evaluators in making the evaluative assessment is a shift from the use of maturity models as vehicles through which practitioners self-assess their own practice. There may be factors associated with ‘external’ assessment or with the positioning of evaluators as working collaboratively with funders and programme teams which encourage or inhibit improvement, but these were not explored in our study. Similarly, further exploration of the impact of the comparative nature of the assessment (comparing progress in different Children’s Communities) may have been helpful in identifying the degree to which the perception of ‘performance’ relative to peers affects the actions of system leaders.
In summary, we conclude that evaluators can usefully address the challenges of evaluating complex systems change processes in different contexts by drawing on both ToC and systems informed maturity models. Despite the limitations of the approach outlined, it has strong potential for evaluators looking to develop structured evaluation designs to assess local systems change in and across defined contexts and to provide evidence informed assessments of progress to help system leaders to improve programme delivery.
Supplemental Material
sj-docx-1-evi-10.1177_13563890221126744 – Supplemental material for Evaluating local system change using a comparative maturity matrix
Supplemental material, sj-docx-1-evi-10.1177_13563890221126744 for Evaluating local system change using a comparative maturity matrix by Mike Coldwell, Sarah Pearson and Ian Wilson in Evaluation
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This study was funded by Save the Children UK.
Supplemental material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
