Applying SCALE 3D for evaluating transformative social innovation

Abstract

Conventional evaluation and strategy approaches insufficiently address the needs of social innovation to adapt to non-linear and emergent change processes. This study addresses this shortcoming by testing a recently developed conceptual framework (3D) for the purpose of adaptive strategy and evaluation. We translated the 3D framework into a practice tool (SCALE 3D [Strategic Capacity development, Leadership and Evaluation in 3 Dimensions]) and applied it in two projects and four workshop settings through an action-research approach, involving networks of community-led sustainability initiatives. We describe practical benefits and suggest process steps for implementing SCALE 3D, as well as overall lessons learnt. We discuss how SCALE 3D can support transformation-oriented networks in alignment with adaptive strategy and evaluation approaches, to support strategic learning as well as reporting, and thereby help practitioners adapt to emerging changes and be accountable to funders. Our findings are relevant for evaluators, action researchers, strategy consultants, funders and social innovation practitioners supporting transformative networks.

Keywords

evaluation practice tool strategic learning strategy transformative social innovation

Introduction

The challenge of evaluating social innovation for transformative change

A growing number of social innovations offer concrete alternatives to established institutions that are increasingly unfit to address the intensifying socio-ecological crises humanity is facing (Howaldt et al., 2016; Pörtner et al., 2022). In themselves, these often very local grass-roots initiatives, like permaculture community gardens, regional food cooperatives or ecovillages, may seem relatively insignificant in the face of the global ‘meta crisis’ (Rowson, 2021; Stein, 2022) of the accelerating climate emergency interconnected with economic, geopolitical, psychological and educational crises. However, thousands of local social innovation initiatives are increasingly organized at regional and global scales through ‘translocal networks’ (Avelino et al., 2019; Loorbach et al., 2020) that seek to support and enable fundamental and large-scale transformative change in and across societal systems, including the economy, energy, agriculture, politics, food and finance. Some social innovation initiatives are aiming not only to improve ecological and social resilience locally but also to challenge worldviews underlying globally dominant institutions of growth-based economics, such as materialism and consumerism, and to offer practical alternatives and potential pathways towards post-capitalist social relations (Henfrey and Ford, 2018; Hobson et al., 2016).

Yet, the ambitions of social innovators for contributing to large-scale transformative change in established institutions are riddled with complex challenges and paradoxes of institutional change (Moore et al., 2014; Pel et al., 2019). Incumbent actors commonly co-opt and re-purpose emerging innovations to conform to established regime interests. While seeking to challenge dominant institutions, innovators can also benefit from, perpetuate and even reinforce them (Bauler et al., 2017; Deflorian, 2021; Pel et al., 2019). This is especially pertinent where organizations and networks supporting social innovations seek funding, support and recognition from governmental or philanthropic organizations, which typically require quantifiable evidence of impacts and delivery of pre-defined plans (Weaver and Kemp, 2017).

In this interface between established institutions and social innovations, evaluation can become both an obstacle but also an enabler for the transformative efforts of social innovators, depending on how evaluation is understood, commissioned and practiced (Chaplowe and Hejnowicz, 2021). Evaluation can enhance the influence of social innovations in governmental decision-making and wider society by strengthening their perceived legitimacy and effectiveness (Hobson et al., 2016). It can also help innovators amplify their impact by guiding development of their strategies and improving delivery through more rigorous and evidence-based ways of planning, reflecting and making decisions or surfacing inconsistencies between their intentions and actions (Milley et al., 2018). However, conventional evaluation approaches employ linear models of causality, focus on individual projects or programmes and primarily serve accountability to funders, which can limit or even corrupt the transformative potential of social innovation. While conventional evaluation methods like formative and summative evaluations are still appropriate for more compliance-oriented initiatives or programmes, they are largely unsuited for transformative innovation, which is characterized by high complexity, uncertainty and emergence (Chaplowe and Hejnowicz, 2021). Instead, evaluation for social innovation needs to support learning and adaptation of strategies and plans, address the complexity and unpredictability of socio-ecological systems and focus on multiple actors and programmes that contribute to desired outcomes and impacts (Milley et al., 2018; Weaver and Kemp, 2017).

In particular, transcending the fixation on accountability is vital for evaluation to support rather than hinder social innovation efforts (Regeer et al., 2016). The different interests of funders and social innovators often create a tension between accountability and learning¹ (Guijt, 2010; Milley et al., 2018; Picciotto, 2018; Reinertsen et al., 2022). Funders typically want to know that project plans are progressing according to plan and intended outcomes are being realized. This can result in evaluation efforts by social innovators that remain committed to initially agreed plans, rather than reviewing and adapting their plans in response to changing conditions and learning from experience. When performance is tied to funding, this can create perverse incentives to overstate results, prioritize funder’s goals over the social mission or focus on activities that can be more readily measured, rather than those that are most generative of desired outcomes, which are often difficult to measure (Milley et al., 2018; Preskill and Beer, 2012; Weaver and Kemp, 2017). In this way, funders may unwittingly obstruct the social change efforts they wish to support (Haldrup, 2022).

Instead of seeing accountability and learning as a trade-off, the two can go hand in hand (Guijt, 2010). Non-conventional evaluation approaches can shift the focus of accountability from delivery of pre-defined plans to the learning process itself. With this reframing, social innovators are ‘acting accountably when they pay careful attention to what is emerging as they work and adapt accordingly [. . .] Innovators are accountable to the learning’ (Preskill and Beer, 2012: 16). Through ‘strategic learning’, evaluation and strategy interact in a reciprocal relationship (Carr et al., 2019; Coffman and Beer, 2011; Preskill and Mack, 2013): The strategic orientation of vision, mission and goals informs what is relevant to measure, while the reflection on data, assumptions and evidence can help to guide strategic thinking and to make better decisions in uncertain and emergent circumstances (Coffman and Beer, 2011).

A common practice for developing both strategy and evaluation is to articulate a Theory of Change (ToC) or logic model. This usually spells out how resources invested in activities and outputs lead to outcomes among target groups and impacts at a societal level. This can then function as a basis for monitoring and evaluation to determine whether and how desired and unexpected changes resulted from the activities performed (Dhillon and Vaca, 2018). A number of criticisms (Arkesteijn et al., 2015) and limitations (Regeer et al., 2016) have been raised against the use of logic models that echo the criticisms of conventional evaluation approaches. However, these criticisms can be addressed by implementing logic models in way that supports both learning and accountability: ‘when used adaptively, logical frameworks can greatly enhance the possibility of achieving impact by enabling reflection upon, and responding, to important contextual changes’ (Botha et al., 2017, 8). To enable strategic learning, logic models need to be used in ways that support ongoing sense-making and adaptive development, ‘to go beyond a “proving” mind-set [. . .] to one that seeks to improve’ (Dhillon and Vaca, 2018: 70). Newer developments in ToC research, such as Systemic Theory of Change (SToC), highlight the importance of addressing complex and non-linear system dynamics (Murphy and Jones, 2020, 2021). Yet in practice, it is challenging to reconcile this complexity with the pragmatic need for articulating in simple-enough terms how specific actions are intended to result in desired outcomes. Murphy and Jones (2020, 2021) suggest that systems-mapping and leverage analysis offers a possible way to address this concern.

A growing number of approaches are emerging for integrating adaptive strategic planning, evaluation and learning in ways that are more suited to complex and unpredictable contexts of social innovation (Milley et al., 2018). These include reflexive monitoring (van Mierlo, 2010), developmental evaluation (Patton, 2011), adaptive management (Armitage et al., 2008; Fabricius and Cundill, 2014; Roux et al., 2022), adaptive learning (LaFond and Haley, 2022), and strategy testing (Ladner, 2015). What these have in common is using monitoring data as an integrated part of structured and ongoing reflection and learning processes to adapt strategies to emerging internal and external conditions that cannot be predicted or controlled in a linear way. However, these approaches lack a specific focus on how they can be usefully applied by networks of social innovators that aim for large-scale transformative change in dominant institutions.

The 3D framework: From conceptual model to practice tool

This study describes how a recently developed conceptual model – the 3D framework – can be used for adaptive evaluation and strategy development, in a way that can support the efforts of transformative social innovation (TSI) networks. In recent studies, we developed the 3D framework (Strasser et al., 2019) and revised it through empirical testing (Strasser et al., 2020, 2022). It builds on transformative social innovation theory (Haxeltine et al., 2017; Pel et al., 2020), which articulates the relationship between social innovation (defined as ‘new social relations and related ways of doing, organizing, knowing and framing’) and transformative societal change (defined as ‘challenging, altering or replacing dominant institutions’). While recognizing a diversity of approaches, perspectives and normative orientations among different research fields studying transformative change processes (Feola, 2015), we specifically position the 3D framework within TSI theory, which seeks to build on many of these traditions (Pel et al., 2020). The 3D framework also focuses specifically on how translocal networks can enable social innovation initiatives to scale their impact, as these translocal networks have been identified as a key mechanism for agency and empowerment of TSI (Avelino et al., 2019; Loorbach et al., 2020). By drawing on TSI theory and further case work, we distilled recurrent patterns across a variety of TSI networks, describing how (1) ‘network leadership’ roles and practices can contribute to the development of a range of (2) ‘transformative capacities’, the collective abilities among TSI networks that are required to achieve (3) ‘transformative impacts’ at a societal level, across three dimensions of depth, width and length (respectively defined as: widespread and coherent influence, structural and cultural embeddedness and persistent and evolving reproduction). See Figure 1 for an overview of the 3D framework and Supplemental Figure S1 in Section A of Supplemental Material One for the complete 3D framework, which includes all the constituent elements.

Figure 1.

Overview of the 3D framework (Strasser et al., 2022). The three layers numbered 1–3 represent network leadership, transformative capacity and transformative impact. The green, blue and red colours represent the three dimensions of width, depth and length. The double-headed arrows indicate interactions across these layers and dimensions.

The overall three-layered structure of network leadership roles, transformative capacities and transformative impacts described in the 3D framework broadly corresponds to the structure of logic models (activities-outcomes-impacts). The specific framework elements point to generic issues that are pertinent to a wide variety of TSI networks and thereby can help these networks develop and refine their own logic models, or theories of change. The SCALE 3D (Strategic Capacity development, Leadership and Evaluation in 3 Dimensions) tool also includes questions for each of the 3D framework elements. These can guide practitioners to develop indicators that are appropriate to measure and reflect on their activities, outcomes and impacts. As discussed in our previous studies (Strasser et al., 2020, 2022), the widening, deepening and lengthening capacities in the 3D framework broadly align with and complement other typologies of how social innovators can scale, diffuse or amplify their efforts to attain institutional changes (Lam et al., 2020). The aforementioned characteristics of the 3D model can help to better understand, measure, evaluate and design for the conditions that TSI networks require to contribute to transformative impact. The underlying ambition for developing the 3D framework is to increase the practical usefulness of the TSI theory for social innovation practitioners (especially those performing network leadership roles²), to strengthen intentional, systematic, equitable and resource-effective transformation efforts.

So far, the empirical application and refinement of the 3D framework has involved collecting feedback from practitioners about the recognizability and perceived usefulness of the framework (Strasser et al., 2020, 2022). Overall, practitioners saw the 3D framework as a recognizable and practically useful lens for understanding transformative impacts and capacities, evaluating strengths and shortcomings and clarifying strategic plans and priorities. We also found that evaluation and strategy can influence the prioritization, inter-linkages and performance of all network leadership roles (Strasser et al., 2022). Building on these findings, the current article specifically focusses on testing the practical applicability of the 3D framework for evaluation and strategy development.

While, in our previous work, we refined the 3D framework content based on empirical observations and the perceptions of practitioners, we now test an approach for how the revised framework can be meaningfully applied to support the practice of social innovation. For this purpose, we translated the 3D framework into a practice tool called ‘SCALE 3D’, as well as suggested a step-by-step application process. This study addresses the following research questions:

(1) What do network leadership practitioners see as the practical benefits of applying SCALE 3D for supporting their transformative efforts, based on actual or intended use?

(2) How can SCALE 3D be implemented in TSI networks, in ways that align with adaptive strategy and evaluation approaches?

By answering these questions, we seek to address the need for adaptive strategy and evaluation approaches that are specifically suited to the context of TSI networks.

In the Methodology section, we describe the methodology used to test SCALE 3D in practice and analyze the results. In the Results section, we describe the experiences and lessons learnt from applying SCALE 3D, regarding its practical benefits and the implementation process. In the Discussion section, we discuss the significance and limitations of SCALE 3D and recommendations for future research. Finally, we discuss how SCALE 3D can be used in the context of transformative networks for adaptive strategy and evaluation.

Methodology

Overall approach

To operationalize the 3D framework for practical application, we first developed an initial application approach (see Supplemental Material One, Section B) including a list of possible application purposes, questions for developing indicators, a draft version of a practice tool as well as a suggested application process (informed by adaptive strategy and evaluation approaches described in the Introduction section). This initial approach was then tested and further developed through action research (Herr and Anderson, 2014; Jessop et al., 2013) in multiple test cases (see Cases section). This involved (1) experimenting with the approach while adapting it to different cases, (2) noting observations about what was done in different cases, challenges encountered and practitioner feedback on perceived benefits and what worked well or could be improved.

Tool development

The following design research principles guided our approach to development and testing of a prototype tool: engaging users in early testing of the tool prototype and iterative steps of tool development based on users’ feedback (Baldassarre et al., 2020; Peffers et al., 2007). The 3D framework was translated into a practice tool, with the acronym ‘SCALE 3D’. This tool consists of both a Google spreadsheet³ (https://bit.ly/SCALE3D-sheet) (See Figures 2 and 3) and a digital canvas (https://bit.ly/SCALE3D-canvas) using the virtual whiteboard software miro⁴ (See Figure 4). The canvas is intended for use in interactive online workshops where multiple people can simultaneously collaborate from different locations and to create a more general overview of (the relationships among) the roles, capacities and impacts that are pertinent to a given use case. The spreadsheet is intended for a more in-depth application in the context of developing and evaluating the strategy of a project, organization or network. The spreadsheet and canvas were used in slightly different ways, depending on what seemed most appropriate to each case.

Figure 2.

SCALE 3D sheet example: logic overview tab. The white fields are filled with examples of strategic impact and capacity goals and network leadership roles that together form a high-level overview of the Theory of Change of a network.

Figure 3.

SCALE 3D sheet example: transformative impact tab. The sheet consists of multiple tabs for each of the three 3D framework layers, allowing for more detailed responses, including assessments of current performance, articulation of strategic goals and priorities for further developments needed. A rating function allows for prioritization.

Figure 4.

SCALE 3D canvas example. The ‘cards’ around the canvas contain the 3D framework elements and can be moved onto the canvas, and the yellow ‘sticky notes’ are examples of how the 3D framework elements can be applied to a given case. The black lines indicate interactions.

The testing experience and practitioner feedback then guided the further development of the canvas and spreadsheet, resulting in two additional tabs in the spreadsheet and an additional template for specific work streams (https://bit.ly/SCALE3D-MnEplan). A second canvas version was created for mapping interactions across network leadership roles (see Supplemental Figure S2 in Section 3 of Supplemental Material Two). Case-specific indicators were developed for measuring outcomes and contributions of work streams in two cases. The initial application process was revised based on the testing experiences and practitioner feedback, by adding more concrete steps, guiding questions and suggested actions for evaluators (See Supplemental Material One, Section C).

Cases

The approach for applying SCALE 3D was tested in the context of six cases: two projects and four workshops. Table 1 describes each case and the test interactions per case. The projects and workshops involved paid staff and unpaid member representatives of organizations that support decentral networks at national (the Netherlands, Scotland, England and Wales), European and global scales. All the chosen cases belong to the wider social innovation field (Pel et al., 2020) of community-led sustainability initiatives. Because we sought to experiment with the process of applying SCALE 3D, the testing process was conducted in slightly different ways,⁵ over different timeframes (four one-off workshops and two project involvements of 4 and 8 months), and with different degrees of detail in the use of 3D framework elements. A detailed description of activities in each case is presented in Supplemental Material One, Section D.

Table 1.

Description of cases and test interactions.

Projects
Case description	Test interactions
Transition Together^a is a project run by Transition Network, in collaboration with SCCAN, seeking to strengthen the Transition movement in England, Scotland and Wales, by developing a networking and support infrastructure.	• Attended four online strategy meetings in September 2021 (unpaid) for understanding strategic ambitions. • Worked as a paid evaluator (December 2021–June 2022): attended weekly team meetings and monthly partner meetings, organized whole-team and bilateral meetings to shape strategic goals, work stream contributions, indicators and the monitoring and learning approach. • The core team consisted of five half-time paid staff.
BLAST^b (Blended Adult Learning for the Socio-ecological Transition) is a European Union-funded Erasmus + project coordinated by ECOLISE, in collaboration with 10 partner organizations supporting community-led sustainability initiatives and offering transformative education.	• Facilitated meetings and asynchronous collaboration on documents to clarify project objectives, capacity contributions of the project and its work streams, and to define an initial list of indicators (November 2019–March 2020, unpaid). • Coordinated the end of project evaluation, including finalizing indicators, conducting surveys and writing a project report (September 2021–November 2022, paid). • The project consortium consisted of 12 paid positions from five partner organizations.
Workshops
Case description	Test interactions
GEN Netherlands^c is a national network connecting and supporting Dutch ecovillages and belongs to the Global Ecovillage Network (GEN).	• Facilitated a 2 hour virtual workshop in November 2020, with three staff members, to discuss strategic goals for 2021.
ECOLISE^d is a European meta-network that connects networks and organizations supporting community-led sustainability initiatives from the ecovillage, permaculture and transition movements, academic institutes and others.	• Facilitated a 2 hour virtual workshop in February 2021, with 12 participants, including 7 staff and 5 member representatives, to clarify goals and the contributions of work streams and their interactions.
Transition Network^e supports citizen groups in towns and cities who are ‘reclaiming the economy, sparking entrepreneurship, reimagining work, reskilling themselves and weaving webs of connection and support.’	• Facilitated a one and a half hour virtual workshop in November 2022 with 12 core team members, to explore how SCALE 3D could be useful for their strategy and evaluation needs.
Scottish Communities Climate Action Network^f (SCCAN) supports and connects Transition groups and other community-led sustainability initiatives in Scotland.	• Facilitated a 2 hour virtual workshop with 13 participants from SCCAN, to explore how they could start using SCALE 3D for the strategic development and evaluation of their activities.

Note: SCALE 3D = Strategic Capacity development, Leadership and Evaluation in 3 Dimensions.

transitiontogether.org.uk

wiki.communitiesforfuture.org/wiki/BLAST

ecodorpennetwerk.nl

ecolise.eu

transitionnetwork.org

www.scottishcommunitiescan.org.uk

The selection of and involvement in all the test cases for this study builds on previous case work for refining the 3D framework (Strasser et al., 2020, 2022). Due to these pre-existing relationships, representatives of the test cases proactively invited the first author to test the application of SCALE 3D, as they saw this as an opportunity to support their practice and shape the further development of SCALE 3D. Both projects (BLAST and Transition Together) were seen as relevant cases because they involve collaborations among partnering organizations that are collectively serving a wider network (Transition Network) or network of networks (ECOLISE). This was deemed important, since the 3D framework is intended to support networks of social innovation initiatives, more than individual projects. The first author had the role of an internal, embedded evaluator in the staff teams of the two projects. The way that SCALE 3D was applied in the two case projects followed a similar approach to how Botha et al. (2017) applied a logic model. This included the involvement of the first author as an evaluator who (1) was embedded in the team, (2) co-designed the monitoring and evaluation approach together with a small internal evaluation team and (3) facilitated and supported the team to apply the 3D model in a flexible way to clarify, reflect on and adapt goals, strategies and monitoring approaches in service of both the project ambitions and accountability to funders and other stakeholders, while (4) being responsive to internal changes and ongoing learning.

Data collection and analysis

Data were collected in qualitative ways, focusing on capturing statements from practitioners about the perceived benefits and their feedback about the process of applying SCALE 3D (what worked well and suggestions for improvement). Specifically, this involved (1) noting observations and reflections during individual and group meetings and (2) interviews and group reflections with workshop participants and project team members. While simple surveys were used initially for collecting quantitative ratings for feedback, this approach was discarded, as it was not possible to receive consistent and sufficient survey responses from practitioners who mainly preferred sharing their reflections verbally during meetings instead of filling forms.

To analyze these data, key passages from workshop and meeting minutes, interviews and observations were collected and broadly categorized by placing them under headings for (1) practical benefits, (2) observations about how SCALE 3D was applied in different ways and (3) suggestions for the process of applying SCALE 3D and the format of the tool. In this stage, the data listed under these categories were organized by test case. In a second step, these passages were analyzed by reading through the material of each case, identifying common themes across cases and selecting examples or quotes that illustrate those themes. We thereby defined initial overall themes of perceived benefits, appropriate steps for applying SCALE 3D and patterns in the testing experiences. These themes, steps and patterns were then refined again by reviewing the data, to describe them – as much as possible – in ways that are specific, representative of the empirical experiences and understandable for practitioners.

The results presented in the following section largely represent the empirical material from the two projects and, to a lesser degree, the four workshops because SCALE 3D was tested in the projects in a more in-depth way and over longer time periods. The entire body of empirical material was analyzed to identify main themes, steps and patterns.

Ethical considerations were addressed by asking members of test cases for informed consent about participation in this study, using the data resulting from testing for the purpose of this research and ensuring confidentiality regarding any potentially sensitive data. Fictive names were used to maintain anonymity of the practitioners referred to in this study.

Results

The following sub-sections present the results from testing SCALE 3D in practice, focusing on how SCALE 3D was observed to be beneficial in practical use applications, the process of implementing SCALE 3D in practice and the main lessons learnt from the experiences of implementation.

Benefits and use applications

Table 2 summarizes the practical benefits of applying SCALE 3D and the more specific use applications, which are based on actual or intended⁶ use. More detailed examples including excerpts from interviews are presented in Supplemental Material Two for each of these benefits and use applications. The five broad ‘benefits’ of SCALE 3D identified by participants were (1) a structure for thinking and communicating strategic plans using an intuitive language, (2) impact-oriented planning and prioritization of goals and activities, (3) cohesion of strategic activities and partnerships, (4) comprehensiveness of strategy development by highlighting possibly overlooked goals and activities and (5) guidance for developing meaningful evaluation approaches that are holistic, adaptive and learning-oriented. While the first four benefits focus more on strategy development, and the fifth more on evaluation per se, all five are part of a process of strategic learning, where strategy and evaluation are coherently integrated. The process of articulating impact and outcome goals, work stream contributions and monitoring and evaluation approaches informed an iterative learning process that resulted in further developments of strategic goals and plans.

Table 2.

Summary of practical benefits and use applications of SCALE 3D.

Benefits	Use applications
Structure for thinking and communicating	• Guiding strategic conversations through a simple, easily adopted and intuitive language • Describing strategic plans in a structured way for potential funders
Impact-orientation for planning and prioritization	• Planning goals and activities based on intended impacts • Prioritizing activities for effective resource allocation • Generating and assessing ideas for activity design
Cohesion of strategies and partnerships	• Articulating the interconnected nature of transformative efforts • Strengthening synergistic relationships among activities • Aligning efforts of organizations and their partners towards shared goals
Comprehensiveness of strategy development	• Identifying important capacities that may be overlooked or underdeveloped • Identifying important network leadership roles that may be missing or underdeveloped
Guidance for developing meaningful evaluation approaches	• Integrating evaluation into strategy design and delivery • Developing holistic and adaptive monitoring and evaluation approaches • Using monitoring data to support ongoing learning and development of strategic activities

Note: SCALE 3D = Strategic Capacity development, Leadership and Evaluation in 3 Dimensions.

Implementation process

As a result of the experiences from testing the approach for applying SCALE 3D in practice, we developed a suggested five-step implementation process that aligns with adaptive strategy and evaluation approaches described in the Introduction section. A more detailed account of the testing experiences is presented in Supplemental Material Three, including challenges encountered and responses to these challenges. The suggested five implementation steps are (1) clarifying the strategic goals (intended transformative impacts and capacity outcomes) and network leadership roles of a network, organization and/or project, (2) defining and designing activities so they contribute to these goals, (3) planning appropriate ways of collecting and reflecting on relevant data to measure progress towards and achievement of goals, (4) learning about and adjusting strategies over time through reflection on monitoring data and emergent developments, and (5) reporting on progress, lessons and achievements. Figure 5 offers a visual and simplified overview of these steps.

Figure 5.

Simplified visual overview of steps for implementing SCALE 3D.

The feedback from test cases suggested that an evaluator who is familiar with the SCALE 3D model needs to guide practitioners through the steps of applying and embedding it in their work activities and organizational culture. Therefore, we also identified what roles an evaluator can play in guiding each of these five steps. Table 3 summarizes the five steps and the roles for evaluators (which are further elaborated in the Overall lessons learnt section). In addition, Supplemental Material One, Section C presents a more detailed guide for practitioners and evaluators, including for each of the five steps: questions to guide strategy development and evaluation, outputs to complete each step, specific actions an evaluator can perform and related use suggestions for the SCALE 3D tool.

Table 3.

Overview of five steps and evaluator roles for implementing SCALE 3D.

Steps	Description	Evaluator roles
1. Clarifying strategic goals and roles	Clearly articulate and prioritize intended transformative impacts and capacity outcomes (in each of the three dimensions) and the network leadership roles needed to achieve these	Guide systematic articulation and prioritization of strategic goals and roles
2. Designing activities to contribute to strategic goals	Clarify how activities of various work streams can strengthen contributions to transformative capacities and how these activities can interact synergistically	Guide the articulation of capacity contributions and synergies across work streams
3. Monitoring and evaluation	Clarify which data need to be collected, how, by whom, for which purposes and by when, so it can inform strategic learning and reporting in a timely manner	Help develop and implement appropriate monitoring and evaluation approaches
4. Strategic learning	Reflect on monitoring data and internal and external developments to guide strategic learning and decision-making and adapt goals, activities and monitoring approaches	Support data-informed and goal-oriented reflection on and adaptation of strategic activities
5. Reporting	Integrate monitoring data and lessons learnt into progress and/or impact reports and make use of findings	Capture lessons learnt, write and disseminate reports

Note: SCALE 3D = Strategic Capacity development, Leadership and Evaluation in 3 Dimensions.

While listed sequentially, these five steps are highly inter-related and need not be followed in a chronological fashion. They partially occur in parallel and influence each other in non-linear iterative ways. The steps should also be seen as generic principles, allowing for a variety of ways to apply SCALE 3D. Supplemental Material One, Section D further includes descriptions of how these steps were performed in this study,⁷ offering empirical examples of how the steps can be implemented in different situations.

Overall, the feedback from practitioners suggests that it is valuable to first clarify long-term impact goals before addressing transformative capacities and the network leadership roles and activities needed to realize those. Yet, some practitioners suggested that two options can be useful and may be performed in parallel: an outward-in approach⁸ (first clarify intended impacts, then capacities, then roles and activities) and an inward-out approach (clarify how currently performed roles can contribute to capacities and impacts). In both projects studied, the latter helped to generate some creative ideas that may not have resulted from a narrower focus on previously agreed goals, which suggests that an inward-out approach may work well when specific strategic goals have not yet been clarified.

Overall lessons learnt

This section summarizes overall conclusions about evaluator roles and SCALE 3D application. These are based on the experiences of implementing SCALE 3D and feedback received from project participants. This analysis (further elaborated in Supplemental Material One, Section C) highlights that evaluators are well-placed to:

Guide practitioners to explore, concretize and prioritize strategic goals for each of the three dimensions and to clarify contributions of work streams to these goals. This helps to articulate their ToC and understand their work in the context of the 3D model.

(Re)organize the results generated through workshops and (bilateral) meetings to align with the layers and dimensions in the 3D framework, (pre)select ideas that seem most relevant and invite practitioners to discuss, refine and use the results to guide further planning and prioritization of activities.

Challenge practitioners to think more systematically about how their activities can be designed to strengthen progress towards outcomes and synergies across work streams, by facilitating group and individual sense-making that zooms out from day-to-day delivery and asking reflective questions to scrutinize intentions, assumptions and underlying reasoning.

Offer practitioners an analysis of strengths and possible gaps, after studying their strategic plans and observing their meetings and ongoing actions from the SCALE 3D perspective. Use this analysis as a basis for more focused (group) reflections on aspects of their strategy that they could further develop or think about in a more rigorous way.

Propose ways for translating strategic goals and work plans into monitoring and evaluation approaches (consisting of evaluation questions, related indicators, data collection methods and responsibilities for data collection), while involving practitioners to co-shape, confirm and take ownership for these.

Clarify times and contexts for reflecting and reporting on monitoring data, to support timely strategic learning, evidence-based decision-making, adaptation of strategic plans over time as well as reporting to funders and other stakeholders.

The lessons we drew from the overall application of SCALE 3D highlighted the need to:

Integrate the use of SCALE 3D into project design or strategy-development process in time to shape the articulation of strategic goals, required roles and related decisions about team members and required resources.

Embed monitoring and evaluation into the core responsibilities of team members (especially of project management and work stream leads), rather than delegating this responsibility to an external evaluator.

Ensure sufficient capacity and willingness for engaging in monitoring, evaluation and learning to shape strategy development: that is, to decide on, collect and reflect on relevant monitoring data and to continue developing and revising goals and work plans over time, in ways that can increase the effectiveness and results of the given project, organization or network.

Put more emphasis on the collective learning process that happens in sense-making conversations that are guided by the 3D model, more than filling in the content of the SCALE 3D tool (the canvas and the sheet).

Discussion

In this action-research study, we tested and further developed an approach for applying the 3D framework, in the form of the SCALE 3D tool, that can support practitioners in social innovation networks to develop their strategies and evaluate their activities and achievements. In this section, we discuss the potential advantages and the limitations of the SCALE 3D tool and the methodology of this study and conclude with recommendations for future research.

Potential advantages of applying SCALE 3D

Regarding the practice benefits of SCALE 3D (Research Question 1), the applications of SCALE 3D confirm and complement reported benefits previously identified (Strasser et al., 2020, 2022) and suggest that SCALE 3D can be useful to support strategic learning (Carr et al., 2019; Coffman and Beer, 2011) specifically for TSI networks. Our findings align with potential benefits of using a ToC for strategy and evaluation (Dhillon and Vaca, 2018), including more coherent and aligned interventions, mutual understanding among staff, communication with funders, explicitly articulated assumptions, more intentional execution and learning and better and timely decisions when adapting to emerging issues. SCALE 3D constitutes an overarching ToC, in the sense of a generic logic model that can be adapted to different social innovation networks, organizations or projects, based on what is relevant to their contexts. Considering that developing a ToC is typically very resource-intensive (Dhillon and Vaca, 2018), SCALE 3D can save practitioners valuable time and energy by using it as a template for designing and evolving their strategy.

Our findings further support the suggestion by Botha et al. (2017) that using a logic model in a dynamic and flexible manner to support strategic learning through ongoing reflection and adaptation begins to address common criticisms of logic models (Arkesteijn et al., 2015). Using SCALE 3D as an adaptive logic model offers a way to reconcile the tension between accountability and learning (Regeer et al., 2016) in the following three ways. It helps to (1) bring rigour to the articulation of impact-oriented goals and the process of adapting and prioritizing goals in a dynamic way as a strategy evolves over time, (2) build a cohesive ToC that shows how multiple interventions jointly contribute to intended outcomes and (3) develop indicators and measurement methods for overall outcomes and work-stream-specific activities that can be used both for reporting and guiding strategic learning.

The use of SCALE 3D transcends conventional uses of logic models. While logic models are usually developed for specific projects or programmes, SCALE 3D can also be used at the levels of networks, where multiple organizations, programmes and projects work to co-produce transformative outcomes. Resulting indicators are not just focused on conventional outputs like number of people reached or outcomes pertaining to the improvement of the lives of specific target groups or changes in specific systems. Rather they focus on changes in social relations and ways of doing, organizing, knowing and framing (Pel et al., 2020) and the capacities of social innovation actors to bring these about. This focuses evaluation and strategy on what makes social innovation potentially transformative in terms of changes in institutionalized social relations, rather than merely on positive results for particular target groups, or specific policy changes. Yet SCALE 3D can still serve the conventional function of logic models to communicate and build trusting relationships with funders because the layers in the 3D framework align with the elements of logic models (activities-outcomes-impacts), which funders are familiar with and often require.

Regarding the implementation process (Research Question 2), we described how SCALE 3D can be applied in a way that targets ongoing learning and development of a strategy, more than merely the creation and measurement of indicators for reporting to funders. This contrasts with how logic models are traditionally applied (in a more linear way, where goals, activities and indicators are defined at the start of a project and usually remain fixed throughout the project timeframe). This underlines the importance of embedding SCALE 3D in a project from the very beginning, in a way that is fully integrated with the design and implementation of a project or organizational strategy and is adequately resourced.

Our results offer practical guidance for implementing SCALE 3D for strategy development and evaluation by describing a generic five-step process that is adaptable to different practice contexts and use applications. These process steps are not entirely new and broadly align with the non-linear, context-dependent and learning-oriented process of adaptive management (Fabricius and Cundill, 2014; Roux et al., 2022) and developmental evaluation (Patton, 2011; Preskill and Beer, 2012). The contribution of this study is understanding how the 3D conceptual framework can be practically applied in contemporary social innovation settings.

The application of the five steps has to be adapted⁹ to the specific context of organizations wanting to work with SCALE 3D, including intended uses and available resources, the position of the evaluator in the programme team and the relationship between funders and grantees, especially funders’ openness to evaluation informing strategy adaptation (Preskill and Beer, 2012). The lessons learnt from testing SCALE 3D (Overall lessons learnt section) and the descriptions of how the suggested process steps were performed in different case contexts (Supplemental Material One, Section D) can offer additional guidance in implementing and adapting the process. We recommend that anyone interested to implement SCALE 3D reads our previous publications (Strasser et al., 2020, 2022) to better understand the 3D model.

Limitations of the SCALE 3D model

Various limitations and uncertainties remain about the ways that the SCALE 3D tool can be used and useful. For example, it became evident during the testing process, that:

Since SCALE 3D considers rather generic patterns of activities, capacities and impacts, it does not (currently) offer guidance on specific tactics, such as how to go about reaching more diverse people, how to cooperate strategically with policymakers or how to address issues of power.

Despite our earlier refinements of the 3D framework (Strasser et al., 2020, 2022), further revisions of the SCALE 3D tool are needed to simplify the language in a way that makes it ‘less academic’ and more accessible to practitioners.

Some aspects of the 3D framework for transformative impacts and capacities can be difficult to measure, so their usefulness may be more limited to the intentional, impact-oriented design of strategic activities than monitoring and reporting of quantitative outcomes and impacts achieved.

Due to the large number of elements to consider in the 3D framework, applying it in an in-depth way can be overwhelming and resource-intensive. Besides prioritizing strategic goals, related evaluation questions and data collection activities, it may also be necessary to further simplify the model and the application process.

It is also unclear how much guidance is required from an external evaluator when applying SCALE 3D. There seemed to be advantages in involving an evaluator familiar with the 3D framework and tool, who is embedded in the core team of a project or organization, proactively analyses and guides development of the strategies of a project or network, brings the results of this analysis into sense-making discussions with practitioners and guides the development and implementation of monitoring, evaluation and learning processes. Some practitioners expressed interest to apply SCALE 3D without external guidance, yet those that tried found this rather challenging. The results of this article may make this more feasible.

Methodological limitations

Some limitations were more methodological. In general, we did not perform a statistical assessment of the relative importance of the benefits described in the Benefits and use applications section, due to the variations in approach and levels of detail in applying SCALE 3D, as well as the variety of case contexts. While we believe our testing experiences offer sufficient support for all the benefits described, further testing in a larger number of cases could help to substantiate the relative importance of these benefits.

Some uncertainties remain regarding the role of SCALE 3D in strategy development. The relatively short-term and low-intensity engagement in the cases that informed our findings limited the scope for observing actual changes in strategy resulting from ongoing evaluation and learning. More extended engagements using SCALE 3D may be needed to achieve more tangible outcomes in terms of strategic development. Context also matters for the applicability of SCALE 3D. There are some indications from our applications of SCALE 3D that its usefulness for strategy development may be greater in situations where practitioners have already defined at least a strategic foundation (overall vision and mission) and an initial version of their strategy (goals and activities). Yet some practitioners suggested that SCALE 3D could make a contribution to early-stage strategy development, such as designing project proposals. These different perspectives could be explored further in future applications of SCALE 3D.

Our study faced various risks of potential bias, for example:

Our selection of cases is biased towards community-led sustainability initiatives. These focus on promoting transformation through a grass-roots approach, by developing local models of alternative practice, provisioning systems and by ways of living, in ways that largely seek cooperation with, or autonomy from, incumbent political systems. We expect that SCALE 3D will also be useful for networks or social movements with more confrontational tactics¹⁰ (such as Extinction Rebellion), but this requires further substantiation.

The cases that previously informed the development and refinement of the 3D framework largely overlap with the test cases in this study, since cases were selected thanks to previous action-research engagements. Hence, the high degree of perceived usefulness may result from the fact that the framework content largely reflects back to practitioners what they already know or believe is important.

The fact that the first author was actively involved in the project teams of these cases possibly introduced a social desirability bias (Bergen and Labonté, 2020), potentially resulting in an overstatement of the perceived benefits. We attempted to address this through anonymous feedback surveys and encouraging critical comments during meetings. On the other hand, some of the benefits of SCALE 3D may be less pronounced where practitioners are more accustomed to working with evaluation frameworks.

Finally this study also did not address the question of power in evaluation (Guijt, 2010; Kaiser, 2021; Struminska-Kutra and Scholl, 2021). This relates to the role of funders and other stakeholders vis-a-vis the project team, and the roles within a team, in shaping goals and deciding what should be measured and how, when and why to adapt strategies and what meaning to ascribe to monitoring data.

These potential limitations and risks of bias could be further clarified in future applications of SCALE 3D.

Future research

Our empirical focus for applying SCALE 3D was on TSI networks (Avelino et al., 2019; Loorbach et al., 2020) or, more specifically, the intermediary organizations (Hargreaves et al., 2013; Kivimaa et al., 2019) and their projects that connect and support local social innovation initiatives. However, these intermediary organizations are also keen to support local groups to conduct monitoring and evaluation processes in a way that is useful to these groups, adapted to their varying needs, activities and capacities, while simultaneously making it possible to collect aggregated data about collective impacts of the network as a whole (Hobson et al., 2016). Future research could explore how versions of SCALE 3D could be applied also at the level of local groups and how this may help to develop locally adaptive, yet globally cohesive, evaluation approaches (i.e. by using the same conceptual language of deepening, widening and lengthening at local and translocal scales). This could help to strengthen both the efforts of local initiatives and intermediary organizations, through better understanding successes, shortcomings and enabling conditions. It may also support their fundraising and advocacy efforts, through developing a consistent evidence base of their (potential) collective impacts and the contributions of intermediary organizations to strengthen these.

Systemic theory of change design, including systems-mapping and leverage analysis, may offer a useful complementary way of applying SCALE 3D for strategy development. This would specifically align with the deepening capacities in the 3D model. This was not done in our study, as the cases and researchers did not deem a deeper systems analysis viable given the available capacity.

The limitations identified previously need to be addressed in future research that tests the use of SCALE 3D. This could, for example, involve:

Implementing SCALE 3D in more diverse cases, especially in networks that operate in the Global South or use more confrontational tactics to challenge existing institutions

Acknowledging and dealing with power relationships between funders and grantees in the evaluation and strategy-development processes more explicitly

Testing SCALE 3D during a longer period of 2 or more years, as an integrated part of the development of a network or project

Exploring how the suggested process resulting from this study can offer adequate guidance for practitioners to apply SCALE 3D in more autonomous ways and how to make it easier for practitioners to use SCALE 3D (e.g. making the tool more user-friendly and using better-suited software than a spreadsheet)

Testing how SCALE 3D could help to stimulate and develop a strategy for projects, organizations and/or networks in the very early stages of their development and how this may help to increase their transformative potential.

Conclusion

In conclusion, we found that SCALE 3D can offer a practically useful framework and process for evaluation and strategy that specifically addresses the contexts and needs of TSI networks, in a way that is compatible with adaptive strategy and evaluation approaches that are more suitable to transformative innovation than conventional strategy and evaluation approaches. SCALE 3D does this in several ways:

It focuses on changes in social relationships that challenge, alter or replace dominant institutions, rather than merely beneficial outcomes for specific target groups (e.g. access to employment or education) in particular geographic locations.

It orients strategies towards and focuses evaluation on recurring patterns of impacts, capacities and network leadership roles that are pertinent to contributing to widespread, fundamental and lasting societal changes.

It can help to navigate complex, non-linear and emergent change processes through a systematic strategic learning process, while also addressing accountability requirements of funders.

It supports cohesion and synergistic interactions among a variety of projects, programmes and partners that are needed to co-produce transformative innovation processes.

It can guide evaluators who are embedded in projects or organizations and proactively support their strategic learning and actions.

Further development of the conceptual model, the tool and the process can help to make it more understandable, simple to use and applicable to a variety of projects, organizations and networks aiming to promote transformative changes in diverse fields. We therefore encourage action researchers, evaluators, strategy consultants and social innovators to experiment with and further develop SCALE 3D and to share their experiences for strengthening a collective learning process about how evaluation and strategy can meaningfully catalyze the transformative impact of social innovation.

Supplemental Material

sj-docx-1-evi-10.1177_13563890231204664 – Supplemental material for Applying SCALE 3D for evaluating transformative social innovation

Supplemental material, sj-docx-1-evi-10.1177_13563890231204664 for Applying SCALE 3D for evaluating transformative social innovation by Tim Strasser and Joop de Kraker in Evaluation

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The first author received payment for his work as a strategy and evaluation consultant from ECOLISE for his work in the BLAST project and from the Transition Network for his work in the Transition Together project. In both cases, the clients consented to the use of information for the purpose of this study. The authors declare that there is no further conflict of interests.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Tim Strasser

Supplemental material

Supplemental material for this article is available online.

Notes

Tim Strasser is a PhD Student at Maastricht University (2014–2023) and a freelance consultant for transformative networks (since 2021), building on his research and practical experience as a network leadership practitioner.

Joop de Kraker is Professor of Sustainability Assessment at Maastricht University and holds the UNESCO Chair of Technology-enhanced Learning for Sustainable Development at Open Universiteit. His research focuses on multi-actor learning support.

References

Arkesteijn

van Mierlo

Leeuwis

(2015) The need for reflexive evaluation approaches in development cooperation. Evaluation 21(1): 99–115.

Armitage

Marschke

Plummer

(2008) Adaptive co-management and the paradox of learning. Global Environmental Change 18(1): 86–98.

Avelino

Dumitru

Cipolla

, et al. (2019) Translocal empowerment in transformative social innovation networks. European Planning Studies 28: 955–977.

Baldassarre

Konietzko

Brown

, et al. (2020) Addressing the design-implementation gap of sustainable business models by prototyping: A tool for planning and executing small-scale pilots. Journal of Cleaner Production 255: 120295.

Bauler

Pel

Backhaus

(2017) Institutionalization processes in transformative social innovation: Capture dynamics in the social and solidarity economy and basic income initiatives. In: Social Change and the Coming of Post-Consumer Society: Theoretical Advances and Policy Implications. Taylor & Francis, 234. Available at: https://books.google.nl/books?hl=en&lr=&id=uzMlDwAAQBAJ&oi=fnd&pg=PA78&dq=Institutionalization+processes+in+transformative+social+innovation&ots=2IqHpa0_4-&sig=5QPlLGsWGCOD1m-FdXoPGC03arw&redir_esc=y#v=onepage&q=Institutionalization%20processes%20in%20transformative%20social%20innovation&f=false

Bergen

Labonté

(2020) ‘Everything is perfect, and we have no problems’: Detecting and limiting social desirability bias in qualitative research. Qualitative Health Research 30: 783–92.

Botha

Coutts

Turner

, et al. (2017) Evaluating for learning and accountability in system innovation: Incorporating reflexivity in a logical framework. Outlook on Agriculture 46(2): 154–60.

Carr

Hembree

Madden

(2019) Strategic learning in practice: A case study of the Kauffman Foundation. The Foundation Review 11: 1451.

Chaplowe

Hejnowicz

(2021) Evaluating outside the box: Evaluation’s transformational potential. Social Innovations Journal 5: 704.

10.

Coffman

Beer

(2011) Evaluation to Support Strategic Learning: Principles and Practices. Center for Evaluation Innovation. Available at: https://www.evaluationinnovation.org/wp-content/uploads/2011/07/Strategic-Learning-Coffman-and-Beer.pdf (accessed 23 September 2022).

11.

Deflorian

(2021) Refigurative politics: Understanding the volatile participation of critical creatives in community gardens, repair cafés and clothing swaps. Social Movement Studies 20(3): 346–63.

12.

Dhillon

Vaca

(2018) Refining theories of change. Journal of Multidisciplinary Evaluation 14(30): 64–87.

13.

Fabricius

Cundill

(2014) Learning in adaptive management: Insights from published practice. Ecology and Society 19: 29.

14.

Feola

(2015) Societal transformation in response to global environmental change: A review of emerging concepts. Ambio 44(5): 376–90.

15.

Guijt

(2010) Accountability and Learning. Capacity Development in Practice. Available at: https://www.academia.edu/1452140/Accountability_and_Learning (accessed 28 October 2022).

16.

Haldrup

(2022) Rethinking monitoring and evaluation in complex systems – when learning is a result in itself. Medium, 20 May 2022. Available at: https://medium.com/@undp.innovation/rethinking-monitoring-and-evaluation-in-complex-systems-when-learning-is-a-result-in-itself-3d1fc90d22fc (accessed 1 July 2022).

17.

Hargreaves

Hielscher

Seyfang

, et al. (2013) Grassroots innovations in community energy: The role of intermediaries in niche development. Global Environmental Change 23(5): 868–80.

18.

Haxeltine

Pel

Wittmayer

, et al. (2017) Building a middle-range theory of transformative social innovation; theoretical pitfalls and methodological responses. European Public & Social Innovation Review 2(1): 59–77.

19.

Henfrey

Ford

(2018) Permacultures of transformation: Steps to a cultural ecology of environmental action. Journal of Political Ecology 25(1): 104–19.

20.

Herr

Anderson

(2014) The Action Research Dissertation: A Guide for Students and Faculty. Thousand Oaks, CA: Sage.

21.

Hobson

Mayne

Hamilton

(2016) Monitoring and evaluating eco-localisation: Lessons from UK low carbon community groups. Environment and Planning A: Economy and Space 48(7): 1393–410.

22.

Holley

Parkhurst

(2019) Shifting mindsets: How meaningful accountability systems can strengthen foundation learning and improve impact. The Foundation Review 11: 1455.

23.

Howaldt

Kaletka

Schröder

, et al. (2016) Comparative Analysis (Mapping 1) Mapping the World of Social Innovation: A Global Comparative Analysis across Sectors and World Regions. SI-DRIVE Social Innovation: Driving Force of Social Change. Available at: https://si-drive.archiv.zsi.at/now-available-first-comparative-analysis-on-the-worldwide-mapping-of-social-innovations/ (accessed 22 July 2022).

24.

Jessop

Moulaert

Hulgård

, et al. (2013) Social innovation research: A new stage in innovation analysis. In: Moulaert

MacCallum

Mehmood

, et al. (eds) The International Handbook on Social Innovation: Collective Action, Social Learning and Transdisciplinary Research. Cheltenham: Edward Elgar, 110–30.

25.

Kaiser

(2021) The art of measuring change. Unity Effect. Available at: https://www.unityeffect.net/resources/the-art-of-measuring-change (accessed 22 July 2022).

26.

Kivimaa

Hyysalo

Boon

, et al. (2019) Passing the baton: How intermediaries advance sustainability transitions in different phases. Environmental Innovation and Societal Transitions 31: 110–25.

27.

Ladner

(2015) Strategy Testing: An Innovative Approach to Monitoring Highly Flexible Aid Programs (Working politically in practice case study 3). San Francisco, CA: The Asia Foundation. Available at: https://www.academia.edu/17713598/Strategy_Testing_An_Innovative_Approach_to_Monitoring_Highly_Flexible_Aid_Programs

28.

LaFond

Haley

(2022) Measuring and Monitoring Adaptive Learning: A Landscape Review. Washington, DC: MOMENTUM Knowledge Accelerator. Available at: https://usaidmomentum.org/resource/adaptive-learning-measures-landscape-review/

29.

Lam

DPM

Martín-López

Wiek

, et al. (2020) Scaling the impact of sustainability initiatives: A typology of amplification processes. Urban Transformations 2(1): 3.

30.

Loorbach

Wittmayer

Avelino

, et al. (2020) Transformative innovation and translocal diffusion. Environmental Innovation and Societal Transitions 35: 251–60.

31.

Milley

Szijarto

Svensson

, et al. (2018) The evaluation of social innovation: A review and integration of the current empirical knowledge base. Evaluation 24(2): 237–58.

32.

Moore

M-L

Tjornbo

Enfors

, et al. (2014) Studying the complexity of change: Toward an analytical framework for understanding deliberate social-ecological transformations. Ecology and Society 19(4): 54.

33.

Murphy

RJA

Jones

(2020) Design management for wicked problems: Towards systemic theories of change through systemic design. In: En 22st DMI: Academic Design Management Conference, Toronto, ON, Canada, 1–16. Available at: https://designdialogues.com/wp-content/uploads/2020/11/Systemic-Theories-of-Change-ADMC-2020.pdf

34.

Murphy

RJA

Jones

(2021) Towards systemic theories of change: High-leverage strategies for managing wicked problems. Design Management Journal 16(1): 49–65.

35.

Patton

(2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press.

36.

Peffers

Tuunanen

Rothenberger

, et al. (2007) A design science research methodology for information systems research. Journal of Management Information Systems 24: 45–77.

37.

Pel

Haxeltine

Avelino

, et al. (2020) Towards a theory of transformative social innovation: A relational framework and 12 propositions. Research Policy 49(8): 104080.

38.

Pel

Wittmayer

Avelino

, et al. (2019) Paradoxes of Transformative Social Innovation: From Critical Awareness Towards Strategies of Inquiry. Available at: https://www.slideshare.net/secret/MriQAl5ZaS1W9C

39.

Picciotto

(2018) Accountability and learning in development evaluation: A commentary on Lauren Kogen’s thesis. Evaluation 24: 363–71.

40.

Pörtner

H-O

Roberts

Adams

, et al. (2022) Climate change 2022: Impacts, adaptation and vulnerability. In: IPCC sixth assessment report (IPCC), Geneva. Available at: https://www.ipcc.ch/report/ar6/wg2/

41.

Preskill

Beer

(2012) Evaluating Social Innovation. Washington, DC: Center for Evaluation Innovation.

42.

Preskill

Mack

(2013) Building a Strategic Learning and Evaluation System for Your Organization. Foundation Strategy Group (FSG). Available at: https://www.fsg.org/publications/building-strategic-learning-and-evaluation-system-your-organization

43.

Regeer

de Wildt-Liesveld

van Mierlo

, et al. (2016) Exploring ways to reconcile accountability and learning in the evaluation of niche experiments. Evaluation 22: 6–28.

44.

Reinertsen

Bjørkdahl

McNeill

(2022) Accountability versus learning in aid evaluation: A practice-oriented exploration of persistent dilemmas. Evaluation 28(3): 356–78.

45.

Roux

Novellie

Smit

IPJ

, et al. (2022) Appraising strategic adaptive management as a process of organizational learning. Journal of Environmental Management 301: 113920.

46.

Rowson

(2021) Tasting the pickle: Ten flavours of meta-crisis and the appetite for a new civilisation. Perspectiva, 9 February 2021. Available at: https://systems-souls-society.com/tasting-the-pickle-ten-flavours-of-meta-crisis-and-the-appetite-for-a-new-civilisation/ (accessed 3 November 2022).

47.

Stein

(2022) Education is the metacrisis. Perspectiva, 27 January 2022. Available at: https://systems-souls-society.com/education-is-the-metacrisis/ (accessed 3 November 2022).

48.

Strasser

de Kraker

Kemp

(2019) Developing the transformative capacity of social innovation through learning: A conceptual framework and research agenda for the roles of network leadership. Sustainability 11(5): 1304.

49.

Strasser

de Kraker

Kemp

(2020) Three dimensions of transformative impact and capacity: A conceptual framework applied in social innovation practice. Sustainability 12(11): 4742.

50.

Strasser

Kraker

de Kemp

(2022) Network leadership for transformative capacity development: Roles, practices and challenges. Global Sustainability 5: e11.

51.

Struminska-Kutra

Scholl

(2021) Taking power seriously: Towards a power-sensitive approach for transdisciplinary action research. Futures 135: 102881.

52.

van Mierlo

(2010) Reflexive Monitoring in Action: A Guide for Monitoring System Innovation Projects. Wageningen: Wageningen University & Research. Available at: http://edepot.wur.nl/149471 (accessed 18 April 2022).

53.

Weaver

Kemp

(2017) A review of evaluation methods relevant for social innovation: With suggestions for their use and development. TRANSIT working paper no. 14, July 2017). TRANSIT: EU SSH.2013.3.2-1 Grant agreement no: 613169. Available at: http://www.transitsocialinnovation.eu/content/original/Book%20covers/Local%20PDFs/267%20TRANSIT_WorkingPaper%2014_Monitoring_final_pmw.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.79 MB