Book Review: Leading Change Through Evaluation: Improvement Science in Action

Leading Change Through Evaluation: Improvement Science in Action, by Kristen L. Rohanna, provides an introduction to continuous improvement tools, improvement science concepts, and systems and complexity science thinking. In this insightful and easily digested book, Rohanna offers both evaluators and non-evaluators a roadmap to navigate complex problems to incite and sustain positive change. Rohanna defines improvement science as a systematic approach that “not only seeks to improve a problem but also the system in which the problem is situated” (p. 14). In the book, improvement science is presented as a type of participatory evaluation with formative aims whereby stakeholders collaborate to understand a problem, identify potential theories about why it exists, and engage in an iterative process of testing potential solutions.

Overview

Leading Change Through Evaluation: Improvement Science in Action, the sixth book in the Evaluation in Practice Series edited by Christina A. Christie and Marvin C. Alkin, consists of eight chapters structured across two sections. The first two chapters provide a beginner-friendly overview of formative evaluation and systems and complexity science concepts, no small feat given the sometimes confusing and often jargon-filled lexicons of these subjects. Rohanna also introduces readers to the Multidisciplinary Model of Evaluation Capacity Building (Preskill & Boyle, 2008), which emphasizes the enhancement of evaluation knowledge, skills, and attitudes in individuals and the transfer of learnings to the organizations or work contexts where the individuals are embedded. This model is an integral part of Rohanna's conceptual framework and, combined with the other ideas and tools discussed in the first two chapters, it lays the foundation for the case study that forms the second section of the book.

Chapters 3 through 8 present a detailed case study of the first year of a newly formed networked improvement community in a large urban school district. Rohanna describes networked improvement communities in education as “inter-organizational networks, whereby multiple schools join forces to tackle complex challenges” (p. 41). The goal of the multi-year initiative was to work with educators and school leaders to create systemic changes in teaching and learning, with a specific focus on students’ math achievement. The first year of their work centered on building network educators’ understanding and familiarity with improvement science. As a result, the book builds the reader's capacity by leading them through a real-world example of how to apply the concepts and tools of complexity science and other approaches discussed in the first section of the book, in addition to illustrating the barriers and facilitators to engaging in participatory evaluation and evaluation capacity-building.

In her closing chapter, Rohanna provides readers with a helpful summary of lessons learned that both emerging and seasoned evaluators can benefit from reviewing. Discussion questions at the end of each chapter and the ample use of graphics further enhance the book's value to instructors and new evaluators.

Translating Concepts into Evaluation Skills

Through a detailed description of the evaluation's design and implementation setbacks and successes, Rohanna underscores the importance of three key skills: bridging theory and practice, exercising reflection and humility, and embracing a system thinking lens. These skills are important for all evaluation practitioners, not only those working in a secondary educational environment.

Bridging Theory and Practice

In showing the application of different theoretical frameworks and tools in the field, including participatory evaluation, complexity science, systems thinking, and formative evaluation, Leading Change Through Evaluation demonstrates the bridging of theory and practice.

A major theme throughout the book is the flexibility needed to apply evaluation approaches within a practical context. For example, the kind of participatory evaluation approach that Rohanna was using in the networked improvement community necessitates building a shared understanding of evaluation goals, activities, and outcomes with stakeholders. Building this understanding can be particularly cumbersome in resource-limited environments and is further complicated when additional concepts and tools, such as those from improvement science, are introduced and incorporated into an evaluation. As many evaluators working in educational settings have experienced, educators and school leaders are busy professionals who typically have limited time to devote to evaluation activities. Recognizing these limitations, Rohanna and her colleagues offered teachers dedicated meeting time to provide support in completing assigned tasks between networked improvement community meetings. This allowed teachers to receive the assistance that they needed to meaningfully engage with the initiative while also reducing any time burden from participating in the network.

Although many evaluation approaches, or theories, exist to inform evaluation design and implementation, there is little guidance on how to navigate contextual challenges, such as limited time for stakeholder engagement in participatory evaluation, when applying specific approaches in the field. Leading Change Through Evaluation serves as a case study of how evaluators identify and implement solutions in the real world. For example, the book addresses the question of the extent to which a given theory can or should be altered in practice without sacrificing adherence to the core elements of the approach. For Rohanna, the solutions involved inviting colleagues’ feedback and perspective, considering stakeholders’ needs and time constraints, and reflecting on lessons learned from previous evaluation experiences. As one of many examples throughout the book, Rohanna explains the logic in her decision-making process and notes:

Of course, the trick is understanding when to be flexible and when to be strict, when to adhere to the righteousness of the process and when to make adaptations. For me, I drew that line at this whole network's purpose: generating meaningful learnings that could solve complex problems of practice. I wanted to accelerate learning. (p. 62)

Exercising Reflection and Humility

Leading Change Through Evaluation showcases how consistent reflection on stakeholders’ needs and other aspects of the evaluation can facilitate implementation and improve methodological rigor. In one of the most prominent examples of this, Rohanna and colleagues first decided to have their smaller networked community join a larger initiative after recognizing their lack of experience in running and developing these types of communities. However, this decision was retracted a few months later, after incongruence between teachers’ needs in their network with the larger group became clear. As Rohanna reports:

Our teachers were frustrated. By November, they had spent 3 months trying to understand the problem of practice. Although we did not want to rush to find solutions, we risked losing their interest and motivation if they did not get to try something new in their classroom soon. (p. 59)

Reflection continued to play a pivotal role throughout the first year of the networked community initiative. With every new introduction of a complexity science tool or new evaluation activity, Rohanna and colleagues carefully considered stakeholders’ understanding and contributions to the progress of goals identified for the network's first year. For example, in one activity, teachers were charged with sharing and critiquing their learnings with one another as part of the completion of a recent implementation cycle. Noticing that the sharing and critiquing was not happening to the extent desired, Rohanna asked herself, “Why did we expect productive inquiry and dialogue to occur just because we put them in a room together?” and, after reflection, “decided to explicitly teach them inquiry and dialogue skills” (p. 69). In this as in other examples, Rohanna and colleagues provided the support needed for networked teachers to participate more critically and thoughtfully in these activities.

Rohanna's examples also demonstrate how reflection and having the humility to change course when needed can strengthen methodological rigor when bridging theory and practice. Rohanna's experience indicates that taking a pause to reflect on the implications of various implementation decisions or deciding to alter plans can result in richer, more representational data without sacrificing evaluation goals and timelines. Making space for intentional reflection also facilitates the learning of lessons that can inform and improve future evaluation practice.

Embracing a System Thinking Lens

Social problems that program interventions attempt to tackle are embedded within a larger system, and no review of Leading Change Through Evaluation: Improvement Science in Action would be complete without acknowledging the central role of systems thinking throughout the book. Rohanna reminds us that, “every system is perfectly designed to get the results that it gets” (p. 29). If the evaluation is to lead to transformational change, it must examine the structures, relationships, and people that contribute to the system to produce the results, or problems, that it gets. This requires meaningfully engaging stakeholders. As Rohanna notes, “potential solutions require the expertise and experience of those closest to the problem” (p. 35).

Facilitated by ideas and tools of improvement science, such as the use of driver diagrams and Plan, Do, Study, Act (PDSA) Cycles, the networked communities provided an opportunity to learn from those closest to the problem by meaningfully engaging teachers. These tools allowed the teachers to identify problems and test for solutions in their own classrooms while the networked community meetings provided a space for teachers to learn from one another. Rohanna and colleagues served specific roles in their work with the network to achieve shared goals and foster understanding of the problem while also learning from the teachers. One member of the leadership team represented a subject-matter expert, Rohanna represented a complexity science expert, and another team member had knowledge and experience with the school systems and teachers involved in the network. In this case, Rohanna's knowledge of systems and complexity sciences served a valuable role in designing an evaluation and using it to harness change among diverse stakeholders in varied settings.

Leading Change Through Evaluation makes a strong argument for systems thinking, even for environments where systemic change may feel futile, and shows how this perspective can work with other approaches. Rohanna makes this thinking, like other concepts introduced, feel very doable. For example, recognizing the importance of defining the system as something with manageable boundaries, Rohanna and colleagues encouraged the teachers in the networked community to think of their classrooms as their system, a place where they have the locus of control and can implement immediate changes, at the same time as they were building teachers’ skills in nonlinear thinking about problems and their potential solutions and in the consideration of underlying values and belief systems that catalyze problems.

Conclusion

In their foundational review of evaluation theories, Shadish et al. (1991) called for the inclusion of a practice component to theories that would allow evaluators to apply the theory under less-than-ideal conditions or contexts that posed particular restraints. This call for practical tips and explicit directives in bridging theory and practice has largely gone unanswered, although efforts by Miller (2010) and, more recently, Alkin et al. (2021) have brought new life to this endeavor. Leading Change Through Evaluation answers the call with its detailed description of how evaluation theories were implemented and tools from different fields were integrated with a specific context. It highlights the interpersonal and intrapersonal skills needed to translate and navigate the application of evaluation approaches in the real world and shows how the American Evaluation Association's (2018) proposed evaluator competencies, such as those in the methodology and context domains, are connected. Rohanna also captures how soft skills, such as reflection and humility, are pivotal to evaluation efforts.

Clearly, the book has many strengths. At the same time, it could have been further strengthened by knowing how the evaluation capacity that Rohanna and colleagues built in the first year of the networked community facilitated work in subsequent years of the initiative. Although the primary objective of Leading Change Through Evaluation was to highlight the initial creation and launch of the networked community to address a complex problem, the story feels incomplete without understanding how year 1 lessons learned and approach utilized contributed to subsequent years and the conclusion of the network. Despite this limitation, the book is a valuable contribution to the evaluation field, especially to graduate students and evaluators interested in learning the concepts of improvement science, evaluation capacity-building, and other approaches as well as details of how to put these evaluation approaches into practice.