Abstract
The Race to the Top (RTTT) program incentivized states to use innovation for systemwide improvement of student outcomes, but little is known about how RTTT-funded innovation was sustained after the RTTT program ended. This mixed-methods study examined state and district approaches to sustaining an international innovation called lesson study, a teacher-driven, collaborative, inquiry-based teacher learning process imported from Japan and promoted statewide in Florida. While the state’s role in sustaining lesson study was limited, we found that districts that integrated lesson study into the district instructional system through a clear expectation and strategic adaptation, supported school and teacher ownership of lesson study practice, and provided necessary support and funding were more likely to sustain lesson study. In contrast, the districts that focused on implementation fidelity and district-led facilitation eventually phased out lesson study. Policy implications for sustaining federally funded professional development innovations are discussed.
Keywords
Introduction
The American Recovery and Reinvestment Act of 2009 (ARRA) signed by the President Obama provided $4.35 billion for Race to the Top (RTTT)—a competitive grant program to encourage and reward state departments of education to implement innovation for improving student outcomes with four years of funding from 2010 to 2014 (USDOE, 2009a). Representative of a new approach to federal education policy that relies on competition-based incentives instead of sanctions to drive state-led reform, the RTTT program mobilized changes in state legislations, especially regarding teacher evaluation and charter schools (Dragoset et al., 2016; McGuinn, 2012). However, improving student outcomes is a long-term process requiring sustained implementation before an innovation affects teacher learning, instructional practice, and student outcomes. Therefore, the success of state educational reform in achieving the goal of improving student outcomes statewide may depend on how states and districts sustain promoted innovations (Durand et al., 2016; Spillane, 1996).
Accordingly, the RTTT program included consideration for the sustainability of innovations as a criterion for funding. The U.S. Department of Education (USDOE, 2009a) encouraged state departments of education to consider “using the fiscal, political, and human capital resources of the State to continue, after the period of funding has ended, those reforms funded under the grant” (p. 7). Despite the importance of reform sustainability, few empirical studies examined how RTTT-funded innovations were sustained after the program ended in 2014, and little is known about what state- and district-level factors are associated with sustained practice of innovations.
The Florida Department of Education (FDOE) received RTTT funding of $700 million in 2010 by proposing a comprehensive instructional reform plan with an innovative professional learning approach imported from Japan called lesson study—a collaborative, inquiry-based process of instructional improvement that integrates standards, assessments, and teaching through a cycle of study, plan, teach, and reflect (Fernandez & Yoshida, 2004; Lewis & Hurd, 2011). FDOE chose lesson study as a main vehicle for implementing college and career readiness standards and proposed to scale up lesson study to achieve student learning goals (FDOE, 2010a).
Because teachers must be provided with rich learning opportunities to transform their beliefs about teaching and learning to improve instructional practice, teacher professional development (PD) is a central policy driver for systemwide improvement of instruction and student learning (Elmore & Burney, 1999; Knapp, 2003; Little, 1989, 1993; Marrongelle et al., 2013). Therefore, it is important to explore the state and district conditions associated with sustained practice of lesson study after the RTTT program. Based on the state and district reform capacity literature (Childs & Russell, 2017; Floden et al., 1995; Goertz et al., 2013; Gottfried et al., 2011; Massell, 1998, 2000; Massell et al., 2012; Murphy & Hill, 2011; Spillane & Thompson, 1997), we conceptualized state and districts’ capacity to sustain lesson study along three dimensions: (1) fiscal resources, (2) knowledge resources, and (3) system integration.
Specifically, we conducted a document analysis of state and district policy documents, a longitudinal survey of 34 district PD leaders from 2014 to 2016, and interviews of select district PD leaders in Florida to address the following research questions:
To what degree was lesson study sustained after RTTT funding ended?
What approaches did the Florida Department of Education (FDOE) take to sustain lesson study based on three key dimensions of state capacity—fiscal resources, human resources, and system integration?
How do three dimensions of district capacity differentiate the districts that sustained lesson study from the other districts that phased out lesson study?
The unique context of Florida, where lesson study was promoted statewide, is ideal for examining state and district approaches to sustaining lesson study. The findings will offer policy implications for how state and district leaders may sustain an innovative PD model to promote instructional improvement after major federal funding is withdrawn.
Background
RTTT, Instructional Reform, and Lesson Study
The RTTT program funded 12 states that proposed a statewide reform for improving student achievement, narrowing achievement gaps, increasing high school graduation rates, and enhancing student preparedness for college and careers in four specified reform areas: (1) standards and assessments, (2) data systems to support instruction, (3) great teachers and leaders, and (4) turning around the lowest-achieving schools (USDOE, 2009a).
This competitive grant program accelerated the standards-based reform and accountability reform from the No Child Left Behind and earlier standards movement since 1980s by promoting the adoption of the Common Core State Standards, teacher evaluation, and charter schools (Coburn et al., 2016; McGuinn, 2012). However, previous policy research identified multiple challenges in realizing its goal of improving student learning (Kober & Rentner, 2011; McGuinn, 2012; Russell et al., 2015; Superfine et al., 2012; Smylie et al., 2004). McGuinn (2012) pointed out the challenges with limited state and federal administrative capacity in a fragmented and decentralized education system. This has led to heavy reliance on nonsystem actors to overcome the state’s limited capacity, according to Russell et al. (2015), which created further challenges in coordinating larger, diverse, and unstable networks of nonsystem actors with a shared vision and goals.
Focusing on the state capacity on teacher workforce improvement, Superfine et al. (2012) further argued that state-level teacher workforce policies lack a strong and efficacious theory of action underlying the reform it promotes, as most workforce development functions and practices remained under-resourced and disconnected as silos (Smylie et al., 2004). As a result, Kober and Rentner (2011) reported a major delay in implementing reforms in improving teacher effectiveness.
Regardless of these challenges, the state approach to instructional reform was central in the RTTT funding decision, and USDOE emphasized the importance of articulating the state’s reform agenda, active participation and support of Local Educational Agencies (LEAs) and stakeholders, and state capacity building for implementing, scaling up, and sustaining the proposed reform (USDOE, 2009a).
Previous research has shown that instructional reform is extremely difficult because it requires teachers to transform deeply held beliefs about knowledge as facts, teaching as telling, and learning as memorizing, and relearn a vision of teaching for understanding and learning as co-construction of knowledge (Marrongelle et al., 2013; Thompson & Zeuli, 1999). This traditional view of teaching and learning is resistant to change because it is embedded in school and society in general, and educators have learned this view through a lifelong “apprenticeship of observation” (Lortie, 1975).
To transform teachers’ beliefs and improve instructional practice, teachers must be provided with rich professional learning opportunities that challenge traditional beliefs and support gradual transformation of these beliefs to integrate a new vision of teaching and learning promoted in instructional reform. In this sense, teacher PD is a central policy driver for systemwide improvement of instruction and student learning (Elmore & Burney, 1999; Knapp, 2003; Little, 1989, 1993; Marrongelle et al., 2013). This is particularly the case for PD such as lesson study, which supports teachers’ shifts in beliefs about teaching and learning through a cycle of study, plan, teach, and reflect within a collaborative teacher group (Hiebert & Stigler, 2017; Hiebert et al., 2002; Lewis et al., 2006).
Specifically, in lesson study, a small group of teachers goes through four stages—(1) study the standards and content of a topic considered challenging for students, examine student understanding of the topic based on the assessment results and student work, and develop a student learning goal; (2) develop a lesson plan for an experimental lesson called the “research lesson”; (3) one team member teaches the research lesson in a classroom with students while other team members observe the lesson to collect student data; and (4) discuss the effectiveness of the lesson based on the collected student data and discuss how to improve the lesson and teaching approaches to achieve the learning goal (Fernandez & Yoshida, 2004; Lewis & Hurd, 2011).
These stages allow teachers to collaboratively examine and reflect on their beliefs and assumptions about student learning and teaching and identify specific elements of instruction that promote student learning based on the observational data collected from students during the research lesson. Therefore, the structure of lesson study affords educators a rich learning opportunity to engage with and incorporate the new ideas promoted in instructional reform into their own teaching practice through collaborative inquiry—an ideal condition for promoting teacher learning identified by previous literature (Ball & Cohen, 1999; Little, 1993; Wilson & Berne, 1999).
Through this structure, teachers can build instructional capacity by developing a professional knowledge base for teaching and making data-driven and coherent instructional decisions based on standards and assessments (Hiebert et al., 2002; Lewis et al., 2006; Martone & Sireci, 2009). Most existing studies on lesson study are small-scale qualitative studies focused on teachers’ experience with lesson study, and few studies examined the effectiveness of lesson study on teacher and student outcomes. The only experimental study available was conducted by Lewis and Perry (2014, 2017) who found that elementary school teachers in experimental lesson study groups supported by rich mathematics resources improved their knowledge and student achievement in mathematics more than control group teachers.
Florida RTTT Context and Lesson Study
The FDOE’s focus on instructional reform through teacher capacity building was evident in its own theory of reform explained in the RTTT proposal: Highly effective teachers and leaders make the difference. Human capital is the core of Florida’s RTTT theory of reform. Florida will change the culture of the profession by ensuring that all teachers and school leaders are well selected, prepared, supported, respected, and accountable for their students’ achievement (FDOE, 2010a, p. 11).
To support this goal, FDOE selected lesson study as a statewide PD innovation, inspired in part by a visit to Japan by the Chancellor of Public Schools around 2008 (Akiba & Wilkinson, 2016). In its RTTT proposal, FDOE explained that “Florida will expand teacher capacity to use college and career-ready standards, multiple types of assessment (summative, formative, and interim), and lesson study to drive continuous improvement of instructional practices” (FDOE, 2010a, p. 80). The goal stated in the RTTT proposal, “By 2013-2014, all participating LEAs will fully implement lesson study supported by high-quality, web-based resources” (FDOE, 2010a, p. 94), demonstrated FDOE’s commitment to scale lesson study as a major driver of instructional reform.
To achieve this goal, FDOE created two projects relevant to lesson study, “Expand Lesson Study” and “Focus Effective Professional Development”, and required participating school districts to propose a plan to implement these two projects—alongside 11 other projects 1 –as conditions for distributing RTTT funding. While all districts were required to implement lesson study, FDOE only established compliance measures for Persistently Lowest Achieving (PLA) schools, requiring their districts to annually submit four deliverables: (1) school schedule with regularly scheduled blocks of time dedicated to lesson study for each grade level or subject area, (2) lesson used to teach, observe, study evidence of student learning and design improved instruction, (3) rosters of lesson study participants, and (4) one improved lesson plan due to lesson study participation (FDOE, 2010b, p. 6).
A total of 65 districts (90% of 72 districts in Florida) submitted a proposal and received a total of $350 million in late 2010—half of the $700 million awarded to Florida. For the remaining $350 million kept at the state level, FDOE proposed to use $42.8 million for PD projects including lesson study. Of this total PD funding, $35.8 million (84%) were dedicated to lesson study: $9.9 million for development of lesson study resources and $25.9 million for lesson study training, participant stipends, and coordinators (FDOE, 2010a, pp. 264–268, pp. 276–278). However, 81% of $42.8 million was allocated for subcontracts with external agencies for resource development and trainings. The remaining 19% was used for participant stipends for a summer training with emphasis on lesson study offered by FDOE. The heavy reliance on subcontractors is consistent with what Russell et al. (2015) found as a common approach taken by RTTT-funded states to offset limited state capacity.
With a goal of scaling lesson study supported by RTTT funding, lesson study spread quickly across the state. By 2014, a statewide survey of district PD leaders showed that lesson study was practiced in 668 schools in 46 districts—79% of 58 districts surveyed (Akiba et al., 2016). FDOE explained in its RTTT proposal that reform initiatives would be sustained in several ways (FDOE, 2010a, p. 51). First, these initiatives would either directly or indirectly address capacity building at both the state and local levels, allowing them to sustain after the RTTT program ended. Second, reforms would be embedded into the system by becoming “the way of work in Florida” (p. 51). Finally, FDOE would continue to reprioritize and repurpose existing funding streams at both the state and local levels to ensure sustainability. However, beyond these general directions for sustaining all RTTT-funded initiatives, there was no specific explanation of how each initiative, including lesson study, would be sustained.
State and District Capacity to Sustain Lesson Study
As specified in the RTTT program by USDOE (2009a), state and district leaders were expected to sustain RTTT-funded projects after the funding ended in 2014. Previous literature on state and district reform capacity (Childs & Russell, 2017; Floden et al., 1995; Goertz et al., 2013; Gottfried et al., 2011; Massell, 1998, 2000; Massell et al., 2012; Murphy & Hill, 2011; Spillane & Thompson, 1997) and professional development policy (Desimone, 2009; Desimone et al., 2002, 2007; Knapp, 2003; Little, 1989, 1993; Marrongelle et al., 2013) point to three areas as important dimensions of state and district capacity to sustain professional development: (1) fiscal resources, (2) knowledge resources, and (3) system integration. Table 1 explains these three dimensions and how they were operationalized in this study as measures of state and district capacity to sustain lesson study based on the relevant professional development literature and lesson study literature (Hiebert & Stigler, 2017; Hiebert et al., 2002; Lewis et al., 2006; Fernandez & Yoshida, 2004).
Conceptual Framework on State and District Capacity to Sustain Lesson Study (LS).
Fiscal resources
An effective use of fiscal resources is an important dimension of state and district capacity for instructional reform (Childs & Russell, 2017; Floden et al., 1995; Goertz et al., 2013; Gottfried et al., 2011; Massell, 1998, 2000; Massell et al., 2012; Murphy & Hill, 2011; Spillane & Thompson, 1997). Reprioritizing and repurposing existing and continuing funding streams such as School Improvement Grants (SIG) and Title II funding at both the state and district levels can be critical to sustaining innovations such as lesson study. The state capacity to redirect these funding streams would allow them to continue to offer workshops and trainings on lesson study for district and school leaders and cover the salaries of expert lesson study facilitators to sustain lesson study implementation.
At the district level, continued provision of funding in the forms of substitutes and stipends for teachers is a critical condition for supporting instructional improvement (Akiba et al., 2015; Desimone et al., 2002; Gamoran, 2003; Spillane & Thompson, 1997). A previous survey of Florida teachers reported that they spent 12 hours over 1.5 months on average for one cycle of lesson study (Akiba et al., 2019). Because of the time-intensive nature of lesson study, the provision of substitutes for planning meetings and research lessons during the regular school hours and extra stipend for meeting outside the regular school hours can serve as major financial incentives for teachers to engage in lesson study (Murata, 2011; Yoshida, 2012). Both SIG and Title II funding distributed to the district level may be used for continued funding for substitutes and teacher stipends.
Knowledge resources
Previous research identified knowledge resources to be a critical dimension of state reform capacity and states rely on a network with large numbers of external organizations to enhance this dimension of state capacity (Childs & Russell, 2017; Goertz et al., 2013; Massell et al., 2012; Russell et al., 2015). Knowledge resources encompass the human and social capital (Spillane & Thompson, 1997) that is needed to understand how to effectively engage in and facilitate PD innovations such as lesson study. Human capital consists of individuals’ commitment, dispositions, and knowledge, and social capital consists of relationships among individuals such as trust, shared goal, and a sense of responsibility for improvement, all of which promote knowledge diffusion and enhancement (Spillane & Thompson, 1997). Human and social capital interacts with each other to produce knowledge resources that the state and districts can generate and distribute to teachers.
At the state level, provision of high-quality trainings on lesson study and technical assistance by expert lesson study facilitators allow teachers to access knowledge resources to learn how to effectively engage in and facilitate lesson study. When lesson study is practiced more effectively, educators experience the benefits of lesson study for deepening teacher knowledge and improving instructional practice and student learning (Lewis & Perry, 2014, 2017; Perry & Lewis, 2009). As a result, they are more likely to see a value in sustaining lesson study.
Likewise, developing and providing high quality lesson study materials openly accessible by district and school leaders and teachers is important not only for learning the lesson study process and facilitation but also for meaningfully engaging in the study and plan phases of lesson study through research-based materials on standards, assessments, and instruction.
Hodge et al. (2016) examined the number and types of external resources for secondary English and Language Arts (ELA) used by 51 states based on the state websites, and identified an important role of states in providing access to external, high-quality curriculum and professional resources. Open-access, web-based lesson study resources could be especially useful for the districts and schools that lack lesson study expertise or resources.
At the district level, teacher training on lesson study, technical assistance with how to engage in lesson study at school sites, and provision of lesson study materials are all important, especially for teachers who do not have access to state-provided training or facilitators. District PD leaders, including instructional specialists and coaches, have the local knowledge that can be used to tailor training, offer technical assistance, and provide materials in ways that address teachers’ learning needs. In addition, continuous support of a lesson study coordinator for facilitating and supporting lesson study at the district level ensures oversight and continuation of lesson study practice across the district. This position may be assumed by a district instructional specialist, coach, or teacher on assignment who can continue to provide knowledge resources to teachers through training, facilitation, and materials on lesson study.
Finally, having a stable PD leadership may also be key to sustaining lesson study, as previous studies have documented the detrimental effects of leadership turnover on sustaining and scaling reform initiatives (Hargreaves & Fink, 2000; McLaughlin & Mitra, 2001). Having the same PD director who oversees the entire PD programs and who prioritizes lesson study as a central learning process for instructional improvement would likely sustain the practice of lesson study after the RTTT program ended.
System integration
Lesson study has potential to coherently address standards, assessments, and instruction by engaging teachers in a cycle of inquiry-based collaboration to examine standards and assessment results and experiment with standards-based instruction (Hiebert et al., 2002; Hiebert & Stigler, 2017; Lewis et al., 2006; Martone & Sireci, 2009). However, integrating this innovative approach to professional learning into the existing system with multiple competing initiatives is challenging, and the state and district leaders must address and overcome this challenge to sustain lesson study.
Despite keen awareness and a growing policy focus on the importance of system integration and coherence (Childs & Russell, 2017; Coburn et al., 2016; Fuhrman et al., 2007; Fullan & Quinn, 2016; Gottfried et al., 2011; Johnson et al., 2015), researchers have argued that new reform initiatives are often simply “added on” to existing fragmented organizations that respond to environmental pressures to meet various and often conflicting interests, instead of “replacing or reforming” them (Cohen et al., 2017). This reality is also present in the Florida RTTT program, in which multiple disconnected initiatives were implemented simultaneously, ranging from lesson study to teacher evaluation to technology to charter schools, reflecting the current policy climate focusing on both standards-based instructional improvement and rigorous accountability (Coburn et al., 2016).
To address this challenge, FDOE and district leaders can approach integrating lesson study into the system by communicating a coherent policy expectation and strategically adapting both lesson study and other system elements. First, state and district requirements are an important policy instrument that state and district leaders can use to communicate PD priorities (Knapp, 2003) and to establish coherence in PD (Firestone et al., 2005). FDOE and districts can communicate their priority and vision for using lesson study as a driver of instructional improvement across the state and districts by establishing a clear expectation for continuously engaging in lesson study after RTTT.
Second, state and district leaders can adapt lesson study and other system elements to achieve integration of lesson study into the instructional system. Previous studies identified adaptation of an innovation while maintaining the core principles as a natural part of scaling up across diverse contexts with different resources (Dede & Honan, 2005; McLaughlin & Mitra, 2001). At the state level, the state professional development standards and teacher evaluation standards can be modified by including lesson study as a model professional learning process and considering teacher participation and contribution in lesson study as part of teacher evaluation criteria. At the district level, consideration of teacher participation and contribution in lesson study in teacher evaluations and prioritizing lesson study over other existing PD programs may help integrate lesson study and support sustained lesson study practice. Local ownership is another factor identified as critical for integrating and promoting innovations (Coburn, 2003; Dede & Honan, 2005; McLaughlin & Mitra, 2001). Coburn (2003) argued that, to scale a reform initiative, reform ownership needs to shift from external to internal controlled by districts, schools, and teachers who will sustain, spread, and deepen reform principles themselves.
To investigate how Florida sustained lesson study after RTTT funding ended, we applied the three dimensions of fiscal resources, knowledge resources, and system integration as a framework to understand the relationship between state and district capacity and the sustainability of an RTTT- supported innovation. We first examined changes in the level of lesson study practice two years after the RTTT funding ended. Then we analyzed state RTTT and other policy documents to examine how the state approached sustaining lesson study along each dimension of capacity. Finally, we used both survey and interview data gathered from district PD leaders to identify the capacity differences between the districts that sustained lesson study and the districts that phased out lesson study. We determined that both quantitative survey data and qualitative interview data are necessary to fully understand the concept of system integration—the third and possibly most important dimension of district capacity. Therefore, quantifiable aspects of system integration such as the existence of a requirement, inclusion of lesson study in teacher evaluation, and number of PD programs were examined through a survey of district PD leaders, and qualitative aspects such as nuanced meanings of expectation, adaptation, and local ownership were explored through interviews with select PD leaders from districts that sustained and phased out lesson study.
Methods
Data Collection
To identify the extent to which lesson study was sustained after the RTTT program and examine the state and district capacity dimensions important for sustaining lesson study, we used three sets of data: (1) federal, state and district policy documents, including RTTT documents and other policy documents, (2) longitudinal survey data collected from district PD leaders in 34 districts in 2014, 2015 and 2016, and (3) interviews with district PD leaders in 10 districts—four sustaining districts and six phasing-out districts conducted from 2014 to 2016.
Longitudinal district survey data were used to address the first research question on the degree to which lesson study was sustained in Florida, and federal and state policy documents were used to address the second research question on the state’s approaches to sustaining lesson study. To address the last question on district capacity associated with sustained practice of lesson study, we used the longitudinal district survey data and interview data, supplemented by the district policy documents to understand the context on district policy and practice of lesson study.
Policy documents
Three types of policy documents were gathered either through web searches or direct communications with FDOE and district PD leaders: (1) USDOE RTTT documents—request for proposal and executive summary for RTTT program (USDOE, 2009a, 2009b) and USDOE annual evaluation reports on Florida (USDOE, 2012, 2013, 2014, 2015), (2) FDOE RTTT documents—RTTT proposal (FDOE, 2010a, 2010b), summary reports on accomplishments and information sharing (FDOE, 2015a, 2015b), process documents on lesson study (e.g., meeting minutes, PPT files, resource documents, funded external subcontractors) produced from 2010 through 2016 including a lesson study guide (Haithcock, 2010), and other related state documents on teacher policy including statutes, administrative codes, rules and regulations, and standards and guidelines on PD and teacher evaluation, (3) district RTTT documents—34 district RTTT proposals (i.e., final scope of work) and district process documents on lesson study (e.g., meeting minutes, PPT files, resource documents) produced from 2010 to 2016. The first two types of federal and state documents were used for addressing the second research question of examining state approaches to sustain lesson study, and the district RTTT documents were used to understand the RTTT district context and district-level lesson study practice to contextualize and supplement the findings from the district survey and interview data.
Longitudinal district survey
A statewide district survey named Lesson Study District Survey was conducted via the Qualtrics online survey tool with all 68 regular school districts 2 in Florida between May and August in 2014, 2015, and 2016 to examine the level of lesson study practice and district approaches to lesson study. In each year, we followed three stages: (1) identification of PD leaders in charge of or most knowledgeable about lesson study practice through web searches, emails and phone calls to district offices, (2) survey administration with these PD leaders, and (3) multiple follow-ups with non-respondents through emails and phone calls.
In all three years, the survey first defined lesson study as “a continuous PD process that involves a group of 3 to 6 teachers collectively engaging in four stages: studying and goal setting, lesson planning, research lesson, and debriefing session,” which is consistent with the FDOE’s explanation of lesson study in initial trainings funded by RTTT. District documents on lesson study were also examined to verify that districts’ lesson study practice was consistent with this definition.
The survey included questions on five major topics: (1) number of schools that practiced lesson study, required or voluntary (as measures of sustained lesson study practice), (2) use of other federal funding and funding provision for substitutes and teacher stipends, (3) knowledge resources measured by lesson study training, technical assistance, materials, lesson study coordinator, and PD coordinator stability, (4) system integration measured by district expectation on lesson study (i.e., the type of schools required to practice lesson study) and adaptation (i.e., consideration of lesson study in teacher evaluation and number of professional development programs). All survey measures are listed in online Appendix A. Survey participants received a link to an online gift card of a major retailer upon completion. 3
A total of 45 districts participated in all three waves of the survey (66.2% response rate), and the study focused on 34 districts after excluding five districts that never practiced lesson study and six districts that started practicing lesson study after the RTTT program ended. A comparison of district background characteristics in poverty level (percentage receiving free or reduced-priced lunch), diversity level (percentage of ethnic minority students), student enrollment, and achievement level in ELA and math showed no statistically significant difference between the 34 participating districts and the other remaining 34 districts in Florida.
Interviews
Semi-structured interviews were conducted from 2014 to 2016 with the PD leaders who participated in the longitudinal district survey. We used maximum variation sampling to invite a total of 14 districts that varied in terms of student enrollment, poverty level, diversity level, and achievement level. Leaders from 12 districts agreed to participate in the interview. This study focused on interviews with PD leaders from 10 districts—four districts that sustained lesson study and six districts that phased out lesson study (four districts that decreased and two districts that stopped practicing lesson study) after excluding two districts that started lesson study after RTTT ended. These districts represent a diverse group of Florida districts ranging from less than 4,000 to over 160,000 students with a mean of 50,368 students. The poverty level ranged from 31% to 54% of students receiving free or reduced-price lunch (FRL) with a mean of 43%, and the diversity level ranged from 19% to 70% ethnic minority students with a mean of 46%.
A total of 14 district leaders in 10 districts were interviewed for 1-2 hours each either face to face or online (i.e., Skype, FaceTime, or Google Hangout). These leaders, selected for their leadership roles in lesson study, held various titles including Director of PD, Supervisor of Leadership Development, Director of Teaching and Learning, Chief Academic Officer, and instructional specialist. All interviewees were also asked to share district documents related to lesson study. Researchers recorded all interviews which were transcribed verbatim. Interview questions were categorized into five topics: (1) influence of the RTTT program and district policy, (2) lesson study approaches and facilitation (e.g., facilitator, schedule, materials), (3) funding, (4) sustainability and changes in lesson study policy and practice, and (5) coherence and integration (see online Appendix B for the interview protocol).
Analysis
To address the first question on the degree to which lesson study was sustained after the RTTT program, PD leaders were asked in a longitudinal survey to report how many schools in total, including both required and volunteered, practiced lesson study in their districts each year, based on teacher reporting of lesson study practice in the district PD reporting system. 4 From the survey responses, we computed two measures: (1) changes from 2014 to 2016 in the total number and percentage of schools that practiced lesson study as a statewide measure, and (2) changes from 2014 to 2016 in the percentage of schools within districts that practiced lesson study as a district-level measure. In the district-level measure, three types of changes were identified—sustained (12 districts), decreased (10 districts), and stopped (12 districts). The districts that decreased and stopped were combined as phasing-out districts (N = 22) and compared with sustaining districts (N = 12) in the analysis of district capacity differences.
To address the second question on state capacity to sustain lesson study, a policy document analysis was conducted by first thoroughly reviewing each document, and coding relevant sections aligned with three dimensions of state capacity: (1) fiscal resources, (2) knowledge resources, and (3) system integration. We first coded broadly using these three dimensions, followed by more specific coding of each dimension using key terms that emerged through multiple readings of each document. These specific codes were compared with the seven measures of state capacity drawn from the PD and lesson study literature listed in Table 1. All the specific codes were then categorized into the seven measures, and we developed a summary of coded sections as supporting data for each measure. Finally, state approaches to sustaining lesson study were described along three dimensions based on the themes that emerged from a comprehensive review and coding of all documents. In describing the state approaches, we differentiated their plan from evidence of actions taken by the state we could obtain from the available documents.
To address our last question on how three dimensions of district capacity differentiate the sustaining districts from phasing-out districts, we analyzed the longitudinal survey data from the 34 district PD leaders and interview data with 10 districts. All the survey measures of three dimensions of district capacity to sustain lesson study are presented in online Appendix A along with the survey questions, years when the data were gathered, and means or percentages of district responses. All variables are available from the 2014 survey, but only select variables were gathered in 2015 and 2016. Longitudinal survey data from district PD leaders were quantitively analyzed using t-tests for mean difference and chi-square tests for percentage difference in three dimensions of district capacity between the districts that sustained lesson study and the districts that phased out lesson study. Due to the small number of districts (N=34), in addition to the statistical significance level, effect sizes were computed using Cox index for percentage difference and Hedges’ g index for mean difference with an effect size of 0.25 or larger or −0.25 or smaller to be considered substantively important (What Works Clearinghouse, 2017).
Because the concept of system integration has important qualitative aspects such as nuanced meanings of expectation, adaptation, and local ownership, semi-structured interviews were analyzed in three stages through systematic, iterative, and constant comparative coding methods (Corbin & Strauss, 2015; Miles et al., 2014). In the first stage, both authors reviewed four interviews from two sustaining and two phasing-out districts and openly coded the interviews and developed a memo for each district independently. The open codes and memos were then reviewed jointly to discuss the ideas and themes identified in interviews, resulting in an initial set of 71 codes.
In the second stage, we coded the same four transcripts using the identified codes. After two rounds of coding and refining, we developed 27 codes grouped into six categories—lesson study changes, reasons for changes, lesson study perspectives and approaches, implementation challenges, leadership, and support. Independent coding of the four interview transcripts resulted in substantial agreement (κ = 0.90) and the two authors discussed all interview segments of discrepancies until both reached agreement on the most suitable codes. In the last stage of coding, the first author coded the remaining six interview transcripts, and discussed with the second author the analytical memos of all 10 districts and emerging themes and findings. These findings were then compared with state and district policy and process documents and the survey responses for triangulation.
Analysis of all data was guided by the pragmatic goal of understanding state and district approaches and capacity for sustaining lesson study (Johnson & Onwuegbuzie, 2004). Aligned with this goal, qualitative analysis aimed to both further explain and inform quantitative findings. In addition to providing further details about district approaches to sustainability, the interviews also provided evidence of the rationales, values, and meanings behind district approaches (Weick, 1995). Earlier stages of initial coding sought to understand district approaches with minimal preconceived notions of what themes might emerge, while later analysis sought to intentionally engage with prior research and our own findings through an iterative process of analysis. Throughout the analysis, the interpretations of both authors were likely influenced by their previous analysis of lesson study implementation around the state and their familiarity with prior research on policy implementation and sustainability. However, these limits to objectivity were also leveraged as strengths in the analysis as they provided an opportunity to compare emergent findings with prior research and experiences within the state context.
Results
Lesson Study Sustainability after RTTT
To examine the level of lesson study implementation at the state and district levels in the last year of RTTT (2014) and one and two years after RTTT funding ended, we present two sets of data in Figure 1: (1) changes in the total number and percentage of Florida K-12 schools that practiced lesson study, and (2) changes in the percentage of K-12 schools within districts that practiced lesson study with three patterns of changes.
Post-RTTT lesson study sustainability.
Of a total of 1,503 schools across 34 districts that participated in our longitudinal survey, 434 schools (29%) were reported to have practiced lesson study in 2014 during the final year of RTTT. This number decreased to 296 school (20%) in 2015 and 250 schools (17%) in 2016.
Table 2 presents the percentage of schools within each district that practiced lesson study in 2014, 2015, and 2016 in each of 34 districts along with district background characteristics—district size, percentage of students receiving free or reduced-price lunch, and percentage of ethnic minority students. Based on this data, we identified three different patterns of changes:
Level of Lesson Study Implementation in 34 Districts.
The district size is reported in range to avoid revealing district identity, but the mean district size in each group was computed based on the actual district size.
The districts where the level of lesson study practice dropped more than 50% from 2014 to 2016 were categorized as “Decreased”.
(1) sustained (12 districts), (2) decreased (10 districts), and (3) stopped (12 districts) as shown in Figure 1. Twelve districts sustained the level of lesson study practice from 2014 to 2016. The level of lesson study practice varied from 10% to 100% (of schools practicing lesson study) across these districts. On average, these districts continued to have about half of their schools practice lesson study.
Ten other districts were practicing lesson study at about the same level as the districts that sustained lesson study in 2014 (50.7% of schools on average), but the level of practice dramatically decreased with more than 50% reduction in all of these districts after RTTT funding ended. In the remaining 12 districts, about one quarter of schools practiced lesson study on average in 2014, yet only four districts practiced lesson study in 2015. By 2016, all of these districts stopped practicing lesson study altogether.
In the next section, we first examined the state RTTT and other policy documents to identify state approaches to sustaining lesson study based on the three dimensions of state capacity—fiscal resources, knowledge resources, and system integration. We explored how they may explain the overall decline in the practice of lesson study across the state. After that, we examined district capacity for sustaining lesson study based on the same three dimensions using both survey and interview data. In these analyses, we combined districts that decreased and stopped lesson study practice as “phasing-out” districts, as they are on the same trajectory to ending lesson study practice and contrast these districts with “sustaining” districts. To fully understand nuanced meaning of expectation, adaptation, and local ownership, we used interview data to explore how sustaining districts approached system integration through these complex concepts in contrast to phasing-out districts.
State Capacity to Sustain Lesson Study
Fiscal resources
An analysis of FDOE RTTT documents showed that the state allocated the responsibility for sustaining RTTT-funded projects to school districts. References to sustaining reforms in the RTTT proposal focused on the role of districts, such as “Districts will be encouraged to use flow-through funding from other federal formula programs. . . in ensuring that successful reforms can be sustained after the end of the grant period” (FDOE, 2010a, p. 49), and “Through a process of redirection and reprioritization of existing funds, districts can ensure the sustainability of reforms put into place during the four-year RTTT period” (FDOE, 2010a, p. 50). In the 2015 FDOE final report (FDOE, 2015a), there was no mention of continued use of federal or state funding by FDOE to sustain lesson study.
Knowledge resources
Our analysis of policy documents showed that FDOE mobilized knowledge resources by partnering with subcontractors for training and resource development, hiring reading and STEM coordinators to work in PLA schools, and partnering with a higher education institution to create a lesson study resource kit from 2010 to 2014.
First, our previous study based on interviews of state leaders and PD providers conducted in 2013 showed that FDOE provided RTTT funding to three higher education institutions, one regional educational lab, and one nonprofit PD provider to provide workshops and trainings on lesson study across the state (Akiba & Wilkinson, 2016). In addition, RTTT documents showed that FDOE offered a Summer Academy focused on lesson study to school leaders in Differentiated Accountability (DA) schools (FDOE, 2015a) and hired 40 reading coordinators and 20 STEM coordinators whose responsibilities were implementation of RTTT-funded initiatives including lesson study at PLA schools (FDOE, 2010a, p. 216).
Furthermore, the FDOE provided a large grant to a higher education institution in Florida to develop open-access lesson study resource kits with video tutorials on how to engage in each stage of a lesson study cycle and how to use the online platform to record lesson study activities with links to high quality lessons, curriculum and instructional materials, assessment tools, and research articles in math and science aligned with the Mathematics Florida Standards and the Next Generation Sunshine State Standards. The kits were shared by FDOE in the USDOE’s initiative for sharing RTTT winning states’ promising practices and lessons learned with other states attempting to implement similar education reform initiatives (FDOE, 2015b).
Despite the active engagement of FDOE in generating knowledge resources during the RTTT funding period, there was no mention of the continued offering of lesson study trainings or technical assistance from the state or any partner organizations after these contracts with external subcontractors ended in 2014. The only knowledge resources that were made continuously available was the web-based lesson study resource kits. While such open-access resources are important for learning the basics of lesson study and accessing relevant materials to guide the process, it is likely that the web content become outdated without an extended contract or a new responsibility for overseeing and updating the website to keep the site current and relevant.
System integration
The state capacity regarding system integration was examined based on two concepts—expectation and adaptation. Although the state set a clear goal for all participating LEAs to fully implement lesson study (FDOE, 2010a), we found no evidence in the available state PD documents and materials produced after 2014 that this expectation continued after the RTTT program.
In the initial RTTT proposal, FDOE stated its plan to support “LEAs’ professional development systems that: (1) provide sustained, job-embedded delivery models such as lesson study, (2) support access to and use of student learning data, and (3) are informed by appraisal results” (FDOE, 2010a, p. 181). However, the actual PD standards implemented since 2010 and during the RTTT funding period did not mention lesson study beyond a reference to learning communities “whose members use a cycle of continuous improvement to achieve goals that align with individual, school, and district goals for student achievement” (FDOE, 2010c, p. 1).
Regarding alignment with teacher evaluation, FDOE stated in the RTTT proposal a goal to “Implement Individual Professional Development Plans for teachers based on analysis of student performance data and results of prior evaluations” (FDOE, 2010a, p. 183). However, the examination of state-level teacher evaluation documents, including the state rules and regulations, guidelines, and web resources, revealed no reference to lesson study either as an evaluation criterion (e.g., teacher participation and contribution to lesson study) or a recommended PD to engage teachers needing improvement in instruction.
In summary, across all three dimensions of state capacity to sustain lesson study, FDOE assumed a limited role in sustaining lesson study. We found no evidence of federal or state funding allocation for lesson study, continued training, or integration of lesson study into the PD standards or teacher evaluation guidelines after the RTTT program ended. This passive role of the state is consistent with a Florida statute (Section 1012.98, F.S.) that designates the responsibility for providing a high-quality professional development system to school districts. Without any expectation, funding or knowledge resources from the state to continue lesson study beyond 2014, the overall decline in the number of schools that practiced lesson study and the major variation across districts are not surprising, since the decision to sustain lesson study was left to each district.
District Capacity to Sustain Lesson Study
Our data showed that 12 out of 34 districts sustained lesson study two years after RTTT ended while the remaining 22 districts phased out lesson study. What differentiate these districts based on the three dimensions of district capacity—fiscal resources, knowledge resources, and system integration?
Before examining the relationship between district capacity and lesson study sustainability, we first examined how demographic characteristics and achievement level differ between sustaining districts and phasing-out districts. Table 3 shows t-test and chi-square test results along with effect sizes for mean or percentage differences between these districts in five sets of factors: (1) percentage of students receiving free or reduced-price lunch (FRL), (2) percentage of ethnic minority students, (3) district student enrollment, (4) mathematics achievement in 2014, 2015, and 2016, and (5) English and Language Arts (ELA) achievement in 2014, 2015 and 2016. Math and ELA achievement levels were measured by the percentage of students who achieved the proficient level or above.
Difference in District Background Characteristics between Sustaining and Phasing-Out Districts (N = 34).
Combination of districts that decreased and stopped practicing lesson study.
Hedges’ g index was computed for mean differences based on g = ῳt/
Average values of 3 years were computed for these three background variables.
Percentage of students who achieved proficient level or above. In 2014, the FDOE implemented the Florida Comprehensive Assessment Test (FCAT) 2.0. In 2015 and 2016, FDOE implemented Florida Standards Assessments (FSA) that are aligned with the new Florida Standards implemented from the 2014–2015 academic year.
p < .10. *p < .05. **Effective size >.25 or <−.25.
An effect size of −0.28 shows that sustaining districts are wealthier than phasing-out districts. There were no statistical differences in the percentage of ethnic minority students or district enrollment. However, the sustaining districts achieved higher in math in 2014 and ELA in 2015 and 2016. These findings show that sustaining districts are more likely to be wealthier and high achieving than phasing-out districts. With these differences in district background characteristics in mind, we first used survey data to examine how measures of fiscal resources, knowledge resources, and system integration differed between sustaining and phasing-out districts as presented in Tables 4 and 5.
Relationship between District Fiscal and Knowledge Resources and Lesson Study (LS) Sustainability (N = 34).
Combination of districts that decreased and stopped practicing lesson study.
Cox index was compted for percentage differences based on dcox = ῳ*LOR/1.65 where Log odds ratio (LOR) = ln(Odds1)−ln(Odds2) (What Works Clearinghouse, 2017). An effect size of .25 or larger or smaller than −.25 is considered to be substantively important (What Works Clearinghouse, 2017).
Percentage of districts that reported using School Improvement Grant (SIG) fund or Title II fund for lesson study.
Percentage of districts that provided funding for substitutes or teacher payment.
Percentage of districts that provided workshop or training on lesson study for teachers. This data was gathered only in 2014.
Percentage of districts that offered assistance or support to schools and teachers on how to practice lesson study. This data was gathered only in 2014.
Percentage of districts that provided web-based or paper materials and resources on lesson study. This data was gathered only in 2014.
Percentage of districts with a designated position for lesson study.
Percentage of districts with the same PD director from the prior academic year.
p < .10, *p < .05, ** Effective size >.25 or <−.25.
Relationship between System Integration and Lesson Study Sustainability (N = 34).
Combination of districts that decreased and stopped practicing lesson study.
Hedges’ g index was computed for mean differences based on g = ῳt/
Percentage of districts with at least one PLA school that was required by FDOE to implement lesson study from 2010–2014.
Lesson study (LS) requirement was coded as 1 = no school required, 2 = some schools are required, and 3 = all schools are required.
Percentage of districts where lesson study was part of teacher evaluation. This data was gathered only in 2014.
Mean number of PD programs implemented by the district each year as reported by the PD director.
p < .10.*p < .05. ** Effective size >.25 or <−.25.
Fiscal resources
Because FDOE encouraged districts to redirect and reprioritize the other federal formula grants, our survey asked district PD leaders if they used SIG funds or Title II funds—two types of federal funds identified in our prior study that some districts used for lesson study (Akiba & Wilkinson, 2016). We also asked whether funding was provided for substitutes and teacher stipends, which were identified in previous studies as critical for teachers to engage in the time-intensive nature of lesson study (Murata, 2011; Yoshida, 2012).
Although about one fourth of the districts reported using SIG funds and 50% of the districts used Title II funds for lesson study, chi-square tests of percentage differences in Table 4 showed no substantial differences in the use of SIG or Title II funds between sustaining and phasing-out districts. However, the provision of substitute funding in 2016 was statistically different with an effect size of 0.37. 66.7% of sustaining districts provided substitute funding compared to only 31.8% of phasing-out districts. Although a higher percentage of sustaining districts provided teacher stipends than phasing-out districts, the percentage differences were not substantial with small effect sizes. The data also showed that provision of teacher stipend is also less common than substitute funding. These findings seem to indicate that (a) how the funding is actually used is more important than whether or not the district used other existing federal funding and (b) provision of substitute funding may be important for sustaining lesson study. 5
Knowledge resources
As shown in the bottom half of Table 4, we examined five indicators of knowledge resources as a dimension of district capacity to sustain lesson study: (1) teacher training on lesson study, (2) technical assistance or support with lesson study implementation, (3) lesson study web-based or paper materials, (4) a designated lesson study coordinator at the district level, and (5) PD director stability.
Despite the importance of knowledge resources identified in the reform capacity literature, only one indicator—PD director stability in 2016 had a large effect size of .38. 75% of sustaining districts had the same PD director in the 2014-15 academic year and the 2015-16 academic year, compared to only 40.9% of phasing-out districts. High turnover in leadership has been identified as detrimental in sustaining reform initiatives (Hargreaves & Fink, 2000; McLaughlin & Mitra, 2001), and stable PD leadership seems to be important for district capacity to sustain lesson study.
Although between 40 to 70% of districts provided other knowledge resources—teacher training, technical assistance, materials, and a designated lesson study coordinator in 2014, there were no significant differences between sustaining districts and phasing-out districts in these factors. This may indicate that simply providing these knowledge resources may not contribute to sustaining lesson study, and there is a need to further examine district leaders’ approaches to training and technical assistance as well as the nature of materials provided to teachers.
System integration
Because the concept of system integration is complex with both quantitative and qualitative aspects, we conducted both survey and interviews. First, using survey data, we examined two quantifiable aspects—expectation and adaptation as presented in Table 5. For expectation, we used two measures—whether the district was required by FDOE to implement lesson study from 2010 to 2014 because it had at least one PLA, and the district’s own lesson study requirement in 2014, 2015, and 2016.
The chi-square test result in Table 5 shows that 41.7% of sustaining districts were required by the state to implement lesson study in PLA schools, compared to 31.8% of phasing-out districts. However, this difference was not significant with a small effect size of .11. Our previous survey of district PD directors in 2013 showed that, even though the state required only 71 PLA schools in 25 districts to implement lesson study (FDOE, n.d.), 34 districts required a total of 611 schools (both PLA schools and other schools) to practice lesson study (Akiba et al., 2016). Therefore, districts played an important role in expanding the lesson study requirement and communicating the district PD priority of lesson study. This suggests the state requirement from 2010 to 2014 seems to have played a limited role regarding whether the districts sustained lesson study or not.
The t-test result on district requirement showed that sustaining districts were significantly more likely to require their schools to practice lesson study than phasing-out districts. Although the overall level of requirement gradually decreased from 2014 to 2016 in both types of districts, a larger number of sustaining districts continued to require lesson study practice compared to phasing-out districts, and the differences were significant with the large effect sizes of .33 in 2014 and 2015, and .88 in 2016.
To quantitatively measure evidence of adaptation for system integration, we used two measures—integration of lesson study into teacher evaluation, and the number of PD programs as an indicator of district focus on lesson study. Teacher evaluation was among the accountability priorities emphasized by the RTTT program, and FDOE established a stringent policy for teacher evaluation by considering value-added modeling (VAM) scores for 50% of the total evaluation scores to determine teacher ratings, which are used to identify professional development and improvement plans and inform decisions on contract renewal or termination (Florida Senate, 2011). 6
Teacher evaluation incorporating VAM scores was fully implemented for the first time in the 2013-14 academic year in Florida. Considering the major impact on teachers, integrating teachers’ participation or contribution to lesson study as part of evaluation criteria likely encourage teachers to continue to engage in lesson study. In 2014, 41.7% of sustaining districts reported integrating lesson study as part of teacher evaluation, compared to only 18.2% of phasing-out districts. The difference was significant with the effect size of .30. Districts’ integration of lesson study in teacher evaluation seems to have allowed them to embed lesson study as a model PD process in the district system of instructional improvement.
The number of PD programs that the district implemented was examined as the second indicator of adaptation for system integration. Previous research has shown that many districts take a “cafeteria” approach to professional development by offering a large number of PD programs to meet the various learning needs of teachers without a clear coordination or integration of teacher learning opportunities (Elmore, 2004; Little, 1993, 1999; Spillane, 2002). If a district continues to offer many disconnected PD programs and simply added lesson study to the list of the existing programs, it is not likely that lesson study is integrated as a core part of the instructional improvement system. Therefore, we asked PD leaders to list the PD programs they implemented, including lesson study, in 2014, 2015, and 2016. The effect sizes of .31 and .37 show that sustaining districts offered fewer PD programs than phasing-out districts in 2014 and 2016. The average number of PD implemented in sustaining districts were 3.9 in 2014 and 3.5 in 2016, compared to 5.1 and 4.5 in phasing-out districts respectively. This means that lesson study was one of 3 to 4 PD programs implemented in the districts that successfully sustained lesson study two years after the RTTT program ended.
In summary, most quantitative measures of system integration as a dimension of district capacity were associated with whether the district could sustain lesson study or not, indicating the importance of system integration for sustainability. A district requirement to communicate the continued priority for lesson study, consideration of lesson study in teacher evaluations, and a smaller number of PD programs characterized the districts that sustained lesson study compared to the other districts that phased out lesson study.
Exploring PD Leaders’ Approaches to Integrate Lesson Study through Interviews
To fully investigate district approaches to system integration that cannot be measured quantitatively in a survey such as nuanced meanings of expectation, adaptation, and local ownership, we interviewed PD leaders from ten districts (four sustaining and six phasing-out districts) from 2014 to 2016 after the RTTT ended. All of the PD leaders reported that they were introduced to lesson study as a result of FDOE’s RTTT award in 2010. This meant they needed to quickly learn how to practice lesson study, and most did so through attending various FDOE-sponsored trainings provided by nonsystem actors (i.e., higher education institutions and non-profit organizations) that relied on pre-packaged lesson study materials during the first few years (Akiba & Wilkinson, 2016). However, all ten PD leaders we interviewed described adapting the lesson study approach introduced in these initial trainings to fit the needs of their districts. Our analysis revealed that the nature of this adaptation was distinctly different between sustaining districts and phasing-out districts.
Sustaining districts adapted lesson study to fit within the stable, core elements of the district instructional system, as they saw lesson study as a flexible process that can be aligned with district priorities. Sustaining districts also communicated a district expectation to practice lesson study but supported school and teacher ownership of the process, which seems to have enabled lesson study to be embedded into various school contexts. On the other hand, phasing-out districts saw lesson study as a rigorous and powerful PD model that needs to be implemented with fidelity. These districts adapted lesson study into a district model with a specific duration, protocols and templates, and ensured the involvement of district leaders in lesson study facilitation. These districts were ultimately not able to meet the high and more exact expectations they set for lesson study, leading to a decreased level of lesson study practice.
Sustaining districts: adapting lesson study to the district instructional system
Embedding lesson study in stable core system elements
In sustaining districts, PD leaders saw a natural fit between lesson study and existing processes in the instructional system. This enabled these districts to successfully embed lesson study into other initiatives reflecting district priorities. In Stonehaven,
7
for example, lesson study was embedded into PLCs that are well established in the district. District leader Ms. Aviles explained: The way it works—the way the process works here—is PLCs are supposed to look at areas of student achievement that need to be supported. So, usually when they decided on an area that needs support, then they do their research, but usually it comes down to classroom practices that need to be changed. So that’s where lesson study comes in, because they located a problem; they researched that problem; they looked through the resources available to them, and then they take it to the classroom level and work on perfecting materials, perfecting the lesson that would support the area of academic concern.
Lesson study is also integrated into teacher evaluation scores in Stonehaven. Teacher groups earn points by sharing their learning with the entire faculty, and individual teachers share and discuss their lesson study experience in end-of-year conferences with the principal.
Likewise, in Ettawa, PD leaders embedded lesson study into the “peer input” component of the teacher evaluation system, which counts as 20% of their total evaluation points. After completing a lesson study cycle, each participating teacher rates team effort and performance on four categories: lesson plan and instruction, observation and debriefing, reflection, and areas for improvement. Ms. Willis in Ettawa explained that almost all teachers receive full points by participating in lesson study, which served as a strong incentive to engage in lesson study. These approaches to integrate lesson study as part of teacher evaluation criteria is consistent with the survey finding that 41.7% of the 12 sustaining districts considered lesson study as part of teacher evaluation.
In Bayport, Ms. Carver-Sparks explained that because of a district focus on technology, lesson study was embedded into a new program called Tech Trailblazers as “part of a great plan” to ensure that the district is prepared for technology-enhanced learning. Because the district did not have enough funding to supply substitutes while teachers engaged in lesson study, school-based coaches without teaching responsibilities from each school were invited to form a lesson study group and to research and teach technology-supported lessons. The end-product of these groups was a catalog of technology-supported lessons shared with teachers.
Communicating district expectation
The examples above show PD leaders interpreting lesson study as a flexible and adaptable process that can be integrated into existing processes or initiatives as a core part of the district’s instructional system. This approach was supported by a district expectation for lesson study as a practice that aligns with district approaches to teacher learning. For example, Ms. Willis in Ettawa described her district’s expectation as follows: We have an expectation of job-embedded PD, and I think we need to keep that expectation in place. You know, this is one of the few things that we do in our districts—meaning that our schools, they can do a lot of what they want. They can opt in, and a lot of times opt out, of things. And so, this is one of the things that you don’t have an opt-in, opt-out model. As a district, we expect you to do something like lesson study. I think that’s a major benefit, an expectation that this is the way that professional learning needs to happen and occur at the school level.
Ms. Richardson in Southden also explained: I think they’re not specific defined expectations, but rather a general expectation that lesson study is a great way to bring teachers together and reflect on their practice. So it’s not like “you need to do two cycles this year” or “we’re going to check off a list that it’s done.”
This is consistent with the survey data result reported above that sustaining districts were more likely to expect teachers to practice lesson study through a district requirement. Interview data suggests that this expectation was communicated with flexibility in lesson study practice at the school level. Although they expected schools to practice lesson study, PD leaders in all four sustaining districts allowed schools and teachers to set the schedule, approach, and facilitation of lesson study.
Supporting school and teacher ownership
Ms. Carver-Sparks explained that school-based coaches in the Tech Trailblazers program in Bayport are given autonomy as to how they approach lesson study. When asked about whether the coaches meet on a weekly basis, she responded, “they’re going to be in their schools, so we’re encouraging them, but we’re going to see how much time they need for it and see how they’re going to do it. We’re not going to set parameters for that.”
To our question of who facilitates lesson study, Ms. Aviles in Stonehaven explained that a designated teacher in charge of PLCs facilitates, and any teacher can facilitate because all district teachers have been trained in lesson study by 2013. When asked about the role of coaches, she said, “our process is that coaches are there to answer teachers, and to be a resource for when teachers need resources.” Such approaches view the district’s role as supporting lesson study practice in response to the needs of those practicing it.
These responses suggest that, within the broader expectation to practice lesson study, sustaining districts supported school leaders’ and teachers’ autonomy and ownership of the lesson study process. This finding could indicate that, if teachers have already embedded lesson study through adaptation and ownership by the end of the RTTT funding period, provision of knowledge resources, especially training or technical assistance, may not be as critical as it is during the beginning phase of lesson study implementation. Sustaining districts embedded lesson study into the relatively stable, core system elements such as PLCs, teacher evaluation, and technology as a coherent part of the instructional system. The further integration of lesson study into various school contexts by promoting school and teacher ownership in specific approaches to lesson study seems to have enabled the sustained practice of lesson study even after the RTTT funding ended.
Phasing-out districts: Adapting lesson study as a district model to ensure fidelity
PD leaders in phasing-out districts described lesson study as a rigorous and powerful PD model, and they were as committed to promoting lesson study as leaders in sustaining districts. These leaders also adapted lesson study to fit with district needs, but their adaptations were often characterized by district-level alterations to the format and structure designed to ensure fidelity of implementation.
Specifying duration and developing protocols and templates
One common approach described by phasing-out districts was to set a specific, and relatively short, duration for lesson study. Rawlings developed a three-day model; Sawgrass and Springland developed a two-day model, and Groveland developed a one-day model of lesson study. Another common approach described by PD leaders was to utilize specific protocols, such as randomly selecting the instructor and classroom for the research lesson, as explained by Ms. Craft at Springland: There was the need to be accountable, and we found very quickly that if you choose the teacher who is going to teach the research lesson up front, then that teacher owns the lesson study. Everybody else just sits back like, “Whew. What do you want to do? We’ll do whatever you want us to do!” And we were trying to avoid that.
Implementing the random selection protocol gave PD leaders a structured way to ensure teachers’ engagement—based on concern that they might be selected for the research lesson. Ms. Craft described this protocol as “the accountability piece,” and Ms. Sharpe from Sawgrass lamented that when they did not require the protocol, teachers were “just back on their phones” during meetings.
Leaders from phasing-out districts also described developing detailed templates for facilitating lesson study. PD leaders in Groveland described one template: This template is what we use when we go to demos. It is how we do business. Everyone learns through this template the same, and this is what they send us for evidence when they are doing it on their own so that we can give them feedback.
The PD leaders’ decision to set a duration and use structured protocols and templates seems to suggest a behaviorist view of teacher learning that assumes teachers need to engage in structured activities in order to learn, instead of a view that emphasizes the situated and cognitive nature of teacher learning (Putnam & Borko, 2000; Spillane, 2002; Thompson & Zeuli, 1999). This tightly structured approach also allowed districts to closely monitor the lesson study process. While it may be challenging to supervise lesson study practice if schools or teacher groups set the schedule and approaches, adaptation of lesson study into a specific district model allowed PD leaders to ensure implementation fidelity.
District-led facilitation for maintaining quality
While acknowledging the need for teacher facilitators, PD leaders in phasing-out districts emphasized the difficulty of facilitation and believed only a small number of teachers had the capacity to facilitate. Ms Castillo in Sawgrass explained: You have to have an understanding of lesson study, deep knowledge on the framework, strategy, Common Core, thinking maps, everything, and we really are the people who get that, because we’re here at district, curriculum and instruction, PD, so a teacher leader can be developed to that point, but I would say we have one person now that I would feel strong about.
Ms. Kassem in Rawlings also explained: We were committed that lesson study would be a district model that people would not morph, and therefore, bastardize the process. You know, so we have been very direct in what our expectations are, and that you don’t go off and do it this way or do it that way, our people must be involved.
These PD leaders saw themselves as gate keepers to ensure that lesson study is led by qualified facilitators. This type of intensive district involvement in lesson study facilitation posed a major challenge, since these districts lacked the capacity to train enough teachers to meet their standards or facilitate numerous lesson studies themselves. About her district’s approach to facilitation, Ms. Cole in Sawgrass explained, “If they all wanted to do it, we couldn’t have met their demands.” Similarly, Ms. Kassem in Rawlings explained “We just can’t respond to anybody. It has to be strategic because we only have two people.”
These leaders were keenly aware of the limited capacity of PD leaders to continue to drive the lesson study process in a large number of schools. However, they felt strongly about the importance of implementing with fidelity a district model of lesson study characterized by high-quality facilitation and specific format and structure. As a result, as district priorities changed and RTTT funding ran out, their time was allocated to other initiatives, which seems to have led to phasing out lesson study.
Discussion and Conclusions
The USDOE invested a significant amount of RTTT funding to promote state-led innovation with a goal of improving student outcomes. Because improvement of student outcome is a long-term process that involves many mediating factors and conditions, sustained implementation of RTTT-funded innovation is critical. Focusing on lesson study as a RTTT-funded innovative PD initiative promoted statewide in Florida, our data showed the major difficulty of sustaining lesson study after funding is withdrawn. The total number of schools that practiced lesson study decreased from 434 (29%) in 2014 to 250 (17%) in 2016, and only 12 out of 34 districts were able to sustain the same level of lesson study practice two years after the RTTT funding ended.
This overall decline in lesson study implementation may be explained by the limited role of FDOE in sustaining lesson study. Our analysis of policy documents produced before, during, and after RTTT revealed that, despite its initial intention to sustain RTTT-funded initiatives, FDOE had limited capacity in terms of fiscal resources, knowledge resources, and system integration to sustain lesson study. This capacity limitation is consistent with findings from previous studies (Kober & Rentner, 2011; McGuinn, 2012; Smylie et al., 2004; Superfine et al., 2012).
The responsibility of continued use of other federal funding such as Title II and SIG funding that could be used for lesson study was delegated to districts and most knowledge-resource investments (e.g., training and technical assistance) ceased altogether, along with the state requirement of lesson study, at the conclusion of the RTTT program. Lesson study was not integrated into the state PD standards or state policy on teacher evaluation, as none of these policy documents referenced lesson study despite the initial intention to make RTTT-funded initiatives “the way of work in Florida” (FDOE, 2010a, p. 51). This finding indicates the need for strategic approaches to sustain federally funded PD innovations by the state when most funding is used for subcontract work with external providers who do not support sustained implementation.
Our district-level data on lesson study implementation and district capacity dimensions for sustaining lesson study, however, showed that the districts that successfully sustained lesson study were more likely to continue to provide substitute funding, had more stable PD leadership, and integrated lesson study through a clear district expectation and strategic adaptation at the district and school levels. The interviews with PD leaders showed that they integrated lesson study into core elements of the instructional system at the district level, such as PLCs, teacher evaluations, and technology initiatives. At the school level, they promoted ownership of the lesson study process by leaving the decision of how to facilitate and organize lesson study to school and teacher leaders. This likely allowed lesson study to become an internal process—a critical condition for scaling reform initiatives identified in previous studies (Coburn, 2003; Dede & Honan, 2005; McLaughlin & Mitra, 2001). Our finding indicates that these district leaders seem to have approached lesson study as a flexible process that can be adapted based on the needs of both district and schools.
On the other hand, our interviews revealed that the districts that phased out lesson study, representing a majority of districts studied here, adapted lesson study as a rigorous PD model by specifying the format and structure for the purpose of ensuring fidelity of implementation. These district leaders did not believe that school and teacher leaders can facilitate lesson study themselves and ensured that it is led by qualified district PD leaders. With a small number of PD leaders to drive lesson study across the district, it became simply infeasible to sustain lesson study when their time was reallocated to other PD initiatives at the conclusion of RTTT program.
These findings point to the critical importance of integrating federally funded PD innovations into state, district, and school contexts to achieve sustainability in ways that support student achievement goals. By its nature, innovation represents a departure from the norms of existing practice. As a PD innovation with an international origin, lesson study introduced a new approach to teacher learning through a continuous cycle of studying, planning, teaching, and reflecting. Its focus on collaborative inquiry, based on investigation of the actual enactment of an instructional practice is a departure from traditional, top-down, short-term PD workshops or even common PLC practices that do not typically involve data collection during an actual lesson with students by a group of teachers and collective in-depth analysis of student responses to specific instructional approaches. Yet, lesson study provides a rich learning opportunity for teachers to reflect on their current beliefs and assumptions about student learning and teaching, and to engage with and incorporate the new ideas promoted in instructional reform into their own teaching practice through collaborative inquiry—an ideal condition for promoting teacher learning identified by previous literature (Ball & Cohen, 1999; Little, 1993; Wilson & Berne, 1999).
In the districts that successfully sustained lesson study after RTTT funding ended, system integration was enabled through a continued expectation to engage in lesson study and strategic adaptation at both district and school levels. The fact that these districts were able to sustain lesson study despite the lack of state support or effort to sustain lesson study shows the critical role of district leaders in sustaining PD innovations that drive instructional reform at the local level (Fuhrman & Elmore, 1990; Spillane, 1996). This finding is consistent with previous research that has also noted limited state capacity to drive large-scale reform (Kober & Rentner, 2011; McGuinn, 2012; Smylie et al., 2004; Superfine et al., 2012), but adds to our understanding of the role of district leadership and approaches to sustaining innovations. The district PD leaders’ emphasis on school and teacher ownership may reflect an understanding that innovation cannot be sustained through a top-down approach focused on fidelity of implementation.
Our district survey data showed that many indicators of fiscal resources and knowledge resources did not differentiate sustaining districts from phasing-out districts. District use of other continuing federal funding such as SIG and Title II funds as well as provision of teacher stipend were not associated with sustained practice of lesson study, although provision of substitute funding was found to be important for allowing teachers time to engage in lesson study during regular school hours. Likewise, indicators of knowledge resources—teacher training, technical assistance, materials, and lesson study coordinator—were not associated with lesson study sustainability. The importance of local adaptation through school and teacher ownership of the process, as our interview data revealed, may explain the limited role of district-led training, technical assistance, or provision of materials.
However, it is important to further examine these two dimensions of district capacity with more detailed measures focused on quality and approaches. Our survey measures simply asked about the provision of these resources, and future qualitative studies should investigate the quality of these fiscal and knowledge resources by paying attention to human and social resources (Spillane & Thompson, 1997)—commitment, dispositions, and knowledge of district PD leaders as well as the relationships and networks to disseminate knowledge and build a shared culture around the principles behind a PD innovation. Fiscal resources can be used to support knowledge resources through investing in the ways to allocate time for school or teacher leaders who successfully adapted and sustained lesson study to share their practice with other schools. Future studies should closely examine specific approaches to using and disseminating these resources to sustain a PD innovation to drive districtwide instructional improvement.
In examining the district background characteristics, we found that the districts that successfully sustained lesson study had less high-poverty students and more high-achieving students than the districts that phased out lesson study. It is important to examine how district poverty and achievement levels impact sustainability of RTTT-funded PD innovations. Although FDOE required all RTTT-funded districts to practice lesson study, compliance measures were applied to only low-achieving and likely high-poverty districts with PLA schools. Despite these more stringent measures, these districts were no more likely to sustain lesson study. Furthermore, these districts are less likely to have stable district PD leadership and to be able to focus on a small number of quality PD initiatives, probably because of multiple top-down initiatives they are often required to implement due to school turnaround reform. This accountability trend of a stronger state oversight on low-achieving schools and districts is also evident in the FDOE’s (2018) plan in response to the 2015 reauthorization of the Every Student Succeed Act (ESSA). Based on our finding on the importance of local adaptation and ownership for sustaining a promising PD innovation, there is a need to examine how current accountability policies with stronger oversight may hinder high-poverty and low-achieving school districts to sustain innovations.
In summary, this study confirmed the major difficulty of sustaining a federally funded PD innovation, identified the state’s limited capacity for sustainability, and highlighted the importance of building district capacity for sustaining PD innovations for instructional improvement. There are important implications for policy and practice based on the study findings. First, state departments of education should consider a long-term plan for sustaining a PD innovation upfront, especially when the PD implementation relies heavily on subcontracts with non-system actors as was the case in Florida. Based on the finding on the importance of adaptation and ownership, the state may partner with district leaders where the PD innovation was successfully sustained to discuss approaches that build local capacity to sustain innovations after funding is withdrawn.
Second, the state can play an important role in setting a clear expectation of continued, meaningful engagement in a PD innovation. This can be done both through PD and teacher evaluation policies to communicate this continued expectation and through promoting partnerships among districts and disseminating successful practices for sustaining PD innovation across the state. Consistent expectation for lesson study was one important factor for integrating lesson study into the district instructional system and having this expectation at the state level would likely have encouraged more districts to sustain lesson study.
Finally, school districts can sustain federally funded PD innovation by paying attention to integration into the instructional system. This requires both a continued expectation and adaptation of both the PD innovation and existing core elements of the instructional system. Because a PD innovation such as lesson study represents new approaches to the existing practice, supporting school and teacher ownership of the process to make sense and integrate it into the school context is important, instead of ensuring implementation fidelity through a tight district involvement in the learning process. This requires district PD leaders to trust the school and teacher leaders and provide necessary support when it is needed. District leaders can also showcase successful, school-led processes of lesson study by organizing an open house for teachers from other schools to visit and by facilitating knowledge sharing to build collective instructional capacity across the district.
Supplemental Material
sj-pdf-1-epx-10.1177_08959048211015619 – Supplemental material for After the Race to the Top: State and District Capacity to Sustain Professional Development Innovation in Florida
Supplemental material, sj-pdf-1-epx-10.1177_08959048211015619 for After the Race to the Top: State and District Capacity to Sustain Professional Development Innovation in Florida by Motoko Akiba and Cassandra Howard in Educational Policy
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based on work supported by the National Science Foundation (Grant DRL-1417585). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Supplemental Material
Supplemental material for this article is available online.
Notes
Author biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
