“Doing the ‘Real’ Work”: How Superintendents’ Sensemaking Shapes Principal Evaluation Policies and Practices in School Districts

Abstract

Almost every state in the United States has revamped its principal evaluation policies since 2009, yet we know little about how they are implemented. Based on interviews and document analysis in 21 small- and medium-sized school districts, we found that superintendents’ sensemaking shaped their implementation of policy. Drawing on their beliefs about principals and evaluation and their understanding of their district context, superintendents in lower performing districts reported that they complied with the processes specified in state principal evaluation policies but strayed from state guidelines regarding maintaining a focus on instructional leadership during evaluation. In contrast, superintendents of higher performing districts reported that they implemented evaluation processes loosely but adhered to their state’s policy emphasis on instructional leadership. Our findings raise questions about whether the implementation of principal evaluation policies disadvantages principals in lower performing districts. We thus caution against attaching high-stakes consequences such as incentive pay or sanctions to these policies.

School district superintendents in the United States and other nations have long faced conflicting pressures and priorities (Björk et al., 2018; Cuban, 1988). In recent years, the image of the “instructional supervisor” has guided their work, focusing their efforts on improving classroom instruction. At the same time, the metaphor of administrative chief has driven their decisions, demanding their attention to efficiency, control, and compliance. These and other distinct visions regarding the “real work” of the superintendency shape how these leaders conceive their role, define their goals, prioritize tasks, delegate responsibilities, and make decisions (Björk et al., 2018). They likely also influence how superintendents interpret and implement state policy in their school districts.

Principal evaluation is one such policy that superintendents enact. Principals play a critical role in providing high-quality education to students by supporting teachers’ instruction, maintaining a focus on student learning, and fostering productive school environments (Branch et al., 2012; Grissom et al., 2013; Louis et al., 2010; Newmann et al., 2001; Robinson et al., 2008; Urick et al., 2018). In recent years, principal evaluation has become a key lever through which districts seek to improve school leaders’ practices. Almost every state in the United States has revamped their principal evaluation policies since 2009, yet we know little about the processes or foci that these new policies prescribe. Furthermore, we know almost nothing about how superintendents make sense of these state policies or how this sensemaking shapes policy implementation. Here, we address this gap by examining principal evaluation policies and superintendents’ sensemaking in 21 small and medium-sized school districts in Connecticut and Michigan. Based on interview data and document analysis, we found that superintendents’ sensemaking varied by district. Drawing on their beliefs about principal leadership and evaluation and their understanding of their district context, superintendents in lower performing districts tended to report that they complied with the processes specified in state principal evaluation policies but strayed from state guidelines regarding maintaining a focus on instructional leadership when evaluating principals. In contrast, superintendents of higher performing districts reported that they implemented evaluation processes loosely but adhered to their state’s policy emphasis on instructional leadership. Our findings raise questions regarding the equity of principal evaluation policies and specifically whether the heavy weight that principal evaluation policies assign to instructional leadership disadvantages principals in lower performing districts, which often serve a disproportionate number of students of color and students from low-income families. Our findings thus caution against attaching high stakes such as incentive pay or sanctions to principals’ evaluation ratings.

Background

School district superintendents play an important role in the implementation of state and district policies, such as principal evaluation, aimed at changing educators’ work in schools (Goertz et al., 1995; Jennings & Spillane, 1996). Districts’ support for and implementation of state policies can vary considerably (Donaldson et al., 2016); Spillane, 1998). Arguably, the superintendent plays an especially central role in policy interpretation and implementation in small and mid-sized districts, which constitute the majority of districts in the nation. ¹

Superintendents’ Role in Policy Implementation

Theoretical and empirical work indicates that superintendents shape the success of new policies several ways. First, they can direct principals’ and teachers’ attention to a policy and signal its importance (Leithwood & Prestine, 2002; Spillane, 1998). Second, they can invest in developing school-level educators’ capacity to implement the policy (Leithwood & Prestine, 2002; Stein & Coburn, 2008). Third, they can keep all members of the organization focused on improving teaching and learning (Daly & Finnigan, 2011; Leithwood & Prestine, 2002).

Moreover, superintendents can signal not only the importance of a policy overall but also convey messages regarding the relative value of different aspects of the policy. Woulfin et al. (2015) reported that superintendents framed one state’s new educator evaluation system primarily as a tool for holding principals and teachers accountable. Finally, superintendents may implement policies in different ways depending on their district context. Superintendents’ leadership practices often vary based on their district poverty rate, urbanicity, and enrollment (Leithwood et al., 2004; Louis et al., 2010; Niño, 2018).

Superintendents and Principal Evaluation Policies

The general influence of superintendents over policy implementation is likely heightened in the case of principal evaluation. Superintendents, especially those in small to mid-size districts similar to those in our sample, often supervise and evaluate principals themselves. Yet most research on district leaders’ involvement in principal evaluation examines principal supervisors in large districts. This research suggests that principal supervisors are increasingly focused on instructional leadership. A study in six large districts found that principal supervisors focused less on monitoring school leaders’ compliance with district priorities and more on supporting their growth as instructional leaders (Anderson & Turnbull, 2016). A second study found that six large districts had revamped principal supervisors’ jobs to emphasize instructional leadership, and they spent the majority of their time observing, coaching, and providing feedback to principals (Goldring et al., 2018).

Is it wise for new principal evaluation systems to emphasize instructional leadership? Grissom and Loeb (2011) found that principals’ managerial leadership was associated with higher levels of student achievement but their instructional leadership did not predict achievement levels. In addition, May et al. (2012) found that principals in high-performing schools allocated more time to personnel and finance issues while leaders in low-performing schools devoted more time to instructional leadership and goal-setting. The authors argue that both groups of principals responded to conditions in their schools; those in low-performing schools faced more external pressure to engage in instructional leadership while those in high-performing schools could concentrate on personnel and budget issues precisely because they did not feel pressure to address instruction.

Beyond the leadership focus of new evaluation systems, research indicates that the district principal evaluation policies and practices may differ. In a study of three large urban districts (e.g., Miami-Dade) and two small, rural Ohio districts, Kimball et al. (2015) found that new principal evaluation systems were more rigorous and complicated than earlier ones. Most districts had enacted goal setting and continuous improvement cycles for principals and the three large districts had reduced supervisor–principal ratios. The authors noted that the close relationship between the superintendent, who was the sole evaluator, and principals in one small district seemed to facilitate coaching.

Despite these advances, we know less about how superintendents respond to state guidelines regarding principal evaluation policy, whether they maintain a focus in evaluating principals on instructional leadership, or whether superintendents vary in policy implementation processes and foci based on their districts’ academic performance levels.

Conceptual Framework

Our inquiry draws on a robust line of research indicating that actors make sense of and adapt policies when enacting them. Policy implementation is influenced by how stakeholders take in and frame information and how they translate this information into action (Evans, 2007; Ingle et al., 2011). Prior research has shown that school and district leaders filter policy messages through their worldviews and professional and personal beliefs (Coburn, 2006; Ingle et al., 2011; Spillane, Diamond et al., 2002). Their framing of problems “opens up and legitimizes certain avenues of action and closes off and delegitimizes others” (Coburn, 2006, p. 344).

While literature on how district leaders make sense of and implement policy is scant, some research examines school leaders’ policy sensemaking. For example, school leaders draw on their experiences, expertise, professional development, principal preparation training, and mentoring to define what it means to be an instructional leader (McGough, 2003; Youngs, 2007), interpret and enact test-based accountability and personnel policies (Donaldson, 2013; Spillane, Reiser et al., 2002), and determine how to incorporate social justice into their leadership practices (Mavrogordato & White, 2020).

Coupled with individual factors, institutional and organizational context also factor prominently into the policy implementation process. Contextual characteristics such as size, capacity, competing priorities, and institutional leadership influence this process (e.g., Coburn, 2001; Cohen, 1990; McLaughlin & Elmore, 1982; Spillane, 1996, 1998, 2000). Local context can open up and close avenues for implementing policy and prompt people to use policy resources in different ways. For example, policy implementers who lack adequate resources to meet ambitious policy expectations often devise solutions to satisfy competing policy demands. Similarly, competing demands can prompt implementers to forego certain ways of enacting policy in an effort to reduce tensions or conflicts among multiple policies.

Moreover, a growing literature explores how educational leaders’ sensemaking determines whether education policies are implemented in ways that enable or constrain educational equity for students. For example, in a study of English learners’ access to college preparatory coursework, Callahan and Shifrer (2016) call upon educational leaders to consider whether adhering to policies on providing English language development services to EL students “precludes, rather than improves, equity” (p. 485). Other researchers encourage educational leaders to leverage policy enactment to promote social justice for students (Mavrogordato & White, 2020). This literature suggests that principal evaluation policy may be implemented in ways that facilitate or undermine educational equity in schools.

In most settings, principal evaluation has received much less attention and oversight from state and district administrators than teacher evaluation (Fuller & Hollingworth, 2014). As a result, superintendents may have a relatively high degree of latitude when interpreting their state’s principal evaluation policy. Moreover, the sensemaking process is heightened during times of change (Anagnostopoulos et al., 2010; Ingle et al., 2011). Thus, as district leaders implement a new high-stakes principal evaluation policy, their sensemaking process will be particularly critical. Despite the importance of principals and the need to attend to their development and appraisal, little empirical work has documented the processes or foci of principal evaluation policies or how superintendents make sense of and enact them. Moreover, we are aware of no studies that investigate these questions in small- to medium-sized districts that comprise the majority of districts across the United States.

Method

Data sources for this study comprised principal evaluation documents and interviews. For each district, we collected policy documents, including the district’s evaluation policy itself and rubrics used to assess principals’ practice. We also conducted one-on-one, hour-long interviews with administrators responsible for conducting principal evaluation in each district. These included 21 superintendents and two assistant superintendents. Our semistructured interview protocol sought to elicit leaders’ sensemaking regarding the principal evaluation policies in their districts. We asked about the processes, foci, and consequences of principals’ evaluations and probed district leaders’ conceptions of principal quality and views on how their districts seek to improve principals’ practice.

Sample

We purposively selected districts in Connecticut and Michigan to build a sample of districts with similar yet distinct policy contexts. Both states granted districts discretion in creating principal evaluation systems that adhered to guidelines set by state policy, yet the state policies, as described below, differed in key ways. Within these states, we selected districts to maximize variation in student enrollment, urbanicity, student demographics, and district principal evaluation policies. Participating districts including 10 in Connecticut and 11 in Michigan and ranged in enrollment from approximately 1,000 to 21,000 students (Table 1). In each district, we interviewed individuals with direct responsibility for evaluating principals. In total, we interviewed 27 leaders, including 15 males and 12 females and 17 superintendents and 10 assistant superintendents/directors of curriculum or instruction. In six instances, we interviewed a superintendent and assistant superintendent/director together. In four instances, the district’s assistant or director oversaw principal evaluation, so we interviewed that person instead of the superintendent. In what follows, we refer to all participants as “superintendents.”

Table 1

Sample Districts

District	State	Enrollment	FRPL (%)	Students of color (%)	English learners (%)	Mean proficiency: ELA (%)	Mean proficiency: Math (%)	Average proficiency (%)	Performance classification	Number of schools
Bradley	CT	21,000	100	90	15	10	15	13	Lower	38
Carleton	MI	4,000	72	52	10	17	10	14	Lower	13
Washington	CT	7,000	59	83	10	31	17	24	Lower	12
Hamilton	MI	2,000	50	26	13	41	30	36	Lower	5
Valliant	CT	3,000	44	31	3	45	33	39	Lower	7
Norwood	MI	1,000	56	20	10	43	36	40	Lower	3
Clearmont	MI	3,000	50	10	20	42	38	40	Lower	5
Gorman	MI	6,000	37	25	3	49	44	47	Mid	10
Elmer	CT	5,000	40	21	2	55	43	49	Mid	9
Rhine	MI	3,000	24	5	1	56	42	49	Mid	6
Parkston	MI	2,000	44	10	0	53	45	49	Mid	4
Ralston	MI	5,000	22	12	7	57	49	53	Mid	9
Morrison	CT	6,000	20	21	3	60	49	55	Mid	13
Barrett	MI	8,000	28	21	3	62	48	55	Mid	12
Spaulding	CT	2,000	12	12	1	61	52	57	Higher	4
Oakwood	CT	2,000	10	10	0	61	54	58	Higher	5
Lambert	MI	3,000	17	11	1	66	52	59	Higher	6
Mayville	CT	1,000	24	24	5	70	66	68	Higher	4
Sienna	CT	3,000	4	8	1	73	70	72	Higher	5
Jefferson	MI	7,000	6	49	16	70	76	73	Higher	8
Gaffney	CT	3,000	9	14	1	79	73	76	Higher	7

Note. All district names are pseudonyms and numbers are rounded to protect the confidentiality of participating districts. Data on economically disadvantaged students in CT retrieved from Kids Count Data Center 2013–2014, kidscount.org. Data on economically disadvantaged students in MI retrieved from CEPI 2014–2015 data, mischooldata.org. All other data collected from the Common Core of Data 2014–2015 LEA survey. FRPL = free and reduced price lunch; ELA = English language arts.

Analytic Approach

To analyze the principal evaluation policies and superintendent interviews, we engaged in a variety of data reduction approaches.

District Principal Evaluation Policies

Consistent with Goldring et al.’s (2009) content analysis of principal evaluation policies, we used an iterative, deductive process to code principal evaluation documents for leadership focus. This framework categorizes principal evaluation policy contents into instruction (e.g., monitors instruction), management (e.g., oversees school facilities), external environment (e.g., advocates for the school), and personal characteristics (e.g., listens; see the appendix). We coded the rubrics associated with the leadership practice portion of each district’s principal evaluation policy at the indicator level, assigning relevant codes (i.e., subdomains) to each portion of rubric text. For example, many rubrics contained an indicator that addressed the completion of classroom observations. We would code these indicators with the subdomain “monitors instruction” and the associated domain “Instruction.” Before engaging in this process, four members of the research team met four times to discuss the meaning of domains and subdomains. All researchers coded multiple rubrics and discussed areas of agreement and disagreement. Subsequently, two researchers double coded every rubric and then the team of four met to discuss any discrepancies. Afterward, we generated aggregate scores for each domain following Goldring et al.’s (2009) procedures by calculating the percentage of indicators on each rubric that were coded in each of the above categories. Values for this variable could range from 0 to 100.

We also coded the district policies for references to the principal evaluation process. We assigned codes such as “observations,” “student scores,” and “summative rating” and engaged in the same process to establish reliability as described above.

Interviews

Each interview was transcribed verbatim. Using Dedoose software, we coded the transcripts and field notes using open, closed, axial, and selective coding (Strauss & Corbin, 1990). Closed codes were selected based on the research literature, including sensemaking, and included “beliefs” and “experience in district.” We created open codes through a process of reading and rereading the transcripts to identify salient concepts and included “implementation logistics” and “congruence with prior evaluation policy.” We then constructed categorical matrices capturing what individual participants said or did related to that theme. All these measures facilitated our use of the constant comparative method (Glaser & Strauss, 1967) to identify emerging themes across participants (Miles & Huberman, 1994). These matrices enabled us to capture each district leader’s perceptions regarding the types of leadership they sought to foster among principals and the process through which they evaluated principals.

To analyze the relationship between principal evaluation policies and superintendents’ evaluative practices, we juxtaposed districts’ coded policies with superintendents’ reported practices and rationales for engaging in them. In these rationales, we found superintendents’ sensemaking regarding the type of leadership they encouraged through principal evaluation, the process through which they evaluated principals, and how and why their implementation of principal evaluation diverged (if at all) from the processes and leadership focus prescribed in state policies. Preliminary analysis suggested that superintendents’ sensemaking about principal evaluation policy seemed to vary according to districts’ performance on standardized assessments. Therefore, we divided the sample to create three groups of districts (i.e., higher, mid-, and lower performing), which allowed us to analyze superintendent sensemaking by district performance level.

Findings

Overall, we found that superintendents reported adhering to state guidelines when they implemented some aspects of principal evaluation while diverging from them when they enacted others. Superintendents’ sensemaking appeared to lead those in lower performing school districts to implement the principal evaluation process laid out by the state, while those in higher and mid-performing districts tended to modify this process. By contrast, superintendents of higher performing tended to report to adhering to state guidelines regarding maintaining a focus on instructional leadership through principal evaluation. Leaders of lower and some mid-performing districts reported straying from the instructional leadership focus to emphasize multiple types of leadership and managerial leadership in particular.

Principal Evaluation Processes

State policy sets out guidelines for principal evaluation in Connecticut and Michigan. Connecticut policy specifies the components of principal evaluation and their weights, with leadership practice, measured by the state rubric or a state-approved locally designed rubric, contributing 40% to a principal’s rating. The balance is obtained through student learning measures in the form of student learning objectives (45%), stakeholder feedback (10%), and teacher effectiveness measures (5%). Michigan similarly delineates components and weights, with 50% obtained from measures of student growth and 50% from other sources. Importantly, Michigan’s two state-approved rubrics cover all measures of principal quality, including student growth measures, which constitute a set of discrete items on the rubrics.

Connecticut’s state policy requires principal evaluators to hold a minimum of three evaluative meetings with principals each school year. They meet in the fall to agree on goals for the principal; at mid-year to evaluate the principal’s progress and revise goals as needed; and at year’s end to discuss summative ratings. The evaluator must observe and score the principal using the rubric at least two times and provide feedback. Michigan requires evaluators to conduct observations and produce summative ratings and a report on the principal’s performance. Thus, both states’ policies emphasize formal meetings and summative ratings and reports.

Adhering to State Policy on the Evaluation Process in Lower Performing Districts

Only 29% (6 of 21) of the leaders in the sample reported that they implemented evaluation processes that were consistent with state guidelines. Leaders of lower performing districts were more likely to report that they adhered to state guidelines regarding process. In all, 43% (3 of 7) of leaders of lower performing districts reported implementing principal evaluation processes as state policy dictated, while only 14% of leaders of higher and mid-performing districts responded similarly.

Adhering to the state guidelines in lower performing districts

For the most part, superintendents of low-performing districts welcomed, or at least tolerated, their state’s principal evaluation guidelines. Superintendents in such districts explained their adherence to state guidelines in terms of their beliefs and professional experiences with principal leadership and evaluation and their district’s context. These superintendents privileged “alignment” and “coherence” as guiding principles, and their sensemaking asserted a highly rational, bureaucratic, accountability-driven model of how their organizations should function and how principal evaluation should occur.

Some superintendents in this group framed their support for the principal evaluation guidelines in terms of their beliefs. When asked if their district’s evaluation system diverged from state guidelines, one superintendent responded: “(W)ith the philosophy that you have here of your leaders, absolutely not” (Carleton, Lower). Other superintendents reported that they implemented the state guidelines with high fidelity because they felt that state directives were reasonable and would help their principals improve. For example, one leader explained that the state’s new rubric was useful because in their words, it “meshes what our teachers are doing with what really our vision of leadership is” and it’s “fair and equitable and all that other good stuff. It focuses on relationships, that’s been a huge focus in our district” (Hamilton, Lower).

Other superintendents in this group argued that principal evaluation processes aligned with preexisting district accountability processes. For example, one Michigan superintendent stated, “I don’t have a problem with the new evaluation system. I think it’s asking us to do what we need to do anyway. And what most people are already doing” (Norwood, Lower). Other superintendents drew connections between principal evaluation practices and other accountability routines in their districts. One leader explained that their district already held regular principals’ meetings to review student achievement: “There’s 15 people in the room. And so I feel like they (principals) already feel like ‘I’m being evaluated on how my school is doing and what I'm doing to improve my school.’" (Carleton, Lower). Similarly, a Connecticut superintendent highlighted the “alignment” between principal evaluation and other processes and documents the district had put in place for accountability purposes. They stated, “(T)he goals of the school improvement plan are also in the [evaluation] goals of our principals and our administrators so that’s very clear to everybody” (Washington, Lower).

Thus, superintendents in lower performing districts tended to report that their districts implemented the principal evaluation processes as specified by their states. They made sense of their decisions to comply with state policies in terms of their beliefs about the importance of compliance, their support for the processes laid out in the guidelines, and the alignment of the policies with other accountability practices in their districts.

Frequent Modifications in Higher and Mid-Performing Districts

In contrast to lower performing districts, most leaders (86% [6 of 7 in each group]) of higher and mid-performing districts in our sample reported modifying their state’s principal evaluation model or guidelines when they enacted their principal evaluation system in their district. Some modifications were allowed by the state (e.g., reducing the scope of the rubric) whereas others were expressly prohibited (e.g., changing the components’ weights). Similar to superintendents in lower performing districts, these leaders’ sensemaking regarding principal evaluation emanated from their beliefs and professional experiences related to principal leadership and evaluation and their understanding of their district contexts.

Implementing an organic process in higher and mid-performing districts

In many higher and mid-performing districts, superintendents reported enacting principal evaluation in a fluid, dynamic, and highly relational manner. They frequently critiqued their state’s evaluation process, arguing that it did not work for their district context. In justifying their changes to evaluative processes, superintendents articulated beliefs about leadership development and the nature of evaluation that supported these modifications. They voiced the belief that coaching was more powerful than evaluation and enabled them to personalize what they perceived to be a static and ineffective process. They also asserted visions of their districts as organic and mission-driven, with a heavy emphasis on change.

For example, one superintendent explained, “I look at and see evaluation as fluid, constantly ongoing” (Rhine, Mid). Another reported using the state guidelines for principal evaluation “as a surface-level tool” instead of following them closely. They implemented principal evaluation in their district in a way that they described as “a lot more organic, and it’s a lot more sliding on that continuum, and I do see myself more as a collaborative coach than a more distant evaluator” (Gaffney, Higher). When asked how they applied the state-mandated weights, one superintendent said, “I kind of disregard them” (Morrison, Mid).

As a result of their organic approach, some district leaders adopted an ongoing evaluative stance. For example, one explained that principal evaluation was “a continuous process of development or understanding of what they are doing with course corrections or dialogues or trying to better understand what they are doing” (Parkston, Mid). In this way, superintendents in higher and mid-performing districts stepped outside the formal structures of evaluation to provide constructive feedback and guide principals toward improvement in an ongoing fashion.

The salience of beliefs

In explaining their reasons for deviating from state guidelines, district leaders revealed sensemaking rooted in their beliefs about principal evaluation and change. They often articulated strong critiques of state guidelines and processes. Some argued that the state’s evaluation system was too static for their district. For example, one superintendent noted,

as the system has evolved, we haven’t adapted . . . the evaluation plan to match where the system is. So, those structures kind of remain static. And then we want to kind of, you know move forward, but I’ve got to keep sending this round peg in a square hole because I have to make sure I do X, Y, Z for the evaluation plan. (Sienna, Higher)

Another argued that the evaluation structure produced negative effects. They stated, “(T)he biggest weakness is that an overly structured system with a four-point grading scale forces them into this fixed mindset that we don’t want them to be in” (Gaffney, Higher).

Similarly, another superintendent of a mid-performing district reported that after spending 1 year in their new district, they concluded,

the administrator eval . . . it wasn’t working for me. It seemed to be a task that . . . the administrators were just going through and it wasn’t really directing the improvement of their school in a way that I think evaluation can. (Mayville, Higher)

Other superintendents articulated broader philosophical opposition to the current focus on educator evaluation. For example, one superintendent questioned the value of evaluation:

I don’t know that it really does much and that’s been my concern about this for 37 years . . . nobody was getting anything out of it. (Elmer, Mid). Another Connecticut superintendent concurred, arguing that evaluation was less productive than other uses of time: “[I]ndividual appraisal delivers less return of investment than everybody’s focusing on the system” (Sienna, Higher). Likewise, a Michigan superintendent acknowledged that they and the principals with whom they worked put little value into evaluation:

The principals themselves I think that they kind of accept that we have to do something and I think they see that this is OK. And, honestly, it just isn’t that important to them and that’s how I want it. (Lambert, Higher)

In short, superintendents in higher and mid-performing districts tended to critique their states’ principal evaluation policies. They held strong and skeptical—if not dismissive—beliefs about the value of performance evaluation, which factored heavily in their sensemaking regarding how to respond to the state guidelines.

As a result of their incredulity about principal evaluation, superintendents reported minimizing the formal structures of the new principal evaluation systems in order to focus on what one called the “real work” of improving principals’ practice. For example, one superintendent stated, “I have used words like cavalier, where I have explained to the team how we would use that model.” They added, “I don’t think the model is more important than the conversations we have or the real work that we do, and so we have acknowledged, to some degree, that we are using the model as a surface-level tool” (Gaffney, Higher).

Similarly, another superintendent said their district enacted the system but expected it to accomplish little other than enabling the district to obtain funding: “(W)e will send the State a number and they will be happy, and they will send it to the feds and they will be happy and the money will flow and you know, it will all do its thing.” This leader added, “Their growth is going to come from conversations about . . . what new ideas do you have for your building, where do you want to take this building, what can I do to help you get there” (Elmer, Mid).

Other superintendents reported that they reshaped principal evaluation to align with their beliefs about promoting continuous improvement through evaluation. For example, the superintendent in Mayville implemented principal evaluation with a focus on professional learning: “I just feel really strongly about professional learning. I just try to keep imbedding that into everything that we do” (Mayville, Higher).

Several superintendents reported that they used evaluation to serve their larger goal of improving principals’ practice. One leader reported that their goal in evaluation was to make principals “feel empowered. I want them to feel mastery. I want them to feel a purpose. I want them to feel, and that’s the way that we’ve shaped or muddied the evaluation process” (Gaffney, Higher). Here the superintendent’s sensemaking is rooted in their beliefs that a sense of mastery and purpose is essential for individuals to grow, learn, and improve. Similarly, another leader of a higher performing district voiced a desire to “empower” principals through “central office streamlining and making certain things a lot easier: curriculum, assessment, evaluation—things like that” (Jefferson, Higher).

The importance of context

In explaining their approach to evaluation, many district leaders drew heavily on their understanding of the district’s context. Some superintendents in this group invoked their district’s performance profile when describing how they implemented principal evaluation. For example, the superintendent in Gaffney argued that the model was not suitable for higher performing districts. They stated, “I do think it’s sort of organic” in their district because “for a high-achieving district, that structure and format (aren’t) necessarily what we need when you’re already performing at a high level.” They added, “I think that I take significant latitude in how we apply and follow the state guidelines . . . I tend to not take the evaluation process itself overly seriously” (Gaffney, Higher).

Other superintendents similarly asserted that higher performance enabled them to deviate from the state policy. In one case, a superintendent downplayed the policy’s emphasis on student achievement, “I’m not really worried about what the data says.” Instead, they focused on what the data “suggest your strengths and weaknesses are, and what are we going to do about it between now and the next time we meet?” (Oakwood, Mid).

In addition, some superintendents made sense of their enactment of principal evaluation as a function of their district’s size. Many of these districts were small, which one superintendent of a four-school district asserted allowed them to enact principal evaluation more informally. As they stated, “I am the central office,” so they could implement principal evaluation in what they called a “loose” way: “It’s in my head and but in a pretty purposeful way because I’m kind of drawing on all of my experiences and the things that I’ve used in other places and using it here” (Mayville, Higher).

Similarly, the superintendent in Sienna contrasted their experience there, a district with seven schools, with their previous experience in a larger district. In the prior district, they would have shared policy documents in order to drive principal evaluation. In contrast, in Sienna they “can get [their] arm around things” and pull the principals “together in the same direction” (Sienna, Higher).

Policy Leadership Focus

In sum, we found that leaders of higher and mid-performing districts were more likely to report diverging from state guidelines with respect to the processes of principal evaluation. However, we found a different pattern regarding the type of leadership emphasized in principal evaluation. Across our sample, district principal evaluation policies weighed instructional leadership more heavily than other forms of principal leadership. On average, 82.34% of rubric indicators were in the Schooling and Instruction domain; 34.38% were in Personal Traits; 31.82% were in External Environment; and 18.01% were in Management (Table 2).

Table 2

Percentage of Indicators Coded With Each Domain (n = 21)

Domains	M (SD)
Schooling and Instruction	82.34% (8.61)
Management	18.01% (6.67)
Personal Traits	34.38% (11.48)
External Environment	31.82% (16.00)

Consistent with the leadership emphasis in the policies, roughly half (52%) of superintendents reported using the principal evaluation system to promote instructional leadership among principals. At the same time, 48% reported modifying the leadership focus as they implemented evaluation. In this case, superintendents of lower performing and mid-performing districts tended to report modifying the leadership focus. In contrast, superintendents of higher performing districts were more likely than administrators in other districts to explicitly name instructional leadership as the type of leadership that they encouraged via principal evaluation. In all, 86% (6 of 7) of superintendents in higher performing districts, 43% (3 of 7) of superintendents in mid-performing districts, and 29% (2 of 7) of superintendents in lower performing districts reported maintaining the leadership focus of principal evaluation on instructional leadership (Schooling and Instruction) as they implemented principal evaluation. Again, superintendents’ sensemaking figured heavily in their decision regarding whether to implement principal evaluation with a focus on instructional leadership as the policies dictated.

Adhering to State Policy on the Evaluation Leadership Focus in Higher Performing Districts

Many superintendents of higher performing districts and some mid-performing districts reported that they enacted principal evaluation with an emphasis on schooling and instruction. For example, one superintendent explained this is the most central part of principals’ work. While principals must manage their buildings and maintain productive relationships with their communities, “if you don’t know how to articulate your own vision and build a shared vision mission and if you don’t understand teaching and learning you’re just, you’re not going to cut it” (Mayville, Higher).

Other superintendents expanded on this argument, describing a shift over time in expectations for principals from an emphasis on management to one on instruction. In doing so, they set up a hierarchy, asserting that managerial leadership entailed a lower level skill set than instructional leadership. For example, one stated: “(T)he emphasis on education right now (is) to take the principals away from being a manager to being an instructional leader and I think this fits in with the instructional leadership components” (Lambert, Higher).

Some superintendents explicitly argued that different forms of principal leadership could be arrayed in a hierarchy, with managerial leadership assuming the bottom rung in the pyramid of leadership skills. For example, one superintendent stated: “If I am a superintendent watching this happen and somebody can’t manage their building, nothing else is going to matter . . . That is just going to be plain ineffective—you can’t manage anything” (Parkston, Mid).

Similarly, some superintendents reported that in certain instances they focused on management despite their more general emphasis on schooling and instruction.

You know, obviously if there were serious deficiencies in other areas they wouldn’t even be able to begin to get to this part of their job. So, they’ve got to do a good job with the areas to even have time to focus on [schooling and instruction]. (Lambert, Higher)

This superintendent articulated the interconnectedness of the domains within principal evaluation. In order to perform well in Instruction, he argued, a principal must perform adequately in the other three domains.

Why maintain the focus on instructional leadership?

In deciding what types of leadership to emphasize and how to implement principal evaluation in their district, superintendents drew on their beliefs about evaluation and principal leadership, prior experiences, understanding of the needs of their principals, and sense of their district and school contexts.

In higher performing districts, superintendents described how their district context affected the leadership focus of principal evaluation. For example, one superintendent framed the activities in the management domain as “remedial work” and expected their school leaders to manage student discipline and staff concerns well: “I don’t want to spend a lot of time talking about that with you. That would be remedial work, as far as I was concerned” (Gaffney, Higher).

In contrast, this superintendent recalled their early days in a lower performing district: “When I got to (my prior district), I needed to have instructional conversations, and that was what I did with the principals.” In their current, higher performing setting, “I do believe that we’ve got seven solid principals here. There’s still a couple that I push hard instructional components” but the others had already mastered these skills (Gaffney, Higher). They described their goal: “I want them to be better at thinking about, talking about, and reflecting on instruction with teachers. I want them to lose the managerial component.”

Superintendents reported they used principal evaluation to encourage principals to build their instructional leadership skills. For example, one superintendent stated that their principals participated in a professional learning community that learned about instructional leadership with principals specializing in different areas:

For example, we have an administrator at one of our elementary schools and early elementary literacy is her thing. She’s an expert at that. So she tends to be the person that leads a lot of those initiatives, which I think is what you want out of a leadership pool. (Barrett, Mid)

Last, one superintendent of a higher performing district argued that their principals had mastered instructional leadership, so they used principal evaluation to promote “change leadership.” They asserted,

The focus became just so narrow on [instructional leadership] and so all-consuming even to this day. I mean because we tried to comply our principals are doing a lot of work that they probably should be using their time in better ways. (Sienna, Higher)

Instead, they shifted the focus of principal evaluation to emphasize “change leadership.” They explained,

you know some of the elements are in there [the rubric], but to really look at the change process, the change agenda to say, “Okay, you know what? That’s a big part now of how our work has surfaced. That should play a prominent role in the evaluation plan.” (Sienna, Higher)

Multiple Foci in Principal Evaluation in Mid- and Lower Performing Districts

In contrast to their counterparts in higher performing districts, leaders of lower and some mid-performing districts did not identify instructional leadership as the main focus of their principal evaluation systems. For example, a superintendent in a lower performing district outlined the breadth of tasks that confronts a principal: They reported that principals in the district needed to focus on student achievement, engage in professional practice, address emergencies, participate in their own learning and development, engage the community and respond to stakeholder feedback. In their words, “(W)e need that well-rounded student, that well-rounded administrator” (Valliant, Lower).

Why not focus on instructional leadership?

Superintendents in this group altered their policy’s leadership focus because they believed their district context demanded a different sort of emphasis. Some superintendents in lower and mid-performing districts reported that they were trying to shift principals’ self-concept and build their skills so they could focus on instruction. Other superintendents of lower performing districts suggested that their principals needed to focus on other forms of leadership before they could implement leadership focused on instruction.

Several leaders of lower performing districts reported that the principals thought of themselves as first and foremost managers and that district leaders were attempting to shift their self-concept. For example, one stated,

The standard in our district used to be that if you managed your building well, you were doing a good job. But it doesn’t work that way anymore. There is so much pressure on accountability and really just to help our kids learn they need to understand the instruction. (Clearmont, Lower)

Similarly, another superintendent recounted that when they started working in their current district the previous year, “All the principals were managers. They were managerial leaders, you know make sure the bells are on time, everything is working, people [are] in their classrooms . . . We are trying to shift them to instruction leaders” (Valliant, Lower). Furthermore, in a mid-performing district, the superintendent reported that instructional leadership “wasn’t necessarily, I don’t want to say it was absent here in (district), but it was never, it hasn’t been intentional.” They said they and their team were trying to encourage principals to engage in instructional leadership by “really being deliberate to make sure that principals see themselves as instructional leaders then managers. And typically, that’s flipped” (Rhine, Mid).

Some superintendents suggested that their principals needed to focus on all forms of leadership because their settings or their principals’ skills demanded it. In a district where 75% of the principals were novices, “Everything feels like a priority,” stated the superintendent (Carleton, Low). Similarly, in another lower performing district, the superintendent articulated a broad focus that spanned management, instruction, and parent and community engagement. She added that principals in her district needed to be entrepreneurial because of the district’s lack of resources: “(S)eeking out grants, that’s an important component in this district because you know we are constantly looking for funding (Washington, Lower).

Other district leaders stated that their principals faced lower order demands that they needed to address before moving on to instructional leadership. One superintendent articulated the daily tasks of principals with a focus on management: “(T)here’s so many roles that a principal has during the day. Referrals. Positive behavior support. Just being visible and, of course, the reporting and paperwork” (Hamilton Lower).

Similarly, a leader in a mid-performing district articulated how basic demands interfered with principals’ ability to exercise instructional leadership. These included a shortage of substitutes that created challenges for classroom observations:

We are trying to address a sub shortage in our buildings. Even down to the evaluation tool, there are times where there is a scheduled evaluation ready to go at 9 am and a sub doesn’t show. Well unwind all of those plans because the principal is now sitting in as that teacher. (Gorman, Mid)

In the lowest performing district in the sample, the superintendent reported that other demands required principals to focus on interpersonal relationships before implementing instructional leadership. As a result, this superintendent was guided by their perceptions about the district context and principals’ needs in identifying building trust as the primary leadership activity she encouraged principals to engage in during evaluation meetings: “My belief system is that [trust] is the foundation for any of the work that we are going to begin to do with principals” (Bradley, Lower). More broadly, this superintendent believed that the principals lacked the resources to be able to perform at a high level according to the principal evaluation system:

I think they are overwhelmed. I think the lack of resources in this area at the school level has been incredibly difficult on them . . . they really honestly don’t have the tools that they need to do the work that they need to do and that is very, it’s you know, very, very difficult. I think they do well with what they have, but you’d be hard pressed to find a K-6 school for example that has two days of a social worker, one day of the speech therapist, two days of a psychologist. I mean there are just not the kind of services that we need to have to allow the principal to do an exemplary job. (Bradley, Lower)

In this superintendent’s view, the principals in their district lacked basic resources—social workers, psychologists—to address significant student needs. Absent these supports, principals could not achieve an “exemplary” rating in the current evaluation system and a focus on instructional leadership proved challenging to sustain.

Discussion

This study examined principal evaluation policies and superintendents’ sensemaking regarding them. Drawing on interviews with district leaders and analyses of district policy documents, we found notable differences between superintendents in lower performing districts and their counterparts in mid- and higher performing districts. Those in lower performing districts were more likely to report complying with state policy regarding the processes of principal evaluation while those in higher performing districts tended to maintain a focus on instructional leadership in evaluating principals. Superintendents’ sensemaking seemed to affect how they enacted these policies and was rooted in their beliefs about what constituted the “real” work of principals, their experiences with evaluation, and the nature of their district contexts (Coburn, 2001; Weick, 1995).

These findings are consistent with prior research documenting ways in which district administrators’ sensemaking was associated with their districts’ enactment of state policies regarding curriculum and assessment (Spillane, 1998). In our study, superintendents in lower performing districts generally valued coherence and believed that state principal evaluation policy processes were well-aligned with existing approaches to accountability in their districts. On the other hand, superintendents in mid- and higher performing districts often critiqued state policy and asserted beliefs about principal development and evaluation that supported their decisions to modify state policy, implementing it in a more organic, informal fashion. In both cases, district leaders’ goal was to enact state policy in ways that they felt would benefit principals, teachers, and students in their particular context (Leithwood & Prestine, 2002).

While district leaders in mid- and higher performing districts in our study deviated from state specified processes for principal evaluation, superintendents in higher performing districts in particular tended to maintain the state’s emphasis on instructional leadership when assessing school leaders, thus reflecting their belief that this type of leadership defined principals’ “real” work. This is consistent with findings in Anderson and Turnbull (2016) and Goldring et al. (2018) that principal supervisors in large districts with considerable resources devoted to principal evaluation primarily concentrated on instructional leadership in their work with principals. In contrast, superintendents in lower performing districts in our study addressed multiple types of leadership, and managerial leadership in particular, when evaluating principals. Some of these superintendents tried to shift their principals’ focus to instructional leadership while others felt that student needs or a lack of resources such as substitute teachers or social workers made it difficult for their principals to focus on instructional leadership in their day-to-day tasks.

What explains these differences across districts in our study? Our finding that superintendent sensemaking about and enactment of principal evaluation varied by district performance is consistent with prior research that superintendent and principal leadership practices often vary based on student characteristics (Bredeson et al., 2011; Grissom & Loeb, 2011; Horng et al., 2010; May et al., 2012). Superintendents in mid- and higher performing districts reported having the autonomy to deviate from state principal evaluation guidelines and, in many cases, focus on developing principals’ development of instructional leadership skills as opposed to evaluating them in a high-stakes way. In contrast, their counterparts in lower performing districts did not perceive themselves to possess the same degree of autonomy to stray from state policy. In addition, their perceptions about the substantial challenges and lack of resources experienced by their principals led superintendents in lower performing districts to emphasize multiple types of leadership as they evaluated principals. In this way, superintendents across our sample drew heavily on their understanding of their district context and their beliefs about the “real” work of principals given this context when making sense of state and district principal evaluation policies and deciding how to implement them.

Limitations and Implications

It is important to note several limitations to this study. First, we included small- and mid-sized Connecticut and Michigan districts in our sample. As a qualitative study based on a purposive sample, findings cannot generalize to larger districts or to districts in other states. Second, we relied on interview data from superintendents about their self-reported enactment of state principal evaluation policies. Superintendents, especially those under pressure to raise student test scores, may have felt reticent to report deviating from state guidelines during our interviews. Future research could include interview data from principals about their perceptions of district administrators’ implementation of such policies.

In addition, our analysis addressed superintendents’ perspectives on the processes and foci of principal evaluation policies, but it did not examine whether or how enactment of such policies is associated with key outcomes. Future research could investigate how district implementation of principal evaluation policies is associated with changes in the quality of principals’ leadership practices and gains in student achievement.

Despite these limitations, this is one of the first studies to examine how superintendents’ sensemaking is related to their districts’ enactment of state principal evaluation policies. The findings reported here can provide important guidance to districts across the United States as they consider ways to modify their educator evaluation systems under the Every Student Succeeds Act (ESSA).

What are the consequences of these differences across districts for equity in general and efforts to ensure that all students have equitable learning opportunities in particular? The decisions that superintendents in lower performing districts made to comply with state principal evaluation policy processes are consistent with a highly rational, bureaucratic, accountability driven model of how their school systems should function. To the extent that such a model more generally promotes equity, then their decisions to comply with state policy could lead to equitable learning opportunities for students. In addition, their decisions to alter the focus of such policies to include multiple forms of principal leadership seem both necessary and consistent with research indicating that principal engagement in managerial leadership is significantly associated with higher levels of student achievement while engagement in instructional leadership does not predict achievement levels (Grissom & Loeb, 2011; Horng et al., 2010).

At the same time, however, the system is structured to reward principals who achieve high levels of instructional leadership through the disproportionate weight assigned to instructional leadership in principals’ summative ratings. This pattern comports with what others have found (Henry & Viano, 2015). If principals in lower achieving districts lack access to basic resources that enable them to exercise instructional leadership at a high level, their evaluation ratings may suffer. If these ratings are attached to rewards, incentive pay, or sanctions, principals in these settings may be disadvantaged.

Therefore, policy makers should carefully weigh the effects of new principal evaluation policies on educators across diverse settings. If additional research confirms our finding that district leaders implemented principal evaluation in divergent ways corresponding to their district’s performance profile, policymakers, practitioners, and researchers should investigate the impact of these differences on principals’ performance ratings, feedback, and growth as leaders over time. They should also consider whether the varied challenges and opportunities encountered by principals in different schools and districts necessitate greater flexibility in principal evaluation. The experiences of superintendents in the lower performing districts in our sample raise questions about whether it is fair and appropriate to evaluate principals against a common rubric across diverse settings that place very different demands on principals while offering them resources that vary—sometimes dramatically—in degree and quality. We must ensure that no policies—principal evaluation included—widen existing gaps in learning opportunities across schools and districts. In many ways, principals in lower performing districts deserve the most supportive principal evaluation policies and ones that are tailored to the particular demands of their job.