Abstract
Debate over the closure of DeVasco High School shows that data-driven accountability was a methodological and administrative processes that produced both transparency and opacity. Data, when applied to a system of accountability, produced new capabilities and powers, and as such were political. It created second-hand representations of important objects of analysis. Using these representations administrators spoke on behalf of the school, the student and the classroom, without having to rely on the first-person accounts of students, teachers or principals. They empowered one group—central city administrators—over another—teachers and principals. After analyzing the form these policies took, this article concludes that it is necessary to rethink the processes that create visibility and invisibility. Public data obscured the voices, experiences and collective traumas of students and faculty within the school. A narrow focus on activities within the schools rendered invisible the structural decisions made by the Department of Education in New York City—to favor small schools over large, comprehensive ones. In order to create understanding, and a sense of common purpose, those who are spoken for in simplified data must also be given the opportunity to debate the representations of their performance and quality.
Introduction
In New York City, the Department of Education (DOE) used a data-driven accountability system to both make policy decisions and share with the public evidence and reasoning for making these choices. The system produced knowledge about schools and the legitimacy to act within them. Administrators used simplified or numeric data on student, teacher, and school performance. They used this data to apply sanctions and rewards to schools, including closing ones that were deemed low performing. The creators of this system hoped parents and educators would use data to hold schools and themselves accountable to high standards.
Leaders of the school reforms also posited that data-driven accountability could create an environment of mutual learning among educators and spur public democratic participation. The first head of the Office of Accountability, James Liebman, wrote: The reform grows out of and contributes to a new form of [administrative] collaboration and new forms of public action. It thus redefines the separation of powers and recasts the administrative state more generally while opening the way to new forms of citizen participation … At the limit, school reform raises the prospect of a broader redefinition of our very democracy. (Liebman and Sabel, 2003: 184)
In New York City and elsewhere, this optimistic vision of data, democracy and school reform rests in tatters with city residents and especially teachers frustrated by a lack of meaningful say in school governance 1 (Henig et al., 2011). Governing with data has resulted in acrimony over the most controversial decisions, especially school closures, and distrust among long-time educators (Otterman, 2010). What follows is an examination of the relationship between city administrators and educators in a school that performed poorly on evaluations. This case is particularly elucidating because both evaluators and educators were deeply engaged in using data and the accountability system to pursue their own aims. What are the local politics of governing with data? How can this practice be brought more closely in line with the ideal of deliberation and democratic participation? To answer these questions this article draws on literature from Actor-network Theory to explore both the perverse and productive forms these politics take.
This analysis shows that the evaluative system produced new capabilities and powers, and as such was political. It created second-hand representations of important objects of analysis. Using these representations administrators spoke on behalf of the school, the student and the classroom, without having to rely on the first-person accounts of students, teachers or principals. They empowered one group—central city administrators—over another—teachers and principals. After analyzing the form these policies took, this article concludes that it is necessary to rethink the processes that create visibility and invisibility. Public data obscured the voices, experiences and collective traumas of students and faculty within the school. A narrow focus on activities within the schools rendered invisible structural decisions made by the Department of Education in New York City—to favor small schools over large, comprehensive ones. These findings suggest that data-driven accountability should be thought of as a process that allows interested parties to debate representations of performance and quality.
The research that follows relies mostly on first-hand accounts of educators and activists at a local high school that had been rated poorly on the Department of Education’s data-driven accountability system. Data, as argued more fully below, is a medium that is decontextualized from local circumstances. Data creates representations that can be understood at a distance from the phenomena they seek to describe (Porter, 1995). Over the course of five months, I re-contextualized the production of data-driven accountability, the circumstances that had led to poor performance at the school, and ultimately the decision to close it.
I focused on the controversy over the closure of a school in the Bronx, which I call DeVasco High School. 2 This research took place while the school was being closed. 3 Because of the debate that ensued, this proved to be an ideal site for understanding data-driven accountability politics. Staff conducted a long-term and active debate with the Department of Education over the findings of its evaluation systems. The research extended beyond the walls of the school into the administrative practices of the Department of Education. I interviewed administrators who created the accountability systems and watched evaluations as they were taking place through participant observation at school reviews. From this debate I reveal the politics of governing with data. 4
An Actor-network Theory of data-driven accountability
Data-driven accountability uses new means to achieve an old goal: to marry the scientific work of knowing and the political work of deciding. As Haridomos Tsoukas wrote in The Tyranny of Light (1997: 828) the making of a transparent self-regulating society rests on an assumption that has been “irresistibly powerful since the first days of the Enlightenment: the more human beings know the more able they will be to control their destiny.” Control is a political problem. In a society that uses knowledge to govern, control requires that experts and administrators have an informed understanding of problems. It also requires that the public accept the claims of these authorities. James Liebman took this one step further, arguing that such systems could become new means of democratic empowerment. The New York City Department of Education under Mayor Bloomberg supported this vision (New York City Department of Education, 2007).
Some scholars have argued that simplified representations of social phenomena result in a loss of meaning, clarity or richness that is inherently harmful. Tsoukas wrote that rich context-specific knowledge becomes mere thin information. Knowledge becomes “objectified, de-contextualized, timeless, impersonal, value free representations to be used instrumentally” (Tsoukas, 1997: 831). For him, this type of information, “undermines the human capacity for understanding. The self-referential world of information, combined with the ocean of instantly available, evanescent images and information items, weaken the human ability to form a coherent understanding of the issues at hand” (p. 839). In relationship to data, accountability, and rational, quantitative management, a variety of theorists have expressed fears of technocracy (Fischer, 1990), technical rationality supplanting more nuanced reason (Toulmin, 2003), and the idea that “power determines what counts as knowledge” (Flyvbjerg, 1998: 226, for an extended account of Flyvbjerg's preference for situated ‘phronetic social science' see Flyvbjerg, 2001). In this telling, data, quantification and technical rationality are reductive; eliding, diminishing and distorting some “real” reality or limiting what would otherwise be deeper forms of knowing.
The analysis that follows approaches the study of quantification and data from a different perspective, that of Actor-network Theory. From this perspective, data-driven accountability is a process of construction. We must understand what it is made of, how it becomes durable, and how it is remade or changed. Actor-network Theory is useful for thinking about these politics because of the focus on technologically mediated authority and citizenship. Its chief proponents, John Law, Bruno Latour, and Michel Callon, called on analysts to focus on the intersection of cultural, social and technological change as a locus of power. In their empirical work they examined how specific scientific and technical changes were constructed through heterogeneous relationships between human and non-human actors (Latour, 2004; Law, 2006, 2009; Callon et al., 2009). They argued that knowledge, technology and even abstract concepts like the economy efficiency, democracy or, in our case accountability, can be usefully interrogated by examining the human–technology relationships that produce them (Latour, 2002; Mitchell, 2011; West 2016). Indeed an important metaphor for these theorists is the cyborg, “a fantastical combination of bodies and machines” that, according to Gandy (2005) produces new kinds of agency. Data-driven accountability is a cyborg entity that rearranges relationships of power and produces new kinds of abilities for actors.
Four key concepts from Actor-network Theory are important when considering the politics of data: spokespersonship, matters of fact/ matters of concern, obligatory passage points and hybrid forums.
Data produces representations designed to speak on behalf of newly constituted objects of analysis. An actor who represents others is a spokesperson (Disch, 2008; Latour, 2002). A spokesperson can be either a human or a non-human actor. In what follows, data representing school quality is a spokesperson, and a teacher with a bullhorn at a school closure meeting is a spokesperson. The crucial capacity of a spokesperson is the ability to represent or speak on behalf of some individual, group or practice. The material qualities of spokespersons influence both actions and consequences. A teacher speaking in her school does not have the same longevity or durability as data that is recorded, analyzed and circulated by administrators with executive power. The success or strength of spokespersonship is related to the person’s entanglement with other human and non-human actors.
When administrators wielding data become the primary spokespersons for schools, problems that are unaccounted for in the data become less apparent. For example, the history, context and traumas of a school may be concealed if they are not readily translated into simple information. Data and technical rhetoric become obligatory passage points that determine who or what is included and excluded from representations (Callon, 1986). Information, experiences, and practices translated into data used in an accountability system move, circulate and become established, while untranslated experience remains local or obscure to decision-making processes. Because of its formatting, some experience may become unspoken-for, invisible, or opaque to the system of data-driven accountability.
Data-driven accountability relies on technical representations such as numbers, letter grades, percentiles or phrases on a spreadsheet or in a report. How this data was constructed (i.e. the work, politics and decisions that went into making it) is often unquestioned or unknown. In the words of Latour such data is matters of fact. As described above, these facts may speak on behalf of some group, collectivity or reality. Those who contest these representations often must find ways to turn what had been seen as simple fact into issues up for debate, or what Latour calls matters of concern. Concern may arise from a re-analysis of how data is constructed, or it may arise from other considerations that precipitate the re-evaluation of the meaning of those facts. Those who successfully contest data-driven accountability must turn simplified matters of fact into publicly debated matters of concern (Latour, 2005: 39).
Consequentially, effective public or political action requires that critics become entangled with the technical means through which data is constructed and deployed. The human–technology relationships that produce accountability also transform how and where public engagement takes place and the strategies that make it successful. As Mathew Gandy (2005) wrote: [This form of] public realm, with its hybridized conceptions of agency, is quite different from that envisaged within Habermasian philosophical traditions because of the radical extension of human agency to encompass those technical and organizational systems which the Frankfurt School philosophers sought to specifically exclude from the realm of ethical and political judgment. (p. 33)
Making and contesting data-driven accountability
The case that follows examines the construction of a data-driven accountability system by the Department of Education in New York City. It also traces the efforts of a group of school-based educators who sought to engage in this process to prevent their school from closing. Based on this case, the conclusion will examine new powers and problems created by the accountability system and what lessons can be learned for improving this and other systems like it.
Accountability system design
In 2006, the New York City Department of Education created a new school accountability system as part of Mayor Michael Bloomberg’s Children First policy. 5 This involved the restructuring of administrative responsibility for schools, changes in public procedures for participating in governance, the creation of new evaluative tools, and the creation of new kinds of sanctions. What follows focuses narrowly on the production of new evaluative processes and then examines how these tools fit into the overall accountability process. According to Liebman, the accountability system was designed to engage students, parents and especially educators in collective inquiry. This inquiry was supposed to achieve two goals: improving all schools, and focusing attention on the most challenging students—those whose first language is not English (called ELL or English Language Learners) and those with learning and behavioral deficiencies. Liebman became head of the office after having written passionately about the potential for public engagement through the realignment of administrative data-gathering practices (Liebman and Sabel, 2003).
To evaluate schools, the Office of Accountability created two tools to be simple enough for the general public to understand and to focus attention on progress made by at-risk students. The first, called the Progress Report, was designed as a “lagging indicator” of school quality, meaning that it assessed data on student outcomes, particularly on standardized tests, credit accumulation and on-time graduation. The second, called Quality Review, was designed as a “leading indicator” of school quality, meaning that the task of evaluators was to examine schools before testing and graduation had occurred to understand the conditions at a particular school. 6
Progress Report
The Progress Report used quantitative information to give each school a summative grade, an A, B, C, D, or F. The Progress Report was the most consequential and controversial aspect of the accountability measures. Schools that received grades below a B were targeted for closure (Interview with Mark 2012).
As displayed in Figure 1 below, Progress Reports were scored out of 100 points with the possibility of 16 bonus points.
Progress report categories and points. Note: Adapted from “High School Progress Report” (Department of Education, 2012).
As Figure 1 indicates, Progress Reports measures of student attainment accounted for 85 percent of an evaluation. This measurement relied on data on the number of students who graduate and the numbers of students who pass classes and tests. Data on the characteristics of students—whether they fit into special needs’ categories—were used in combination with measures of attainment for the Closing the Achievement Gap score. The School Environment came from a school-wide survey which accounted for the final 15 points.
Liebman sought to embed three ethical goals into the Progress Report (Interview with Liebman 2012). First he created a platform and methodology for evaluating schools with the goal of making it comprehensible and calculable for parents and teachers. The Progress Report displayed a simple letter grade at the top and explained the components that made up that grade on a single page with graphics to make it easily understandable (See Appendix 1 for an example). The methodology of the progress report used only basic math, rather than more complicated statistical techniques, so parents and educators could understand how a school was graded (Interview with Liebman, 2012, Interview with Adriana, 2012, Interview with David 2012).
Second, Liebman and his successors created a system of evaluation that emphasized improved outcomes for the most challenging students. This is most obviously reflected in the Closing the Achievement Gap category, which awards points for improved outcomes for vulnerable students. It is also reflected in the way the Student Progress metric is calculated. The metric scored progress towards graduation of all students in one calculation and progress towards graduation specifically for students in the lowest third of academic performance in a second. 7 Both were given equal weight (Interview with Liebman, 2012, Interview with Adriana, 2012, Interview with David 2012).
Third, he set out to create a system that would grade schools based on improved academic outcomes for students while controlling for the socioeconomic and educational background of the student population. The calculation was to “reflect each school’s contribution to student achievement, no matter where each child begins his or her journey” (New York City Department of Education, Office of Accountability, 2012a). To achieve this goal, the Progress Report measured school outcomes relative to 40 other schools with similar educational makeup using the Peer Index. “The Peer Index is used to sort schools on the basis of student population. A lower Peer Index indicates a higher-need population” (New York City Department of Education, Office of Accountability, 2012a). A mathematical formula was used to determine the peer schools for each school being evaluated: The formula was as follows
Seventy-five percent of the Student Progress category was derived through this calculation, and the other 25 percent was derived by comparing the school under evaluation to all city schools. Using this formula, the Progress Report was designed to control for the characteristics of the student population and measure the impact of the school. The Progress Report was, for the most part, 8 a reorganization of administrative data tied with new calculative practices and tools for sharing information. The Quality Review, by contrast, generated new data through school-based evaluations.
Quality review
The core component of the Quality Review was a three-day school visit by highly experienced educators, such as former principals and superintendents. 9 Based on this assessment, schools were given a grade of (U) Underdeveloped, (D) Developing, (P) Proficient, or (WD) Well Developed. Reviews were scheduled ahead of time, allowing a school to prepare for evaluation. The review itself consisted of classroom observation and meetings with the principal, students, parents, and teachers.
When launched, independent consultants were used as evaluators. They often found schools that had performed poorly on the Progress Report were well-functioning or were facing challenges outside of their control. According to David, who redesigned the Quality Review in 2008, the Department of Education and the Office of Accountability were dissatisfied when too many schools performed well on the review after approximately 85 percent of Quality Reviews deemed schools proficient or better during the first years it was in use 10 (Interview with David, 2012). In 2009, the Department of Education began administering the Quality Review and reorganized to make the reviews more difficult. This resulted in 60 percent of schools being rated proficient or above, a decrease of 25 percent in two years. To achieve this goal, David created a system that tied the first-hand assessment of experienced educators more tightly with narrow categories and data.
The revised system used new assessment tools to tie reviewers’ observations to categories of school practices defined by the Department of Education in the Quality Review Conceptual Framework. Reviewers were required to use a rubric that was contained within this document. The rubric identified three categories and 10 indicators for assessment, which are listed in Figure 2 below.
Quality review rubric. Note: Adapted from “2012–2013 Quality Review Conceptual Framework” (New York City Department of Education, Office of Accountability, 2012b).
For each indicator, evaluators categorized school performance to be Underdeveloped, Developing, Proficient, or Well Developed. The Quality Review Conceptual Framework document defined each category for each indicator, so that, for example, the curriculum indicator defined underdeveloped, in part, as “School leaders do not consistently align curricula to State standards …” (New York City Department of Education, Office of Accountability, 2012b).
The rubric prioritized two kinds of practices: First it encouraged teaching and learning that was “aligned” with State standards. Aligned practices were oriented towards mastery of concepts deemed important by the New York State Board of Regents, which was reflected in the Regents Examinations.
11
An aligned curriculum as defined in the rubric would teach students what they needed to know to perform well on the Regents Examinations. Pedagogy was assessed to the extent which it was aligned to the curriculum. Assessment, to take a third, was to be “aligned with key standards and curricula” (New York City Department of Education, Office of Accountability, 2012b). Below, each category that stressed alignment is bolded in Figure 3 below.
Quality review rubric categories that emphasize “alignment”. Note: Adapted from “2012–2013 Quality Review Conceptual Framework” (New York City Department of Education, Office of Accountability, 2012b).
The Quality Review rubric encouraged reviewers to ensure that teaching and management practices were aligned with the goal of preparing students for Regents Examinations.
Using school data was the second practice encouraged by the rubric. For example, a “well developed” curriculum was “planned and refined using student work and data.”
“Well developed” goals and action plans were “informed by a comprehensive, data-driven needs assessment and ongoing data gathering and analysis” (New York City Department of Education, Office of Accountability, 2012b). Below, in Figure 4, each category that stressed using data is underlined.
Quality review rubric categories that emphasize “data”. Note: Adapted from “2012–2013 Quality Review Conceptual Framework” (New York City Department of Education, Office of Accountability, 2012b).
Evaluators focused on the extent to which teachers used data because the Quality Review rubric emphasized this practice.
In addition to the Quality Review rubric, reviewers relied on, past Progress Reports to guide their evaluation of schools. Prior to visiting schools, reviewers used Progress Reports to identify areas that appeared successful or problematic. With this tool reviewers would structure their evaluation and train their focus on specific areas. During trainings, evaluators were taught how to use Progress Reports as part of their evaluative practice. 12
The Quality Review process, which relied on the first-hand experience of well-trained educators, emphasized school-based practices focused either on data or on alignment with the tests that would improve that data. In this way the most qualitative portion of the evaluation process served to further embed the norm of emphasizing quantitative data as the benchmark for school performance. The first-person accounts of highly experienced reviewers were controlled and circumscribed through use of the rubric.
Research from outside the Department of Education indicated that despite the aim of not penalizing schools with challenging students, the assessments on both the Progress Report and the Quality Review were correlated with the presence of students who were in vulnerable categories (Independent Budget Office (IBO), 2012). These analyses indicated that what was revealed by the Progress Report also concealed other realities taking place in schools. To understand these dynamics more fully, we now turn to the case of DeVasco.
DeVasco High School closure controversy reveals structural changes and collective trauma
DeVasco was a large public high school in the Bronx, at one time enrolling about 4,500 students. After years of poor performance, it was slated for phase out in the spring of 2010, meaning that it would stop taking incoming classes of students (Leanne, 2012). From 2002 to 2015 when the school closed, DeVasco experienced changes that were not within the control of the school-based educators and had dramatic impact on teaching and learning. Two kinds of changes are highlighted in what follows: First, the Department of Education undertook structural and administrative changes that favored small schools over large comprehensive ones (like DeVasco). Second, the Department of Education burdened the school with too many students over the course of several years, leading to an unproductive and ultimately unsafe working environment. This led to collective traumas that endured for years.
Small schools, data and “protection”
The fate of DeVasco is linked with a long-term trend in school policy that pre-dated the creation of the data-driven accountability system. Starting in the mid-1990s the Board of Education began closing large public schools and replacing them with smaller schools of between approximately 150 and 400 students. As an administrator who had been active at the time said:
13
Julia Richmond (the first school closed) was an appallingly poor performing high school. It was shut down in the mid-1990s, phased out and replaced with four or five schools that are there now … Those schools are more successful than the large high school they replaced, but I’d say close to six or seven years went by before anyone acknowledged that this was an effective reform strategy.” (Interview with Edward 2012)
Proponents of these reforms argued that creating small schools and holding poor performing schools to standards was a way of supporting and developing innovative programs. Small schools were described as firms that would innovate and focus on improvement. As a top administrator in the Department of Education said: Regardless of whether you’re a manager of a private sector or public sector endeavor the job is always the same. You have to go out and recruit the best people you can find. You have to support them. You have to develop them. You have to incent[ivize] them when they do good work. You have to protect them from outside interference, and you have to hold them accountable to the highest standards of performance, and that is probably the best definition of management I’ve ever heard. (Interview with Edward, 2012)
Data, rhetoric, and dissent
A critic of small schools argued that these innovations were being protected in another way: statistically. Crystal, an education specialist who worked for the United Federation of Teachers (UFT) at DeVasco, said: The need to prove the statistical point about small schools, I think, drove them to shelter the schools in the first two years of their existence. They did not need to take English Language Learners or special needs students unless they were schools designed specifically to serve those populations. (Interview with Crystal, 2012)
Activists at DeVasco argued that the statistical protection of new small schools led to vulnerability of their comprehensive high school. This group of activists-consisting of teachers, administrators and parents-coalesced around two leaders: Crystal, a union representative, teacher and administrator, and Leanne, the principal. In response to the first threat of closure in 2008, they organized themselves into an ad hoc research group. Through this research they showed that their school had been subject to an influx of high needs students 14 (Interview with Leanne, 2013; Interview with Crystal, 2012).
They argued that the growth of small schools was an important factor in the changing population at DeVasco. Small schools, for the first two years of their operation, were allowed to undertake selective enrollment, capping the numbers of ELL and special education students. When small schools were opened up within the DeVasco building, these more difficult students would often end up in the original DeVasco High School. Moreover, students who failed to do well in the new smaller schools were “counseled out” or encouraged to enroll elsewhere. Leanne, the principal at De Vasco, described the process as follows: [Other new schools in the building] take all the kids with undesirable attendance and grade point averages … and they bring them into the principal’s office and the principal goes, “so which transfer school do you want to go to?”. (2012)
DeVasco was an open-enrollment school, meaning that it had to accept all students, including those excluded from or counseled out of small schools in the building or nearby. Moreover Leanne had a strong ethic of enrolling students that other schools had asked to leave (Interview with Leanne, 2013; Interviews with Teachers 1 and 3, 2013). From 2002 to 2010 DeVasco took on more under-performing students.
The principal also argued that small schools “poached” the strongest programs from DeVasco. Small schools enrolled particularly motivated students into programs that focused on, for example, mathematics, or teacher preparation (Interview with Leanne, 2013). Administrators and educators at large schools argued that they were tasked with teaching the students small schools left behind. In some cases this was blatant. At DeVasco, in 2004, the Department of Education co-located a small school specializing in mathematics that drew in many of the students who were the strongest academic performers. This smaller school hired the top teaching talent at DeVasco who had run an advanced mathematics program (Interview with Leanne, 2013). One group of critics referred to large schools, like DeVasco as the “host” and the smaller schools as “parasites” (Interview with Jon and Alan, 2013).15
With closure looming, Crystal and Leanne sought to mobilize insights about the changing population at DeVasco and its consequences in a form that would be relevant to Department of Education administrators. Using the same categories of analysis as the Progress Report, they produced reports to describe the problems with sorting students. One way of measuring the change in students at DeVasco was to look at the past performance of incoming ninth-grade students. In a letter sent to a New York State school administrator, activists showed that, in 1998, 41 percent and 56 percent of entering freshmen and sophomores met testing standards in reading and math respectively. In 1999, the percentage of students meeting standards at entry dropped precipitously to 28 percent in reading and 12 percent in math. Subsequent cohorts in reading had consistently lower percentages of students meeting standards at entry, falling to 6 percent in reading in 2005. The eighth-grade math scores of entering students held steady. DeVasco High School also received many students departing schools specifically designed to address high-needs children. In 2008 and 2009 this included 34 students from District 75 16 and 34 from District 79 (Letter to Regent Tisch, 2009). 17
In this same report the principal of DeVasco used data to argue that DeVasco had become a catch-all for students transferring between schools. She showed that of the 2005 cohort, which was graduating in 2009, only 14 percent arrived on the first day of class as freshmen, while 42 percent arrived subsequently that year, 21 percent entered as sophomores, 13 percent entered as juniors, and 10 percent entered during senior year. In 2009, for example, DeVasco received 359 students after the first day of class. These students came in waves, with approximately 130 arriving in September and October (80 in October alone) and well over 100 arriving from February through June, with nearly 80 arriving in February. Transfers came after leaving other schools within the district, serving time in prison, and transferring from out of the district, state or country. 18 Often times they required remedial help because of the disruption in their education, or socio-emotional help because of their dislocation. Also, teachers had less time to prepare these students before state tests were administered in March (Letter to Regent Tisch, 2009).
The small schools sharing building space with DeVasco enrolled fewer students, leading to overcrowding at DeVasco. In fall 2002 DeVasco shared space with one small school, but by fall 2003, it shared space with five. In the 2003 school year, DeVasco had so many students that it held double shifts of school, with one group of students coming for classes in the morning and a second group arriving in the early afternoon (Interview with Crystal, 2012). Classes were held in the library, gym and auditorium, often with separate classes being taught in different corners of the same room. The school was crowded with 124 students per classroom that year. According to a report created by Crystal, in 2002 the utilization rate19 at DeVasco was 158 percent, the highest of any school in the region. In 2004, the utilization rate peaked at 180 percent and then dipped to 163 percent in 2005 (Letter to Regent Tische, 2009).
The overcrowding created a depressing and unsafe learning and working environment, damaged the school’s reputation, and set it apart from the co-located small schools. In an interview, the principal described the demoralization students and staff felt: It was crazy, [we had juniors and seniors come at] so let’s say seven o’clock till, I think it was 12 o’clock. Then the freshmen and sophomores came in a little after to 12 and they didn’t leave till 6. Only [DeVasco High School]; all the other schools had a normal schedule. That was atrocious. It was just so horrible for the kids, for the parents. We were treated differently. I mean just the whole fact that we were treated differently was such a mental attack, emotional attack on my kids. It’s horrendous. It’s unforgivable what they did to the kids. (Leanne Interview, 2013)
In addition to showing that DeVasco faced more challenging demographics and over-enrollment, Crystal used the data from the accountability system to come up with still other explanations for poor performance. Crystal’s research showed that the accountability system produced perverse effects in two ways. First, she found that because their school was at the very bottom of all schools in New York City in terms of performance and in terms of the students they were serving, the peer schools they were being compared to had significantly fewer of the most challenging students. In 2004, for example, DeVasco enrolled nearly double the number of ELL and special education students as the peer schools to which it was compared (Letter to Regent Tisch, 2009).
Second, activists found they were being penalized for taking on students who had no data history. Students with no data history may have been coming into the New York City education system for the first time, often from abroad, or they may have been returning from juvenile detention or another hiatus in their education. By the 2007 school year, DeVasco had a high percentage of these students. The Department of Education used an approximation of what the performance of such students should be on future tests and other metrics. So, for example, if a student entered the school without an eighth-grade test score, she would be assumed to be able to perform at a certain level of proficiency over the course of her career. Activists at DeVasco used city-wide data to argue that the estimates of the Department of Education were incorrect in the assumed test scores. If a more accurate estimate of the performance of students with no data history was used, the school would have performed well enough to avoid closures (Crystal 2013).
Ultimately, a final decision to close the school put an end to the research and activism collective assembled at DeVasco. On the strength of their research, Crystal and other researchers at DeVasco were able to garner the support of prominent officials in the State Board of Education and officials within New Visions for New Schools. These individuals had no actual power in deciding if a school would stay open or close. Activists conveyed their arguments to city officials through email and at public closure hearings. City officials with power over the school acknowledged the issues identified by activists but said their process for reviewing the accountability system would take place after a final decision about school closure was made (Crystal 2013).
In the public process for making final decisions about school closure, the voices of opposition were ignored. A seven-person panel appointed by the mayor made final decisions about school closures. The Panel for Education Policy held a single public hearing for each closed school. At the closure meetings the panel did not engage in discussion with individuals in favor of or opposed to school closures. The meetings, usually held in a high school auditorium, consisted of the panel making a presentation on stage and then members of the audience making their arguments and pleas. At public hearings, city officials were not required to—and did not—reply to the claims of activists (Otterman, 2010; Editorial Board of The New York Times, 2010). This very limited process resulted in a decision by the Department of Education Panel for Education Policy to endorse the closure of DeVasco. 20
Analysis and conclusion: New powers, new opacities, and democratic possibilities
Based on the analysis of the instruments and procedures used to evaluate schools and the example of DeVasco, what are the politics of data-driven accountability evident in this case? Focusing the systems technologies and public processes that produce accountability gives a set of answers to this question that is largely unexplored in existent literature on school politics in New York City, which tend to view data and accountability as either objective conduits of information or high-handed impositions by the mayor and a cadre of well-funded education reform advocates. Lost in this debate is an understanding of the specific kinds of powers that the data-driven accountability created, the kinds of problems that these new power relationships engendered and what can be done to come closer to Liebman’s goal of enhancing democracy.
The powers of accountability
As Latour and other ANT scholars argue, we must explain “how domination has become so efficacious and through which unlikely means” (Latour, 2005: 86). This requires returning continuously to the technological and human assemblages that produced new capacities in the case. These powers are related to the concepts explored earlier:
– Spokespersonship—the ability to represent, or speak on behalf of groups and individuals; – Matters of fact and matters of concern—the capability to determine what is settled fact and what is up for debate; and – Obligatory passage points—the capacity to share specific kinds of information more widely; – Hybrid forums—the ability of actors to share, evaluate and modify information is shaped by the technological, bureaucratic and physical spaces where information is exchanged and debated.
These concepts will help to render more precise the kinds of power that data-driven accountability created in what follows.
The first kind of power that arises from governing with data is changing the location of expertise. Translating schools into simplified data using the Progress Report and Quality Review enhanced the power of those outside the school and diminished the power of the educators and schools subject to scrutiny. The voice of socially sanctioned professionals, such as teachers at DeVasco, was replaced with trust in the accountability system and the diagnoses of analysts working in the DOE bureaucracy.21 Before data-driven accountability became a common practice, scholars of school management lamented the fact that teachers could ignore reform efforts because of their autonomy within the classroom (Hess, 1999; Weick, 1976). Now, with audit systems in place, administrators can make determinations about the quality of a student without talking to her, the pedagogy of a teacher without entering her classroom or the performance of a school without ever visiting it. “Transparency inevitably presupposes a subject: transparent to who?” Tsoukas (1997: 834). The Progress Report and Quality Review made schools transparent to central city administrators by becoming a new, powerful spokesperson for school success. 22
A second power was creating new categories, discourses and techniques for representing objects of analysis. The designers of the assessment tools reinforced the importance of certain categories, especially ELL and special education students. Geoffrey Bowker and Susan Leigh Star have been particularly interested in the material and ideological production of standards and categories. They have argued “each category valorizes some point of view and silences another. This was not inherently a bad thing—indeed it is inescapable. But it was an ethical choice, and as such it is dangerous—not bad, but dangerous” (1999: 6). The designers of the data-driven accountability system sought to produce widely valued outcomes by reinforcing the importance of categories that group and represent vulnerable students. 23
When tied with sanction and reward, the uniformity of these categories became an object of scrutiny and strategy for principals through the enrollment process. New schools on the DeVasco campus sorted out students with a poor data history, who showed little likelihood of improving, or who had parents who were less committed to their success. When educators used their local and situated knowledge to invite or discourage students to enroll in specific schools, they turned categories that appeared as matters of fact into matters of concern, opening up new and unintended modes of action.
A third, related, kind of power arising from data-driven accountability comes from the ability to make selective parts of reality visible or invisible. In the extreme Strathern (2000a: 309) argues that “visibility as a conduit for knowledge is elided with visibility as an instrument for control.” The auditor has the power to define what aspects of social reality “count” by making certain aspects of social reality that were formerly concealed more widely visible to new audiences. In this sense, “one kind of reality is knowingly eclipsed” and another is valorized and used for making policy decisions (Strathern, 2000a: 309). Systems of accountability render opaque phenomena that are not measured or represented in data.
The power to make visible also produced its opposite, opacities that undermined understanding but enhanced control. The tools and procedures of creating data-driven accountability created three important opacities: Opacity One: Accountability was a one-way window focused narrowly on teachers and schools: By focusing on the impacts of teachers and schools and ‘controlling for’ other variables, it excluded from view the roles of parents, students and even the Department of Education itself. Opacity Two: The perspective of school-based personnel was not meaningfully taken into account. During the evaluative processes neither school-based educators nor even the reviewers tasked with evaluating schools for the Quality Review had significant opportunity to make observations that fell outside of a tightly controlled rubric or defend their own stories about the strengths and weaknesses of schools. Opacity Three: History, context and trauma impact school performance in ways that are invisible to the accountability system. DeVasco underwent traumas and slights that had cumulative effects which could not be addressed statistically or through the use of the Peer Index method. Over the course of 10 years, DeVasco received an influx of more challenging students, became overcrowded, lost its prestige programs and became a refuge for students rejected elsewhere. This slow accretion of slights culminated in a year-long overcrowding ordeal punctuated by an episode of violent trauma. These cumulative and collective traumas and slights effected student performance data used in the accountability system. Students in the building during these disruptions struggled to perform on tests, to pass courses and to graduate. The underlying causes and historical context leading to these problems went undetected because this history and context could not be translated into a system that measured year-to-year performance change through data.
The last power of accountability was changing the essential qualities of what is being studied. As Steven Woolgar writes: Audit cultures redefine and re-establish the very nature of ideas and their values. They do so through various processes of stripping, decontextualization and abstraction and through the introduction and use of technologies of measurement. Like all technologies of measurement, these have the capacity to redefine the essential qualities of what is being measured. (2004: 453)
Practices of accountability reoriented school-based activities around the production and examination of data based on school tests. Progress Reports used test scores to derive the majority of its letter grades for schools. Other scholars have noted that this created an incentive to improve performance by teaching to the test, narrowing what is taught to what is tested (Ravitch, 2010: 155). The Quality Review judged curriculum based on its alignment with testing questions. It also judged whether schools were embedding the analysis of data in their pedagogic practice. When data, rather than professional relationships, became the medium for displaying success or failure, it radically changed what it set out to measure.
Democratic possibilities
With regards to data, accountability, and democracy, it is important to recall the idealistic sentiments of James Liebman, the first chief of the Office of Accountability and the designer of the evaluative tools. As quoted above he argued that making school practices more visible through data and accountability had the potential to transform administrative practice, and “at the limit, democracy itself.” There is much in the story of DeVasco to cast doubt on this hopeful vision, and there are also lessons that can be drawn to come closer to this goal. Activists and critics pushed for public deliberation not only with the bullhorn and the placard, but also with the spreadsheet and calculations. From their work we can take away three important lessons for making data-driven accountability more democratic.
First, making data democratic requires the kind of hybrid public space described above by Gandy, in which interested parties can examine, recalculate, debate over, and expand upon the meaning of data. Such a hybrid forum would more fully invite the evaluated into the process of defining the terms of evaluation. This new kind of agora would occur in places of public dialogue designed for exchanging ideas. Rather than an auditorium with a stage, we can envision a room in which administrators, educators, and the public could face each other as equals. In additional to this physical space, we must also imagine a digital, calculative space, in which all interested parties and wider publics can parse and question the meaning of data. Pushing our imagination further, we can imagine organizing, educating and creating tools that enable non-experts to become involved in the process of governing with data. Such spaces, practices, and tools would promote multi-directional accountability, rather than a one-way window.
Second, since accountability is inherently political, it requires checks and balances. The Department of Education held both evaluative and administrative authority in the case of DeVasco. Under such conditions the deliberative and experimental potential of governing with data was stymied. An educator is less likely to participate earnestly and honestly in analysis of data if her sole motivation is avoiding being fired. The public is less likely to trust the numbers and participate in producing them if they have no say in what is being counted and how the information will be used. Administrative and evaluative authority should not be held by the same body. An outside party should take responsibility for school evaluation. 24
Finally, school evaluation cannot be successful if it is unable to account for history, context, and lived experience. An incessant focus on forms of data that have been deemed important by both federal and local departments of education has severed the passage between the experiences and practices that are relevant to practitioners and those that count as school success. Accountability systems must widen the obligatory passage point through which information flows. To do so, other kinds of data must become part of evaluation. School histories, faculty profiles, and long-form examples of successes and failures in writings, videos or other media would contain important and relevant information that is currently missed by a focus on quantitative assessment. If the grind of teaching challenging students in a hostile administrative environment was visible to wider audiences, it might elicit empathy and support, as well as the civic engagement Liebman so prized. Accountability reconceived could be a conduit for such understanding.
Primary sources
Department of Education Interview Subjects
Adriana Interview. 2012 Interview by John West. Digital Recording.
Cathy Interview. 2012 Interview by John West. Digital Recording.
David Interview. 2012 Interview by John West. Written Notes.
Edward Interview. 2012 Interview by John West. Digital Recording.
Jane Interview. 2012 Interview by John West. Digital Recording.
James Liebman Interview. 2012. Interview by John West. Written notes.
Mark Interview. 2012 Interview by John West. Digital Recording.
Mary Interview. 2012 Interview by John West. Digital Recording.
Pat Interview. 2012 Interview by John West. Digital Recording.
Roger Interview. 2012 Interview by John West. Digital Recording.
Valerie Interview. 2012 Interview by John West. Digital Recording.
DeVasco High School Interview Subjects
Crystal Interview. 2012 Interview by John West. Digital Recording.
Jaime Interview. 2012. Interview by John West. Digital Recording.
Leanne Interview. Interview. 2012 Interview by John West. Digital Recording.
Leanne Interview. Interview. 2013 Interview by John West. Digital Recording.
Jon and Alan Interview. 2013. Interview by John West. Digital Recording.
Teacher 1 Interview. 2013. Interview by John West. Digital Recording.
Teacher 2 Interview. 2013. Interview by John West. Digital Recording.
Teacher 3 Interview. 2013. Interview by John West. Digital Recording.
Academic, Union and Philanthropy Interviews
Jamie Interview. Interview. 2013 Interview by John West. Digital Recording.
Mitchel Interview. Interview. 2013 Interview by John West. Digital Recording.
School Based Research Documents
Crystal and Leanne. 2009. “Letter to Regent Tisch”
Crystal. 2009. “Letter to Sternberg”
Crystal and Pat. 2009. Internal Email Communication.
Crystal. 2013. “The Progress Report: An analysis of the 2009-10 progress report for Christopher Columbus High School”
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
This article is a part of special theme on Urban Governance. To see a full list of all articles in this special theme, please click here: http://journals.sagepub.com/page/bds/collections/urban-governance.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
