Sage Journals: Discover world-class research

Abstract

As districts across the United States move to implement comprehensive computer science education (CSed) initiatives, there is increasing interest in utilizing data-informed strategies such as continuous improvement to address systemic inequities. However, the specifics of how education institutions move from collecting and monitoring data into processes of systems improvement are not straightforward. This study explores the critical intermediary step of diagnostic problem framing which connects the collection of data to the identification of root causes of inequities so that adequate action can be taken to change an instructional system. Without taking the step of diagnostic problem framing, district leaders run the risk of using data to change systems in ways that simply reproduce or worsen existing inequities. Through four cases from three school districts implementing comprehensive (e.g. K12, at scale) CSed initiatives, we explore the dynamics of diagnostic problem framing around each of the four phenomena related to equitable CSed as specified in the CAPE framework: system and teacher capacity around CSed, student access to CS learning, participation rates in those opportunities by subgroups, and the quality of the experience of those participating. Our findings highlight five factors mediating diagnostic problem framing: conceptualization of equity as a multifaceted phenomenon, problem attribution to systems, not students, ability to analyze data for indicators of inequities, knowledge of local instructional systems, and embeddedness in instructional networks. Our results affirm the importance of developing robust data systems in CSed initiatives, but also show how attention to broader institutional factors can improve the capacity of these systems to address inequities.

Keywords

Computer Science education equity systems change data use practices diagnostic problem framing problem identification policy implementation organizational change organizational learning

Introduction

This study explores how district leaders use data to identify underlying root causes of inequities in the context of district-wide K12 computer science education (CSed) initiatives, highlighting exemplars of data use practices, along with contextual dynamics that supported and mediated the possibilities of using data to understand causes of inequities.

As efforts to transform K12 school systems to include equitable CSed have spread in the United States, there is increasing interest in utilizing data-driven strategies such as continuous improvement (DeLyser et al., 2020). Acknowledging that simply making CS learning opportunities available to students is insufficient, Fletcher and Warner (2020) provided a model encouraging education institutions to monitor data on four equity-oriented phenomena in order to advance equity-related goals: system and teacher capacity around CSed, student access to CS learning opportunities, participation rates in those opportunities by subgroups, and the quality of the experience of those participating (CAPE). However, the specifics of how education institutions move from collecting equity-related data in CSed into systems change processes that address inequities are not straightforward.

In this study, we focus on the critical intermediary step of diagnostic problem framing in which a problem is interpreted in ways that determine its parameters and responsible stakeholders (Coburn et al., 2009; Snow and Benford, 1992). Within decision-making processes, diagnostic problem framing sits between monitoring data and taking action to change an element of an instructional system. We explored the work district leaders perform to investigate the root causes of inequities within the context of comprehensive CSed initiatives. Our results highlighted how data on their own did not offer clear-cut implications for where and how to intervene in an instructional system to reduce inequities. We addressed the following questions:

1. How do district actors utilize data infrastructures and routines in order to understand possible root causes of inequity within comprehensive CSed initiatives?

2. What factors support or mediate such processes of diagnostic problem framing?

This study draws on data from four retrospective case studies from three school districts of varying size, region, and demographics. Each has implemented CSed initiatives for at least 2 years, providing a robust opportunity to examine variations in data use practices as they enacted multi-year reforms. Through interviews and district provided documentation, we collected data on how districts both conceptualized and addressed issues of equity in terms of who had access to CS, how CS was taught, and what computer science was taught (Santo et al., 2019). We selected cases related to one of each of the four phenomena outlined by Fletcher and Warner (2020) in the CAPE framework—system and teacher capacity, student access, participation, and experience. Next, we analyzed each case in terms of the institutional dynamics at play—organizational relationships, routines, roles, interpretation, learning, and decision-making—within the process of diagnostic problem framing and identification of causes of inequities that district actors observed in their data.

Our findings highlight five specific factors that mediated processes of diagnostic problem framing: (1) conceptualization of equity as a multifaceted phenomenon, (2) problem attribution to systems, not students, (3) ability to analyze data for indicators of inequities, (4) knowledge of local instructional systems, and (5) embeddedness in instructional networks. Problem attribution to systems, not students, is of particular importance in the context of Computer Science education reform efforts given the persistence of identity stereotypes about which students Computer Science is for, and which it is not (Lewis et al., 2016). The need for an orientation toward diagnostic problem framing is reinforced by prior research, as detailed in the background section below, that shows how school leaders can use data in ways that reproduce, rather than counteract, inequitable narratives, practices, and systems that negatively impact students (Datnow and Park, 2018; Dodman et al., 2021; Safir and Dugan, 2021).

Our results affirm the importance of developing robust data systems in CSed initiatives, but also show how attention to broader institutional factors and root causes can improve the capacity of these systems to address inequities.

Background

Within the field of computer science education, emerging scholarship as well as practitioner approaches have attended to whole district change as a critical site of intervention (e.g. Fanscali et al., 2018; Goode and Margolis, 2011; Proctor et al., 2019; Reed et al., 2015; Santo et al., 2020). Often, these efforts actively draw from frameworks around institutional learning and reform that have been established in other areas such as data-driven decision-making (DeLyser and Wright, 2019; DeLyser et al., 2020; Santo et al., 2019). Although data-driven approaches to reforming educational institutions vary in their size, scope, and strategy, policy discourse and data use guides tend to portray data-driven decision-making as an iterative cycle of the following steps: collecting and monitoring data to evidence problems such as disparate educational outcomes for students; characterizing the problem and solution space to identify root causes and responsible stakeholders; and taking action within the system to respond effectively to the problem (Coburn et al., 2009; Dodman et al., 2021).

The focus on district-level change in this study is set within a broader context of the Computer Science Education (CSed) field, which has actively been oriented toward issues of historical inequities within computing (Margolis et al., 2017; Master et al., 2016). Within CSed, there has been a need for frameworks that address the question of what forms of data might be collected related to equitable implementation at scale within school systems. One prominent framework, CAPE, which was developed in the context of the Expanding Computing Education Pathways Alliance (Ericson et al., 2016; Leftwich et al., 2020), offers four areas of focus for equity-centered systems change data: system and teacher capacity, student access to learning, participation in learning by subgroups, and quality of student experience in learning (Fletcher and Warner, 2020). Within this framework, these four areas are referred to as capacity, access, participation, and experience (CAPE) respectively.

Using the CAPE framework, then, can guide district leaders in their efforts to create whole district change that addresses inequities at multiple levels in their CSed work. The scholarship on data-driven approaches to institutional change offers wide-ranging practical guidance for how to collect and monitor data that could be applied to measuring capacity, access, participation, and experience. Scholars have shared guidance, for example, on how to build data infrastructures (Datnow et al., 2007), have leadership that promotes data use (Copland, 2003; Halverson et al., 2007), and establish organizational routines of data use and interpretation (Boudett et al., 2020; Spillane, 2012). Likewise, the scholarship on data use provides practical advice for how to enact change at a systemic level. Scholars have shared guidance, for example, on how to create dedicated spaces within institutions to continually design, implement, and track the effectiveness of new institutional practices (Sannino et al., 2016), and how to equitably partner with students (Phelps and Santo, 2021), families (Ishimaru, 2019), and communities (Bang et al., 2016), to create change together as well as to build capacity for future change. Within the context of CSed, scholars have also shared guiding principles of transformative justice and specific examples of how to enact these principles within institutional change efforts (Erete et al., 2021).

Yet, there is little research with clear guidance for district leaders on the intermediary step of diagnostic problem framing that exists between data collection and systems change. Whereas data collection and monitoring points to the existence, prevalence, and magnitude of a system-level inequity, diagnostic problem framing surfaces the underlying root causes of the inequity and locates the causes within the interplay of various elements of an educational institution. Scholars argue that uncovering the root causes of an inequity is slow and messy work, partly because educational institutions are riddled with uncertainties and complexities that obscure the root causes of inequities (Yurkofsky, 2022). Additionally, inequities have historically settled into the dominant narratives, practices, and systems of an educational institution such that they can continuously hide in plain sight (Safir and Dugan, 2021). Further still, educational institutions are populated with stakeholders that have competing agendas and conflicting interpretations of what an underlying root cause might be such that it can be difficult to reach agreement and consensus (Coburn et al., 2009).

Scholars have shown that district leaders who primarily couple their data use with accountability policies (such as the No Child Left Behind Act and the Every Child Succeeds Act in the United States) to measure school success on norm-referenced tests tend to skip over the slow and messy work entailed by diagnostic problem framing, and instead focus exclusively on developing instructional management solutions to remediate students rather than identify and respond to root causes of inequities (Dodman et al., 2021; Horn, 2018). Drawing on a number of cautionary tales from the research literature, we will show below how district leaders’ data-driven decision-making, within a high stakes accountability context, can reproduce inequitable narratives, practices, and systems within their educational institutions when diagnostic problem framing is done poorly and root causes of inequities remain intact.

Inequitable narratives can be reproduced, for example, when school leaders use student benchmark data to generate problem and solution frames around “closing the achievement gap.” The narrative and rhetoric surrounding the “achievement gap” has been shown to problematically position low-income and minoritized students as deficient and in need of remediation while maintaining measures of success that privilege middle-class White students (Gutiérrez, 2008; Ladson-Billings, 2006; Nasir, 2011). In their study of 52 educators across seven school districts, Lasater et al. (2020) found that data use that positioned assessment data as the sole measure of students’ academic capability “prompted” teachers to engage in deficit narratives (p. 5; italics in the original). Similarly in their study of school leaders at six middle schools, Bertrand and Marsh (2015) revealed that teachers’ sense-making around assessment data reinforced existing deficit narratives and low expectations for students designated as ELL and in special education.

Additionally, inequitable practices can be reproduced during the data use process. Within the context of accountability pressures, scholars have shown how school leaders use data to take up inequitable practices such as solely teaching to the test (Diamond and Cooper, 2007; Garner et al., 2017), performing educational triage only on students right below the benchmark cusp (Booher-Jennings, 2005; Horn, 2018), or reshaping the test pool by referring students last minute to special education services (Ehren and Swanborn, 2012; Jacob, 2005). When school leaders use data to inform these practices, it is not to reform inequitable educational institutions, but rather to engage in ceremonial, and sometimes harmful, compliance to accountability policies (Spillane, 2012).

Further still, inequitable systems can be reproduced through data-driven approaches. Horn and colleagues who closely followed the data use practices of teacher workgroups (Garner et al., 2017; Horn, 2018) argue that the narrow focus on instructional management solutions such as student triage and student remediation, marked a missed opportunity for workgroups to create more equitable solutions by transforming aspects of the instructional system. According to Horn (2018) equitable transformations of systems might include improving instruction to provide more robust and rigorous learning opportunities that leveraged students’ assets, or investing in professional development to better support teachers in improving instruction.

Taken together, these and related studies reveal how the predominant coupling of student achievement data with policy contexts leads to data-driven decision-making meetings that fail to engage in diagnostic problem-framing work (Garner et al., 2017; Horn, 2018), fail to evidence equity issues as part of the problem-framing working (Gannon-Slater et al., 2017; Schildkamp et al., 2012), or that explain equity issues in a way that attributes blame to the students themselves (Bertrand and Marsh, 2015; Booher-Jennings, 2005; Evans et al., 2019; Oláh et al., 2010).

On the flip side, when diagnostic problem framing is done in a thoughtful and responsible way, school leaders can identify and respond to root causes that reproduce harmful inequities in students’ day-to-day schooling experiences. For example, in the context of CSed, Sheena Erete and colleagues (Erete et al., 2021) worked with school leaders to better understand the root causes within a school district—stemming from historical injustices, ongoing state violence, and negative stereotypes—that were shaping the opportunities and willingness for Black and Latinx female students to participate in CSed learning programs. The root causes explored in this study disproportionately narrowed the entry points into CS learning opportunities for Black and Latinx female students as well as produced negative experiences within AP CS classrooms for Black and Latinx students who felt isolated due to experiencing a lack of representation of teachers, mentors, and peers that they could identify with.

By more intentionally exploring these root causes, the researchers and school leaders were informed to take action to remediate many components of the school system including the design of CSed learning opportunities, recruitment efforts, mentor hiring, and professional development. This study reveals the crucial role that diagnostic problem framing, when paired with a transformative justice approach to explicitly counter structural racism and oppression, plays in bridging the steps of data use and transformative change to provide more just outcomes for students in CSed.

The study we present builds on these findings by providing an in-depth analysis of the specific work school leaders performed to conduct diagnostic problem framing in a way that successfully surfaced and responded to the root causes of inequities in their CSed contexts. Given the lack of guidance on diagnostic problem framing within policy discourse, data use handbooks, and the research literature, and given the crucial role diagnostic problem framing plays for identifying and responding to inequitable narratives, practices, and systems, it is critically important to develop guidance and exemplar cases of how district leaders can use diagnostic problem framing to support their equitable systems change work in CSed.

Methods

Our study was guided by two research questions: (1) How do district actors utilize data infrastructures and routines in order to understand possible root causes of inequity within comprehensive CSed initiatives? (2) What factors support or mediate such processes of diagnostic problem framing? We address these questions through analysis of four case studies of district data use related to addressing equity issues in CSed initiatives, detailed below.

Context and data

To analyze the work entailed in district actors’ data practices for understanding barriers to equity in their CSed initiatives, this study utilizes data from a broader study of eight school districts conducted by the authors. While these districts were geographically and demographically diverse, all shared a commonality of implementing comprehensive CSed initiatives for at least 2 years. This provided ample opportunity to examine the two research questions. We define “Comprehensive CSed initiatives” as ones in which districts are actively working to implement coherent, district-wide computer science learning opportunities across their K12 grade bands, with an intention to achieve equitable access and participation within these learning opportunities. This stands in contrast to districts where there might simply be “pockets” of CS learning opportunities (for instance, only at the high school level, or only in particular schools) and no larger or coherent district effort to broaden these opportunities across schools and grade bands within the district.

We conducted 44 1-hour interviews with district actors including administrators, coaches, teachers, and research partners. These interviews were conducted using a theoretically guided protocol that focused on surfacing historical actions and issues related to promoting equitable CSed in terms of who CSed is for, how CS was taught, and what CS was taught (Santo et al., 2019). Interview data was triangulated with 130 internal and external district planning and implementation documents as well as with video recordings of six 90-min virtual professional learning community meetings. In these meetings, conducted by the research team, representatives from each district met to present their equitable CSed work and share common challenges and emerging best practices. Through our case development process, we identified a more targeted set of cases and associated data from within this corpus that served as the basis for this analysis (see further details in section Analysis). With the exception of Chicago Public Schools, which, in consultation with the district, we decided to identify due to the already public nature of some of the work we analyzed, all districts and individuals are pseudonymized.

Analysis

In developing our findings, we employed the methodological approach of “telling cases” (Mitchell, 1984). Telling cases entails “the detailed presentation of ethnographic data related to some sequence of events from which the analyst seeks to make some theoretical inference” (p. 238). In seeking to make theoretical inferences related to the process of district actors’ process of identifying root causes of inequities in K12 CS education through usage of data, our unit of analysis was instances in which district actors attempted to engage in practices of diagnostic problem framing. Our analytic process took place across three phases: (1) case identification and selection, (2) case development, and (3) cross-case analysis.

To select cases for deeper analysis, we first identified 17 illuminative cases from our broader data corpus that involved district actors building and leveraging data infrastructures, as well as the routines they engaged in to identify root causes of barriers to equity within their CS instructional systems. These cases spanned six of the eight districts involved in our broader study. Related data included 19 semi-structured hour-long interviews across 18 participants. From this sample, we selected four representative cases based on alignment with the CAPE framework (Fletcher and Warner, 2020). Our rationale for focusing on these specific cases was based on the criteria that each case spoke to one of the four different elements of the Fletcher and Warner framework. We selected one case for capacity, one for access, one for participation, and one for experience issues. The intent in doing so was to highlight how diagnostic problem framing plays out across a range of phenomena related to CSed equity. These four cases are from 3 districts, and are based on a total of 7 interviews across six participants, as well as on related district documentation including published papers, internal reports, and presentations.

To develop the cases further and answer our first research question, we created thick descriptions of each case (Denzin, 1989; Denzin and Lincoln, 2005). This involved writing analytic memos centered on the various dynamics at play in district actors’ data infrastructures and routines as well as the different types of work involved in using data to investigate and understand the root causes of inequities within each case. The writing of these thick descriptions was guided by a sociocultural view of institutional change (March and Olson, 2006; Meyer and Rowan, 2012). This view centers on exploring how intertwined dynamics of institutional relationships, norms, routines, interpretation, learning, and decision-making unfold in relation to valued actions and outcomes within organizational contexts. For each case, we utilized a combination of member checking as well as triangulation across multiple interviews and district documents in order to ensure the accuracy of case details.

To answer our second research question concerning underlying factors that mediated diagnostic problem framing (i.e. which factors hindered or supported the process of district leaders to identify root causes of educational inequities), we engaged in the constant comparative method (Glaser, 1965) to conduct a cross-case analysis. This analysis was conducted simultaneously “bottom up” from the data while also being guided by our review of the broader literature on data-informed improvement in schools outlined in the background established above. For example, Mandinach and Gummer (2016) established the importance of data literacy generally within educational systems change work. As such, we sought evidence of whether this capacity actively mediated diagnostic problem framing specifically. Within this analysis, we sought to identify common factors present across each of the cases that we saw as supportive mediators of the process of identifying root causes of inequities highlighted by data.

District descriptions

Chicago Public Schools is the third largest school district in the United States, serving close to 400,000 students across almost 650 schools. CPS' computer science education efforts, while in existence since 2008, were formalized into a comprehensive initiative in 2013. Of particular relevance to this analysis was the existence of (1) a CS graduation requirement established in 2016 and (2) a long-term research-practice partnership, CAFÉCS, which served to support the district’s CSed efforts through development of a robust data and research infrastructure.

Yorkville School District is a small-to-mid size district located on the East Coast. Their computer science offerings began with AP Computer Science A in 2012 and have since grown into a robust pathway of computer science courses at the time of writing. In recent years they have focused on increasing student enrollment, teacher capacity, and the number of entry points into the CS course pathways for students with minimal CS experience.

Rosewood Public Schools is a small rural district in the Northwest. Rosewood serves a high number of economically disadvantaged students with limited STEM and CS opportunities in their rural community. With a focus on economic mobility, Rosewood has built out a computer science pathway that transitions students from block to line coding (in grades 3–8) and an optional college and career focused computer science pathway at the high school level (Table 1).

Table 1.

Summary of focal cases.

Equity issue focus within case	District	District size	Urbanity
Teacher capacity in introductory CSed sources	Chicago Public Schools	Large	Urban
Access to secondary CS courses for English language learners	Chicago Public Schools	Large	Urban
“Undermatching” and gendered participation in secondary CS courses	Yorkville School District	Small-to-Mid	Suburban
Student experience and disengagement in elementary CS courses	Rosewood Public Schools	Small	Rural

Findings

Our findings consist of two analyses. The first analysis is organized around the four elements of the CAPE framework (Fletcher and Warner, 2020). This analysis focused on how data and associated diagnostic problem framing around inequities played out within issues of teacher capacity around CS, student access to CS learning experiences, student participation in CS learning opportunities, and the quality of the experience students had within those CS learning opportunities. The second analysis compared across each case to tease out and identify recurring mediating factors that were consequential for district leaders’ problem-framing process.

Diagnosing root causes of CSed inequities in capacity, access, participation, and experience

Capacity: surfacing teacher capacity factors in introductory CS courses

Chicago Public Schools is one of the largest school districts in the United States, and has been engaged in a comprehensive, district-wide CSed initiative since 2013. Notable among the work the district has done to advance CSed was the establishment of a CS graduation requirement for high school students in 2016 (Dettori et al., 2018). The policy change was a substantial step toward reducing inequities in CSed. This policy created a mandate for not only offering, but promoting full participation in, CSed experiences. However, the policy also contained risks around potentially reproducing broader inequities: if CSed implementation was not well-supported, it could result in students not graduating from high school.

In the second year of the graduation policy’s implementation, the team involved in both implementation and research began to broadly hear district stakeholder concerns related to these risks. Specifically, there was a sense that the introductory course (Exploring Computer Science: Goode et al., 2014) that was in widespread implementation in CPS high schools was too difficult and that there was a high course failure rate. If this concern was well-founded, it would present a substantial risk for both students experiencing course failure as well as for the sustainability of the broader CPS CSed initiative itself.

In response, the CPS Office of Computer Science (OCS) collaborated with CAFÉCS, a longstanding research-practice partnership supporting the CPS CSed initiative (Henrick et al., 2019) to understand failure rates within their ECS courses. To support the analysis, leaders in OCS and CAFÉCS took advantage of existing data that had been collected since the introduction of the ECS course, as well as collaborative structures they had in place to engage in discussions of data in relation to the initiative’s strategic goals. Steven McGee, a leader of the CAFÉCS partnership, described the process this way:

“These meetings are two to three dozen people that get together. [...] The idea there is we want to create an atmosphere where we are doing this shared visioning, shared interpretation of data, getting the whole community input so that we’re guided by everyone’s voice. So it was at one of these monthly meetings that we then said, hey, there’s this issue and we brainstormed. Everyone had an opportunity to say ‘well, I think these are the factors that I think affect the failure rate.’ GPA, attendance, the quality of the teaching, there was some hypotheses about, well, maybe it’s higher in the second semester because that’s where there’s more collaborative work, that’s where attendance really would become an issue for people, well there are school level factors, teacher level factors on content knowledge.”

This described process of hypothesis generation around possible factors affecting failure rates was significant—in response to the larger concern that was raised about failure rates, the group simply could have gone to their historical data to answer the question of whether or not there was, in fact, a problem with failure rates. Doing so would mean using available data to simply monitor the degree to which an equity issue around course failure was present. Instead, they engaged in a process of both answering the initial question of the degree of course failure but also used additional data they had available to them diagnostically—asking what might be “underneath” cases where course failure did occur in terms of potential root causes (e.g. “GPA,” “attendance,” “quality of teaching,” noted in the excerpt above). Essentially, engaging in a process of diagnostic problem framing vis-a-vis course failure rates.

The analysis proved to have important impacts when it came to how the initiative approached questions of systems change, in particular when it came to issues of teacher capacity (e.g. teacher content knowledge). McGee shared that the results, first and foremost, showed that the initial concern around failure rates did not exist to the degree that those who raised them may have imagined.

“The failure rate was about 11%, which as near as we could tell, was not any different than any other course [in the district]… And in fact the consortium [University of Chicago Consortium on School Research], who actually has access to all subject areas, found that the grades in computer science were actually higher on average than grades on other courses.”

But just because the degree of course failure in reality did not match the stakeholder concerns did not mean there was no room for improvement. The deeper exploration of factors that correlated with course failure offered important insights around potential root causes of the problem. Outlined in a technical report by McGee et al. (2018), the results pointed to two critical factors that could be focused on to reduce failure rates: (1) Whether teachers had participated in ECS professional development and (2) the number of times a teacher had taught the course. Each was found to be statistically significant factors correlated with students passing the course.

These results, in turn, were then able to be mobilized to encourage administrators to make evidence-based decisions on both supporting teacher participation in professional development as well as in assigning teachers to the course. As Andy Rasmussen, an administrator within Chicago’s OCS, shared:

“We were able to see from the research that the failure rate for students in classes taught by somebody who had actually been through the professional development was nearly half that of teachers who had not been through that. So that’s been validating in one sense, but it also gives us a really good talking point when speaking to principals and others about the importance of starting and completing the professional development sequence and taking advantage of what we have to offer.”

McGee shared a similar view on how the team was able to mobilize findings around teacher experience in teaching ECS:

“That was then data that the [OCS] implementation specialists or integration specialists could take back to the principals to say, ‘Hey, if you’re worried about failure rate, please make sure you assign the same teacher, don’t keep changing teachers every year.’”

If the group had simply monitored course failure data, rather than taking a stance of diagnostic problem framing around what factors might be behind instances of course failure, they may have missed an important opportunity to reduce inequities. The group did report that in the year after the report’s results were shared with administrators, they saw increased attendance in ECS professional development, a result suggesting that the findings prompted evidence-based decision-making on the part of principals with regard to engaging with teacher professional development opportunities.

Access: reaching into district systems to understand access issues for English Language Learners in secondary CS

A second case shared by the Chicago CS team concerned lack of access to secondary CS courses by students who were English Language Learners (ELLs). As Andy Rasmussen of the CPS Office of Computer Science (OCS) shared:

“I’d been looking at intersectional slices of who has not had access to a Computer Science course by the time that they’re in their senior year. So there were a small, small percentage of students that hadn’t been offered a class, and we’re trying to figure out why. And looking at every slice, so like gender and race, or diverse learner status and race, looking at all these sorts of demographics. One thing that really stuck out is that if you’re an English Language Learner you had a much higher chance of not having been offered the course. And this isn’t something that had popped out to us before.”

The group had surfaced an equity issue around access for this particular subgroup, but did not have a clear sense of what was causing the problem. As in the previous case, the graduation requirement that CPS had put in place around secondary CS meant that all students should have, in theory, been offered the opportunity to take a CS course in order to fulfill the requirement.

In contrast to the case involving course failure rates, the group did not have a clear set of hypotheses around what might explain this discrepancy. To better understand the problem, the group reached out to a representative from CPS’s Office of Language and Cultural Education (OLCE), which focused on English Language Learners within the district. This representative was invited to one of OCS’s aforementioned “consultancy” meetings where stakeholders regularly explored data and how it might be used to improve goals. As another OCS administrator shared:

“Her presence was invaluable, because she was able to articulate why schools have problems scheduling English language learner students into computer science. I didn’t realize that there were these tiers of English Language Learners, and depending on what tier they fell in, that determined their English proficiency. And some students are never elevated to the level of proficiency where they’re able to [participate]... They have to continue... I think it’s if you’re level one or level two, then you have to continue to take all of these English classes until you get to a level of proficiency where you can partake in other things.”

In consultation with the representative from OLCE, the computer science office came to understand a set of district policies they had, up to that point, been unaware of: ELL students had particular proficiency requirements that resulted in certain courses not being available to them. They determined that this policy likely explained the under-enrollment trend in ELL students that they had found in their secondary CS course data. Andy Rasmussen, an OCS administrator, described the importance of gathering information and diagnostic problem framing around this issue:

“She was just this fount of information about all that schools and students have to think about when they are a foreign transfer student, or any other thing where their English level is low, and so they’re treated very differently by the school. So we just hadn’t had that insight very much. We kind of knew from conversation, but hadn’t had the insight there.”

At the time of writing, OCS was considering how to respond, exploring the possibility of putting in place a waiver for certain students when it comes to the CS graduation requirement.

In reflecting on this case and others like it, Rasmussen noted that “there’s a fair amount of data work that’s sort of gumshoe work.”, pointing to the reality of needing to track down both forms of data or people with knowledge of the inner workings of particular aspects of the instructional system that might be impacting data:

“There’s so many things we didn’t know to ask the question. [...] It’s often one person in the district that manages that thing. And you have to be able to know that the person exists, know the right questions to ask and how things relate. So, there’s a fair amount of the data work that’s sort of gumshoe work, and figuring out what are the things we need to be thinking about, who are the students that are hidden? What are the experiences that they have that are hidden? And learning to track some of that down.”

In contrast to the first case shared around CPS’s data work, this one did not involve either direct generation of hypotheses to explain trends they were seeing in their broader monitoring data, or formal processes of “asking questions of the data.” Instead, the diagnostic problem-framing process took the form of consultations with other district actors who understood elements of the system that they were unaware of. As stated by Rasmussen, this “gumshoe work” allowed them to figure out how to address inequities they were observing in the data.

Participation: Explaining trends in “undermatching” within secondary CS courses

In Yorkville school district, two stakeholders monitored and analyzed the enrollment data of AP Computer Science A over multiple years across student subgroups: a high school CS teacher, Rosalyn, and the math department supervisor who managed the district’s secondary CS work, Nicole. When examining the percentage of female enrollment in AP CS-A, they noticed the highest it ever reached was a one-to-two ratio: for every female enrolled in AP CS-A there were two males enrolled. To make salient how problematic these enrollment numbers were for CS specifically, they compared them to the district’s female enrollment in AP Calculus which held a one-to-one ratio. Rosalyn explained her use of AP Calculus rather than AP Physics as a comparison point:

“Calc we’re fifty-fifty, that’s not the problem, but that’s why it’s a really good standard. It doesn’t make sense to compare it with physics. In other words, the people missing in AP Physics are pretty much the same people missing in AP CS-A. It’s females and minorities.”

Rosalyn’s point here echoes Barbara Ericson’s (2020) research that shows AP CS-A has the second lowest female enrollment rates among all AP courses, second only to AP Physics. Instead, AP Calculus has a relatively balanced percentage at approximately 50% female.

Keeping track of this data was not straightforward, however, especially as more teachers were added and involved. As Rosalyn describes,

“I like to keep track of the percent of minority and then the percent of female, and then the percent of free and reduced lunch across all courses. And again, it was easier when I was the only teacher. Now that there’s so many other teachers involved, it needs to be done through a Google Form, all teachers need to administer that form. It’s grown.”

Procuring a comprehensive picture of their enrollment numbers across all classes helped them identify the occurrence, magnitude, and trends in disparities over time. However, this process did not, in of itself, help Rosalyn or Nicole to sort out the underlying variables that might be acting as root causes for the disparity in the first place. This needed to be accomplished through diagnostic problem framing.

Rosalyn began to identify a possible root cause, in her opinion, as she witnessed another phenomenon happening in her AP Computer Science Principles class. There were more female students who she felt were “undermatched,” and should have been enrolled in AP CS-A based on Rosalyn’s perception of their proficiency. Rosalyn had a hunch that guidance counselor recommendations might explain the undermatching participation trends she was seeing, and thus the low enrollment numbers for AP CS-A. She described that this hunch may have emerged from a combination of factors. First, Rosalyn had numerous experiences with faculty in her district who voiced stereotypical views about the kind of students computer science was well-suited for (i.e. male technology enthusiasts). Second, she attended national training sessions that articulated the worldwide pervasiveness of these stereotypes and how they worked to keep women out of Computer Science. During one such training she reflected on her own high school experiences as a student talking with guidance counselors, and realized that “I am one of those statistics.”

Acting on this hunch, Rosalyn spoke with the “misplaced” female students who were not in AP CS-A to figure out who their guidance counselors were, which led her to notice a pattern as to which counselors were undermatching female students. She then spoke with those specific guidance counselors to both advocate for female students as well as to figure out if there were other female students who might be undermatched. As Rosalyn described this informal data practice:

“Some of that data is word of mouth. So I’ll have a student that’s in my class, Abby, she was in CS Principles, and she should have been in [AP CS] A based on her abilities, and I’ll ask her who was her counselor, and she’ll tell me. And then two periods later, it’s September, we’re still trying to work out the kinks. And again, I’ve got Sally, who’s in the class, and she should have been in CS-A as well and it’s the same counselor, so now that’s data. And now I’m gonna go talk to that counselor and explain, like I said, adult to adult, ‘This is what I’m seeing. Are you aware of this and what happened here?’ So that it doesn’t happen in the future. ‘And do you think it might have happened to anybody else? Anybody that wasn’t in my class?’ So that’s an example of data that’s sort of informal.”

Rosalyn’s diagnostic problem framing was informed by the combination of various sources of data: annual formal comparisons of enrollment numbers between AP Calculus and AP CS-A, Rosalyn’s own formative understandings of students’ abilities and whether they are misplaced or not, and beginning-of-the-year informal word-of-mouth data from students and their counselors. Rosalyn attributed the issue of low female enrollment in AP CS-A not to the students themselves (as if females were reluctant to push themselves), but to the recommendations of guidance counselors. She suspected that the problem may be related to counselors being unfamiliar with or even afraid of computer science as well as unconscious stereotypical views they may have held about who CS is for. This diagnostic problem framing set her up to enact solutions that focused on inviting counselors to see themselves as part of the solution to increasing female enrollment in computer science. To do this, Rosalyn had a series of conversations with counselors to familiarize them with Computer Science, debunk the myths of who Computer Science is for, and to back up her claims about the severity of the problem by directly sharing the low enrollment data for females in Computer Science.

Experience: unpacking student disengagement in CS across grades 3–6

Beverly served as Rosewood Public Schools’ data, assessment and technology facilitator since 2018. In this role, she collected student data across a variety of content areas and presented these back to the administration to inform improvement efforts. In other subject areas, she had well-established methods of tracking growth over time. However, in CS, she lacked any formalized benchmarks such as state-based, standards aligned tests. In lieu of these, Beverly decided to focus on student engagement in CS. To measure engagement, she went directly into CS classrooms with a stopwatch and a notebook and collected what she described as “[Beverly’s] mucky data.” While she considered these data to be “informal” and “anecdotal,” she was in fact fairly systematic in her approach, both iterating on how she considered accounting for disengagement and collecting data longitudinally and across grade levels.

Her approach to “mucky data” evolved over time. First, she recorded “tallies of how many times did I see kids not engage.” Then, she used her phone’s stopwatch feature to document, “here’s the longest that everybody was engaged. And then I restart the timer. Okay, here’s the next longest, here’s the next longest.” As her classroom observations progressed over the school year, Beverly noticed, for example at the sixth grade level, that when the school year started students were “scared to death of the teacher. So they were quiet…but then as time went on, and they built rapport and they got to know her…they loosened up a bit.” As engagement seemed to increase at the aggregate level, she modified her approach to her mucky data again, this time attending to individual students.

“It made more sense to go back to looking at, okay, here’s a kid that’s not engaged. How long is that not happening? And seriously, there were such high levels of engagement that I could just count like I used to coach. So I would just count one thousand one, one thousand two, one thousand three, just to see how long they were not engaged.”

When this method revealed to her that some individual students were not engaging for extended periods of time (e.g. 2–3 min) she would talk with the students’ teacher about how they might more effectively engage them. However, these conversations about individual students were not always fruitful as teachers tended to attribute the problem of disengagement back to the students themselves. Beverly would try to gather additional data about students’ engagement:

“How do you know that students are grasping this? And then I’d name a specific student. So then saying, depending on whatever the answer was, which was usually, ‘I don't know.’ ... Or ‘That student just isn’t getting it.’ To then being able to say, ‘Okay. What are some ways that this instruction can change so that they do get it or they are able to engage more and they’re not sitting back and just staring at their computer?'”

Our data suggest that Beverly had a hunch about one factor affecting student engagement: the lack of immediate assistance when a student became stuck during a CS activity. Beverly guessed that this was specifically true during whole group instruction, which has tendencies to lose student engagement when struggling students were “stuck” and needed to wait for teacher assistance. Due to her work as data, assessment, and technology facilitator across multiple content areas, she had a unique vantage point that made her familiar with what she saw as the shortcomings of whole group instruction:

“So in the past, we did everything whole group. We’re getting away from that more because we have kids that are at all different levels and if we do solely and just whole-group instruction and then set them loose, we know that our strugglers are not learning. And so that’s true whether it’s computer science, or whether it’s math, or whether it’s reading.”

She also appreciated that placing students in smaller groups was not enough. For example, if “the strugglers” were all placed together, students did not have the opportunity for “more differentiated” groups that could influence their growth. This assumption was reflected in the solutions to student disengagement that she devised with the computer science teachers. At the third-fifth grade level she worked to create differentiated small groups. At the sixth grade level, the teacher wanted to continue with whole group instruction, but was willing to bring in a teacher aide student from her upper-level programming class. These moves suggest that Beverly suspected that student disengagement was caused not by student disinterest in computer science, but by a specific feature of whole group instruction—students becoming stuck and having to wait at their computer until a teacher was free to come over and help them move forward again.

Based on her measurements around time on task as a proxy for engagement, Beverly’s actions suggest that she saw prolonged periods of disengagement as resulting from lack of individualized support within whole group instruction, which then drove her actions to bring in more individualized supports for students by reconfiguring the third-to-fifth grade classrooms into small differentiated groups where peers could support each other, and by bringing in a student teacher aide in the sixth grade classroom. Beverly described the resultant change in engagement for the sixth grade classroom this way:

“She has been able to strategically pick a teacher aide student who is also in her upper level programming class to be able to be a teacher aide for her, say, an eighth grader who can work with a sixth grader and be able to... The two of them, after she gives a short 10-minute lesson on whatever the skill is for the day, and they can go around the room and work individually with students, or work with pairs and the students. And that’s been interesting to watch, because the level of engagement has amazed me in that room. Because kids, they don’t sit wondering what they should be doing, they have a pretty clear picture on what they should be doing and if they have a question, it gets answered pretty quickly. So they can move on.”

Yet, midway through the year Beverly discovered that whole group instruction was not the only problem interfering with student engagement. As part of her work, Beverly also offered computer science professional development to science teachers in her district. After 5 days of training, these science teachers were able to offer computer science in their science rooms. At the fifth grade level, students had computer science opportunities both in their science classroom and in a 30-min specialist class with a CS instructional specialist. Unbeknownst to Beverly, however, was that the science teachers felt like they had to teach the same lessons that the CS instructional specialist was teaching, and unbeknownst to both Beverly and the science teachers was that a handful of students were attending the CS specialist course right before attending the science classroom for a CS lesson—the rest of the students were going to other specialist courses such as P.E. or the library. The result of this was that some but not all fifth graders were immediately repeating the same CS lessons. As Beverly describes it, “so the kids that already got it felt like they were being punished.” In response to this situation, Beverly started gathering a new form of data—fifth graders’ schedules—and shared these with the science teachers. Together, they devised solutions to increase the engagement of students receiving double doses of CS by providing them with additional pathways for engagement such as helping other students or creating their own CS projects, instead of simply repeating a CS lesson.

Underlying factors that mediate diagnostic problem framing

The first analysis presented four individual cases of the work entailed in district leaders’ diagnostic problem-framing processes as they addressed equity issues with regard to capacity, access, participation, and engagement, respectively. The second analysis, below, builds on the first by using a cross-case comparison approach to tease out and identify a set of recurring factors that appeared to consequentially mediate district leaders’ problem-framing processes.

Conceptualization of equity as a multifaceted phenomenon

Within each of our cases, we see how ideas about what counts as “equitable” informed and shaped district leaders’ practices of diagnostic problem framing. In the first case concerning capacity, after learning that ECS failure rates were not, in fact, higher than average compared to other subject areas, district staff could have set the issue aside. However, they did not. They maintained a more expansive equity goal of reducing failure rates, a goal which then drove further inquiry into root causes. In the second case regarding access, an underlying conceptualization of equity through the lens of reducing subgroup inequities in access led district actors to seek out explanations for the disparities they identified in their data. In Yorkville’s participation case, a conceptualization of not simply enrollment, but enrollment in courses that were appropriately matched to female student competencies, drove Rosalyn to seek out causes for undermatching. Finally, in Rosewood, Beverly conceptualized equity within the context of student experiences within courses, leading her to explore why disengagement might be occurring. In each, a normative stance concerning what might be considered equitable, or not, undergirded the process of then diagnosing underlying causes. Indeed, it’s possible to imagine that different conceptualizations of equity, especially ones that are perhaps more narrow, would constrain which activities, if any, district leaders might have engaged in around problem diagnosis.

Problem attribution to systems, not students

Rather than explaining disparities in system outcomes or opportunities as the fault of students themselves, district leaders used diagnostic problem framing to show that the system itself consisted of shortcomings, contradictions, and redundancies, and/or was steeped in processes that reproduced historical inequities. In Chicago, district leaders countered the claim that the course failure rate was evidence that computer science was simply too difficult for students, by showing, instead, that when teachers had adequate professional development and previous experience teaching the course, students were successfully learning computer science.

In Yorkville, district leaders encountered a stereotype-laden view amongst counselors that female students were not well-suited for participating in CS courses. Instead of blaming female students themselves for not choosing more advanced courses, district leaders attributed the problem to the presence of historically inaccurate stereotypes that were being reproduced in everyday interactional dynamics between students and their teachers as well as in counselors’ course selection advice.

Lastly, in Rosewood, Beverly confronted a deficit problem framing from certain teachers when they explained students’ disengagement as “that student just isn’t getting it.” She worked to counter this framing, by redirecting the nature of the problem of engagement to the system itself: “Okay. What are some ways that this instruction can change so that they do get it or they are able to engage more and they’re not sitting back and just staring at their computer.”

Ability to analyze data for indicators of inequities

District leaders made one or more intentional data analysis moves to search for potential inequities. These included disaggregating data sets into student groups, comparing data sets across instructional systems, and closely analyzing cases of specific students. District leaders in Chicago and in Yorkville worked to disaggregate student data into various student groups to glean the presence of disparities in student access or student participation in computer science courses, respectively. District leaders in both Chicago and Yorkville also compared computer science data sets such as course failure rates and enrollment rates (of males and females) to other subject areas that served as appropriate points of reference (such as AP Calculus rather than AP Physics in Yorkville), to grasp the severity of a given inequity. Last, district leaders in both Yorkville and Rosewood worked to build relationships with specific students and gather individual data points on them to notice irregularities that suggested an underlying equity issue such as students being undermatched in Yorkville and students not receiving timely support in Rosewood. This problem-framing work entailed more than monitoring and collecting data, but rather strategically triangulating data across multiple sources to infer and surface underlying inequities.

Knowledge of local instructional systems

As part of their problem-framing work, district leaders drew on their pre-existing knowledge of local instructional systems to locate the root causes of the inequities they surfaced. In Yorkville, Rosalyn connected ideas between her knowledge of under-enrollment of female students in AP CS-A courses, her formative knowledge of capabilities around CS displayed by female students in the AP CS-P course she taught, and her familiarity with counselors’ stereotype-laden views about who CS is for. This led to her diagnostic problem framing to form a “hunch” about the cause of under-enrollment being related to guidance counselor brokerage and recommendation practices.

In Rosewood, Beverly was familiar with how issues of whole group instruction were inadequately serving struggling students across her district’s different subject areas. This familiarity allowed her to extrapolate the issue into the context of computer science courses as a potential explanation for student disengagement. She eventually located an additional root cause as she became familiar with additional aspects of the local instructional system. This root cause was a teaching practice in which science teachers provided the same computer science lessons as the CS specialist, and a scheduling quirk that allowed certain students to get a double dose of these lessons back-to-back.

This points to the importance of district leaders working not just to connect the dots between their pre-existing knowledge of local instructional systems, but to expand their purview of other potentially relevant aspects of instructional systems. In Chicago, for example, district leaders worked to learn about the division of English Language Learners into tiers and the scheduling and resultant access issues that emerged from that. Likewise, in the experience case, Beverly’s intuition about whole group instruction being a root cause of disengagement was only a part of the explanation.

Embeddedness in instructional networks

The cases also highlight the reality that diagnostic problem framing does not occur in a social vacuum. District leaders were able to advance their problem framing through actively embedding themselves in larger organizational systems and building personal relationships. In Chicago, district leaders were able to engage in processes of hypothesis generation through a standing consultancy meeting that involved a large number of district administrators and research partners, which led to formal analysis of large datasets on the part of the CAFÉCS research partners in order to identify factors correlated with course failure. Additionally, in their problem-framing work around access, district leaders in Chicago leveraged their social networks with the district to broker a meeting with a representative from the district office responsible for policies surrounding ELL students. Their notion encompassed not about what an explanation might be, but who might possess it. By gathering additional background knowledge from the ELL administrator, the team was able to receive knowledge of ELL instructional policy that was highly plausible as the root cause of the access issues at play.

In Yorkville, Rosalyn utilized her positionality as a teacher of AP-CSP who had direct relationships with students, her role collaborating with Nicole on monitoring enrollment data, and her experiences at national equity-focused trainings in order to develop her hunch around “undermatching” and its relationship to guidance counselor practices. In Rosewood, Beverly’s positionality as both the actor responsible for data monitoring as well as professional development around CSed positioned her to be exposed to the underlying “double-dosing” issue that explained why some students were disengaged in CS lessons.

In all of these examples of diagnostic problem framing, district leaders unpacked and made sense of the data they collected and monitored by partnering with researchers, leveraging brokers, attending equity-focused trainings, and building relationships with students impacted by a specific equity issue.

Discussion

The examples shared above highlight how data were utilized within the context of “debugging” inequities within CS instructional systems, spanning phenomena of capacity, access, participation and experiences of CSed. Although districts varied by size, geographic region, and student demographics, they each had in place data infrastructures (Datnow et al., 2007), leadership (Copland, 2003; Halverson et al., 2007), and organizational routines of interpretation (Boudett et al., 2020; Spillane, 2012) that supported their data monitoring work. Further still, although districts varied in the types of inequities they identified, as well as in their processes of surfacing the root causes to those inequities, their diagnostic problem-framing work was directly mediated by five factors that took them above and beyond data monitoring: (1) conceptualization of equity as a multifaceted phenomena, (2) problem attribution to systems, not students, (3) ability to analyze data for indicators of inequities, (4) knowledge of local instructional systems, and (5) embeddedness in instructional networks.

We argue that these five factors are not unique to the four cases presented above, nor only pertinent to the work of implementing equitable computer science education. Rather, we believe these factors are constitutive of data-informed diagnostic problem-framing practices, more generally, that aim to transform inequities at various points within instructional systems. That is, their presence or absence directly mediates the kind of problem-framing work that is possible for district actors.

Evidence to support this claim can be found in previous studies wherein scholars have taken up each of these factors and their interrelations within the context of data-informed diagnostic problem-framing work. First, the importance of these factors when it comes to supporting diagnostic problem framing is affirmed by previous research in which scholars take up each factor individually as their focal phenomenon of interest. This includes scholarship that focuses on conceptualization of equity as a multifaceted phenomena (Fletcher and Warner, 2020; Gutiérrez, 2012; Santo et al., 2019), problem attribution to systems, not students (Ladson-Billings, 2006; Valencia, 2012), ability to analyze data for indicators of inequities (Blake et al., 2016; Johnson and La Salle, 2010), knowledge of local instructional systems (Schildkamp and Kuiper, 2010), and embeddedness in instructional networks (Daly, 2012; Hubers et al., 2018; Marsh et al., 2015; Park, 2018).

Second, the interrelatedness of these five factors—not simply presence of them, but interaction among them—within processes of diagnostic problem framing have also been taken up by scholars. For example, research on equity-mindedness in districts’ data use practices, (Bensimon and Harris, 2007; Bensimon et al., 2016; Dowd and Bensimon, 2015; Galloway and Ishimaru, 2019; Irby and Clark, 2018) shows the importance of conceptualizing equity issues as historical and systemic phenomenon in which disparities are accumulated over time, of not blaming students for those accumulated disparities, and of disaggregating data by student groups to evidence the presence and severity of an inequity.

Likewise, research on data literacy (Mandinach and Gummer, 2016) in districts’ work to “identify problems and frame questions” (p. 369, italics in the original) also conceptualizes the importance of examining multiple aspects of a system including “a student, group of students, a topical area, the curriculum, or an aspect of instruction” (p. 370) as well as understanding the local context at both the student and school level. Additionally, they conceptualize the work of identifying problems to “involve other participants or stakeholders, including students” (p. 370).

Last, the co-presence of all five factors can be found in an extended case of successful diagnostic problem framing for equity in the research literature, even though the factors were not explicitly theorized as such. Specifically, Park et al. (2013) presented a case in which Hispanic students at the high school level were not being encouraged to take college preparatory courses. The diagnostic problem-framing work in this case entailed district leaders to (a) shift their conceptualization of inequity from achievement gaps to opportunity gaps, to (b) acknowledge how a culture of low academic rigor and expectations for Hispanic students was a pervasive part of their instructional system, to (c) disaggregate data sets by student population to discover that Hispanic students were half as likely as Asian students to be enrolled in College preparatory courses, to (d) be familiar with district practices, policies, and course offerings, and to (e) work collaboratively so that administrators and teachers were learning from each other to generate a fuller picture of the problem (Park et al., 2013).

These disparate threads of pre-existing research showcase other similar representations of diagnostic problem framing found in this study, affirming the potential generalizability of this study’s findings At the same time, it showcases the need for additional research that shows the co-presence and interrelations of these aspects of problem-framing work in the context of data use practices that ultimately support equitable systems change. The inclusion of more exemplar cases in the research literature that provide detailed and full accounts of what is entailed in diagnostic problem-framing work will greatly support district leaders in enacting these processes in ways that go beyond mere compliance to accountability policies or that act on narrow conceptions of equity which aim to remediate students, rather than systems.

Conclusion

Working within the context of both historical inequities in computing along with complex institutional arrangements, district leaders, scholars, and advocates see promise in utilizing data-informed change processes to advance their goals around establishing comprehensive, equitable computer science education initiatives. In this paper, we affirmed the importance of gathering and monitoring data as a starting point, but highlighted that it is just that—a critical first step. We showed how processes of utilizing these data to identify the source of inequities is necessary in order to understand where and how to intervene, processes that offer their own complexities and contextual contingencies. For district leaders, our findings also offer direction on how to more effectively identify the underlying causes of inequity—the mediating factors we identified point to areas of activity to cultivate and guiding principles to follow as they develop equity-focused data initiatives. For scholars, we hope that our findings provide a useful theoretical roadmap for both studying processes of district change and engaging in successful research-practice partnerships with district leaders, as many of the processes outlined here are applicable to such arrangements. Broadly, this study highlights the importance of uncovering the facets of the real work involved in utilizing data to advance equity, something that while not necessarily straightforward, can indeed be viable if the conditions are right.

Footnotes

Acknowledgments

The authors would like to thank our district partners who generously and openly shared about their work to bring computer science education to their students. This research was made possible through support from Google’s Computer Science Education Research (CS-ER) award.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Google (CS-ER Program).

ORCID iDs

David Phelps

Rafi Santo

Author Biographies

David Phelps, PhD, is a researcher committed to serving youth and educators by designing, facilitating and studying innovative learning environments. This passion grew from his experiences working in a Reggio-Emilia preschool in Vermont, a community school in Peru, a care farm in the Netherlands, and philosophy for children programs in the Midwest and Pacific Northwest. Over the past 10 years he has had the privilege to professionally co-design and co-facilitate a wide range of learning environments that engage youth in collective interest-driven inquiry. This strand of research champions the incredible inquiry competencies of young learners. Simultaneously, he studies how issues of power and equity are navigated at a system-wide level as educators attempt to implement and scale innovative learning environments. He received his PhD in Learning Sciences and Human Development from the University of Washington.

Rafi Santo, PhD, is a learning scientist focused on the intersection of digital culture, education, and institutional change. His scholarship spans multiple levels of activity—from understanding youth learning pathways across settings to investigating policy implementation and organizational network design—in order to develop practical insights that come from a holistic perspective. His work has been supported by the Spencer Foundation, the MacArthur Foundation, the Wallace Foundation, the National Science Foundation, the Mozilla Foundation, the Susan Crown Exchange, Google, and the Corporation for Public Broadcasting, and he holds affiliations with CSforALL, the Joan Ganz Cooney Center at Sesame Workshop, and the GovLab at New York University. Rafi holds a PhD in Learning and Developmental Sciences from Indiana University.

References

Bang

Faber

Gurneau

, et al. (2016) Community-based design research: learning across generations and strategic transformations of institutional relations toward axiological innovations. Mind, Culture and Activity 23(1): 28–41.

Bensimon

Harris

III (2007) Accountability, equity, and practitioner learning and change. Metropolitan Universities 18(3): 28–45.

Bensimon

Dowd

Witham

(2016) Five principles for enacting equity by design. Diversity and Democracy 19(1): 1–8.

Bertrand

Marsh

(2015) Teachers’ sensemaking of data and implications for equity. American Educational Research Journal 52(5): 861–893.

Blake

Gregory

James

, et al. (2016) Early warning signs: identifying opportunities to disrupt racial inequities in school discipline through data-based decision making. School Psychology Forum 10: 289–306.

Booher-Jennings

(2005) Below the bubble: “Educational triage” and the Texas accountability system. American Educational Research Journal 42(2): 231–268.

Boudett

City

Murnane

(eds) (2020) Data Wise, Revised and Expanded Edition: A Step-by-step Guide to Using Assessment Results to Improve Teaching and Learning. Cambridge, MA: Harvard Education Press.

Coburn

Toure

Yamashita

(2009) Evidence, interpretation, and persuasion: instructional decision making at the district central office. Teachers College Record 111(4): 1115–1161.

Copland

(2003) Leadership of inquiry: building and sustaining capacity for school improvement. Educational Evaluation and Policy Analysis 25(4): 375–395.

10.

Daly

(2012) Data, dyads, and dynamics: exploring data use and social networks in educational improvement. Teachers College Record 114(11): 1–38.

11.

Datnow

Park

(2018) Opening or closing doors for students? Equity and data use in schools. Journal of Educational Change 19(2): 131–152.

12.

Datnow

Park

Wohlstetter

(2007) Achieving with Data: How High-Performing School Systems Use Data to Improve Instruction for Elementary Students. Los Angeles, CA: New Schools Venture Fund.

13.

DeLyser

Wright

(2019) A systems change approach to CS education: creating rubrics for school system implementation. In: Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, New York, NY, 02 July 2019, pp. 492–498.

14.

DeLyser

Wright

Wortel-London

, et al (2020) Evaluating a systems approach to district CS education implementation. In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education, New York, NY, 261120 February 2020, pp. –1126.

15.

Denzin

(1989) Interpretive Interactionism. Newbury Park, CA: Sage.

16.

Denzin

Lincoln

(eds) (2005) Handbook of Qualitative Research. 3rd edition. Thousand Oaks, CA: Sage.

17.

Dettori

Greenberg

McGee

, et al (2018) CS as a graduation requirement: catalyst for systemic change. In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, New York, NY, 21406 February 2018, pp. –407.

18.

Diamond

Cooper

(2007) Chapter 10 the uses of testing data in urban elementary schools: some lessons from Chicago. Yearbook of the National Society for the Study of Education 106(1): 241–263.

19.

Dodman

Swalwell

DeMulder

, et al. (2021) Critical data-driven decision making: a conceptual model of data use for equity. Teaching and Teacher Education 99: 103272.

20.

Dowd

Bensimon

(2015) Engaging The“ Race Question”: Accountability and Equity in US Higher Education. New York, NY: Teachers College Press.

21.

Ehren

Swanborn

(2012) Strategic data use of schools in accountability systems. School Effectiveness and School Improvement 23(2): 257–280.

22.

Erete

Thomas

Nacu

, et al. (2021) Applying a transformative justice approach to encourage the participation of Black and Latina Girls in computing. ACM Transactions on Computing Education 21(4): 1–24.

23.

Ericson

(2020) ‘AP CS Data for 2020’, Computing For Everyone, 11 December. Available at: https://cs4all.home.blog/2020/12/11/ap-cs-data-for-2020/ (Accessed 21 January 2021).

24.

Ericson

Adrion

Fall

, et al. (2016) State-based progress towards computer science for all. ACM Inroads 7(4): 57–60.

25.

Evans

Teasdale

Gannon-Slater

, et al. (2019) How did that happen? Teachers' explanations for low test scores. Teachers College Record 121(2): 1–40.

26.

Fancsali

Tigani

Toro Isaza

, et al (2018) A landscape study of computer science education in nyc: early findings and implications for policy and practice. In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, New York, NY, 2144 February 2018, pp. –49.

27.

Fletcher

Warner

(2020) Summary of the CAPE Framework For Assessing Equity in Computer Science Education. Retrieved from: https://www.tacc.utexas.edu/epic/research (Accessed May 26, 2020)

28.

Galloway

Ishimaru

(2019) Leading equity teams: the role of formal leaders in building organizational capacity for equity. Journal of Education for Students Placed at Risk 25(2): 107–125.

29.

Gannon-Slater

La Londe

Crenshaw

, et al. (2017) Advancing equity in accountability and organizational cultures of data use. Journal of Educational Administration 55: 361–375.

30.

Garner

Thorne

Horn

(2017) Teachers interpreting data for instructional decisions: where does equity come in? Journal of Educational Administration 55: 407–426.

31.

Glaser

(1965) The constant comparative method of qualitative analysis. Social problems 12(4): 436–445.

32.

Goode

Margolis

(2011) Exploring computer science: a case study of school reform. ACM Transactions on Computing Education 11(2): 1–16.

33.

Goode

Margolis

Chapman

(2014) Curriculum is not enough: the educational theory and research foundation of the exploring computer science professional development model. In: Proceedings of the 45th ACM Technical Symposiumon Computer Science Education, New York, NY, 05 March 2014, pp. 493–498.

34.

Gutiérrez

(2008) A" gap-gazing" fetish in mathematics education? Problematizing research on the achievement gap. Journal for Research in Mathematics Education 39: 357–364.

35.

Gutiérrez

(2012) Context matters: how should we conceptualize equity in mathematics education? In: Equity in Discourse for Mathematics Education. Dordrecht: Springer, pp. 17–33.

36.

Halverson

Grigg

Prichett

, et al. (2007) The new instructional leadership: creating data-driven instructional systems in school. Journal of School Leadership 17(2): 159–194.

37.

Henrick

McGee

Greenberg

, et al. (2019) Assessing the effectiveness of computer science RPPs: the case of CAFÉCS. 2019 Research on Equity and Sustained Participation In Engineering, Computing, and Technology (RESPECT). New York, NY: IEEE, 1–5.

38.

Horn

(2018) Accountability as a design for teacher learning: sensemaking about mathematics and equity in the NCLB era. Urban Education 53(3): 382–408.

39.

Hubers

Moolenaar

Schildkamp

, et al. (2018) Share and succeed: the development of knowledge sharing and brokerage in data teams’ network structures. ReseaRch PaPeRs in Education 33(2): 216–238.

40.

Irby

Clark

(2018) Talk it (Racism) out: race talk and organizational learning. Journal of Educational Administration 56: 504–518.

41.

Ishimaru

(2019) Just Schools: Building Equitable Collaborations with Families and Communities. New York, NY: Teachers College Press.

42.

Jacob

(2005) Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics 89: 761–796.

43.

Johnson

La Salle

(2010) Data Strategies to Uncover and Eliminate Hidden Inequities: The Wallpaper Effect. Thousand Oaks, CA: Corwin Press.

44.

Ladson-Billings

(2006) From the achievement gap to the education debt: understanding achievement in US schools. Educational Researcher 35(7): 3–12.

45.

Lasater

Bengtson

Albiladi

(2020) Data use for equity?: how data practices incite deficit thinking in schools. Studies in Educational Evaluation 69: 100845.

46.

Leftwich

Garvin

Dunton

, et al (2020) The landscape of broadening participation in computing, using state and national datasets to advocate for equity in computer science education. In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education, New York, NY, 261403 February 2020, pp. –1403.

47.

Lewis

Anderson

Yasuhara

(2016) I don't code all day" fitting in computer science when the stereotypes don't fit. In: Proceedings of the 2016 ACM Conference on International Computing Education Research, New York, NY, 2523 August 2016, pp. –32.

48.

Mandinach

Gummer

(2016) What does it mean for teachers to be data literate: laying out the skills, knowledge, and dispositions. Teaching and Teacher Education 60: 366–376.

49.

March

Olsen

(2006) Elaborating the “new institutionalism”. The Oxford handbook of political institutions. New York, NY: Oxford University Press, Vol. 5, 3–20.

50.

Margolis

Estrella

Goode

, et al. (2017) Stuck in the Shallow End: Education, Race, and Computing. Cambridge, MA: MIT press.

51.

Marsh

Bertrand

Huguet

(2015) Using data to alter instructional practice: the mediating role of coaches and professional learning communities. Teachers College Record 117(4): 1–40.

52.

Master

Cheryan

Meltzoff

(2016) Computing whether she belongs: stereotypes undermine girls’ interest and sense of belonging in computer science. Journal of Educational Psychology 108(3): 424–437.

53.

McGee

Greenberg

Dettori

, et al. (2018) An examination of factors correlating with course failure in a high school computer science course. The Learning Partnership Technical Report 5. Western Springs, IL.

54.

Meyer

Rowan

HDMB

(2012) New Institutionalism in Education. Albany, NY: The Suny Press.

55.

Mitchell

(1984) Typicality and the case study. In: Ellens

(ed) Ethnographic Research: A Guide To General Conduct. New York, NY: Academic Press, pp. 238–241.

56.

Nasir

(2011) Racialized Identities: Race and Achievement Among African American Youth. Stanford, CA: Stanford University Press.

57.

Oláh

Lawrence

Riggan

(2010) Learning to learn from benchmark assessment data: how teachers analyze results. Peabody Journal of Education 85(2): 226–245.

58.

Park

(2018) Leading data conversation moves: toward data-informed leadership for equity and learning. Educational Administration Quarterly 54(4): 617–647.

59.

Park

Daly

Guerra

(2013) Strategic framing: how leaders craft the meaning of data use for equity and learning. Educational Policy 27(4): 645–675.

60.

Phelps

Santo

(2021) Student leadership, systems change: opportunities and tensions for youth impact on district-wide computer science initiatives. ACM Transactions on Computing Education 21(4): 1–39. DOI: 10.1145/3461716

61.

Proctor

Bigman

Blikstein

(2019) Defining and designing computer science education in a k12 public school district. In: Proceedings of the 50th ACM Technical Symposium On Computer Science Education, New York, NY, 22314 July 2022, pp. –320.

62.

Reed

Wilkerson

Yanek

, et al. (2015) How exploring computer science (ECS) came to Chicago. ACM Inroads 6(3): 75–77.

63.

Safir

Dugan

(2021) Street Data: A Next-Generation Model for Equity, Pedagogy, and School Transformation. Thousand Oaks, CA: Corwin.

64.

Sannino

Annalisa

Engeström

Yrjö

Lemos

Monica

(2016) Formative Interventions for Expansive Learning and Transformative Agency. Journal of the Learning Sciences 25(4): 599–633. DOI: 10.1080/10508406.2016.1204547

65.

Santo

DeLyser

Ahn

, et al. (2019) Equity in the who, how and what of computer science education: K12 school district conceptualizations of equity in ‘CS for all ’initiatives. 2019 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT). New York, NY: IEEE, 1–8.

66.

Santo

DeLyser

Ahn

(2020) Negotiating equity priorities within systems change: a case study of a district-level initiative to implement K12 computer science education. Computing in Science & Engineering 22(5): 7–19.

67.

Schildkamp

Kuiper

(2010) Data-informed curriculum reform: which data, what purposes, and promoting and hindering factors. Teaching and teacher education 26(3): 482–496.

68.

Schildkamp

Rekers-Mombarg

Harms

(2012) Student group differences in examination results and utilization for policy and school development. School Effectiveness and School Improvement 23(2): 229–255.

69.

Snow

Benford

(1992) Master frames and cycles of protest. Frontiers in Social Movement Theory 133: 155.

70.

Spillane

(2012) Data in practice: conceptualizing the data-based decision-making phenomena. American Journal of Education 118(2): 113–141.

71.

Valencia

(ed) (2012) The Evolution of Deficit Thinking: Educational Thought and Practice. London: Routledge.

72.

Yurkofsky

(2022) Environmental, technical, and representational uncertainty: a framework for making sense of the hidden complexity of educational change. Educational Researcher 58(2): 300–346.

Debugging inequities: Data use,“gumshoe work,” and problem identification in district-wide computer science education initiatives

Abstract

Keywords

Introduction

Background

Methods

Context and data

Analysis

District descriptions

Findings

Diagnosing root causes of CSed inequities in capacity, access, participation, and experience

Capacity: surfacing teacher capacity factors in introductory CS courses

Access: reaching into district systems to understand access issues for English Language Learners in secondary CS

Participation: Explaining trends in “undermatching” within secondary CS courses

Experience: unpacking student disengagement in CS across grades 3–6

Underlying factors that mediate diagnostic problem framing

Conceptualization of equity as a multifaceted phenomenon

Problem attribution to systems, not students

Ability to analyze data for indicators of inequities

Knowledge of local instructional systems

Embeddedness in instructional networks

Discussion

Conclusion

Footnotes

Acknowledgments

Declaration of conflicting interests

Funding

ORCID iDs

Author Biographies

References