Sage Journals: Discover world-class research

Abstract

Low-income students and students of color are faced with pervasively lower levels of opportunity to learn compared with their peers, creating unequal opportunities for educational success. Textbooks, which serve as the backbone of the curriculum in most mathematics classrooms, present a potentially powerful tool to help mitigate unequal opportunity to learn across students. Using the Surveys of Enacted Curriculum framework, we investigate the content of commonly used eighth-grade math textbooks in California and the extent to which they align with the Common Core State Standards. We also explore the relationship between the variation in content coverage and alignment and student characteristics. We find poor alignment between the textbooks in our sample and the Common Core State Standards and low overall levels of cognitive demand, but only limited evidence of systematic differences in alignment or cognitive demand coverage associated with student characteristics at the school or district level.

Keywords

content analysis curriculum descriptive analysis equity mathematics education regression analyses textbooks

In 2004, the Williams v. State of California lawsuit contended that California public schools serving low-income students and students of color disproportionately lacked basic educational necessities like textbooks and safe facilities. Underlying these arguments was the notion of pervasive differences in students’ opportunity to learn (OTL), the simple idea that students cannot learn material which they have not been given the opportunity to study (McDonnell, 1995). The suit reached a settlement where over $1 billion would be directed toward guaranteeing that all California students have high-quality facilities, textbooks, and teachers. Although more than 15 years have passed since Williams, inequities persist, resulting in unequal opportunities for students to learn.

A growing body of research indicates that textbooks may be particularly powerful tools in the effort to equalize OTL. The vast majority of teachers use textbooks in most of their lessons, making them the backbone of the curriculum (Polikoff, 2018) and ubiquitous in a typical student’s schooling experience (Blazar et al., 2020). Research suggests that textbook choice may be important for student achievement, with several studies indicating that the choice of textbook can affect standardized test scores in mathematics (Agodini et al., 2010; Bhatt et al., 2013; Bhatt & Koedel, 2012; Koedel et al., 2017), though a recent large study found no such effects (see Blazar et al., 2020). In the context of the highly decentralized textbook market in the United States, where states, districts, schools, and even individual teachers may be involved in choosing from among dozens of possible materials (Polikoff, 2021), textbook choice may thus present policy makers with a potentially powerful tool to help equalize learning opportunities and improve education outcomes for disadvantaged students.

As part of a broader effort to improve teaching and prepare students for success in college and careers, California joined most other states in adopting the Common Core State Standards (CCSS) in 2010. Four years later, in 2014, the state also adopted a list of mathematics textbooks they approved as being aligned with the standards. While at certain times in California’s history these adopted textbook lists have been mandatory, they are now advisory—districts are free to adopt on or off the state-approved list. As we describe below, most California districts indeed adopt textbooks approved by the state as being aligned with the CCSS. While adoption by the state of California presumably signals some degree of alignment and quality, we know that textbooks often vary in their content (Polikoff, 2015). Thus, we wondered whether the problems highlighted in Williams—systematic variation in student opportunity to learn—remain issues in present-day California.

In this article, we examine two dimensions considered to promote student OTL as they apply to textbooks: content alignment and cognitive demand. Using the Surveys of Enacted Curriculum (SEC) framework (Porter, 2002), we evaluate many popular eighth-grade math textbooks according to their alignment to the CCSS and the level of student expectation (cognitive demand) involved in the presentation of content. We then analyze the extent to which variation in alignment and cognitive demand are systematically related to the characteristics of students in schools and districts using those books. In short, we address the following research questions:

Research Question 1: To what extent do eighth-grade math textbooks commonly used in California vary in terms of their alignment with content standards and cognitive demand?

Research Question 2: To what extent is variation in alignment with content standards and cognitive demand systematically related to student characteristics?

This analysis is one of the first to apply a rigorous content analysis method to a large number of textbooks and is the first to do so in the context of middle school math. The methods therefore have applicability in nations where there are multiple approved curriculum materials used in schools. It also answers important questions about students’ equitable exposure to aligned and cognitively demanding content through curriculum materials.

Review of Literature

The Origins of Opportunity to Learn

The notion of OTL has roots in international comparative studies, including the First International Mathematics Study (FIMS). Serving as a validity check, OTL questionnaires asked teachers whether they had covered each topic on the test to ensure that countries did not appear to be failing students when in fact certain topics simply had not yet been covered (Husen, 1967). While the questionnaires became more nuanced over the ensuing decades, bolstering the legitimacy of these international comparisons remained the purpose of OTL. Therefore, variation in OTL was nearly always analyzed between countries rather than within a single country (Floden, 2002). Contemporary international comparative studies like the Trends in International Mathematics and Science Study (TIMSS) and the Program for International Student Assessment (PISA) still use OTL in this way.

OTL in Research

Following the release of A Nation at Risk in 1983, American discourse treated middling U.S. results on international exams as a security and economic crisis (National Commission on Excellence in Education, 1983). Improving scores on these assessments became an important policy goal, and in the early 1990s, American researchers began treating OTL as a construct worthy of study in itself rather than just a covariate to improve the validity of international assessments (e.g., McDonnell, 1995; Porter, 1995). The importance of U.S. competitiveness on measures of international educational performance and increased attention on OTL led to efforts to measure OTL more precisely (e.g., Greenwald, 1997; Knapp, 1995; Niemi, 1996); using frequent teacher log entries (Smithson & Porter, 1994) and even early online tools (Ball et al., 1999).

The approach with the most sustained impact was pioneered by a team of researchers studying OTL in the context of high school mathematics (Gamoran et al., 1997). This research sought to explain between-class variation in student learning using survey measures of teachers’ instruction. They explored measures based on topic coverage only (e.g., multiplying fractions, solving two-step problems), cognitive demand coverage only (e.g., memorize facts, understand concepts), or the intersection of topic and cognitive demand. They found that, when defined as the intersection of topic and cognitive demand, OTL explained nearly all the between-class variation in achievement gains (in other words, that differences in OTL were driving differences in student learning between classes). This study led to the development of a suite of tools for measuring OTL (see Porter, 2002), which have since been used in numerous studies of teachers’ instruction (e.g., Polikoff, 2012a, 2012b), the content of various policy instruments like curriculum materials and content standards (e.g., Polikoff et al., 2015; Porter et al., 2011), and the associations of OTL with student learning (e.g., Polikoff & Porter, 2014).

OTL for Measuring Equity

OTL has been a particularly useful concept in research analyzing differential educational opportunities (e.g., Floden, 2002; Guiton & Oakes, 1995). Researchers reasoned that if OTL was predictive of student learning, then differences in OTL could be used to understand and contextualize oft-studied student performance gaps. OTL measures have found a foothold in studies focused on explaining between-group differences in performance. For instance, studies showed that low achievement among English learners (Callahan, 2005) and historically disadvantaged racial/ethnic groups (Schiller et al., 2010) could be explained by low OTL in these students’ classes, rather than by factors like race, gender, or English proficiency level. Traditionally underserved students’ learning experiences have often been characterized as offering weak OTL (e.g., see Cooper & Liou, 2018; Heafner & Fitchett, 2015; Santibañez & Fagioli, 2016; Tate, 2001). These types of OTL studies have provided evidence of educational inequities, such as those cited in the Williams case.

Current Conceptions of OTL

Much OTL literature has conceived of curriculum in three tiers (e.g., see Martin et al., 1999; McDonnell, 1995; Mullis et al., 2000; Mullis et al., 2005; Törnroos, 2005). The “intended curriculum” refers to a system-wide official curriculum, typically a set of academic standards. The “enacted curriculum” refers to how teachers bring that content to life in their classroom. The “attained curriculum” refers to the understanding students actually gain during a lesson. Each tier influences, but does not dictate, the tier below it. Tarr et al. (2006) add the “written curriculum,” consisting of textbook content, as a useful fourth tier. The written curriculum is informed by the intended curriculum and in turn informs teachers’ enacted curriculum. While all tiers of curriculum influence the knowledge students attain, improving written curriculum may offer policy makers a particularly high-leverage reform tactic (e.g., Koedel & Polikoff, 2017), since teachers use and students interact with their textbooks directly (Chingos & Whitehurst, 2012). Therefore, we focus our inquiry on textbook content as a way to document and improve students’ OTL. It is also worthy to study variation in the enacted curriculum, but this is much more complex and difficult to carry out at any reasonable scale.

Textbook Variation and Implications for OTL

Despite the importance of written curriculum to OTL, not all textbooks used in schools appear to be equally effective in promoting student learning. In particular, a number of experimental and nonexperimental studies have shown that textbooks differ meaningfully in their effects on student achievement (e.g., Agodini et al., 2010; Bhatt et al., 2013; Koedel et al., 2017), although a recently published study of textbook effectiveness across six states failed to find a significant effect (Blazar et al., 2020). These possible differences in textbook effectiveness suggest textbook choice as a potentially important intervention aimed at improving written curriculum.

Furthermore, research suggests that textbooks may be the most central component of written curriculum, especially in mathematics. Blazar et al. (2020) surveyed almost 1,200 mathematics teachers across six states and found that over 90% used an official textbook in more than half their lessons, with three fourths using the official textbook in 75% of lessons or more. While very few math teachers report exclusive use of the textbook in their instruction (Blazar et al., 2020; Opfer et al., 2017), many seem to consider officially adopted textbooks to be fundamental to their curriculum (Polikoff, 2018). In studying textbook content, then, we study a major determinant of both teachers’ enacted curriculum and students’ attained curriculum. Moreover, while textbooks are not the only component of students’ OTL, the link between textbooks and student achievement, the central role of textbooks as a learning tool, and the high-leverage policy reform intervention that they present underscore the importance of analyzing their connection to OTL.

Systematic Variation in OTL

The Williams settlement highlighted growing concerns about the unequal distribution of OTL across schools in California’s diverse communities and the need to adopt common standards of quality across schools (Venezia & Maxwell-Jolly, 2007). Rather than focus on funding equality, an approach that had yielded unsatisfactory results in increasing educational opportunities for the state’s disadvantaged students, Williams focused on resource adequacy as a means to more equitably distribute learning opportunities (Glenn & Picus, 2007). The plaintiffs in the Williams lawsuit were a group of students and parents in California public schools who alleged that the State failed to provide thousands of students in public schools, particularly students of color and students in low-income communities, with the basic necessities for an education as required by the Constitution and laws of California (American Civil Liberties Union Southern California, n.d.; Williams v. State of California, 2000). The plaintiff’s complaint cited chronic problems in California’s public schools as evidence, pointing to public schools staffed with emergency credentials and school facilities with “terrible slum conditions” (First Amended Complaint in Williams v. State of California, 2000, p. 9). The complaint specifically highlighted textbooks as a cause of education inequity, pointing out that many students lacked textbooks in core academic subjects or worked from textbooks that contained false or outdated information. “The social studies textbook Luther Burbank [Middle School in San Francisco] students use is so old that it does not reflect the breakup of the former Soviet Union” (First Amended Complaint in Williams v. State of California, 2000, p. 27). Four years after the case’s initial filing, Governor Arnold Schwarzenegger reached a settlement with the plaintiffs.

The Williams settlement, and its implementing legislation, earmarked funds in three issue areas: facility maintenance and repair, instructional materials, and teacher assignments and qualifications. Of the $1.2 billion dollars allotted in the settlement, $138.7 million was dedicated to instructional materials funding. The settlement, which focused certain resources on only the very lowest performing schools, produced some skepticism among education experts, leading to predictions that it would come up short in producing equal opportunity for learning among students (Glenn & Picus, 2007). Advocates, while acknowledging improvements in some areas, including current textbook access (Allen, 2005), continue to cite the Williams promise of every student having sufficient educational resources to support their learning as a “work in progress” (Chung, 2013, p. 8).

Recent research continues to emphasize persistently lower levels of OTL for disadvantaged students compared with their more advantaged counterparts (e.g., Abedi & Herman, 2010; Albano & Rodriguez, 2013; Barnard-Brak et al., 2018; Lafontaine et al., 2015; Ottmar et al., 2014; Santibañez & Fagioli, 2016; Wang, 2010). Lower levels of OTL are reported for English language learner (ELL; Abedi & Herman, 2010), special education (SPED; Kurz et al., 2014), female (Heafner & Fitchett, 2015), racial/ethnic minority (Heafner & Fitchett, 2015; Morton & Riegle-Crumb, 2020; Polikoff & Struthers, 2013), and low-income students (Heafner & Fitchett, 2015; Polikoff & Struthers, 2013), as well as students in urban schools (Polikoff & Struthers, 2013), relative to students not in these groups.

Research demonstrating variation in OTL based on student characteristics has relied on a variety of tools and methods to measure OTL. Heafner and Fitchett (2015), for example, rely on student survey data from the National Assessment of Educational Progress (NAEP), which includes items on instructional experiences and exposure to curriculum. Abedi and Herman (2010), draw from teacher and eighth-grade student surveys on Algebra I first semester topic coverage and an assessment of achievement in initial Algebra I content for ELL and non-ELL students. Kurz et al. (2014) study OTL among students with and without disabilities using data collected from self-reported daily logs of standards coverage, time allocated to each standard along five cognitive process expectations, instructional practices, and grouping formats (individual, small group, or whole class) from a small sample of teachers. Polikoff and Struthers (2013) use data the most similar to ours—teacher surveys of topic and cognitive demand based on the SEC. That these studies use such disparate methods but reach the same general conclusion bolsters the argument that there remain systematic disparities in OTL across student groups.

We further investigate systematic variation in OTL by focusing on differences in both content and rigor of textbooks across differences in school and district demographics. In doing so, we make three primary contributions to this field of literature. First, while many other studies of OTL rely on survey or observation data more limited in size, we take advantage of a statewide panel of schools. Our statewide data give us assurance that our findings are representative of the state (which is itself highly diverse). Second, the recent timing of our data allows for analysis of systematic variation in OTL specific to the post-CCSS implementation context, making the findings relevant at the present day, when the large majority of states still use the CCSS or a close facsimile (as of 2016, 36 states along with Washington, D.C. were still implementing the CCSS, for either all or a significant part of their curriculum, with another 11 states announcing major rewrites or replacements for the CCSS, and 4 states remaining that never adopted the CCSS; Ujifusa, 2016). Last, we analyze statewide demographic data in relation to detailed content analysis of textbooks, allowing us to measure OTL in terms of both content and rigor at a fine-grained level not possible in most other OTL studies.

The Current Study

To investigate the relationship between textbooks and OTL, we first describe variation in content coverage of standards and cognitive demand in twenty widely used eighth-grade mathematics (pre-Algebra and Algebra I) textbooks that were used in California schools in 2015–2016, when schools were transitioning to textbooks aligned with the adopted CCSS. We then analyze the extent to which student OTL, as characterized by our measures of textbook content and cognitive demand, is equitably distributed according to school- and district-level demographics.

Data

Textbook Selection

One obstacle to carrying out large-scale OTL studies of written curriculum is that most American states do not record information on school and district textbook usage. They may have a list of state-approved textbooks, but no way of telling which books are actually being used in which districts (Chingos & Whitehurst, 2012). As a result of the Williams settlement, however, California requires schools to publish an annual School Accountability Report Card (SARC), which includes information on adopted textbooks (typically the titles and adoption years are reported). We construct our eighth-grade math textbook data set, which includes full-course pre-Algebra and Algebra I books, by manually downloading and recording the SARC information for all schools serving eighth graders for school year 2015–2016 across 2,666 schools in 870 California districts serving about 1,656,742 students, including 446,328 eighth-grade students.¹ While schools are required to publish their yearly SARCs, there is no standardized process for reporting textbook information, resulting in a time-consuming process for gathering these data (for additional detail about our decisions in cleaning the SARC textbook data see Koedel et al., 2017; Polikoff, Campbell, et al., 2020). Although all the textbooks in our sample were in use in schools during the 2015–2016 school year, some books in the sample were published and adopted prior to CCSS and claim no alignment to the CCSS, while other books were published and adopted post-CCSS and do claim alignment.

Another obstacle to studying OTL through textbooks is that content analyzing these materials is time consuming and resource intensive. While we were unable to content analyze every textbook used by any California eighth grader, we selected and analyzed 20 textbooks that provided us the greatest coverage of California’s eighth-grade population: in 2015–2016, 46% of California eighth graders attended one of the 974 schools across 347 districts with at least one of our selected textbooks adopted.² Table 1 shows a full list of textbooks included in our sample, as well as the number of districts overall, uniform textbook adoption districts,³ schools and students represented by each textbook.

Table 1

Textbook Descriptions

Textbook	Publisher	Publication year	No. of districts	No. of uniform adoption districts	No. of schools	Total student enrollment	Eighth-grade student enrollment
Algebra Connections, Volume 1–2	College Preparatory Mathematics	2008	13	6	26	18,714	6,364
Big Ideas Math Course 3	Big Ideas Learning LLC	2015	27	16	106	82,673	25,749
Carnegie Learning Math Series Course 3: A Common Core Math Approach, Volume 1–2	Carnegie Learning	2011	24	17	50	30,765	9,931
CGP Education California Mathematics Course Two	CGP	2007	1	0	9	10,485	3,518
Core Connections Algebra	College Preparatory Mathematics	2013	19	10	53	35,680	12,117
Core Connections Course 3	College Preparatory Mathematics	2013	55	33	118	68,295	21,771
Glencoe Math Course 3, Volume 1–2	McGraw-Hill	2013	42	32	150	114,829	38,659
Glencoe McGraw-Hill California Algebra 1: Concepts, Skills, and Problem Solving	McGraw-Hill	2008	7	4	11	5,494	1,281
Go Math	Houghton Mifflin Harcourt	2014	144	109	346	220,336	59,796
Holt Algebra 1	Holt, Rinehart and Winston	2008	25	16	60	46,918	16,050
Holt California Algebra Readiness, Volume 1–4	Holt, Rinehart and Winston	2008	4	3	13	12,678	4,317
Holt California Mathematics Course 2: Pre-Algebra	Holt, Rinehart and Winston	2008	19	12	45	34,913	12,018
McDougal-Littell Algebra Readiness	McDougal Littell	2008	7	4	12	6,447	2,665
McDougal-Littell California Math Algebra 1	McDougal Littell	2008	18	10	29	16,373	5,081
McDougal-Littell California Mathematics Concepts and Skills, Course 2 Pre-Algebra	McDougal Littell	2001	13	7	28	17,345	6,129
Prentice Hall California Pre-Algebra	Pearson Prentice Hall	2009	17	8	54	40,395	13,060
Prentice Hall Mathematics Algebra 1	Pearson Prentice Hall	2009	23	9	71	57,392	15,812
Prentice Hall Mathematics Algebra Readiness	Pearson Prentice Hall	2009	7	2	17	14,207	5,099
Springboard Algebra 1	College Board	2014	4	3	31	21,101	8,713
Springboard Mathematics Course 3	College Board	2014	7	2	34	20,289	7,885

School Characteristics

To obtain school demographic characteristics, we draw from the National Center for Education Statistics’ Common Core of Data (CCD), the Office of Civil Rights’ Civil Rights Data Collection (CRDC), and the California Department of Education for school year 2015–2016. We obtain the following characteristics at the school level from the CCD: total school enrollment, as well as enrollment by grade, gender, and race/ethnicity; number of students eligible for free or reduced-price lunch (FRL); urbanicity (city, suburban, town, or rural), alternative school status, and charter status. From the CRDC we obtain school counts of students classified as SPED or ELL. From the California Department of Education, we obtain mean scale scores on the California Assessment of Student Performance and Progress (CAASPP) standardized math test for students.

Methods

Research Question 1

To address our first research question focusing on the variation in alignment to content standards and cognitive demand in textbooks, we apply the SEC framework for mathematics to quantitatively code the content in the standards and textbooks. The SEC provides a taxonomy of mathematics content at the intersection of specific topics and levels of cognitive demand. The instrument has been in use for over two decades, undergoing developments along the way to reflect changes in content standards (see Porter, 2002, for a history; for other examples, see Polikoff, 2012a, 2012b; Porter et al., 2007), including those changes brought on by the adoption of the Common Core (see Porter et al. [2011] for an analysis of the Common Core using the SEC and Polikoff, Gasparian, et al. [2020] for recent work to update the SEC framework).

The SEC mathematics framework used in this study is the most recent version described by Polikoff, Gasparian, et al. (2020). It includes 228 topics (e.g., surface area, quadratic functions), that are grouped under 16 broader categories (e.g., measurement, basic Algebra). The SEC content language is intended to be inclusive of any content that might be covered in typical K–12 mathematics instruction. Orthogonal to the topics are six levels of cognitive demand, which are based on a modified version of Bloom’s taxonomy. Figure 1 shows an image from the SEC teacher survey that illustrates the topic-by-cognitive demand organization of the instrument (we refer to the intersection of a topic and a cognitive demand level as a “cell”). The SEC framework can be applied to analyze content standards, assessments, curriculum materials, and teachers’ instructional content. An outline of the SEC framework in mathematics, including a list of topics and cognitive demand levels, is provided in Supplemental Appendix A, available in the online version of this article.

Figure 1.

Surveys of Enacted Curriculum (SEC) teacher survey.

We apply the SEC framework to the textbooks in our sample following the methods described in detail, with a worked example, in Polikoff (2015). The first step is “chunking” the textbooks, by which we mean breaking each textbook down into its finest-grained level of detail. The “chunks” in textbooks typically consist of sections of text (typically set off by headings) and problems for students to complete (including example problems, each problem treated individually). These are all equally weighted. We hired an experienced mathematics teacher who had familiarity with the SEC to go through the 20 books and “chunk” them, preparing the coding sheets for reviewers to use. In total the number of chunks in the 20 books ranged from 2,857 (Go Math) to 10,789 (Holt Algebra 1) with most in the 4,000 to 8,000 range.

Once the books have been chunked, the task is to analyze each chunk in terms of its content and cognitive demand coverage. Content analysts examine each chunk and code it as covering up to six topic-by-cognitive demand combinations.⁴ The weight for that chunk is then evenly divided among the SEC cells that were selected. Finally, the codes for all of the chunks in a book are aggregated and the results are turned into proportions. If multiple raters are used, the codes for the multiple raters are then averaged to arrive at the final content analysis, which takes the form of a matrix of proportions, one for each cell, indicating the percentage of the textbook’s content on that particular topic-by-cognitive demand combination. The procedure for analyzing content standards is identical. The “chunks” in content standards are objectives, which are again coded as covering up to six SEC cells apiece.

Since analyzing full textbook content using the SEC framework is time intensive, the analyses presented here rely primarily on a single coder—the same experienced mathematics teacher who created the coding sheet. Prior examinations of the quality of textbook analyses indicate that these are highly reliable, with generalizability coefficients well above .90 for even two raters (Polikoff et al., 2015). This implies that the difference between the ratings from one coder and the ratings from another would be negligible. Here, we also check the reliability of our coding by randomly sampling one chapter within each of four randomly selected textbooks to be analyzed by a second coder—a doctoral student and former elementary teacher with advanced quantitative training. No training was provided, just a brief orientation to the process and content language. We calculated an alignment index for each chapter, comparing the main coder’s ratings with the second coder’s ratings. In all cases, these alignment indices were .94 or above, indicating a high degree of agreement, and supporting the consistency with which the SEC framework is applied in this study.

We similarly apply the SEC to eighth-grade mathematics standards, following the same general procedure (for standards the “chunks” are simply objectives). We focus on the grade-level standards because those are the standards against which students are assessed, and also because our research questions require an apples-to-apples comparison given the uneven distribution of students across Algebra/pre-Algebra classes both within and between districts.

After applying the SEC to textbooks and content standards, we create indicators for both content alignment and for cognitive demand coverage. We calculate two alignment indices. The main alignment index (Porter, 2002) is

A l i g n m e n t I n d e x 1 = 1 - \sum_{i} \frac{| x_{i} - y_{i} |}{2}

Here, x_i represents the proportion of document x (e.g., a textbook) on cell i, while y_i represents the proportion of document y (e.g., a set of standards) on that same cell. The resulting value, which ranges from 0 to 1, indicates the proportion of documents x and y that are in exact proportional agreement on content. A relaxed version of Equation 1 is as follows (Polikoff, 2012a):

A l i g n m e n t I n d e x 2 = \sum_{i | y_{i} > 0} x_{i}

The difference here is that exact proportional agreement is no longer required. This index also ranges from 0 to 1 and represents the proportion of document x’s content (i.e., textbook content) that is on SEC cells that are covered at all in the standards. In other words, this index only penalizes textbooks for covering content that is not emphasized at all in the standards. For the books analyzed here, the correlation between the two indices is .40, indicating they provide related, but distinct information about alignment.

In addition to these two alignment indices, we also calculate the simple proportion of each textbook’s content at each of the six levels of cognitive demand. This analysis is based on previous work in English language arts (Polikoff & Struthers, 2013), which found that students in more disadvantaged schools were more likely to be exposed to lower cognitive demand skills during the No Child Left Behind Act era.

Research Question 2

To analyze our second research question, focusing on the relationship between school demographics and variation in standards alignment and cognitive demand, we rely on ordinary least squares regression at the school level.⁵ Using each of our indices as an outcome, we apply the following model:

\begin{array}{l} Y_{i} = E n r o l l m e n t_{i} + R a c e_{i} + F R L_{i} + S P E D_{i} + E L L_{i} + M a t h S c o r e_{i} \\ + U r b a n i c i t y_{i} + C h a r t e r_{i} + A l t e r n a t i v e_{i} + P r e P u b l i c a t i o n_{i} + ε_{i} \end{array}

Here, $Y_{i}$ is our content measure of interest (measuring either alignment or cognitive demand) for the textbooks used in school i;⁶ $E n r o l l m e n t_{i}$ indicates total eighth-grade school enrollment, which we divide by 100; $R a c e_{i}$ , $F R L_{i}$ , $S P E D_{i}$ , and $E L L_{i}$ , indicate the racial/ethnic, FRL, SPED, and ELL distribution of the school⁷; $M a t h S c o r e_{i}$ denotes mean scale score on the CAASPP eighth-grade math test; $U r b a n i c i t y_{i}$ indicates whether a school is located in a city, suburban, town, or rural; and $C h a r t e r_{i}$ and $A l t e r n a t i v e_{i}$ are indicators for whether a school is a charter and/or alternative school.⁸ $P r e P u b l i c a t i o n_{i}$ indicates the percentage of books in our sample at a school that list adoption dates occurring before the publication dates of the editions we coded.⁹ $ε_{i}$ indicates the error term. Standard errors are clustered at the district level. In addition to two alignment indices and six individual cognitive demand levels, we also implement a version of this model that combines Cognitive Demand Levels B and C since these two levels are both the two most basic levels of cognitive demand and together these two levels comprise an average of about 84% of textbook content in our sample. In online Supplemental Appendix B, we report results from a similar model run at the district level in uniform-adopting districts, where we use the corresponding district-level count, percentage, or average for all measures discussed above, weighting a district’s average eighth-grade math score across schools by the number of students with scores at each school. We do not use corrections for multiple hypothesis testing in our analyses because we view the research as more exploratory and are more concerned about reducing Type II error (i.e., finding all the relevant predictors of OTL inequities that might exist). That said, applying the Romano–Wolf stepdown approach (Romano & Wolf, 2016) to our results substantially reduces the number of statistically significant coefficients in the school-level models, and removes all statistical significance in the district-level models. The corrected results are available on request.

Results

Research Question 1

Alignment Differences Across Books

We begin analysis for our first research question by focusing on the extent to which textbooks align with the CCSS using either of our alignment indices (see Table 2). The results for the main alignment index indicate that only about one fifth to one third of textbook content aligns with the CCSS in terms of both topic and cognitive demand, with an average alignment of 0.25 (SD 0.03). Using the second alignment index, we find that the proportion of textbook content covering content found within the CCSS ranges from 0.36 to 0.60, with an average alignment of 0.51 (SD 0.06). The difference between these two alignment indices ranges from 16 to 37 percentage points across books, indicating the degree to which textbooks are overemphasizing content relative to what our analysts say the CCSS calls for. Subtracting the second alignment index from 1 indicates that 40% to 64% of textbook content is on topic/cognitive demand combinations not included in the standards at all.

Table 2

Textbook Alignment With California’s Common Core State Standards (CCSS)–Alignment Index 1 Ranking

Textbook	Pre- or post-CCSS	Subject	Alignment Index 1	Alignment Index 2	Difference	Alignment Index 1 ranking	Alignment Index 2 ranking
Big Ideas Math Course 3	Post-CCSS	Pre-Algebra	0.319	0.553	0.234	1	6
Go Math	Post-CCSS	Pre-Algebra	0.314	0.526	0.212	2	9
Springboard Mathematics Course 3	Post-CCSS	Pre-Algebra	0.304	0.512	0.207	3	12
Glencoe Math Course 3, Volume 1–2	Post-CCSS	Pre-Algebra	0.293	0.515	0.223	4	11
Core Connections Course 3	Post-CCSS	Pre-Algebra	0.285	0.536	0.250	5	7
Carnegie Learning Math Series Course 3: A Common Core Math Approach, Volume 1–2	Post-CCSS	Pre-Algebra	0.282	0.504	0.222	6	13
Algebra Connections, Volume 1–2	Pre-CCSS	Algebra 1	0.268	0.580	0.312	7	2
Core Connections Algebra	Post-CCSS	Algebra 1	0.251	0.530	0.279	8	8
Springboard Algebra 1	Post-CCSS	Algebra 1	0.245	0.522	0.277	9	10
Holt California Mathematics Course 2: Pre-Algebra	Pre-CCSS	Pre-Algebra	0.243	0.470	0.227	10	15
CGP Education California Mathematics Course Two	Pre-CCSS	Pre-Algebra	0.235	0.450	0.215	11	16
Holt Algebra 1	Pre-CCSS	Algebra 1	0.234	0.562	0.328	12	5
Glencoe McGraw-Hill California Algebra 1: Concepts, Skills, and Problem Solving	Pre-CCSS	Algebra 1	0.231	0.575	0.344	13	4
Prentice Hall Mathematics Algebra 1	Pre-CCSS	Algebra 1	0.229	0.596	0.366	14	1
McDougal-Littell Algebra Readiness	Pre-CCSS	Pre-Algebra	0.229	0.431	0.202	15	18
McDougal-Littell California Mathematics Concepts and Skills, Course 2 Pre-Algebra	Pre-CCSS	Pre-Algebra	0.227	0.486	0.259	16	14
McDougal-Littell California Math Algebra 1	Pre-CCSS	Algebra 1	0.226	0.577	0.351	17	3
Prentice Hall California Pre-Algebra	Pre-CCSS	Pre-Algebra	0.223	0.412	0.189	18	19
Prentice Hall Mathematics Algebra Readiness	Pre-CCSS	Pre-Algebra	0.208	0.448	0.239	19	17
Holt California Algebra Readiness, Volume 1–4	Pre-CCSS	Pre-Algebra	0.199	0.362	0.164	20	20
Textbook Average			0.252	0.507	0.255
Pre-CCSS Pre-Algebra Textbook Average			0.224	0.437	0.214
Post-CCSS Pre-Algebra Textbook Average			0.299	0.524	0.225
Pre-CCSS Algebra Textbook Average			0.238	0.578	0.340
Post-CCSS Algebra Textbook Average			0.248	0.526	0.278

These alignment values are somewhat lower than those found for third-grade mathematics textbooks (0.28 to 0.40 for Alignment Index 1, 0.64 to 0.80 for Alignment Index 2) in the one prior published study of mathematics textbook alignment to standards using the SEC (Polikoff, 2015). Prior research on alignment indices cautions against direct comparisons of alignment indices outside the confines of a single study, however, because features of the alignment process such as the number of cells in the coding framework can affect the expected magnitude of alignment indices (Polikoff & Fulmer, 2013).

The alignment indices also reveal patterns by adoption date and subject. We find that, while overall alignment with the CCSS remains low, textbooks that were adopted after CCSS implementation are in the top half of the rankings for Alignment Index 1 (eight of the top nine most aligned books), while textbooks that were adopted prior to CCSS implementation rank primarily among the bottom half. In other words, the books claiming CCSS alignment were indeed more aligned to the CCSS than prior textbooks. Perhaps not surprisingly given that the CCSS moved a good deal of Algebra content to high school, we also find that post-CCSS pre-Algebra books rank higher than any other textbooks on the first alignment index.

Interestingly, as shown in Table 2, comparing books on the alternate alignment index does change the rankings. For instance, post-CCSS books are no longer the most aligned when the alignment index no longer requires exact proportional agreement. Based on the second alignment index, the five books measured as most closely aligning with the CCSS are pre-CCSS pre-Algebra books. In the next section, we explore areas of misalignment and explain these differences between the two indices.

Exploring Areas of Misalignment

To shed some descriptive light on the misalignment between textbooks and the CCSS, we examine differences in content coverage at the coarse-grained topic level. For these analyses, we first calculate the percentage of CCSS content under each of the 16 coarse-grained topics (only 11 of the 16 are represented at all in the eighth-grade CCSS; see online Supplemental Appendix C). Then, we calculate the percentage of textbook content on CCSS-covered cells under those same 16 coarse-grained topics. Finally, we take the difference. So, for instance, we find that 3.5% of CCSS eighth-grade mathematics content is on whole number operations, and Big Ideas Math Course 3 allocates an additional 2.1% more of its content to the CCSS cells covered under whole number operations (so 5.6% of its total content).

Based on this analysis, we find that all textbooks overemphasize SEC cells within the coarse-grained topic area of whole number operations by an overall average of about six percentage points (the CCSS allocate about 3.5% of their content to this topic; textbooks allocate about 9.4% of their content to it). Pre-CCSS pre-Algebra books fare especially poorly here, allocating about 12% of their content to this topic versus just 6.1% for post-CCSS pre-Algebra books. In contrast, all textbooks underemphasize CCSS content within most other coarse-grained topics. Geometric concepts, in particular, is dramatically underemphasized in textbooks relative to the standards. While about 30% of CCSS content is on geometric concepts, textbooks average just 5.4% content on geometric concept cells covered by the standards.

We also analyze the content in the books by aggregating the textbook content on cells not covered at all in the CCSS under each coarse-grained area (see online Supplemental Appendix D). Based on this analysis, we reach several conclusions. First, textbooks do not cover much content in coarse-grained areas that are not covered at all in the standards (these are consumer applications, probability, analysis, trigonometry, special topics). Only about 1% of textbook coverage on average is on these areas. In contrast, about 48% of textbook content on average is found within coarse-grained topics that are covered in the standards (the remaining 51% of textbook content is Alignment Index 2, i.e., it is on cells covered in the standards). Areas that are particularly emphasized in textbooks include basic algebra (about 13% of textbook content on average is on basic Algebra cells that are not covered at all in the CCSS); fraction, decimal, and percentage of operations (about 9%); and whole number operations (about 7%).

We find interesting patterns when comparing pre- with post-CCSS pre-Algebra books. Pre-CCSS pre-Algebra books emphasize non-CCSS content under whole number operations (9.0%); fraction, decimal, and percentage of operations (17.0%); and number sense/properties/relationships (9.3%) to a far greater extent than do post-CCSS pre-Algebra books (5.1%, 5.7%, 3.9%, respectively). In contrast, post-CCSS pre-Algebra books actually fare worse than pre-CCSS pre-Algebra books in their emphasis of non-CCSS basic Algebra content (12.8% to 8.6%, respectively).

Cognitive Demand in the Textbooks

In addition to examining differences between textbooks and the CCSS by coarse-grained topics, we also analyze cognitive demand. To do this, we sum any fine-grained topic covered at each of the six levels of cognitive demand (regardless of whether the cell is covered by the CCSS). This allows us to determine the total proportion of each textbook, as well as the CCSS, that is devoted to coverage at each level of cognitive demand. In fact, we find striking differences in the cognitive demand levels emphasized in the textbooks in our sample compared with the CCSS.

Table 3, ordered by Alignment Index 1, displays the proportion of each textbook devoted to coverage of each level of cognitive demand, as well as the proportion of coverage devoted to each of these levels in the CCSS in the bottom row. We find that, on average, the textbooks in our sample and the CCSS devote roughly the same proportion of coverage, about 7%, to topics using memorize/recall techniques (Level B), which comprise the most basic level of cognitive demand but are likely fundamental to the development of more complex cognitive skills. Similarly, we find almost matching levels of coverage when it comes to cognitive approaches requiring students to make generalizations (Level F): both our sample of textbooks and the CCSS devote about 4% of coverage. Beyond these two categories of cognitive demand, however, we find little agreement between our textbook sample and the CCSS in the level of rigor required by content coverage.

Table 3

Textbook Total Coverage of Common Core State Standards (CCSS) by Coarse-Grained Cognitive Demand Category Alignment Index 1 Ranking

Textbook	Alignment Index 1 rank	Adoption	Subject	B (memorize/recall)	C (perform procedures)	D (demonstrate/communicate understanding)	E (justify/evaluate)	F (Generalize)	G (apply to real-world problems)
Big Ideas Math Course 3	1	Post-CCSS	Pre-Algebra	0.064	0.719	0.109	0.052	0.053	0.003
Go Math	2	Post-CCSS	Pre-Algebra	0.099	0.699	0.105	0.037	0.056	0.005
Springboard Mathematics Course 3	3	Post-CCSS	Pre-Algebra	0.059	0.723	0.107	0.036	0.063	0.012
Glencoe Math Course 3, Volume 1–2	4	Post-CCSS	Pre-Algebra	0.102	0.716	0.097	0.030	0.047	0.009
Core Connections Course 3	5	Post-CCSS	Pre-Algebra	0.039	0.717	0.122	0.050	0.072	0.001
Carnegie Learning Math Series Course 3: A Common Core Math Approach, Volume 1–2	6	Post-CCSS	Pre-Algebra	0.078	0.678	0.130	0.044	0.066	0.004
Algebra Connections, Volume 1–2	7	Pre-CCSS	Algebra 1	0.062	0.684	0.147	0.062	0.042	0.004
Core Connections Algebra	8	Post-CCSS	Algebra 1	0.034	0.729	0.080	0.051	0.105	0.002
Springboard Algebra 1	9	Post-CCSS	Algebra 1	0.065	0.733	0.093	0.043	0.064	0.002
Holt California Mathematics Course 2: Pre-Algebra	10	Pre-CCSS	Pre-Algebra	0.085	0.807	0.068	0.021	0.018	0.002
CGP Education California Mathematics Course Two	11	Pre-CCSS	Pre-Algebra	0.069	0.836	0.054	0.024	0.010	0.006
Holt Algebra 1	12	Pre-CCSS	Algebra 1	0.064	0.801	0.057	0.057	0.019	0.001
Glencoe McGraw-Hill California Algebra 1: Concepts, Skills, and Problem Solving	13	Pre-CCSS	Algebra 1	0.050	0.830	0.045	0.050	0.023	0.002
Prentice Hall Mathematics Algebra 1	14	Pre-CCSS	Algebra 1	0.063	0.854	0.030	0.035	0.017	0.001
McDougal-Littell Algebra Readiness	15	Pre-CCSS	Pre-Algebra	0.064	0.826	0.057	0.036	0.014	0.003
McDougal-Littell California Mathematics Concepts and Skills, Course 2 Pre-Algebra	16	Pre-CCSS	Pre-Algebra	0.092	0.794	0.051	0.047	0.009	0.007
McDougal-Littell California Math Algebra 1	17	Pre-CCSS	Algebra 1	0.076	0.824	0.040	0.036	0.018	0.005
Prentice Hall California Pre-Algebra	18	Pre-CCSS	Pre-Algebra	0.107	0.783	0.071	0.025	0.013	0.001
Prentice Hall Mathematics Algebra Readiness	19	Pre-CCSS	Pre-Algebra	0.067	0.849	0.047	0.028	0.009	0.001
Holt California Algebra Readiness, Volume 1–4	20	Pre-CCSS	Pre-Algebra	0.092	0.781	0.060	0.061	0.005	0.000
Textbook Average				0.071	0.769	0.078	0.041	0.036	0.004
Pre-CCSS Pre-Algebra Textbook Average				0.082	0.811	0.058	0.035	0.011	0.003
Post-CCSS Pre-Algebra Textbook Average				0.073	0.709	0.112	0.041	0.060	0.006
Pre-CCSS Algebra Textbook Average				0.063	0.799	0.064	0.048	0.024	0.003
Post-CCSS Algebra Textbook Average				0.049	0.731	0.086	0.047	0.085	0.002
CCSS				0.066	0.378	0.320	0.181	0.040	0.015

In general, we find that textbooks devote a much higher proportion of coverage to content using techniques relying on performing procedures (Level C) at the expense of techniques that require students to demonstrate/communicate understanding (Level D) and provide justifications/evaluations (Level E). On average, textbooks in our sample cover over three quarters of content through techniques involving performing procedures, whereas the CCSS devotes just over a third of coverage to this cognitive demand level. The highest levels of emphasis on procedures can be found in pre-CCSS books for pre-Algebra and Algebra, with an average of about 81% and 80% for each category of textbooks, respectively. In short, our findings on cognitive demand indicate that these commonly used eighth-grade math textbooks are universally underdelivering on the cognitive demand expectations called for by the CCSS.

Overall, our alignment indices reveal major discrepancies that remain between the CCSS and textbooks in our sample, and we find that these differences remain prominent along both dimensions of coarse-grained topic and coarse-grained cognitive demand. While the main alignment index does show that post-CCSS pre-Algebra books are the most aligned with the standards, when we use other alignment indices the results are not as positive. Furthermore, all of the books, regardless of whether they were written pre- or post-CCSS, overemphasize procedural skills relative to what is called for in the standards.

Research Question 2

We next examine the extent to which the variation in textbook alignment with the CCSS, as well as cognitive demand coverage of the adopted textbooks, are associated with student characteristics. We begin by describing our school-level findings (see Table 4). We briefly discuss our district-level findings—those results are available in the online Supplemental Appendix B.

Table 4

Fitted Models Estimating the Relationship Between Alignment Indices/Cognitive Demand Levels and School-Level Variables

Variable	Alignment Index 1	Alignment Index 2	Cognitive Level B	Cognitive Level C	Cognitive Level D	Cognitive Level E	Cognitive Level F	Cognitive Level G	Cognitive Level BC combo
Variable	b (SE)	b (SE)	b (SE)	b (SE)	b (SE)	b (SE)	b (SE)	b (SE)	b (SE)
Enrollment/100–eighth grade	−0.00260* (0.00131)	−0.00236* (0.00099)	0.00033 (0.00086)	0.00256 (0.00179)	−0.00132 (0.00090)	−0.00025 (0.00031)	−0.00134^† (0.00074)	0.00001 (0.00011)	0.00289^† (0.00169)
Race/ethnicity (Reference: % Race/ethnicity White–eighth grade)
% Race/ethnicityBlack–eighth grade	0.00004 (0.00026)	−0.00022 (0.00021)	0.00004 (0.00029)	−0.00014 (0.00023)	−0.00003 (0.00010)	−0.00011 (0.00008)	0.00018 (0.00014)	0.00006** (0.00002)	−0.00010 (0.00025)
% Race/ethnicityLatinx–eighth grade	−0.00011 (0.00014)	0.00005 (0.00008)	−0.00006 (0.00009)	0.00014 (0.00018)	−0.00009 (0.00009)	0.00002 (0.00003)	−0.00001 (0.00006)	0.00001 (0.00001)	0.00007 (0.00016)
% Race/ethnicityAsian–eighth grade	0.00021 (0.00015)	0.00001 (0.00009)	−0.00007 (0.00011)	−0.00040* (0.00018)	0.00020* (0.00009)	0.00005 (0.00003)	0.00020* (0.00008)	0.00002^† (0.00001)	−0.00047** (0.00016)
% Race/ethnicityAmerican Indian/Pacific Islander/2 or more races combined–eighth grade	−0.00027 (0.00025)	0.00008 (0.00019)	−0.00014 (0.00014)	0.00007 (0.00034)	−0.00002 (0.00020)	0.00003 (0.00005)	0.00006 (0.00012)	−0.00000 (0.00002)	−0.00006 (0.00034)
% Free or reduced-price lunch	0.00011 (0.00014)	−0.00020* (0.00010)	0.00015* (0.00007)	−0.00025 (0.00020)	0.00010 (0.00010)	−0.00002 (0.00003)	0.00002 (0.00007)	0.00001 (0.00001)	−0.00010 (0.00017)
% Special education	−0.00058^† (0.00034)	0.00040^† (0.00024)	−0.00029 (0.00024)	0.00095* (0.00045)	−0.00042^† (0.00024)	0.00005 (0.00009)	−0.00031^† (0.00018)	0.00002 (0.00003)	0.00066 (0.00043)
% English language learner	−0.00018 (0.00021)	0.00005 (0.00014)	0.00004 (0.00010)	0.00028 (0.00028)	−0.00014 (0.00015)	−0.00005 (0.00005)	−0.00012 (0.00010)	−0.00000 (0.00001)	0.00031 (0.00026)
Eighth-grade math mean scale score/100	−0.00097 (0.00070)	0.00030 (0.00060)	−0.00132^† (0.00070)	0.00087 (0.00095)	−0.00009 (0.00051)	0.00023 (0.00025)	0.00024 (0.00036)	0.00007 (0.00005)	−0.00045 (0.00094)
Charter	0.00060 (0.00539)	−0.00433 (0.00463)	0.00031 (0.00418)	−0.00297 (0.00737)	0.00277 (0.00407)	−0.00054 (0.00145)	0.00045 (0.00320)	−0.00003 (0.00038)	−0.00266 (0.00790)
Alternative	−0.01050 (0.01087)	−0.00282 (0.00705)	0.00733^† (0.00408)	0.01587 (0.01431)	−0.01244^† (0.00748)	−0.00324* (0.00139)	−0.00746 (0.00459)	−0.00007 (0.00057)	0.02320^† (0.01203)
Urbanicity (Reference: City)
Suburban	−0.00125 (0.00576)	−0.00357 (0.00388)	0.00890** (0.00313)	0.00050 (0.00685)	−0.00220 (0.00359)	−0.00332* (0.00133)	−0.00437 (0.00279)	0.00049 (0.00038)	0.00940 (0.00680)
Town	0.00417 (0.00642)	−0.00416 (0.00436)	0.00033 (0.00485)	−0.01427^† (0.00824)	0.00811^† (0.00433)	−0.00051 (0.00150)	0.00644^† (0.00381)	−0.00012 (0.00046)	−0.01393^† (0.00833)
Rural	−0.00396 (0.00719)	−0.00438 (0.00551)	0.00469 (0.00405)	0.00010 (0.00974)	−0.00115 (0.00511)	−0.00102 (0.00143)	−0.00298 (0.00374)	0.00036 (0.00042)	0.00479 (0.00906)
% observations with pre-pub date	0.00001 (0.00013)	0.00013^† (0.00008)	−0.00012*** (0.00003)	0.00025^† (0.00015)	−0.00007 (0.00008)	0.00006* (0.00003)	−0.00010* (0.00005)	−0.00002*** (0.00000)	0.00013 (0.00015)
_cons	0.32576*** (0.02065)	0.53035*** (0.01769)	0.10888*** (0.01798)	0.69644*** (0.03026)	0.10614*** (0.01642)	0.03663*** (0.00644)	0.05070*** (0.01064)	0.00120 (0.00151)	0.80532*** (0.02906)
N	877	877	877	877	877	877	877	877	877
R ²	.045	.097	.108	.094	.075	.124	.121	.123	.094
Adjusted R²	.028	.082	.092	.078	.059	.108	.106	.108	.078
df_model	15	15	15	15	15	15	15	15	15
df_error	301	301	301	301	301	301	301	301	301
RMSE	0.032	0.023	0.020	0.043	0.023	0.008	0.018	0.002	0.042
F	1.207	2.186	5.647	2.150	2.453	2.246	2.820	2.743	3.129

Note. Standard errors clustered at the district level. RMSE = root mean square error.

†

p < .10. *p < .05. **p < .01. ***p < .001.

School Level

Looking first at the regressions predicting alignment indices, we find limited evidence of differences in OTL as measured by alignment. The only consistent finding between the two models is for school size, with larger schools adopting slightly less aligned textbooks on both indices (p < .05).

Relationships of alignment with student demographics are variable, but all evidence indicates any statistically significant associations are small in magnitude. The only student demographic characteristic that is statistically significant at the .05 level is for the proportion of students receiving FRL on the second alignment index. This magnitude of this coefficient indicates that a 10 percentage point increase in the proportion of FRL students is associated with a 0.2 percentage point decrease, or about 6% the standard deviation of textbook alignment indices, on Alignment Index 2.

The remaining columns of Table 4 are for the relationship of school and student characteristics with cognitive demand coverage. We include both the six individual cognitive demand levels and, in the final column, an aggregate of Cognitive Demand Levels B (memorize) and C (perform procedures). Again, the broad pattern is one of limited or weak associations, though there are individual coefficients that are statistically significant at the .05 level in each model. The most robust finding is that schools with more Asian students adopt higher cognitive demand textbooks (significantly lower on cognitive demand Level C and the B/C combo, significantly higher on Level D and F). There is some weaker evidence that schools with more students receiving free/reduced-price lunch adopt books emphasizing more memorization and schools with more students with disabilities adopt books emphasizing more procedures. In contrast, schools serving more Black students adopt textbooks emphasizing more application (Level G). In general, the magnitude of these relationships is again relatively modest. The largest coefficient is for students with disabilities on Cognitive Demand Level C—a 10 percentage point increase in the percentage of students with disabilities is associated with about 1% more procedural content.

District Level

Because many, but not all, textbook adoptions happen at the district level, we also ran all analyses in these “uniform-adopting” districts by aggregating school-level results to the district level. As these are a subset of districts represented in the school-level analysis, we would not expect identical results. Indeed, the results shown in the online Supplemental Appendix B indicate even fewer clear or consistent relationships of either alignment or cognitive demand with district characteristics. None of the main findings that were statistically significant at the .05 level in the school-level model remains significant in the district-level model.

Limitations

Our work aims to provide a thorough descriptive analysis of textbook OTL. In doing so, we make no causal claims about any relationships identified between school and district characteristics and our OTL measures.

It is also important to note three limitations in our data. First, SARC data, which is self-reported by schools and/or districts, may contain errors. When we are able to identify likely errors, we attempt to remedy these errors when possible as previously detailed. However, not all issues of missing or incorrectly entered data can be remedied. Moreover, in some cases, even when a school has valid SARC data, demographic data for the school may be missing. Second, while our data capture a single year snapshot of trends when examining the relationship between school and district characteristics and our OTL measures, we are not able to examine possible changes in trends across years. Third, due to the time-intensive analysis required to code textbook and CCSS alignment, our sample is limited to 20 textbooks analyzed by one primary coder. We attempt to mitigate the impact of these limitations by focusing our analysis on more commonly used textbooks in California and, for a limited portion of the analysis, confirming high levels of interrater reliability with a second coder.

As a consequence of our data limitations, some schools and districts are dropped from our sample. We aim to be transparent about our resulting sample by reporting the number of eighth-grade serving schools in the SARC data, the number of schools remaining after we limit our sample to schools that we determine to use at least one of our 20 textbooks, and the number of these schools with nonmissing demographic data used in regression calculations. However, we do acknowledge that these limitations result in a sample of schools that are not necessarily representative of all California schools.

Discussion

Our results reveal that problems of poor alignment of textbooks to standards reported elsewhere (e.g., Polikoff, 2015) were also present in eighth-grade mathematics, even after the statewide adoption of “standards-aligned” textbooks. While post-CCSS textbooks do appear to be more aligned than pre-CCSS textbooks, even the most highly ranked textbooks are not very well-aligned with the CCSS. The highest scoring textbook in our sample, for example, only achieves about a third of cell-based alignment with the CCSS using our primary alignment index. Even using the alternative version of our alignment index, which more generously scores textbook alignment with the CCSS since there is no penalty for overemphasis, we still find that, depending on the textbook, 40% to 64% of textbook content is on topic/cognitive demand combinations not covered by the CCSS.

Examining the data by coarse-grained topic and cognitive demand level, we find that textbooks dramatically underemphasize the geometry content in the standards and overemphasize operations. This could imply that the textbooks are devoting considerable space to prior-grades content, which would align with findings about the large amount of redundancy of content across grades in mathematics instruction (Polikoff, 2012c). We also find that the eighth-grade math textbooks in our sample overemphasize performing procedures and de-emphasize techniques relying on higher level cognitive functions requiring students to demonstrate/communicate understanding and provide justifications/evaluations. This aligns with work on elementary mathematics textbooks (Polikoff, 2015). Post-CCSS textbooks do fare somewhat better on this measure. Given the centrality of curriculum materials in mathematics teachers’ instruction, these results suggest that teachers may need to supplement core textbooks with more cognitively demanding material if students are to experience instruction on these more ambitious skills.

Given the challenges of determining textbook and CCSS alignment, it is perhaps also not surprising that we find only limited evidence of systematic differences in textbook alignment associated with school or district characteristics. When we do see significant relationships between school or district characteristics and our OTL measures, the relationships tend to be fairly small in magnitude and primarily occur with our measures of cognitive demand rather than alignment indices. Moreover, most of the small differences at the school level tend to disappear when we focus our analysis on the district level, although doing so of course omits districts without uniform adoptions from our sample entirely. In short, though the books themselves do differ from one another in their alignment to standards and their coverage of topics and cognitive demand levels, we find only modest evidence that these differences are systematically associated with the kinds of schools and districts that choose them.

Where we do find any statistically significant associations of school or student characteristics with our measures of textbook content, they are often—but not always—in the direction we might expect based on theory and prior literature. For instance, schools with more students receiving FRL adopt slightly less aligned textbooks and schools serving more students with disabilities or students receiving FRL adopt textbooks that emphasize lower levels of cognitive demand more strongly. Though far from overwhelming, these findings comport with recent research finding ongoing inequities in student OTL (e.g., New Teacher Project, 2018).

Our conclusions are particularly important in the context of the Williams settlement for two reasons. First, although more than 15 years have passed since Williams, our results suggest that California still has a long way to go in order to ensure that the intended and written curriculum are well aligned. This problem is unique neither to California classrooms nor to mathematics (see Polikoff, Wang, et al., 2020, for a recent analysis of curriculum coherence in English language arts). We must get better aligned materials in classrooms, and recent policy efforts (e.g., in Louisiana) and nonprofit activities like textbook reviews through EdReports.org are models for this kind of work. That said, because all our analyzed materials were published in 2015 or prior, we cannot say anything about the alignment of newer materials to standards (and ratings of alignment on EdReports.org have been trending upward over time).

Second, although we do not generally see large systematic differences in textbook OTL by school/district characteristics, it is important to also note that there are many other ways in which historically underserved student groups receive less access to OTL in and out of school. For instance, recent research finds that teachers of these student groups often lower the challenge of standards-aligned lessons in implementation, in essence denying students the OTL the materials seek to provide (New Teacher Project, 2018). And of course there are undoubtedly inequities in out-of-school learning opportunities such as access to enrichment and tutoring activities. Thus, while our findings suggest that textbook content may not be a primary driver of inequities in access to learning opportunities, clearly textbooks are not contributing much to the solution of these inequities.

Supplemental Material

sj-xlsx-1-ero-10.1177_23328584211065712 – Supplemental material for The Equitable Distribution of Opportunity to Learn in Mathematics Textbooks

Supplemental material, sj-xlsx-1-ero-10.1177_23328584211065712 for The Equitable Distribution of Opportunity to Learn in Mathematics Textbooks by Morgan S. Polikoff, Sarah J. Rabovsky, Daniel Silver and Rosalynn Lazar-Wolfe in AERA Open

Footnotes

Acknowledgements

We gratefully acknowledge financial support for this research from the National Science Foundation Grant No. 1445654, as well as the William T. Grant Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF or the William T. Grant Foundation.

Data and Code

ORCID iD

Daniel Silver

Notes

Authors

MORGAN S. POLIKOFF is an associate professor of education at the University of Southern California Rossier School of Education. He researches standards and curriculum policy.

SARAH J. RABOVSKY is a research analyst at the Tennessee Department of Education. She researches alternative schools in U.S. education.

DANIEL SILVER is a PhD candidate at the University of Southern California Rossier School of Education. He researches teachers’ selection and use of supplementary curriculum materials.

ROSALYNN LAZAR-WOLFE is a teacher at the Ojai Valley School in Ojai, California.

References

Abedi

Herman

(2010). Assessing English language learners’ opportunity to learn mathematics: Issues and limitations. Teachers College Record, 112(3), 723–746. https://doi.org/10.1177/016146811011200301

Agodini

Harris

Thomas

Murphy

Gallagher

(2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders. U.S. Department of Education, Institute of Education Sciences. https://files.eric.ed.gov/fulltext/ED512551.pdf

Albano

A. D.

Rodriguez

M. C.

(2013). Examining differential math performance by gender and opportunity to learn. Educational and Psychological Measurement, 73(5), 836–856. https://doi.org/10.1177/0013164413487375

Allen

B. M.

(2005). The Williams v. California settlement: The first year of implementation. American Civil Liberties Union of Southern California. https://decentschools.org/settlement/WilliamsReportWeb2005.pdf

American Civil Liberties Union Southern California. (n.d.). Williams v. State of California. https://www.aclusocal.org/en/cases/williams-v-state-california

Ball

D. L.

Camburn

Correnti

Phelps

Wallace

(1999). New tools for research on instruction and instructional policy: A web-based teacher log. https://www.education.uw.edu/ctp/content/new-tools-research-instruction-and-instructional-policy-web-based-teacher-log

Barnard-Brak

Lan

W. Y.

Yang

(2018). Differences in mathematics achievement according to opportunity to learn: A 4pL item response theory examination. Studies in Educational Evaluation, 56(March), 1–7. https://doi.org/10.1016/j.stueduc.2017.11.002

Bhatt

Koedel

(2012). Large-scale evaluations of curricular effectiveness: The case of elementary mathematics in Indiana. Educational Evaluation and Policy Analysis, 34(4), 391–412. https://doi.org/10.3102/0162373712440040

Bhatt

Koedel

Lehmann

(2013). Is curriculum quality uniform? Evidence from Florida. Economics of Education Review, 34(June), 107–121. https://doi.org/10.1016/j.econedurev.2013.01.014

10.

Blazar

Heller

Kane

T. J.

Polikoff

Staiger

D. O.

Carrell

Goldhaber

Harris

D. N.

Hitch

Holden

K. L.

Kurlaender

(2020). Curriculum reform in the Common Core era: Evaluating math textbooks across six U.S. states. Journal of Policy Analysis and Management, 39(4), 966-1019. https://doi.org/10.1002/pam.22257

11.

Callahan

R. M.

(2005). Tracking and high school English learners: Limiting opportunity to learn. American Educational Research Journal, 42(2), 305–328. https://doi.org/10.3102/00028312042002305

12.

Chingos

Whitehurst

G. J.

(2012). Choosing blindly: Instructional materials, teacher effectivness, and the Common Core. Brown Center on Education Policy at Brookings. https://www.brookings.edu/wp-content/uploads/2016/06/0410_curriculum_chingos_whitehurst.pdf

13.

Chung

(2013). Williams v. California: Lessons from nine years of implementation. American Civil Liberties Union of Southern California. https://www.aclusocal.org/en/publications/williams-v-california-lessons-nine-years-implementation

14.

Cooper

Liou

D. D.

(2018). The structure and culture of information pathways: Rethinking opportunity to learn in urban high schools during the ninth grade transition. High School Journal, 91(1), 43–56. https://doi.org/10.1353/hsj.2007.0020

15.

Floden

R. E.

(2002). The measurement of opportunity to learn. In Porter

A. C.

Gamoran

(Eds.), Methodological advances in crossnational surveys of educational achievement (pp. 231–266). National Academies Press.

16.

Gamoran

Porter

A. C.

Smithson

White

P. A.

(1997). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation and Policy Analysis, 19(4), 325–338. https://doi.org/10.3102/01623737019004325

17.

Glenn

W. J.

Picus

L. O.

(2007). The Williams settlement and the prospects for future school finance adequacy litigation in California. Journal of Education Finance, 32(3), 382–394. https://www.jstor.org/stable/40704301

18.

Greenwald

A. G.

(1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52(11), 1182–1186. https://doi.org/10.1037/0003-066X.52.11.1182

19.

Guiton

Oakes

(1995). Opportunity to learn and conceptions of educational equality. Educational Evaluation and Policy Analysis, 17(3), 323–336. https://doi.org/10.3102/01623737017003323

20.

Heafner

T. L.

Fitchett

P. G.

(2015). An opportunity to learn US history: What NAEP data suggest regarding the opportunity gap. High School Journal, 98(3), 226–249. https://doi.org/10.1353/hsj.2015.0006

21.

Husen

(1967). International study of achievement in mathematics: A comparison of 12 countries (Vol. 2). Almquist & Wiksell.

22.

Knapp

M. S.

(1995). Teaching for meaning in high-poverty classrooms. Teachers College Press.

23.

Koedel

Polikoff

M. S.

Hardaway

Wrabel

S. L.

(2017). Mathematics curriculum effects on student achievement in California. AERA Open, 3(1). https://doi.org/10.1177/2332858417690511

24.

Koedel

Polikoff

(2017, January 5). Big bang for just a few bucks: The impact of math textbooks in California. Brookings: Evidence Speaks. https://www.brookings.edu/research/big-bang-for-just-a-few-bucks-the-impact-of-math-textbooks-in-california/

25.

Kurz

Elliott

S. N.

Lemons

C. J.

Zigmond

Kloo

Kettler

R. J.

(2014). Assessing opportunity-to-learn for students with disabilities in general and special education classes. Assessment for Effective Intervention, 40(1), 24–39. https://doi.org/10.1177/1534508414522685

26.

Lafontaine

Baye

Vieluf

Monseur

(2015). Equity in opportunity-to-learn and achievement in reading: A secondary analysis of PISA 2009 data. Studies in Educational Evaluation, 47(December), 1–11. https://doi.org/10.1016/j.stueduc.2015.05.001

27.

Martin

M. O.

Mullis

I. V. S.

Gonzalez

E. J.

Smith

T. A.

Kelly

D. L.

(1999). School contexts for learning and instruction: IEA’s third international mathematics and science study (TIMSS). TIMSS International Study Center.

28.

McDonnell

L. M.

(1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17(3), 305–322. https://doi.org/10.3102/01623737017003305

29.

Morton

Riegle-Crumb

(2020). Is school racial/ethnic composition associated with content coverage in algebra? Educational Researcher, 49(6), 441–447. https://doi.org/10.3102/0013189X20931123

30.

Mullis

I. V. S.

Martin

M. O.

Foy

(2005). IEA’s TIMSS 2003 international report on achievement in the mathematics cognitive domains. TIMSS International Study Center. https://files.eric.ed.gov/fulltext/ED494652.pdf

31.

Mullis

I. V. S.

Martin

M. O.

Gonzalez

E. J.

Gregory

K. D.

Garden

R. A.

O’Connor

K. M.

Chrostowski

S. J.

Smith

T. A.

(2000). TIMSS 1999 international mathematics report: Findings from IEA’s repeat of the Third International Mathematics and Science Study at the eighth grade. TIMSS International Study Center. https://www.iea.nl/publications/study-reports/international-reports-iea-studies/timss-1999-international-mathematics

32.

National Commission on Excellence in Education. (1983). A nation at risk: The imperative for education reform. U.S. Government Printing Office. https://edreform.com/wp-content/uploads/2013/02/A_Nation_At_Risk_1983.pdf

33.

New Teacher Project. (2018). The opportunity myth: What students can show us about how school is letting them down—and how to fix it. Author. https://tntp.org/publications/view/student-experiences/the-opportunity-myth

34.

Niemi

(1996). Assessing conceptual understanding in mathematics: Representations, problem solutions, justifications, and explanations. Journal of Educational Research, 89(6), 351–363. https://doi.org/10.1080/00220671.1996.9941339

35.

Opfer

Kaufman

Thompson

(2017). Implementation of K–12 state standards for mathematics and English language arts and literacy: Findings from the American Teacher Panel. RAND. https://www.rand.org/pubs/research_reports/RR1529-1.html

36.

Ottmar

E. R.

Decker

L. E.

Cameron

C. E.

Curby

T. W.

Rimm-Kaufman

S. E.

(2014). Classroom instructional quality, exposure to mathematics instruction and mathematics achievement in fifth grade. Learning Environments Research, 17(2), 243–262. https://doi.org/10.1007/s10984-013-9146-6

37.

Polikoff

(2018, June 28). The challenges of curriculum materials as a reform lever. Brookings: Evidence Speaks. https://www.brookings.edu/research/the-challenges-of-curriculum-materials-as-a-reform-lever/

38.

Polikoff

(2021). Beyond standards: The fragmentation of education governance and the promise of curriculum reform. Harvard Education Press.

39.

Polikoff

M. S.

(2012a). Instructional alignment under no child left behind. American Journal of Education, 118(3), 341–368. https://doi.org/10.1086/664773

40.

Polikoff

M. S.

(2012b). The association of state policy attributes with teachers’ instructional alignment. Educational Evaluation and Policy Analysis, 34(3), 278–294. https://doi.org/10.3102/0162373711431302

41.

Polikoff

M. S.

(2012c). The redundancy of mathematics instruction in U.S. elementary and middle schools. Elementary School Journal, 113(2), 230-251. https://doi.org/10.1086/667727

42.

Polikoff

M. S.

(2015). How well aligned are textbooks to the Common Core Standards in mathematics? American Educational Research Journal, 52(6), 1185–1211. https://doi.org/10.3102/0002831215584435

43.

Polikoff

M. S.

Campbell

Korn

White

Wrabel

Hardaway

Gasparian

Koedel

(2020). CEPEG policy brief: Curriculum counts: Math and science textbook adoptions and effects. Center on Education Policy, Equity and Governance, Rossier School of Education, University of Southern California. https://www.readkong.com/page/cepeg-policy-brief-curriculum-counts-math-and-science-3914093

44.

Polikoff

M. S.

Fulmer

G. W.

(2013). Refining methods for estimating critical values for an alignment index. Journal for Research on Educational Effectiveness, 6(4), 380-395. https://doi.org/10.1080/19345747.2012.755593

45.

Polikoff

M. S.

Gasparian

Korn

Gamboa

Porter

A. C.

Smith

Garet

M. S.

(2020). Flexibly using the Surveys of Enacted Curriculum to study alignment. Educational Measurement: Issues and Practice, 39(2), 38–47. https://doi.org/10.1111/emip.12292

46.

Polikoff

M. S.

Porter

A. C.

(2014). Instructional alignment as a measure of teaching quality. Educational Evaluation and Policy Analysis, 36(4), 399–416. https://doi.org/10.3102/0162373714531851

47.

Polikoff

M. S.

Struthers

K. S.

(2013). Changes in the cognitive complexity of English instruction: The moderating effects of school and classroom characteristics. Teachers College Record, 115(8), 1–26. https://doi.org/10.1177/016146811311500802

48.

Polikoff

M. S.

Wang

E. L.

Haderlein

S. K.

Kaufman

J. H.

Woo

Silver

Opfer

V. D.

(2020). Exploring coherence in English language arts instructional systems in the Common Core era. RAND. https://doi.org/10.7249/RRA279-1

49.

Polikoff

M. S.

Zhou

Campbell

S. E.

(2015). Methodological choices in the content analysis of textbooks for measuring alignment with standards. Educational Measurement: Issues and Practice, 34(3), 10–17. https://doi.org/10.1111/emip.12065

50.

Porter

(1995). The uses and misuses of opportunity-to-learn standards. Educational Researcher, 24(1), 21–27. https://doi.org/10.2307/1176117

51.

Porter

McMaken

Hwang

Yang

(2011). Common core standards: The new U.S. intended curriculum. Educational Researcher, 40(3), 103–116. https://doi.org/10.3102/0013189X11405038

52.

Porter

A. C.

(2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3–14. https://doi.org/10.3102/0013189X031007003

53.

Porter

A. C.

Smithson

Blank

Zeidner

(2007). Applied measurement in education alignment as a teacher variable. Applied Measurement in Education, 20(1), 27–51. https://doi.org/10.1080/08957340709336729

54.

Romano

J. P.

Wolf

(2016). Efficient computation of adjusted p-values for resampling-based stepdown multiple testing. Statistics & Probability Letters, 113(June), 38–40. https://doi.org/10.1016/j.spl.2016.02.012

55.

Santibañez

Fagioli

(2016). Nothing succeeds like success? Equity, student outcomes, and opportunity to learn in high- and middle-income countries. International Journal of Behavioral Development, 40(6), 517–525. https://doi.org/10.1177/0165025416642050

56.

Schiller

K. S.

Schmidt

W. H.

Muller

Houang

R. T.

(2010). Hidden disparities: How courses and curricula shape opportunities in mathematics during high school. Equity & Excellence in Education, 43(4), 414–433. https://doi.org/10.1080/10665684.2010.517062

57.

Smithson

J. L.

Porter

A. C.

(1994). Measuring classroom practice: Lessons learned from efforts to describe the enacted curriculum: The Reform Up Close Study (CPRE Research Report Series Report #31). https://eric.ed.gov/?id=ED377563

58.

Tarr

J. E.

Chávez

Ó.

Reys

R. E.

Reys

B. J.

(2006). From the written to the enacted curricula: The intermediary role of middle school mathematics teachers in shaping students’ opportunity to learn. School Science and Mathematics, 106(4), 191–201. https://doi.org/10.1111/j.1949-8594.2006.tb18075.x

59.

Tate

(2001). Science education as a civil right: Urban schools and opportunity-to-learn considerations. Journal of Research in Science Teaching, 38(9), 1015–1028. https://doi.org/10.1002/tea.1045

60.

Törnroos

(2005). Mathematics textbooks, opportunity to learn and student achievement. Studies in Educational Evaluation, 31(4), 315–327. https://doi.org/10.1016/j.stueduc.2005.11.005

61.

Ujifusa

(2016, November 2). Map: Tracking the Common Core State Standards. Education Week. https://www.edweek.org/teaching-learning/map-tracking-the-common-core-state-standards

62.

Venezia

Maxwell-Jolly

(2007). The unequal opportunity to learn in California’s schools: Crafting standards to track quality (Working Paper No 07-2). Policy Analysis for California Education, University of California and Stanford University. https://files.eric.ed.gov/fulltext/ED510177.pdf

63.

Wang

A. H.

(2010). Optimizing early mathematics experiences for children from low-income families: A study on opportunity to learn mathematics. Early Childhood Education Journal, 37(4), 295–302. https://doi.org/10.1007/s10643-009-0353-9

64.

Williams v. State of California. (2000). Case No. 312236 Superior Court of the State of California, County of San Francisco. https://www.aclusocal.org/en/cases/williams-v-state-california

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB