Does ESSA Assure the Use of Evidence-based Educational Practices?

Abstract

The Every Student Succeeds Act (ESSA) of 2015 requires that K-12 educational agencies invest federal education funds in evidence-based practices. We estimated what percentage of Title I funds at a large school district are invested in practices supported by a single study meeting one of the top three tiers of evidence as defined by ESSA. Over 95% met this bar. When studies about each practice from four research repositories were considered, the percentage of funds invested in practices with overall positive or mostly positive ratings fell below 60%. These proportions did not change substantially after the introduction of ESSA.

Keywords

educational policy evaluation and assessment evidence federal policy public education school districts

In recent years, the U.S. federal government has allocated $41 billion to $58 billion to K-12 education under the Every Student Succeeds Act (ESSA), the 2015 reauthorization of the Elementary and Secondary Education Act of 1965 (ESEA) (U.S. Census Bureau, 2018, 2019, 2020). An additional $190 billion has been provided by the Elementary and Secondary School Emergency Relief (ESSER) Fund (Office of Elementary and Secondary Education, 2021). ESSA encourages investment of federal funds in evidence-based activities, strategies, or interventions. These are defined as practices which have been shown to improve “student outcomes or other relevant outcomes” (ESSA, 2015, p. 388) by at least one well-designed and well-implemented experimental study (Tier I evidence), quasi-experimental study (Tier II evidence), or correlational study (Tier III evidence). Absent such studies, and to accommodate innovative practices (Jochim & Gross, 2016), Tier IV evidence requires a “rationale based on high-quality research findings or positive evaluation” and “ongoing efforts to examine the effects of such activity, strategy, or intervention” (p. 388–389).

ESSA’s tiered evidence approach differs substantially from the prior iteration of ESEA: the No Child Left Behind Act (NCLB) of 2002. NCLB exhorted education agencies to offer educational activities based on scientifically based research, defined as experimental or quasi-experimental studies. Despite the apparent demand for rigor, the lack of such studies at that time effectively rendered this requirement difficult to meet or enforce (West, 2016). Proponents of ESSA’s more permissive definition anticipated greater compliance by education agencies because it is more feasible to meet the evidence requirement and, at least for Tiers I–III, clearer what constitutes acceptable evidence. Supporters also anticipated education agencies would spend ESSA funds on producing local evidence by evaluating activities in which they invested the funds (e.g., Kane, 2017). Other commentators cautioned that ESSA could instead harm the evidence-based movement because almost any activity could be implemented under Tier IV (Slavin, 2017). Alternatively, the pressure to use evidence might encourage adoption of evidence-based practices unsuitable for the local context (Yoshizawa, 2021).

Use of evidence to inform education agency decisions is expected to improve academic and other outcomes for students (Heinrich, 2007; Massell et al., 2012; U.S. Department of Education, 2016). In addition to setting standards for what counts as evidence and when it should be used, ESSA asks education agencies to provide activities that are responsive to the needs of their local populations. The U.S. Department of Education’s guidance integrates these two requirements by recommending that the evidence used to guide investment of federal funds be based on studies of similar types of students in similar settings. Ideally, the evidence must be both rigorous and relevant to local conditions. In addition, the U.S. Department of Education recommends consideration of the full body of evidence regarding the effectiveness of an activity, in contrast to the ESSA requirement which allows a single study to justify an investment of funds. No explanation is given for why the full body of evidence should be considered rather than a single study and the U.S. Department of Education acknowledges that its guidance is “nonbinding and does not create or impose new legal requirements” (p. 2).

Despite the federal emphasis on use of evidence-based practices, no prior research has established what proportion of federal education funds are invested in evidence-based practices according to ESSA’s definition, the degree to which applying the U.S. Department of Education’s more rigorous definition yields a different answer, or whether this proportion changed after the introduction of ESSA. Answers to these questions could help assess the effectiveness of federal policy in influencing the use of evidence-based practices by local education agencies (LEAs) and inform policymakers how use of evidence should be defined and encouraged.

In this paper, we first briefly address the research on use of evidence in education decision-making and review the relationship between the largest category of federal education funds, Title I, and student achievement. Given inconsistent findings on the association between federal funding amounts and student achievement, but mostly positive findings when the funds are used to support implementation of specific practices, we designed a study to establish the extent to which Title I funding is allocated to practices which may be reasonably expected to improve student achievement. We describe our collaboration with a large, southern, urban district (SUD) to examine what proportion of its Title I funds are spent on activities for which there is evidence of effectiveness; how the answer differs depending on whether only one study is deemed sufficient to support the use of funds or whether multiple studies are considered; and whether the proportion changed after ESSA went into effect in 2017–2018. We did not aim to ascertain whether evidence was used to make these investment decisions but to quantify the proportion of Title I funds spent on activities for which we ourselves could find evidence meeting ESSA’s top three tiers of evidence. We focused on the top three tiers because the fourth tier is vaguely defined: it is unclear what counts as an adequate research-based rationale or ongoing evaluation.

Use of Evidence in Education Decision-Making

Many scholars have documented the relatively limited extent to which research-based evidence is used in education decision-making and the many barriers precluding greater use (Asen et al., 2013; Farley-Ripple et al., 2018; Farrell & Coburn, 2017; Finnigan et al., 2013; Honig & Coburn, 2008; Honig et al., 2017; King & Pechman, 1984; Neal et al., 2018; Nutley et al., 2007; Penuel et al., 2016, 2018; Spillane, 2012; Tseng, 2012; Tseng & Nutley, 2014; Weiss, 1977). However, none of these studies have quantified the proportion of education agency funds that are actually invested in evidence-based activities. Several efforts have been made to assess the extent to which education agencies are working to meet ESSA requirements (Duff & Wohlstetter, 2019; Results for America, 2017, 2018; Stark Rentner et al., 2019; Troppe et al., 2020) and have documented actions taken in this direction. For example, Results for America (2017, 2018) evaluated the 51 state “ESSA plans” approved by the U.S. Department of Education for implementation in the 2017–2018 school year and found that 46 of the plans included at least one promising practice for building and using evidence to improve student outcomes. These practices were concentrated on Title I funding which constitutes the largest portion of ESSA funding, totaling $16.5 billion for fiscal year 2021 (ESEA Network, n.d.; U.S. Department of Education, 2021). But again, the extent to which Title I funds are invested in evidence-based activities, and whether ESSA has changed this, has not yet been quantitatively evaluated.

Title I Funds

Title I of ESEA is designed to help children from families of low socio-economic status (SES) meet state academic standards (Dynarski & Kainz, 2015; Stullich et al., 2007; U.S. Department of Education, 2015). According to the U.S. Department of Education (2018), national spending on Title I increased from $14.5 billion in 2010 to $15.9 billion in 2019, reaching over 13,000 LEAs across all states. ESSER funds have been awarded to state education agencies in the same proportion as Title I funds were distributed in 2019. ESSA repeatedly emphasizes the need to invest Title I funds in evidence-based practices, limiting the use of certain streams to practices supported by the top three tiers of evidence.

Pre-ESSER, Title I funding accounted for 2% of total school funding (U.S. Census Bureau, 2018) and added approximately 5% more funding per student than would otherwise be available (Dynarski & Kainz, 2015). The U.S. Government Accountability Office (2011) demonstrated that school districts spend Title I funds on instructional staff, professional development, after-school and summer programs, technology, supplemental services including interventions and tutoring, and class size reductions. Contreras (2011) categorized Title I investments into eight spending categories including personnel, staff development, reading programs, and math programs, but found no significant relationship between the proportion of funds allocated to each category and student achievement. However, Contreras did not investigate whether the specific practices on which the funds were spent were effective for improving student learning.

Studies investigating the impact of Title I funds on student achievement have, overall, been inconclusive. While some have found positive effects (e.g., Borman & D’Agostino, 1996; Torgesen et al., 2007) others have not (e.g., Deke et al., 2012; Puma et al., 1997; Sousa & Armor, 2016; van der Klaauw, 2008). Dynarski and Kainz (2015) propose two explanations for the mixed evidence: first, Title I funding is not enough to substantially improve the learning experience for students; second, a large proportion of the funding has been spent on ineffective practices. Additionally, Gordon (2004) demonstrated that local contributions to education funding fall in reaction to receipt of Title I funds, yielding no overall increase in revenues.

Studies evaluating the effectiveness of Title I-funded practices at a more granular level indicate that certain programs or activities are more effective than others. For example, Kainz (2019) found that students in classrooms with a large proportion of low-income Black and Latinx students showed greater gains in reading when Title I funds were used for class size reduction, whereas they showed greater gains in math when funds were used for professional development. Cook et al., (2015) found that intensive, individualized academic intervention programs for struggling 9th and 10th graders improved math scores. Even among relatively low-cost schoolwide interventions that are the focus of the majority of Title I investments, there is evidence for significant variation in effects. For example, researchers found substantial variation in the effectiveness of math curricula and textbooks in California (Koedel et al., 2017; Koedel & Polikoff, 2017). These studies suggest that highly customized investments, tailored to the needs of particular student populations, age groups, subjects, and teacher and school contexts, accompanied by rigorous evidence to justify an investment, could enhance the impact of Title I and school funding in general.

Using SUD’s investment of Title I funds as a case study to investigate the extent to which federal education funds are expended on evidence-based practices and how ESSA has affected this, we designed a descriptive study to address the following research questions:

1) What proportion of Title I funds are spent by a large, urban school district on practices supported by a single study that meets ESSA Tier I–III evidence requirements?

2) What proportion of Title I funds are spent by a large, urban school district on practices supported by a wider body of evidence?

3) Did the proportion of Title I funds spent by a large, urban school district on evidence-based practices increase after the introduction of ESSA?

We believe the analytical strategy we demonstrate could be adopted by any LEA as a means to evaluate its own progress toward investing in evidence-based strategies. Such a self-evaluation could serve as a needs assessment to determine the extent to which efforts are warranted to support school and district office administrators in using or producing evidence to inform investment of educational funds. Furthermore, the approach provides a means to test the theory that schools or districts investing in evidence-based practices will produce better student outcomes.

Methods

Setting

SUD is a large, urban school district in the U.S. with an enrollment of over 210,000 students. In comparison to the average demographics of American urban school districts serving over 100,000 students (National Center for Education Statistics [NCES], 2021, n.d.), SUD serves more non-white students (91% vs. 78%), in particular Hispanic and Latinx students (62% vs. 42%). SUD also serves a higher percentage of English language learners (29% vs. 17%) and economically disadvantaged students (80% vs. 67%). The graduation rate at SUD is 79%, four percentage points lower than the average. The district’s Title I funding allocation in 2018 was $599 per student compared with an average of $580 per student in any large, urban district serving over 100,000 students (U.S. Department of Education, 2018). The authors have established an informal, ongoing research-practice partnership (RPP) with SUD. The RPP’s focus on improving the availability and use of evidence in SUD’s decisions about the programs implemented in its schools made it a suitable and convenient choice of site for this study which demands a high level of trust given the nature of the information being shared.

SUD schools serving at least 40% of students from low-SES backgrounds are eligible for Title I schoolwide programs which serve all students as opposed to only targeting students from low-income families (U.S. Department of Education, 2015). At SUD in 2017–2018, this rule applied to 156 elementary, 52 middle, and 46 high schools. Only 20 SUD schools received no Title I funds. SUD provides substantial training and monitoring related to use of Title I funds, including clear and specific guidance on types of allowable expenditures. As a result, we expected use of the funds would be similar across schools, varying most by age of students served. A sample of four of SUD’s Title I schools, selected by the Title I Director as typical of how SUD schools spend Title I funds, participated in our study in addition to the central district office. To include the range of practices in which Title I funds are invested across the continuum of schooling, the school sample consisted of two elementary, one middle, and one high school.

Each year, approximately two-thirds of SUD’s $90 million–$100 million Title I funds are distributed to school campuses which they can invest in certain personnel positions and in non-personnel items such as contracted services (e.g., professional development), classroom supplies, reading materials, and technology. The SUD central office uses the remaining funds to support programs which school campuses can opt into based on their student and community needs. Figure 1 provides an overview of Title I fund distribution between the SUD district office and schools.

Figure 1.

Approximate distributions of annual title 1 funding at SUD.

Overview of Analytic Strategy

Our first step toward associating Title I expenditures with evidence was to obtain line-item financial records documenting all outlays of Title I funds by the SUD central office and the sample of four schools over 3 years. For each line item, we identified one or more associated educational practices. To identify research evidence associated with each practice, we followed Alexander’s (2020) recommendations for conducting systematic reviews, including specifying the corpus of search terms, delimiting the search, establishing inclusion and exclusion criteria, adhering to quality standards, and tying back to critical questions. For each practice, we assigned one or more search terms that we used in each of four reputable research repositories. We first determined whether at least one study returned by the search term constituted ESSA Tier I–III evidence. We also assigned a summary rating to each search term based on all studies meeting our criteria. In a final step, we tied the evidence ratings back to each line item dollar amount to calculate the proportion of funds that were spent on evidence-based practices. Below, we describe each step in greater detail.

Data

SUD provided summary and line-item spreadsheets documenting SUD’s Title I expenditures for school years 2016–2017, 2017–2018, and 2018–2019. Most expenditures were already categorized by program such as teacher coaching, summer school, and staff development. The exception was General Administration, which accounted for $14.9 million total over the 3 years and included district office services such as Accounting, Budgeting, Research and Accountability, and the External Funding departments. Research and Accountability evaluates programs funded by Title I, documenting outcomes based on the program goals and objectives. The other departments manage the state reimbursement process, ensure grant compliance, check for balanced and accurate budgets, and monitor expenditures to ensure they are spent on allowable items.

The largest category of funds was for Title I schoolwide programs, an amount of $48 million to $49 million per year (approximately 50% of the SUD total) which is allocated to individual eligible schools, as described in the Setting section. Because these spreadsheets included no further details on how schoolwide program funds were used by schools, we requested Title I expenditure records over the same 3 years from the sample of four schools. The four school spreadsheets, detailing a total of $3.97 million in Title I expenditures, were organized by items purchased, but no specific educational activity was associated with each item. In addition to the expenditure records, SUD provided documents related to Title I funds, including a several-hundred-page manual detailing how funds could be spent, annual accountability reports, and SUD evaluations of programs implemented with the Title I funds.

Associating Expenditures With Educational Practices

For each Title I program funded using the central office’s Title I allocation, we reviewed SUD’s annual reports and other Title I documentation such as internal program evaluations for program descriptions. Staff from SUD’s Title I office and Research and Accountability department also contributed their knowledge of the programs via ongoing electronic and verbal communications. We specifically aimed to document which types of students were targeted by each program (elementary, middle, or high school; special or general education; English Learners) and what educational practices were involved so that we could label each expenditure with one or more practices.

Some expenditures were for items that were too vague or broad to assign a specific practice. In some of these cases, the SUD Office of External Funding was able to provide more detailed information on the practices supported. For example, 79% of the expenditure on private-non-profit schools was split evenly across four practices: pull-out interventions for reading and math, push-in interventions for reading and math, academic counseling, and summer school; 8% was spent on professional development, and less than 1% was assigned to five named reading or math programs. The remaining 11% supported general administration which could not be associated with a specific educational practice. For general administration and other items for which we could not identify a specific educational practice, including the aforementioned $14.9 million in general administration, we labeled the spending “Title I funding.”

We pursued two approaches for associating the approximately $48 million in expenditures on Title I schoolwide programs with educational practices. First, we assigned the eponymous practice of “Title I schoolwide programs” to the entire amount and treated it as a single lump sum. As an alternative approach, we assumed that all SUD Title I schools invested their individual allotments of funds for schoolwide programs in the same practices as the four sample schools.

We broke out expenditures and their associated practices by the relevant school levels: elementary, middle, and high. This allowed us to later apply evidence ratings specific to the age group(s) targeted because, for some practices, the research evidence may be positive for one school level and mixed, negative, or absent for other levels. For practices that targeted all three school levels such as summer school and student health care programs, we split the expenditures to reflect the proportion of SUD’s total student enrollment at each school level.

For Title I expenditures at the individual schools, we first reviewed each expenditure and associated it with an educational practice based on allowable uses prescribed in SUD’s Title I manual or using information from other internal documents, or from the SUD and vendor websites. Subsequently, we conducted semi-structured interviews with administrative assistants and principals at the two elementary schools in our sample to confirm our assumptions and fill in gaps where it was unclear how the purchased item might have been used. These two schools had the most diverse investments of the four schools in our sample and there was high overlap in items purchased across all four schools so we assumed they were used for similar practices at each school unless informed otherwise. We interviewed the principal at the high school who provided information about items that were unique to the secondary schools such as AVID and a test-prep item.

Assigning Evidence Search Terms to Practices

Each program or practice funded by the central office Title I allocation (approximately $30 million) was labeled with one to seven terms that we could use to search for studies on effectiveness at improving student outcomes. If an expenditure supported a well-recognized practice (e.g., tutoring intervention for reading, professional development, summer school), this practice name alone was used as a search term. For investments in vendor-provided programs (e.g., Imagine Learning), we used the program name as a search term. Some SUD programs were developed in-house and delivered as packages of activities and practices, for example, turnaround efforts or parent engagement initiatives. In these cases, we assigned search terms to reflect any of the educational practices involved. For the practice of “Title I schoolwide programs,” we assigned the search terms “Title I schoolwide programs” and “Title I funding.” In total, we assigned 52 search terms to 35 practices on which the central office Title I allocation was spent (see Supplemental Table 1).

Expenditures in the four school spreadsheets were not organized by educational practice or program but were simply associated with the actual item purchased. For example, $620.98 was spent on batteries at an elementary school in 2016–2017. We associated a total of approximately 1,300 personnel and non-personnel expenditures with one or more of 109 educational practices. The batteries, for example, were associated with a robotics program based on information from our interviewees. Most personnel expenditures and a few non-personnel expenditures supported just one educational practice but others were associated with multiple practices. For instance, colored paper was used for home-school communication, for experiential learning, and as general supplies. Unless otherwise directed by our interviewees, we split the dollar amount for these items evenly among the uses (one third each in the case of colored paper). Each of the 109 educational practices was then labeled with one or more of 129 search terms (see Supplemental Table 2). A single educational practice might be associated with multiple search terms because it involves several activities at once. For example, the Reading Mini-lessons series was initially described as a professional development practice, but it encompasses three more specific educational practices: professional learning communities, professional book study, and mini-lesson delivery. We used multiple search terms for these compound activities to capture all potentially relevant evidence.

Searching for Evidence

After associating each Title I expenditure with one or more educational practices and related search terms, we entered the search terms in each of four reputable databases of educational research to identify relevant studies: What Works Clearinghouse (WWC), Evidence for ESSA, Education Endowment Foundation, and Education Resources Information Center. We limited our search to these sources partly due to time and funding constraints but also because these sources of research studies are freely available to the public and therefore represent what is likely to be accessible for the typical K-12 decision-maker. For each search term, we identified studies that employed randomized controlled trials (RCTs), quasi-experimental designs (QEDs), or correlational designs to investigate the effects of the associated educational practice. We also collected meta-analyses of studies that used correlational or more rigorous research designs. Although ESSA refers to “student outcomes and other relevant outcomes” (p. 388), there is no specification as to what constitutes other relevant outcomes so we only retained studies reporting on academic, behavioral, socio-emotional, or other outcomes for students. We did not consider studies that might serve as ESSA Tier IV evidence because it is not clear what counts as an adequate “rationale based on high quality evidence or positive evaluation” (ESSA, 2015, p. 388).

We reviewed each study to identify the target student population, target outcomes, and findings regarding the effectiveness of the practice at improving the stated target outcomes. To answer our first research question, for each search term, we determined whether any single study based on an RCT, QED, correlational study, or meta-analysis that included studies using correlational or more rigorous approaches, indicated statistically significant improvements in one or more of the target outcomes at the school level (elementary, middle, high) in which the associated educational practice was implemented at SUD. To address our second research question, we assigned one of the seven summary evidence ratings shown in Table 1 to each search term based on all the studies we found meeting our criteria for the relevant school level. In both cases, we only included studies targeting the same general population of students for which SUD used the practice (elementary, middle, or high school; special or general education; English Learners) and that aimed to improve outcomes similar to those targeted by SUD (e.g., reading or math achievement).

Table 1.

Ratings Applied to Evidence on Educational Practices Based on a Wider Body of Evidence.

Evidence rating	Ratings based on all studies meeting our criteria in four research repositories
Positive	All studies based on RCTs, QEDs, correlational analyses, or meta-analyses based on studies using these designs, find only positive effects of the practice (i.e., T performs significantly better than C or correlational models suggest a positive relationship between the practice and student outcomes).
Mostly positive	Most of the studies find positive effects while some show either mixed results or no difference between T and C. Also, no studies find a purely negative impact. If only one study is found, the evidence shows partially positive effects.
Mixed	Effects include some mix of positive, negative, mixed, and/or null results.
Mostly negative^a	The majority of studies find negative impacts while some show no difference.
Negative	All studies find negative effects (i.e., C performs significantly better than T, or correlational models suggest a negative relationship between the practice and student outcomes).
No significant positive or negative evidence	All studies find no significant difference between T and C or no significant correlation between independent variables and outcome variables.
No studies	No studies were found about the practice at the relevant school level.

Note. T = treatment group; C = comparison group.

This rating did not apply to any of the practices we investigated. Additionally, there were no practices for which there was some negative evidence and some showing no difference.

Tying Evidence Back to Expenditure Amounts

After assigning an evidence rating to each search term, we connected the evidence ratings back to the expenditure amounts. For example, a district office expenditure of $1,606,669 for a mentoring program offered at all school levels was labeled with the search term “mentoring for female students.” It was assigned an evidence rating of “no studies” for elementary school, “positive” for middle school, and “positive” for high school based on one correlational study (Stoeger et al., 2019). For practices associated with multiple search terms, we assigned the strongest level of evidence found among the relevant search terms to the full dollar amount. Using the example of the Reading Mini-lessons series, an elementary school expenditure of $1,188 for this line item was labeled with four search terms: “professional development,” “professional learning community,” “professional book study,” and “professional development for mini-lesson delivery.” We found positive evidence for professional learning communities, mixed evidence for professional development at the elementary school level, and no studies on professional book study or professional development for mini-lesson delivery. We therefore assigned the highest level of evidence found—positive—to the $1,188 expenditure. In a sensitivity analysis, we assigned the lowest level of evidence found among the relevant search terms to determine how this would affect the results.

Once each line item for the four schools was assigned an evidence rating, we calculated the percentage of the total school-based expenditures supported by one study meeting ESSA evidence Tiers I–III. We also calculated the percentage supported by positive or mostly positive evidence according to our summary rating which considered all studies reviewed for each practice associated with an expenditure. We did the same for the district office expenditures but, in each case, using each of the two approaches described above for the Title I schoolwide programs category of funds: first assigning the evidence rating attributed to “Title I schoolwide programs” or “Title I funding” to the full amount; and, second, assuming that the ratings assigned to Title I expenditures at the four schools were, on average, applicable proportionally to all SUD Title I schoolwide expenditures.

Results

Proportion of Title I Funds Spent on Practices Supported by a Single Study Meeting ESSA Tier I–III Evidence Requirements

As described in the methods section, we used two approaches to assign evidence to the large portion of SUD’s Title I funds that were distributed to school campuses for schoolwide programs. Under the first approach in which we reviewed studies returned by the search terms “Title I funding” and “Title I schoolwide programs,” we found positive evidence at all school levels. Applying this rating to the full $48 million–$49 million amount allocated to schoolwide programs, we found that 99% of the SUD’s Title I investments were supported by at least one study meeting ESSA Tier I–III evidence requirements (see Figure 2, left panel). Under the second approach, we applied our findings for the sample of four schools to the full $48 million–$49 million amount. Ninety-five percent of the schools’ expenditures were spent on practices supported by at least one study meeting ESSA Tier I–III evidence requirements. Applying this percentage to all SUD Title I schoolwide program expenditures resulted in an overall percentage of 96% of SUD’s total Title I investments being supported by at least one study meeting ESSA Tier I–III evidence requirements (see Figure 2, right panel).

Figure 2.

Percentage of SUD’s title I funds invested in evidence-based practices applying ESSA’s definition of evidence.

Proportion of Title I Funds Spent on Practices Supported by a Wider Body of Evidence

When we applied evidence ratings to each expenditure based on all relevant studies in the four repositories, many of the ratings were downgraded from positive to mixed or below due to inconsistent findings across studies. This includes the ratings for Title I schoolwide programs for middle and high schools. Using these ratings, under the approach in which we applied the evidence ratings for Title I schoolwide programs and Title I funding to the $48 million–$49 million portion of SUD’s Title I funds distributed to schools for schoolwide programs, we found positive or mostly positive evidence to support 49% of SUD’s total Title I expenditures, down from 99% based on ESSA evidence Tiers I–III (see Figure 3, left panel).

Figure 3.

Percentage of SUD’s title I funds invested in evidence-based practices according to a wider body of evidence.

When considering the wider body of evidence, the percentage of Title I investments at the four sample schools supported by positive or mostly positive evidence dropped from 95% to 74%. Applying this percentage to the full amount of SUD’s Title I schoolwide funds resulted in an overall percentage of 58% of SUD’s total 2016–2019 Title I spending being supported by positive or mostly positive evidence (see Figure 3, right panel).

Proportion of Title I Funds Spent on Evidence-based practices After the Introduction of ESSA

The percentage of SUD’s Title I funds invested in practices supported by ESSA Tier I–III evidence dropped one to two percentage points from 2016–2017 to 2017–2018, the year in which LEAs were first expected to adhere to the ESSA evidence requirements. In 2018–2019, the proportion increased one percentage point. This pattern was the same regardless of whether we used a single evidence rating for all Title I schoolwide program funds or generalized the ratings from the four schools. In our analysis based on a wider body of evidence and generalizing evidence ratings from the four schools to all SUD funds allocated to Title I schoolwide programs, the percentage of funds expended on practices supported by positive or mostly positive evidence increased from 56% in 2016–2017 to 60% in 2017–2018 and 59% in 2018–2019. The left panel in Figure 4 shows the percentage of funds invested in practices supported by ESSA Tier I–III evidence when we generalized the school ratings to Title I schoolwide programs. The right panel shows the results for practices supported by a fuller body of evidence.

Figure 4.

Percentage of SUD’s title I funds invested in evidence-based practices over a 3-year period.

Our sensitivity analysis in which we assigned the lowest level of evidence instead of the highest to practices for which varying levels of evidence were found for multiple search terms resulted in lower percentages of Title I funds being invested in practices supported by positive or mostly positive evidence. For our first approach in which we assigned an evidence rating to “schoolwide programs” as a single practice, 6 out of 35 summary ratings changed from positive or mostly positive to mixed. These changes accounted for a total of approximately $84 million ($80 million of which is schoolwide programs at the elementary school level) out of the total $288 million Title I funds, and led to a large drop in the overall percentage of funds invested in practices supported by positive or mostly positive evidence from 49% to 20%. For the second approach in which we assigned evidence ratings to all schoolwide program funds based on the results from the four schools, using the lowest evidence rating instead of the highest caused 5 of the same 6 ratings to change from positive or mostly positive to mixed. These changes accounted for a total of approximately $3.6 million. For the practices at the individual schools, 28 of 109 dropped in evidence rating, resulting in a fall in the percentage of practices supported by positive or mostly positive evidence from 74% to 68%. Applying the lower evidence ratings for practices at the four schools to the overall district allocation of $146 million for schoolwide programs, $8.9 million was associated with less positive evidence. In total, over $12.5 million out of the total $288 million Title I funds was no longer supported by positive or mostly positive evidence when the lowest evidence rating was prioritized. This led to a drop in the overall percentage of funds invested in practices supported by positive or mostly positive evidence from 58% to 54%.

Discussion

Our study demonstrates strategies for documenting the extent to which LEAs invest funds in evidence-based practices. Depending on whether evidence-based practices are defined as those supported by evidence meeting the top three tiers of evidence according to ESSA, or as being supported by a wider body of evidence—more in alignment with the U.S. Department of Education’s (2016) guidance—the results vary dramatically at SUD from over 95% to under 60%. The results also vary, but much less so, depending on whether funds allocated by the central district office to schools are categorized simply as “Title I schoolwide funds,” or whether they are assigned evidence ratings based on how the schools actually use the funds. Overall, our findings suggest that SUD is complying with ESSA’s evidence requirements but that, even after ESSA went into effect, 42–51% of Title I funds are still invested in practices that are not supported by a wider body of evidence. If this finding is typical of other school districts, it may help explain why it is difficult to show consistently that greater investment in education leads to better outcomes (Jackson, 2020).

The study highlights a number of limitations in the evidence base for educational practices. We found no studies at the relevant school level for 26 practices in which the four sample SUD schools invest Title I funds. For the district office Title I expenditures, we found no evidence for seven practices at one or more of the relevant school levels served and no evidence for five practices at any school level. It was also difficult to find studies relevant to SUD’s particular implementation of a practice. For example, a rigorous study demonstrating positive impacts of small-group tutoring on the mathematical problem-solving performance of third-grade children with math difficulties (e.g., Jitendra et al., 2013) does not assure that a parallel strategy would be equally effective for other subjects or grade levels as implemented in SUD.

For some practices, while studies are available for a generalized form of the practice, there is no evidence on more specific forms. For example, hundreds of studies have been conducted on professional development for teachers, but we could find none on on-site consulting for teachers, motivational speakers, or conferences. A similar challenge is that, in some instances, several educational practices are bundled to create a program and evaluated as a package, making it difficult to pinpoint which components are effective. For example, field trips are a component of youth programs that have been evaluated and reviewed by WWC (e.g., Cardenas et al., 1992; Herrera et al., 2013; Snipes et al., 2015). While the youth programs show positive outcomes, it is impossible to determine whether field trips contributed significantly to the outcomes. We could find only one correlational study specific to field trips (Whitesell, 2016) and this was for middle school while the practice was used at an elementary school in SUD.

Limitations and Future Directions

There are a number of limitations in our analysis which could be improved upon in future studies. First, we only investigated use of Title I funds by one school district central office and a sample of four schools. Our results may be typical of other large, urban districts because the guidelines for Title I funding substantially constrain the types of investments that can be made. However, further studies at districts of different sizes and in a variety of geographical locations would better illustrate variations in the extent to which Title I funds are invested in evidence-based practices. We note that such work requires close collaboration with the school district because publicly-available records of LEA expenditures would not permit the depth of analysis we were able to pursue based on line-item expenditure data and access to personnel who could explain how items were used for specific educational activities. Even with the collaboration of the district, for items that were used for multiple purposes, we had to make assumptions about how to split their value across the various uses.

Secondly, we limited our definition of an evidence-based practice to render the study feasible. We only retained studies that reported student outcomes and we did not assess whether a practice was supported by the lowest tier of ESSA evidence. However, including Tier IV evidence could only add 1–4% to the already high percentages of practices that met ESSA evidence Tiers I–III. Our definition of a wider body of evidence was limited to studies in four research repositories which exclude the most recent studies due to the time required for review and inclusion in a repository. To determine whether a study was relevant to the SUD context, we considered a number of factors: grade level of the study population; whether the students were designated as general or special education students, or English Learners; and whether the outcomes measured aligned with SUD’s targeted outcomes. However, we did not eliminate studies on the basis of racial composition or SES of the student population. While the latter variables are clearly important contextual factors (Kao & Thompson, 2003; Roscigno & Ainsworth-Darnell, 1999), the evidence base is currently too limited to allow for this level of matching.

Another avenue for investigating the effects of ESSA on evidence use that we did not pursue is to document the number of local evaluations that have been conducted by state or local education agencies of their ESSA-funded programs and determine what percentage constitutes ESSA evidence Tiers I–III. SUD conducts internal annual evaluations of a number of its Title I and other programs and nine of these appeared among the studies we found in the four research repositories, in addition to two external evaluations of SUD’s programs.

Recommendations

To assist district office administrators and school principals in investing Title I and other funds in evidence-based practices, we recommend that the U.S. Department of Education or state education agencies survey LEAs to identify the most widely implemented activities and direct research funding to investigate the effectiveness of these activities in a variety of contexts. Such “transferability studies” or “modified replications” (Schneider, 2021) would vary one or more factors such as student groups served, grade level, urbanicity of location, or delivery mechanism. For practices found to be consistently ineffective, funding should be directed to identify, develop, or scale up more effective alternatives, or to find ways to improve effectiveness of the existing practice if the underlying theory of change is sound.

The utility of repositories of research could be increased by allowing users to search by practice, by eligibility for investment of specific streams of federal funding, by student characteristics and needs, and by specific outcomes targeted. These repositories should also include information about the practical demands of implementation including resource requirements, costs, and other contextual or political factors that may influence likelihood of attaining positive results.

Researchers who study interventions for which evidence is mixed could aim to explain apparent contradictions or inconsistent findings by investigating differences in implementation, context, and populations served. More attention to subgroup analysis could highlight for which students certain interventions are more or less effective. Identifying the key components of programs and practices that lead to better student outcomes could help school administrators focus efforts on implementing these aspects with high fidelity. Economic evaluations could help assess the feasibility of implementation as well as informing the most efficient use of limited resources. For districts that must provide services to meet a certain student need or a state mandate without evidence on the effectiveness of the services, the district office research department can take advantage of the ability to use Title I funds for ongoing evaluation of programs to develop its own evidence base. Districts without internal capacity could contract with outside researchers for this purpose. Sharing results across districts or collaborating with other districts on studies could help build a more locally-relevant repository of evidence.

A future reauthorization of ESEA should provide a more rigorous standard for what counts as evidence of program effectiveness, directing decision-makers to consider multiple studies in addition to context and feasibility of implementation. Most critically, research is needed to determine whether use of evidence in decision-making does indeed lead to better student outcomes.

Supplemental Material

sj-pdf-1-epx-10.1177_08959048221127989 – Supplemental material for Does ESSA Assure the Use of Evidence-based Educational Practices?

Supplemental material, sj-pdf-1-epx-10.1177_08959048221127989 for Does ESSA Assure the Use of Evidence-based Educational Practices? by Yuan Chang Ginsberg, Fiona M. Hollands, Venita R. Holmes, Robert Shand, Pamela Evans, Ryan Blodgett, Yixin Wang and Laura Head in Educational Policy

Supplemental Material

sj-pdf-2-epx-10.1177_08959048221127989 – Supplemental material for Does ESSA Assure the Use of Evidence-based Educational Practices?

Supplemental material, sj-pdf-2-epx-10.1177_08959048221127989 for Does ESSA Assure the Use of Evidence-based Educational Practices? by Yuan Chang Ginsberg, Fiona M. Hollands, Venita R. Holmes, Robert Shand, Pamela Evans, Ryan Blodgett, Yixin Wang and Laura Head in Educational Policy

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the William T. Grant Foundation under Grant number 187095 awarded to Teachers College, Columbia University.

ORCID iD

Fiona M. Hollands

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Yuan Chang Ginsberg is a researcher in the Department of Education Policy & Social Analysis at Teachers College, Columbia University.

Fiona M. Hollands is a senior researcher in the Department of Education Policy & Social Analysis at Teachers College, Columbia University.

Venita R. Holmes is a manager in the Houston Independent School District, Department of Research and Accountability.

Robert Shand is an assistant professor in the School of Education at American University.

Pamela Evans is a director in the Houston Independent School District, External Funding Department.

Ryan Blodgett is a Grants Administrator in the Houston Independent School District.

Yixin Wang is an education data consultant at UNICEF.

Laura Head is a researcher in the Department of Education Policy & Social Analysis at Teachers College, Columbia University.

References

Alexander

P. A.

(2020). Methodological guidance paper: The art and science of quality systematic reviews. Review of Educational Research, 90(1), 6–23. https://doi.org/10.3102/0034654319854352

Asen

Gurke

Conners

Solomon

Gumm

(2013). Research evidence and school board deliberations: Lessons from three Wisconsin school districts. Educational Policy, 27(1), 33–63. https://doi.org/10.1177/0895904811429291

Borman

G. D.

D’Agostino

J. V.

(1996). Title I and student achievement: A meta-analysis of federal evaluation results. Educational Evaluation and Policy Analysis, 18(4), 309–326. https://doi.org/10.3102/01623737018004309

Cardenas

J. A.

Montecel

M. R.

Supik

J. D.

Harris

R. J.

(1992). The Coca-Cola valued youth program: Dropout prevention strategies for at-risk students. Texas Researcher, 3, 111–130.

Contreras

(2011). The relationship between Title I funding allocations and student achievement. [Doctoral dissertation, California State University]. ProQuest Dissertations Publishing.

Cook

P. J.

Dodge

Farkas

Fryer

R. G.

Guryan

Ludwig

Mayer

Pollack

Steinberg

(2015). Not too late: Improving academic outcomes for disadvantaged youth. (Working Paper WP–15–01). Institute for Policy Research, Northwestern University. https://www.ipr.northwestern.edu/our-work/working-papers/2015/ipr-wp-15-01.html

Deke

Dragoset

Bogen

Gill

(2012). Impacts of Title I supplemental educational services on student achievement (NCEE 2012–40530). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/pubs/20124053/pdf/20124053.pdf

Duff

Wohlstetter

(2019). Negotiating intergovernmental relations under ESSA. Educational Researcher, 48(5), 296–308. https://doi.org/10.3102/0013189X19854365

Dynarski

Kainz

(2015). Why federal spending on disadvantaged students (Title I) doesn’t work. Evidence Speaks Reports, 1(7), 1–5. https://www.brookings.edu/wp-content/uploads/2016/07/Download-the-paper-2.pdf

10.

ESEA Network. (n.d.). Federal education programs summary. Author. Retrieved November 12, 2021, from https://www.eseanetwork.org/about/esea-programs-summary

11.

ESSA. (2015). Every Student Succeeds Act of 2015, Pub. L. No. 114–95 § 114 Stat. 1177 (2015–2016).

12.

Farley-Ripple

May

Karpyn

Tilley

McDonough

(2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235–245. https://doi.org/10.3102/0013189X18761042

13.

Farrell

C. C.

Coburn

C. E.

(2017). Absorptive capacity: A conceptual framework for understanding district central office learning. Journal of Educational Change, 18(2), 135–159. https://doi.org/10.1007/s10833-016-9291-7

14.

Finnigan

Daly

Che

(2013). Systemwide reform in districts under pressure. Journal of Educational Administration, 51(4), 476–497. https://doi.org/10.1108/jea.2013.07451daa.002

15.

Gordon

(2004). Do federal grants boost school spending? Evidence from Title I. Journal of Public Economics, 88(9–10), 1771–1792. https://doi.org/10.1016/j.jpubeco.2003.09.002

16.

Heinrich

C. J.

(2007). Evidence-based policy and performance management: Challenges and prospects in two parallel movements. The American Review of Public Administration, 37(3), 255–277. https://doi.org/10.1177/0275074007301957

17.

Herrera

Grossman

J. B.

Linden

L. L.

(2013). Staying on track: Testing higher achievement’s long-term impact on academic outcomes and high school choice. A Public/Private Ventures project distributed by MDRC. https://doi.org/10.15868/socialsector.15918

18.

Honig

M. I.

Coburn

(2008). Evidence-based decision making in school district central offices: Toward a policy and research agenda. Educational Policy, 22(4), 578–608. https://doi.org/10.1177/0895904807307067

19.

Honig

M. I.

Venkateswaran

McNeil

(2017). Research use as learning: The case of fundamental change in school district central offices. American Educational Research Journal, 54(5), 938–971. https://doi.org/10.3102/0002831217712466

20.

Jackson

C. K.

(2020). Does school spending matter? The new literature on an old question. In Tach

Dunifon

Miller

D. L.

(Eds.), Confronting inequality: How policies and practices shape children's opportunities (pp. 165–186). American Psychological Association. https://doi.org/10.1037/0000187-008

21.

Jitendra

A. K.

Dupuis

D. N.

Rodriguez

M. C.

Zaslofsky

A. F.

Slater

Cozine-Corroy

Church

(2013). A randomized controlled trial of the impact of schema-based instruction on mathematical outcomes for third-grade students with mathematics difficulties. The Elementary School Journal, 114(2), 252–276. https://doi.org/10.1086/673199

22.

Jochim

Gross

(Eds.). (2016). Maximizing opportunities under ESSA. (The SEA of the Future, Volume 6). Building State Capacity & Productivity Center at Edvance Research, Inc. http://www.bscpcenter.org/resources/publications/SEAF_6_11.2016_r2.pdf

23.

Kainz

(2019). Early academic gaps and Title I programming in high poverty, high minority schools. Early Childhood Research Quarterly, 47, 159–168. https://doi.org/10.1016/j.ecresq.2018.08.012

24.

Kane

T. J.

(2017). Making evidence locally. Education Next, 17(2), 52–58.

25.

Kao

Thompson

J. S.

(2003). Racial and ethnic stratification in educational achievement and attainment. Annual Review of Sociology, 29(1), 417–442. https://doi.org/10.1146/annurev.soc.29.010202.100019

26.

King

J. A.

Pechman

E. M.

(1984). Pinning a wave to the shore: Conceptualizing evaluation use in school systems. Educational Evaluation and Policy Analysis, 6(3), 241–251. https://doi.org/10.3102/01623737006003241

27.

Koedel

Polikoff

M. S.

Hardaway

Wrabel

S. L.

(2017). Mathematics curriculum effects on student achievement in California. AERA Open, 3(1), 1–22. https://doi.org/10.1177/2332858417690511

28.

Koedel

Polikoff

(2017). Big bang for just a few bucks: The impact of math textbooks in California. Evidence Speaks Reports, 2(5), 1–7.

29.

Massell

Goertz

M. E.

Barnes

C. A.

(2012). State education agencies’ acquisition and use of research knowledge for school improvement. Peabody Journal of Education, 87(5), 609–626.

30.

National Center for Education Statistics. (2021). Table 215.10. Selected statistics on enrollment, teachers, dropouts, and graduates in public school districts enrolling more than 15,000 students in 2018: Selected years, 1990 through 2018. Author. https://nces.ed.gov/programs/digest/d20/tables/dt20_215.10.asp

31.

National Center for Education Statistics. (n.d.). Common core of data public school districts search tool. Author. https://nces.ed.gov/ccd/districtsearch/

32.

Neal

J. W.

Neal

Z. P.

Lawlor

J. A.

Mills

K. J.

McAlindon

(2018). What makes research useful for public school educators? Administration and Policy in Mental Health and Mental Health Services Research, 45(3), 432–446. https://doi.org/10.1007/s10488-017-0834-x

33.

Nutley

S. M.

Walter

Davies

(2007). Using evidence: How research can inform public services. The Policy Press.

34.

Office of Elementary and Secondary Education. (2021). Elementary and secondary school emergency relief fund. Author. https://oese.ed.gov/offices/education-stabilization-fund/elementary-secondary-school-emergency-relief-fund/

35.

Penuel

W. R.

Briggs

D. C.

Davidson

K. L

Herlihy

Sherer

Hill

H. C.

Farrell

C. C.

Allen

A-R.

(2016). Findings from a national survey of research use among school and district leaders (Technical Report No. 1). National Center for Research in Policy and Practice.

36.

Penuel

W. R.

Farrell

C. C.

Allen

A.-R.

Toyama

Coburn

C. E.

(2018). What research district leaders find useful. Educational Policy, 32(4), 540–568. https://doi.org/10.1177/0895904816673580

37.

Puma

M. J.

Karweit

Price

Ricciuti

Thompson

Vaden-Kiernan

(1997). Prospects: Student outcomes. Final report (ED 413411). ERIC. https://files.eric.ed.gov/fulltext/ED413411.pdf

38.

Results for America. (2017). Leverage Points: Thirteen opportunities for state education agencies to use their ESSA state plans to build and use evidence to improve student outcomes. Author. http://results4america.org/wp-content/uploads/2017/05/R4A_LP_REV_May-2017.pdf

39.

Results for America. (2018). ESSA leverage points: 50-state report on promising practices for using evidence to improve student outcomes. Author. http://results4america.org/wp-content/uploads/2018/01/RFA-ESSA-50-State-Report_final.pdf

40.

Roscigno

V. J.

Ainsworth-Darnell

J. W.

(1999). Race, cultural capital, and educational resources: Persistent inequalities and achievement returns. Sociology of Education, 72(3), 158–178. https://doi.org/10.2307/2673227

41.

Schneider

(2021, November). Gathering speed with SEERNet, prize competitions, and more [Blog post]. https://ies.ed.gov/director/remarks/11-9-2021.asp

42.

Slavin

R. E.

(2017, December 6). Evidence and the ESSA. Huffington Post. https://www.huffpost.com/entry/evidence-and-the-essa_b_8750480?

43.

Snipes

Huang

C.-W.

Jaquet

Finkelstein

(2015). The effects of the Elevate Math summer program on math achievement and algebra readiness (REL 2015–096). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory West. http://ies.ed.gov/ncee/edlabs

44.

Sousa

Armor

(2016). The effectiveness of Title I: Synthesis of national-level evidence from 1966 to 2013. Universal Journal of Educational Research, 4(1), 205–311. https://doi.org/10.13189/ujer.2016.040126

45.

Spillane

J. P.

(2012). Data in practice: Conceptualizing the data-based decision-making phenomena. American Journal of Education, 118(2), 113–141. https://doi.org/10.1086/663283

46.

Stark Rentner

Kober

Braun

. (2019). State leader interviews: How states are responding to ESSA’s evidence requirements for school improvement. Center on Education Policy. https://files.eric.ed.gov/fulltext/ED597203.pdf

47.

Stoeger

Debatin

Heilemann

Ziegler

(2019). Online mentoring for talented girls in STEM: The role of relationship quality and changes in learning environments in explaining mentoring success. New Directions for Child and Adolescent Development, 2019(168), 75–99. https://doi.org/10.1002/cad.20320

48.

Stullich

Eisner

McCrary

(2007). National assessment of Title I, final report: Volume I: Implementation. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/pdf/20084012_rev.pdf

49.

Torgesen

Schirm

Castner

Vartivarian

Mansfield

Myers

Stancavage

Durno

Javorsky

Haan

(2007). National assessment of Title I, final report: Volume II: Closing the reading gap, findings from a randomized trial of four reading interventions for striving readers. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/pdf/20084013.pdf

50.

Troppe

Isenberg

Milanowski

Garrison-Mogren

Rizzo

Gill

B. P.

Ross

Dillon

(2020). The transition to ESSA: State and district approaches to implementing Title I and Title II-A in 2017–18 (NCEE 2021-002). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/pubs/2021002/

51.

Tseng

(2012). The uses of research in policy and practice and commentaries. Social Policy Report, 26(2), 1–24. https://doi.org/10.1002/j.2379-3988.2012.tb00071.x

52.

Tseng

Nutley

(2014). Building the infrastructure to improve the use and usefulness of research in education. In Finnigan

K. S.

Daly

A. J.

(Eds.). Using research evidence in education: From the schoolhouse door to Capitol Hill (Vol. 2, pp. 163–175). Springer Science & Business Media.

53.

U.S. Census Bureau. (2018). 2018 public elementary-secondary education finance data. Author. https://www.census.gov/data/tables/2018/econ/school-finances/secondary-education-finance.html

54.

U.S. Census Bureau. (2019). 2019 public elementary-secondary education finance data. Author. https://www.census.gov/data/tables/2019/econ/school-finances/secondary-education-finance.html

55.

U.S. Census Bureau. (2020). 2020 public elementary-secondary education finance data. Author. https://www.census.gov/data/tables/2020/econ/school-finances/secondary-education-finance.html

56.

U.S. Department of Education. (2015). Improving basic programs operated by local educational agencies (Title I, Part A). Author. https://www2.ed.gov/programs/titleiparta/index.html

57.

U.S. Department of Education. (2016). Non-regulatory guidance: Using evidence to strengthen education investments. Author. https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf

58.

U.S. Department of Education. (2018). Estimated ESEA Title I LEA allocations—FY 2018. Author. https://www2.ed.gov/about/overview/budget/titlei/fy18/index.html

59.

U.S. Department of Education. (2021, January 8). Education appropriations. ED Review. Author. https://www2.ed.gov/news/newsletters/edreview/2021/0108.html

60.

U.S. Government Accountability Office. (2011). Disadvantaged students: School districts have used Title I funds primarily to support instruction. Report to Congressional Committees. https://www.gao.gov/new.items/d11595.pdf

61.

van der Klaauw

. (2008). Breaking the link between poverty and low student achievement: An evaluation of Title I. Journal of Econometrics, 142(2), 731–756. https://doi.org/10.1016/j.jeconom.2007.05.007

62.

Weiss

C. H.

(1977). Research for policy’s sake: The enlightenment function of social research. Political Analysis, 3(4), 531–545.

63.

West

M. R.

(2016). From evidence-based programs to an evidence-based system: Opportunities under the Every Student Succeeds Act. The Brookings Institution. https://www.brookings.edu/research/from-evidence-based-programs-to-an-evidence-based-system-opportunities-under-the-every-student-succeeds-act/

64.

Whitesell

E. R.

(2016). A day at the museum: The impact of field trips on middle school science achievement. Journal of Research in Science Teaching, 53(7), 1036–1054. https://doi.org/10.1002/tea.21322

65.

Yoshizawa

(2021). Fidelity, rigor, and relevance: How SEAs are approaching the ESSA evidence requirements. Educational Policy. https://doi.org/10.1177/08959048211029025

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB

0.13 MB