Comparing and Contrasting the National Research Council Report On Evaluating Curricular Effectiveness With the What Works Clearinghouse Approach

Abstract

This article summarizes the findings of the National Research Council (NRC) report On Evaluating Curricular Effectiveness and examines the reviews in middle grades mathematics undertaken by the What Works Clearinghouse (WWC). The NRC report reviewed and assessed 147 key evaluations of 13 National Science Foundation–supported K–12 mathematics curricula and six commercially generated curricula. The report found that the evaluations overall were not sufficiently robust to permit confident judgments on individual programs, so it instead focused on how to define effectiveness in conducting future evaluations. Effectiveness was defined as “an integrated judgment based on interpretation of a number of scientifically valid evaluations that combine social values, empirical evidence, and theoretical rationales” (NRC, 2004, p. 4). The report introduced a model for curricular evaluation that includes program theory, implementation, and outcome measures, and reviewed three major methodologies found in the literature: content analysis, comparative analysis, and case study. This article then examines the What Works Clearinghouse’s exclusive emphasis on experimental and quasi-experimental designs from the perspective of an author of the NRC report. The two reports agree in recognizing the need for significant improvement in evaluation quality; however, they differ in four areas: standards for individual studies, need for multiple methods, the way to accumulate information across a set of studies, and how to communicate results with the public. This article concludes with a call for focused efforts to address several shared targets needed to make further progress on how to establish curricular effectiveness.