Upping the Stakes: Using Gain Scores to Judge Local Program Effectiveness in Chapter 1

Abstract

Since 1978, the national evaluation and reporting system for Title I/Chapter 1 of the Elementary and Secondary Education Act has depended upon a norm-referenced evaluation system to determine the effectiveness of compensatory education programs funded under the act. For a decade, policy directives urged caution in interpreting local results and encouraged use of additional indicators. Since 1988, legislative amendments have required that the national evaluation system serve, without adjustment or interpretation, to identify individual school projects in need of improvement. In this article I argue that the system is technically unsuited for this purpose. Problems with regression and measurement error are illustrated with empirical data. Policy should encourage the use of multiple indicators of program quality at the local level, including applied performance tests and process indicators.