Measuring Change Conventionally and Adaptively

Abstract

The ordinary difference or gain score is known generally to be unreliable. However, it is not widely known that when a difference score is used to measure change, the difficulty of the pretest can bias the amount of gain observed in groups that differ in initial achievement. This type of bias we call scale distortion. Using item response theory, one may compute gain scores based on differences in estimated Os (the latent trait being measured). The principal focus of the present inquiry was the degree to which scale distortion in the ordinary difference score could be removed by using differences based on estimated Os in either conventional or adaptive testing situations. Conventional tests that use these 0-based gains are shown to remove much of the scale distortion inherent in ordinary difference scores. Adaptive tests using estimated Os also offer substantial improvement in reflecting actual changes in latent proficiency.