Two Measures of Change in the Gaps Between the CDFs of Test-Score Distributions

Abstract

Two ways of measuring the gap between two cumulative distribution functions (CDFs) are examined—vertical and horizontal distance. The applications of interest are to the distributions of scores on tests. Vertical distance leads to gap-measures based on differences between “percents above cut-points” for a given score along the score scale. Horizontal distance leads to gap-measures based on differences between the percentiles of the two distributions for a given percent value. Both methods of measuring gaps between CDFs of score distributions are widely used. My focus here is on how the two methods behave when they are used to measure changes in gaps over time. Using a simple model in which the CDFs simply change by small shifts to the right or left, as they often do in real data (the “shift model”), I demonstrate how these two approaches can easily lead to contradictory conclusions about small changes in gaps. The theoretical analysis is illustrated using real data from the National Assessment of Educational Progress. It is recommended that when considering changes in gaps over time, gap-measures be supplemented with fuller data-displays to clarify the directions that changes in such measures can appear to indicate.