A Note on the Reliability of Ranked Items

Abstract

Instruments may be designed to use either Likert-type items (ratings) or ranks to obtain priority rankings of items. Models commonly used in generalizability theory (GT) provide a basis for assessing the reliability of item measurements based on ratings. However, these commonly used models do not apply when respondents are required to rank a set of items. Unlike ratings, ranks force subject effects to zero. Furthermore, ranks violate independence assumptions, and the dependence between ranks assigned to different items affects the standard error of item mean differences and, consequently, the reliability of the item comparisons. A GT analysis for ranked items is presented and used to derive standard errors for differences between item means as well as an item measurement reliability coefficient. This analysis uses a linear model that is not typically used in GT analyses but that models those features unique to ranks.