Detection and Diagnosis of Person Misfit From Patterns of Summed Polytomous Item Scores

Abstract

For valid decision making, it is essential to both the person being measured and the person or organization that is having the person measured that the observed scores adequately represent the underlying trait. This study deals with person-fit analysis of polytomous item scores to detect unusual patterns of sum scores on subsets of items. This approach has the advantage that it allows for a diagnostic approach in which specific hypotheses of person misfit can be tested. In a simulation study, the false-positive and detection rates have been investigated under varying test and item characteristics and different types and levels of aberrant response behavior. The performance of the sum-score—based approach is compared with the perforance of the l^P _z person-fit measure. The simulations show that the person-fit analysis based on sum-score patterns is useful and performs best for detecting aberrant response behavior that manifests itself locally in the pattern, such as careless responding to reverse-worded tems. Statistic l^P _z performs better for detecting aberrant response behavior that affects the responses globally, such as a tendency to choose extreme response options. The person-fit measures discussed are illustrated using real data from the Neuroticism—Extraversion—Openness Personality Inventory—Revised.