Using a Theorem by Andersen and the Dichotomous Rasch Model to Assess the Presence of Random Guessing in Multiple Choice Items

Abstract

Andersen (1995, 2002) proves a theorem relating variances of parameter estimates from samples and subsamples and shows its use as an adjunct to standard statistical analyses. The authors show an application where the theorem is central to the hypothesis tested, namely, whether random guessing to multiple choice items affects their estimates in the Rasch model. Taking random guessing to be a function of the difficulty of an item relative to the proficiency of a person, the authors describe a method for creating a subsample of responses, which is least likely to be affected by guessing. Then using Andersen’s theorem, the authors assess the difference in difficulty estimates between responses from the whole sample and the subsample for each item. To demonstrate the effectiveness of the procedure, data are simulated according to a class of models in which random guessing is a function of the proficiency of a person relative to the difficulty of an item. The procedure is also applied to an empirical data set from Raven’s Advanced Progressive Matrices, with the results indicating that guessing is present in a substantial number of items. It is noted that one especially important application in which estimating the correct relative difficulty of items is required is where the items will form part of an item bank and where on subsequent occasions the items will be administered interactively. In this case, items too difficult for a person are not administered and therefore unlikely to attract random guessing.