Anomalous Responses on Confidence-Scored Tests

Abstract

Confidence testing allows subjects to respond to each option of a multiple-choice question with their perceived likelihood that it is the correct one. Scoring procedures subsequently provide measures not only of knowledge, but of the individual's "confidence" or realism in the assessment of whether the response is correct. There is an underlying assumption, however, of a significant correlation between assigned probabilities and actual likelihood for choosing a correct response, for we would naturally assume that subjects would, in general, be more certain when correct and less certain when wrong. Significant negative correlation or correlation not different from zero suggests bizarre responses in which the subject becomes more certain of the answer as mastery declines. Such anomalous responses were isolated from 530 tests of clinical knowledge administered to highly motivated medical students. These were rare (less than 3%0). Thus a crucial assumption underlying the theory of confidence testing is validated and a useful method of identifying outliers is devised.