Using Results on K Out of N System Reliability to Study and Characterize Tests'

Abstract

For a specific achievement test item and a randomly selected examinee, let p be the probability of correctly determining whether the examinee knows the correct response. Various techniques have been proposed for estimating p. This paper describes and illustrates how results in the engineering literature on "k out of n system reliability" can be used to study and characterize tests based on the estimated values of p. In particular, we can empirically determine the minimum number of distractors required for multiple-choice tests. If we estimate p with an answer-until-correct scoring procedure, we can also determine the minimum number of examinees needed to be reasonably certain about whether y is less than or greater than some predetermined constant, where y = [UNKNOWN] _pi and p_i is the value of p for the ith item on an n-item test. In other words, we can determine whether the expected number of correct decisions on an n-item test is reasonably large.