Correction of Item-Test Correlations and Attempts at Improving Reproducibility in Item-Analysis: an Experimental Approach

Abstract

The indices of discrimination determined for items in multiple choice examinations are known to be spuriously inflated (biased), due to their part-whole correlation. The magnitude of this bias for the phi-coefficient was investigated, using computer simulated examinations in which all the students had equal knowledge (null situation). The bias for the Item-Total ϕ was found to be .8/√M(M = number of items), with variance equal to 1/N (N = number of candidates). The Item-Remainder ϕ reduced but did not eliminate the bias. When the ϕ was calculated for items in one arbitrary half of the examination, using the student's ranking as determined by the other half of the examination, the bias was eliminated, but variance remained unchanged. When ϕ-correlations were calculated for each item against each of the other M-1 items, in turn, the average ϕ for each item was unbiased, and the variance considerably reduced. When applied to real examinations, however, none of these modifications of the ϕ, succeeded in improving its reproducibility, when items are re-used on an equivalent student group. This seriously detracts from the usefulness of the ϕ in item analysis.