Estimation of the Reliability of Composite Tests by a Computer Simulation Method

Abstract

A computer program simulated variability in test scores by generating the values of random variables, X_ij, each having possible values 1 and 0 with probability p_ij and 1 — p_ij, where the subscript i refers to items and the subscript j to persons. A matrix of p_ij values was considered to be a representation of a given test. Reliability was interpreted as a measure of the consistency of X_ij values over replications of the procedure and was expressed as a function of expectations and variances of the p_ij values in any matrix. Reliability formulas such as the KR 20 and KR 21 were interpreted as cases to which the general formula reduces when restrictions are placed on the rows and columns of the matrix. Beginning with selected arrays of p_ij values, none of which met conditions for all the formulas, the program found estimates of reliability from the X_ij values which were generated. The procedure gave an indication of the bias and the efficiency of the estimates when there are departures from assumptions made in derivation of the formulas.