A Comparison of Item Calibration Media in Computerized Adaptive Testing

Abstract

A concern in computerized adaptive testing is whether data for calibrating items can be collected from either a paper-and-pencil (P&P) or a computer ad ministration of the items. Fixed blocks of power test items were administered by computer to one group of examinees and by P&P to a second group. These data were used to obtain computer-based and P&P-based three-parameter logistic model parameters of the items. Then each set of parameters was used to esti mate item response theory pseudo-adaptive scores for a third group of examinees who had received all of the items by computer. The effect of medium of adminis tration of the calibration items was assessed by com parative analyses of the adaptive scores using structural modeling. The results support the use of item parameters calibrated from either P&P or com puter administrations for use in computerized adaptive power tests. The calibration medium did not appear to alter the constructs measured by the adaptive test or the reliability of the adaptive test scores. Index terms: computerized adaptive testing, item calibration, item parameter estimation, item response theory, me dium of administration, trait level estimation.