An Investigation of Context Effects for Item Randomization within Testlets

Abstract

This study investigated the statistical and practical significance of context effects for items randomized within testlets for administration during a series of computerized non-adaptive tests. One hundred and twenty-five items from four primary school reading tests were studied. Logistic regression analyses identified from one to four items for each test as having statistically significant differences in item difficulty when administered in different positions in the testlet. Subsequent evaluation indicated that the effect sizes for most of these items were in the small category, and no consistent characteristic was identified to support a causative factor for these items. In addition, the few items flagged at each grade and their small effect sizes would argue against any effect on students' total scores. These results not only extended the study of context effects to primary school grades and computerized tests, but also supported testlets as useful to balance psychometric and practical testing needs. Further research into context effects due to item randomization within testlets should increase the number of items per testlet and the content areas studied.