Effects of Item Context on Intrajudge Consistency of Expert Judgments via the Nedelsky Standard Setting Method

Abstract

Judgmental methods for estimating passing scores generally are accomplished by having testing specialists evaluate items in an intact test form. Alternatively, judges are asked to rate individual items from an item pool. Cutscores are determined by these ratings when the items are chosen from the pool to form the operational test. This approach to establishing cutscores assumes (a) stable definitions of minimal competency from the time of review to time of assembly into the operational form and (b) invariance of item rating judgments to test form contextual variables. This study was concerned with the impact of overall test length and difficulty on the expert judgments of item performance by using the Nedelsky method. Results suggest that the judges are fairly consistent in their ratings of items regardless of overall test length or difficulty.