Reconciling Experts’ Differences in Setting Cut Scores for Pass-Fail Decisions

Abstract

Standard setting is a routine procedure in educational testing. Each examinee in the administration of a test is assigned a score, and the responses of a sample of the examinees are reviewed by a panel of experts. Each expert rates every selected examinee as pass or fail. Based on these ratings a cut score is to be established. A random effects logistic regression method is applied to set the cut score and to estimate the associated standard error. Two examples illustrate that taking account of between-rater differences is essential.