Accuracy of Item Performance Predictions Based on the Nedelsky Standard Setting Method

Abstract

The Nedelsky standard setting procedure utilizes an option elimination strategy to estimate the probability that a minimally competent candidate (MCC) will answer a multiple-choice item correctly. The purpose of this study was to investigate the accuracy of predicted item performance from the Nedelsky ratings. The results indicate that test taking behavior of MCCs does not match the underlying Nedelsky assumption that MCCs randomly guess among the options judges believe should be attractive. Further, the accuracy of predicted item performance appears to vary as a function of item difficulty and content domain. However, an analysis of the relationship between judges' rating of distractor difficulty and proportion of examinees selecting item distractors indicated that useful information about examinee item performance is obtainable from Nedelsky-based judgments.

Get full access to this article

View all access options for this article.

References

Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.) Educational Measurement. Washington, DC: American Council on Education.

Bejar, I. I. (1983). Subject matter experts' assessment of item statistics. Applied Psychological Measurement, 7, 303-310.

Bernknopf, S. , Curry, A. , and Bashaw, W. L. (April, 1979). A defensible model for determining a minimal cutoff score for criterion referenced tests. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.

Brennan, R. L. and Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement. 4, 219-240.

Cross, L. H. , Impara, J. C. , Frary, R. B. , and Jaeger, R. M. (1984). A comparison of three methods for establishing minimal standards on the National Teacher Examinations. Journal of Educational Measurement. 21, 113-129.

Gulliksen, H. (1950). Theory of mental tests. New York: Wiley. p. 368-369.

Jaeger, R. M. (1978). A proposal for setting a standard on the North Carolina High School Competency Test. Paper presented at the spring meeting of the North Carolina Association for Research In Education. Chapel Hill, North Carolina.

Lorge, I. and Kruglov, L. K. (1953). The improvement of the estimates of test difficulty. Educational And Psychological Measurement, 13, 34-46.

Melican, G. J. and Thomas, N. (April, 1984). Identification of items that are hard to rate accurately using Angoffs standard setting method. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.

10.

Nedelsky, L. (1954). Absolute grading standards for objective tests. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 14, 3-19.

11.

Thorndike, R. L. (1982). Item and score conversion by pooled judgment. In Holland, P. W. and Rubin, D. B. (Eds.). Test Equating. Academic Press, New York.