Evaluating the Quality of Ratings Obtained from Standard-Seting Judges

Abstract

The purpose of this study is to describe a new approach based on Rasch measurement theory for examining the quality of ratings obtained from standard-setting judges. This new approach is based on the adaptation and modification of several indicators of rating errors. To explore the usefulness of the Rasch measurement theory, the ratings of the nine judges who took part in the standard-setting process on a certification examination for nurses were examined. Ratings were obtained for 213 multiple-choice items on this examination. The data suggest that the judges vary significantly in their views of the essential items required for certification. Furthermore, the data suggest that there is statistically significant variability in the judged essentiality of these items.

Get full access to this article

View all access options for this article.

References

Andrich, D. (1982). An index of person separation in latent trait theory, the traditional KR.20 index and the Guttman Scale response pattern. Education Research and Perspectives, 9, 95-104.

Andrich, D. (1988). Rasch models for measurement. Newbury Park, CA: Sage

Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council of Education.

Cizek, G. J. (1996). Setting passing scores. Educational Measurement: Issues and Practice, 15(2), 20-31

Du, Y , & Wright, B. D. (1997). Effects of item characteristics in large-scale direct writing assessment. In M. Wilson , G. Engelhard , & K. Draney (Eds.), Objective measurement: Theory into practice (Vol. 4, pp. 1-24). Norwood, NJ: Ablex.

Ebel, R. L. (1972). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice Hall.

Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31(2), 93-112.

Grosse, M. E. , & Wright, B. D. (1986). Setting, evaluating, and maintaining certification standards with the Rasch model. Evaluation and the Health Professions, 9, 267-285.

Hedges, L. V. , & Okin, I. (1985). Statistical methods for meta-analysis. San Diego: Academic Press.

10.

Jaeger, R. M. (1982). An iterative structured judgment process for establishing standards on competency tests: Theory and application. Educational Evaluation and Policy Analysis, 4, 461-476.

11.

Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago: MESA Press.

12.

Linacre, J. M. , & Wright, B. D. (1994). A user's guide to FACETS: Rasch measurement computer program. Chicago: MESA Press.

13.

Lunz, M. E. , Wright, B. D. , & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3(4), 331-345.

14.

National Certification Corporation (NCC). (1996). Certification programs: Information and registration materials. Chicago: Author.

15.

Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.

16.

Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press.

17.

Saal, F E. , Downey, R. G. , & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413-428.

18.

Stone, G. E. (1996). Objective standard setting. Unpublished doctoral dissertation, the University of Chicago.

19.

Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.

20.

Zieky, M. J. (1994, October). A historical perspective on standard setting. Paper presented at the Joint Conference on Standard Setting for Large-Scale Assessments, Washington, DC.