Interrater Reliability Reconsidered

Abstract

The purpose of this study was to determine if adequate levels of interrater reliability could be attained on a national standardizedperformance test, using one examiner per observation. Traditionally, performance assessments have relied on two examiners because of the subjectivity involved in grading. However, for nationally administered health-professions performance tests involving process evaluation, cost and logistical issues render two examiners unrealistic. Datafor this study were obtained during a national clinical skills examination for optometry. Surplus examiners were available to serve as secondary examiners in four evaluation stations, yielding 101 paired candidate observations. The results indicated high levels of interrater reliability. At each station, the agreement between examiners in their dichotomous yes-no observations was 90% or greater The agreement among examiner-determined scores based on the differentially weighted items was greater These results indicate that adequate, psychometrically sound examiner judgments and scores can be attained with one examiner per observation.

Get full access to this article

View all access options for this article.

References

Colliver, J. A. , Verhulst, S. J. , Williams, R. G. , & Norcini, J. J. (1989). Reliability of performance on standardized-patient cases: Acomparison of consistency measures based on generalizability theory." Teaching and Learning in Medicine, 1, 31-37.

Gronlund, N. E. (1976). Measurement and evaluation in teaching. New York: Macmillan.

Gross, L. J. (1985). Setting cutoff scores on credentialing examinations: A refinement in the Nedelsky procedure. Evaluation & the Health Professions, 8, 469-493.

Guidelines for non-written examinations. (1990). Washington, DC: National Commission for Certifying Agencies.

Littlefield, J. H. , Anthracite, N. E. , Herbert, R. , & McKendree, J. (1983, April). Adjusting observational ratings to improve interrater consistency. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Miller, E. V. (1982). Performance checklist to evaluate anesthesia skills. In J. S. Lloyd (Ed.), Evaluation of noncognitive skills and clinical performance (pp. 139-144). Chicago: American Board of Medical Specialties.

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(Suppl.), S63-S67.

Nunnally, J. (1967). Psychometric theory. New York: McGraw-Hill.

Raymond, M. R. , & Houston, W. M. (1990, April). Detecting and correcting for rater effects in performance assessment. Paper presented at the annual meeting of the American Educational Research Association, Boston.

10.

Reznick, R. , Smee, S. , Rothman, A. , Chalmers, A. , Swanson, D. , Dufresne, L. , Lacombe, G. , Baumber, J. , Poldre, P. , Levasseur, L. , Cohen, R. , Mendez, J. , Patey, P. , Boudreau, D. , & Berard, M. (1992). An objective structured clinical examination for the licentiate: Report of the pilot project on the Medical Council of Canada. Academic Medicine, 67, 487-494.

11.

Stahl, J. A. , Lunz, M. E. , & Wright, B. D. (1991, April). Equating examinations that include judges (multiple facets). Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.

12.

Swanson, D. B. (1987). A measurement framework for performance-based tests. In I. Hart & R. Harden (Eds.), Further developments in assessing clinical competence. Montreal: Can-Heal.

13.

Van der Vleuten , C.P.M. , & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2, 58-76.