Sage Journals: Discover world-class research

Abstract

The purpose of this study was to compare the continuous method of scoring a performance test composed of standardized patients with a derivative method that assigned each examinee a dichotomous score and to explore the use of Ang off 's method at the case level with these two scoring methods. Both scoring systems produced reasonable means and distributions of scores. The continuous scores were somewhat more reproducible than the dichotomous scores although neither was very relay?ducible. Pass rates for both scoring systems were appropriately very high and thus the reproducibility of the pass/fail decisions was also high. Regardless of the scoring system, the application of Angoff's method reported in this article has the advantages of efficiency and enhanced credibility to the experts.

Get full access to this article

View all access options for this article.

References

Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement. Washington, DC: American Council on Education.

Barrows, H. S. (1971). Simulated patients. Springfield, IL: Charles C Thomas.

Brennan, R. L. (1983). Elements of generalizability theory. Iowa City: American College Testing Program.

Brennan, R. L. , & Lockwood, R. E. (1980). A comparison of two cutting score procedures. Applied Psychological Measurement, 4, 105-126.

Colliver, J. A. , Verhulst, S. J. , Williams, R. G. , & Norcini, J. J. (1989). Reliability of performance on standardized patient cases: Acomparison of consistency measures based on generalizability theory. Teaching and Learning in Medicine, 1, 31-37.

Elstein, A. S. , Shulman, L. S. , & Sprafka, S. A. (1978). An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.

Kane, M. , & Wilson, J. (1984). Errors of measurement and standard-setting in mastery testing. Applied Psychological Measurement, 8, 107-115.

Krome, R. L. , Wagner, D. K. , & Munger, B. S. (1983). Standardization of oral examinations. In J. S. Lloyd (Ed.), Oral examinations in medical specialty board certification (pp. 25-53). Chicago: American Board of Medical Specialties.

Stillman, P. L. , Swanson, D. B. , Regan, M. B. , Philbin, M. M. , Nelson, V. , Ebert, T. H. , Ley, B. , Parrino, T. , Shorey, J. , Stillman, A. , Alpert, E. , Caslowitz, J. , Clive, D. , Florek, J. , Hamolsky, M. , Hatem, C. , Kizirian, J. , Kopelman, R. , Levenson, D. J. , Levinson, G. , McCue, J. , Pohl, H. , Schiffman, F. , Schwartz, J. , Thane, M. , & Wolf, M. (1991). Assessment of clinical skills of residents utilizing standardized patients. Annals of Internal Medicine, 114, 393-401.

10.

Stillman, P. L. , Swanson, D. B. , Smee, S. , Stillman, A. E. , Ebert, T. H. , Emmel, V. S. , Caslowitz, J. , Greene, H. L. , Hamolsky, M. , Hatem, C. , Levenson, D. J. , Levin, R. , Levinson, G. , Ley, B. , Morgan, G. J. , Parrino, T. , Robinson, S. , & Wlllms, J. (1986). Assessing clinical skills of residents with standardized patients. Annals of Internal Medicine, 105, 762-771.

11.

Subkoviak, M. J. (1988). A practitioner's guide to computation and interpretation of reliability indices for mastery tests. Journal of Educational Measurement, 25, 47-55.

12.

Sutnick, A. I. , Stillman, P. L. , Norcini, J. J. , Friedman, M. , Regan, M. B. , Williams, R. G. , Kachur, E. K. , Haggerty, M. A. , & Wilson, M. P. (in press). ECFMG clinical competence assessment study. Proceedings of the Fifth Ottawa International Conference on Assessing Clinical Skills.

Scoring and Standard Setting with Standardized Patients

Abstract

Get full access to this article

References