A Preliminary Study of the Impact of Case Specificity on Computer-Based Assessment of Medical Student Clinical Performance

Abstract

The implications of case specificity of two computer-based clinical simulation examination cases (CBX) were examined by a classical measurement approach and by a Bayesian analysis of test characteristics. The CBXs (a surgery and an ob/gyn case) were designed by the National Board of Medical Examiners and administered to 163 University of Michigan Medical School students. The results indicate that the students performed differently on the two cases, the surgery case appearing to be more difficult. The ob/gyn case had greater sensitivity (more accuracy in passing competent students), whereas the surgery case had greater specificity (more accuracy in failing noncompetent students). The differences between the cases and evidence of case specificity raise the issue of an exam's objective and the acceptable type of classification error These results suggest that additional studies are required before widespread use of such exams can be implemented in "high stakes" situations for licensure purposes.

Get full access to this article

View all access options for this article.

References

Algina, J. (1978). On the validity of examinations for making promotion decisions in medical education. Medical Education, 12, 82-87.

Clyman, S. G. , & Orr, N. A. (1990). Status report on the NBME's computer-based testing. Academic Medicine, 65, 235-241.

Elstein, A. S. , Shulman, L. S. , & Sprafka, S. A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.

Elstein, A. S. , Shulman, L. S. , & Sprafka, S. A. (1990). Medical problem solving: A ten-year retrospective. Evaluation and the Health Professions, 13(1), 5-36.

Freund, R. J. , Littell, R. C. , & Spector, P. C. (1986). SAS system for linear models. Cary, NC: SAS Institute.

McGaghie, W. C. (1993). Evaluating competence for professional practice. In L. Curry & J. F. Wergin (Eds.), Educating professionals (pp. 229-261). San Francisco, CA: Jossey-Bass.

Millman, J. (1989). If at first you don't succeed: Setting passing scores when more than one attempt is permitted. Educational Researcher, 18(6), 5-9.

Norman, G. R. , & Feightner, J. W. (1981). A comparison of behavior on simulated patients and patient management problems. Medical Education, 15, 26-32.

Swanson, D. B. , & Stillman, P. L. (1990). Use of standardized patients for teaching and assessing clinical skills. Evaluation and the Health Professions, 13(1), 79-103.

10.

van der Vleuten, C.P.M. , & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2(2), 58-76.

11.

Volle, R. L. (1990). Standardized testingof patient management skill: Acomputer-based method. Clinical Orthopaedics and Related Research, 257, 47-51.

12.

Wolf, F. M. (1991). Toward a theory of clinical evidence: An empirical scientific perspective. Professions Education Researcher Quarterly, 12(4), 3-7.