Evaluation Issues in the Development of Expert Systems in Medicine

Abstract

Evaluation of medical decision support software (MDSS) -computer programs to assist health professionals with diagnostic and/or therapeutic decisions -has not kept pace with the development of such programs. This article describes the following formative evaluation issues that must be addressed by developers of MDSS to evaluate these programs properly: (1) How can systematic feedback be obtained about an evolving program? (2) How can enough data to evaluate the program be obtained? (3) How much instruction is necessary? (4) What are the most important aspects for users to evaluate? and (5) How can the appropriate use of a developing MDSS be assured? Data from an ongoing evaluation of an existing MDSS, Quick Medical Reference, are used to illustrate the issues and to suggest recommendations for addressing them.

Get full access to this article

View all access options for this article.

References

ADAMS, E. S. and M. W. GRAY (1987) "Strict liability for the malfunction of a medical expert system," pp. 93-99 in Proceedings of the Eleventh Annual Symposium of Computer Applications in Medical Care. Washington, DC: Computer Society of the IEEE.

BARNETT, G. O. , J. J. CIMINO , J. A. HUPP , and E. P. HOFFER (1987a) "DXplain-An evolving diagnostic decision-support system." J. of the Amer. Med. Assn. 258 (1): 67-74.

BARNETT, G. O. , J. J. CIMINO , J. A. HUPP , and E. P. HOFFER (1987b) "DXplain: Experience with knowledge acquisition and program evaluation," pp. 150-154 in Proceedings of the Eleventh Annual Symposium of Computer Applications in Medical Care. Washington, DC: Computer Society of the IEEE.

BRANNIGAN, V. M. and R. E. DAYHOFF (1986) "Medical informatics: the revolution in law, technology, and medicine." J. Legal Medicine 7 (1): 1-53.

BUCHANAN, B. G. and E. H. SHORTLIFFE [eds.] (1984) Rule Based Expert Systems. Reading, MA: Addison-Wesley.

FINCHER, R. E. , A. M. ABDULLA , R. S. MARANDAPALLI, J. L. HOUGHTON , and J. S HENKE (1987) "Teaching fundamental electrocardiography to medical students: computer assisted learning compared with weekly seminars." Research in Medical Education 26: 197-202.

LUNDSGAARDE, H. P. (1987) "Evaluating medical expert systems." Social Sci. Medicine 24 (10): 805-819.

McGUIRE, C. H. and R. W. TYLER (1984) "Comparing the use of research in other professions with research in education," pp. 13-28 in P. L. Hosford (ed.) Using What We Know About Teaching. VA: Association for Supervision and Curriculum Development.

MILLER, R. A. (1984) "INTERNIST I/caduceus: problems facing expert consultant programs." Methods of Information in Medicine 23: 9-14.

10.

MILLER, R. , F. MASARIE , and J. MYERS (1986) "Quick medical reference (QMR) for diagnostic assistance." MD Computing 3 (5): 34-48.

11.

MILLER, R. , M. McNEIL , S. CHALLINOR , F. MASARIE , and J. MYERS (1986) "The INTERNIST-I/Quick Medical Reference project-status report." Western J. of Medicine 145: 816-822.

12.

MILLER, R. , H. POPLE , and J. MYERS (1982) "Internist-I, an experimental computer-based diagnostic consultant for general internal medicine." Ncw England J. of Medicine 307: 468-476.

13.

MILLER, R. , K. SCHAFFNER , and A. MEISEL (1985) "Ethical and legal issues related to the use of computer programs in clinical medicine." Annals of Internal Medicine 102 (4): 529-536.