This research focused on the design of a decision-support system to assist blood bankers in identifying alloantibodies in patients' blood. It was hypothesized that critiquing, a technique in which a computer monitors human performance for errors, would be an effective role for such a decision-support system if the error monitoring was unobtrusive and if the critiquing was in response to both intermediate and final conclusions made by the user. A prototype critiquing system monitored medical technologists for (a) errors of commission and errors of omission, b) failure to follow a complete protocol, (c) answers inconsistent with the data collected, and (d) answers inconsistent with prior probability information. Participants using the critiquing system had significantly better performance (completely eliminating misdiagnosis rates for 3 out of 4 test cases) than a comparable control group. Detailed analysis of the behavioral protocols provided insights into how specific design features influenced performance. Practical applications of this research include its use (after refinements) as a tool for routine antibody identification in blood banks.
Get full access to this article
View all access options for this article.
References
1.
Aikins, J., Kunz, J., & Shortliffe, E. (1983). PUFF: An expert system for interpretation of pulmonary function data. Computers and Biomedical Research, 16, 199–208.
2.
Andert, E. (1992). Integrated knowledge-based system design and validation for solving problems in uncertain environments. International Journal of Man-Machine Studies, 36, 357–373.
3.
Bankowitz, R., McNeil, M., Challinor, S., Parker, R., Kapoor, W., & Miller, R. (1989). A computer-assisted medical diagnostic consultation service: Implementation and prospective evaluation of a prototype. Annals of Internal Medicine, 110, 824–832.
4.
Bernard, J. A. (1989). Applications of artificial intelligence to reactor and plant control. Nuclear Engineering and Design, 113, 219–227.
5.
Bernelot Moens, H. J. (1992). Validation of the AI/RHEUM knowledge base with data from consecutive rheumatological outpatients. Methods of Information in Medicine, 31, 175–181.
6.
Berner, E., Brooks, C., Miller, R., Masarie, F., & Jackson, J. (1989). Evaluation issues in the development of expert systems in medicine. Evaluation and the Health Professions, 12(3), 270–281.
7.
Berner, E., Webster, G., Shugerman, A., Jackson, J., Algina, J., Baker, A., Ball, E., Cobbs, C., Dennis, V., Frenkel, E., Hudson, L., Mancall, E., Rackley, C., & Taunton, O. (1994). Performance of four computer-based diagnostic systems. New England Journal of Medicine, 330, 1792–1796.
8.
Fischer, G., Lemke, A., Mastaglio, T., & Morch, A. (1990). Using critics to empower users. In CHI '90 Conference Proceedings: Human Factors in Computing Systems (pp. 337–347). New York: Association for Computing Machinery.
9.
Forsythe, D., & Buchanan, B. (1992). Broadening our approach to evaluating medical information systems. In Proceedings of the 15th Annual Symposium on Computer Applications in Medical Care (pp. 3–7). Baltimore: IEEE Computer Society.
10.
FrancLois, P., Robert, C., Astruc, J., Begue, P., Borderon, J., Floret, D., Lagardere, B., Mallet, E., Pautard, J., & Demongeot, J. (1993). Comparative study of human expertise and an expert system: Application to the diagnosis of child's meningitis. Computers and Biomedical Research, 26, 383–392.
11.
Gregory, D. (1986). Delimiting expert systems. IEEE Transactions on Systems, Man, and Cybernetics, 16, 834–843.
12.
Guerlain, S., Smith, P. J., Gross, S. M., Miller, T. E., Smith, J. W., Svirbely, J. R., Rudmann, S., & Strohm, P. (1994). Critiquing vs. partial automation: How the role of the computer affects human-computer cooperative problem solving. In M. Mouloua & R. Parasuraman(Eds.), Human performance in automated systems: Current research and trends (pp. 73–80). Mahwah, NJ: Erlbaum.
13.
Guerlain, S., Smith, P. J., Obradovich, J. H., Rudmann, S., Strohm, P., Smith, J. W., Svirbely, J. R., & Sachs, L. (1997). Critiquing as a form of decision-support: An empirical study (CSEL Tech. Report 1996-14). Columbus, OH: Ohio State University.
14.
Guerlain, S., Smith, P. J., Obradovich, J., Smith, J. W., Rudmann, S., & Strohm, P. (1995). The Antibody Identification Assistant (AIDA), an example of a cooperative computer support system. In Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics (pp. 1909–1914). New York: International Institute of Electrical and Electronics Engineers.
15.
Harris, S. D., & Owens, J. M. (1986). Some critical factors that limit the effectiveness of machine intelligence technology in military systems applications. Journal of Computer-Based Instruction, 13(2), 30–34.
16.
Hickam, D., Shortliffe, E., Bischoff, M., Scott, A., & Jacobs, C. (1985). The treatment advice of a computer-based cancer chemotherapy protocol advisor. Annals of Internal Medicine, 103(6 pt 1), 928–936.
17.
Josephson, J., & Josephson, S. (Eds.). (1994). Abductive inference: Computation, philosophy, technology.New York: Cambridge University Press.
18.
Langlotz, C. P., & Shortliffe, E. H. (1983). Adapting a consultation system to critique user plans. International Journal of Man-Machine Studies, 19, 479–496.
19.
Layton, C., Smith, P. J., & McCoy, E. (1994). Design of a cooperative problem-solving system for enroute flight planning: An empirical evaluation. Human Factors, 36, 94–119.
20.
Linnarson, R. (1993). Decision support for drug prescription integrated with computer-based patient records in primary care. Medical Informatics, 18(2), 131–142.
21.
Miller, P. (1986). Expert critiquing systems: Practice-based medical consultation by computer.New York: Springer-Verlag.
22.
Miller, R. A. (1984). INTERNIST-/CADUCEUS: Problems facing expert consultant programs. Methods of Information in Medicine, 23, 9–14.
23.
Miller, R. A., & Maserie, F. (1990). The demise of the “Greek oracle” model for medical diagnostic systems. Methods of Information in Medicine, 29, 1–2.
24.
Nelson, S. J., Blois, M. S., Tuttle, M. S., Erlbaum, M., Harrison, P., Kim, H., Winkelmann, B., & Yamashita, D. (1985). Evaluating RECONSIDER, a computer program for diagnostic prompting. Journal of Medical Systems, 9, 379–388.
25.
Norman, D. (1981). Categorization of actions slips. Psychological Review, 88, 1–15.
26.
Norman, D. (1988). The psychology of everyday things.New York: Basic Books.
27.
Parasuraman, R., Molloy, R., & Singh, I. (1993). Performance consequences of automation-induced “complacency”. International Journal of Aviation Psychology, 3(1), 1–23.
28.
Parasuraman, R., Mouloua, M., & Molloy, R. (1994). Monitoring automation failures in human-machine systems. In M. Mouloua & R. Parasuraman(Eds.), Human performance in automated systems: Current research and trends (pp. 45–49). Mahwah, NJ: Erlbaum.
29.
Plugge, L., Verhey, F., & Jolles, J. (1990). A desktop expert system for the differential diagnosis of dementia. International Journal of Technology Assessment in Health Care, 6, 147–156.
30.
Roth, E., Bennett, K., & Woods, D. (1988). Human interaction with an “intelligent” machine. In E. Hollnagel, G. Mancini, & D. D. Woods(Eds.), Cognitive engineering in complex worlds (pp. 23–69). London: Academic.
31.
Sassen, A., BuieUl, E., & Hoegee, J. (1994). A laboratory evaluation of a human operator support system. International Journal of Human-Computer Studies, 40, 895–931.
32.
Shamsolmaali, A., Collinson, P., Gray, T., Carson, E., & Cramp, D. (1989). Implementation and evaluation of a knowledge-based system for the interpretation of laboratory data. In J. Hunter, J. Cookson, & J. Wyatt(Eds.), Proceedings of AIME '89: Second European Conference on Artificial Intelligence in Medicine (pp. 167–176). Berlin: Springer-Verlag.
33.
Shortliffe, E. H. (1976). Computer-based medical consultations: MYCIN.New York: Elsevier.
34.
Shortliffe, E. H. (1990). Clinical decision-support systems. In E. Shortliffe & L. Perreault(Eds.), Medical informatics: Computer applications in health care (pp. 466–500). New York: Addison-Wesley.
35.
Shortliffe, E. H., Scott, A. C., Bischoff, M., Campbell, A. B., Melle, W. van, & Jacobs, C. (1981). ONCOCIN: An expert system for oncology protocol management. In Proceedings of the Seventh International Joint Conference on Artificial Intelligence (pp. 815–822). Vancouver, BC, Canada: University of British Columbia.
36.
Silverman, B. G. (1992, April). Building a better critic. Recent empirical results. IEEE Expert, 18--25.
37.
Smith, P. J., Galdes, D., Fraser, J., Miller, T., Smith, J. W., Svirbely, J. R., Blazina, J., Kennedy, M., Rudmann, S., & Thomas, D. L. (1991). Coping with the complexities of multiple-solution problems: A case study. International Journal of Man-Machine Studies, 35, 429–453.
38.
Smith, P. J., Miller, T., Gross, S., Guerlain, S., Smith, J., Svirbely, J., Rudmann, S., & Strohm, P. (1992). The transfusion medicine tutor: A case study in the design of an intelligent tutoring system. In Proceedings of the 1992 Annual Meeting of the IEEE Society of Systems, Man, and Cybernetics (pp. 515–520). New York: Institute of Electrical and Electronics Engineers.
39.
Sutton, G. C. (1989, October 14). How accurate is computeraided diagnosis? The Lancet, 905--908.
40.
Verdaguer, A., Patak, A., Sancho, J., Sierra, C., & Sanz, F. (1992). Validation of the medical expert system PNEUMON-IA. Computers and Biomedical Research, 25, 511–526.
41.
Weiner, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill.
42.
Wellwood, J., Johannessen, S., & Spiegelhalter, D. J. (1992). How does computer-aided diagnosis improve the management of acute abdominal pain?Annals of the Royal College of Surgeons of England, 74, 40–46.
43.
Wyatt, J., & Spiegelhalter, D. (1992). Field trials of medical decisionaids: Potential problems and solutions. Proceedings of the 14th Annual Symposium on Computer Applications in Medical Care (pp. 3–7). Los Alamitos, CA: AMIA.
