A Neural Network-Based Differential Diagnosis Assessment Instrument

Abstract

Medical educators have been unable to produce convincing evidence of the construct validity of written or simulation-based assessments of differential diagnosis (DDX) competencies. In 1987, a team of investigators at our institution introduced preliminary reports regarding the psychometric properties of an artificial intelligence-derived DDX assessment instrument. These investigations produced evidence of the construct validity (experts' DDX performance > novices') of the measures derived from this instrument, a linear, fuzzy set-like expert system.

In this investigation, the authors used a non-linear, “Back Propagation” neural network as a DDX assessment instrument. An Acute Chest Pain knowledge base was acquired from each of twenty-four board certified emergency medicine specialists and seventy-four junior and senior medical students. The neural network used these knowledge bases to simulate and assess each subject's individual DDX performance against twenty Acute Chest Pain/Myocardial Infarction test cases.

Student-t test revealed that the DDX performance of experts was significantly superior to novices (p < .001). This finding provides converging evidence of the validity of DDX performance measures produced by both linear and non-linear, artificial intelligence-derived assessment instruments. These instruments may prove to be a useful and powerful new assessment methodology.

Get full access to this article

View all access options for this article.

References

Hart

and Harden

(eds.), Further Developments in Assessing Clinical Competence, Can-Heal, Montreal, 1987.

Stillman

P. L.

, Results of a Survey on the Use of Standardized Patients to Teach and Evaluate Clinical Skills, Academic Medicine, 65, pp. 288–292, 1990.

van der Vleuten

C. P. M.

and Swanson

D. B.

, Assessment of Clinical Skills with Standardized Patients: State of the Art, Teaching, Learning Medicine, 2, pp. 58–76, 1990.

Einstein

A. S.

Shulman

L. S.

, and Sprafka

S. A.

, Medical Problem Solving: An Analysis of Clinical Reasoning, Harvard University Press, Cambridge, 1978.

Norcini

J. J.

Swanson

D. B.

Grosso

L. J.

, and Webster

G. D.

, A Comparison of Several Methods for Scoring Patient Management Problems, Proceedings of the 22nd Conference on Research in Medical Education, Washington, D.C., 1983.

Woolliscroft

J. O.

Swanson

D. B.

, and Case

S. M.

, Matching versus Short-Answer Response Formats with Pattern Recognition Items, presented at the Fifth Ottawa International Conference on Assessment of Clinical Competence, 1992.

Swanson

D. B.

and Norcini

J. J.

, Factors Influencing Reproducibility of Tests Using Standardized Patients, Teaching, Learning Medicine, 4, pp. 158–166, 1989.

Norman

G. R.

, Knowledge and Clinical Problem-solving, Medical Education, 19, pp. 344–356, 1985.

Crocker

and Algina

, Introduction to Classical and Modern Test Theory, Holt Rinehart & Winston, QA Location, 1986.

10.

Parsaye

and Chignell

, Expert Systems for Experts, John Wiley & Sons, Inc., New York, 1988.

11.

Papa

F. J.

and Meyer

, An Expert Program Shell Designed for Extracting “Disease Prototypes” and Their Use as Models for Exploring the “Strong Problem Solving Methods” Employed in Clinical Reasoning, in Further Developments in Assessing Clinical Competence, Hunt

I. R.

(ed.), Heal Publications, Quebec, Canada, pp. 354–364, 1987.

12.

Papa

F. J.

Shores

J. H.

, and Meyer

, The Use of a Pattern Recognition-based Prototype-Driven Research Tool to Study the Cognitive Constructs in Medical Decision Making, in Current Developments in Assessing Clinical Competence, Hart

I. R.

Harden

R. M.

, and Des Marchais

, (eds.), Can-Heal Publications, Ottawa, pp. 210–213, 1992.

13.

Stone

R. C.

Papa

F. J.

, and McIntosh

Aldrich D. G.

, The Logistical Advantage of a Neural Network-Based Differential Diagnostic Assessment Procedure, in Approaches to the Assessment of Clinical Competence (Part 2), Harden

R. M.

Hart

I. R.

, and Mulholland

(eds.), pp. 802–804, 1992.

14.

Levine

D. S.

, Introduction to Neural and Cognitive Modeling, Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1991.

15.

Churchland

P. S.

and Senjowski

T. J.

, The Computational Brain, The MIT Press, Cambridge Massachusetts, 1992.

16.

Papa

F. J.

and Meyer

S. A.

, Computer-assisted Learning Tool Designed to Improve Problem Solving Skills, Annals of Emergency Medicine, 18, pp. 273–296, 1989.