Sage Journals: Discover world-class research

Abstract

A long-standing problem of medical diagnosis is why rates of observer disagreement are so high. To study this question a model of test interpretation is defined which accounts for disagreement when observers use identical diagnostic strategies. Test subjects are assigned to one of two disease classes on the basis of two parameters: the test score and A, the diagnostic certainty, a parameter analogous to signal-to-noise ratio. The rate of disagreement and the mean error rate of independent ideal observers are derived as a function of diagnostic strategy. It follows from the model that: 1) Maximization of the ratio of disagreement to error defines an optimal diagnostic strategy with advantages over maximum likelihood and Bayes diagnostic strategies. 2) Disagreement exceeds error if the sensitivity and specificity of in dependent observers exceed one-half for all A. 3) The ratio of disagreement to error can be no greater than two if test interpretation is dichotomous but can approach infinity for tricho tomous interpretation, i.e., if observers are allowed to interpret cases as indeterminate.

Keywords

Key words: Bayesian statistics maximum likelihood signal-to-noise ratio. (Med Decis Making 1989;9:196-206)

Get full access to this article

View all access options for this article.

References

Quinn MF , Lundell CJ , Klotz TA , et al. Reliability of selective pulmonary arteriography in the diagnosis of pulmonary embolism. Am J Roentgenol. 1987 ;149:469-72.

Thomas Gdh , Dixon MF , Smeeton NC , Williams NS Observer variation in histologic grading of rectal carcinoma. J Clin Pathol. 1983;36:385-91.

Grove WM , Andreason NC , McDonald-Scott P. , Keller MB , Shapiro RW Reliability studies of psychiatric diagnosis: theory and practice. Arch Gen Psychiatry. 1981;38:408-13.

Shrout PE , Spitzer RL , Fleiss JL Quantification of agreement in psychiatric diagnosis revisited. Arch Gen Psychiatry. 1987;44:172-7.

Fleiss JL The measurement of interrater agreement. In: Statistical methods for rates and proportions. 2nd ed. New York: Wiley, 1981;211-36.

Cohen J. A coefficient of agreement for nominal scales. Education and Psychological Measurement. 1960;20:37-46.

Kraemer HC Ramifications of a population model for kappa as a coefficient of reliability . Psychometrika. 1979;44:461-72.

Uebersax JS Diversity of decision-making models in the measurement of interrater agreement . Psychol Bull. 1987;101:140-6.

Yerushalmy J. Statistical problems in assessing methods of medical diagnosis with special reference to x-ray techniques. Public Health Rep. 1947;62:1432-49.

10.

Birkelo CC , Chamberlain WE , Phelps PS , Schools PE , Zacks D. , Yerushalmy J. A comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA. 1947;133:359-65.

11.

Feinstein AR A bibliography of publications on observer variability . J. Chronic Dis. 1985;38:619-32.

12.

Lusted LB Decision processes and observer error in medical diagnosis. In: Introduction to medical decision making. Spring-field, IL: Charles C Thomas, 1968;98-140.

13.

Green DM , Swets JA Signal detection theory and psychophysics. Huntington, NY: R. E. Krieger, 1974.

14.

Feinstein AR Observer variability. In: Clinical epidemiology. Philadelphia, W.B. Saunders, 1985 ;632-48.

15.

Swets JA Measuring the accuracy of diagnostic systems. Science. 1988;240:1285-93.

16.

Swets JA , Pickett RM Evaluation of diagnostic systems, Methods from signal detection theory. New York: Academic Press, 1982.

17.

McNeil BJ , Hanley JA Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making. 1984;4:137-50.

18.

Diamond GA ROC steady: a receiver operating characteristic curve that is invariant relative to selection bias. Med Decis Making. 1987;7238-43.

19.

Egan JP Signal detection theory and ROC analysis. New York: Academic Press, 1975.

20.

Metz CE ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720-33.

21.

Ziemer RE , Tranter WH Digital data transmission. In: Principles of communications systems, modulation and noise. 2nd ed. Boston: Houghton Mifflin, 1985;358-439.

22.

Davenport Jr WB , Root WL Statistical detection of signals. In: An introduction to the theory of random signals and noise. New York: McGraw-Hill, 1958;312-63.

23.

Afifi AA , Azen SP Classification of an individual into one of two populations. In: Statistical analysis, a computer oriented approach. 2nd ed. New York: Academic Press , 1979;288-300.

24.

Hanley JA The robustness of the "binormal" assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197-203.

Relation of Observer Agreement to Accuracy According to a Two-receiver Signal Detection Model of Diagnosis

Abstract

Keywords

Get full access to this article

References