The application of the receiver operating characteristic (ROC) curve for computer-aided diagnostic systems is reviewed. A statistical framework is presented and different methods of evaluating the classification performance of computer-aided diagnostic systems, and, in particular, systems for ultrasonic tissue characterization, are derived.
Most classifiers that are used today are dependent on a separation threshold, which can be chosen freely in many cases. The separation threshold separates the range of output values of the classification system into different target groups, thus conducting the actual classification process. In the first part of this paper, threshold specific performance measures, e.g., sensitivity and specificity; are presented. In the second part, a threshold-independent performance measure, the area under the ROC curve, is reviewed. Only the use of separation threshold-independent performance measures provides classification results that are overall representative for computer-aided diagnostic systems.
The following text was motivated by the lack of a complete and definite discussion of the underlying subject in available textbooks, references and publications. Most manuscripts published so far address the theme of performance evaluation using ROC analysis in a manner too general to be practical for everyday use in the development of computer-aided diagnostic systems. Nowadays, the user of computer-aided diagnostic systems typically handles huge amounts of numerical data, not always distributed normally. Many assumptions made in more or less theoretical works on ROC analysis are no longer valid for real-life data. The paper aims at closing the gap between theoretical works and real-life data. The review provides the interested scientist with information needed to conduct ROC analysis and to integrate algorithms performing ROC analysis into classification systems while understanding the basic principles of classification.
Software for estimation of ROC curves: ‘ROCKIT’ of the University of Chicago: xray.bsd.uchicago.edu/krl.
2.
Software for estimation of ROC curves: ‘MedCalc’ of the University og Gent: www.medcalc.be.
3.
Software for estimation of ROC curves: ‘SPSS’ of SPSS Inc.: www.spss.com.
4.
Software for estimation or ROC curves: ‘SCHROC’ of the University of Bochum: www.scheipers.org.
5.
ScheipersULorenzAPesaventoA. Ultrasonic multifeature tissue characterization for the early detection of prostate cancer, in Proc IEEE Ultrason Symp, pp. 1265–1268 (2001).
6.
ScheipersUErmertHLorenzA. Neuro-fuzzy inference system for ultrasonic multifeature tissue characterization for prostate diagnostics, in Proc IEEE Ultrason Symp, pp. 1347–1350 (2002).
7.
ScheipersUErmertHSommerfeldHJ. Ultrasonic multifeature tissue characterization for prostate diagnostics, Ultrasound Med Biol29, 1137–1149 (2003).
8.
ScheipersUKönigKSommerfeldHJ. Ultrasonic tissue characterization for the classification of prostate tissue, in Proc World Cong Ultrasound, pp. 637–640 (2003).
9.
ScheipersUErmertHSommerfeldHJ. Ultrasonic tissue characterization for prostate diagnostics: spectral parameters vs. texture parameters, Biomed Technik48, 122–129 (2003).
10.
ScheipersUKönigKSommerfeldH-J. Diagnostics of prostate cancer based on ultrasonic multifeature tissue characterization, in Proc IEEE Ultrason Symp, pp. 2153–2156 (2004).
11.
PerreyCScheipersUBojaraW. Computerized segmentation of blood and luminal borders in intravascular ultrasound, in Proc IEEE Ultrason Symp, pp. 1122–1125 (2004).
12.
SiebersSGeierBScheipersU. Staging of deep venous thrombosis using ultrasound, Biomed Technik49, 846–847 (2004).
13.
SiebersSGeierBScheipersU. Classification of venous thrombosis combining ultrasound elastography and tissue characterization, in Proc IEEE Ultrason Symp, pp. 1761–1764 (2004).
14.
SiebersSSchwabeMScheipersU. Evaluation of ultrasonic texture and spectral parameters for coagulated tissue characterization, in Proc IEEE Ultrason Symp, pp. 1804–1807 (2004).
15.
HansenCScheipersUHüttebräukerN. Trends in time series of parameters from ultrasonic images due to metabolic activities of the human liver, in MeinzerH-P: Bildverarbeitung für die Medizin, pp. 445–449 (2005).
16.
ScheipersUSiebersSGottwaldF. Sonohistology for the computerized differentiation of parotidgland tumors, Ultrasound Med Biol31, 1287–1296 (2005).
17.
BassetOSunZMestasJL. Texture Analysis of ultrasonic images of the prostate by means of co-occurrence matrices, Ultrasonic Imaging15, 218–237 (1993).
18.
FeleppaEJPorterCRKetterlingJ. Recent developments in tissue-type imaging (TTI) for planning and monitoring treatment of prostate cancer, Ultrasonic Imaging26, 163–172 (2004).
19.
HartmanPCOosterveldBJThijssenJM. Detection and differentiation of diffuse liver disease by quantitative echography. A retrospective assessment, Invest Radiol28, 1–6 (1993).
20.
JenderkaK VGärtnerTCobet. Tissue characterization by imaging the local frequency dependent relative backscatter coefficient, in Ultrasonic Imaging and Signal Processing, Proc SPIE3982, pp. 270–277 (2000).
21.
LangMErmertHHeuserL. In vivo study of on-line liver tissue classification based on envelope power spectrum analysis, Ultrasonic Imaging16, 77–86 (1994).
22.
LizziFLFeleppaEJAstorM. Statistics of ultrasonic spectral parameters and liver examinations, IEEE Trans Ultrason Ferroelec Freq Control44, 935–942 (1997).
23.
NairAKubanBDObuchowskiN. Assessing spectral algorithms to predict atherosclerotic plaque composition with normalized and raw intravascular ultrasound data, Ultrasound Med Biol27, 1319–1331 (2001).
24.
O'BrienSPSigelBJustinJ. Carotid plaque spaces relate to symptoms and ultrasound scattering, Ultrasound Med Biol30, 611–616 (2004).
25.
OelzeMLO'BrienWDJr.BlueJP. Differentiation and characterization of rat mammary fibroadenomas and 4T1 mouse carcinomas using quantitative ultrasound imaging, IEEE Trans Med Imaging23, 764–771 (2004).
26.
OosterveldBJThijssenJMHartmanPC. Ultrasound attenuation and texture analysis of diffuse liver disease: methods and preliminary results, Phys Med Biol36, 1039–1064 (1991).
27.
SchmitzGErmertHSengeT. Tissue characterization and imaging of the prostate using radio frequency ultrasonic signals, IEEE Trans Ultrason Ferroelec Freq Contr46, 126–138 (1999).
28.
SigelBFeleppaEJSwamiV. Ultrasonic tissue characterization of blood clots, Surg Clin North Am.70, 13–29 (1990).
ThijssenJMOosterveldBJHartmanPC. Correlations between acoustic and rexture parameters from rf and B-mode liver echograms, Ultrasound Med Biol19, 13–20 (1993).
BöhmeJF. Stochastische Signale (B. G. Teubner, Stuttgart/Germany, 1993).
35.
HanleyJAMcNeilBJ. A method for comparing the areas under receiver operating characteristic curves derived from the same cases, Radiology148, 839–843 (1983).
36.
VerlindePCholletGAcheroyM. Multi-modal identity verification using expert fusion, Information Fusion1, 17–33 (2000).
37.
ZhouXHGatsonisCA. A simple method for comparing correlated ROC curves using incomplete data, Statistics Med15, 1687–1693 (1996).
38.
WebbS. The Physics of Medical Imaging (Adam Hilger Publishing, Bristol/United Kingdom, 1988).
AnastasioMAKupinskiMANishikawaRM. Optimization of FROC analysis of rule-based schemes using a multiobjective approach, IEEE Trans Med Imag17, 1089–1093 (1998).
41.
OglesbyJ. What's in a number? Moving beyond the equal error rate, Speech Comm17, 193–208 (1995).
DeLongERDeLongDMClarke-PearsonDL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach, Biometrics44, 837–845 (1988).
44.
MetzCEWangP-LKronmanHB. A new approach for testing the significance of differences between ROC curves measured from correlated data, in Deconinck F: Information Processing in Medical Imaging, pp. 432–445 (The Hague, Nijhoff, (1984).
45.
BamberD. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph, J Math Psych12, 387–415 (1975).
46.
MetzCEKronmanHB. Statistical significance tests for binormal ROC curves, J Math Psych22, 218–243 (1980).
47.
EganJPGreenbergGZSchulmanAI. Operating characteristics, signal detectability, and the method of free response, J Acoust Soc Amer33, 993–1007 (1961).
48.
HanleyJAMcNeilBJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology143, 29–36 (1982).
49.
GreyDR. Some aspects of ROC curve-fitting: normal and logistic models, J Math Psych9, 128–139 (1972).
HastieTTibshiraniRFriedmanJ. TheElements of Statistical Learning: Data Mining, Inference, and Prediction, (Springer Verlag, Series in Statistics, 2001).
52.
WagnerRFBeidenSVCampbellG. Assessment of medical imaging and computer-assist systems: lessons from recent experience, Acad Radiol9, 1264–1277 (2002).
53.
BeidenSVMaloofMWagnerRF. A general model for finite-sample effects in training and testing of competing classifiers, IEEE Trans Patt Anal Mach Intell25, 1561–1569 (2003).
54.
BishopCM. Neural Networks for Pattern Recognition (Oxford University Press, Oxford/United Kingdom, 1995).
55.
HanleyJAMcNeilBJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology143, 29–36 (1982).
56.
ScheipersUSiebersSAshfaqM. Ultrasonic tissue characterization for the differentiation of parotid gland tumors, in Proc IEEE Ultrason Symp (2005, in press).
57.
RocketteHEObuchowskiN. Statistical issues in ROC curve analysis, in SPIE 1234, Medical Imaging IV: PACS System Design and Evaluation, pp. 111–119 (1990).
