Issues related to ROC curves are addressed. The original article on the subject by Lee Lusted, describing the "state of the art" 20 years ago, is reviewed. The concepts that Lusted addressed are then expanded, suggesting the current state of the art. New issues that have arisen with regard to ROC curves and their use in medicine are addressed. Finally, potential areas for future investigation are suggested.
Lusted LBSignal detectability and medical decision-making. Science.1971;171:1217-9.
2.
Hanley JAReceiver operating characteristic (ROC) methodology: the state of the art . Crit Rev Diagn Imaging.1989;29:307-35.
3.
McNeil BJ, Keeler E., Adelstein SJPrimer on certain elements of medical decision making. N Engl J Med.1975;293:211-5.
4.
Simpson AJ, Fitter MJWhat is the best index of detectability? Psychol Bull.1973;80:481-8.
5.
Bamber D.The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psych.1975;12:387-415.
6.
Dorfman DD, Alf E. Jr. Maximum likelihood estimation of parameters of signal detection theory—a direct solution. Psychometrika.1968;33:117-24.
7.
Ogilvie JC, Creelman CDMaximum likelihood estimation of ROC curve parameters. J Math Psych.1968;5:377-91.
8.
Dorfman DD, Alf E.Jr. Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals — rating—method data . J Math Psych.1969;6:487-96.
9.
Grey DR, Morgan Bjt.Some aspects of ROC curve-fitting: normal and logistic models. J Math Psych.1972;9:128-39.
10.
Hanley JAThe robustness of the "binormal" assumptions used in fitting ROC curves . Med Decis Making.1988;8:197-203.
11.
Rockette HE , Obuchowski NA, Gur D.Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems. Invest Radiol.1990;25:835-7.
12.
Goddard MJ, Hinberg I.Receiver operator characteristic (ROC) curves and non-normal data: an empirical study. Stat Med.1990;9:325-37.
13.
Hanley JA, McNeil BJThe meaning and use of the area under the receiver operating characteristic (ROC) curve. Radiology.1982;143:29-36.
Centor RMA Visicalc program for estimating the area under a receiver operating characteristic (ROC) curve. Med Decis Making.1985;5:139-48.
16.
Centor RM, Schwartz JSAn evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve. Med Decis Making.1985;5:149-56.
17.
England WLAn exponential model used for optimal threshold selection on ROC curves . Med Decis Making.1988;8:120-31.
18.
Diamond GAROC steady: a receiver operating characteristic curve that is invariant relative to selection bias. Med Decis Making.1987;7:238-43.
19.
Hanley JA, Begg CBResponse to ROC steady. Med Decis Making.1987;7:244-6.
20.
Hanley JA, McNeil BJA method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology.1983 ;148:839-43.
21.
Metz CE, Kronman HBStatistical significance tests for binormal ROC curves. J Math Psych.1980;22:218-43.
22.
Metz CE, Wang P-L., Kronman HBA new approach for testing the significance of differences between ROC curves from correlated data. In: Deconink F, ed. Information processing in medical imaging. The Hague: Nijhoff; 1984:432-45.
23.
Moise A., Clement B., Ducimetiere P., Bourassa MGComparison of receiver operating curves derived from the same population: a bootstrapping approach. Comput Biomed Res.1985;18: 125-31.
24.
DeLong ER, DeLong DM, Clarke-Pearson DLComparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics.1988; 44:837-45.
25.
McClish DKComparing the areas under more than two independent ROC curves. Med Decis Making.1987;7:149-55.
26.
Moise A., Clement B., Raissis M., Nanopoulos P.A test for crossing receiver operating characteristic (ROC) curves . Commun Statist Theory Meth. 1988;17:1985-2003.
27.
McNeil BJ, Hanley JAStatistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making.1984;4:137-50.
28.
McClish DKAnalyzing a portion of the ROC curve. Med Decis Making.1989;9:190-5.
29.
Thompson ML , Zucchini W.On the statistical analysis of ROC curves . Stat Med.1989;8:1277-90.
30.
McClish DKDetermining a range of false-positive rates for which ROC curves differ . Med Decis Making.1990;10:283-7.
31.
Centor RM, Keightley GEReceiver operating characteristic (ROC) curve area analysis using the ROC ANALYZER. SCAMC Proceedings.1989;13:222-6.
32.
Metz CEBasic principles of ROC analysis. Semin Nucl Med.1978;8:283-98.
