Constructing “Proper” ROCs from Ordinal Response Data Using Weighted Power Functions

Abstract

Background. Receiver operating characteristic (ROC) analysis is the standard method for describing the accuracy of diagnostic systems where the decision task involves distinguishing between 2 mutually exclusive possibilities. The popular binormal curve-fitting model usually produces ROCs that are improper in that they do not have the ever-decreasing slope required by signal detection theory. Not infrequently, binormal ROCs have visible hooks that falsely imply worse-than-chance diagnostic differentiation where the curve lies below the no-information diagonal. In this article, we present and evaluate a 2-parameter, weighted power function (WPF) model that always results in a proper ROC curve with a positive, monotonically decreasing slope. Methods. We used a computer simulation study to compare results from binormal and WPF models. Results. The WPF model produces ROC curves that are less biased and closer to the true values than are curves obtained using the binormal model. The better performance of the WPF model follows from its design constraint as a necessarily proper ROC. Conclusions. The WPF model fits a broader variety of data sets than previously published power function models while maintaining straightforward relationships among the original decision variable, specific operating points, ROC curve contours, and model parameters. Compared with other proper ROC models, the WPF model is distinctive in its simplicity, and it avoids the flaws of the conventional binormal ROC model.

Keywords

receiver operating characteristic ROC analysis proper ROC weighted power function

Get full access to this article

View all access options for this article.

References

Pepe

. The Statistical Evaluation of Medical Tests for Classification and Prediction. New York: Oxford University Press; 2003.

Zhou

X-H

Obuchowski

McClish

. Statistical Methods in Diagnostic Medicine. New York: Wiley-Interscience; 2002.

Wagner

Metz

Campbell

. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol. 2007;14:723–48.

Pesce

Metz

Berbaum

. On the convexity of ROC curves estimated from radiological test results. Acad Radiol. 2010;17:960–8.

Mossman

. Resampling techniques in the analysis of non-binormal ROCs. Med Decis Making. 1995;15:358–66.

Swets

. Indices of discrimination or diagnostic accuracy: their ROCs and implied models. Psychol Bull. 1986;99:100–17.

Swets

. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 1986;99:181–98.

Swets

Pickett

Whitehead

. Assessment of diagnostic technologies. Science. 1979;205:753–9.

Hanley

. The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197–203.

10.

Dorfman

Alf

. Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—rating method data. J Math Psychol. 1969;6:487–96.

11.

Somoza

. Eccentric diagnostic tests: redefining sensitivity and specificity. Med Decis Making. 1996;16:15–23.

12.

Hillis

Berbaum

. Using the mean-to-sigma ratio as a measure of the improperness of binormal ROC curves. Acad Radiol. 2011;18:143–54.

13.

Green

Swets

. Signal Detection Theory and Psychophysics. New York: Wiley; 1966.

14.

Egan

. Signal Detection Theory and ROC Analysis. New York: Academic Press; 1975.

15.

Pan

Metz

: The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad Radiol. 1997;4:380–9.

16.

Metz

. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–45.

17.

Dorfman

Berbaum

Metz

. Proper receiver operating characteristic analysis: the bigamma model. Acad Radiol. 1997;4:138–49.

18.

Carroll

Feinberg

Greden

. A specific laboratory test for the diagnosis of melancholia: standardization, validation, and clinical utility. Arch Gen Psychiatry. 1981;38:15–22.

19.

Mossman

Somoza

: Maximizing diagnostic information from the dexamethasone suppression test: an approach to criterion selection using receiver operating characteristic analysis. Arch Gen Psychiatry. 1989;46:653–60.

20.

Metz

Herman

Roe

. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Med Decis Making. 1998;18:110-21. ROCKIT is available from: URL: http://www-radiology.uchicago.edu/krl/KRL_ROC/software_index6.htm.

21.

Birdsall

. The theory of signal detectability: ROC curves and their character. Doctoral thesis. Ann Arbor, MI: University of Michigan; 1966.

22.

Metz

Pan

. “Proper” binormal ROC curves: theory and maximum likelihood estimation. J Math Psychol. 1999;43:1–33.

23.

Pesce

Metz

. Reliable and computationally efficient maximum likelihood estimation of “proper” binormal ROC curves. Acad Radiol. 2007;14:814–29.

24.

Gönen

Heller

. Lehmann family of ROC curves. Med Decis Making. 2010;30:509–17.

25.

Dorfman

Berbaum

. A contaminated binormal model for ROC data, part II: a formal model. Acad Radiol. 2000;7:427–37.

26.

Berkson

. Minimum chi-square, not maximum likelihood! Ann Stat. 1980;8:457–87.

27.

Keating

Mason

Sen

. Pitman’s Measure of Closeness: A Comparison of Statistical Estimators. Philadelphia, PA: Society for Industrial and Applied Mechanics; 1993.

28.

Lunn

Spiegelhalter

Thomas

Best

. The BUGS project: evolution, critique and future directions. Stat Med. 2009;28:3049–67.

29.

Dorfman

Alf

. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psychol. 1969;6:487–96.

30.

Schönemann

Tucker

. A maximum likelihood solution for the method of successive intervals allowing for unequal stimulus dispersions. Psychometrika. 1967;32:403–17.

31.

Egan

Greenberg

Schulman

. Operating characteristics, signal detectability, and the method of free response. J Acoust Soc Am. 1961;33:993–1007.

32.

Barzman

Brackenbury

Sonnier

. Brief Rating of Aggression by Children and Adolescents (BRACHA): development of a tool to assess risk of inpatients’ aggressive behavior. J Am Acad Psychiatry Law. 2011;39:170–9.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.01 MB