Restricted ROC curves are useful tools to evaluate the performance of tumour markers

Abstract

In Clinical Epidemiology, receiver operating characteristic (ROC) analysis is a standard approach for the evaluation of the performance of diagnostic tests for binary classification based on a tumour marker distribution. The area under a ROC curve is a popular indicator of test accuracy, but its use has been questioned when the curve is asymmetric. This situation often happens when the marker concentrations overlap in the two groups under study in the range of low specificity, corresponding to a subset of values useless for classification purposes (non-informative values). The partial area under the curve at a high specificity threshold has been proposed as an alternative, but a method to identify an optimal cut-off that separates informative from non-informative values is not yet available. In this study, a new statistical approach is proposed to perform this task. Furthermore, a statistical test associated with the area under a ROC curve corresponding to informative values only (restricted ROC curve) is provided and its properties are explored by extensive simulations. Finally, the proposed method is applied to a real data set containing peripheral blood levels of six tumour markers proposed for the diagnosis of neuroblastoma. A new approach to combine couples of markers for classification purposes is also illustrated.

Keywords

Tumour markers receiver operating characteristic analysis diagnostic tests restricted receiver operating characteristic curve

Get full access to this article

View all access options for this article.

References

Erdreich

. Use of relative operating characteristic analysis in epidemiology. Am J Epidemiol 1981; 114: 649–662.

Pepe

. The statistical evaluation of medical tests for classification and prediction, Oxford, UK: Oxford University Press, 2003.

Krzanowski

Hand

. ROC curves for continuous data, Boca Raton, FL: CRC Press, 2009.

Baker

. Improving the biomarker pipeline to develop and evaluate cancer screening tests. J Natl Cancer Inst 2009; 101: 1–4.

Bamber

. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychology 1975; 12: 387–415.

Mann

Whitney

. On a test whether one of two random variables is stochastically larger than another. Ann Math Stat 1947; 18: 50–60.

Hoeffding

. A class of statistics with asymptotically normal distribution. Ann Math Stat 1948; 19: 293–325.

Hanley

McNeil

. A method of comparing the area under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148: 839–843.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

10.

Rosner

Glynn

. Power and sample size estimation for the Wilcoxon rank sum test with application to comparisons of c statistics from alternative prediction models. Biometrics 2009; 65: 188–197.

11.

Baker

Kramer

. Peirce, Youden, and receiver operating characteristic curves. Am Statistician 2007; 61: 343–346.

12.

Perkins

Schisterman

. The inconsistency of “oprimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol 2006; 163: 670–675.

13.

Pepe

Longton

Anderson

. Selecting differentially expressed genes from microarray experiments. Biometrics 2003; 59: 133–142.

14.

Hanley

. The use of the ‘binormal’ model for parametric ROC analysis of quantitative diagnostic tests. Stat Med 1996; 15: 1575–1585.

15.

Schisterman

Reiser

Faraggi

. ROC analysis for markers with mass at zero. Stat Med 2006; 25: 623–638.

16.

Baker

. Identifying combinations of cancer markers for further study as triggers of early intervention. Biometrics 2000; 56: 1082–1087.

17.

Dodd

Pepe

. Partial AUC estimation and regression. Biometrics 2003; 59: 614–623.

18.

Sprott

. Numerical recipes software: numerical recipes: routine and examples in BASIC, New York, NY, USA: Cambridge University Press, 1998.

19.

McIntosh

Pepe

. Combining several screening tests: optimality of the risk score. Biometrics 2002; 58: 657–664.

20.

Corrias

Haupt

Carlini

. Multiple target molecular monitoring of bone marrow and peripheral blood samples from patients with localized neuroblastoma and healthy donors. Pediatr Blood Cancer 2012; 58(1): 43–49.

21.

Maris

. Recent advances in neuroblastoma. N Engl J Med 2010; 362: 2202–2211.

22.

Beiske

Burchill

Cheung

. Consensus criteria for sensitive detection of minimal neuroblastoma cells in bone marrow, blood and stem cell preparations by immunocytology and RT-QPCR: recommendations by the International Neuroblastoma Risk Group Task Force. Br J Cancer 2009; 100: 1627–1637.

23.

Livak

Schmittgen

. Analysis of relative gene expression data using real-time quantitative PCR and the 2(-Delta Delta C(T)) Method. Methods 2001; 25: 402–408.

24.

Davison

Hinkley

The basic bootstrap. In: Davison

Hinkley

(eds). Bootstrap methods and their application, New York: Cambridge University Press, 2006, pp. 11–69.

25.

Lee

Hsiao

. Alternative summary indices for the receiver operating characteristic curve. Epidemiology 1996; 7: 605–611.

26.

Kagaris D and Yiannoutsos CT. A multi-index ROC-based methodology for high throughput experiments in gene discovery. Int J Data Min Bioinform 2011; in press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.15 MB