The choice of methods in determining the optimal cut-off value for quantitative diagnostic test evaluation

Abstract

Introduction

The choice of criteria in determining optimal cut-off value is a matter of concern in quantitative diagnostic tests. Several indexes such as Youden’s index, Euclidean index, product of sensitivity and specificity in receiver operator characteristic space and diagnostic odds ratio have been used in clinical practices but their advantages and limitations are not well understood by clinicians. This study aimed to compare these methods in determining optimal cut-off values for quantitative diagnostic test.

Methods

The different configurations of binormal and bilogistic distributions with equal and unequal variances for nondiseased and diseased subjects were examined. The cut-off values with increment of 0.1 in Z-scale were varied. Then, the Youden’s index, Euclidean index, product of sensitivity and specificity, and diagnostic odds ratio were calculated over various cut-off values under distributional assumptions with confirmed parameters.

Results

According to the obtained data from binormal model and equal variances, the optimal cut-off values derived from Youden’s index, Euclidean index, and product method were similar but the diagnostic odds ratio yielded either extremely low or extremely high optimal cut-off value. For bilogistic pair distributions with equal variances, the Youden’s, Euclidean indexes and product method resulted in an identical cut-off value but the diagnostic odds ratio was constant over various cut-points. By both binormal and bilogistic data with more variations in nondiseased population, the Youden’s index produced a higher sensitive optimal cut-off value; but with more variation for diseased distribution, the Euclidean index showed a more sensitive optimal cut-off. For bilogistic data with unequal variance, the log(diagnostic odds ratio) had a straight line relationship over cut-off values with either positive or negative slope.

Conclusion

As a measure of association, diagnostic odds ratio cannot be informative in determining an optimal cut-off value. The advantage of receiver operator characteristic analysis to obtain the optimal cut-off value is to use Youden’s index, Euclidean index, or product index which is recommended. The choice between them depends on variability of test results in diseased and nondiseased subjects and the desired sensitivity.

Keywords

ROC analysis optimal cut-off Youden’s index Euclidean index product index diagnostic odds ratio

Get full access to this article

View all access options for this article.

References

Hanley

. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imag 1989; 29: 307–335.

Greiner

Pfeiffer

Smith

. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med 2000; 30: 23–41.

Hajian-Tilaki

. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med 2013; 4: 627–635.

Kumar

Indrayan

. Receiver operating characteristic (ROC) curve for medical researchers. Ind Pediatr 2011; 48: 277–287.

Kramer

. Clinical epidemiology and biostatistics: a primer for clinical investigation and decision making, Berlin: Springer-Verlag, 1988, pp. 201–219.

Raslich

Markert

Stutes

. Selecting and interpreting diagnostic tests. Biochem Med 2007; 17: 151–161.

Linnet

. Comparison of quantitative diagnostic tests: type I error, power, and sample size. Stat Med 1987; 6: 147–158.

Hajian-Tilaki

Hanley

Joseph

et al.

A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Mak 1997; 17: 94–102.

Hajian-Tilaki

Gholizadehpasha

Bozogzadeh

et al.

Body mass index and waist circumference are predictor biomarkers of breast cancer risk in Iranian women. Med Oncol 2011; 28: 1296–1301.

10.

Hanley

McNeil

. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36.

11.

Swets

. Measuring the accuracy of diagnostic systems. Science 1988; 240: 1285–1293.

12.

Fluss

Faraggi

Reiser

. Estimation of Youden index and its associated cut-off point. Biomed J 2005; 47: 458–472.

13.

Sharma

Jain

. Right choice of a method for determination of cut-off values: a statistical tool for diagnostic test. Asian J Med Sci 2014; 5: 30–34.

14.

Glas

Lijmer

Prins

et al.

The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol 2003; 56: 1129–1135.

15.

Magder

Fix

. Optimal choice of a cut point for a quantitative diagnostic test performed for research purposes. J Clin Epidemiol 2003; 56: 956–962.

16.

Perkins

Schisterman

. The inconsistency of optimal cut-points obtained using two criteria based on receiver operating characteristic curve. Am J Epidemiol 2006; 163: 670–675.

17.

Lopez-Raton M, Rodriguez-Alvarez MX, Cadarso-Suarez C, et al. Optimal cutpoints: An R package for selecting optimal cut-points in diagnostic tests. J Stat Software 2014; 61: 1–35, http//www.jstatsoft.org.

18.

Liu

. Classification accuracy and cut point selection. Stat Med 2012; 31: 2676–286.

19.

McGee

. Simplifying likelihood ratios. J Gen Intern Med 2002; 17: 647–650.

20.

Chio

. Slopes of a receiver operating characteristic curve and likelihood ratio for a diagnostic test. Am J Epidemiol 1998; 148: 1127–1132.

21.

Linnet

Bossuet

PMM

Moons

KGM

et al.

Quantifying the accuracy of a diagnostic test or marker. Clin Chem 2012; 58: 1292–1301.

22.

Parikh

Arun

et al.

Likelihood ratios: clinical application in day-to-day practice. Ind J Ophthalmol 2009; 57: 217–221.

23.

Pepe

Janes

Longton

et al.

Limitation of odd ratio in gauging the performance of a diagnostic, prognostic or screening marker. Am J Epidemiol 2004; 159: 882–890.

24.

Böhning

Holling

Patilea

. A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Stat Meth Med Res 2011; 20: 541–550.

25.

Budczies

Klauschen

Sinn

et al.

Cutoff finder: a comprehensive and straightforward web application enabling rapid Biomarker cutoff optimization. PLoS ONE 2012; 7: e51862–e51862.

26.

Böhning

. Youden’s index and the likelihood ratio positive in diagnostic testing. Meth Inf Med 2015; 54: 382–383.