Assessing Test Accuracy and its Clinical Consequences: A Primer for Receiver Operating Characteristic Curve Analysis

Beck JR Shultz EK The use of relative operating characteristic (ROC) curves in test performance evaluation. Arch Pathol Lab Med 1986; 110: 13–20

Swets JA Measuring the accuracy of diagnostic systems. Science 1988; 240: 1285–93

Solberg HE Discriminant analysis. Crit Rev Clin Lab Sci 1978; 9: 209–42

Krouwer JS Cumulative distribution analysis graphs—an alternative to ROC curves. Clin Chem 1987; 33: 2305–6

Galen RS Statistics. In: Sonnenwirth AC Jarett L , eds. Gradwohl's Clinical Laboratory Methods and Diagnosis, 8th edn. St Louis: CV Mosby Company, 1980: 41–68

Galen RS Peters T Analytical goals and clinical relevance of laboratory procedures. In: Tietz NW , ed. Textbook of Clinical Chemistry. Philadelphia: WB Saunders Company, 1986: 387–409

Bold AM Timms P Evaluation and introduction of new tests. In: Williams DL Marks V , eds. Principles of Clinical Biochemistry: Scientific Foundations, 2nd edn. Oxford: Heinemann Medical Books, 1988: 13–21

McLauchlan DM Gowenlock AH The ‘normal range’—reference values. In: Gowenlock AH McMurray JR McLauchlan DM , eds. Varley's Practical Clinical Biochemistry, 6th edn. London: Heinemann Medical Books, 1988: 273–84

Buncher CR Weiner D Reference values. In: Kaplan LA Pesce AJ , eds. Clinical Chemistry: Theory, Analysis, and Correlation, 2nd edn. St Louis: CV Mosby Company, 1989: 262–9

Howanitz JH Howanitz PJ Principles of laboratory medicine. In: Howanitz JH Howanitz PJ , eds. Laboratory Medicine: Test Selection and Interpretation. New York: Churchill Livingstone, 1991: 1–10

Feinstein AR Clinical Biostatistics. St Louis: The CV Mosby Company, 1977: 1–468

Feinstein AR Clinical Epidemiology: The Architecture of Clinical Research. Philadelphia: WB Saunders Company, 1985: 1–812

Lee TH Goldman L Serum enzyme assays in the diagnosis of acute myocardial infarction: recommendations based on a quantitative analysis [Review]. In: Sox HC , ed. Common Diagnostic Tests: Use and Interpretation, 2nd edn. Philadelphia: American College of Physicians, 1990: 35–66

Gerhardt W Katus H Ravkilde J Hamm C Jørgensen PJ Peheim E S-troponin T in suspected ischemic myocardial injury compared with mass and catalytic concentrations of S-creatine kinase isoenzyme MB. Clin Chem 1991; 37: 1405–11

Sackett DL Haynes RB Guyatt GH Tugwell P Clinical Epidemiology. A Basic Science for Clinical Medicine, 2nd edn. Boston: Little, Brown and Company, 1991: 1–441

Riegelman RK Hirsch RP Studying a Study and Testing a Test: How to Read the Medical Literature, 2nd edn. Boston: Little, Brown and Company, 1989: 1–349

Moossa AR Diagnostic tests and procedures in acute pancreatitis. N Engl J Med 1984; 311: 639–43

Clavien P-A Burgan S Moossa AR Serum enzymes and other laboratory tests in acute pancreatitis. Br J Surg 1989; 76: 1234–43

Clavien P-A Robert J Meyer P Borst F Hauser H Herrmann F Acute pancreatitis and normoamylasemia: not an uncommon combination. Ann Surg 1989; 210: 614–20

Sherlock S Diseases of the Liver and Biliary System, 8th edn. Oxford: Blackwell Scientific Publications, 1989: 1–749

Maharaj B Maharaj RJ Leary WP Cooppan RM Naran AD Pirie D Sampling variability and its influence on the diagnostic yield of percutaneous needle biopsy of the liver. Lancet 1986; i: 523–5

Goldman L Cook EF Brand DA Lee TH Rouan GW Weisberg MC A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med 1988; 318: 797–803

Kassirer JP Kopelman RI Learning Clinical Reasoning. Baltimore: Williams & Wilkins, 1991: 1–332

Yerushalmy J Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Pub Health Rep 1947; 62: 1432–49

Galen RS Gambino SR Beyond Normality: The Predictive Value and Efficiency of Medical Diagnoses. New York: John Wiley & Sons, 1975: 1–237

Sox HC Blatt MA Higgins MC Marton KI Medical Decision Making. Boston: Butterworths, 1988: 1–406

Simel DL Feussner JR Delong ER Matchar DB Intermediate, indeterminate, and uninterpretable diagnostic test results. Med Dec Making 1987; 7: 107–14

Armitage P Berry G Statistical Methods in Medical Research, 2nd edn. Oxford: Blackwell Scientific Publications, 1987: 115–20

Gardner MJ Altman DG Calculating confidence intervals for proportions and their differences. In: Gardner MJ Altman DG , eds. Statistics with Confidence—Confidence Intervals and Statistical Guidelines. London: British Medical Journal, 1989: 28–33

Diem K Seldrup J Geigy Scientific Tables. Volume 2: Introduction to Statistics, Statistical Tables, & Mathematical Formulae—The Binomial Distribution, 8th edn. Basle: Ciba-Geigy, 1982: 73–107

Gardner MJ Gardner SB Winter PD Confidence Interval Analysis Microcomputer Program. London: British Medical Journal, 1989: 1–77

Puleo PR Guadagno PA Roberts R Scheel MV Marian AJ Churchill D Perryman MB Early diagnosis of acute myocardial infarction based on assay for subforms of creatine kinase-MB. Circulation 1990; 82: 759–64

Gerhardt W The Bayes approach: systematic graphic evaluation of diagnostic tests. In: Keller H Trendelenburg C , eds. Data Presentation/Interpretation. New York: Walter de Gruyter, 1989: 161–204

Pellar TG Galbraith LV Leung FY Henderson AR A computer program to determine diagnostic decision thresholds and likelihood ratios illustrated with aspartate aminotransferase activities after a myocardial infarction. Ann Clin Biochem 1989; 26: 533–7

Ransohoff DF Feinstein AR Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978; 299: 926–30

Henderson AR McQueen MJ Patten RL Krishnan S Wood DE Webb S Testing for creatine kinase and creatine kinase-2 in Ontario: reference ranges and assay types. Clin Chem 1992; 38: 1365–70

Diamond GA Clinical epidemiology of sensitivity and specificity. J Clin Epidemiol 1992; 45: 9–13

Hlatky MA Mark DB Harrell FE Lee KL Califf RM Pryor DB Rethinking sensitivity and specificity. Am J Cardiol 1987; 59: 1195–8

Lofgren RP The dynamic nature of sensitivity and specificity [Editorial]. J Gen Intern Med 1987; 2: 452–3

Godyn JJ Tomaszewski JE Zmijewski CM Specificity of a reproducible qualitative urine examination is not a constant test characteristic [Letter]. Am J Clin Pathol 1991; 95: 265–6

Godyn JJ Zmijewski CM Tomaszewski JE Reference value changes and improvement of test performance. Lab Med 1991; 22: 411–14

Lachs MS Nachamkin I Edelstein PH Goldman J Feinstein AR Schwartz JS Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Ann Intern Med 1992; 117: 135–40

Hlatky MA Pryor DB Harrell FE Califf RM Mark DB Rosati RA Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariate analysis. Am J Med 1984; 77: 64–71

Fletcher RH Carcinoembryonic antigen. Ann Intern Med 1986; 104: 66–73

Levy D Labib SB Anderson KM Christiansen JC Kannell WB Castelli WP Determinants of sensitivity and specificity of electrocardiographic criteria for left ventricular hypertrophy. Circulation 1990; 81: 815–20

Detrano R Janosi A Lyons KP Marcondes G Abbassi N Froelicher VF Factors affecting sensitivity and specificity of a diagnostic test: the exercise thallium scintogram [Review]. Am J Med 1988; 84: 699–710

Leung FY Galbraith LV Jablonsky G Henderson AR Re-evaluation of the diagnostic utility of serum total creatine kinase and creatine kinase-2 in myocardial infarction. Clin Chem 1989; 35: 1435–40

Peters T Westgard JO Evaluation of methods. In: Tietz NW , ed. Textbook of Clinical Chemistry. Philadelphia: WB Saunders Company, 1986: 410–23

United Kingdom Trial of Early Detection of Breast Cancer Group. Specificity of screening in United Kingdom trial of early detection of breast cancer. BMJ 1992; 304: 346–9

Anonymous. The value of diagnostic tests [Editorial]. Lancet 1979: i: 809–10

Altman DG Gore SM Gardner MJ Pocock SJ Statistical guidelines for contributors to medical journals. In: Gardner MJ Altman DG , eds. Statistics with Confidence—Confidence Intervals and Statistical Guidelines. London: British Medical Journal, 1989: 83–100

Feinstein AR Clinical biostatistics—XXXI: On the sensitivity, specificity, and discrimination of diagnostic tests. Clin Pharmacol Ther 1975; 17: 104–16

Youden WJ Index for rating diagnostic tests. Cancer 1950; 3: 32–5

Zweig MH Inefficiency of diagnostic efficiency. Clin Chem 1992; 38: 163–4

Vecchio TJ Predictive value of a single diagnostic test in unselected populations. N Engl J Med 1966; 274: 1171–3

Sackett DL Rennie D The science of the art of the clinical examination [Editorial]. JAMA 1992; 267: 2650–2

Sackett DL A primer on the precision and accuracy of the clinical examination. JAMA 1992; 267: 2638–44

Egan JP Signal Detection Theory and ROC Analysis. New York: Academic Press, 1975: 1–277

Swets JA Pickett RM Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. New York: Academic Press, 1982: 1–253

Wu AHB Gornet TG Wu VH Brockie RE Nishikawa A Early diagnosis of acute myocardial infarction by rapid analysis of creatine kinase isoenzyme-3 (CK-MM) sub-types. Clin Chem 1987; 33: 358–62

Pellar TG Leung FY Henderson AR A computer program for rapid generation of receiver operating characteristic curves and likelihood ratios in the evaluation of diagnostic tests. Ann Clin Biochem 1988; 25: 411–16

Zweig MH Campbell G Receiver operating curve (ROC) plots: a fundamental evaluation tool in clinical medicine [Review]. Clin Chem 1993; 39: 561–77

Hanley JA McNeil BJ The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36

Bamber D The area above the ordinal dominance graph and the area below the receiver operating graph. J Math Psych 1975; 12: 387–415

Bhayana V Cohoe S Leung FY Jablonsky G Henderson AR Diagnostic evaluation of the assays for creatine kinase-2 mass and creatine kinase-3 and -2 isoform ratios in the early diagnosis of acute myocardial infarction. Clin Chem 1993; 39: 488–95

Hanley JA McNeil BJ A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148: 839–43

McNeil BJ Hanley JA Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Dec Making 1984; 4: 137–50

Albert A On the use and computation of likelihood ratios in clinical chemistry. Clin Chem 1982; 28: 1113–19

Miettinen OS Estimability and estimation in case-referent studies. Am J Epidemiol 1976; 103: 226–35

Beck JR Likelihood ratios: another enhancement of sensitivity and specificity [Editorial]. Arch Pathol Lab Med 1986; 110: 685–6

van Steirteghem AC Zweig MH Robertson EA Bernard RM Putzeys GA Bieva CJ Comparison of the effectiveness of four clinical chemical assays in classifying patients with chest pain. Clin Chem 1982; 28: 1319–24

Werner M Brooks SH Mohrbacher RJ Wasserman AG Diagnostic performance of enzymes in the discrimination of myocardial infarction. Clin Chem 1982; 28: 1297–302

Bayes T An essay towards solving a problem in the doctrine of chances. Phil Trans Roy Soc 1763; 53: 370–418. (Reprinted with a biographical note by G. A. Barnard In: Biometrika, 1958; 45: 293–315)

Howson C Urbach P Scientific Reasoning. La Salle, Illinois: Open Court Publishing Company, 1989: 1–312

Hall GH The clinical application of Bayes' theorem. Lancet 1967; ii: 555–7

Lindley DV Making Decisions, 2nd edn. London: John Wiley & Sons, 1985: 1–207

Weinstein MC Fineberg HV Clinical Decision Analysis. Philadelphia: WB Saunders Company, 1980: 1–351

Fagan TJ A nomogram for applying likelihood ratios. N Engl J Med 1975; 293: 257