An investigation of new toxicity test method performance in validation studies: 1. toxicity test methods that have predictive capacity no greater than chance

Abstract

An approach commonly used to measure new toxicity test method (NTM) performance in validation studies is to divide toxicity results into positive and negative classifications, and then identify true positive (TP), true negative (TN), false positive (FP) and false negative (FN) results. After this step is completed, the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are calculated. Although these statistics are widely used and often the only statistics used to assess the performance of toxicity test methods, there is little specific guidance in the validation literature on what values for these statistics indicate adequate performance. The purpose of this study was to begin developing data-based answers to this question by characterizing the CPS obtained from an NTM whose data have a completely random association with a reference test method (RTM). Determining the CPS of this worst-case scenario is useful because it provides a lower baseline from which the performance of an NTM can be judged in future validation studies. It also provides an indication of relationships in the CPS that help identify random or near-random relationships in the data. The results from this study of randomly associated tests show that the values obtained for the statistics vary significantly depending on the cut-offs chosen, that high values can be obtained for individual statistics, and that the different measures cannot be considered independently when evaluating the performance of an NTM. When the association between results of an NTM and RTM is random the sum of the complementary pairs of statistics (sensitivity+ specificity, NPV+PPV) is approximately 1, and the prevalence (i.e., the proportion of toxic chemicals in the population of chemicals) and PPV are equal. Given that combinations of high sensitivity–low specificity or low specificity–high sensitivity (i.e., the sum of the sensitivity and specificity equal to approximately 1) indicate lack of predictive capacity, an NTM having these performance characteristics should be considered no better for predicting toxicity than by chance alone.

Keywords

correlation negative predictive value positive predictive value prediction interval prevalence random association sensitivity specificity toxicity test battery toxicity test performance measurement toxicity test validation validation

References

Balls M , et al. Report and recommendations of the CAAT/ ERGATT workshop on the validation of toxicity test procedures . Alternatives Lab Anim 1990; 18: 313–336 .

Frazier JM. Scientific criteria for validation of in vitro toxicity tests. OECD environment monograph number 36. Paris: Organisation for Economic Co-operation and Development (OECD) , 1990.

Balls M , et al. Practical aspects of the validation of toxicity test procedures . Alternatives Lab Anim 1995; 23: 129–147 .

Anonymous . Validation and regulatory acceptance of toxicological test methods. A report of the ad hoc Interagency Co-ordinating Committee on the Validation of Alternative Methods. Research Triangle Park, NC: National Institutes of Health, NIH Publication 97-3981 , 1997: 1–105.

Anonymous . Final report of the OECD workshop on harmonisation of validation and acceptance criteria for alternative toxicological test methods. Paris: Organisation for Economic Co-operation and Development (OECD) , ENV/MC/CHEM/TG(96)9, 1996: 1–51.

Bruner LH , Carr GJ , Chamberlain M , Curren RD . Validation of alternative methods for toxicity testing . Toxicol In Vitro 1996; 10: 479–501 .

Bruner LH , Carr GJ , Chamberlain M , Curren RD . No prediction model, no validation study . Alternatives Lab Anim 1996; 24: 139–142 .

Holzhutter HG , et al. Recommendations for the application of biostatistical methods during the development and validation of alternative toxicological method. s ECVAM Biostatistics Task Force Report 1. Alternatives Lab Anim 1996; 24: 511–530 .

Weiss, NS. Clinical epidemiology: the study of outcome of disease. New York, NY: Oxford University Press , 1989.

10.

Feinstein AR. On the sensitivity, specificity, and discrimination of diagnostic tests . Clin Pharmacol Ther 1975; 17: 104–116 .

11.

Cooper JA , Saracci R , Cole P . Describing the validity of carcinogen screening tests . Br J Cancer 1979; 39: 87–89 .

12.

Scala RA. Theoretical approaches to validation . Altern methods Toxicol 1987; 5: 1–9 .

13.

Purchase IF. Validation of tests for carcinogenicity . IARC Sci Publ 1980; 27: 343–349 .

14.

Bruner LH , Carr GJ , Harbell JW , Curren RD . An investigation of new toxicity test method performance in validation studies: 2. comparative assessment of three measures of toxicity test performance . Hum Exp Toxicol 2002; 21: 305–315 .

15.

Bruner LH , Carr GJ , Harbell JW , Curren RD . An investigation of new toxicity test method performance in validation studies: 3. sensitivity and specificity are not independent of prevalence or distribution of toxicity . Hum Exp Toxicol 2002; 21: 317–326 .

16.

Balls M , Botham PA , Bruner LH , Spielmann H . The EC/ HO international validation study on alternatives to the Draize eye irritation test . Toxicol In Vitro 1995; 9: 871–929 .

17.

Brantom PG , et al. A summary report of the COLIPA International validation study on alternatives to the Draize rabbit eye irritation test . Toxicol In Vitro 1997; 11: 141–179 .

18.

Spielmann H , et al. The International EU/COLIPA in vitro phototoxicity validation study: results of phase II (blind trial): Part 1: The 3T3 NRU phototoxicity test . Toxicol In Vitro 1998; 12: 305–327 .

19.

Spielmann H , et al. EEC/COLIPA project on in vitro phototoxicity testing: first results obtained with a BALB/C 3T3 cell phototoxicity assay . Toxicology In Vitro. 1994; 8: 793–796 .

20.

Spielmann H , Liebsch M , Kalweit S , et al. Results of a validation study in Germany on two in vitro alternatives to the Draize eye irritation test, the HET-CAM test and the 3T3-NRU cytotoxicity test . Alternatives Lab Anim 2000; 24: 741–858 .

21.

Spielmann H , et al. EEC/COLIPA in vitro photoirritancy program: results of the first stage of validation . Curr Probl Dermatol 1995; 23: 256–264 .

22.

Fentem JH , et al. The ECVAM international validation study on in vitro tests for skin corrosivity: 2. Results and evaluation by the management team . Toxicol In Vitro 1998; 12: 483–524 .

23.

Kanno J , et al. The OECD program to validate the rat uterotrophic bioassay to screen compounds for in vivo estrogenic responses: phase I . Environ Health Perspect 2002; 109: 785–793 .

24.

Bruner LH , et al. Validation of alternative methods for toxicity testing. In Marzulli FN , Maibach HI , eds. Dermatotoxicology. Washington, DC: Taylor and Francis , 1996.

25.

Anonymous . Evaluation of the validation status of toxicological methods: general guidelines for submissions to ICCVAM. Research Triangle Park, NC: National Toxicology Programme Inter Agency Center for the Evaluation of Alternative Toxicological Methods (NICE ATM) , 1999: 1–13.

26.

Botham PA , et al. The skin corrosivity test in vitro: results of an inter-laboratory trial . Toxicol In Vitro 1992; 6: 191–194 .

27.

Liebsch M , et al. Application of the human dermal model skin2 ZK 1350 to phototoxicity and skin corrosivity testing . Toxicol In Vitro 1995; 9: 557–562 .

28.

Haseman JK , et al. Predicting rodent carcinogenicity from four in vitro genetic toxicity assays: an evaluation of 114 chemicals studied by the National Toxicology Program . J Am Stat Assoc 1990; 85: 964–971 .

29.

Aardema MJ , Isfort RJ , Thompson ED , LeBoeuf RA . The low pH Syrian hamster embryo (SHE) cell transformation assay: a revitalized role in carcinogen prediction . Mutat Res 1996; 356: 5–9 .

30.

Olson H , et al. Concordance of the toxicity of pharmaceuticals in humans and in animals . Regul Toxicol Pharmacol 2000; 32: 56–67 .

31.

Forthofer RN , Lee J . Introduction to biostatistics. A guide to design, analysis, and discovery. San Diego: Academic Press , 1995.