An area that requires further research is how best to measure test method performance in validation studies and how to set criteria that should be used to judge the adequacy of this performance. The studies reported here were designed to begin an investigation of these questions. Computer simulations were used to generate data sets similar to those that might be obtained from a large validation study. These data were then analysed using three procedures including determination of the 95% prediction interval (PI), calculation of Pearson's correlation coefficient and calculation of the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The results of this work suggest that of the three approaches examined, quantitative measurements with calculation of the 95% PI provide the most information to allow discrimination between the performance of several different NTMs. The results also suggest that dividing data sets into positive and negative toxicity classifications followed by the calculation of CPS leads to considerable information loss. This loss of information may be so significant that it is not possible in certain circumstances to distinguish between NTMs that are adequate and those that are not.
Anonymous . Validation and regulatory acceptance of toxicological test methods, A report of the ad hoc Interagency Co-ordinating Committee on the Validation of Alternative Methods. Research Triangle Park, NC: NIH Publication 97–3981 , 1997: 1–105.
2.
Anonymous . Final report of the OECD workshop on harmonisation of validation and acceptance criteria for alternative toxicological test methods. Paris: Organisation for Economic Co-operation and Development (OECD) , ENV/MC/CHEM/TG(96)9, 1996: 1-51.
3.
Balls M , et al. Report and recommendations of the CAAT/ ERGATT workshop on the validation of toxicity test procedures . Alternatives Lab Anim1990; 18: 313–336 .
4.
Balls M , et al. Practical aspects of the validation of toxicity test procedures . Alternatives Lab Anim1995; 23: 129–147 .
5.
Bruner LH , Carr GJ , Chamberlain M , Curren RD . Validation of alternative methods for toxicity testing . Toxicol In Vitro1996; 10: 479–501 .
6.
Holzhutter HG , et al. Recommendations for the application of biostatistical methods during the development and validation of alternative toxicological methods . ECVAM Biostatistics Task Force Report 1. Alternatives Lab Anim1996; 24: 511–530 .
7.
Bruner LH , Carr GJ , Curren RD , Chamberlain M . Validation of alternative methods for toxicity testing . Environ Health Perspect1998; 106: 477–484 .
8.
Balls M , Karcher M . The validation of alternative test methods . Alternatives Lab Anim1995; 23: 884–886 .
9.
Balls M , Fentem JH . Progress toward the validation of alternative methods . Alternatives Lab Anim1997; 25: 33–43 .
10.
Frazier JM.Scientific criteria for validation of in vitro toxicity tests. OECD Environment Monograph Number 36. Paris: Organisation for Economic Co-operation and Development (OECD) , 1990: 36: 1-62.
11.
Feder P , et al. Statistical planning and analysis considerations in the evaluation of in vitro alternatives to whole animal use for eye irritation testing . Food Chem Toxicol1997; 35: 167–174 .
12.
Scala RA , Springer J . IRAG working group 6. Guidelines for the evaluation of eye irritation alternative tests: criteria for data submission. Interagency Regulatory Alternatives Group . Food Chem Toxicol1997; 35: 13–22 .
13.
Gettings SD , et al. The CTFA Evaluation of Alternatives Program: an evaluation of in vitro alternatives to the Draize primary eye irritation test. (Phase III) Surfactant-based formulations . Food Chem Toxicol1996; 34: 79–117 .
14.
Brantom PG , et al. A summary report of the COLIPA International validation study on alternatives to the Draize rabbit eye irritation test . Toxicol In Vitro1997; 11: 141–179 .
15.
Cooper JA , Saracci R , Cole P . Describing the validity of carcinogen screening tests . Br J Cancer1979; 39: 87–89 .
16.
Forthofer RN , Lee J.Introduction to biostatistics. A guide to design, analysis, and discovery. San Diego: Academic Press , 1995.
17.
Feinstein AR.On the sensitivity, specificity, and discrimination of diagnostic tests . Clin Pharmacol Ther1975; 17: 104–116 .
18.
Vose D.Monte Carlo simulations: a guide to Monte Carlo simulation modelling. New York, NY: John Wiley & Sons , 1996.
19.
Southee JA , et al. The performance of the tissue equivalent assay using the Skin2TM ZK1200 model in COLIPA international validation study on alternatives to the Draize eye irritation test . Toxicol In Vitro1999; 13: 355–373 .
20.
Draize JH , Woodard G , Calvery JO . Methods for the study of irritation and toxicity of substances applied topically to the skin and mucous membranes . J Pharmacol Exp Pharmacol1944; 82: 377–390 .
21.
Weil CS , Scala RA . Study of intra-and interlaboratory variability in the results of rabbit eye and skin irritation tests . Toxicol Appl Pharmacol1971; 19: 276–360 .
22.
Cormier EM , et al. Determination of the intra-and interlaboratory reproducibility of the low volume eye test and its statistical relationship to the Draize eye test . Regul Toxicol Pharmacol1996; 23: 156–161 .
23.
ECETOC . Eye irritation. Brussels: Reference Chemicals Data Bank , 1992: 48-48. Technical Report No. 48.
24.
Balls M , Botham PA , Bruner LH , Spielmann H . The EC/ HO international validation study on alternatives to the Draize eye irritation test . Toxicol In Vitro1995; 9: 871–929 .
25.
Lordo RA , Feder PI , Gettings SD . Comparing and evaluating alternative (in vitro) tests on their ability to predict the Draize maximum average score . Toxicol In Vitro1999; 13: 45–72 .
26.
Bagley DM , et al. An evaluation of five potential alternatives in vitro to the rabbit eye irritation test in vivo . Toxicol In Vitro1992; 6: 275–284 .
27.
Fentem JH , et al. The ECVAM international validation study on in vitro tests for skin corrosivity? 2. Results and evaluation by the management team . Toxicol In Vitro1998; 12: 483–524 .
28.
Spielmann H , et al. Validation studies of alternatives to the Draize eye irritation test in Germany: cytotoxicity testing and HET-CAM test with 136 industrial chemicals . Toxicol In Vitro1993; 7: 505–510 .
29.
Spielmann H , Liebsch M , Kalweit S , et al. Results of a validation study in Germany on two in vitro alternatives to the Draize eye irritation test, the HET-CAM test and the 3T3-NRU cytotoxicity test . Alternatives Lab Anim2000; 24: 741–858 .
30.
Spielmann H , et al. EEC/COLIPA project on in vitro phototoxicity testing: first results obtained with a BALB/C 3T3 cell phototoxicity assay . Toxicol In Vitro1994; 8: 793–796 .
31.
Spielmann H , et al. The international EU/COLIPA in vitro phototoxicity validation study: results of phase II (blind trial): Part 1. The 3T3 NRU phototoxicity test . Toxicol In Vitro1998; 12: 305–327 .
32.
Spielmann H , Liebsch M , Doering B , Moldenhauer F. First results of the EU/COLIPA validation trial ‘in vitro phototoxicity testing’ . In Vitro Toxicol1996; 9: 325–338 .
33.
Curren RD , et al. The role of prevalidation in the development, validation and acceptance of alternative methods . Alternatives Lab Anim1995; 23: 211–217 .
34.
Bruner LH , et al. Validation of alternative methods for toxicity testing. In Marzulli FN , Maibach HI , eds. Dermatotoxicology. Washington, DC: Taylor and Francis , 1996.
35.
Bagley D , et al. The SDA alternatives program phase III: comparison of in vitro data with animal eye irritation data on solvents, surfactants, oxidizing agents, and prototype cleaning products . J Toxicol Cutan Ocul Toxicol1994; 13: 127–155 .
36.
Gettings SD , et al. The CTFA evaluation of alternatives program: an evaluation of in vitro alternatives to the Draize primary eye irritation test. (Phase II) Oil/water emulsions . Food Chem Toxicol1994; 32: 943–976 .
37.
Gettings SD , et al. Comparison of low-volume, Draize and in vitro eye irritation test data: I. Hydroalcoholic formulations . Food Chem Toxicol1996; 34: 737–749 .
38.
Gettings SD , Lordo RA , Feder PI , Hintze KL . A comparison of low volume, Draize and in vitro eye irritation test data: III. Surfactant-based formulations . Food Chem Toxicol1998; 36: 209–231 .
39.
Gettings SD , Lordo RA , Feder PI , Hintze KL . A comparison of low volume, Draize and in vitro eye irritation test data: II. Oil/water emulsions . Food Chem Toxicol1998; 36: 47–59 .
40.
Spielmann H , et al. Validation study of alternatives to the Draize eye irritation test in Germany: cytotoxicity testing and HET-CAM test with 136 industrial chemicals . Toxicol In Vitro1993; 7: 505–510 .
41.
Spielmann H , et al. EEC/COLIPA in vitro photoirritancy program: results of the first stage of validation . Curr Probl Dermatol1995; 23: 256–264 .
42.
Feder P , et al. Statistical planning and analysis considerations in the evaluation of in vitro alternatives to whole animal use for eye irritation testing . Food Chem Toxicol1997; 35: 167–174 .
43.
Bland JM , Altman DG . Measurement error and correlation coefficients . BMJ1996; 313: 41–42 .
44.
Bland JM , Altman DG . Measuring agreement in method comparison studies . Stat Methods Med Res1999; 8: 135–160 .
45.
Bland JM , Altman DG . Comparing methods of measurement: why plotting difference against standard method is misleading . Lancet1995; 346: 1085–1087 .
46.
Bland JM , Altman DG . Calculating correlation coefficients with repeated observations: Part 1. Correlation within subjects . BMJ1995; 310: 446–446 .
47.
Bland JM , Altman DG . Statistical methods for assessing agreement between two methods of clinical measurement . Lancet1986; 1: 307–310 .
48.
Hogg RV , Ledholter J . Applied statistics for engineers and physical scientists. New York, NY: Macmillan , 1992.
