Statistical Methods and Software for Validation Studies on New In Vitro Toxicity Assays

Abstract

When a new in vitro assay method is introduced, it should be validated against the best available knowledge or a reference standard assay. For assays resulting in a simple binary outcome, the data can be displayed as a 2 × 2 table. Based on the estimated sensitivity and specificity, and the assumed prevalence of true positives in the population of interest, the positive and negative predictive values of the new assay can be calculated. We briefly discuss the experimental design of validation experiments and previously published methods for computing confidence intervals for predictive values. The application of the methods is illustrated for two toxicological examples, by using tools available in the free software, namely, R: confidence intervals for predictive values are computed for a validation study of an in vitro test battery, and sample size calculation is illustrated for an acute toxicity assay. The R code necessary to reproduce the results is given.

Keywords

confidence interval diagnostic test predictive value sample size

Get full access to this article

View all access options for this article.

References

Hoffmann

& Hartung

(2005). Diagnosis: Toxic! — Trying to apply approaches of clinical diagnostics and prevalence in toxicology considerations. Toxicological Science 85, 422–428.

Tsamou

, Jennen

D.G.J.

, Claessen

S.M.H.

, Magkoufopoulou

, Kleinjans

J.C.

& van Delft

J.H.

(2012). Performance of in vitro γH2AX assay in HepG2 cells to predict in vivo genotoxicity. Mutagenesis 27, 645–652.

Brenner

& Gefeller

(1997). Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Statistics in Medicine 16, 981–991.

Brown

L.D.

, Cai

T.T.

& DasGupta

(2001). Interval estimation for a binomial proportion. Statistical Science 16, 101–128.

Pepe

M.S.

(2003). The Statistical Evaluation of Medical Tests for Classification and Prediction, 302 pp. Oxford, UK: Oxford University Press.

Mercaldo

N.D.

, Lau

K.F.

& Zhou

X.H.

(2007). Confidence intervals for predictive values with emphasis to case-control studies. Statistics in Medicine 26, 2170–2183.

Steinberg

D.M.

, Fine

& Chappell

(2009). Sample size for positive and negative predictive value in diagnostic research using case-control designs. Biostatistics 10, 94–105.

R Core Team (2013). R: A language and Environment for Statistical Computing, R 3.0.2. Vienna, Austria: R Foundation for Statistical Computing. Available at: http://www.R-project.org/ (Accessed 23.09.14).

Scherer

(2013). PropCIs: Various Confidence Interval Methods for Proportions, R package version 0.2-0. Available at: http://CRAN.R-project.org/package=PropCIs (Accessed 20.09.14).

10.

Stevenson

T.N.

, Heuer

, Marshall

, Sanchez

, Thornton

, Reiczigel

, Robison-Cox

, Sebastiani

, Solymos

, Yoshida

& Firestone

(2014). epiR: An R Package for the Analysis of Epidemiological Data, R package version 0.9-57. Available at: http://CRAN.R-project.org/package=epiR (Accessed 20.09.14).

11.

Stock

& Hielscher

(2014). DTComPair: Comparison of Binary Diagnostic Tests in a Paired Study Design, R package version 1.0.3. Available at: http://CRAN.R-project.org/package=DTComPair (Accessed 20.09.14).

12.

Stamey

J.D.

& Holt

M.M.

(2010). Bayesian interval estimation for predictive values from case-control studies. Communications in Statistics: Simulation & Computation 39, 101–110.

13.

Schaarschmidt

(2014). bdpv: Inference and Design for Predictive Values in Binary Diagnostic Tests, R package version 1.1. Available at: http://CRAN.R-project.org/package=bdpv (Accessed 20.09. 14).

14.

Zhang

, Wu

V.Q.

, Chow

S-C.

& Zhang

H.G.

(2013). TrialSize: R Functions in Chapter 3,4,6,7,9, 10,11,12,14,15, R package version 1.3. Available at: http://CRAN.R-project.org/package=TrialSize (Accessed 20.09.14).

15.

Zhang

, Bilder

, Biggerstaff

& Schaarschmidt

(2012). binGroup: Evaluation and Experimental Design for Binomial Group Testing, R package version 1.1-0. Available at: http://CRAN.R-project.org/package=binGroup (Accessed 20.09.14).

16.

Kirkland

, Aardema

, Mueller

& Hayashi

(2006). Evaluation of the ability of a battery of three in vitro genotoxicity tests to discriminate rodent carcinogens and non-carcinogens II. Further analysis of mammalian cell results, relative predictivity and tumour profiles. Mutation Research 608, 29–42.

17.

Bulgheroni

, Kinsner-Ovaskainen

, Hoffmann

, Hartung

& Prieto

(2009). Estimation of acute oral toxicity using the No Observed Adverse Effect Level (NOAEL) from the 28 day repeated dose toxicity studies in rats. Regulatory Toxicology & Pharmacology 53, 16–19.

18.

Lange

& Brunner

(2013). Analysis of predictive values based on individual risk factors in multi-modality trials. Diagnostics 3, 192–209.