The large-scale multiple testing problems resulting from the measurement of thousands of genes in microarray experiments have received increasing interest during the past several years. This article describes some commonly used criteria for controlling false positive errors, including familywise error rates, false discovery rates and false discovery proportion rates. Various statistical methods controlling these error rates are described. The advantages and disadvantages of these methods are discussed. These methods are applied to gene expression data from two microarray studies and the properties of these multiple testing procedures are compared.
Get full access to this article
View all access options for this article.
References
1.
DeRisi J. , Penland L., Brown PO et al. Use of a cDNA microarray to analyse gene expression patterns in human cancer. Nature Genetics1996; 14: 457-60.
2.
Schena M., Shalon D., Davis RW, Brown POQuantitative monitoring of gene expression patterns with a complementary DNA microarray. Science1995; 270: 467-70.
3.
Lockhart DJ, Dong HL, Byrne MC et al. Expression monitoring by hybridization to high-density oligonucleotide arrays. Nature Biotechnology1996; 14: 1675-80.
4.
Wodicka L., Dong H., Mittmann M., Ho MH, Lockhart DJGenome-wide expression monitoring in Saccharomyces cerevisiae. Nature Biotechnology1997; 15(13): 1359-67.
Parmigiani G, Garrett ES, Irizarry RA, Zeger SL, eds. The analysis of gene expression data. Springer, New York; 2003.
8.
Dudoit S., van der Laan MJ, Pollard KSMultiple testing Part I. Single-step procedures for control of general Type I error rates. Statistical Applications in Genetics and Molecular Biology2004; 3(1): Article 13.
9.
Callow MJ, Dudoit S., Gong EL, Speed TP, Rubin EMMicroarray expression profiling identifies genes with altered expression in HDL deficient mice. Genome Research2000 ; 10(12): 2022-29.
10.
Golub TR, Slonim DK, Tamayo P. et al. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science1999 ; 286: 531-7.
11.
Welch BLThe significance of the difference between two means when the population variances are unequal. Biometrika1938; 29: 350-62.
12.
Westfall PH , Young SSResampling-based multiple testing: examples and methods for p-value adjustment . John Wiley & Sons, New York ; 1993.
13.
Benjamini Y. , Hochberg Y.Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B1995; 57: 289-300.
14.
Korn EL, Troendle JF, McShane LM, Simon R.Controlling the number of false discoveries: Application to high dimensional genomic data. Journal of Statistical Planning and Inference2004; 124: 379-98.
15.
Romano JP, Wolf M.Exact and approximate stepdown methods for multiple hypothesis testing . Journal of the American Statistical Association2005; 100(469): 94-108.
16.
Genovese C. , Wasserman L.Operating characteristics and extensions of the false discovery rate procedure. Journal of the Royal Statistical Society B2002; 64: 499-517.
17.
Taylor J., Tibshirani R., Efron B.The ‘miss rate’ for the analysis of gene expression data . Biostatistics2005; 6(1): 111-7.
18.
Lehmann EL, Romano JPGeneralization of the familywise error rate. The Annals of Statistics2005; 33: 1138-54.
19.
Fernando RL , Nettleton D., Southey BR, Dekkers JCM, Rothschild MF, Soller M.Controlling the proportion of false positives in multiple dependent tests. Genetics2004 ; 166: 611-9.
20.
Storey JDA direct approach to false discovery rates. Journal of the Royal Statistical Society B2002; 64: 479-98.
21.
Storey JD, Taylor JE, Siegmund D.Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society B2004; 66: 187-205.
22.
Seeger P.A note on a method for the analysis of significance en masse. Technometrics1968; 10(3): 586-93.
23.
Benjamini Y. , Yekutieli D.The control of the false discovery rate in multiple hypothesis testing under dependency. The Annals of Statistics2001; 29(4): 1165-88.
24.
Tusher V., Tibshirani R., Chu G.Significance analysis of microarrays applied to the ionizing radiation response . Proceedings of the National Academy of Sciences2001; 98: 5116-21.
25.
Efron B., Tibshirani R., Storey JD, Tusher V.Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association2001; 96(456): 1151-60.
26.
Efron B.Microarrays, emprical bayes, and the two-groups model; 2006. Department of Statistics, Stanford University. Available at: http://www.-stat.stanford.edu/″brad/papers
27.
Bonferroni CETeoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze1936; 8: 3-62.
28.
Holm S.A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics1979; 6: 65-70.
29.
Šidák Z.Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association1967; 62: 626-33.
30.
Romano JP, Shaikh AMStepup procedures for control of generalisations of the familywise error rate . Annals of Statistics2006a; 34(4): 1850-73.
31.
Romano JP, Shaikh AMOn stepdown control of the false discovery proportion. In Rojo J. ed. Optimality: The Second Erich L. Lehmann Symposium, IMS Lecture Notes: Monograph Series, Vol. 49, Institute of Mathematical Statistics, Bethesda; 2006b: 33-50.
32.
Ge Y., Sealfon SC, Tseng CH, Speed TPA Holm-type procedure controlling the false discovery rate. Statistics and Probability Letters2007, 77: 1756-62.
33.
Ge Y., Dudoit S., Speed TPResampling-based multiple testing for microarray data analysis. Test2003; 12(1): 1-44. With discussion 44-77.
van der Laan MJ, Dudoit S., Pollard KSMultiple testing.Part II. Step-down procedures for control of the family-wise error rate. Statistical Applications in Genetics and Molecular Biology2004; 3(1): Article 14.
36.
Ge Y., Li X., Sealfon SC, Speed TPAn upper prediction bound for the false discovery proportion. In JSM, Statistical Computing Section, American Statistical Association, Alexandria, VA; 2005 : 2093-100.
37.
Meinshausen N., Rice J.Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. The Annals of Statistics2006; 34(1): 373-93.
38.
Meinshausen N.False discovery control for multiple tests of association under general dependence. Scandinavian Journal of Statistics2006; 33(2): 227-37.
39.
Genovese C. , Wasserman L.A stochastic process approach to false discovery control. The Annals of Statistics2004; 32: 1035-61.
40.
Bolstad BM, Irizarry RA, Astrand M., Speed TPA comparison of normalisation methods for high density oligonucleotide array data based on variance and bias. Bioinformatics2003; 19: 185-93.
41.
Owen ABVariance of the number of false discoveries. Journal of the Royal Statistical Society, Series B2005; 67(3): 411-26.
42.
Yekutieli D. , Benjamini Y.Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical Planning and Inference1999 ; 82: 171-96.
43.
Efron B. Large-scale simultaneous hypothesis testing: the choice of a null hypothesis. Journal of the American Statistical Association 2004; 99 (465): 96-104. 44 Pawitan Y, Michiels S, Koscielny S, Gusnanto A, Ploner A. False discovery rate and sample size for microarray studies. Bioinformatics. 2005; 21: 3017-24.
