The cross-entropy and minimum cross-entropy methods are well-known Monte Carlo simulation techniques for rare-event probability estimation and optimization. In this paper, we investigate how these methods can be eXtended to provide a general non-parametric cross-entropy framework based on φ-divergence distance measures. We show how the χ2 distance, in particular, yields a viable alternative to the Kullback—Leibler distance. The theory is illustrated with various eXamples from density estimation, rare-event simulation and continuous multi-eXtremal optimization.
Rubinstein, R.Y. and D.P. Kroese.2004. The Cross-Entropy Method , New York: Springer .
2.
Rubinstein, R.Y.2005. The stochastic minimum cross-entropy method for combinatorial optimization and rare-event estimation . Methodology and Computing in Applied Probability7, 5—50 .
3.
Kapur, J.N. and H.K. Kesavan.1989. The generalized maximum entropy principle. IEEE Transactions on Systems , Man, and Cybernetics19, 1042—1052 .
4.
Havrda, J.H. and F. Charvát.1967. Quantification methods of classification processes: Concepts of structural α entropy . Kybernatica3, 30—35 .
5.
Botev, Z.I. and D.P. Kroese.2006, http://espace.library.uq.edu.au/view.php?pid=UQ:12759. The generalized cross entropy method, with applications to probability density estimation. Electronic Preprint.
6.
Scott, D.W.1992. Multivariate Density Estimation. Theory, Practice and Visualization. John Wiley & Sons .
7.
Wand, M.P. and M.C. Jones.1995. Kernel Smoothing. Chapman & Hall .
8.
Homem-de-Mello, T. and R.Y. Rubinstein.Rare event probability estimation for static models via cross-entropy and importance sampling. Technical Report 2002, Ohio State University : http://www.iwse.eng.ohio-state.edu/ISEFaculty/tito/pubs/rarevents.pdf.
9.
Botev, Z.I.2005. Stochastic Methods for Optimization and Machine Learning . Technical Report: ePrintsUQ, http://eprint.uq.edu.au/archive/00003377/.
10.
Kapur, J.N. and H.K. Kesavan.1987. Generalized Maximum Entropy Principle (With applications). Waterloo, Ontario, Canada: StandfordEducational Press , University of Waterloo .
11.
Kuhn, H.W. and A.W. Tucker.1951. Nonlinear programming . In Proceedings of the 2nd Berkeley Symposium, pp. 481—492. Berkeley: University of California Press.
12.
Decarreau, A. , D. Hilhorst , C. Lemarechal , and J. Navaza.1992. Dual methods in entropy maximization: applications to some problems in crystalography . SIAM Journal of Optimization.
13.
Borwein, J.M. and A.S. Lewis.1991. Duality relationships for entropy-like minimization problems . SIAM Journal of Control and Optimization29, 325—338 .
14.
Ben-Tal, A. and M. Teboulle.1987. Penalty functions and duality in stochastic programming via Á divergence functionals . Mathematics of Operations Research12, 224—240 .
15.
Kapur, J.N.1994. Measures of Information and their Applications. New Delhi, India: John Wiley & Sons .
16.
Devroye, L. and L. Gyofri.1985. Nonparametric Density Estimation: The L1 View. Wiley Series In Probability And Mathematical Statistics, New York: Wiley .
17.
Gelman, A. , J.B. Carlin , H.S. Stern , and D.B. Rubin.2003. Bayesian Data Analysis. Chapman & Hall , 2nd edition.
18.
Pawitan, Y.2001. In All Likelihood: Statistical Modeling and Inference Using Likelihood. Oxford: Clarendon Press .
19.
Bowman, A.W.1985. A comparative study of some kernel-based non-parametric density estimators . Journal of Statistical Computation and Simulation21, 313—327 .
20.
Silverman, B.W.1986. Density Estimation for Statistics and Data Analysis. Chapman and Hall .
21.
Simonoff, J.S.1996. Smoothing Methods in Statistics , New York: Springer .
22.
Jones, M.C. , J.S. Marron , and S.J. Sheather.1996. Progress in data-based bandwidth selection for kernel density estimation . Computational Statistics11, 337—381 .
23.
Hall, P. and B.A. Turlach.1999. Reducing bias in curve estimation by use of weights . Computational Statistics and Data Analysis30, 67—86 .
24.
Robert, C.P. and G. Casella.2004. Monte Carlo Statistical Methods. New York: Springer , 2nd edition.
25.
Vapnik, V.1998. Statistical Learning Theory, New York : John Wiley & Sons .
26.
Anderson, J.A. , K. Whale , J. Williamson , and W.W. Buchanan.1972. A statistical aid to the diagnosis of keratoconjunctivitis sicca . Quarterly Journal of Medicine41, 175—189 .
27.
Aitchison, J. and C. Aitken.1976. Multivariate binary discrimination by the kernel method . Biometrika63, 413—420 .
