Supervised feature extraction algorithm based on improved polynomial entropy

Abstract

Based on information entropy theory, a novel feature extraction algorithm based on improved polynomial entropy (IPE) is set up. Firstly, the concepts and their properties of information entropy and cross entropy (CE) are analysed and studied. On this foundation, we prove that symmetrical cross entropy (SCE) proposed here based on CE satisfies three axioms of the distance, i.e. nonnegativity, symmetry and triangle inequation. So SCE is a kind of distance measure, which can be used to measure the degree of variation between two random variables. Secondly, for convenience, we propose a new concept of improved polynomial entropy (IPE) based on polynomial entropy (PE), and explain that IPE is equivalent to SCE. Thirdly, we make IPE separability criterion of the classes for feature extraction, and design a novel feature extraction algorithm based on IPE. Fianlly, the experimental results demonstrate that the algorithm proposed here is valid and reliable, and it provides a new research approach for feature extraction, data mining and pattern recognition.

Keywords

information entropy theory cross entropy (CE)symmetrical cross entropy (SCE)polynomial entropy (PE)improved polynomial entropy (IPE)supervised feature extraction

Get full access to this article

View all access options for this article.

References

R. O. Duda and P. E. Hart (eds), Pattern Classification and Scene Analysis ( Wiley, New York , 1973).

L. Devroye , L. Gyorfi and G. Lugosi (eds), A Probabilistic Theory of Pattern Recognition ( Springer-Verlag, New York , 1996).

K. Fukunaga (ed.), Introduction to Statistical Pattern Recognition ( Academic Press, New York , 1990).

D. J. Hand (ed.), Discrimination and Classification ( Wiley, New York , 1981).

M. Nadler and E. P. Smith (eds), Pattern Recognition Engineering ( Wiley, New York , 1993).

J. G. Gy and C. E. Brodley , Feature subset selection and order identification for unsupervised learning . In P. Langley (ed.), International Conference on Machine Learning: Proceedings of 17th International Conference on Machine Learning (ICML 2000) (Morgan Kaufmann, San Francisco, 2000) 247-254 .

S. Ranganath and K. Arun , Face recognition using transform features and neural networks , Pattern Recognition 30(10) (1997) 1615-1622 .

M. Turk and A. Pentland , Eigenfaces for recognition , Journal of Cognitive Neuroscience 3(1) (1991a) 71-86 .

J. Yang and J. Y. Yang , A generalized K-L expansion method that can deal with small sample size and high-dimensional problems , Pattern Analysis Applications 6(6) (2003) 47-54 .

10.

J. Yang and J. Y. Yang . Generalized K-L transform based combined feature extraction , Pattern Recognition 35(1) (2002) 295-297 .

11.

T. Young and K. S. Fu (eds), Handbook of Pattern Recognition and Image Processing ( Academic Press, New York , 1986).

12.

C. E. Shannon , A mathematical theory of communication , Bell Systems Technical Journal (1948) 379-423 .

13.

T. M. Cover and J. A. Thomas (eds), Elements of Information Theory ( Wiley, New York , 1991).

14.

Z. X. Ye (ed.), Information Theory Foundation ( Higher Education Press, Beijing , 2003).

15.

X. H. Cao and Y. C. Zhang (eds), Information Theory and Coding ( Beijing University of Post and Telecommunications Press, Beijing , 2001).

16.

P. Nath , On the measures of errors in information , Journal of Mathematical Sciences 3 (1968) 1-16 .

17.

S. Watanabe , A new entropic method of dimensionality reduction specially designed for multiclass discrimination , Pattern Recognition Letters 2(1) (1983) 1-4 .

18.

S. H. Chen and C. L. Lu , An entropic approach to dimensionality reduction in the representation space on discrete processes , Journal of Nanjing Institute of Meteorology 24(1) (2001) 74-82 .