Dimensionality optimization by heuristic greedy learning vs. genetic algorithms in knowledge discovery and data mining

Abstract

Dimensionality optimization involves optimizing the size of data sets from both dimensions, variable and observation selections. The ultimate objective of dimensionality optimization is to obtain the induced data space, by reducing both dimensionalities in such a way that the reduced subset could retain sufficient information. In most real-world applications, it is not known what the best subset is and what should be contained in such a subset. Selecting the appropriate subset is extremely important in effectively mining over large data sets in the sense that it is the only source for any data mining and knowledge discovery algorithm to work with the data of interest reliably.

The statistical as well as artificial intelligence community has provided good methods in this domain, but still a lot of improvements need to be made, especially for data mining applications. This paper introduces a heuristic methodology that integrates heuristic greedy search methods and tree-structured SampleC4.5 to efficiently find the optimal subset of very large data sets from both dimensions simultaneously. A GA-based optimization approach is also proposed in the paper. Experimental results are presented which illustrate the effectiveness of our approaches in digging out the important underlying patterns, and indicate the potential advantages of the proposed techniques to improve the optimizing process while staying out of misleading dilemma. The results of our experiments also show the robustness of our approaches and complementary characteristics for knowledge discovery and data mining tasks.