NICFS: A novel feature selection method applied to lexicon based sentiment analysis

Abstract

The technique used for feature selection immensely affects the performance of classification in case of high dimensional datasets. The bag-of-word model is often used for sentiment classification using machine learning. The set of unique words in a text based dataset constitute the feature vector which has a high dimension. In this work, a new feature selection method has been proposed which chooses features that are highly relevant to the class using the Information Gain score and least redundant with respect to the selected feature set using the CHI square statistic. The two scores were normalized to make them comparable. The proposed method was applied to lexicon based sentiment analysis using the lexicon SentiWordNet. Previously, IG and mRMR have been shown to be the best filter feature selection methods for sentiment term selection. The performance of our proposed method in terms of classification accuracy in sentiment analysis is significantly higher than IG and mRMR methods. Experiments were performed on three datasets in different domains. Also, the feature subset obtained by removing words with zero polarity score in SentiWordNet led to a significant reduction in feature vector size with no effect on performance.