Comprehensive empirical investigation for prioritizing the pipeline of using feature selection and data resampling techniques

Abstract

The contemporary real-world datasets often suffer from the problem of class imbalance as well as high dimensionality. For combating class imbalance, data resampling is a commonly used approach whereas for tackling high dimensionality feature selection is used. The aforesaid problems have been studied extensively as independent problems in the literature but the possible synergy between them is still not clear. This paper studies the effects of addressing both the issues in conjunction by using a combination of resampling and feature selection techniques on binary-class imbalance classification. In particular, the primary goal of this study is to prioritize the sequence or pipeline of using these techniques and to analyze the performance of the two opposite pipelines that apply feature selection before or after resampling techniques i.e., F + S or S + F. For this, a comprehensive empirical study is carried out by conducting a total of 34,560 tests on 30 publicly available datasets using a combination of 12 resampling techniques for class imbalance and 12 feature selection methods, evaluating the performance on 4 different classifiers. Through the experiments we conclude that there is no specific pipeline that proves better than the other and both the pipelines should be considered for obtaining the best classification results on high dimensional imbalanced data. Additionally, while using Decision Tree (DT) or Random Forest (RF) as base learner the predominance of S + F over F + S is observed whereas in case of Support Vector Machine (SVM) and Logistic Regression (LR), F + S outperforms S + F in most cases. According to the mean ranking obtained from Friedman test the best combination of resampling and feature selection techniques for DT, SVM, LR and RF are SMOTE + RFE (Synthetic Minority Oversampling Technique and Recursive Feature Elimination), Least Absolute Shrinkage and Selection Operator (LASSO) + SMOTE, SMOTE + Embedded feature selection using RF and SMOTE + RFE respectively.

Keywords

Imbalanced data feature selection machine learning oversampling undersampling

Get full access to this article

View all access options for this article.

References

Al-Asadi , Mustafa

, Sakir Tasdemír , Empirical comparisons for combining balancing and feature selection strategies forcharacterizing football players using FIFA video game system, IEEE Access 9 (2021), 149266–149286.

Al-Ghraibah

, Boucheron

L.E.

, McAteer

R.J.

A study of feature selection of magnetogram complexity features in an imbalanced solar flare prediction data-set. In 2015 IEEE International Conference on Data Mining Workshop (ICDMW) (2015) (pp. 557–564). IEEE.

Al-Ghraibah , Amani , Laura Boucheron

, James McAteer

R.T.

A study of feature selection of magnetogram complexity features in an imbalanced solar flare prediction data-set, 2015 IEEE International Conference on Data Mining Workshop (ICDMW). IEEE, 2015.

Alibeigi , Mina , Sattar Hashemi , Ali Hamzeh , DBFS: An effective Density Based Feature Selection scheme for small sample size andhigh dimensional imbalanced data sets, Data & KnowledgeEngineering 81 (2012), 67–103.

Amin

et al., Comparing oversampling techniques to handle the class imbalance problem: A customer churn prediction case study, IEEE Access 4 (2016), 7940–7957.

Ang

J.C.

, Mirzal

, Haron

and Hamed

H.N.A.

, Supervised,unsupervised, and semi-supervised feature selection: a review ongene selection, IEEE/ACM Transactions on Computational Biologyand Bioinformatics 13(3) (2015), 971–989.

Asim , Muhammad Nabeel et al. Comparison of feature selection methods in text classification on highly skewed datasets, 2017 First International Conference on Latest trends in Electrical Engineering and Computing Technologies (INTELLECT). IEEE, 2017.

Asuncion Arthur , David Newman UCI machine learning repository (2007).

Batista , Gustavo

E.A.P.A.

, Ronaldo Prati

, Maria Carolina Monard , A study of the behavior of several methods for balancing machine learning training data, ACM SIGKDD Explorations Newsletter 6.1 (2004), 20–29.

10.

Bhatia Anshul , Anuradha Chug , Amit Prakash Singh , Plant disease detection for high dimensional imbalanced dataset using an enhanced decision tree approach, International Journal of Future Generation Communication and Networking 13.4 (2020), 71–78.

11.

Bolón-Canedo

, Sánchez-Maroño

, Alonso-Betanzos

Feature selection for high-dimensional data. Cham: Springer International Publishing (2015).

12.

Cervantes

, Garcia-Lamont

, Rodríguez-Mazahua

and Lopez

, A comprehensive survey on support vector machineclassification: Applications, challenges and trends, Neurocomputing 408 (2020), 189–215.

13.

Chawla Nitesh

et al., SMOTE: synthetic minority over-sampling technique, Journal of Artificial Intelligence Research 16 (2002), 321–357.

14.

Chen Hongmei

et al., Feature selection for imbalanced data based on neighborhood rough sets, Information Sciences 483 (2019), 1–20.

15.

Chong

, Tjurin

, Niemelä

, Jämsä

and Farrahi

, Machine-learning models for activity class prediction: A comparative study of feature selection and classification algorithms, Gait & Posture 89 (2021), 45–53.

16.

de Haro-García , Aida , Gonzalo Cerruela-García , Nicolás García-Pedrajas , Ensembles of feature selectors fordealing with class-imbalanced datasets: A proposal and comparativestudy, Information Sciences 540 (2020), 89–116.

17.

Demšar , Janez , Statistical comparisons of classifiers overmultiple data sets,, The Journal of Machine learning Research 7 (2006), 1–30.

18.

Derrac

et al., Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework, J. Mult. Valued Logic Soft Comput 17 (2015).

19.

Derrac Joaquín

et al., A practical tutorial on the use ofnonparametric statistical tests as a methodology for comparingevolutionary and swarm intelligence algorithms, Swarm andEvolutionary Computation 1.1 (2011), 3–18.

20.

Desuky Abeer

, Sadiq Hussain , An improved hybrid approach for handling class imbalance problem, Arabian Journal for Science and Engineering 46 (2021), 3853–3864.

21.

Ding Chris , Hanchuan Peng , Minimum redundancy feature selection from microarray gene expression data, Journal of Bioinformatics and Computational Biology 3.02 (2005), 185–205.

22.

Drotár Peter , Juraj Gazda , Zdenek Smékal , An experimentalcomparison of feature selection methods on two-class biomedicaldatasets, Computers in Biology and Medicine 66 (2015), 1–10.

23.

Du Guodong

et al., Joint imbalanced classification and feature selection for hospital readmissions, Knowledge-Based Systems 200 (2020), 106020.

24.

Fallahpour Saeid , Eisa Norouzian Lakvan , Mohammad Hendijani Zadeh , Using an ensemble classifier based on sequential floating forward selection for financial distress prediction problem, Journal of Retailing and Consumer Services 34 (2017), 159–167.

25.

Faris Hossam

et al., Improving financial bankruptcy prediction in a highly imbalanced class distribution using oversampling and ensemble learning: a case from the Spanish market, Progress in Artificial Intelligence 9 (2020), 31–53.

26.

Feng Fang

et al., Using cost-sensitive learning and feature selection algorithms to improve the performance of imbalanced classification, IEEE Access 8 (2020), 69979–69996.

27.

Fu Guang-Hui

et al., Hellinger distance-based stable sparse feature selection for high-dimensional class-imbalanced data, BMC Bioinformatics 21.1 (2020), 1–14.

28.

Fu Guang-Hui , Lun-Zhao Yi , Jianxin Pan , LASSO-based false-positive selection for class-imbalanced data in metabolomics, Journal of Chemometrics 33.10 (2019), e3177.

29.

Galar Mikel

et al., An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes, Pattern Recognition 44.8 (2011), 1761–1776.

30.

Ghorbani Ramin , Rouzbeh Ghousi , Comparing different resampling methods in predicting students’ performance using machine learning techniques, IEEE Access 8 (2020), 67899–67911.

31.

Ghosh Arka , Bing Xue , Mengjie Zhang Binary Differential Evolution based Feature Selection Method with Mutual Information for Imbalanced Classification Problems, 2021 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2021.

32.

Gillala Rekha

et al., An efficient chaotic salp swarm optimization approach based on ensemble algorithm for class imbalance problems, Soft Computing 25.23 (2021), 14955–14965.

33.

Hang

, Yang

and Xing

, Diagnosis of rolling bearing based on classification for high dimensional unbalanced data, IEEE Access 7 (2019), 79159–79172.

34.

Haq , Anwar Ul et al., Combining multiple feature-ranking techniques and clustering of variables for feature selection, IEEE Access 7 (2019), 151482–151492.

35.

Hasanin

, Khoshgoftaar

T.M.

, Leevy

, Seliya

Investigating random undersampling and feature selection on bioinformatics big data. In 2019 IEEE Fifth International Conference on Big Data Computing Service and Applications (BigDataService) (2019) (pp. 346–356). IEEE.

36.

He Yuanyu

et al., A class imbalance-aware Relief algorithm for the classification of tumors using microarray gene expression data, Computational Biology and Chemistry 80 (2019), 121–127.

37.

Huang

M.W.

, Chiu

C.H.

, Tsai

C.F.

and Lin

W.C.

, On combining feature selection and over-sampling techniques for breast cancer prediction, Applied Sciences 11(14) (2021), 6574.

38.

Jonathan Panca Hadi Putra , Yova Ruldeviyani, Observation imbalanced data text to predict users selling products on female daily with smote, tomek, and smotetomek, 2020 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT). IEEE, 2020.

39.

Jović

, Brkić

, Bogunović

A review of feature selection methods with applications. In 2015 38th international convention on information and communication technology, electronics and microelectronics (MIPRO) (2015) (pp. 1200–1205). Ieee.

40.

Kamalov Firuz , Fadi Thabtah , Ho Hon Leung Feature selection in imbalanced data, Annals of Data Science (2022), 1–15.

41.

Kaur Prabhjot , Anjana Gosain , FF-SMOTE: a metaheuristic approach to combat class imbalance in binary classification, Applied Artificial Intelligence 33.5 (2019), 420–439.

42.

Khaldy

M.A.

and Kambhampati

, Resampling imbalanced class and theeffectiveness of feature selection methods for heart failuredataset, International Robotics & Automation Journal 4(1) (2018), 1–10.

43.

Khushi Matloob

et al., A comparative performance analysis of data resampling methods on imbalance medical data, IEEE Access 9 (2021), 109960–109975.

44.

Kim Jongmo , Jaewoong Kang , Mye Sohn , Ensemble learning-based filter-centric hybrid feature selection framework for high-dimensional imbalanced data, Knowledge-Based Systems 220 (2021), 106901.

45.

Kumar Lov , Ashish Sureka Feature selection techniques to counter class imbalance problem for aging related bug prediction: aging related bug prediction, Proceedings of the 11th Innovations in Software Engineering Conference. 2018.

46.

Kumari Ritika , Jaspreeti Singh , Anjana Gosain , SmS: SMOTE-stacked hybrid model for diagnosis of polycystic ovary syndrome using feature selection method, Expert Systems with Applications 225 (2023), 120102.

47.

Lachheta Pawan , Seema Bawa Combining synthetic minority oversampling technique and subset feature selection technique for class imbalance problem, Proceedings of the International Conference on Advances in Information Communication Technology & Computing. 2016.

48.

, Fong

, Zhuang

Optimizing SMOTE by metaheuristics with neural network and decision tree. In 2015 3rd International Symposium on Computational and Business Intelligence (ISCBI) (2015) (pp. 26–32). IEEE.

49.

Li Kewen

et al., Feature selection method based on weighted mutual information for imbalanced data, International Journal of Software Engineering and Knowledge Engineering 28.08 (2018), 1177–1194.

50.

Liu Haoyue

et al. Weighted Gini index feature selection method for imbalanced data, 2018 IEEE 15th international conference on networking, sensing and control (ICNSC), IEEE, 2018.

51.

Liu Haoyue , MengChu Zhou , Qing Liu , An embedded feature selection method for imbalanced data classification, IEEE/CAA Journal of Automatica Sinica 6.3 (2019), 703–715.

52.

Liu Yi

et al., A classification method based on feature selection for imbalanced data, IEEE Access 7 (2019), 81794–81807.

53.

Friedman

, A comparison of alternative tests of significance for the problem of m ranking, Annals of Math Statistics 11 (1940), 86–92.

54.

Maldonado Sebastián , Julio López , Carla Vairetti , Analternative SMOTE oversampling strategy for high-dimensionaldatasets, Applied Soft Computing 76 (2019), 380–389.

55.

Matharaarachchi Surani , Mike Domaratzki , Saman Muthukumarana , Assessing feature selection method performance with class imbalance data, Machine Learning with Applications 6 (2021), 100170.

56.

Moayedikia Alireza

et al., Feature selection for high dimensional imbalanced class data using harmony search, Engineering Applications of Artificial Intelligence 57 (2017), 38–49.

57.

More Ajinkya Survey of resampling techniques for improving classification performance in unbalanced datasets, arXiv preprint arXiv:1608.06048 (2016).

58.

Nagpal Arpita , Vijendra Singh , Feature selection from high dimensional data based on iterative qualitative mutual information, Journal of Intelligent & Fuzzy Systems 36.6 (2019), 5845–5856.

59.

Omar Boutkhoum

et al., Minimizing the overlapping degree to improve class-imbalanced learning under sparse feature selection: application to fraud detection, IEEE Access 9 (2021), 28101–28110.

60.

Pes Barbara , , Learning from high-dimensional and class-imbalanced datasets using random forests, Information 12.8 (2021), 286.

61.

Pes Barbara , Learning from high-dimensional biomedical datasets: the issue of class imbalance, IEEE Access 8 (2020), 13527–13540.

62.

Pudjihartono Nicholas

et al., A review of feature selection methods for machine learning-based disease risk prediction, Frontiers in Bioinformatics 2 (2022), 927312.

63.

Radovic Milos

et al., Minimum redundancy maximum relevance feature selection approach for temporal gene expression data, BMC Bioinformatics 18.1 (2017), 1–14.

64.

Ramos-Pérez , Ismael et al., When is resampling beneficial forfeature selection with imbalanced wide data? Expert Systemswith Applications 188 (2022), 116015.

65.

Shahee Shaukat Ali , Usha Ananthakumar , An effective distance based feature selection approach for imbalanced data, Applied Intelligence 50 (2020), 717–745.

66.

Shanab

A.A.

, Khoshgoftaar

T.M.

Is gene selection enough for imbalanced bioinformatics data?. In 2018 IEEE International Conference on Information Reuse and Integration (IRI) (2018) (pp. 346–355). IEEE.

67.

Shanab Ahmad Abu

et al. Comparison of approaches to alleviate problems with high-dimensional and classimbalanced data, 2011 IEEE International Conference on Information Reuse & Integration. IEEE, 2011.

68.

Sharma Ajay , Pramod Kumar Mishra Performance analysis of machine learning based optimized feature selection approaches for breast cancer diagnosis, International Journal of Information Technology (2021), 1–12.

69.

Shi Xiujin , Feifei Kong , Hao Li Research on Credit Evaluation Model for High-Dimensional Imbalanced Data, 2021 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA). IEEE, 2021.

70.

Singh Deepika , Anjana Gosain , Anju Saha , Weighted k-nearest neighbor based data complexity metrics for imbalanced datasets, Statistical Analysis and Data Mining: The ASA Data Science Journal 13.4 (2020), 394–404.

71.

Singh Deepika , Anju Saha , Anjana Gosain , wCM based hybridpre-processing algorithm for class imbalanced dataset, Journalof Intelligent & Fuzzy Systems 41.2 (2021), 3339–3354.

72.

Sun Lin

et al., TFSFB: Two-stage feature selection via fusing fuzzy multi-neighborhood rough set with binary whale optimization for imbalanced data, Information Fusion 95 (2023), 91–108.

73.

Thaher Thaer

et al. Wrapper-based feature selection for imbalanced data using binary queuing search algorithm, 2019 2nd International Conference on New Trends in Computing Sciences (ICTCS). IEEE, 2019.

74.

Tsai Chih-Fong , Wei-Chao Lin , Feature selection and ensemble learning techniques in one-class classifiers: an empirical study of two-class imbalanced datasets, IEEE Access 9 (2021), 13717–13726.

75.

Ul Hassan

, Ali

R.H.

, Ul Abideen

, Khan

T.A.

and Kouatly

, Significance of machine learning for detection of malicious websites on an unbalanced dataset, Digital 2(4) (2022), 501–519.

76.

Viegas Felipe

et al., A genetic programming approach for feature selection in highly dimensional skewed data, Neurocomputing 273 (2018), 554–569.

77.

Wang

, Jia

, Xu

, Wang

, Zhu

and Li

, Sample and feature selecting based ensemble learning for imbalanced problems, Applied Soft Computing 113 (2021), 107884.

78.

Wasikowski Mike , Xue-wen Chen , Combating the small sample class imbalance problem using feature selection, IEEE Transactions on Knowledge and Data Engineering 22.10 (2009), 1388–1400.

79.

Yang Pengyi,

et al. Ensemble-based wrapper methods for feature selection and class imbalance learning, Advances in Knowledge Discovery and Data Mining: 17th Pacific-Asia Conference, PAKDD 2013, Gold Coast, Australia, April 14-17, 2013, Proceedings, Part I 17. Springer Berlin Heidelberg, 2013.

80.

Yin Hua , Keke Gai An empirical study on preprocessing high-dimensional class-imbalanced data for classification, 2015 IEEE 17th international conference on high performance computing and communications, 2015 IEEE 7th international symposium on cyberspace safety and security, and 2015 IEEE 12th international conference on embedded software and systems. IEEE, 2015.

81.

Yin Liuzhi

et al., Feature selection for high-dimensional imbalanced data, Neurocomputing 105 (2013), 3–11.

82.

Zhang Chongsheng

et al., An empirical study on the joint impact of feature selection and data resampling on imbalance classification, Applied Intelligence 53.5 (2023), 5449–5461.

83.

Zhang Chunkai

et al. Feature selection for high dimensional imbalanced class data based on F-measure optimization, 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC). IEEE, 2017.

84.

Zhang Chunkai

et al., Research on classification method of high-dimensional class-imbalanced datasets based on SVM, International Journal of Machine Learning and Cybernetics 10 (2019), 1765–1778.

85.

Zhang Yong

et al., Clustering-guided particle swarm feature selection algorithm for high-dimensional imbalanced data with missing values, IEEE Transactions on Evolutionary Computation 26.4 (2021), 616–630.

86.

Zhou Hong

et al., A hybrid feature selection method RFSTL for manufacturing quality prediction based on a high dimensional imbalanced dataset, IEEE Access 9 (2021), 29719–29735.