Variable selection is crucial for improving interpretation quality and forecasting accuracy. To this end, it is very interesting to choose an effective dimension reduction technique suitable for processing data according to their specificity and characteristics. In this paper, the problem of variable selection for linear and nonlinear regression is deeply investigated. The curse of dimensionality issue is also addressed. An intensive comparative study is performed between Support Vector Regression(SVR) and Random Forests (RF) for the purpose of variable importance assessment then for variable selection. The main contribution of this work is twofold: to expose some experimental insights about the efficiency of variable ranking and selection based on SVR and on RF, and to provide a benchmark study that helps researchers to choose the appropriate method for their data. Experiments on simulated and real-world datasets have been carried out. Results show that the SVR score ∂ Gα is recommended for variable ranking in linear situations whereas the RF score is preferable in nonlinear cases. Moreover, we found that RF models are more efficient for selecting variables especially when used with an external score of importance.
AmaldiE. and KannV., On the approximability of minimizing non zero variables or unsatisfied relations in linear systems, Theoretical Computer Science209(1-2) (1998), 237-260.
2.
AmbroiseC. and McLachlanG.J., Selection bias in gene extraction on the basis of microarray gene-expression data, National Academy of Sciences99(10) (2002), 6562-6566.
3.
AndersenC.M. and BroR., Variable selection in regression - a tutorial, Journal of Chemometrics24 (2010), 728-737.
4.
BlakeC. and MerzC.,UCI Repository of Machine Learning Databases, http://www.ics.uci.edu/\symbol{126}mlearn/MLRepository. html, University of California, Irvine, Department of Information and Computer Sciences, 1998.
5.
BlumA. and LangleyP., Selection of relevant features and examples in machine learning, Artificial Intelligence97(1-2) (1997), 245-271.
6.
BoserB.E., GuyonI. and VapnikV.N., A training algorithm for optimal margin classifiers, in: Proceedings of the Fifth Annual Workshop on Computational Learning Theory COLT '92, New York, NY, USA: ACM Press, Pittsburgh, (1992), 144-152.
7.
BreimanL., FriedmanJ.H., OlshenR.A. and StoneC.J., Classification and Regression Trees, Wadsworth and Brooks, 1984.
BreimanL., Random forests, Machine Learning45 (2001), 5-32.
10.
BühlmannP., Boosting for high-dimensional linear models, Ann Statist34 (2006), 559-583.
11.
BurgesC., A tutorial on support vector machines for pattern recognition, in: Data Mining and Knowledge Discovery, Kluwer Academic Publishers, Boston, Vol. 2, 1998.
12.
ChangM.W. and LinC.J., Leave-one-out bounds for support vector regression model selection, Neural Computation17(4) (2005), 1-26.
13.
ChapelleO., VapnikV.N., BousquetO. and MukherjeeS., Choosing multiple parameters for support vector machines, Machine Learning46(1-3) (2002), 131-159.
14.
Dataset provided by Prof. Marc Meurens, Université catholique de Louvain, BNUT, meurens@bnut.ucl.ac.be. Dataset available from www.ucl.ac.be/mlg/.
15.
Díaz-UriarteR. and Alvarez de AndrésS., Gene selection and classification of microarray data using random forest, BMC Bioinformatics7(3) (2006).
16.
DonohoD.L., High-dimensional data analysis: The curses and blessings of dimensionality, in: Aide-Memoire of a Lecture at AMS Conference on Math Challenges of the 21st Century, (2000).
17.
DouhaL., BenoudjitN. and MelganiF., A robust regression approach for spectrophotometric signal analysis, Journal of Chemometrics26 (2012), 400-405.
18.
EfronB., HastieT., JohnstoneI. and TibshiraniR., Least angle regression (with discussion), Ann Statist32 (2004), 407-499.
19.
FanJ. and LiR., Statistical challenges with high dimensionality: Feature selection in knowledge discovery, in: Proceedings of the International Congress of Mathematicians, Sanz-SoleM., SoriaJ., VaronaJ.L. and VerderaJ., eds, Vol. III, 2006, pp. 595-622.
20.
FekiA., Ben IshakA. and FekiS., Feature selection using bayesian and multiclass support vector machines approaches: Application to bank risk prediction, Expert Systems with Applications39(3) (2012), 3087-3099.
21.
FormanG., An extensive empirical study of feature selection metrics for text classification, JMLR3 (2003), 1289-1306.
22.
GenuerR., PoggiJ.M. and TuleauC., Variable selection using random forests, Pattern Recognit Lett31(14) (2010), 2225-2236.
23.
GhattasB. and Ben IshakA., Sélection de variables pour la classification binaire en grande dimension: Comparaisons et application aux données de biopuces, Journal de la Société Française de Statistiques149(3) (2008), 43-66.
24.
GrömpingU., Variable importance assessment in regression: Linear regression versus random forest, The American Statistical Association63(4) (2009), 308-319.
25.
GunnS.R., BrownM. and BossleyK.M., Network performance assessment for neuro-fuzzy data modelling, in: Intelligent Data Analysis, LiuX., CohenP. and BertholdM., eds, volume 1208 of Lecture Notes in Computer Science, 1997, pp. 313-323.
26.
GuyonI., WestonJ., BarnhillS. and VapnikV.N., Gene selection for cancer classification using support vector machines, Machine Learning46(1-3) (2002), 389-422.
27.
GuyonI. and ElisseeffA., An introduction to variable and feature selection, JMLR3 (2003), 1157-1182.
28.
GuyonI., LiJ., MaderT., PletscherP.A., SchneiderG. and UhrM., Competitive baseline methods set new standards for the NIPS 2003 feature selection benchmark, Pattern Recognit Lett28(12) (2007), 1438-1444.
29.
HastieT., TibshiraniR. and FriedmanJ., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer-Verlag (2nd edition), New York, 2009.
30.
HillS.M., NeveR.M., BayaniN., KuoW.L., ZiyadiS., SpellmanP.T., GrayJ.W. and MukherjeeS., Integrating biological knowledge into variable selection: An empirical Bayes approach with an application in cancer biology, Bioinformatics28(18) (2012), 2342-2348.
31.
HirstJ.D., KingR.D. and SternbergM.J.E., Quantitative structure-activity relationships by neural networks and inductive logic programming: I. The inhibition of dihydrofolate reductase by pyrimidines, Journal of Computer-Aided Molecular Design8 (1994), 405-420.
32.
HirstJ.D., KingR.D. and SternbergM.J.E., Quantitative structure-activity relationships by neural networks and inductive logic programming: II. The inhibition of dihydrofolate reductase by triazines, Journal of Computer-Aided Molecular Design8 (1994), 421-432.
33.
IshwaranH., Variable importance in binary regression trees and forests, Electronic Journal of Statistics1 (2007), 519-537.
34.
KingR.D., HirstJ.D. and SternbergM.J.E., A comparison of artificial intelligence methods for modelling QSARs, Applied Artificial Intelligence9(2) (1995), 213-233.
35.
KohaviR. and JohnG., Wrappers for feature selection, Artificial Intelligence97(1-2) (1997), 273-324.
36.
RakotomamonjyA., Variable selection using SVM-based criteria, JMLR3 (2003), 1357-1370.
37.
RakotomamonjyA., Analysis of SVM regression bounds for variable ranking, Neurocomputing70(7-9) (2007), 1489-1501.
38.
Shawe-TaylorJ. and CristianiniN., Kernel Methods for Pattern Analysis, Cambridge University Press, Cambridge, UK, 2004.
39.
SmolaA.J. and ScholkopfB., A tutorial on support vector regression. NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London, UK, 1998.
40.
StroblC., BoulesteixA.-L., KneibT., AugustinT. and ZeileisA., Conditional variable importance for random forests, Bioinformatics9(307) (2008).
41.
TibshiraniR., Regression shrinkage and selection via the LASSO, J Roy Statist Soc Ser B58 (2004), 267-288.
42.
VapnikV.N., The Nature of Statistical Learning Theory, Springer Verlag, New York, 1995.
43.
VapnikV.N., GolowichS. and SmolaA., Support vector method for function approximation regression estimation and signal processing, in: Advances in Neural Information Processing Systems Pages, MozerM., JordanM. and PetscheT., eds, Cambridge, MA, MIT Press, 1997.
44.
VapnikV.N., Statistical Learning Theory, Wiley, New York, 1998.
45.
VapnikV.N. and ChapelleO., Bounds on error expectation for support vector machines, Neural Computation12(9) (2000), 2013-2036.
46.
ZengX.-Q. and LiG.-Z., Incremental partial least squares analysis of big streaming data, Pattern Recognit47(11) (2014), 3726-3735.
47.
YangC.C. and ShiehM.D., A support vector regression based prediction model of affective responses for product form design, Computers & Industrial Engineering59(4) (2010), 682-689.
