We consider the problem of estimation and variable selection for general linear regression models. Regularized regression procedures have been widely used for variable selection, but most existing methods perform poorly in the presence of outliers. We construct a new penalized procedure that simultaneously attains full efficiency and maximum robustness. Furthermore, the proposed procedure satisfies the oracle properties. The new procedure is designed to achieve sparse and robust solutions by imposing adaptive weights on both the decision loss and the penalty function. The proposed method of estimation and variable selection attains full efficiency when the model is correct and, at the same time, achieves maximum robustness when outliers are present. We examine the robustness properties using the finite-sample breakdown point and an influence function. We show that the proposed estimator attains the maximum breakdown point. Furthermore, there is no loss in efficiency when there are no outliers or the error distribution is normal. For practical implementation of the proposed method, we present a computational algorithm. We examine the finite-sample and robustness properties using Monte Carlo studies. Two datasets are also analyzed.
FrankIFriedmanJ. A statistical view of some chemometrics regression tools. Technometrics1993; 35: 109–135.
2.
TibshiraniR. Regression shrinkage and selection via the lasso. J Royal Stat Soc Ser B1996; 58: 267–288.
3.
FanJLiR. Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc2001; 96: 1348–1360.
4.
ZouH. The adaptive lasso and its oracle properties. J Am Stat Assoc2006; 101: 1418–1429.
5.
ZouHHastieT. Regularization and variable selection via the elastic net. J Royal Stat Soc Ser B2005; 67: 301–320.
6.
ZouHZhangHH. On the adaptive elastic-net with a diverging number of parameters. Ann Stat2009; 37: 1733–1751.
7.
ZhangC. Nearly unbiased variable selection under minimax concave penalty. Ann Stat2010; 38: 894–942.
8.
HuberPJ. Robust statistics, New York, NY: Wiley, 1981.
9.
WangHLiGJiangG. Robust regression shrinkage and consistent variable selection through the LAD-Lasso. J Business Econ Stat2007; 25: 347–355.
10.
LiGPengHZhuL. Nonconcave penalized m-estimation with a diverging number of parameters. Statistica Sinica2011; 21: 391–419.
11.
Lambert-LacroixSZwaldL. Robust regression through the Huber’s criterion and adaptive lasso penalty. Electronic J Stat2011; 5: 1015–1053.
12.
ArslanO. Weighted LAD-LASSO method for robust parameter estimation and variable selection in regression. Computat Stat Data Analys2012; 56: 1952–1965.
13.
WuYLiuY. Variable selection in quantile regression. Stat Sinica2009; 19: 801–817.
14.
WangLWuYLiR. Quantile regression for analyzing heterogeneity in ultra-high dimension. J Am Stat Assoc2012; 107: 214–222.
15.
KaiBLiRZouH. New efficient estimation and variable selection methods for semiparametric varying-coefficient partially linear models. Ann Stat2011; 39: 305–332.
16.
ZouHYuanM. Composite quantile regression and the oracle model selection theory. Ann Stat2008; 36: 1108–1126.
LengC. Variable selection and coefficient estimation via regularized rank regression. Stat Sinica2010; 20: 167–181.
20.
ChenXWangJMcKeownA. Asymptotic analysis of robust LASSOs in the presence of noise with large variance. IEEE Transact Inform Theory2010; 56: 5131–5149.
21.
BradicJFanJWangW. Penalized composite quasi-likelihood for ultrahigh dimensional variable selection. J Royal Stat Soc Ser B2011; 73: 325–349.
22.
WangXJiangYHuangMet al.Robust variable selection with exponential squared loss. J Am Stat Assoc2013; 108: 632–643.
23.
FanJFanYBarutE. Adaptive robust variable selection. Ann Stat2014; 42: 324–351.
24.
AlfonsACrouxCGelperS. Sparse least trimmed squares regression for analyzing high-dimensional large data sets. Ann Appl Stat2013; 7: 226–248.
25.
ÖllererVCrouxCAlfonsA. The influence function of penalized regression estimators. Statistics2015; 49: 741–765.
26.
SmuclerEYohaiVJ. Robust and sparse estimators for linear regression model. Computat Stat Data Analys2017; 111: 116–130.
27.
YohaiV. High breakdown-point and high efficiency robust estimates for regression. Ann Stat1987; 15: 642–656.
28.
LohP-L. Statistical consistency and asymptotic normality for high-dimensional robust M-estimators. Ann Stat2017; 45: 866–896.
29.
HampleF. The influence curve and its role in robust estimation. J Am Stat Assoc1974; 69: 383–393.
30.
HuberPJ. Final sample breakdown of M- and P-estimators. Ann Stat1984; 12: 119–126.
31.
MaronnaRAMartinDRYohaiVJ. Robust statistics: theory and methods, New York, NY: Wiley, 2006.
32.
GerviniDYohaiV. A class of robust and fully efficient regression estimators. Ann Stat2002; 30: 583–616.
33.
MaronnaRAYohaiVJ. Robust and efficient estimation of multivariate scatter and location. Computat Stat Data Analys2017; 109: 64–75.
34.
ChristensenRSunSK. Alternative goodness-of-fit tests for linear models. J Am Stat Assoc2010; 105: 291–301.
35.
Le CamL. Théorie Asymptotique de la Décision Statistique, Les Presses de l’Université de Montréal, 1969.
36.
BickelPJKlaassenCRitovYet al.Efficient and adaptive estimation for semiparametric models, New York, NY: Springer, 1998.
37.
van der VaartAW. Asymptotic statistics, Cambridge, UK: Cambridge University Press, 2000.
38.
ZouHLiR. One-step sparse estimates in nonconcave penalized likelihood models. Ann Stat2008; 36: 1509–1533.
39.
EfronBHastieTJohnstoneIet al.Least angle regression. Ann Stat2004; 32: 407–499.
40.
FriedmanJHastieTHöflingHet al.Pathwise coordinate optimization. Ann Appl Stat2007; 1: 302–332.
41.
BoydSParikhNChuEet al.Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations Trends Mach Learn2011; 3: 1–122.
42.
ArnoldTBTibshiraniRJ. Efficient implementations of the generalized lasso dual path algorithm. J Computat Graphic Stat2016; 25: 1–27.
43.
ZhuY. An augmented ADMM algorithm with application to the generalized lasso problem. J Computat Graph Stat2017; 26: 195–204.
44.
HampelFRRonchettiEMRousseeuwPJet al.Robust statistics: the approach based on influence functions, New York, NY: Wiley, 1986.
45.
HampelF. A global qualitative definition of robustness. Ann Math Stat1971; 42: 1887–1895.
46.
DonohoDHuberPThe notion of breakdown point. In: BickelPJDoksumKAHodgesJL, Jr. (eds). A Festschrift for E. L. Lehmann, Belmont, CA: Wadsworth, 1983, pp. 157–184.
47.
RousseeuwPJYohaiV. Robust regression by means of S-estimators. In: Robust and nonlinear time series analysis. Lecture Notes in Statistics, New York, NY: Springer, 1984, pp. 256–272.
48.
GerviniD. A robust and efficient adaptive reweighted estimator of multivariate location and scatter. J Multivariate Analys2003; 84: 116–144.
49.
RousseeuwPJLeroyA. Robust regression and outlier detection, New York, NY: Wiley, 1987.
50.
MartinRDYohaiVJZamarRH. Min-max bias regression. Ann Stat1989; 17: 1608–1630.
51.
YohaiVJZamarRH. A minimax-bias property of the least-quantile estimates. Ann Stat1993; 21: 1824–1842.
52.
RousseeuwPJDriessenKV. A fast algorithm for the minimum covariance determinant estimator. Technometrics1999; 41: 212–223.
53.
SheYOwenAB. Outlier detection using nonconvex penalized regression. J Am Stat Assoc2011; 106: 626–639.
54.
Wei L. Simultaneous variable selection and outlier detection using LASSO with applications to aircraft landing data analysis. PhD Thesis: Rutgers University, NJ, 2012.
55.
LindenDE. The challenges and promise of neuroimaging in psychiatry. Neuron2012; 73: 8–22.
56.
NestlerEJHymanSE. Animal models of neuropsychiatric disorders. Nat Neurosci2010; 13: 1161–1169.
57.
WangLZangYHeYet al.Changes in hippocampal connectivity in the early stages of Alzheimer’s disease: evidence from resting state fMRI. Neuroimage2006; 31: 496–504.
58.
GreiciusMDFloresBHMenonVet al.Resting-state functional connectivity in major depression: abnormally increased contributions from subgenual cingulate cortex and thalamus. Biol Psychiatr2007; 62: 429–437.
59.
RomboutsSADamoiseauxJSGoekoopRet al.Model free group analysis shows altered BOLD FMRI networks in dementia. Human Brain Map2009; 30: 256–266.
60.
ADHD-200Consortium. The ADHD-200 consortium: a model to advance the translationa l potential of neuroimaging in clinical neuroscience. Frontiers Syst Neurosci2012; 6: 62–62.
61.
BellecPChuCChouinard-DecorteFet al.The Neuro bureau ADHD-200 preprocessed repository. Neuroimage2017; 144: 275–286.
62.
Tzourio-MazoyerNLandeauBPapathanassiouDet al.Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage2002; 15: 273–289.
63.
TianLJiangTWangYet al.Altered resting-state functional connectivity patterns of anterior cingulate cortex in adolescents with attention deficit hyperactivity disorder. Neurosci Lett2006; 400: 39–43.
64.
CastellanosFXMarguliesDSKellyCet al.Cingulate-precuneus interactions: a new locus of dysfunction in adult attention-deficit/hyperactivity disorder. Biol Psychiatr2008; 63: 332–337.
65.
McDonaldGCSchwingRC. Instabilities of regression estimates relating air pollution to mortality. Technometrics1973; 15: 463–482.
66.
McCullaghPNelderJA. Generalized linear models., 2nd ed. London: Chapman and Hall, 1989.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
