Variable Selection in Linear Regression

Abstract

We present a new Stata program, vselect, that helps users perform variable selection after performing a linear regression. Options for stepwise methods such as forward selection and backward elimination are provided. The user may specify Mallows's Cp, Akaike's information criterion, Akaike's corrected information criterion, Bayesian information criterion, or R² adjusted as the information criterion for the selection. When the user specifies the best subset option, the leaps-and-bounds algorithm (Furnival and Wilson, Technometrics 16: 499–511) is used to determine the best subsets of each predictor size. All the previously mentioned information criteria are reported for each of these subsets. We also provide options for doing variable selection only on certain predictors (as in [R] nestreg) and support for weighted linear regression. All options are demonstrated on real datasets with varying numbers of predictors.

Keywords

st0213 vselect variable selection regress nestreg

References

Akaike

1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control 19: 716–723.

Efron

, Hastie

, Johnstone

, and Tibshirani

2004. Least angle regression. Annals of Statistics 32: 407–499.

Frank

, and Asuncion

2010. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml/datasets/Housing.

Furnival

G. M.

, and Wilson

R. W.

1974. Regression by leaps and bounds. Technometrics 16: 499–511.

Hastie

, Tibshirani

R. J.

, and Friedman

2001. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer.

Hocking

R. R.

1976. A Biometrics invited paper: The analysis and selection of variables in linear regression. Biometrics 32: 1–49.

Hurvich

C. M.

, and Tsai

C.-H.

1989. Regression and time series model selection in small samples. Biometrika 76: 297–307.

Izenman

A. J.

2008. Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. New York: Springer.

Lawless

J. F.

, and Singhal

1978. Efficient screening of nonnormal regression models. Biometrics 34: 318–327.

10.

Mallows

C. L.

1973. Some comments on Cp. Technometrics 15: 661–675.

11.

, and Huo

2005. Enhanced leaps-and-bounds method in subset selections with additional optimality tests. http://www3.informs.org/site/qsr/downloadfile.php?i=17e62166fc8586dfa4d1bc0e1742c08b.

12.

Raftery

A. E.

1995. Bayesian model selection in social research. Sociological Methodology 25: 111–163.

13.

Schwarz

1978. Estimating the dimension of a model. Annals of Statistics 6: 461–464.

14.

Sheather

S. J.

2009. A Modern Approach to Regression with R. New York: Springer.

15.

Simonoff

J. S.

2003. Analyzing Categorical Data. New York: Springer.

16.

Tryfos

1998. Methods for Business Analysis and Forecasting: Text and Cases. New York: Wiley.