Estimation of the Squared Cross-Validity Coefficient in the Context of Best Subset Regression

Abstract

A monte carlo study was conducted to examine the performance of several strategies for estimating the squared cross-validity coefficient of a sample regres sion equation in the context of best subset regression. Data were simulated for populations and experimental designs likely to be encountered in practice. The re sults indicated that a formula presented by Stein (1960) could be expected to yield estimates as good as or better than cross-validation, or several other for mula estimators, for the populations considered. Fur ther, the results suggest that sample size may play a much greater role in validity estimation in subset se lection than is true in situations where selection has not occurred. Index terms: Best subset regression, Cross-validity coefficient, Multiple regression, Predic tive validity, Variable selection.

Get full access to this article

View all access options for this article.

References

Browne, M. (1975). Predictive validity of a linear regression equation . British Journal of Mathematical and Statistical Psychology , 28, 79-87.

Cattin, P. (1980). Estimation of the predictive power of a regression model. Journal of Applied Psychology, 65, 407-414.

Claudy, J.G. (1978). Multiple regression and validity estimation in one sample. Applied Psychological Measurement, 2, 595-607.

Cohen, J. , & Cohen, P. (1975). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale NJ: Erlbaum.

Darlington, R.B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69, 161-182.

Diehr, G. , & Hoflin, D.R. (1974). Approximating the distribution of the sample R2 in best subset regressions. Technometrics, 16, 317-320.

Drasgow, F. , Dorans, N.J. , & Tucker, L.R. (1979). Estimators of the squared cross-validity coefficient: A monte carlo investigation. Applied Psychological Measurement , 3, 387-399.

Herzberg, P. (1969). The parameters of cross-validation. Psychometrika Monograph, 34 (2, Pt. 2, No. 16).

Hockings, R.R. (1976). The analysis and selection of variables in linear regression. Biometrics, 32, 1-49.

10.

Klienbaum, D.G. , & Kuper, L.L. (1978). Applied regression analysis and other multivariable methods. North Scituate MA: Duxbury Press.

11.

Lerner, J.V. , & Games, P.A. (1981). Maximum R2 improvement and stepwise multiple regression as related to over-fitting. Psychological Reports, 48, 979-983.

12.

Lord, F.M. (1950). Efficiency of prediction when a regression equation from one sample is used in a new sample (Research Bulletin No. 50-40). Princeton NJ: Educational Testing Service.

13.

Moiser, C.I. (1951). Problems and designs of cross-validation. Educational and Psychological Measurement, 11, 5-11.

14.

Pedhazur, E.J. (1982). Multiple regression in behavioral research: Explanation and prediction. New York: CBS College Publishing.

15.

Picard, R.R. , & Cook, R.D. (1984). Cross-validation of regression models. Journal of the American Statistical Association, 79, 575-583.

16.

Pope, P.T. , & Webster, J.T. (1972). The use of an F-statistic in stepwise regression procedures. Technometrics, 14, 327-340.

17.

Rencher, A.C. , & Pun, F.C. (1980). Inflation of R2 in best subset regression. Technometrics, 22, 49-53. Rozeboom, W.W. (1978). Estimation of cross-validated multiple correlations: A clarification. Psychological Bulletin, 85, 1348-1351.

18.

Schmitt, N. , Coyle, B.W. , & Rauschenberger, J. (1977). A Monte Carlo evaluation of three formula estimators of cross-validated multiple correlation. Psychological Bulletin, 84, 751-758.

19.

Stein, C. (1960). Multiple regression. In I. Olkin et al. (Eds.), Contributions to probability and statistics. Stanford CA: Stanford University Press.

20.

Wherry, R.J. (1931). A new formula for predicting the shrinkage of the coefficient of multiple correlation. Annals of Mathematical Statistics, 2, 440-457.

21.

Wherry, R.J. (1975). Underprediction from overfitting: 45 years of shrinkage. Personnel Psychology, 28, 1-18.