An empirical method (PRESS) for examining and contrasting the cross-validated prediction accuracies of some popular algorithms for weighting predictor variables was advanced and examined. The weighting methods that were considered were ordinary least squares, ridge regression, regression on principal components, and regression on an equally weighted composite. PRESS was executed on several data sets having varied characteristics, with each of the weighting techniques obtaining the greatest accuracy under some conditions. The degree of advantage or disadvantage offered by these alternate weighting algorithms relative to ordinary least squares was considered. As it was not possible to determine a priori which weighting technique would be most accurate for a particular data set from theoretical knowledge or from simple sample data characteristics, the sample specific PRESS method was proffered as possibly most appropriate when the researcher wishes to select from among the several alternate predictor weighting algorithms in order to achieve maximum cross-validated prediction accuracy.
Get full access to this article
View all access options for this article.
References
1.
Allen, D. M. (1971). The prediction sum of squares as a criterion for selecting predictor variables (Tech. Rep. No. 23). University of Kentucky, Department of Statistics.
2.
Allen, D. M. (1972). Biased prediction using multiple linear regression (Tech. Rep. No. 36). University of Kentucky, Department of Statistics.
3.
Allen, D. M. and Cady, F. B. (1982). Analyzing experimental data by regression. Belmont, CA: Wadsworth.
4.
Bartlett, M. S. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3, 77-85.
5.
Bartlett, M. S. (1951). An inverse matrix adjustment arising in discriminant analysis. Annals of Mathematical Statistics, 22, 107-111.
6.
Burkett, G. R. (1964). A study of reduced rank models for multiple prediction. Psychometric Monographs, No 12.
7.
Cattin, P. (1981). The predictive power of ridge regression: Some quasi-simulation results. Journal of Applied Psychology, 66, 282-290.
8.
Claudy, J. G. (1972). A comparison of five variable weighting procedures. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 32, 311-322.
9.
Darlington, R. B. (1978). Reduced-variance regression. Psychological Bulletin, 85, 1238-1255.
10.
Dawes, R. M. and Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.
11.
Dempster, A. P. , Schatzoff, M., and Wermuth, N. (1977). A simulation study of alternatives to ordinary least squares. Journal of the American Statistical Association, 72, 77-91.
12.
Dorans, N. and Drasgow, F. (1978). Alternate weighting schemes for linear prediction. Organizational Behavior and Human Performance, 21, 316-345.
13.
Egerton, M. F. and Laycock, P. J. (1981). Some criticisms of stochastic shrinkage and ridge regression, with counterexamples. Technometrics, 23, 155-159.
14.
Einhorn, H. J. and Hogarth, R. M. (1975). Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13, 171-192.
15.
Gabriel, R. M. (1980, September). Using composite variables in multivariate analysis: Why weight? Paper presented at the meeting of the American Psychological Association, Montreal, Quebec, Canada.
16.
Gibbons, D. G. (1981). A simulation study of some ridge estimators. Journal of the American Statistical Association, 76, 131-139.
17.
Gollob, H. F. (1967). Cross-validation using samples of size one. Paper presented at the meeting of the American Psychological Association, Washington, D.C.
18.
Hocking, R. R. (1983). Developments in linear regression methodology: 1959-1982. Technometrics, 25, 219-230.
19.
Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Applications to nonorthogonal problems. Technometrics, 12, 69-82.
20.
Horst, P. (1963). Matrix algebra for social scientists. New York: Holt, Rinehart, and Winston.
21.
Huberty, C. J. (1984). Issues in the use and interpretation of discriminant analysis. Psychological Bulletin, 95, 156-171.
22.
Huberty, C. J. and Mourad, S. A. (1980). Estimation in multiple correlation/prediction. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 40, 101-112.
23.
Kaiser, H. F. (1960). The application of electronic computers to factor analysis. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 20, 141-151.
24.
Lachenbruch, P. A. and Mickey, M. R. (1968). Estimation of error rates in discriminant analysis. Technometrics, 10, 1-11.
25.
Laughlin, J. E. (1978). Comment on "Estimating coefficients in linear models: It don't make no nevermind."Psychological Bulletin, 85, 247-253.
26.
Lawless, J. F. and Wang, P. (1976). A simulation study of ridge and other regression estimators. Communications in Statistics—Theory and Methods, A5(4), 307-323.
27.
Lawshe, C. H. and Schucker, R. E. (1959). The relative efficiency of four test weighting methods in multiple prediction. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 19, 103-114.
28.
Morris, J. D. (1979). A comparison of regression prediction accuracy on several types of factor scores. American Educational Research Journal, 16, 17-24.
29.
Morris, J. D. (1981, April). An extension to "A comparison of regression accuracy on several types of factor scores." Paper presented at the meeting of the American Educational Research Association, Los Angeles, CA.
30.
Morris, J. D. (1982). Ridge regression and some alternate weighting techniques: A comment on Darlington. Psychological Bulletin, 91, 203-210.
31.
Morris, J. D. (1983). Stepwise ridge regression: A computational clarification. Psychological Bulletin, 94, 363-366.
32.
Morris, J. D. (1984). Cross-validation with Gollob's estimator: A computational simplification. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 44, 151-154.
33.
Morris, J. D. and Guertin, W. H. (1977). The superiority of factor scores as predictors. Journal of Experimental Education, 45, 41-44.
34.
Mosteller, F. and Tukey, J. W. (1968). Data analysis, including statistics. In G. Lindzey & E. Aronson (Eds.), Handbook of Social Psychology: Vol. 2. (pp. 80-203). Reading MA: Addison-Wesley.
35.
Musgrage, B.Marquette, J. and Newman, I. (1982). On using the average intercorrelation among predictor variables and eigenvalue orientation to choose a regression solution. Multiple Linear Regression Viewpoints, 11, 1-21.
36.
Pagel, M. D. and Lunneborg, C. E. (1985). Empirical evaluation of ridge regression. Psychological Bulletin, 97, 342-355.
37.
Pruzek, R. M. and Frederick, B. C. (1978). Weighting predictors in linear models: Alternatives to least squares and limitations of equal weights. Psychological Bulletin, 85, 254-266.
38.
Rozeboom, W. W. (1979). Ridge regression: Bonanza or beguilement?Psychological Bulletin, 86, 242-249.
39.
Schmidt, F. L. (1971). The relative efficiency of regression and simple unit predictor weights in applied differential psychology. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 31, 699-714.
40.
Smith, G. and Campbell, F. (1980). A critique of some ridge regression methods. Journal of American Statistical Association, 75, 74-81.
41.
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36, 111-133.
42.
Thisted, R. A. and Morris, C. N. (1980). Theoretical results for adaptive ordinary ridge regression estimators (Tech. Rep. No. 94). Chicago: University of Chicago.
43.
Trattner, M. H. (1963). Comparison of three methods for assembling aptitude test batteries. Personnel Psychology, 16, 221-232.
44.
Wainer, H. (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin, 83, 213-217.
45.
Wainer, H. (1978). On the sensitivity of regression and regressors. Psychological Bulletin, 1978, 85, 267-273.
46.
Wesman, A. G. and Bennett, G. K. (1959). Multiple regression vs. simple addition of scores in prediction of college grades. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 19, 243-246.
