The authors review and critically evaluate three major approaches to cross-validation in the context of predictive validity: (1) sample-splitting, (2) resampling plans such as bootstrapping, and (3) a method that simultaneously estimates parameters and cross-validates. Because of the information loss involved and the availability of shrinkage formulas, they argue that data-splitting is inferior to the simultaneous approach. An empirical example illustrates how the simultaneous approach can be used in conjunction with bootstrap resampling to construct prediction intervals that are superior to classical prediction intervals.
Get full access to this article
View all access options for this article.
References
1.
AkaikeH. (1973), “Information Theory and an Extension of the Maximum Likelihood Principle,” Second International Symposium on Information Theory, PetrovB. N. and CzakiF., eds. Budapest: Akademia Kiado.
2.
BasuD. (1980), “Randomization Analysis of Experimental Data: The Fisher Randomization Test,” Journal of the American Statistical Association, 75 (September), 575–81.
3.
BreimanL. and FreedmanD. (1983), “How Many Variables Should Be Entered in a Regression Equation,” Journal of the American Statistical Association, 78 (March), 131–6.
4.
CattinPhilippe (1978), “On the Use of Formulas of the Predictive Validity of Regression in Consumer Research,” Proceedings, Association for Consumer Research, Vol. 6, 284–7.
5.
CattinPhilippe, GelfandAlan E., and DanesJeffrey (1983), “A Simple Bayesian Procedure for Estimation in a Conjoint Model,” Journal of Marketing Research, 20 (February), 29–35.
6.
CraskMelvin R. and PerreaultWilliam D.Jr. (1977), “Validation of Discriminant Analysis in Marketing Research,” Journal of Marketing Research, 14 (February), 60–8.
7.
DarlingtonRichard (1968), “Multiple Regression in Psychological Research and Practice,” Psychological Bulletin, 69 (March), 161–82.
8.
DoransNeil J. and DrasgowFritz (1980), “A Note on Cross-Validating Prediction Equations,” Journal of Applied Psychology, 65 (December), 728–9.
9.
EfronBradley (1982), “The Jackknife, the Bootstrap, and Other Resampling Plans,” SIAM Monograph #38. Philadelphia: Society for Industrial and Applied Mathematics.
10.
EfronBradley (1983), “Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation,” Journal of the American can Statistical Association, 78 (June), 316–31.
11.
EfronBradley and GongGail (1983), “A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation,” The American Statistician, 37 (February), 36–48.
12.
FenwickIan (1979), “Techniques in Market Measurement: The Jackknife,” Journal of Marketing Research, 16 (August), 410–14.
13.
FreedmanD. A. (1981), “Bootstrapping Regression Models,” Annals of Statistics, 9 (December), 1218–28.
14.
GeisserSeymour (1975), “The Predictive Sample Reuse Method with Applications,” Journal of the American Statistical Association, 70 (June), 320–8.
15.
HorstP. (1941), Prediction of Personal Adjustment.New York: Social Science Research Council, Bulletin 48.
16.
JamesLawrence R., DemareeRobert G., and MulaikStanley A. (1986), “A Note on Validity Generalization Procedures,” Journal of Applied Psychology, 71 (August), 440–50.
17.
KerlingerFred N. and PedhazurElazar J. (1973), Multiple Regression in Behavioral Research.New York: Holt, Rinehart and Winston, Inc.
18.
LachenbruchPeter A. and MickeyM. Ray (1968), “Estimation of Error Rates in Discriminant Analysis,” Technometrics, 10 (February), 1–11.
19.
LarsonSelmer C. (1931), “The Shrinkage of the Coefficient of Multiple Correlation,” Journal of Educational Psychology, 22 (January), 45–55.
20.
LordFrederick M. and NovickMelvin R. (1968), Statistical Theories of Mental Test Scores.Reading, MA: Addison-Wesley Publishing Company.
21.
McCarthyPhilip J. (1976), “The Use of Balanced Half-Sample Replication in Cross-Validation Studies,” Journal of the American Statistical Association, 71 (September), 596–604.
22.
MendenhallWilliam and SincichTerry (1986), A Second Course in Business Statistics: Regression Analysis.San Francisco: Dellen Publishing Co.
23.
MosierCharles I. (1951), “Problems and Designs of Cross-Validation,” Educational and Psychological Measurement, 11 (Spring), 5–11.
24.
MostellerFrederick and TukeyJohn W. (1977), Data Analysis and Regression.Reading, MA: Addison-Wesley Publishing Company.
25.
MurphyKevin R. (1983), “Fooling Yourself with Cross-Validation: Single Sample Designs,” Personnel Psychology, 36 (Spring), 111–18.
26.
NunnallyJum C. (1978), Psychometric Theory.New York: McGraw-Hill Book Company.
27.
PicardRichard R. and CookR. Dennis (1984), “Cross-Validation of Regression Models,” Journal of the American Statistical Association, 79 (September), 575–83.
28.
RobertsHarry V. and LingRobert E. (1982), Conversational Statistics with IDA.New York: McGraw-Hill Book Company.
SneeRonald D. (1977), “Validation of Regression Models: Methods and Examples,” Technometrics, 19 (November), 415–28.
31.
StineRobert A. (1985), “Bootstrap Prediction Intervals for Regression,” Journal of the American Statistical Association, 80 (December), 1026–31.
32.
StoneM. (1974), “Cross-Validatory Choice and Assessment of Statistical Predictions” (with discussion), Journal of the Royal Statistical Society, Series B, 36, 111–47.
33.
StoneM. (1977a), “Asymptotic For and Against Cross-Validation,” Biometrika, 64 (April), 29–35.
34.
StoneM. (1977b), “An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion,” Journal of the Royal Statistical Society, Series B, 39, 44–7.
