Stepwise methods are frequently employed in educational and psychological research, both to select useful subsets of variables and to evaluate the order of importance of variables. Three problems with stepwise applications are explored in some detail. First, computer packages use incorrect degrees of freedom in their stepwise computations, resulting in artifactually greater likelihood of obtaining spurious statistical significance. Second, stepwise methods do not correctly identify the best variable set of a given size, as illustrated by a concrete heuristic example. Third, stepwise methods tend to capitalize on sampling error and thus tend to yield results that are not replicable.
Get full access to this article
View all access options for this article.
References
1.
Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
2.
Cliff, N. (1987). Analyzing multivariate data. San Diego, CA: Harcourt Brace Jovanovich.
3.
Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443.
4.
Cohen, J. (1994). The earth is round (p < .05>). American Psychologist, 49, 997-1003.
5.
Craeger, J. (1969). The interpretation of multiple regression via overlapping rings. American Educational Research Journal, 6, 706-709.
6.
Huberty, C. (1989). Problems with stepwise methods-Better alternatives. In B. Thompson (Ed.), Advances in social science methodology (Vol. 1, pp. 43-70). Greenwich, CT: JAI.
7.
Huberty, C. (1994). Applied discriminant analysis. New York: Wiley.
8.
Hudson, W. D. (1969). The is/ought question. London: Macmillan.
9.
Hume, D. (1957). An inquiry concerning human understanding. New York: Liberal Arts Press.
10.
Knapp, T. R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416.
11.
McCabe, G. P. (1975). Computations for variable selection in discriminant analysis. Technometrics, 17, 103-109.
12.
Snyder, P. (1991). Three reasons why stepwise regression methods should not be used by researchers. In B. Thompson (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 99-105). Greenwich, CT: JAI.
13.
Snyder, P. , & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
14.
Strike, K. A. (1979). An epistemology of practical research. Educational Researcher, 8(1), 10-16.
15.
Thompson, B. (1989). Why won't stepwise methods die?Measurement and Evaluation in Counseling and Development, 21(4), 146-148.
16.
Thompson, B. (1990). Finding a correction for the sampling error in multivariate measures of relationship: A Monte Carlo study. Educational and Psychological Measurement, 50, 15-31.
17.
Thompson, B. (1991). A primer on the logic and use of canonical correlation analysis. Measurement and Evaluation in Counseling and Development, 24(2), 80-95.
18.
Thompson, B. (Ed.). (1993). Special issue on statistical significance testing, with comments from various journal editors. Journal of Experimental Education, 61(4).
19.
Thompson, B. (1994a). The concept of statistical significance testing (an ERIC/AE Clearing-house Digest #EDO-TM-94-1). Measurement Update, 4(1), 5-6. (ERIC Document Reproduction Service No. ED 366 654)
20.
Thompson, B. (1994b). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
21.
Thompson, B. (1994c). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62(2), 157-176.
22.
Thompson, B. , & Borrello, G. M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, 203-209.
