Conventional statistical significance tests do not inform the researcher regarding the likelihood that results will replicate. One strategy for evaluating result replicability is to employ a "bootstrap" re sampling of a study's data so that the stability of results across numerous configurations of the subjects can be explored. This article illustrates the use of the bootstrap in a canonical correlation analysis.
Get full access to this article
View all access options for this article.
References
1.
Bartko, J. J. (1991). Proving the null hypothesis. American Psychologist, 46, 1089.
2.
Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
3.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304-1312.
4.
Diaconis, P. , & Efron, B. (1983). Computer-intensive methods in statistics. Scientific American, 248(5), 116-130.
5.
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1-26.
6.
Fan, X. (1992, April). Canonical correlation analysis as a general data-analytic model. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. ED 348 383)
7.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Haase, T. , & Thompson, B. (1992, January). The homogeneity of variance assumption in ANOVA: What it is and why it is required. Paper presented at the annual meeting of the Southwest Educational Research Association, Houston.
9.
Hays, W. L. (1981). Statistics (3rd ed.). New York: Holt, Rinehart & Winston.
10.
Holzinger, K. J. , & Swineford, F (1939). A study in factor analysis: The stability of a bi-polar solution (No. 48). Chicago: University of Chicago.
11.
Huberty, C. J. (1987). On statistical testing. Educational Researcher, 16(8), 4-9.
12.
Jöreskog, K. G. , & Sorbom, D. (1989). LISREL 7: A guide to the program and applications (2nd ed.). Chicago: SPSS.
13.
Kaiser, H. F (1976). Review of Factor analysis as a statistical method. Educational and Psychological Measurement, 36, 586-589.
14.
Knapp, T. R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416.
15.
Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43, 635-642.
16.
Lunneborg, C. E. (1990). Review of Computer intensive methods for testing hypotheses. Educational and Psychological Measurement, 50, 441-445.
17.
McGraw, K. O. (1991). Problems with the BESD: A comment on Rosenthal's "How are we doing in soft psychology?"American Psychologist, 46, 1084-1086.
18.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
19.
Morrison, D. E. , & Henkel, R. E. (Eds.). (1970). The significance test controversy. Chicago: Aldine.
20.
Neale, J. M. , & Liebert, R. M. (1986). Science and behavior: An introduction to methods of research (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
21.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
22.
Rosenthal, R. (1991). Effect sizes: Pearson's correlation, its display via the BESD, and alternative indices. American Psychologist, 46, 1086-1087.
23.
Rosnow, R. L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
24.
Salzman, K. L. (1989). A significantly significant approach to significant research findings: The Salzman All-Significant F test. In G. C. Ellenbogen (Ed.), The primal whimper (pp. 158-162). New York: Guilford.
25.
Schneider, A. L. , & Darcy, R. E. (1984). Policy implications of using significance tests in evaluation research. Evaluation Review, 8, 573-582.
26.
Thompson, B. (1984). Canonical correlation analysis: Uses and interpretation. Beverly Hills, CA: Sage.
27.
Thompson, B. (1987, April). The use (and misuse) of statistical significance testing: Some recommendations for improved editorial policy and practice. Paper presented at the annual meeting of the American Education Research Association, Washington, DC. (ERIC Document Reproduction Service No. ED 287 868)
28.
Thompson, B. (1988a, April). Canonical correlation analysis: An explanation with comments on correct practice. Paper presented at the annual meeting of the American Educational Research Association, New Orleans. (ERIC Document Reproduction Service No. ED 295 957)
29.
Thompson, B. (1988b). Program FACSTRAP: A program that computes bootstrap estimates of factor structure. Educational and Psychological Measurement, 48, 681-686.
30.
Thompson, B. (1989a). Asking "what if" questions about significance tests. Measurement and Evaluation in Counseling and Development, 22, 66-68.
31.
Thompson, B. (1989b). Meta-analysis of factor structure studies: A case study example with Bem's androgyny measure. Journal of Experimental Education, 57, 187-197.
32.
Thompson, B. (1989c). The place of qualitative methods in contemporary social science: The importance of post-paradigmatic thought. In B. Thompson (Ed.), Advances in social science methodology (Vol. 1, pp. 1-42). Greenwich, CT: JAI.
33.
Thompson, B. (1989d). Prerotation and postrotation eigenvalues shouldn't be confused: A reminder. Measurement and Evaluation in Counseling and Development, 22(3), 114-116.
34.
Thompson, B. (1989e). Statistical significance, result importance, and result generalizability: Three noteworthy but somewhat different issues. Measurement and Evaluation in Counseling and Development, 22, 2-6.
35.
Thompson, B. (1990). Finding a correction for the sampling error in multivariate measures of relationship: A Monte Carlo study. Educational and Psychological Measurement, 50, 15-31.
36.
Thompson, B. (1991a). Invariance of multivariate results. Journal of Experimental Education, 59, 367-382.
37.
Thompson, B. (1991b). A primer on the logic and use of canonical correlation analysis. Measurement and Evaluation in Counseling and Development, 24(2), 80-95.
38.
Thompson, B. (1991c). Review of Data analysis for research designs. Educational and Psychological Measurement, 51, 500-510.
39.
Thompson, B. (1992a). DISCSTRA: A computer program that computes bootstrap resampling estimates of descriptive discriminant analysis function and structure coefficients and group centroids. Educational and Psychological Measurement, 52, 905-911.
40.
Thompson, B. (1992b). Misuse of ANCOVA and related "statistical control" procedures. Reading Psychology, 13, iii-xviii.
41.
Thompson, B. (1992c). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
42.
Thompson, B.(1993). Statistical significance testing in contemporary practice: Proposed alternatives with comments from journal editors. Special issue of theJournal of Experimental Education, 61(4).
43.
Thompson, B. (in press). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality.
44.
Thompson, B. , & Borrello, G. M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, 203-209.
