In spite of arguments to the contrary, psychologists, it is shown here, believe statistical significance (SS) signifies that a finding will replicate. The most visible argument that SS is not an index of replicability, one that is based in notions of Bayesian statistical inference, is considered and shown to be flawed. Two different arguments are presented that demonstrate the irrelevance of SS to replicability: (a) SS may not be taken as a sign of the truth of the research hypothesis; and (b) statistical significance tests do not generate verifiable predictions of replication attempts. Direct tests of replicability and effect-size measures of replicability are shown to have comparable problems. A solution to the replicability problem is proposed for atheoretical research that replaces `once-and-for-all' tests of replicability with the requirement that the treatment effect be demonstrable (a) in the individual, (b) on a continuing basis, and (c) in a way that is clearly discernible. The question of the role of significance testing in research is answered by denying a role to such testing.
Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
2.
Baril, G.L. , & Cannon, T. (1995). What is the probability that null hypothesis testing is meaningless?American Psychologist, 50, 1098-1099.
3.
Barlow, D.H. , & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). New York: Pergamon.
4.
Bem, D.J. , & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115, 4-18.
5.
Bergin, A.E. , & Strupp, H.H. (1970). New directions in psychotherapy research. Journal of Abnormal Psychology, 76, 13-26.
6.
Carver, R.P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
7.
Carver, R.P. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
8.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
9.
Elmes, D.G. , Kantowitz, B.H., & Roediger, H. L., III (1995). Research methods in psychology (5th ed.). St Paul, MN: West.
Evans, J.D. (1985). Invitation to psychological research. New York: Holt, Rinehart & Winston.
12.
Falk, R. (1986). Misconceptions of statistical inference. Journal of Structural Learning, 9, 83-96.
13.
Falk, R. , & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
14.
Fisher, R.A. (1925). Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society, 22, 700-725.
15.
Fisher, R.A. (1960). The design of experiments (7th ed.). New York: Hafner.
16.
Gigerenzer, G. (1993). The Superego, the Ego, and the Id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311-339). Hillsdale, NJ: Erlbaum.
17.
Gigerenzer, G. , & Murray, D.J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
18.
Goodman, S.N. (1992). A comment on replication, p-values and evidence. Statistics in Medicine, 11, 875-879.
19.
Greenwald, A.G. , Gonzalez, R., Harris, R.J., & Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated?Psychophysiology, 33, 175-183.
20.
Hagood, M.J. (1970). The notion of a hypothetical universe. In D. E. Morrison & R. E. Henkel (Eds.), The significance test controversy-A reader (pp. 65-78). Chicago, IL: Aldine.
21.
Judd, C.M. , McClelland, G.H., & Culhane, S.E. (1995). Data analysis: Continuing issues in the everyday analysis of psychological data. Annual Review of Psychology, 46, 433-465.
22.
Levin, J.R. (1993). Statistical significance testing from three perspectives. Journal of Experimental Education, 61, 378-382.
23.
Levy, P. (1967). Substantive significance of significant differences between two groups. Psychological Bulletin, 67, 37-40.
Lykken, D.T. (1991). What's wrong with psychology anyway? In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. 1. Matters of public interest (pp. 3-39). Minneapolis: University of Minnesota Press.
26.
Melton, A.W. (1962). Editorial. Journal of Experimental Psychology, 64, 553-557.
27.
Neyman, J. , Iwaszkiewicz, K., & Kolosziejczyk, S. (1935). Statistical problems in agricultural experimentation. Supplement to Journal of the Royal Statistical Society, 2, 107-154.
28.
Nunnally, J.C. (1975). Introduction to statistics for psychology and education. New York: McGraw-Hill.
29.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
30.
Phillips, L.D. (1973). Bayesian statistics for social scientists. London: Nelson.
31.
Pollard, P. , & Richardson, J. T. E. (1987). On the probability of making Type I errors. Psychological Bulletin, 102, 159-163.
32.
Reaves, C.C. (1992). Quantitative research for the behavioral sciences. New York: Wiley.
33.
Rorer, L.G. (1991). Some myths of science in psychology. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. 1. Matters of public interest (pp. 61-87). Minneapolis: University of Minnesota Press.
34.
Rosenthal, R. (1993). Cumulating evidence. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 519-559). Hillsdale, NJ: Erlbaum.
35.
Rozeboom, W.W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416-428.
36.
Shaughnessy, J.J. , & Zechmeister, E.B. (1994). Research methods in psychology (3rd ed.). New York: McGraw-Hill.
37.
Sidman, M. (1960). Tactics of scientific research. New York: Basic Books.
38.
Sohn, D. (1993). Psychology of the scientist: LXVI. The idiot savants have taken over the psychology labs! Or why in science using the the rejection of the null hypothesis as the basis for affirming the research hypothesis is unwarranted. Psychological Reports, 73, 1167-1175.
39.
Steering Committee of the Physicians' Health Study Research Group (1988). Preliminary report: Findings of the aspirin component of the ongoing physicians' health study. New England Journal of Medicine, 318, 262-264.
40.
Thompson, B. (1993). The use of significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
41.
Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157-176.
42.
Tversky, A. , & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
43.
Winch, R.F. , & Campbell, D.T. (1969). Proof? No. Evidence? Yes. The significance of tests of significance. The American Sociologist, 4, 140-143.
