The historical growth in the popularity of statistical significance testing is examined using a random sample of annual data from 12 American Psychological Association (APA) journals. The results replicate and extend the findings of Hubbard, Parsa, and Luthy, who used data from only the Journal of Applied Psychology. The results also confirm Gigerenzer and Murray’s allegation that an inference revolution occurred in psychology between 1940 and 1955. An assessment of the future prospects for statistical significance testing is offered. It is concluded that replication with extension research, and its connections with meta-analysis, is a better vehicle for developing a cumulative knowledge base in the discipline than statistical significance testing. It is conceded, however, that statistical significance testing is likely here to stay.
Get full access to this article
View all access options for this article.
References
1.
Abelson, R. P. (1997). A retrospective on the significance test ban of 1999 (if there were no significance tests, they would be invented). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 117-141). Mahwah, NJ: Lawrence Erlbaum.
2.
Allen, M. , & Preiss, R. (1993). Replication and meta-analysis: A necessary connection. Journal of Social Behavior and Personality, 8, 9-20.
3.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
4.
Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
5.
Berkson, J. (1942). Tests of significance considered as evidence. Journal of the American Statistical Association, 37, 325-335.
6.
Box, J. F. (1978). R. A. Fisher: The life of a scientist. New York: John Wiley.
7.
Bozarth, J. D. , & Roberts, R. R. (1972). Signifying significant significance. American Psychologist, 27, 774-775.
8.
Busche, K. , & Kennedy, P. (1984). On economist’s belief in the law of small numbers. Economic Inquiry, 22, 602-603.
9.
Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
10.
Chow, S. L. (1988). Significance test or effect size?Psychological Bulletin, 103, 105-110.
11.
Chow, S. L. (1996). Statistical significance: Rationale, validity and utility. Thousand Oaks, CA: Sage.
12.
Chow, S. L. (1998). Précis of statistical significance: Rationale, validity and utility (with comments). Behavioral and Brain Sciences, 21, 169-239.
13.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
14.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
15.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
16.
Cortina, J. M. , & Dunlap, W. P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2, 161-172.
17.
Cowles, M. , & Davis, C. (1982). On the origins of the .05 level of statistical significance. American Psychologist, 37, 553-558.
18.
Dar, R. (1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. American Psychologist, 42, 145-151.
19.
Dar, R. , Serlin, R. C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62, 75-82.
20.
Falk, R. , & Greenbaum, C. W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
21.
Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver and Boyd.
22.
Fisher, R. A. (1935). The design of experiments. Edinburgh, Scotland: Oliver and Boyd.
23.
Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390.
24.
Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In G. Keren & C. A. Lewis (Eds.), A handbook for data analysis in the behavioral sciences-methodological issues (pp. 311-339). Hillsdale, NJ: Lawrence Erlbaum.
25.
Gigerenzer, G. , & Murray, D. J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Lawrence Erlbaum.
26.
Gigerenzer, G. , Swijtink, Z., Porter, T., Daston, L., Beatty, J., & Kruger, L. (1989). The empire of chance. New York: Cambridge University Press.
27.
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
28.
Greenwald, A. G. , Gonzalez, R., Harris, R. J., & Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated?Psychophysiology, 33, 175-183.
29.
Guttman, L. (1985). The illogic of statistical inference for cumulative science. Applied Stochastic Models and Data Analysis, 1, 3-10.
30.
Hagen, R. L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52, 15-24.
31.
Hagen, R. L. (1998). A further look at wrong reasons to abandon statistical testing. American Psychologist, 53, 801-803.
32.
Harlow, L. L. , Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests?Mahwah, NJ: Lawrence Erlbaum.
33.
Harris, R. J. (1997). Reforming significance testing via three-valued logic. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 145-174). Mahwah, NJ: Lawrence Erlbaum.
34.
Hubbard, R. (1995). The earth is highly significantly round (p < .001). American Psychologist, 50, 1098-1098.
35.
Hubbard, R. , & Armstrong, J. S. (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing, 11, 233-248.
36.
Hubbard, R. , & Armstrong, J. S. (1997). Publication bias against null results. Psychological Reports, 80, 337-338.
37.
Hubbard, R. , Parsa, R. A., & Luthy, M. R. (1997). The spread of statistical significance testing in psychology: The case of the Journal of Applied Psychology, 1917-1994. Theory & Psychology, 7, 545-554.
38.
Hubbard, R. , & Vetter, D. E. (1996). An empirical comparison of published replication research in accounting, economics, finance, management, and marketing. Journal of Business Research, 35, 153-164.
39.
Hubbard, R. , Vetter, D. E., & Little, E. L. (1998). Replication in strategic management: Scientific testing for validity, generalizability, and usefulness. Strategic Management Journal, 19, 243-254.
40.
Huberty, C. J (1993). Historical origins of statistical testing practices: The treatment of Fisher versus Neyman-Pearson views in textbooks. Journal of Experimental Education, 61, 317-333.
41.
Huberty, C. J , & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp. 1-22). Stamford, CT: JAI.
42.
Humphreys, L. G. (1980). The statistics of failure to replicate: A comment on Buriel’s (1978) conclusions. Journal of Educational Psychology, 72, 71-75.
43.
Hunter, J. E. , & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
44.
Kihlstrom, J. F. (1997). Editorial. Psychological Science, 8, 1-1.
45.
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
46.
Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43, 635-642.
47.
Kupfersmid, J. , & Wonderly, D. M. (1994). An author’s guide to publishing better articles in better journals in the behavioral sciences. Brandon, VT: Clinical Psychology.
48.
Levin, J. R. (1998). To test or not to test H?Educational and Psychological Measurement, 58, 313-333.
49.
Lindsay, R. M. , & Ehrenberg, A.S.C. (1993). The design of replicated studies. American Statistician, 47, 217-228.
50.
Loftus, G. R. (1996). Psychology will be a much better science when we change the way we analyze data. Psychological Science, 7, 161-171.
51.
Lovie, A. D. (1979). The analysis of variance in experimental psychology: 1934-1945. British Journal of Mathematical and Statistical Psychology, 32, 151-178.
52.
Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.
53.
Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.
54.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
55.
Morrison, D. E. , & Henkel, R. E. (Eds.). (1970). The significance test controversy—A reader. Chicago: Aldine.
56.
Mulaik, S. A. , Raju, N. S., & Harshman, R. A. (1997). There is a time and a place for significance testing. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 65-115). Mahwah, NJ: Lawrence Erlbaum.
57.
Nelder, J. A. (1985). Discussion on the initial examination of data (by C. Chatfield). Journal of the Royal Statistical Society, Ser. A, 148, 238-238.
58.
Nelder, J. A. (1986). Statistics, science and technology. Journal of the Royal Statistical Society, Ser. A, 149, 109-121.
59.
Nester, M. R. (1996). An applied statistician’s creed. Applied Statistics, 45, 401-410.
60.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: John Wiley.
61.
Robinson, D. H. , & Levin, J. R. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26, 21-26.
62.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638-641.
63.
Rosenthal, R. , & Rosnow, R. L. (1984). Essentials of behavioral research: Methods and data analysis. New York: McGraw-Hill.
64.
Rosnow, R. L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
65.
Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416-428.
66.
Rucci, A. J. , & Tweney, R. D. (1980). Analysis of variance and the “second discipline” of scientific psychology: A historical account. Psychological Bulletin, 87, 166-184.
67.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
68.
Schmidt, F. L. , & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum.
69.
Shrout, P. E. (1997). Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science, 8, 1-2.
70.
Snedecor, G. W. (1934). Calculation and interpretation of the analysis of variance and covariance. Ames, IA: Collegiate.
71.
Snedecor, G. W. (1937). Statistical methods. Ames, IA: Collegiate.
72.
Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.
73.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54, 30-34.
74.
Sterling, T. D. , Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108-112.
75.
Thompson, B. (1994a). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
76.
Thompson, B. (1994b). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157-176.
77.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.
78.
Thompson, B. (1997). Editorial policies regarding statistical significance tests: Further comments. Educational Researcher, 26, 29-32.
79.
Thompson, B. (1999a). If statistical significance tests are broken/misused, what practices should supplement or replace them?Theory & Psychology, 9, 165-181.
80.
Thompson, B. (1999b). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
81.
Thompson, B. (1999c). Why “encouraging” effect size reporting is not working: The etiology of researcher resistance to changing practices. Journal of Psychology, 133, 133-140.
82.
Thompson, B. , & Snyder, P. A. (1997). Statistical significance testing practices in The Journal of Experimental Education. Journal of Experimental Education, 66, 75-83.
83.
Tversky, A. , & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
84.
Utts, J. (1991). Replication and meta-analysis in parapsychology. Statistical Science, 6, 363-378.
85.
Vacha-Haase, T. , Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
86.
Wilkinson, L. , & The APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. [reprint available through the APA home page: http://www.apa.org/journals/amp/amp548594.html]
87.
Yates, F. (1951). The influence of Statistical Methods for Research Workers on the development of the science of statistics. Journal of the American Statistical Association, 46, 19-34.
