Reformers have long argued that misuse of Null Hypothesis Significance Testing (NHST) is widespread and damaging. The authors analyzed 150 articles from the Journal of Applied Psychology (JAP) covering 1940 to 1999. They examined statistical reporting practices related to misconceptions about NHST, American Psychological Association (APA) guidelines, and reform recommendations. The analysis reveals (a) inconsistency in reporting alpha and p values, (b) the use of ambiguous language in describing NHST, (c) frequent acceptance of null hypotheses without consideration of power, (d) that power estimates are rarely reported, and (e) that confidence intervals were virtually never used. APA guidelines have been followed only selectively. Research methodology reported in JAP has increased greatly in sophistication over 60 years, but inference practices have shown remarkable stability. There is little sign that decades of cogent critiques by reformers had by 1999 led to changes in statistical reporting practices in JAP.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association . (1957). Publication manual of the American Psychological Association. Washington, DC: Author.
2.
American Psychological Association . (1967). Publication manual of the American Psychological Association. Washington, DC: Author.
3.
American Psychological Association . (1974). Publication manual of the American Psychological Association (2nd ed.). Washington, DC: Author.
4.
American Psychological Association . (1983). Publication manual of the American Psychological Association (3rd ed.). Washington, DC: Author.
5.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
6.
American Psychological Association Council of Editors . (1952). Publication manual of the American Psychological Association. Psychological Bulletin, 49, 389-450.
7.
Bakan, D. (1967). On method. San Francisco: Jossey-Bass.
8.
Berkson, J. (1938). Some difficulties of interpretation encountered in the application of the chi-square test. Journal of the American Statistical Association, 33, 526-542.
9.
Bobko, P. (1995). Editorial. Journal of Applied Psychology, 80, 3-5.
10.
Boring, E. G. (1919). Mathematical versus statistical significance. Psychological Bulletin, 16, 335-338.
11.
Campbell, J. P. (1982). Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691-700.
12.
Carver, R. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
13.
Chandler, R. E. (1957) The statistical concepts of confidence and significance. Psychological Bulletin, 54(5), 429-430.
14.
Chase, L. J. , & Chase, R. B. (1976). A statistical power analysis of applied psychological research. Journal of Applied Psychology, 61, 234-237.
15.
Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
16.
Cohen, J. (1965). Some statistical issues in psychological research. In B. Wolman (Ed.), Handbook of clinical psychology (pp. 95-121). New York: McGraw-Hill.
17.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
18.
Cohen, J. (1994). The earth is round (p < 0.05). American Psychologist, 49, 997-1003.
19.
Falk, R. , & Greenbaum, C. W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
20.
Finch, S., Thomason, N., & Cumming, G. (2001). Past and future APA guidelines for statistical practice. Manuscript in preparation.
21.
Gigerenzer, G. (1993). The superego, the ego and the id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioural sciences: Methodological issues (pp. 311-339). Hillsdale, NJ: Erlbaum.
22.
Gigerenzer, G. , & Murray, D. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
23.
Grayson, D. , Pattison, P., & Robins, G. (1997). Evidence, inference and the “rejection” of the significance test. Australian Journal of Psychology, 49(2), 64-70.
24.
Guion, R. M. (1983). Editorial: Comments from the new editor. Journal of Applied Psychology, 68, 547-551.
25.
Hammond, G. (1996). The objections to the null hypothesis as a means of analysing psychological data. Journal of Australian Psychology, 48, 104-106.
26.
Harlow, L. , Mulaik, S., & Steiger, J. (1997). What if there were no significance tests?Mahwah, NJ: Erlbaum.
27.
Heldref Foundation . (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95-96.
28.
Hubbard, R. , Parsa, R. A., & Luthy, M. R. (1997). The spread of statistical significance testing in psychology: The case of the Journal of Applied Psychology 1917-1994. Theory & Psychology, 7, 545-554.
29.
Hubbard, R. , & Ryan, P. A. (2000). The historical growth of statistical significance testing in psychology—And its future prospects. Educational and Psychological Measurement, 60, 661-681.
30.
Huberty, C. J. , & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp. 1-22). Stamford, CT: JAI.
31.
Hunter, J. , & Schmidt, F. (1990). Methods of meta-analysis. Newbury Park, CA: Sage.
32.
Hunter, J. E. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3-7.
33.
International Committee of Medical Journal Editors . (1997). Uniform requirements for manuscripts submitted to biomedical journals. Journal of the American Medical Association, 277, 927-934.
34.
John, I. D. (1992). Statistics as rhetoric in psychology. Australian Psychologist, 27, 144-149.
35.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
36.
Loftus, G. R. (1993). A picture is worth a thousand p-values: On the irrelevance of hypothesis testing in the microcomputer age. Behavior Research Methods, Instruments & Computers, 25, 250-256.
37.
Loftus, G. R. (1996). Why psychology will never be a real science until we change the way we analyze data. Current Directions in Psychological Science, 5(6), 161-171.
38.
Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.
39.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting & Clinical Psychology, 4, 806-843.
40.
Mittag, K. C. , & Thompson, B. (2000). A national survey of AERA members’ perceptions of statistical significance tests and other statistical issues. Educational Researcher, 29(4), 14-20.
41.
Muchinsky, P. M. (1979). Some changes in the characteristics of articles published in the Journal of Applied Psychology over the past 20 years. Journal of Applied Psychology, 64, 455-459.
42.
Murphy, K. R. (1997). Editorial. Journal of Applied Psychology, 82, 3-5.
43.
Nelson, N. , Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 41, 1299-1301.
44.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioural sciences. Chichester, UK: Wiley.
45.
Pollard, P. (1993). How significant is “significance”? In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioural sciences: Methodological issues (pp. 449-460). Hillsdale, NJ: Erlbaum.
46.
Pollard, P. , & Richardson, J. (1987). On the probability of making Type I errors. Psychological Bulletin, 102, 159-163.
47.
Reichardt, C. S. , & Gollob, H. F. (1997). When confidence intervals should be used instead of statistical tests, and vice versa. In L. Harlow, S. Muliak, & J. Steiger (Eds.), What if there were no significance tests? (pp. 259-284). Mahwah, NJ: Lawrence Erlbaum.
48.
Rosenthal, R. , & Gaito, J. (1963). The interpretation of level of significance by psychological researchers. Journal of Psychology, 55, 33-38.
49.
Rosnow, R. L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
50.
Rossi, J. S. (1997). A case study in the failure of psychology as a cumulative science: The spontaneous recovery of verbal learning. In L. Harlow, S. Muliak, & J. Steiger (Eds.), What if there were no significance tests? (pp. 175-197). Mahwah, NJ: Lawrence Erlbaum.
51.
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
52.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
53.
Schmitt, N. (1989). Editorial. Journal of Applied Psychology, 74, 843-845.
54.
Sedlmeier, P. , & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309-315.
55.
Shrout, P. (1997). Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science, 8, 1-2.
56.
Thompson, B. (1993). Foreword. Journal of Experimental Education, 61(4), 285-286.
57.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
58.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
59.
Thompson, B. , & Snyder, P. A. (1997). Statistical significance testing practices in The Journal of Experimental Education. Journal of Experimental Education, 66, 75-83.
60.
Thompson, B. , & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent Journal of Counseling & Development research articles. Journal of Counseling & Development, 76, 436-441.
61.
Tversky, A. , & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.
62.
Tyron, W. W. (1998). The inscrutable null hypothesis. American Psychologist, 53, 796-796.
63.
Vacha-Haase, T. , & Ness, C. N. (1999). Statistical significance testing as it relates to practice: Use within Professional Psychology: Research and Practice. Professional Psychology: Research and Practice, 30(1), 104-105.
64.
Vacha-Haase, T. , & Nilsson, J. E. (1998). Statistical significance reporting: Current trends and uses in MECD. Measurement and Evaluation in Counseling and Development, 31, 46-57.
65.
Vacha-Haase, T. , Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
66.
Wilkinson, L. , & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
67.
Zuckerman, M. , Hodgins, H. S., Zuckerman, A., & Rosenthal, R. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, 49-53.
