Reporting p values from statistical significance tests is common in psychology's empirical literature. Sir Ronald Fisher saw the p value as playing a useful role in knowledge development by acting as an `objective' measure of inductive evidence against the null hypothesis. We review several reasons why the p value is an unobjective and inadequate measure of evidence when statistically testing hypotheses. A common theme throughout many of these reasons is that p values exaggerate the evidence against H0. This, in turn, calls into question the validity of much published work based on comparatively small, including .05, p values. Indeed, if researchers were fully informed about the limitations of the p value as a measure of evidence, this inferential index could not possibly enjoy its ongoing ubiquity. Replication with extension research focusing on sample statistics, effect sizes, and their confidence intervals is a better vehicle for reliable knowledge development than using p values. Fisher would also have agreed with the need for replication research.
Abelson, R.P. (1997). A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). In L.L. Harlow , S.A. Mulaik , & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 117—141). Mahwah, NJ: Erlbaum.
2.
Baird, D. (1988). Significance tests, history and logic. In S. Kotz & N.L. Johnson (Eds.), Encyclopedia of statistical sciences (pp. 466—471). New York: Wiley .
3.
Bakan, D. (1966). The test of significance in psychological research . Psychological Bulletin, 66, 423—437.
4.
Bayarri, M.J. , & Berger, J.O. (1999). Quantifying surprise in the data and model verification (with comments). In J.M. Bernardo , J.O. Berger , A.P. Dawid , & A.F.M. Smith (Eds.), Bayesian statistics (Vol. 6, pp. 53—82). Oxford: Clarendon.
5.
Bayarri, M.J. , & Berger, J.O. (2000). P values for composite null models . Journal of the American Statistical Association, 95, 1127—1142.
6.
Bayarri, M.J. , & Berger, J.O. (2004). The interplay of Bayesian and frequentist analysis . Statistical Science, 19, 58—80.
7.
Bayarri, M.J. , & Mayoral, A.M. (2002). Bayesian design of `successful' replications . The American Statistician, 56, 207—214.
8.
Berger, J.O. (1986). Are p-values reasonable measures of accuracy? In I.S. Francis , B. F.J. Manly , & F.C. Lam (Eds.), Pacific Statistical Congress (pp. 21—27). Amsterdam: Elsevier.
9.
Berger, J.O. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing? (with comments) . Statistical Science, 18, 1—32.
10.
Berger, J.O. , & Berry, D.A. (1988). Statistical analysis and the illusion of objectivity . American Scientist, 76, 159—165.
11.
Berger, J.O. , Boukai, B. , & Wang, Y. (1997). Unified frequentist and Bayesian testing of a precise hypothesis (with comments) . Statistical Science, 12, 133—160.
Berger, J.O. , & Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence (with comments) . Journal of the American Statistical Association, 82, 112—139.
14.
Berkson, J. (1942). Tests of significance considered as evidence . Journal of the American Statistical Association,37, 325—335.
15.
Birnbaum, A. (1962). On the foundations of statistical inference (with comments) . Journal of the American Statistical Association , 57, 269—326.
16.
Carver, R.P. (1978). The case against statistical significance testing . Harvard Educational Review, 48, 378—399.
17.
Casella, G. , & Berger, R.L. (1987). Reconciling Bayesian and frequentist evidence in the one-sided testing problem (with comments) . Journal of the American Statistical Association, 82, 106—139.
18.
Chatfield, C. (1995). Model uncertainty, data mining and statistical inference (with comments) . Journal of the Royal Statistical Society A, 158, 419—466.
Chow, S.L. (1998). Précis of statistical significance: Rationale, validity and utility (with comments) . Behavioral and Brain Sciences, 21, 169—239.
21.
Cohen, J. (1994). The earth is round (p .05) . American Psychologist, 49, 997—1003.
22.
Cortina, J.M. , & Dunlap, W.P. (1997). On the logic and purpose of significance testing . Psychological Methods, 2, 161—172.
23.
Cumming, G. , & Finch, S. (2001). A primer on the understanding, use and calculation of confidence intervals that are based on central and noncentral distributions . Educational and Psychological Measurement, 61, 532—574.
24.
Cumming, G. , & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data . American Psychologist, 60, 170—180.
25.
Dickey, J.M. (1987). Comment on Berger and Sellke . Journal of the American Statistical Association,82, 129—130.
26.
Dollinger, M.B. , Kulinskaya, E. , & Staudte, R.G. (1996). When is a p-value a good measure of evidence? In H. Rieder (Ed.), Robust statistics, data analysis and computer intensive methods (pp. 119—134). New York: Springer Verlag.
Edwards, W. , Lindman, H. , & Savage, L.J. (1963). Bayesian statistical inference for psychological research . Psychological Review, 70, 193—242.
29.
Ehrenberg, A.S.C. , & Bound, J.A. (1993). Predictability and prediction (with comments) . Journal of the Royal Statistical Society A, 156, 167—206.
30.
Falk, R. (1998). Replication—A step in the right direction: Commentary on Sohn . Theory & Psychology, 8, 313—321.
31.
Falk, R. , & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception . Theory & Psychology, 5, 75—98.
32.
Fidler, F. , Thomason, N. , Cumming, G. , Finch, S. , & Leeman, J. (2004). Editors can lead researchers to confidence intervals, but can't make them think: Statistical reform lessons from medicine . Psychological Science, 15, 119—126.
33.
Fidler, F. , Thomason, N. , Cumming, G. , Finch, S. , & Leeman, J. (2005). Still much to learn about confidence intervals: Reply to Rouder and Morey (2005)Psychological Science, 16, 494—495.
34.
Fisher, R.A. (1925). Statistical methods for research workers. Edinburgh: Oliver & Boyd. Fisher, R.A. (1935). The design of experiments. Edinburgh: Oliver & Boyd.
35.
Fisher, R.A. (1959). Statistical methods and scientific inference (2nd ed.). Edinburgh: Oliver & Boyd.
36.
Fisher, R.A. (1966). The design of experiments (8th ed.). Edinburgh: Oliver & Boyd.
37.
Fisher Box, J. (1978). R.A. Fisher: The life of a scientist. New York: Wiley.
38.
Freeman, P.R. (1993). The role of p-values in analysing trial results . Statistics in Medicine, 12, 1443—1452.
39.
Freund, J.E. , & Perles, B.M. (1993). Observations on the definition of P-values . Teaching Statistics, 15, 8—9.
40.
Frick, R.W. (1996). The appropriate use of null hypothesis testing . Psychological Methods, 1, 379—390.
41.
Gelman, A. , & Stern, H. (2006). The difference between `significant' and `not significant' is not itself statistically significant . The American Statistician, 60, 328—331.
42.
Gibbons, J.D. (1986). P-Values. In S. Kotz & N.L. Johnson (Eds.), Encyclopedia of statistical sciences (pp. 366—368). New York : Wiley.
43.
Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In G. Keren & C.A. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311—339). Hillsdale, NJ: Erlbaum.
44.
Gigerenzer, G. , Krauss, S. , & Vitouch, O. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp. 391—408). Thousand Oaks, CA: SAGE.
45.
Gigerenzer, G. , & Murray, D.J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
46.
Gigerenzer, G. , Swijtink, Z. , Porter, T. , Daston, L. , Beatty, J. , & Kruger, L. (1989). The empire of chance. New York : Cambridge University Press.
47.
Glover, S. , & Dixon, P. (2004). Likelihood ratios: A simple and flexible statistic for empirical psychologists . Psychonomic Bulletin & Review , 11, 791—806.
48.
Goldstein, H. , & Healy, M.J.R. (1995). The graphical interpretation of a collection of means . Journal of the Royal Statistical Society A, 158, 175—177.
49.
Good, I.J. (1981). Some logic and history of hypothesis testing. In J.C. Pitt (Ed.), Philosophy in economics (pp. 149—174). Dordrecht: D. Reidel.
50.
Goodman, S.N. (1993). P values, hypothesis tests, and likelihood: Implications for epidemiology of a neglected historical debate . American Journal of Epidemiology, 137, 485—496.
51.
Goodman, S.N. (1999). Toward evidence-based medical statistics. 1: The P value fallacy . Annals of Internal Medicine, 130, 995—1004.
52.
Goodman, S.N. , & Royall, R.M. (1988). Evidence and scientific research . American Journal of Public Health, 78, 1568—1574.
53.
Guttman, L. (1977). What is not what in statistics . The Statistician, 26, 81—107.
54.
Hagen, R.L. (1997). In praise of the null hypothesis statistical test . American Psychologist, 52, 15—24.
55.
Hand, D.J. (1998). Data mining: Statistics and more ? The American Statistician, 52, 112—118.
56.
Hogben, L. (1957). Statistical theory. New York : Norton.
57.
Hubbard, R. (2004). Alphabet soup: Blurring the distinctions between p's and α's in psychological research . Theory & Psychology , 14, 295—327.
58.
Hubbard, R. , & Armstrong, J.S. (2006). Why we don't really know what statistical significance means: Implications for educators . Journal of Marketing Education, 28, 114—120.
59.
Hubbard, R. , & Bayarri, M.J. (2003a). Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing (with comments) . The American Statistician, 57, 171—182.
60.
Hubbard, R. , & Bayarri, M.J. (2003b). P values are not error probabilities Institute of Statistics and Decision Sciences, Working Paper, No. 03—26. Durham, NC: Duke University Working Papers Series, 27708—0251.
61.
Hubbard, R. , & Bayarri, M.J. (2005). Comment on Christensen . The American Statistician, 59, 353 .
62.
Hubbard, R. , & Ryan, P.A. (2000). The historical growth of statistical significance testing in psychology—and its future prospects . Educational and Psychological Measurement, 60, 661—681.
63.
Jeffreys, H. (1939). Theory of probability. Oxford : Clarendon.
64.
Johnstone, D.J. (1986). Tests of significance in theory and practice (with comments) . The Statistician, 35, 491—504.
65.
Krämer, W. , & Gigerenzer, G. (2005). How to confuse with statistics or: The use and misuse of conditional probabilities . Statistical Science, 20, 223—230.
66.
Krantz, D.H. (1999). The null hypothesis testing controversy in psychology . Journal of the American Statistical Association, 44, 1372—1381.
67.
Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method . American Psychologist, 56, 16—26.
Lindley, D.V. (1993). The analysis of experimental data: The appreciation of tea and wine . Teaching Statistics, 15, 22—25.
70.
Lindley, D.V. (1999). Comment on Bayarri and Berger. In J.M. Bernardo , J.O. Berger , A.P. Dawid , & A.F.M. Smith (Eds.), Bayesian Statistics (Vol. 6, p. 75). Oxford: Clarendon .
71.
Lindsay, R.M. (1995). Reconsidering the status of tests of significance: An alternative criterion of adequacy . Accounting, Organizations and Society, 20, 35—53.
72.
Lindsay, R.M. , & Ehrenberg, A.S.C. (1993). The design of replicated studies . The American Statistician, 47, 217—228.
Loftus, G.R. (1996). Psychology will be a much better science when we change the way we analyze data . Current Directions in Psychological Science, 5, 161—171.
75.
Marden, J.I. (2000). Hypothesis testing: From p values to Bayes factors . Journal of the American Statistical Association, 95, 1316—1320.
76.
Mulaik S.A. , Raju, N.S. , & Harshman, R.A. (1997). There is a time and a place for significance testing. In L.L. Harlow , S.A. Mulaik , & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 65—115). Mahwah, NJ: Erlbaum.
77.
Nelder, J.A. (1986). Statistics, science and technology (with comments) . Journal of the Royal Statistical Society A, 149, 109—121.
78.
Nelder, J.A. (1999). From statistics to statistical science (with comments) . The Statistician, 48, 257—269.
79.
Nester, M.R. (1996). An applied statistician's creed . The Statistician, 45, 401—410.
80.
Neuliep, J.W. , & Crandall, R. (1990). Editorial bias against replication research . Journal of Social Behavior and Personality, 5, 85—90.
81.
Neuliep, J.W. , & Crandall, R. (1993). Reviewer bias against replication research . Journal of Social Behavior and Personality, 8, 22—29.
82.
Neyman, J. (1977). Frequentist probability and frequentist statistics . Synthese, 36, 97—131.
83.
Nickerson, R.S. (2000). Null hypothesis statistical testing: A review of an old and continuing controversy . Psychological Methods , 5, 241—301.
84.
Ottenbacher, K.J. (1996). The power of replications and replications of power . The American Statistician, 50, 271—275.
85.
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling . London, Edinburgh and Dublin Philosophical Magazine and Journal of Science , 50, 157—175.
86.
Rosenbaum, P.R. (1999). Choice as an alternative to control in observational studies (with comments) . Statistical Science, 14, 259—304.
87.
Rosenbaum, P.R. (2001). Replicating effects and biases . The American Statistician, 55, 223—227.
88.
Rosenthal, R. (1990). Replication in behavioral research . Journal of Social Behavior and Personality, 5, 1—30.
89.
Rosnow, R.L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science . American Psychologist , 44, 1276—1284.
90.
Royall, R.M. (1986). The effect of sample size on the meaning of significance tests . The American Statistician, 40, 313—315.
91.
Royall, R.M. (1997). Statistical evidence: A likelihood paradigm. New York: Chapman & Hall.
92.
Schenker, N. , & Gentleman, J.F. (2001). On judging the significance of differences by examining the overlap between confidence intervals . The American Statistician, 55, 182—186.
93.
Schervish, M.J. (1996). P values: What they are and what they are not . The American Statistician, 50, 203—206.
94.
Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers . Psychological Methods, 1, 115—129.
95.
Sellke, T. , Bayarri, M.J. , & Berger, J.O. (2001). Calibration of p values for testing precise null hypotheses . The American Statistician, 55, 62—71.
96.
Smithson, M. (2003). Confidence intervals. Thousand Oaks, CA: SAGE.
97.
Sohn, D. (1998). Statistical significance and replicability: Why the former does not presage the latter . Theory & Psychology , 8, 291—311.
98.
Steiger, J.H. (1990). Structural model evaluation and modification: An interval estimation approach . Multivariate Behavioral Research , 25, 173—180.
99.
Thompson, B. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results . Journal of Personality, 62, 157—176.
100.
Thompson, B. (1999). If statistical significance tests are broken/misused, what practices should supplement or replace them ? Theory & Psychology, 9, 165—181.
101.
Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes . Educational Researcher, 31, 25—32.
102.
Tryon, W.W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests . Psychological Methods, 6, 371—386.
103.
Tukey, J.W. (1991). The philosophy of multiple comparisons . Statistical Science, 6, 100—116.
