Statistical inference in psychology has traditionally relied heavily on p-value significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes. The second is to replace inference with Bayesian measures of evidence, such as the Bayes factor. The authors provide a practical comparison of p values, effect sizes, and default Bayes factors as measures of statistical evidence, using 855 recently published t tests in psychology. The comparison yields two main results. First, although p values and default Bayes factors almost always agree about what hypothesis is better supported by the data, the measures often disagree about the strength of this support; for 70% of the data sets for which the p value falls between .01 and .05, the default Bayes factor indicates that the evidence is only anecdotal. Second, effect sizes can provide additional evidence to p values and default Bayes factors. The authors conclude that the Bayesian approach is comparatively prudent, preventing researchers from overestimating the evidence in favor of an effect.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2.
BemD.J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425.
BergerJ.O.SellkeT. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82, 112–139.
5.
BergerJ.O.WolpertR.L. (1988). The likelihood principle (2nd ed.). Hayward, CA: Institute of Mathematical Statistics.
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
7.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
8.
CortinaJ.M.DunlapW.P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2, 161–172.
9.
CummingG. (2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300.
10.
DennisS.LeeM.KinnellA. (2008). Bayesian analysis of recognition memory: The case of the list-length effect. Journal of Memory and Language, 59, 361–376.
11.
DienesZ. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan.
12.
DienesZ. (2011). Bayesian versus orthodox statistics: Which side are you on?Perspectives on Psychological Science, 6, 274–290.
13.
DixonP. (2003). The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology, 57, 189–202.
ErdfelderE. (2010). A note on statistical analysis. Experimental Psychology, 57, 1–4.
16.
FisherR.A. (1935). The design of experiments. Edinburgh: Oliver and Boyd.
17.
FrickR.W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379–390.
18.
GallistelC. (2009). The importance of proving the null. Psychological Review, 116, 439–453.
19.
GelmanA.HillJ. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, England: Cambridge University Press.
20.
GigerenzerG. (1993). The Superego, the ego, and the id in statistical reasoning. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
21.
GigerenzerG. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.
22.
GönenM.JohnsonW.O.LuY.WestfallP.H. (2005). The Bayesian two-sample t test. American Statistician, 59, 252–257.
23.
GoodI.J. (1983). Good thinking: The foundations of probability and its applications. Minneapolis: University of Minnesota Press.
24.
GoodI.J. (1985). Weight of evidence: A brief survey. In BernardoJ.M.DeGrootM.H.LindleyD.V.SmithA.F.M. (Eds.), Bayesian statistics2 (pp. 249–269). New York: Elsevier.
25.
HagenR.L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52, 15–24.
26.
HowardG.MaxwellS.FlemingK. (2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315–332.
27.
IoannidisJ.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696–701.
28.
JaynesE.T. (2003). Probability theory: The logic of science. Cambridge, UK: Cambridge University Press.
29.
JeffreysH. (1961). Theory of probability. Oxford, UK: Oxford University Press.
30.
KassR.E.RafteryA.E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377–395.
31.
KilleenP.R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.
32.
KilleenP.R. (2006). Beyond statistical inference: A decision theory for science. Psychonomic Bulletin & Review, 13, 549–562.
KruschkeJ.K. (2010b). Doing Bayesian data analysis: A tutorial introduction with R and BUGS. Burlington, MA: Academic Press.
35.
KruschkeJ.K. (2010c). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14, 293–300.
36.
KruschkeJ.K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299–312.
37.
LeeM.D. (2008). Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review, 15, 1–15.
38.
LeeM.D. (in press). How cognitive modeling can benefit from hierarchical Bayesian models. Journal of Mathematical Psychology.
39.
LeeM.D.WagenmakersE.-J. (2005). Bayesian statistical inference in psychology: Comment on Trafimow (2003). Psychological Review, 112, 662–668.
40.
LiangF.PauloR.MolinaG.ClydeM.BergerJ. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410.
41.
LindleyD.V. (1972). Bayesian statistics; a review. Philadelphia: Society for Industrial and Applied Mathematics.
42.
LoftusG.R. (1996). Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science, 5, 161–171.
43.
LunnD.J.ThomasA.BestN.SpiegelhalterD. (2000). WinBUGS—a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.
44.
MussweilerT. (2006). Doing is for thinking!Psychological Science, 17, 17–21.
45.
MyungI.J.ForsterM.R.BrowneM.W. (2000). A special issue on model selection. Journal of Mathematical Psychology, 44.
46.
NickersonR.S. (2000). Null hypothesis statistical testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
47.
PittM.A.MyungI.J.ZhangS. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472–491.
48.
PressS.ChibS.ClydeM.WoodworthG.ZaslavskyA. (2003). Subjective and objective Bayesian statistics: Principles, models, and applications. Hoboken, NJ: Wiley-Interscience.
49.
RichardF.D.BondC.F.J.Stokes-ZootaJ.J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.
50.
RosenthalR. (1990). How are we doing in soft psychology?American Psychologist, 45, 775–777.
51.
RosenthalR.RubinD. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166–169.
52.
RouderJ.N.LuJ. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604.
53.
RouderJ.N.SpeckmanP.L.SunD.MoreyR.D.IversonG. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.
54.
SchmidtF.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
55.
ScottJ.BergerJ. (2006). An exploration of aspects of Bayesian multiple testing. Journal of Statistical Planning and Inference, 136, 2144–2162.
56.
ThompsonB. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25–32.
57.
WagenmakersE.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.
58.
WagenmakersE.-J.GrünwaldP. (2006). A Bayesian perspective on hypothesis testing. Psychological Science, 17, 641–642.
59.
WagenmakersE.-J.LodewyckxT.KuriyalH.GrasmanR. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the Savage-Dickey method. Cognitive Psychology, 60, 158–189.
60.
WagenmakersE.-J.WetzelsR.BorsboomD.van der MaasH.L.J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432.
61.
WainerH. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 4, 212–213.
62.
WassermanL. (2004). All of statistics: A concise course in statistical inference. New York: Springer.
63.
WetzelsR.LeeM.WagenmakersE.-J. (2010). Bayesian inference using WBDev: A tutorial for social scientists. Behavior Research Methods, 42, 884–897.
64.
WetzelsR.RaaijmakersJ.JakabE.WagenmakersE.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review, 16, 752–760.
65.
WilkinsonL.the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
