Researchers are often confused about what can be inferred from significance tests. One problem occurs when people apply Bayesian intuitions to significance testing—two approaches that must be firmly separated. This article presents some common situations in which the approaches come to different conclusions; you can see where your intuitions initially lie. The situations include multiple testing, deciding when to stop running participants, and when a theory was thought of relative to finding out results. The interpretation of nonsignificant results has also been persistently problematic in a way that Bayesian inference can clarify. The Bayesian and orthodox approaches are placed in the context of different notions of rationality, and I accuse myself and others as having been irrational in the way we have been using statistics on a key notion of rationality. The reader is shown how to apply Bayesian inference in practice, using free online software, to allow more coherent inferences from data.
ArmitageP.BerryG.MatthewsJ.N.S. (2002). Statistical methods in medical research (4th ed.). Malden, MA: Blackwell.
2.
ArmitageP.McPhersonC.K.RoweB.C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society: Series A. Statistics in Society, 132, 235–244.
3.
BaguleyT. (2009). Standardized or simple effect size: What should be reported?British Journal of Psychology, 100, 603–617.
4.
BaguleyT. (in press). Beyond analysis of variance. Hampshire, England: Palgrave Macmillian.
5.
BaguleyT.KayeW.S. (2010). Review of “Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference”. British Journal of Mathematical & Statistical Psychology, 63, 695–698.
6.
BarghJ.A.ChenM.BurrowsL. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230–244.
7.
BergerJ. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing? Statistical Science, 18, 1–32.
8.
BirnbaumA. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 53, 259–326.
9.
BlumeJ. D. (in press). Likelihood and its evidential framework. In BandyopadhyayP.S.ForsterM. (Eds.), Handbook of the philosophy of statistics. Amsterdam: Elsevier.
DienesZ. (2010). Measuring the conscious status of knowledge. In MirandaL.A.P.MadariagaA.I. (Eds.), Advances in cognitive science: Learning, evolution and social action (pp. 113–128). Bilbao, Spain: University of the Basque Country Press.
14.
FidlerF.ThomasonN.CummingG.FinchS.LeemanJ. (2004). Editors can lead researchers to confidence intervals, but can’t make them think. Psychological Science, 15, 119–126.
15.
GigerenzerG. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587–606.
HarlowL.L.MulaikS.A.SteigerJ.H. (Eds.) (1997). What if there were no significance tests?Mahwah, NJ: Erlbaum.
18.
HoijtinkH.KlugkistI.BoelenP.A. (Eds.) (2008). Bayesian evaluation of informative hypotheses. New York: Springer.
19.
HowardG.S.MaxwellS.E.FlemingK.J. (2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315–332.
20.
HowsonC.UrbachP. (2006). Scientific reasoning: The Bayesian approach (3rd ed.). Chicago: Open Court.
21.
JaynesE.T. (2003). Probability theory: The logic of science. Cambridge, England: Cambridge University Press.
22.
JeffreysH. (1961). The theory of probability (3rd ed.). Oxford, England: Oxford University Press.
23.
JohanssonT. (in press). Hail the impossible: p values, evidence, and likelihood. Scandinavian Journal of Psychology.
24.
KerrN.L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.
KruschkeJ.K. (2010b). Doing Bayesian data analysis: A tutorial with R and BUGS. Burlington, MA: Academic Press.
27.
KruschkeJ.K. (2010c). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14, 293–300.
28.
KruschkeJ.K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299–312.
29.
Lakatos. I. (1978). The methodology of scientific research programmes: Philosophical papers (Vol. 1). Cambridge, England: Cambridge University Press.
30.
LakatosI.FeyerabendP. (1999). For and against method. Chicago: University of Chicago Press.
31.
MayoD. (1996). Error and the growth of experimental knowledge. Chicago: University of Chicago Press.
32.
MayoD.G. (2004). An error statistical philosophy of evidence. In TaperM.L.LeleS.R. (Eds.), The nature of scientific evidence: Statistical, philosophical and empirical considerations (pp. 79–96). Chicago: University of Chicago Press.
33.
MeehlP. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.
34.
MillerD. (1994). Critical rationalism: A restatement and defence. Chicago: Open Court.
35.
MussweilerT. (2006). Doing is for thinking!Psychological Science, 17, 17–21.
36.
OakesM. (1986). Statistical inference: A commentary for the social and behavioural sciences. Hoboken, NJ: Wiley.
37.
PopperK. (1963). Conjectures and refutations. London: Routledge.
38.
RouderJ.N.MoreyR.D.SpeckmanP.L.PratteM.S. (2007). Detecting chance: A solution to the null sensitivity problem in subliminal priming. Psychonomic Bulletin & Review, 14, 597–605.
39.
RouderJ.N.SpeckmanP.L.SunD.MoreyR.D.IversonG. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.
SavageL.J. (1962). The foundations of statistical inference: A discussion. London: Methuen.
42.
SiviaD.S.SkillingJ. (2006). Data analysis: A Bayesian tutorial (2nd ed.). Oxford, England: Oxford University Press.
43.
TaperM.L.LeleS.R. (2004). The nature of scientific evidence: Statistical, philosophical and empirical considerations. Chicago: University of Chicago Press.
44.
von MisesR. (1957). Probability, statistics and truth. London: Macmillan.
45.
WagenmakersE.J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.
46.
WestfallP.H.JohnsonW.O.UttsJ.M. (1997). A Bayesian perspective on the Bonferroni adjustment. Biometrika, 84, 419–427.
47.
WetzelsR.MatzkeD.LeeM.D.RouderJ.N.IversonG.J.WagenmakersE.J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291–298.
48.
WrightD.B. (2010). Ten statisticians and their impacts for psychologists. Perspectives on Psychological Science, 4, 587–597.
49.
ZiliakS.T.McCloskeyD.N. (2008). The cult of statistical significance: How the standard error cost us jobs, justice and lives. Ann Arbor: University of Michigan Press.
