Critics of null hypothesis significance testing suggest that (a) its basic logic is invalid and (b) it addresses a question that is of no interest. In contrast to (a), I argue that the underlying logic of hypothesis testing is actually extremely straightforward and compelling. To substantiate that, I present examples showing that hypothesis testing logic is routinely used in everyday life. These same examples also refute (b) by showing circumstances in which the logic of hypothesis testing addresses a question of prime interest. Null hypothesis significance testing may sometimes be misunderstood or misapplied, but these problems should be addressed by improved education.
BakkerM.Van DijkA.WichertsJ. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. doi:10.1177/1745691612459060
2.
BataneroC. (2000). Controversies around the role of statistical tests in experimental research. Mathematical Thinking and Learning, 2, 75-97. doi:10.1207/S15327833MTL0202_4
3.
BergerJ. O.BerryD. A. (1988). Statistical analysis and the illusion of objectivity. American Scientist, 76, 159-165.
4.
BerksonJ. (1942). Tests of significance considered as evidence. Journal of the American Statistical Association, 37, 325-335. doi:10.1080/01621459.1942.10501760
5.
CampbellS. K. (1974). Flaws and fallacies in statistical thinking. Englewood Cliffs, NJ: Prentice-Hall.
6.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003. doi:10.1037//0003-066X.49.12.997
7.
CoxD. R. (1958). Some problems connected with statistical inference. Annals of Mathematical Statistics, 29, 357-372. doi:10.1214/aoms/1177706618
8.
CummingG.FidlerF. (2009). Confidence intervals: Better answers to better questions. Zeitschrift für Psychologie, 217, 15-26. doi:10.1027/0044-3409.217.1.15
9.
EfronB. (2013). Bayes’ theorem in the 21st century. Science, 340, 1177-1178. doi:10.1126/science.1236536
10.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555-561. doi:10.1177/1745691612459059
11.
FrancisG. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975-991. doi:10.3758/s13423-012-0322-y
12.
GelmanA. (2013). Commentary: P values and statistical practice. Epidemiology, 24, 69-72.
13.
GigerenzerG.GaissmaierW.Kurz-MilckeE.SchwartzL. M.WoloshinS. (2008). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8, 53-96. doi:10.1111/j.1539-6053.2008.00033.x
14.
GigerenzerG.MarewskiJ. N. (2015). Surrogate science: The idol of a universal method for scientific inference. Journal of Management, 41, 421-440. doi:10.1177/0149206314547522
15.
GrantD. A. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 69, 54-61. doi:10.1037/h0038813
16.
HallerH.KrausS. (2002). Misinterpretations of significance: A problem students share with their teachers?Methods of Psychological Research, 7, 1-20.
17.
HollandB. K. (2007). A classroom demonstration of hypothesis testing. Teaching Statistics, 29, 71-73. doi:10.1111/j.1467-9639.2007.00269.x
18.
HubbardR.LindsayR. M. (2008). Why p values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18, 69-88. doi:10.1177/0959354307086923
19.
IoannidisJ. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124
20.
JohnL. K.LoewensteinG.PrelecD. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532. doi:10.1177/0956797611430953
21.
KalinowskiP.FidlerF.CummingG. (2008). Overcoming the inverse probability fallacy: A comparison of two teaching interventions. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4, 152-158.
22.
KlineR. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
23.
LeekJ. T.PengR. D. (2015). P values are just the tip of the iceberg. Nature, 520, 612.
24.
NosekB. A.SpiesJ. R.MotylM. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.
25.
PollardP.RichardsonJ. T. E. (1987). On the probability of making Type I errors. Psychological Bulletin, 102, 159-163. doi:10.1037/0033-2909.102.1.159
26.
RipsL. J.MarcusS. L. (1977). Suppositions and the analysis of conditional sentences. In JustM. A.CarpenterP. A. (Eds.), Cognitive processes in comprehension (pp. 185-220). Hillsdale, NJ: Lawrence Erlbaum.
27.
RouderJ. N.SpeckmanP. L.SunD.MoreyR. D.IversonG. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237. doi:10.3758/PBR.16.2.225
28.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632
29.
TrafimowD.MarksM. (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1-2. doi:10.1080/01973533.2015.1012991
30.
TsilidisK. K.PanagiotouO. A.SenaE. S.AretouliE.EvangelouE.HowellsD. W.… IoannidisJ. P. A. (2013). Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biology, 11(7), e1001609. doi:10.1371/journal.pbio.1001609
31.
WagenmakersE. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804.
32.
WainerH.RobinsonD. H. (2003). Shaping up the practice of null hypothesis significance testing. Educational Researcher, 32, 22-30. doi:10.3102/0013189X032007022
33.
WasonP. C. (1968). Reasoning about a rule. Quarterly Journal of Experimental Psychology, 20, 273-281. doi:10.1080/14640746808400161
34.
WetzelsR.RaaijmakersJ. G. W.JakabE.WagenmakersE. J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review, 16, 752-760. doi:10.3758/PBR.16.4.752
35.
WilkinsonL., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. doi:10.1037/0003-066X.54.8.594
36.
ZiliakS. T.McCloskeyD. (2008). The cult of statistical significance: How the standard error costs us jobs, justice, and lives. Ann Arbor: University of Michigan Press.
