Six decades-worth of published information has shown irrefutably that null-hypothesis significance tests (NHSTs) provide no information about the reliability of research outcomes. Nevertheless, they are still the core of editorial decision-making in Psychology. Two reasons appear to contribute to the continuing practice. One, survey information suggests that a majority of psychological researchers incorrectly believe that p values provide information about reliability of results. Two, a position sometimes taken is that using them to make decisions has been essentially benign. The mistaken belief has been pointed out many times, so it is briefly covered because of the apparent persistence of the misunderstanding. The idea that NHSTs have been benign is challenged by seven “side-effects” that continue to retard effective development of psychological science. The article concludes with both a few suggestions about possible alternatives and a challenge to psychological researchers to develop new methods that actually assess the reliability of research findings.
AbelsonR. P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science, 8, 12–15.
2.
AllportG. W. (1937). Personality: A psychological interpretation. New York, NY: Holt, Rinehart, & Winston.
3.
AnscombeF. J. (1973). Graphs in statistical analysis. American Statistician, 27, 17–21.
4.
BakanD. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423–437.
5.
BaumeisterR. F.VohsK. D.FunderD. C. (2007). Psychology as the science of self-reports and finger movements: Whatever happened to actual behavior?Perspectives on Psychological Science, 2, 396–403.
6.
BeaudetteC. G. (2002). Excess heat: Why cold fusion research prevailed. South Bristol, ME: Oak Grove Press.
7.
BergerJ. O.BerryD. A. (1988). Statistical analysis and the illusion of objectivity. American Scientist, 76, 159–165.
8.
BergerJ. O.SelkeT. (1987). Testing a point null hypothesis: The irreconcilability of P values and evidence. Journal of the American Statistical Association, 82, 112–122.
9.
BerksonJ. (1942). Tests of significance considered as evidence. Journal of the American Statistical Association, 37, 325–335.
10.
BranchM. N.PennypackerH. S. (2012). Generality and generalization of research findings. In MaddenG. J. (Ed.), APA handbook of behavior analysis (Vol. 1, pp. 151–175). Washington, DC: American Psychological Association.
11.
BrownG. K.BeckA. T.SteerR. A.GrishamJ. R. (2000). Risk factors for suicide in psychiatric outpatients: A 20-year prospective study. Journal of Consulting and Clinical Psychology, 68, 371–377.
12.
CarverR. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378–399.
13.
ClevelandW. S. (1994). The elements of graphing data. Murray Hill, NJ: AT&T Bell Laboratories.
14.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 69, 145–153.
15.
CohenJ. (1988). Statistical power analysis for the behavior sciences. Hillsdale, NJ: Erlbaum.
16.
CohenJ. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
17.
CohenJ. (1994). The world is round (p < .05). American Psychologist, 49, 997–1003.
18.
Committee on CMMP 2010, Solid State Sciences Committee, National Research Council. (2007). Condensed-matter and materials physics: The science of the world around us. Washington, DC: National Academies Press.
19.
CummingG.MaillardetR. (2006). Confidence intervals and replication: Where will the next mean fall?Psychological Methods, 11, 217–227.
20.
EbbinghausH. (1885). Über das Gedächtnis [On memory]. Leipzig, Germany: Verlag von Duncker & Humber.
21.
FalkR.GreenbaumC. W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75–98.
22.
GelmanA.CarlinJ.SternH.RubinD. B. (1995). Bayesian data analysis. London, UK: Chapman & Hall.
23.
GigerenzerG. (1993). The superego, the ego, and the id in statistical reasoning. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
24.
GigerenzerG.GaissmeyerW.Kurz-MilckeE.SchwartzL. M.WoloshinS. (2008). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8, 53–96.
25.
GoodmanS. N. (1999). Toward evidence-based medical statistics. 1: The p value fallacy. Annals of Internal Medicine, 130, 995–1004.
26.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
27.
HallerS.KraussS. (2002). Misinterpretations of significance: A problem students share with their teachers?Methods of Psychological Research, 7, 1–20.
28.
HaysW. L. (1981). Statistics (3rd ed.). New York, NY: Holt, Rinehart, & Winston.
29.
HungH. M.O’NeillR. T.BauerP.KohneK. (1997). The behavior of the P-value when the alternative hypothesis is true. Biometrics, 53, 11–22.
30.
IoannidisJ. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 0696–0701.
31.
JohnstonJ. M.PennypackerH. S. (2009). Strategies and tactics of human behavioral research (3rd ed.). New York, NY: Routledge.
32.
JonesD.MatloffN. (1986). Statistical hypothesis testing in biology: A contradiction in terms. Journal of Economic Entomology, 79, 1156–1160.
33.
KalinowskiP.FidlerF.CummingG. (2008). Overcoming the inverse probability fallacy: A comparison of two teaching interventions. Experimental Psychology, 4, 152–158.
34.
KentD. M.HaywardR. A. (2007a). Limitations of applying summary results of clinical trials to individual patients: The need for risk stratification. Journal of the American Medical Association, 298, 1209–1212.
35.
KentD.HaywardR. (2007b). When averages hide individual differences in clinical trials: Analyzing the results of clinical trials to expose individual patients’ risks might help doctors make better treatment decisions. American Scientist, 95, 1016–1019.
36.
KlineR. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: APA Books.
37.
LambdinC. (2012). Significance tests as sorcery: Science is empirical—significance tests are not. Theory & Psychology, 22, 67–90. doi: 10.1177/0959354311429854
38.
LoftusG. R. (1996). Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science, 5, 161–171.
39.
MeehlP. E. (1967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.
40.
MeehlP. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
41.
MeehlP. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244.
42.
MichelsonA. A.MorleyE. W. (1887). On the relative motion of the earth and the luminiferous ether. American Journal of Science, 34, 333–345.
43.
MillerJ. (2009). What is the probability of replicating a statistically significant effect?Psychonomic Bulletin & Review, 16, 617–640.
44.
MittagK. C.ThompsonB. (2000). A national survey of AERA members’ perceptions of statistical significance tests and other statistical issues. Educational Researcher, 29, 14–20.
45.
MorganD. L.MorganR. K. (2001). Single-participant research design: Bringing science to managed care. American Psychologist, 56, 119–127.
46.
MotulskyH. (2010). Intuitive biostatistics: A nonmathematical guide to statistical thinking. New York, NY: Oxford University Press.
47.
MulaikS. A.RajuN. S.HarshmanR. A. (1997). There is a time and place for significance testing. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 65–116). Hillsdale, NJ: Erlbaum.
48.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
49.
NieuwenhuisS.ForstmannB. U.WagenmakersE-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105–1107.
50.
OakesM. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York, NY: Wiley.
51.
PavlovI. P. (1927). Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex (AnrepG. V.Ed. & Trans.). Oxford, UK: Oxford University Press.
52.
PenstonJ. (2005). Large-scale randomized trials: A misguided approach to clinical research. Medical Hypotheses, 64, 651–657.
53.
PiagetJ. (1928). The child’s conception of the world. London, UK: Routledge and Kegan Paul.
54.
PopperK. (1959). The logic of scientific discovery. New York, NY: Basic Books.
55.
RosenthalR. (1966). Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts.
56.
RosenthalR.FodeK. L. (1963). The effect of experimenter bias on the performance of the albino rat. Behavioral Science, 8, 183–189.
57.
SchmidtF. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
58.
SidmanM. (1960). Tactics of scientific research. New York, NY: Basic Books.
59.
SiegfriedT. (2010, March). Odds are, it’s wrong: Science fails to face the shortcomings of statistics. Science News, 177, 26–37. Retrieved from http://www.sciencenews.org/view/feature/id/57091
60.
SkinnerB. F. (1938). The behavior of organisms: An experimental analysis. New York, NY: Appleton-Century-Crofts.
SohnD. (1998). Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology, 8, 291–311. doi: 10.1177/0959354398083001
63.
TaubesG. (1993). Bad science: The short life and weird times of cold fusion. New York, NY: Random House.
64.
ThompsonB. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361–377.
65.
ThompsonB. (1994). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157–176.
ThompsonB. (1999). Statistical significance tests, effect size reporting and the vain pursuit of pseudo-objectivity. Theory & Psychology, 9, 191–196. doi: 10.1177/095935439992007
68.
TukeyJ. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
69.
TukeyJ. W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100–116.
70.
WilliamsB. A. (2010). Perils of evidence-based medicine. Perspectives in Biology and Medicine, 53, 106–120.
71.
ZakzanisK. K. (1998). Brain is related to behavior (p < .05). Journal of Clinical and Experimental Neuropsychology, 20, 419–427.
72.
ZiliakS. T.McCloskeyD. N. (2008). The cult of statistical significance: How the standard error costs us jobs, justice, and lives. Ann Arbor: University of Michigan Press.
