Significance testing has been controversial since Neyman and Pearson published their procedure for testing statistical hypotheses. Fisher, who popularized tests of significance, first noticed the emerging confusion between that procedure and his own, yet he could not stop their hybridization into what is nowadays known as Null Hypothesis Significance Testing (NHST). Here I hypothesize why similar attempts to clarify matters have also failed; namely because both procedures are designed to be confused: their names may not match purpose, both use null hypotheses and levels of significance yet for different goals, and p-values, errors, alternative hypotheses, and significance only apply to one procedure yet are commonly used with both. I also propose a reconceptualization of the procedures to prevent further confusion.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2.
BergerJ. O. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing?Statistical Science, 18(1), 1–32.
3.
ChristensenR. (2005). Testing Fisher, Neyman, Pearson, and Bayes. The American Statistician, 59(2), 121–126.
4.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
5.
CohenJ. (1994). The Earth is round (p < .05). American Psychologist, 49(12), 997–1003.
6.
CortinaJ. M.DunlapW. P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2(2), 161–172.
7.
FisherR. A. (1925). Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.
8.
FisherR. A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society, Series B (Methodological), 17(1), 69–78.
9.
FisherR. A. (1959). Statistical methods and scientific inference (2nd ed.). Edinburgh, UK: Oliver and Boyd.
10.
FisherR. A. (1960). The design of experiments (7th ed.). Edinburgh, UK: Oliver and Boyd.
11.
FrickR. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1(4), 379–390.
12.
GigerenzerG. (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587–606.
13.
GoodmanS. N. (1993). P values, hypothesis tests, and likelihood: Implications for epidemiology of a neglected historical debate. American Journal of Epidemiology, 137(5), 485–496.
14.
GoodmanS. N. (1999). Toward evidence-based medical statistics: 1. The p-value fallacy. Annals of Internal Medicine, 130(12), 995–1004.
15.
HagerW. (2013). The statistical theories of Fisher and of Neyman and Pearson: A methodological perspective. Theory & Psychology, 23, 251–270. doi:10.1177/0959354312465483
16.
HalpinP. F.StamH. J. (2006). Inductive inference or inductive behavior: Fisher and Neyman–Pearson approaches to statistical testing in psychological research (1940–1960). American Journal of Psychology, 119(4), 625–653.
17.
HubbardR. (2004). Alphabet soup: Blurring the distinctions between p’s and α’s in psychological research. Theory & Psychology, 14, 295–327. doi:10.1177/0959354304043638
18.
HubbardR. (2011). The widespread misinterpretation of p-values as error probabilities. Journal of Applied Statistics, 38(11), 2617–2626.
19.
HubbardR.BayarriM. J. (2003). Confusion over measures of evidence (p’s) versus errors (α’s) in classical statistical testing. The American Statistician, 57(3), 171–178. doi:10.1198/0003130031856
20.
HubertyC. J. (1993). Historical origins of statistical testing practices: The treatment of Fisher versus Neyman–Pearson views in textbooks. Journal of Experimental Education, 61(4), 317–333.
21.
JohnstoneD. J. (1986). Tests of significance in theory and practice. Journal of the Royal Statistical Society, Series D (The Statistician), 35(5), 491–504.
22.
JonesL. V.TukeyJ. W. (2000). A sensible formulation of the significance test. Psychological Methods, 5(4), 411–414. doi:10.1037//1082–989X.5.4.411
23.
KlineR. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: APA.
24.
KruegerJ. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56(1), 16–26. doi:10.1037//0003–066X.56.1.16
25.
LehmannE. I. (1993). The Fisher, Neyman–Pearson theories of testing hypotheses: One theory or two?Journal of the American Statistical Association, 88(424), 1242–1249.
26.
LindquistE. F. (1940). Statistical analysis in educational research. Boston, MA: Houghton Mifflin.
27.
LouçãF. (2008). The widest cleft in statistics: How and why Fisher opposed Neyman and Pearson (Working Papers WP/02/2008/DE/UECE). Lisbon, Portugal: School of Economics and Management, Technical University of Lisbon. Retrieved from https://www.repository.utl.pt/bitstream/10400.5/2327/1/wp022008.pdf
28.
NeymanJ. (1953). First course in probability and statistics. New York, NY: Henry Holt.
29.
NeymanJ.PearsonE. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20(A), 175–263.
30.
NeymanJ.PearsonE. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337.
31.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5(2), 241–301. doi:10.1037//1082–989X.5.2.241
32.
PearsonE. S. (1955). Statistical concepts in the relation to reality. Journal of the Royal Statistical Society, Series B (Methodological), 17(2), 204–207.
33.
RosnowR. L.RosenthalR. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276–1284.
34.
SchmidtF. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173–1181.
35.
SchwederT. (1988). A significant version of the basic Neyman–Pearson theory for scientific hypothesis testing. Scandinavian Journal of Statistics, 15(4), 225–242.
36.
WainerH. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 4(2), 212–213.
37.
WilkinsonL., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604.
