It is often stated that results which are not statistically significant at conventional levels (5% is the common benchmark) represent neither evidence against the null hypothesis nor for it. That is, tests of significance have no logical facility to support the hypothesis tested. This logical standpoint appears in the writings of R. A. Fisher and is endorsed in various textbooks. Notwithstanding the position taken by Fisher (who appears later to have changed his mind), statistical theorists, both orthodox and Bayesian, accept that results which are statistically insignificant may represent strong evidence in favor of the null hypothesis, particularly if the sample size is very large (power is high). This is contrary to the interpretation advocated by a great many researchers in applied disciplines. Among those who use classical (Fisherian) tests of significance, Fisher's initial standpoint remains widely accepted.
Get full access to this article
View all access options for this article.
References
1.
BergerJ. O. (1985) Statistical decision theory and Bayesian analysis. (2nd ed.) New York: Springer-Verlag.
2.
BergerJ. O.SellkeT. (1987) Testing a point null hypothesis: the irreconcilability of P values and evidence. (With discussion)Journal of the American Statistical Association, 82, 112–139.
3.
BerksonJ. (1942) Tests of significance considered as evidence. Journal of the American Statistical Association, 37, 325–335.
4.
BirnbaumA. (1977) The Neyman-Pearson theory as decision theory, and as inference theory; with a criticism of the Lindley-Savage argument for Bayesian theory. Synthese, 36, 19–49.
5.
CoxD. R. (1982) Statistical significance tests. British Journal of Clinical Pharmacology, 14, 325–331.
6.
CoxD. R.HinkleyD. V. (1974) Theoretical statistics. London: Chapman & Hall.
7.
DeGrootM. H. (1985) Probability and statistics. (2nd ed.) Reading, MA: Addison-Wesley.
8.
EdwardsW.LindmanH.SavageL. J. (1963) Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242.
9.
FisherR. A. (1936) ‘The coefficient of racial likeness’ and the future of craniometry. Journal of the Royal Anthropological Institute, 66, 57–63.
10.
FisherR. A. (1951) Statistics. In HeathA. E. (Ed.), Scientific thought in the twentieth century. London: Watts. Pp. 31–55.
11.
FisherR. A. (1955) Statistical methods and scientific induction. Journal of the Royal Statistical Society, B, 17, 69–78.
12.
FisherR. A. (1973) Statistical methods and scientific inference. (3rd ed.) New York: Hafner.
13.
GibbonsJ. D.PrattJ. W. (1975) P-values: interpretation and methodology. The American Statistician, 29, 20–25.
14.
GiereR. N. (1972) The significance test controversy. The British Journal for the Philosophy of Science, 23, 170–181.
15.
GoodI. J. (1980) The diminishing significance of a P-value as the sample size increases. Journal of Statistical Computation and Simulation, 11, 307–309.
16.
GoodI. J. (1981) Some logic and history of hypothesis testing. In PittJ. C. (Ed.), Philosophy in economics. Dordrecht: Reidel. Pp. 149–174.
17.
GoodI. J. (1982) Discussion. In G. Shafer, Lindley's paradox. Journal of the American Statistical Association, 77, 325–351.
18.
GoodI. J. (1983) The diminishing significance of a fixed P-value as the sample size increases: a discrete model. Journal of Statistical Computation and Simulation, 16, 312–314.
19.
GoodI. J. (1984) The tolerant Bayesian's interpretation of a tail-area probability. Journal of Statistical Computation and Simulation, 19, 300–302.
20.
GoodI. J. (1985) Weight of evidence: a brief survey. (With discussion) In BernardoJ. M. (Eds.), Bayesian statistics 2. Amsterdam: North Holland. Pp. 249–269.
21.
GreenwaldA. G. (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
22.
JeffreysH. (1939) Theory of probability. Oxford, Eng.: Oxford Univer. Press.
23.
JohnstoneD. J. (1987a) Tests of significance following R. A. Fisher. The British Journal for the Philosophy of Science, 38, 481–499.
24.
JohnstoneD. J. (1987b) On the interpretation of hypothesis tests following Neyman and Pearson. In ViertlR. (Ed.), Probability and Bayesian statistics. New York: Plenum. Pp. 267–277.
25.
JohnstoneD. J. (1988) Comments on Oakes on the foundations of statistical inference in the social and behavioral sciences: the market for statistical significance. Psychological Reports, 63, 319–331.
26.
JohnstoneD. J. (1990a) Consistency and relative plausibility. Journal of Statistical Computation and Simulation, in press.
27.
JohnstoneD. J. (1990b) Sample size and the strength of evidence: a Bayesian interpretation of binomial tests of the information content of qualified audit reports. (With discussion)Abacus, 26, 17–36.
28.
KalbfleischJ. G. (1985) Probability and statistical inference. Vol. 2. Statistical inference. (2nd ed.) New York: Springer-Verlag.
29.
KennedyW. J.GentleJ. E. (1980) Statistical computing. New York: Marcel Dekker.
30.
LindleyD. V. (1957) A statistical paradox. Biometrika, 44, 187–192.
31.
NeymanJ. (1955) The problem of inductive inference. Communications on Pure and Applied Mathematics, 8, 13–45.
32.
OakesM. (1986) Statistical inference: a commentary for the social and behavioural sciences. Chichester, Eng.: Wiley.
33.
PitzG. F. (1968) An example of Bayesian hypothesis testing: the perception of rotary motion in depth. Psychological Bulletin, 70, 252–255.
34.
PrattJ. W. (1961) Review of E. L. Lehmann: ‘Testing Statistical Hypotheses.’ Journal of the American Statistical Association, 56, 163–167.
35.
PrattJ. W. (1976) A discussion of the question: For what use are tests of hypotheses and tests of significance. Communication in Statistics—Theory and Methods, A5, 779–787.
36.
PrattJ. W. (1977) ‘Decisions’ as statistical evidence and Birnbaum's ‘confidence concept.’Synthese, 36, 59–69.
37.
RosenkrantzR. D. (1977) Inference, method and decision: towards a Bayesian philosophy of science. Dordrecht: Reidel. Royall, R. M. (1986) The effect of sample size on the meaning of significance tests. The American Statistician, 40, 313–315.
38.
SeidenfeldT. (1979) Philosophical problems of statistical inference: learning from R. A. Fisher. Dordrecht: Reidel.
39.
SterlingT. D. (1960) What is so peculiar about accepting the null hypotheis. Psychological Reports, 7, 363–364.
40.
TachibanaT. (1989) Reproducibility across experiments of behavioral results obtained under strictly standardized conditions: another view of the results of the Collaborative Behavioral Teratology Study. Psychological Reports, 64, 559–579.
41.
ZellnerA. (1984) Basic issues in econometrics. Chicago, IL: Univer. of Chicago Press.
