Most of the debates around statistical testing suffer from a failure to identify clearly the features specific to the theories invented by Fisher and by Neyman and Pearson. These features are outlined. The hybrids of Fisher’s and Neyman–Pearson’s theory are briefly addressed. The lack of random sampling and its consequences for statistical inference are also highlighted, leading to the recommendation to dispense with inferences and perform approximate randomization tests instead. A possible scheme for the appraisal of substantive hypotheses is offered, the corroboration of which is a necessary prerequisite for scientific explanations and predictions. The scheme is partly based on the Neyman–Pearson theory. This theory, though not perfect, is superior to its competitors, especially when examining substantive hypotheses. The many statistical and extra-statistical decisions prior to experimentation and the inevitable subjectivity of our research endeavors are emphasized. If feasible, statistical problems should be discussed from an extra-statistical methodological/epistemological viewpoint.
AbelsonR. P. (1995). Statistics as principled argument. Hillsdale, NJ: Erlbaum.
2.
AbelsonR. P. (1997). On the surprising longevity of flogged horses: Why there is a case for significance tests. Psychological Science, 8, 12–15.
3.
AndersonM. J.Ter BraakC. F. J. (2003). Permutation tests for multifactorial analysis of variance. Journal of Statistical Computation and Simulation, 73, 85–113.
4.
BerensonM. L. (1982). A comparison of several k sample tests for ordered alternatives in completely randomized designs. Psychometrika, 47, 265–280.
5.
BoikR. J. (1987). The Fisher–Pitman permutation test: A non-robust alternative to the normal theory F when variances are heterogeneous. British Journal of Mathematical and Statistical Psychology, 40, 26–42.
6.
ChalmersA. F. (1999). What is this thing called science? (3rd ed.). St. Lucia, Australia: University of Queensland Press.
7.
ChowS. L. (1996). Statistical significance: Rationale, validity and utility. London, UK: Sage.
8.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
9.
CohenJ. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
10.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
11.
CookT. D.CampbellD. T. (1979). Quasi-experimentation. Boston, MA: Houghton Mifflin.
12.
CortinaJ. M.DunlapW. P. (1997). On the logic and purpose of significance testing. Psychological Methods, 2, 161–172.
13.
CowlesM. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Erlbaum.
14.
DavisonA. C.HinkleyD. (2006). Bootstrap methods and their application (8th ed.). Cambridge, UK: Cambridge University Press.
15.
EdgingtonE. S. (1966). Statistical inference and nonrandom samples. Psychological Bulletin, 66, 485–487.
16.
EdgingtonE. S. (1969). Approximate randomization tests. The Journal of Psychology, 62, 143–149.
17.
EdgingtonE. S. (1995). Randomization tests (3rd ed.). New York, NY: Dekker.
18.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. Andover, MA: Chapman & Hall.
19.
FalkR.GreenbaumC. W. (1995). Significance tests die hard. Theory & Psychology, 5, 75–98.
20.
FisherR. A. (1935). The logic of inductive inference. Journal of the Royal Statistical Society, 98, 39–82.
21.
FisherR. A. (1947). Development of the theory of experimental design. Proceedings of the International Statistical Conferences, 3, 434–439.
22.
FisherR. A. (1950). Statistical methods for research workers (11th ed.). Edinburgh, UK: Oliver & Boyd.
23.
FisherR. A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society, B, 17, 69–78.
24.
FisherR. A. (1956). Statistical methods and scientific inference. Edinburgh, UK: Oliver & Boyd.
25.
FisherR. A. (1966). The design of experiments (8th ed.). Edinburgh, UK: Oliver & Boyd.
26.
FrickR. W. (1995). Accepting the null hypothesis. Memory & Cognition, 23, 132–138.
27.
FrickR. W. (1998). Interpreting statistical testing: Process and propensity, not population and random sampling. Behavior Research Methods, Instruments, and Computers, 30, 527–535.
28.
FrickR. W. (1999). Defending the statistical staus quo. Theory & Psychology, 9, 183–189.
29.
GigerenzerG. (1987). Probabilistic thinking and the fight against subjectivity. In KrügerL.GigerenzerG.MorganM. (Eds.), The probabilistic revolution: Vol. 2. Ideas in the sciences (pp. 11–33). Cambridge, MA: MIT Press.
30.
GigerenzerG. (1993). The superego, the ego, and the id in statistical reasoning. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
31.
GigerenzerG.MurrayD. J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
32.
GonzalezR. (1994). The statistics ritual in psychological research. Psychological Science, 5, 321, 325–328.
33.
HackingI. (1965). Logic of statistical inference. Cambridge, UK: Cambridge University Press.
34.
HagerW. (2004). Testplanung zur statistischen Prüfung psychologischer Hypothesen [Power analysis for the statistical examination of psychological hypotheses]. Göttingen, Germany: Hogrefe.
35.
HalpinP. F.StamH. J. (2006). Inductive inference or inductive behavior: Fisher and Neyman–Pearson approaches to statistical testing in psychological research (1940–1960). American Journal of Psychology, 119, 625–653.
36.
HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.). (1997). What if there were no significance tests?Mahwah, NJ: Erlbaum.
HaysW. L. (1994). Statistics (5th ed.). Belmont, CA: Wadsworth.
39.
HubbardR. (2004). Alphabet soup: Blurring the distinction between p’s and α’s in psychological research. Theory & Psychology, 14, 295–327.
40.
HubertyC. J. (1987). On statistical testing. Educational Researcher, 16(8), 4–9.
41.
HubertyC. J. (1993). Historical origins of statistical testing practices: The treatment of Fisher vs. Neyman–Pearson in textbooks. Journal of Experimental Education, 61, 317–333.
42.
HubertyC. J.PikeC. J. (1999). On some history regarding statistical testing. In ThompsonB. (Ed.), Adcances in social science methodology (Vol. 5, pp. 1–22). Stamford, CT: JAI Press.
43.
HunterM. A.MayR. B. (2003). Statistical testing and null distributions: What to do when samples are not random?Canadian Journal of Experimental Psychology, 57, 176–188.
44.
KilleenP. R. (2005). An alternative to null hypothesis significance tests. Psychological Science, 16, 345–353.
45.
KraemerH. C.ThiemannS. (1987). How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.
46.
KuhnT. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago, IL: University of Chicago Press.
47.
LakatosI. (1978). The methodology of scientific research programmes. In WorrallJ.CurrieG. (Eds.), Imre Lakatos: Philosophical papers (Vol. 1, pp. 8–101). Cambridge, UK: Cambridge University Press.
48.
MayoD. G. (1996). Error and the growth of experimental knowledge. Chicago, IL: University of Chicago Press.
49.
MeehlP. E. (1967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.
50.
MeehlP. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress in psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
51.
MeehlP. E. (1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 393–425). Mahwah, NJ: Erlbaum.
52.
NeymanJ. (1942). Basic ideas and some recent results of the theory of testing statistical hypotheses. Journal of the Royal Statistical Society, 105, 292–327.
53.
NeymanJ. (1950). First course in probability and statistics. New York, NY: Holt, Rinehart & Winston.
54.
NeymanJ. (1952). Lectures and conferences on mathematical statistics and probability (2nd ed.). Washington, DC: U.S. Department of Agriculture.
55.
NeymanJ. (1957). “Inductive behavior” as a basic concept of philosophy of science. Revue d’Institute Internationale de Statistique, 25, 7–22.
56.
NeymanJ. (1967). R. A. Fisher (1890–1962): An appreciation. Science, 156, 1456–1460.
57.
NeymanJ. (1971). Foundations of behavioristic statistics (with comments). In GodambeV. P.SprottD. A. (Eds.), Foundations of statistical inference (pp. 1–19). Toronto, Canada: Holt, Rinehart & Winston.
58.
NeymanJ. (1976). Tests of statistical hypotheses and their use in studies of natural phenomena. Communications in Statistics—Theory and Methods, A5, 737–751.
59.
NeymanJ. (1977). Frequentist probability and frequentist statistics. Synthese, 36, 97–131.
60.
NeymanJ.PearsonE. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika, 20A, Pt. I: 175–240; Pt. II: 263–294.
61.
NeymanJ.PearsonE. S. (1933a). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society, A, 231, 289–337.
62.
NeymanJ.PearsonE. S. (1933b). The testing of hypotheses in relation to probabilities a priori. Proceedings of the Cambridge Philosophical Society, 29, 492–510.
63.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
64.
OakesM. (1986). Statistical inference: A comment for the social and behavioural sciences. Chichester, UK: Wiley.
65.
PaivioA. (1971). Imagery and verbal processes. New York, NY: Holt, Rinehart & Winston.
66.
PaivioA. (1990). Mental representations. A dual coding approach (2nd ed.). New York, NY: Oxford University Press.
67.
PearsonE. S. (1955). Statistical concepts in their relation to reality. Journal of the Royal Statistical Society, B, 17, 204–207.
68.
PearsonE. S. (1962). Some thoughts of statistical inference. Annals of Mathematical Statistics, 33, 394–403.
69.
PolkinghorneD. (1983). Methodology for the human sciences: Systems of inquiry. Albany: State University of New York Press.
70.
PopperK. R. (1981). Conjectures and refutations: The growth of scientific knowledge (4th ed.). London, UK: Routledge & Kegan Paul.
71.
PopperK. R. (1992). The logic of scientific discovery. London, UK: Hutchinson. (Original work published 1934)
72.
PreeceD. A. (1982). The design and analysis of experiments: What has gone wrong?Utilitas Mathematica, Series A, 21, 201–244.
73.
PruzekR. M. (1997). An introduction to Bayesian inference and its applications. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 287–318). Mahwah, NJ: Erlbaum.
74.
ReichardtC. S.GollobH. F. (1999). Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. Psychological Methods, 4, 117–128.
75.
RindskopfD. M. (1997). Testing “small,” not null, hypotheses: Classical and Bayesian approaches. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 319–332). Mahwah, NJ: Erlbaum.
76.
RosnowR. L.RosenthalR. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
77.
SchefféH. (1959). The analysis of variance. New York, NY: Wiley.
78.
SchmidtF. L.HunterJ. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Erlbaum.
79.
SerlinR. C.LapsleyD. K. (1993). Rational appraisal of psychological research and the good-enough principle. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 199–228). Hillsdale, NJ: Erlbaum.
80.
SpielmanS. (1973). A refutation of the Neyman–Pearson theory of testing. British Journal for the Philosophy of Science, 24, 201–222.
81.
StillA. W.WhiteA. F. (1981). The approximate randomization test as an alternative to the F test in analysis of variance. British Journal of Mathematical and Statistical Psychology, 34, 243–252.
82.
ThompsonB. (1993). The use of statistical significance tests in research: Bootstraps and other alternatives. Journal of Experimental Education, 61, 361–377.
83.
WestermannR. (2000). Wissenschaftstheorie und Experimentalmethodik [Philosophy of science and experimental methods]. Göttingen, Germany: Hogrefe.
84.
WilkinsonL., and the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
85.
WinerB. J.BrownD. R.MichelsK. M. (1991). Statistical principles in experimental design (3rd ed.). New York, NY: McGraw-Hill.
