Most basic researchers who collect data do so with the goal of testing theories. However, there is disagreement among realists versus pragmatists about whether theories are best characterized in terms of truth or verisimilitude, or in terms of problem-solving ability. Nonetheless, authorities in both philosophical camps agree that empirical hypotheses can be true or false. Consequently, tests of empirical hypotheses are straightforward. In contrast, the present thesis is that even tests of empirical hypotheses may be less straightforward than researchers appreciate. Gain-probability thinking can clarify crucial caveats and qualifications.
AzzaliniA. (1985). A class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 12(2), 171–178. https://www.jstor.org/stable/4615982
2.
AzzaliniA. (2014). The skew-normal and related families (Vol. 3). Cambridge University Press.
BlancaM. J.ArnauJ.López-MontielD.BonoR.BendayanR. (2013). Skewness and kurtosis in real data samples. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 9(2), 78–84. https://doi.org/10.1027/1614-2241/a000057
CamererC. F.DreberA.HolzmeisterF.HoT.-H.HuberJ.JohannessonM.KirchlerM.NaveG.NosekB. A.PfeifferT.AltmejdA.ButtrickN.ChanT.ChenY.ForsellE.GampaA.HeikenstenE.HummerL.ImaiT., . . . WuH. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behavior, 2(9), 637–644. https://doi.org/10.1038/s41562-018-0399-z
7.
CialdiniR. B.SchallerM.HoulihanD.ArpsK.FultzJ.BeamanA. L. (1987). Empathy-based helping: Is it selflessly or selfishly motivated?Journal of Personality and Social Psychology, 52(4), 749–758. https://doi.org/10.1037/0022-3514.52.4.749
8.
DawidR. (2006). Underdetermination and theory succession from the perspective of string theory. Philosophy of Science, 73(3), 298–322. https://doi.org/10.1086/515415
9.
DuhemP. (1954). The aim and structure of physical theory (WienerP., Trans.). Princeton University Press. https://doi.org/10.1515/9780691233857 (Original work published 1914)
10.
EinsteinA. (1961). Relativity: The special and general theory (LawsonR. W., Trans.). Bonanza Books.
11.
EllisG.SilkJ. (2014). Scientific method: Defend the integrity of physics. Nature, 516(7531), 321–323. https://doi.org/10.1038/516321a
12.
FiedlerK.ErmarkF. (2023). Replication in social and personality psychology. In ReisH. T.WestT.JuddC. M. (Eds.), Handbook of research methods in social and personality psychology (3rd ed.). Cambridge University Press.
13.
FiedlerK.TrafimowD. (2024). Using theoretical constraints and the TASI taxonomy to delineate predictably replicable findings. Psychonomic Bulletin & Review, 31, 2581–2598. https://doi.org/10.3758/s13423-024-02521-4
14.
FunderD. C.OzerD. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245920979282
15.
GreenlandS. (2019). Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values. The American Statistician, 73(Supp. 1), 106–114. https://doi.org/10.1080/00031305.2018.1529625
16.
GriceJ. W.MedellinE.JonesI.HorvathS.McDanielH.O’IansenC.BakerM. (2020). Persons as effect sizes. Advances in Methods and Practices in Psychological Science, 3(4), 443–455. https://doi.org/10.1177/2515245920922982
17.
HoA. D.YuC. C. (2015). Descriptive statistics for modern test score distributions: Skewness, kurtosis, discreteness, and ceiling effects. Educational and Psychological Measurement, 75(3), 365–388. https://doi.org/10.1177/0013164414548576
18.
Iso-AholaS. E. (2024). Science of psychological phenomena and their testing. American Psychologist. Advance online publication. https://doi.org/10.1037/amp0001362
19.
LakatosI. (1970). Falsification and the methodology of scientific research programs. In LakatosI.MusgraveA. (Eds.), Criticism and the growth of knowledge (pp. 91–195). Cambridge University Press.
20.
LakatosI. (1978). The methodology of scientific research programmes: Vol. 1. Philosophical papers. Cambridge University Press.
21.
LaudanL. (1977). Progress and its problems. Routledge & Kegan Paul.
22.
LaudanL. (1984). Science and values: The aims of science and their role in scientific debate. University of California Press. https://doi.org/10.1086/289329
MaxwellS. E.KelleyK.RauschJ. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563. https://doi.org/10.1146/annurev.psych.59.103006.093735
26.
MeehlP. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1
27.
MeehlP. E. (1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In HarlowL.MulaikS. A.SteigerJ. H. (Eds.), What if there were no significance tests? (pp. 393–425). Lawrence Erlbaum Associates.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
30.
PopperK. R. (1959). The logic of scientific discovery. Basic Books.
31.
PopperK. R. (1963). Conjectures and refutations. Routledge.
32.
PopperK. R. (1972). Objective knowledge. Oxford University Press.
33.
PopperK. R. (1983). Realism and the aim of science. Routledge.
34.
RosenthalR.RubinD. B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74(2), 166–169. https://doi.org/10.1037/0022-0663.74.2.166
35.
StephanW. G.StephanC. W. (2000). An integrated threat theory of prejudice. In OskampS. (Ed.), Reducing prejudice and discrimination (pp. 23–45). Lawrence Erlbaum Associates.
36.
TateC. U. (2015). On the overuse and misuse of mediation analysis: It may be a matter of timing. Basic and Applied Social Psychology, 37(4), 235–246. https://doi.org/10.1080/01973533.2015.1062380
37.
TrafimowD. (2012). The role of auxiliary assumptions for the validity of manipulations and measures. Theory & Psychology, 22(4), 486–498. https://doi.org/10.1177/0959354311429996
38.
TrafimowD. (2019). A taxonomy of model assumptions on which P is based and implications for added benefit in the sciences. International Journal of Social Research Methodology, 22(6), 571–583. https://doi.org/10.1080/13645579.2019.1610592
39.
TrafimowD. (2023). A new way to think about internal and external validity. Perspectives on Psychological Science, 18(5), 1028–1046. https://doi.org/10.1177/17456916221136117
40.
TrafimowD. (2024). Methodological issues in psychology: Concept, method, and measurement. Routledge.
41.
TrafimowD.FiedlerK. (2024). An exploration of physics envy with implications for desiderata of psychology theories. American Psychologist. Advance online publication. https://doi.org/10.1037/amp0001416.
42.
TrafimowD.RothN.XuL.ToomasianD.PerilloA.TongT.WangT.ChoyS. T.ChenX.WangC.HuL. (2023). Surprising implications of differences in locations versus differences in means. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 19(2), 152–169. https://doi.org/10.5964/meth.10969
43.
TrafimowD.TongT.WangT.ChoyS. T. B.HuL.ChenX.WangC.WangZ. (2024). Improving inferential analyses pre-data and post-data. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000697
44.
WhewellW. (1840). Philosophy of the inductive sciences. Parker.
