A Bayesian perspective on Ioannidis’s (2005) memorable statement that “Most Published Research Findings Are False” suggests a seemingly inescapable trade-off: It appears as if research hypotheses are based either on safe ground (high prior odds), yielding valid but unsurprising results, or on unexpected and novel ideas (low prior odds), inspiring risky and surprising findings that are inevitably often wrong. Indeed, research of two prominent types, sexy hypothesis testing and model testing, is often characterized by low priors (due to astounding hypotheses and conjunctive models) as well as low-likelihood ratios (due to nondiagnostic predictions of the yin-or-yang type). However, the trade-off is not inescapable: An alternative research approach, theory-driven cumulative science, aims at maximizing both prior odds and diagnostic hypothesis testing. The final discussion emphasizes the value of pluralistic science, within which exploratory phenomenon-driven research can play a similarly strong part as strict theory-testing science.
AlloyL. B.AbramsonL. Y. (1979). Judgment of contingency in depressed and nondepressed students: Sadder but wiser?Journal of Experimental Psychology: General, 108, 441–485.
2.
BellR.BuchnerA. (2012). How adaptive is memory for cheaters?Current Directions in Psychological Science, 21, 403–408.
3.
BjorkR. A. (1994). Memory and metamemory considerations in the training of human beings. In MetcalfeJ.ShimamuraA. P.MetcalfeJ.ShimamuraA. P. (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: The MIT Press.
4.
BrandstätterE.GigerenzerG.HertwigR. (2006). The priority heuristic: Making choices without tradeoffs. Psychological Review, 113, 409–432.
5.
BrunswikE. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193–217.
6.
CampbellD. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.
7.
CampbellD. T. (1966). Pattern matching as an essential in distal knowing. In HammondK. R. (Ed.), The psychology of Egon Brunswik (pp. 81–106). New York, NY: Holt, Rinehart and Winston.
8.
CampbellD. T.KennyD. A. (1999). A primer on regression artifacts. New York, NY: Guilford Press.
9.
CummingG. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
10.
DannerD.HagemannD.FiedlerK. (2015). Mediation analysis with structural equation models: Combining theory, design, and statistics. European Journal of Social Psychology, 45, 460–481.
11.
De HouwerJ. (2007). A conceptual and theoretical analysis of evaluative conditioning. The Spanish Journal of Psychology, 10, 230–241.
12.
DenrellJ. (2005). Why most people disapprove of me: Experience sampling in impression formation. Psychological Review, 112, 951–978.
13.
DenrellJ.Le MensG. (2011). Seeking positive experiences can produce illusory correlations. Cognition, 119, 313–324.
14.
DhamiM. K.HertwigR.HoffrageU. (2004). The role of representative design in an ecological approach to cognition. Psychological Bulletin, 130, 959–988.
15.
DiekmannA. (2011). Are most published research findings false?Jahrbücher für Nationalökonomie und Statistik, 321, 628–635.
16.
DoughertyM. P.GettysC. F.OgdenE. E. (1999). MINERVA-DM: A memory processes model for judgments of likelihood. Psychological Review, 106, 180–209.
ErevI.WallstenT. S.BudescuD. V. (1994). Simultaneous over-and underconfidence: The role of error in judgment processes. Psychological Review, 101, 519–527.
19.
FeldG. B.BornJ. (2015, May29). Exploiting sleep to modify bad attitudes. Science, 348, 971–972.
20.
FiedlerK. (1991). The tricky nature of skewed frequency tables: An information loss account of distinctiveness-based illusory correlations. Journal of Personality and Social Psychology, 60, 24–36.
21.
FiedlerK. (1996). Explaining and simulating judgment biases as an aggregation phenomenon in probabilistic, multiple-cue environments. Psychological Review, 103, 193–214.
22.
FiedlerK. (2000). Beware of samples! A cognitive-ecological sampling approach to judgment biases. Psychological Review, 107, 659–676.
23.
FiedlerK. (2008). The ultimate sampling dilemma in experience-based decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 186–203.
24.
FiedlerK. (2010). How to study cognitive decision algorithms: The case of the priority heuristic. Judgment and Decision Making, 5, 21–32.
25.
FiedlerK.ArmbrusterT. (1994). Two halfs may be more than one whole: Category-split effects on frequency illusions. Journal of Personality and Social Psychology, 66, 633–645.
26.
FiedlerK.FreytagP.MeiserT. (2009). Pseudocontingencies: An integrative account of an intriguing cognitive illusion. Psychological Review, 116, 187–206.
27.
FiedlerK.KruegerJ. I. (2012). More than an artifact: Regression as a theoretical construct. In KruegerJ. I.KruegerJ. I. (Eds.), Social judgment and decision making (pp. 171–189). New York, NY: Psychology Press.
28.
FiedlerK.KutznerF.KruegerJ. I. (2012). The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661–669.
29.
FiedlerK.SchottM.MeiserT. (2011). What mediation analysis can (not) do. Journal of Experimental Social Psychology, 47, 1231–1236.
30.
FiedlerK.UnkelbachC. (2014). Regressive judgment: Implications of a universal property of the empirical world. Current Directions in Psychological Science, 23, 361–367.
31.
FiedlerK.UnkelbachC.FreytagP. (2009). On splitting and merging categories: A regression account of subadditivity. Memory & Cognition, 37, 383–393.
32.
FurbyL. (1973). Interpreting regression toward the mean in developmental research. Developmental Psychology, 8, 172–179.
33.
GaltonF. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263.
34.
GigerenzerG.GoldsteinD. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103, 650–669.
35.
GildenD. L. (2001). Cognitive emissions of 1/f noise. Psychological Review, 108, 33–56.
36.
GoldvargE.Johnson-LairdP. N. (2001). Naive causality: A mental model theory of causal meaning and reasoning. Cognitive Science, 25, 565–610.
37.
GonzalezC.DuttV. (2011). Instance-based learning: Integrating sampling and repeated decisions from experience. Psychological Review, 118, 523–551.
38.
HannulaD. E.RanganathC. (2009). The eyes have it: Hippocampal activity predicts expression of memory in eye movements. Neuron, 63, 592–599.
39.
HigginsE. T. (1992). Increasingly complex but less interesting articles: Scientific progress or regulatory problem?Personality and Social Psychology Bulletin, 18, 489–492.
40.
HigginsE. T. (2004). Making a theory useful: Lessons handed down. Personality and Social Psychology Review, 8, 138–145.
41.
HintzmanD. L. (1974). Theoretical implications of the spacing effect. In SolsoR. L.SolsoR. L. (Eds.), Theories in cognitive psychology: The Loyola Symposium (pp. 78–99). Oxford, England: Erlbaum.
42.
HoffmanP. J. (1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57, 116–131.
43.
IoannidisJ. P. (2005). Why most published research findings are false. Chance, 18(4), 40–47.
44.
JuslinP.PerssonM. (2002). PROBabilities from EXemplars (PROBEX): A “lazy” algorithm for probabilistic inference from generic knowledge. Cognitive Science, 26, 563–607.
45.
KassinS. M. (2008). False confessions: Causes, consequences, and implications for reform. Current Directions in Psychological Science, 17, 249–253.
46.
KatsikopoulosK. V.GigerenzerG. (2008). One-reason decision-making: Modeling violations of expected utility theory. Journal of Risk and Uncertainty, 37, 35–56.
47.
KleinS. B. (2012). A role for self-referential processing in tasks requiring participants to imagine survival on the savannah. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38, 1234–1242.
48.
KouchakiM.OveisC.GinoF. (2014). Guilt enhances the sense of control and drives risky judgments. Journal of Experimental Psychology: General, 143, 2103–2110.
49.
KruegerJ.MuellerR. (2002). Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance. Journal of Personality and Social Psychology, 82, 180–188.
50.
KruglanskiA. W. (2001). That “vision thing”: The state of theory in social and personality psychology at the edge of the new millennium. Journal of Personality and Social Psychology, 80, 871–875.
51.
KutznerF. L.FiedlerK. (2015). No correlation, no evidence for attention shift in category learning: Different mechanisms behind illusory correlations and the inverse base-rate effect. Journal of Experimental Psychology: General, 144, 58–75.
52.
LakatosI. (1970). Falsification and the methodology of scientific research programmes. In LakatosI.MusgraveA. (Eds.), Criticism and the growth of knowledge (pp. 91–195). London, England: Cambridge University Press.
53.
LefcourtH. M.Davidson-KatzK.KuenemanK. (1990). Humor and immune-system functioning. Humor: International Journal of Humor Research, 3, 305–321.
54.
McGartyC.HaslamS. A.TurnerJ. C.OakesP. J. (1993). Illusory correlation as accentuation of actual intercategory difference: Evidence for the effect with minimal stimulus information. European Journal of Social Psychology, 23, 391–410.
55.
McGrawA. P.MellersB. A.RitovI. (2004). The affective costs of overconfidence. Journal of Behavioral Decision Making, 17, 281–295.
56.
MeehlP. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108–141.
57.
MellersB. A.SchwartzA.HoK.RitovI. (1997). Decision affect theory: Emotional reactions to the outcomes of risky options. Psychological Science, 8, 423–429.
58.
MellersB. A.SchwartzA.RitovI. (1999). Emotion-based choice. Journal of Experimental Psychology: General, 128, 332–345.
59.
MooreD. A.HealyP. J. (2008). The trouble with overconfidence. Psychological Review, 115, 502–517.
60.
NairneJ. S.PandeiradaJ. S.ThompsonS. R. (2008). Adaptive memory: The comparative value of survival processing. Psychological Science, 19, 176–180.
61.
OskampS. (1965). Overconfidence in case study judgments. Journal of Consulting Psychology, 29, 261–265.
62.
ParducciA. (1968, December). The relativism of absolute judgments. Scientific American, 219(6), 84–90.
63.
PennebakerJ. W. (1989). Confession, inhibition, and disease. In BerkowitzL.BerkowitzL. (Eds.), Advances in experimental social psychology (Vol. 22, pp. 211–244). San Diego, CA: Academic Press.
64.
ReichenbachH. (1952). Experience and prediction. Chicago, IL: University of Chicago Press. (Original work published 1938)
65.
RobertsS.PashlerH. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107, 358–367.
66.
RoeR. M.BusemeyerJ. R.TownsendJ. T. (2001). Multialternative decision field theory: A dynamic connectionist model of decision making. Psychological Review, 108, 370–392.
67.
RoedigerH. I.MarshE. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 1155–1159.
68.
RoweG.HirshJ. B.AndersonA. K.SmithE. E. (2007). Positive affect increases the breadth of attentional selection. PNAS: Proceedings of the National Academy of Sciences of the United States of America, 104, 383–388.
69.
RulonP. J. (1941). Problems of regression. Harvard Educational Review, 11, 213–223.
70.
SchallerM.ParkJ. H. (2011). The behavioral immune system (and why it matters). Current Directions in Psychological Science, 20, 99–103.
71.
SchmidtF. (2010). Detecting and correcting the lies that data tell. Perspectives on Psychological Science, 5, 233–242.
72.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
SheffieldV. F. (1949). Extinction as a function of partial reinforcement and distribution of practice. Journal of Experimental Psychology, 39, 511–526.
75.
ShermanJ.KruschkeJ.ShermanS.PercyE.PetrocelliJ.ConreyF. (2009). Attentional processes in stereotype formation: A common model for category accentuation and illusory correlation. Journal of Personality and Social Psychology, 96, 305–323.
76.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
77.
SteeleC. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613–629.
78.
StevensS. S. (1957). On the psychophysical law. Psychological Review, 64, 153–181.
79.
StewartN.ChaterN.BrownG. A. (2006). Decision by sampling. Cognitive Psychology, 53, 1–26.
80.
StewartN.ChaterN.StottH. P.ReimersS. (2003). Prospect relativity: How choice options influence decision under risk. Journal of Experimental Psychology: General, 132, 23–46.
81.
SwetsJ. A.DawesR. M.MonahanJ. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.
82.
TangneyJ. P.BaumeisterR. F.BooneA. L. (2004). High self-control predicts good adjustment, less pathology, better grades, and interpersonal success. Journal of Personality, 72, 271–322.
83.
TropeY.ThompsonE. P. (1997). Looking for truth in all the wrong places? Asymmetric search of individuating information about stereotyped group members. Journal of Personality and Social Psychology, 73, 229–241.
84.
TverskyA.KahnemanD. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5, 297–323.
85.
TverskyA.KoehlerD. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101, 547–567.
86.
UnkelbachC.FiedlerK.BayerM.StegmüllerM.DannerD. (2008). Why positive information is processed faster: The density hypothesis. Journal of Personality and Social Psychology, 95, 36–49.
87.
Van LangeP. M. (2013). What we should expect from theories in social psychology: Truth, abstraction, progress, and applicability as standards (TAPAS). Personality and Social Psychology Review, 17, 40–55.
88.
Van RooyD.Van OverwalleF.VanhoomissenT.LabiouseC.FrenchR. (2003). A recurrent connectionist model of group biases. Psychological Review, 110, 536–563.
89.
VerhagenJ.WagenmakersE. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 143, 1457–1475.
90.
WasonP. C. (1960). On the failure to eliminate hypotheses in a conceptual task. The Quarterly Journal of Experimental Psychology, 12, 129–140.
91.
WellsG. L.WindschitlP. D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25, 1115–1125.
92.
WestenD.RosenthalR. (2003). Quantifying construct validity: Two simple measures. Journal of Personality and Social Psychology, 84, 608–618.
93.
WestfallJ.KennyD. A.JuddC. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.
