Robust Evidence for Knowledge Attribution and Luck: A Comment on Hall et al. (2024)

Knowledge is essential to human culture and civilization, and acquiring it constitutes an important cognitive achievement. Psychologists have increasingly become interested in how people credit others with this achievement (Phillips et al., 2021) and particularly whether people think it can ever be achieved through luck.

There are some ways in which knowledge and luck are clearly incompatible. For example, suppose a woman purchases a gem from a flea market, puts it in her pocket, and tells a friend “There’s a diamond in my pocket.” Even if she is right, the woman might not know there is a diamond in her pocket. For example, she might have concluded the gem is a diamond simply because it looks like one, without any other evidence. In this case, she would be right only as a matter of luck.

Turri et al. (2015) investigated how knowledge attributions are affected by many kinds of luck. Their first experiment focused on a protagonist who was lucky in a different way from the example above. This time, suppose the woman bought a diamond from a reputable jewelry store, but then a sneaky pickpocket tried and failed to nab it. The woman’s claim that there is a diamond in her pocket is again correct only as a matter of luck—the woman did not notice the failed attempt, and the thief could have very easily succeeded in taking it from her pocket. Turri et al. found that participants mostly attributed knowledge to protagonists in these types of lucky situations. The researchers then went on to conduct subsequent experiments involving other types of luck and found that people continue to attribute knowledge with some kinds of luck and deny it for others.

Hall et al. (2024) presented a replication of Turri et al.’s (2015) first experiment (i.e., which focused on just one type of luck). They claimed to find different results and to show that people deny knowledge to lucky agents. And they inferred from this finding that there is a common psychological tendency to deny that true-by-luck beliefs are knowledge (sometimes glossed by researchers as “the Gettier intuition”). We worry these claims might leave readers with some misimpressions. To help prevent this, we make three main points: (a) Hall et al.’s results do replicate the original ones; (b) opposite to what they conclude, their participants were largely tolerant of luck and do attribute knowledge to agents in certain lucky cases; and (c) understanding how people attribute knowledge requires abandoning general theories about luck and knowledge and instead focusing on specific mechanisms underlying knowledge attribution. We conclude by suggesting that these discussions may have more general implications about replication in science and the goals of conducting Registered Replication Reports.

Different Methods, Same Results

In Turri et al. (2015), American MTurk workers read a single vignette about a protagonist named Darrel and then answered a forced-choice question about whether he knows or only believes something. Things were different in Hall et al.’s (2024) replication: Participants were primarily university students, a population more likely to be exposed to philosophical debates about knowledge; they were tested in several languages, with textual changes to ease translation; they read three vignettes—a version of the “Darrel” story plus two others; and they responded using Likert scales, which were later treated only dichotomously. In sum, the original and replication had different participant populations, materials, and dependent variables.

Remarkably, this did not seem to affect the findings. In the original, Darrel was seen as knowledgeable by 81% of participants in a no-luck control condition and by 67% in a condition in which his belief was true by luck; these rates did not significantly differ from one another (Turri et al., 2015, Table 1). In Hall et al. (2024), participants saw Darrel as knowledgeable at slightly lower rates—71% versus 60% in the knowledge control condition and true-by-luck condition, respectively (Hall et al., 2024, Table 6). This difference between conditions is smaller than in the original but statistically significant because the replication had a much larger sample.

On the basis of this difference in significance patterns, Hall et al. (2024) claimed to not replicate the original findings. Yet the difference cannot stem from a larger difference between conditions in Hall et al. because this is not what they found—on the contrary, they observed a smaller difference. Things are similar when results from all three of their vignettes are combined. Now the difference between the two main conditions is 14%, which is identical to the difference in Turri et al. (2015). Hence, despite claiming to not replicate Turri et al., the results of Hall et al. are basically the same.

Luck Tolerance in Knowledge Attribution

Hall et al. (2024) claimed that their participants showed the Gettier intuition by denying knowledge when agents were lucky. But if anything, the opposite is true. Simply taking the results at face value, knowledge attribution rates in two of their three lucky vignettes were not below chance. Although participants did not overwhelmingly attribute knowledge to lucky protagonists, they did not deny it either.

However, determining whether knowledge attributions are sensitive to luck requires going beyond face value. It requires examining luck cases relative to closely matched nonlucky controls (i.e., which provide a baseline estimate of people’s inclination to ever attribute knowledge in the first place). One way to do this is to derive scores by dividing lucky-condition rates by those from a no-luck control condition (Starmans & Friedman, 2020). On this measure, lower scores indicate sensitivity to luck; higher ones indicate that attributions are insensitive to luck or that they are luck tolerant. For Turri et al. (2015), the score is 83; for Hall et al. (2024), it is 85 for the Darrel vignette and 75 across all vignettes. In other words, participants in Hall et al. attributed knowledge in lucky-Darrel vignettes 85% of the time they would attribute it in a vignette intended to convey paradigmatic knowledge and at even higher rates than in the original study. Perhaps some participants were intolerant of luck, but this was hardly the norm. When taken together, the fact that such similarly high scores are observed between studies despite differences in materials and methods is powerful evidence for just how luck-tolerant knowledge attribution can sometimes be.

In sum, Hall et al. (2024) did not provide strong evidence for luck intolerance (aka, the Gettier intuition). On the contrary, their findings are among the strongest evidence against the intuition to date. But there is further irony here: Hall et al. reported finding evidence for the Gettier intuition while framing Turri et al. (2015) as not finding it. But although Turri et al.’s first experiment provided evidence against the intuition for one kind of luck, they found evidence for it with other kinds of luck in their subsequent experiments. We turn to the implications of this variation next.

Mechanisms Over Constructs

Questions such as “Do people deny knowledge when beliefs are true by luck?” and “Do people show the Gettier intuition?” are too vague to be answered. Beliefs can be lucky in numerous ways, and scenarios now labeled as “Gettier cases” bear little relation to one another and to those originally discussed by the philosopher Edmond Gettier (1963), who they are named after. Moreover, people are sensitive to these differences. They attribute knowledge for some forms of luck and deny it for others (Beebe & Shea, 2013; Buckwalter, 2014; Colaço et al., 2014; Gonnerman et al., 2023; Kim & Yuan, 2015; Machery, Stich, Rose, Alai, et al., 2017; Machery, Stich, Rose, Chatterjee, et al., 2017; Nagel et al., 2013; Starmans & Friedman, 2012, 2013, 2020; Turri, 2016). The large vignette effects that Hall et al. (2024) found between their different cover stories may even suggest further dimensions or categories of luck.

Attempting to answer vague questions, such as whether people show the Gettier intuition, masks and obfuscates the mechanisms underlying knowledge attribution. This was very much Turri et al.’s (2015) original point. For example, they referred to Gettier cases as a “theoretically useless category” (p. 387) and went on to suggest more fine-grained taxonomies (Blouw et al., 2018) that could be useful. Moving forward, progress will likely require giving up vague questions and labels in favor of focusing on specific forms of luck and developing theories about why each might (or might not) matter when people attribute knowledge.

Implications for Replication Reports

Our concerns about Hall et al. (2024) may have more general implications for metascience that stem from a conflict between two goals of replication. One goal of replication is to directly assess the reliability of published effects, no matter what they mean. A related but distinct goal is to understand the shape and significance of real-world phenomenon more deeply. Although replication initiatives often involve a blend of both goals, there can also be tension between them. For example, replicating one experiment from a target article may allow researchers to begin to assess the reliability of a reported outcome. But the value of this assessment ultimately stems from the meaning and significance researchers associate with the original result. Attempting to answer this larger question about the significance of a result can lead researchers to embrace theoretical frameworks that go beyond observed results, deviate from experimental procedures used by prior researchers, and present data in certain ways when multiple interpretations are possible. The lesson, we suggest, is to appreciate the tension between these two goals from the outset and to carefully consider the choices one makes as a researcher when pursuing them.