Some Reflections on the Many Labs 2 Replication of Norenzayan,Smith,Kim,and Nisbett’s (2002) Study 2: Cultural Preferences for Formal Versus Intuitive Reasoning

In an ambitious effort to assess the evidentiary value of 28 published findings across 36 countries and territories, Klein et al (2018, this issue) replicated Study 2 of Norenzayan, Smith, Kim, and Nisbett (2002). Here, I reflect on their effort and offer thoughts and clarifications.

The original study had two hypotheses. The first was that we would find an overall greater preference for the rule-based strategy in the categorization (belong-to) condition and an overall greater preference for the family-resemblance-based strategy in the similarity-judgment (similar-to) condition. The second was that we would find less preference for the rule-based strategy and more preference for the family-resemblance-based strategy in the East Asian sample than in the European American sample in both experimental conditions (with Asian Americans in between), a hypothesis derived from the theory of holistic versus analytic cognition in cultural psychology (Nisbett, Peng, Choi, & Norenzayan, 2001). Our results showed evidence consistent with the first hypothesis, except that the European American sample showed equally high levels of preference for the rule-based strategy in the two experimental conditions. As to the second hypothesis, we concluded that the observed cultural difference in the similar-to condition was consistent with what we had predicted, but, contrary to our prediction, there was no cultural difference in the belong-to condition.

Klein et al. preregistered and tested the hypothesis that the percentage of rule-based responses would be greater in the belong-to condition compared with the similar-to condition, as we had hypothesized in the original study. This aspect of Klein et al.’s efforts followed the original study design closely, and this experimental effect was found in their vastly larger and more diverse cross-cultural sample.

Although the stimuli and experimental instructions in the replication were closely matched to the earlier study, there were notable differences in sampling and implementation in the two studies. In the original study, we used a narrow sampling strategy: The participants were university students at an elite American public university and were closely matched on cognitive abilities and education levels, but differed in cultural background. All participants completed this one judgment task on desktop computers in individual cubicles in a standardized environment. These steps were taken to ensure a high degree of experimental control. But because Klein et al. were pursuing a different set of research questions, 13 or 15 different tasks were administered to each participant in a randomized order, and the samples, which came from a range of countries, varied in age, education, and other sociodemographic characteristics. In addition, the large sample size provided very high statistical power.

Moreover, there were two additional critical divergences in the cross-cultural comparisons, and these differences render Klein et al.’s findings particularly difficult to interpret in relation to the original study’s second hypothesis. First, Klein et al. introduced a novel procedure in their cross-cultural comparisons: On the basis of Henrich, Heine, and Norenzayan’s (2010) concept of WEIRD (Western, educated, industrialized, rich, and democratic) cultures, they computed a WEIRDness score for each country, drawing on the letters of the WEIRD acronym. They then categorized the individual samples as WEIRD or less WEIRD and compared these two groups. This approach is not derived from Henrich et al.’s conceptualization of WEIRD sampling in psychology, nor is it found in the original study. There are many reasons to be skeptical of its merits. Instead, WEIRDness can be more appropriately and precisely conceptualized as a measure of cultural distance (Muthukrishna et al., 2018); moreover, the variation in WEIRDness is multidimensional, which is why it is crucial to theoretically identify the precise basis and level of cultural variability as they apply to a particular psychological phenomenon (see Henrich et al., 2010; Norenzayan & Heine, 2005). This is a broader issue that deserves detailed discussion, but it is beyond the scope of this brief Commentary.

Second, the hypothesis we tested was whether European Americans would be more likely than East Asians to prefer the rule-based strategy in both the belong-to and the similar-to conditions. But Klein et al. tested a different hypothesis: whether the magnitude of the effect of experimental instruction (“belong to” vs. “similar to”) showed heterogeneity across all samples and, in an exploratory test, whether this effect was different for samples classified as WEIRD and those classified as less WEIRD. They found that this effect was stronger for WEIRD than for less WEIRD samples. Although this was not a question of interest in the original study, it is a noteworthy finding in and of itself, considering that a recent study by Murphy, Bosch, and Kim (2017) failed to show this effect of experimental instruction in five American samples (a result consistent with the original study’s findings for the European American sample). But Murphy et al. did observe this effect in a Korean sample. Thus, the pattern of results Murphy et al. obtained is at odds with the findings of Klein et al. (Note, however, that Murphy et al.’s sample size, like the sample size in the original study, is dwarfed in comparison with the massive sample size in Many Labs 2.) As far as I am aware, there are no a priori theoretical reasons from cultural psychology that would explain a cultural difference in responsiveness to experimental instruction; it is an unexpected finding that might be of interest to researchers studying cultural variability in cognition.