Mentioning the Sample’s Country in the Article’s Title Leads to Bias in Research Evaluation

Abstract

Psychology research from Western, educated, industrialized, rich, and democratic (WEIRD) countries, especially from the United States, receives more scientific attention than research from non-WEIRD countries. We investigate one structural way that this inequality might be enacted: mentioning the sample's country in the article title. Analyzing the current publication practice of four leading social psychology journals (Study 1) and conducting two experiments with U.S. American and German students (Study 2), we show that the country is more often mentioned in articles with samples from non-WEIRD countries than those with samples from WEIRD countries (especially the United States) and that this practice is associated with less scientific attention. We propose that this phenomenon represents a (perhaps unintentional) form of structural discrimination, which can lead to underrepresentation and reduced impact of social psychological research done with non-WEIRD samples. We outline possible changes in the publication process that could challenge this phenomenon.

Keywords

culture/ethnicity WEIRD diversity culture structural discrimination publication bias publication practice

Psychological research published in highly ranked journals in the English language is characterized by a lack of diversity in studied populations. An overwhelming proportion of the research includes participants from Western, educated, industrialized, rich, and democratic (WEIRD) countries—mostly the United States, and is authored by U.S. scientists (Cheek, 2017; Rad et al., 2018). As a result, the current psychological knowledge is based on only 11–15% of the world’s population (Arnett, 2008; Cole, 2006; O’Gorman et al., 2012; Sue, 1999; Thalmayer et al., 2021) and is not representative of humanity as a whole (Henrich et al., 2010). This sampling bias not only reduces the generalizability of research findings and their interpretive power (i.e., the ability to understand individuals’ experiences and behaviors in relation to their cultural contexts; Brady et al., 2018), but it also portrays WEIRD populations and their experiences as the prototype or normative center for humanity (Medin et al., 2017).

Structural Inequalities in Psychological Science

Structural inequalities in the field are evident not only in the number of publications but also in citations; journal articles with first authors from the United States appear to receive a disproportionate share of the citations (Brown & Goh, 2016; Cheek, 2017). Therefore, the problem is not just the lack of published non-WEIRD research (from Asia, Africa, Latin America, and the Middle East) but also its low impact on the field. Here, we examined one possible form of structural discrimination that might be undermining the impact of non-WEIRD research: mentioning the sample’s country in the title of the article.

At first glance, in a climate that seeks to diversify psychological science (Apicella et al., 2020; Hruschka et al., 2018), mentioning the name of the sample’s country in the title of an article might seem beneficial. It might highlight the inclusiveness of the journal and the editor’s openness to sample diversity. However, these potential benefits may not outweigh the potential disadvantages. Mentioning a specific country in the title might signal that the article’s findings lack generalizability and, thus, are less relevant to one’s own research. The broadly held view that social psychology as a field ought to predict social phenomena that are relevant to humanity across different contexts and cultures (Rozin, 2006) may lead researchers to avoid—either consciously or unconsciously—reading or citing articles that contain cues that they lack generalizability. Consequently, researchers might avoid including articles that mention a specific country in their own work, which would result in fewer citations and in turn impede the visibility and impact of the research.

Previous research argues that social phenomena cannot be well defined independent of their context (Yin, 1982), even when studying basic cognitive processes (e.g., Nisbett & Masuda, 2003; Nisbett et al., 2001). Culture plays a critical role in shaping cognition, emotion, and behavior, and overlooking cultural influences may seriously jeopardize our understanding of the examined phenomena (Brady et al., 2018). However, if emphasizing the origin of a sample occurs more consistently in articles when samples are drawn from non-WEIRD countries, this may serve to reinforce the view of WEIRD samples as the default category, the norm to which other cultures are compared. Not mentioning the country name in the title when the sample was drawn from a WEIRD country reinforces the view that these samples provide the norm for human behavior as a whole. In other words, samples drawn from WEIRD countries are supposedly human universals, whereas samples from other countries must be marked as “special cases.” Thus, the exclusive and consistent mentioning of non-WEIRD countries in article titles may not be highlighting the importance of cultural variation but instead may be unintendedly exacerbating the appearance of research from non-WEIRD countries as something that is only relevant for those who are interested in specific variants.

As mentioned earlier, psychological science is very much Americanized (Cheek, 2017). According to U.S. sources, more than 90% of psychology’s most eminent researchers have spent their careers in the United States (Diener et al., 2014). Moreover, research, which is published in leading psychology journals, focuses heavily on U.S. participants (Arnett, 2008; although some change has occurred in the 10 years since Arnett’s analysis; Thalmayer et al., 2021). A recent study that analyzed over 5,000 published psychology articles showed that within those papers, the U.S. samples are less often specified in the article titles than samples from other WEIRD and non-WEIRD countries (Cheon et al., 2020). We do not assume that researchers and editors are aware of these practices. It is much more likely that this bias stems from hypocognition—the tendency that privilege is invisible to people who have it (Wu & Dunning, 2020). However, we assume that the practice of mentioning the country in the title reduces the scientific attention an article receives.

The Present Research

In the present research, consisting of two studies, we examined (a) whether mentioning the sample’s country of origin in the title of an article is more prevalent when the sample is from a non-WEIRD country than from a WEIRD country (in line with Cheon et al., 2020) and (b) whether this serves as a subtle mechanism that leads to discrimination in researchers' judgments of the utility of these articles for their own work (manifested in fewer citations and exclusion).

Specifically, in Study 1, we analyzed four leading social psychology journals to examine whether mentioning the sample’s country in the article’s title is more common when the sample is drawn from non-WEIRD countries than from WEIRD countries and especially from the United States (Hypothesis 1). We also tested whether naming the sample’s country in the title is associated with an article accumulating fewer citations which, in combination with H1, would reduce the impact of psychological research from non-WEIRD countries (Hypothesis 2). In Study 2, we experimentally examined whether mentioning a country in the title reduces the likelihood that potential readers will consider the article to be relevant (Hypothesis 3) and will reduce their willingness to integrate the article into their academic work (Hypothesis 4).

Study 1

Method

We selected four leading social psychology and personality outlets (highly ranked by the Social Science Citation Index), including two general journals—Social Psychological and Personality Science (SPPS), Personality and Social Psychology Bulletin (PSPB)—and two focused on gender-related issues—Sex Roles (SR) and Psychology of Women Quarterly (PWQ). To ensure a sufficient number of papers that include samples from a variety of countries, we avoided leading journals, such as Psychological Science and Journal of Personality and Social Psychology, which are known to be strongly focused on the United States and other WEIRD countries (Rad et al., 2018; Thalmayer et al, 2021). Instead, we chose leading psychology journals known to be more inclusive. SPPS, for example, is a collaboration of the Association for Research in Personality, the European Association of Social Psychology, the Society of Experimental Social Psychology, the Society for Personality and Social Psychology, the Asian Association of Social Psychology, and the Society of Australasian Social Psychologists.

In addition, some subfields of psychological science (e.g., women and gender psychology and racial/ethnic psychology) have been more open to sample diversity in an effort to pay attention to cultural factors that might influence social phenomena (Brady et al., 2018; Frieze & Dittrich, 2008). Thus, we also included the two leading journals at the intersection of psychology and gender. In PWQ and SR, authors are often encouraged to elaborate on the cultural context of their sample—both to better understand the social structures that shape people's experiences (Hennessy, 2012; Magnusson & Marecek, 2010), and to reflect on whether the studied constructs and applied measures are relevant for that context. We, therefore, expected that these journals would have more national diversity in their samples than general social psychology journals.

All four journals had similar impact factors (5-year impact factor ranged from 3.171 to 4.039). For each journal, we collected original articles from issues published between January 1, 2015, and December 31, 2017. We chose this interval so enough time would have passed after the debate on the WEIRD bias (7–9 years after the Arnett, 2008, and 5–7 years after the Henrich et al., 2010, publication; see Rad et al., 2018), which might have increased interest in sample diversity. As we were interested in the number of citations, we tracked a period of time that allowed published articles to generate citations.

We only included original articles in our analysis—editorials, introductions to special issues, meta-analyses, and research methods articles were excluded. Articles that included studies from several countries, such as cross-cultural comparisons, were not included in the analysis as they include too many countries to be mentioned in the title (e.g., “examination of X across 62 nations”) or since by definition, they assume that observations from one sample may not generalize to another (e.g., “X influences Y among Japanese but not Americans”). Articles with nonhuman samples and articles without any information regarding the sample’s country were not included in the analysis. Overall, 86 articles were not included in the analysis. The corpus consisted of 855 journal articles (PSPB, n = 310; SPPS, n = 243; SR, n = 211; PWQ, n = 91).

Procedure and Measures

For each article, we first coded the location from where the sample was drawn using eight categories based on schemes used by Arnett (2008) and Rad et al. (2018): Africa, Asia, Europe, Latin America, the Middle East, the United States, other English-speaking countries (Australia, New Zealand, Canada, and the United Kingdom), and Israel. We then collapsed these to produce three categories of country type: (a) non-WEIRD countries (Africa, Asia, Latin America, and the Middle East), (b) WEIRD countries other than the United States¹ (Europe, other English-speaking countries, and Israel), and (c) the United States. If the article included samples from two countries, we considered it to represent the most WEIRD country. We then coded whether the country was mentioned in the title (yes, no). We also coded whether the country was mentioned in the abstract to assure that title mention (rather than any prominent mention) is the key factor influencing citations. This additional analysis concerning the impact of country mention in the abstract and the data files are available on the Open Science Framework (OSF) (osf.io/grpy7). Finally, we recorded Google Scholar citation counts for each article as of March 2020.²

Results

Country Mentions in Title

A χ² test showed that articles from WEIRD countries (including the United States) mention the sample’s country in the title less often (6.70%) than articles from non-WEIRD countries (47.5%), χ²(1) = 80.09, p < .001. Frequencies are presented in Table 1.

Table 1.

Frequencies of Country Mentions in the Title in Different Countries, Study 1.

Country Type		Country not Mentioned in the Title	Country Mentioned in the Title	Total
United States	N	546	34	580
United States	% within country	94.10%	5.90%	580
WEIRD countries other than the United States	N	214	21	235
WEIRD countries other than the United States	% within country	91.10%	8.90%	235
Non-WEIRD countries	N	21	19	40
Non-WEIRD countries	% within country	52.50%	47.5%	40
Total	N	781	74	855
Total	% within all	91.30%	8.70%	855

Note. WEIRD = Western, educated, industrialized, rich, and democratic.

A logistic regression was then conducted, using SPSS Version 25, to examine whether non-WEIRD countries and WEIRD countries other than the United States significantly differ from the United States in mentioning the sample’s country in the title, such that the United States served as the reference category. The difference between articles featuring U.S. samples and articles featuring samples from WEIRD countries other than the United States was not significant (Wald = 2.48, p = .116, 95% CI [−0.51, 0.06]). However, the difference between articles featuring U.S. samples and articles featuring samples from non-WEIRD countries was significant (Wald = 54.47, p < .001, 95% CI [−1.69, −0.98]), such that the odds of the sample’s country being included in the title were 14.53 times higher for articles with samples from non-WEIRD countries in comparison to articles with U.S. samples.

A second logistic regression was then conducted to examine whether WEIRD countries other than the United States and non-WEIRD countries significantly differ from each other. WEIRD countries other than the United States served as the reference category. The difference between the two was significant (Wald = 32.35, p < .001, 95% CI [− 1.49, −0.73]), such that the odds of the sample’s country being included in the title were 9.22 times higher for articles with samples from non-WEIRD countries compared to articles with samples from WEIRD countries other than the United States.

Number of Citations

Next, we examined whether articles that contain the name of the country in the title received fewer citations. A 3 × 2 analysis of covariance (ANCOVA) was conducted with number of citations as the dependent variable. The predictors were country type (United States, WEIRD countries other than the United States, non-WEIRD countries) and whether the country was mentioned in the title (yes, no). Journal type (general/special focus) and year of publication were entered as covariates.

Preliminary analysis revealed that although the assumption of homogeneity of variance was met (p = .208), the data were not normally distributed (Shapiro–Wilk, p < .001). In line with McClelland’s (2014) recommendation, we transformed the data to fit the normality assumptions (Q–Q plots for a set of reasonable transformations are available in the OSF) and removed outliers. To do so, we repeated the statistical test across all possible combinations of reasonable transformations and outlier exclusion options to quantify the degree to which conclusions might change based on arbitrary decisions of the researcher (Steegen et al., 2016; see Cesario et al., 2019, for a similar approach). As can be seen in Table 2, the title effect remained significant in the majority of robustness analyses. In fact, we observed two consequences of applying transformations to satisfy analysis of variance (ANOVA) assumptions. First, we obtained smaller p values for the title effect. Second, although less support was found for the main effect for country, the interaction between title and country was significant in the majority of analyses. Interpreting the interaction using the Bonferroni adjustment for log10 transformation without exclusion of outliers revealed that mentioning the name of the sample's country in the title (in comparison to not mentioning it) was associated with less citations for non-WEIRD (p = .009) and Western countries other than the United States (p = .030), while it did not affect articles with U.S. samples (p = .780). The results of the hypothesized model without transformations or outlier exclusion are presented in Table 3.

Table 2.

Multiverse Analysis Across Options of Transformation, Covariates, and Exclusion of Outliers.

Exclusion Type + Covariates	No Transformation	Sqrt	Log10
All covariates included
No exclusion of outliers	In title: p = .048 Country: p = .091 Interaction: p = .102	In title: p = .014 Country: p = .050 Interaction: p = .039	In title: p = .003 Country: p = .052 Interaction: p = .025
Standardized < 4	In title: p = .036 Country: p = .061 Interaction: p = .011	In title: p = .013 Country: p = .051 Interaction: p = .027	In title: p = .003 Country: p = .052 Interaction: p = .025
Cook distance < 4/N	In title: p = .025 Country: p = .105 Interaction: p = .153	In title: p = .041 Country: p = .039 Interaction: p = .039	In title: p = .006 Country: p = .055 Interaction: p = .012
Excluding journal type as a covariate
No exclusion of outliers	In title: p = .074 Country: p = .111 Interaction: p = .131	In title: p = .024 Country: p = .064 Interaction: p = .055	In title: p = .005 Country: p = .062 Interaction: p = .033
Standardized < 4	In title: p = .044 Country: p = .068 Interaction: p = .013	In title: p = .022 Country: p = .064 Interaction: p = .032	In title: p = .005 Country: p = .062 Interaction: p = .033
Cook distance < 4/N	In title: p = .025 Country: p = .103 Interaction: p = .174	In title: p = .052 Country: p = .048 Interaction: p = .055	In title: p = .013 Country: p = .044 Interaction: p = .029
Reduced model: In title and year
No exclusion of outliers	In title: p = .140	In title: p = .060	In title: p = .015
Standardized < 4	In title: p = .211	In title: p = .065	In title: p = .015
Cook distance < 4/N	In title: p = .018	In title: p = .042	In title: p = .022

Note. For year, p < .001 across all analyses. Interaction = In Title × Country. Each square displays the p values for the main effects and interaction examined in the model.

Table 3.

Analysis of Variance Results for Number of Citations Without Transformations, or Outlier Exclusions, Study 1.

	SS	df	MS	F	p	$η_{p}^{2}$
Intercept	45,274.12	1	45,274.12	84.95	.000	.09
Year	26,817.24	1	26,817.24	50.32	.000	.06
Journal type	615.91	1	615.91	1.16	.283	<.01
Country type	2,556.61	2	1,278.31	2.40	.091	.01
Country mention in title	2,084.64	1	2,084.64	3.91	.048	.01
Country Type × Country mention in the title	2,435.82	2	1,217.91	2.29	.102	.01
Error	451,412.79	847	532.96
Total	1,012,681.00	855

Note. R² = .07 (adjusted R² = .06).

Discussion

Study 1 revealed that articles featuring samples from non-WEIRD countries mentioned the sample’s country in the title more often than articles featuring WEIRD samples, and especially those featuring U.S. samples. This result supports our prediction that non-WEIRD samples are more likely to be marked as “special cases” and is in line with previous research that examined other fields and outlets in psychology (Cheon et al., 2020). Most importantly, our results suggest that mentioning the sample’s country in the title is associated with fewer citations for articles with samples from outside the United States, supporting our claim that these mentions can dampen interest and subsequent use of these articles by other scholars.

Importantly, because articles with WEIRD samples seldom mention the country in the title and almost half of the articles with samples from non-WEIRD countries mention their sample’s country, the possible negative impact of this practice is felt more strongly for articles with samples from non-WEIRD countries.

Study 2

In two preregistered studies, we experimentally examined the possibility that mentioning the sample’s country in the title would affect initial interest in the article. Such practice would be especially harmful for research including non-WEIRD samples, since they are those who are most likely to mention the sample’s country in the title. Study 2a was conducted in the United States and Study 2b was conducted in Germany.

Based on the assumption that our manipulation will produce a medium-sized effect, an a priori power analysis using G*Power (Erdfelder et al., 2009), for four groups within a between-subject ANOVA, revealed that we needed 180 participants to achieve 80% test power at α =.05.

Study 2a - U.S. Sample

Participants

We recruited 188 participants (132 women, 52 men, 4 non-binary/agender, M_age = 19.96, SD = 1.79, range = 18–28) to participate in an online study that supposedly examined how psychology students deal with the large amount of research available today. Participants were recruited from the undergraduate subject pool of a large U.S. Midwestern university in exchange for introductory psychology course credit. In addition, we recruited participants through emails and advertisement on the Psychology School’s website (for a chance to win US$30 lottery). Most participants were BA students (n = 139) and the rest were MA/PhD students. Participants identified as European American (n = 92), Asian American (n = 48), multiracial (n = 11), African American (n = 8), Middle Eastern (n = 8), Latino/a (n = 5), American Indian (n =1), or other (n = 15).

Procedure and Measures

After providing informed consent, participants read the following text: “When we are screening results in Google Scholar, we usually look at the title and make a fast decision about whether the paper is relevant or non-relevant for our purposes.” Participants were asked to imagine that they had to choose articles for a research assignment on behavior change. They were randomly presented with one of four article titles that differed only in terms of whether it mentioned the United States, another WEIRD country (France), a non-WEIRD country (China), or did not mention a country. Participants indicated on a scale, ranging from 1 (not at all) to 5 (very much), how relevant the article was for their research assignment.

Next, participants saw the results of a fictitious Google Scholar search showing the titles of eight real journal articles on behavior change. Two titles mentioned the United States, two mentioned another WEIRD country (France and Italy), two mentioned a non-WEIRD country (Japan and China), and two did not mention a country. Because journal articles that are written by first authors from the United States appear to receive more citations (Brown & Goh, 2016; Cheek, 2017), neither the authors’ names nor the number of citations was presented on the fabricated Google scholar page. Participants were instructed to pick the two articles that they wanted to include in their research assignment. At the end of the study, participants were thanked and debriefed.

Results

Relevance Ratings

An ANOVA was conducted comparing the perceived relevance of the articles across the four conditions. The difference between the four groups was not significant, F(3,184) = 1.12, p = .312, η² _p = .02.

Inclusion Choices

A χ² test was conducted to examine whether there are differences between the observed and expected frequency of choices across the four categories of articles. The results (see Figure 1) suggest a statistically significant difference, χ²(3) = 76.75, p < .001. Pairwise comparisons with Bonferroni adjustments revealed that articles, which mentioned the United States in the title, were chosen more often than the other three categories, χ²(1) > 32.33, p_s < 0.001. No differences were found between the other three categories, χ²(1) < 1.34, p_s > 0.793.³

Figure 1.

American students’ choice pattern, Study 2a. Note. The line represents the expected N.

A series of Binomial probability tests was also conducted, using R, to examine whether the relative frequencies of choosing two, one, or none of the articles from a given category were different from what can be expected under the assumption of randomness. The results, presented in Table 4, revealed that the proportions of choosing one or two articles with U.S. mentions were significantly higher than the expected values. Choosing two articles with no country mentions was also higher than the expected value. The proportions of the other options were lower or not different than the expected values.

Table 4.

Proportion of Participants Choosing Different Numbers of Articles From a Given Category, Study 2a.

Category	Two ArticlesExpected Value = .04		One ArticleExpected Value = .43		No ArticlesExpected Value = .54
Category	Observed	95% CI	Observed	95% CI	Observed	95% CI
No mention	.07**	[.04, .12]	.27***	[.20, .34]	.66***	[.59, .73]
United States	.19***	[.13, .25]	.52*	[.44, .59]	.30***	[.23, .37]
WEIRD non-U.S.	.01	[<.01, .04]	.34**	[.27, .41]	.65**	[.58, .72]
Non-WEIRD	.03	[.01, .06]	.29***	[.22, .36]	.69**	[.61, .75]

Note. N = 188. The probability of randomly choosing both articles from the same category of articles was 2/8 × 1/7, while the probability of choosing a single article from a given category of articles was 1 − (6/8×5/7) − (2/8 × 1/7). The probability of never choosing an article from a given category was (6/8) × (5/7). WEIRD = Western, educated, industrialized, rich, and democratic.

*p < .05. **p < .01. ***p < .001.

Study 2b - German Sample

Participants

We recruited 274 participants (196 women, 77 men, 1 non-binary, M_age = 31.41, SD = 9.83; range = 18–66) from a subject pool of a university in Germany in exchange for course credit. Most participants were BA students (n = 256) and the rest were MA students. The majority were born in Germany (n = 206) and the rest indicated that they had a migration background.

Procedure and Measures

The procedures and materials in Study 2b were identical to those used in Study 2a.

Results

Relevance Ratings

An ANOVA was conducted comparing the perceived relevance of the articles across the four conditions. The difference between the four groups was not significant, F(3,268) = 1.55, p = .202, η² _p = .02.

Inclusion Choices

A χ² test showed a significant difference between the observed and expected distribution of the frequency of choices across the four categories of articles, χ²(3) = 41.02, p < .001 (see Figure 2). Pairwise comparisons with Bonferroni adjustments revealed that the frequency of choosing articles without country mention in the title was higher than all other categories, χ²(1) > 11.99, p_s < 0.001. No differences were found between the other categories, χ²(1) < 5.26, p_s > 0.132.

Figure 2.

German students’ choice pattern, Study 2b. Note. The line represents the expected N.

A series of Binomial probability tests revealed that the proportion of participants choosing two articles with no country mention was higher than the expected value; the remaining proportions were smaller or not different than the expected value. The proportion of participants never choosing articles without country mentions or with U.S. mentions was not different from the expected value, yet choosing articles with WEIRD none U.S. mentions and especially with non-WEIRD mentions, were higher than the expected values. The results are presented in Table 5.

Table 5.

Proportion of Participants Choosing Different Numbers of Articles From a Given Category, Study 2b.

Category	Two ArticlesExpected Value = .04		One ArticleExpected Value = .43		No ArticleExpected Value = .54
Category	Observed	95% CI	Observed	95% CI	Observed	95% CI
No mention	.22***	[.17, .27]	.29***	[.24, .35]	.49	[.43, .55]
United States	.04	[.02, .07]	.42	[.36, .48]	.54	[.48, .60]
WEIRD non-U.S.	.03	[.01, .06]	.35*	[.30, .41]	.61*	[.55, .67]
Non-WEIRD	.04	[.02, .08]	.28***	[.22, .33]	.68***	[.62, .74]

Note. N = 272. WEIRD = Western, educated, industrialized, rich, and democratic.

*p < .05. **p < .01. ***p < .001.

Discussion

Although no differences were found in participants’ perception of the relevance of the papers, the distribution of papers they selected suggests a meaningful pattern of preference across the different categories of articles. Pointing to what appears to be an in-group preference, in Study 2a, American students showed a strong preference for articles mentioning the United States in the title compared to the other three categories of articles. Their tendency to choose two articles with U.S. mentions was especially pronounced. In Study 2b, where the German sample was not offered the option of choosing their own country, participants showed a preference for articles with no country mentioned in the title. Their tendency to choose two articles with no country mentions was especially pronounced. This suggests that when in-group preference is removed, people will prefer articles that suggest greater generalizability of the research by leaving the sample’s country of origin out of the title. Again, this tendency to avoid articles with country mentions in the title will be especially problematic for articles with non-WEIRD samples, as they are more likely to have titles that include the name of the country.

General Discussion

Beyond being underrepresented in psychological science, research that includes non-WEIRD samples also seems to receive less attention. The results of the present research point to an important form of structural (although likely unintentional) discrimination that appears to contribute to this reduced attention and subsequent impact of non-WEIRD research: mentioning the sample’s country in the title. Study 1 provided evidence that mentioning the country in a title of an article is associated with fewer citations for research with samples from countries other than the United States and that country mentions occur more often in articles that include non-WEIRD samples. Consequently, research with non-WEIRD samples is more negatively affected by this practice. Study 2 provides experimental evidence of a, perhaps unsurprising, in-group bias at least among U.S. readers, such that articles with one’s own country mentioned in the title are particularly attractive. However, when the possibility of in-group bias was removed in a German sample, articles with any country mentions in the title receive less research attention. Again, given that country names are much more likely to appear in titles of articles with non-WEIRD samples, this preference for articles without country names will be more problematic for research with non-WEIRD samples.

Changing the Current Practice

The impact of efforts to address the lack of sample diversity in psychology (Adair & Huynh, 2012, Bauer, 2020) by publishing work with samples from non-WEIRD countries will be limited if these articles do not receive the same attention from the scientific community as articles with samples from WEIRD countries. Citation counts serve as a proxy for impact and inclusion. Preferentially citing articles that include WEIRD (and especially U.S.) samples, especially when articles with non-WEIRD samples are available, creates academic literatures that perpetuate and reinforce the dominant position of these high-status nations, as well as the view that WEIRD-populations are the prototype for human psychology. Omitting country mentions in articles with WEIRD samples, and especially U.S. samples, may contribute to the perception that these samples are human universals, whereas non-WEIRD samples are deviant in the way that they require to be explicitly mentioned.

The decision of whether to include the country from which the sample was drawn in the title can result from various motivations on the part of the editors/reviewers. For example, they may think that non-WEIRD samples are more interesting or important and that highlighting them might attract readers. This might also be a genuine response to research showing the importance of including samples from non-WEIRD countries. This tendency may also reflect very legitimate motivations of the author/s themselves. For example, authors researching non-WEIRD samples may think that they are expected to include this information in the title or that including it will bring attention to the novelty of the research. Despite these laudable motives, it appears that this practice represents a systemic problem within our editing system that can undermine both the diversification of the literature and the impact and opportunities of researchers working in non-WEIRD countries. However, this practice is not only a problem for the science itself, it also creates a form of subtle institutional discrimination against researchers who focus on non-WEIRD samples. The citation record is crucial for researchers' academic success, the ranking of their institution, and decisions regarding funding and career options. If mentioning the sample’s country in the title reduces the probability that an article will get cited, then encouraging authors studying non-WEIRD countries to do so can unintentionally undermine their future success.

Interestingly, the preference for titles including one’s own country (Study 2a) or no country (Study 2b) occurred only when participants were instructed to pick articles out of several options but not when they were asked to rate the relevance of a single article. This suggests that these preferences do not stem from a conscious intent to discriminate. Readers might find a paper interesting and relevant yet might choose not to include it in their work when they have numerous options to choose from. However, deciding to include an article in one’s own article or in a teaching syllabus might eventually determine whether that article will become a part of the core literature.

Limitations and Future Direction

Our results document the phenomenon but do less to unravel the psychological mechanisms responsible for it. In-group bias, the tendency to favor “us” over “them” (e.g., Brewer, 1979), provides a reasonable explanation for the preference for articles that mention one’s own country. A preference for universality or generalizability—that is, for research that does not appear to be exclusively relevant to a specific country or culture—may provide an explanation for the tendency to select articles with no country mentions. However, future research is needed to provide direct tests of these underlying mechanisms and to consider other possible mechanisms that could explain these tendencies. In addition, we focused on a small subset of journals, and Study 2 included student samples only from WEIRD countries. Future research would benefit from considering a broader sample of outlets and examining whether academics from a wider array of countries show a similar pattern of favoring studies from their countries.

Conclusions

Despite efforts to make psychological science more inclusive, inequality is often maintained through subtle mechanisms, like a bias toward citing articles with U.S. samples or with no country mentions in the title. Revealing such mechanisms is important if we want our literatures to be more representative of the world’s population. Because a complete understanding of psychological research requires the consideration of context, we believe that mentioning the sample’s country is critical. Editors and reviewers who evaluate research should, however, be aware of the costs of asking authors from non-WEIRD countries to mention their samples’ country specifically in the title, as should these authors themselves. One way to overcome such unintentional bias and double standards in research practice is to establish guidelines for journals, editors, and reviewers. Based on our results that mentioning the country in the title leads to bias in evaluation, and since it is not necessarily important for scientific communication, it is not recommended. Instead, we suggest that a proper description of the sample, including country, should appear in the abstract. We hope that raising awareness to such subtle mechanisms may help make the core literatures in psychology more inclusive and representative of all human behavior.

Footnotes

Author Contribution

Rotem Kahalon and Verena Klein contributed equally to this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: H2020 Marie Skłodowska-Curie Actions awarded to Verena Klein (Grant No. 845508). Swiss Government Excellence Scholarship awarded to Rotem Kahalon (Grant No. 2020.0085).

ORCID iD

Rotem Kahalon

Johannes Ullrich

Notes

References

Adair

J. G.

Huynh

C.-L.

(2012). Internationalization of psychological research: Publications and collaborations of the United States and other leading countries. International Perspectives in Psychology: Research, Practice, Consultation, 1, 252–267. https://doi.org/10.1037/a0030395

Apicella

Norenzayan

Henrich

(2020). Beyond WEIRD: A review of the last decade and a look ahead to the global laboratory of the future. Evolution and Human Behavior, 41, 319–329. https://doi.org/10.1016/j.evolhumbehav.2020.07.015

Arnett

J. J.

(2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63, 602–614. https://doi.org/10.1037/0003-066X.63.7.602

Bauer

P. J.

(2020). Expanding the reach of psychological science. Psychological Science, 31, 3–5. https://doi.org/10.1177/0956797619898664

Brady

L. M.

Fryberg

S. A.

Shoda

(2018). Expanding the interpretive power of psychological science by attending to culture. Proceedings of the National Academy of Sciences of the United States of America, 115, 11406–11413. https://doi.org/10.1073/pnas.1803526115

Brewer

M. B.

(1979). In-group bias in the minimal intergroup situation: A cognitive-motivational analysis. Psychological Bulletin, 86, 307–324. https://doi.org/10.1037/0033-2909.86.2.307

Brown

A. J.

Goh

J. X.

(2016). Some evidence for a gender gap in personality and social psychology. Social Psychological and Personality Science, 7, 437–443. https://doi.org/10.1177/1948550616644297

Cesario

Johnson

D. J.

Terrill

. (2019). Is there evidence of racial disparity in police use of deadly force? Analyses of officer-involved fatal shootings in 2015–2016. Social Psychological and Personality Science, 10, 586–595.

Cheek

N. N.

(2017). Scholarly merit in a global context: The nation gap in psychological science. Perspectives on Psychological Science, 12, 1133–1137. https://doi.org/10.1177/1745691617708233

10.

Cheon

B. K.

Melani

Hong

Y. Y.

(2020). How USA-centric is psychology? An archival study of implicit assumptions of generalizability of findings to human nature based on origins of study samples. Social Psychological and Personality Science, 11, 928–937. https://doi.org/10.1177/1948550620927269

11.

Cole

(2006). Internationalism in psychology: We need it now more than ever. American Psychologist, 61, 904–917. https://doi.org/https://doi.org/10.1037/0003-066X.61.8.904

12.

Diener

Oishi

Park

(2014). An incomplete list of eminent psychologists of the modern era. Archives of Scientific Psychology, 2, 20–31. https://doi.org/10.1037/arc0000006

13.

Erdfelder

Faul

Buchner

Lang

A. G.

(2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160. https://doi.org/10.3758/BRM.41.4.1149

14.

Frieze

I. H.

Dittrich

. (2008). Sex roles: An international journal. Sex Roles, 58, 751.

15.

Harzing

A-W.

Alakangas

(2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106, 787–804. https://doi.org/10.1007/s11192-015-1798-9

16.

Hennessy

(2012). Materialist feminism and the politics of discourse (RLE Feminist Theory). Routledge. https://doi.org/10.4324/9780203087435

17.

Henrich

Heine

S. J.

Norenzayan

(2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X

18.

Hruschka

D. J.

Medin

D. L.

Rogoff

Henrich

(2018). Pressing questions in the study of psychological and behavioral diversity. Proceedings of the National Academy of Sciences, 115, 11366–11368. https://doi.org/10.1073/pnas.1814733115

19.

Jacsó

(2005). Google Scholar: The pros and the cons. Online Information Review, 29, 208–214.

20.

Klein

Savaş

Ö.

Conley

(2021). How WEIRD and androcentric is sex research? Global inequities in study populations. Journal of Sex Research. https://doi.org/10.1080/00224499.2021.1918050

21.

Magnusson

Marecek

. (2010). Sociocultural means to feminist ends: Discursive and constructionist psychologies of gender. In Magnusson

Marecek

(Eds.), The sociocultural turn in psychology: The contextual emergence of mind and self (pp. 88–110). Columbia University Press.

22.

McClelland

G. H.

(2014). Nasty data: Unruly, ill-mannered observations can ruin your analysis. In Reis

H. T.

Judd

C. M.

(Eds.), Handbook of research methods in social and personality psychology (pp. 608–626). Cambridge University Press.

23.

Medin

Ojalehto

Marin

Bang

(2017). Systems of (non-)diversity. Nature Human Behaviour, 1, 1–5. https://doi.org/10.1038/s41562-017-0088

24.

Nisbett

R. E.

Masuda

. (2003). Culture and point of view. Proceedings of the National Academy of Sciences, 100, 11163–11170. https://doi.org/10.1073/pnas.1934527100

25.

Nisbett

R. E.

Peng

Choi

Norenzayan

(2001). Culture and systems of thought: Holistic versus analytic cognition. Psychological Review, 108, 291–310. https://doi.org/10.1037/0033-295X.108.2.291

26.

O’Gorman

Shum

D. H. K.

Halford

W. K.

Ogilvie

(2012). World trends in psychological research output and impact. International Perspectives in Psychology: Research, Practice, Consultation, 1, 268–283. https://doi.org/10.1037/a0030520

27.

Rad

M. S.

Martingano

A. J.

Ginges

(2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences of the United States of America, 115, 11401–11405. https://doi.org/10.1073/pnas.1721165115

28.

Rozin

. (2006). Domain denigration and process preference in academic psychology. Perspectives on Psychological Science, 1, 365–376. https://doi.org/10.1111/j.1745-6916.2006.00021.x

29.

Steegen

Tuerlinckx

Gelman

Vanpaemel

. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712. https://doi.org/10.1177/1745691616658637

30.

Sue

(1999). Science, ethnicity, and bias: Where have we gone wrong? American Psychologist, 54, 1070–1077. https://doi.org/10.1037/0003-066X.54.12.1070

31.

Thalmayer

A. G.

Toscanelli

Arnett

J. J.

(2021). The neglected 95% revisited: Is American psychology becoming less American? American Psychologist, 76, 116–129. https://doi.org/10.1037/amp0000622

32.

Dunning

(2020). Hypocognition and the invisibility of social privilege. In Thye

S. R.

Lawler

E. J.

(Eds.), Advances in group processes (Vol. 37, pp. 1–23). Emerald Publishing Limited. https://doi.org/10.1108/S0882-614520200000037001

33.

Yin

R. K.

(1982). Studying phenomenon and context across sites. American Behavioral Scientist, 26, 84–100.