Splitting Evaluative Association Measures Reveals How Sociolinguistic Context Shapes Implicit Attitudes Towards Regional and Minority Languages

Abstract

We present three studies examining evaluative associations as manifestations of implicit attitudes towards languages in three sociolinguistically distinct bilingual communities: Lombard–Italian, Moselle Franconian–German, and Welsh–English. Using a Go/No-Go Association Task (GNAT) that controls for spurious priming effects and subsequently allows split analyses, study 1 evidenced strongly negative attitudes towards Lombard across measures. Study 2 uncovered a similar trend towards Moselle Franconian, but only in response to pictures following words that required a Go response, not for pictures that followed non-target words. In study 3, although the GNAT failed to detect any significant associations, split analysis revealed that participants responded faster to positive pictures following non-target English words. Therefore, by splitting data according to the Go/NoGo status of the attitude object, we can achieve finer characterizations of implicit attitudes, while highlighting the prominent role of sociolinguistic context in modulating activation of each language in regional/minority language communities.

Introduction

This paper investigates evaluative associations as a manifestation of attitudes towards regional and/or minority languages within three sociolinguistically distinct bilingual communities: Lombard–Italian (Italy), Moselle Franconian-German (Belgium), and Welsh-English (Wales, UK). Selection of these communities was based on the prominent differences in their language policies and the levels of socio-political recognition received by their regional/minority language (RML).

At one end we find Lombard, which is not officially recognized under Italian state law (Coluzzi, 2019), and therefore does not benefit from any active policy for its maintenance, leading to negative attitudes as measured by explicit methods (Brasca, 2023; Brasca et al., 2024; Tamburelli, 2014). This contrasts with the situation in the Belgian Eifel, where—although the community is recognized as a linguistic minority (Möller, 2017)—recognition solely references “German” and does not extend to Moselle Franconian. Moselle Franconian is therefore only indirectly supported, insofar as it is considered a variety of German, despite it being occasionally perceived as a separate language, at least by some speakers (Weber, 2009).

At the other end of the spectrum is Welsh, which enjoys full socio-political recognition in Wales under the Welsh Language Act of 1993. Welsh speakers have the right to use Welsh in a court of law, and Welsh-medium education is well-established. Several measures protect and encourage the use of Welsh, such as the national action plan Iaith Pawb (“Everyone's Language”) issued by the Welsh Assembly Government (2003). These provisions have had positive attitudinal effects, for example the improvement of attitudes towards Welsh immersion education in south-east Wales (Hodges, 2012; Williams, 2024). Welsh has also gained strong association with higher prestige contexts, particularly education (Price & Tamburelli, 2016).

In this paper, we present an adaptation of the Go/No-Go Association Task (GNAT), modified to incorporate auditory stimuli and an added control for spurious priming effects, allowing us to obtain a layered measure of implicit bias towards languages in the three bilingual communities introduced above.

Literature Review

Language Attitudes as Measures of Vitality

Although evidence for the link between attitudes and RML use is somewhat tenuous, being almost entirely based on self-reports (see Tamburelli et al., 2025 for a discussion), speakers’ attitudes in particular are consistently associated with language vitality and shift, and thus ultimately with language use (e.g., Garrett, 2010; Kircher & Zipp, 2022; Sallabank, 2013). As Grenoble and Whaley (2021, p. 922) put it: “language attitudes [are] […] a matter of introducing (or reinforcing) language use.” The association between language attitudes and use is therefore central to the study of language maintenance and the maintenance of linguistic diversity, and as such has been at the basis of research on many minority and/or endangered language situations, including, for example, Irish (Laoire, 2006), Nahuatl (Gomashie, 2023), and Quebec French (Kircher, 2022).

The Role of Implicit Measures

Research in social psychology has long focused on the link between attitudes and behavior (e.g., Petty et al., 2013 for an extensive overview). While researchers disagree as to whether implicit methods tap on a specific attitude representation within an individual's mind (i.e., a specifically implicit attitude representation, see for example Greenwald & Nosek, 2009) or whether a single attitude representation exists and implicit methods simply measure different aspects of it (see e.g., Fazio & Olson, 2003; Olson et al., 2009), there is nevertheless agreement that implicit methods are better predictors of habitual and spontaneous behavior (e.g., Devos, 2008; Gawronski & Hahn, 2018; Perugini, 2005). This makes implicit methods particularly important in the assessment of vitality in RMLs, especially when interest in language attitudes is based on their potential ability to inform us about a speakers’ likelihood to engage in language use. If implicit methods are good predictors of language use, they should be at the center of language attitude research. This insight has led to a growing body of literature applying implicit methods to measure language attitudes, mostly employing the Implicit Association Test or IAT (Greenwald et al., 1998). Although still scarcely applied to language when compared to its widespread use in the broader social psychology literature, there has been some interest in the IAT among researchers investigating implicit attitudes towards language variation. Specifically, Pantos and Perkins (2013) use the IAT to investigate attitudes towards foreign accent versus U.S. accent in English speakers, and found more positive implicit associations with U.S. accented speakers. In a similar study, McKenzie and Carrie (2018) employed the IAT to investigate attitudes towards Northern and Southern English speech in England. Their participants produced lower response latencies when Southern English was associated with positive attributes, indicating more positive implicit associations with Southern English.

Some research is also emerging where the IAT is specifically used to investigate attitudes towards minority languages. Ianos et al. (2020) employed the IAT in their work on Catalan–Spanish adolescents, and found a moderate positive bias towards Catalan over Spanish, although this was not the case for participants who had Spanish as their home language, leaving open the question of whether the preference is for Catalan itself or simply for one's home language. Furthermore, a positive correlation between IAT scores and motivation to learn Catalan was found, indicative of the strength of association between implicit measures and language use. The overall preference for Catalan over Spanish has also been found in young adults (Ianos et al., 2023), with further results showing correlations between IAT scores and reported language use.

The IAT has also been employed to study attitudes in Welsh–English bilingual adolescents (Lee, 2015, 2016) and adults (Gruffydd et al., 2025). Lee (2015, 2016) reports that Welsh-medium educated pupils from English-speaking homes show more positive attitudes towards Welsh than their English-medium educated counterparts. Gruffydd et al. (2025) found no implicit preference for either language among adults in the Welsh-speaking heartland of north-west Wales, though evidence emerged linking IAT scores to exposure to Welsh during primary school age as measured in self-reports.

Some of these IAT studies also replicated previous findings showing that attitudes from implicit measures often conflict with those obtained via explicit measures. McKenzie and Carrie (2018) found a discrepancy between IAT results and results from self-reports on accent preference, which echoed the findings reported by Pantos and Perkins (2013) on foreign accent bias. Gruffydd et al. (2025) found a positive correlation between IAT results and early exposure to Welsh. However, no such effect was found when using an explicit method (the Attitudes towards Languages Questionnaire of Schoel et al., 2012), suggesting that implicit measures may be potentially more sensitive to fine-grained attitudinal tendencies in bilingual communities where both languages are highly regarded and socio-politically supported.

While the results reviewed above are promising, the use of implicit methods in the study of language attitudes remains limited, and almost entirely restricted to the IAT. This contrasts with the broader literature, where other implicit methods have also been employed. The GNAT (Nosek & Banaji, 2001), for instance, characterizes implicit attitudes by measuring the strength of association between a target category (e.g., fruit vs. vegetable) and an evaluative dimension (i.e., positive vs. negative). A criterion is specified through instruction at the start of each experimental block (e.g., “Press if a fruit word or a good word,” Banfield et al., 2006, p. 1843), requiring participants to provide a response for stimuli that meet the criterion (“Go” trials) and withhold their response when the criterion is not met (“NoGo” trials). Go trials typically elicit slower response times (RTs) when the criterion is incongruency between a target category and the evaluative item (e.g., “peach” and “revolting” both requiring a Go response) as compared to Go trials where the criterion is congruency (e.g., “peach” and “delicious” requiring a Go response). RT differences index the strength of automatic associations, with stronger perceived associations leading to larger RT differences. The GNAT has been successfully applied to the measurement of implicit attitudes towards smoking (Detandt et al., 2017; Wagner-Altendorf et al., 2021) and the elderly (Van Der Lugt et al., 2012). While it has also been employed to measure attitudes towards linguistic items, specifically brand names (Thomas et al., 2013) and names of cities (Wagner-Altendorf et al., 2023), to our knowledge, this paper is the first to adopt it to test attitudes towards languages, and particularly regional/minority languages, in bilingual populations.

The GNAT has seen several adaptations, particularly in relation to stimulus presentation, but also modality (notably the work of Monk et al., 2016, which measured responses to olfactory stimuli). In the original implementation (Nosek & Banaji, 2001), both target categories and evaluative items consisted of words appearing on a screen, while subsequent studies have made use of pictures instead of words (e.g., Detandt et al., 2017; Wagner-Altendorf et al., 2021; Wittleder et al., 2024). This is a welcome development, as attitude measures that rely on written words exclude a priori many of the world's languages, since only a small subset of languages benefit from a standardized orthographic system.

In this paper, we employed the GNAT paradigm to examine attitudes towards regional and/or minority languages within three sociolinguistically distinct bilingual communities: Lombard–Italian (Italy), Moselle Franconian–German (Belgium), and Welsh–English (Wales). To fully extend the GNAT paradigm to these sociolinguistic contexts, the linguistic stimuli also needed to be presented in a way that bypassed the lack of standardized orthography (which applies in two of our three populations), and therefore we presented stimuli for the target category in audio format. This enabled us to adopt the GNAT to study regional and/or minority languages, thus extending its use and contributing to addressing the well-known lack of linguistic diversity in multilingualism research (e.g., Bylund et al., 2024).

This audio-visual setup required additional considerations to control for spurious semantic priming effects that could occur between words and pictures. The resulting control measure allowed us to not only measure traditional GNAT effects but also distinguish RT modulations that arise following Go and NoGo trials separately (detailed in the Go/NoGo split-analysis section). To our knowledge, it is the first time that extraneous priming effects have been controlled in a GNAT experiment, as well as the first time that such paradigm has been applied to measure implicit attitudes in bilingual communities who speak a majority language and regional/minority or minoritized language. It is also the first time that the GNAT is implemented using auditory stimuli. Therefore, besides making empirical contributions to research on language attitudes, this paper also aims to foster methodological innovation in the study of attitudes more broadly.

Study 1: Lombard–Italian

Method

An adaptation of the GNAT (Nosek & Banaji, 2001) was used for this study, comprising of an auditory modification and restricting randomization in order to control for spurious priming effects. All else followed a standard GNAT paradigm as implemented in attitude studies (e.g., Van Der Lugt et al., 2012; Wagner-Altendorf et al., 2021), relying on reaction times to capture the extent to which participants automatically associate a target category (attitude object) with an attribute category (emotional valence: positive or negative).

In order to control for priming effects, the order of stimulus presentation was counterbalanced across conditions. While in the original GNAT any target stimulus can appear following any attribute stimulus, we counterbalanced target-attribute pairs to avoid unwanted priming effects relating to spurious semantic relationships between words and pictures. For example, the word “water” was prevented from preceding a picture showing a body of water (to avoid direct or repetition priming), or a picture of a bottle (to avoid associative priming). Similarly, the Italian word ferro (“iron”), for instance, was prevented from preceding a picture showing a metallic object (to avoid conceptual or taxonomic priming). Counterbalancing was implemented through random selection from four semantically unrelated pictures associated with each word, yielding four possible picture–word pairs. Each of the four possible pairs was randomly assigned to a different block, such that all combinations were presented to each participant, but only once per block, and in a random order. This allowed Go conditions (i.e., conditions where participants are required to give a response) to be analyzed according to the Go/NoGo status of the attitude object as well as for the standard GNAT effect.

A further modification involved the use of audio stimuli. While in the standard GNAT both target and evaluation stimulus are presented visually, we employed auditory stimuli for the target category. This was done in order to accommodate for the fact that Lombard does not have a standardized orthography, and therefore use of written word stimuli would have likely compromised neutrality. To our knowledge, this is the first time that an auditory GNAT design has been used for the study of language attitudes.

Participants

Forty-four Lombard–Italian bilinguals (20 female, 24 Male, 4 left-handed, age range 22–36, M_age = 28.95, SD = 3.74) took part in this study. Selection criteria required participants to be native speakers of both Lombard and Italian, but also admitted those who were “very familiar with” or had “a good understanding of (any Bergamasque variety of) Lombard.” This range of criteria was due to the fact that fluent Lombard speakers are in severe decline (Brasca et al., 2024; Coluzzi et al., 2018). This situation of decline also played a central role in determining sample size, which we decided in advance based on the potential participant pool, as established by our previous experiences in recruiting participants within the Lombard-speaking community.

Materials

In our GNAT implementation, target categories were represented by spoken words of neutral valence in either Lombard (the regional/minoritized language) or Italian (the majority/dominant language) and evaluative items consisted of pictures with a positive or negative valence (e.g., a field in bloom or a dilapidated building).

As our aim is to identify attitudes towards languages rather than towards words, we needed to select words of neutral valence in order to exclude the possibility that any (in)congruency effects may be due to semantic properties of the words rather than to the language in which they were spoken. The main study was thus preceded by a norming procedure, described in detail in Supplemental Materials. Further controls pertaining to word length, frequency, semantic classification, word-onset structure, and the exclusion of homophony are also detailed in the Supplemental Materials.

Similarly, pictures that represented the attribute category were normed for valence. These were taken from the Nencki Affective Picture System (NAPS, Marchewka et al., 2014), a database of standardized, high-resolution pictures with four themes: people, animals, objects, and landscapes, ¹ and subsequently pre-normed to ensure maximal contrast for valence among the population under investigation. Finally, the speaker who recorded the words was selected based on norming in order to ensure that they would be perceived as suitably authentic in both languages. Norming procedures are outlined in the Supplemental Materials.

Design

The attitude objects’ target category was language (Lombard vs. Italian) represented by 40 words of neutral valence per language (n = 80 auditory stimuli, 40 in Italian, and 40 in Lombard, see Table 1, Supplemental Materials). The attribute category was emotional valence (positive vs. negative), with a total of 80 visual stimuli (40 positive and 40 negative).

Sample size was determined based primarily on feasibility considerations related to the recruitment capacity of the target population, while also taking into account a range of sample sizes from previous Go/NoGo studies (Chowdhury et al., 2017).

Procedure

The auditory GNAT paradigm was implemented using PsychoPy 2023.1.3 open-source software (Peirce et al., 2019). Participants performed eight experimental blocks (two blocks per condition, four conditions) each consisting of 80 experimental trials. The first block was preceded by 40 practice trials in order to ensure familiarity with the task. Subsequent blocks were preceded by eight practice trials allowing participants to familiarize themselves with the Go and NoGo conditions specific to the current block. The same words and pictures were used within each block, but category-attribute associations between language and picture valence were different across blocks. This setup minimally departs from the original GNAT paradigm, where item selection is entirely random, ² thus leaving spurious priming effects uncontrolled. This modification also allowed us to investigate the difference between RTs in response to positive and negative pictures following words in a particular language and as a function of whether the preceding word required a response. While standard GNAT implementations tend to measure either a combination of responses to targets and evaluative attributes or focus exclusively on responses to targets, it has been understood since Nosek and Banaji’s (2001) foundational study that signal in the GNAT comes from both the target category and the evaluative attribute, with some measurements—notably false alarms—being driven primarily by evaluative attributes. Therefore, this adaptation allowed us to restrict measurement to picture-based responses, thereby avoiding additional variance associated with the temporal smearing that occurs during processing of spoken stimuli, while also enabling us to measure responses on a component of the standard GNAT signal structure.

Visual stimuli were presented as 7.2 × 7.2 cm pictures centered vertically and horizontally, and audio-stimuli were presented through headphones without accompanying visual stimulus, as illustrated in Figure 1. Participants were instructed to press the space bar as quickly and as accurately as possible to categorize the target categories (Lombard and Italian words) and attribute categories (positive and negative valence of pictures). At the start of each trial, a white fixation cross was centrally presented on a black background for a randomly selected duration of between 400 and 500 ms. This was immediately followed by a word which lasted between 727 and 1,137 ms, followed by another fixation cross for a duration of 400–500 ms, which in turn was followed by a picture which remained on the screen for a randomly selected duration of between 500 and 900 ms, followed again by a fixation cross for 400–500 ms until the end of the trial. Except for the first instance, fixation crosses provided feedback to the participant: a cross turning green after a response indicated a correct GNAT response, while a cross turning red indicated an incorrect response. Figure 1 shows a schematic representation of the trial procedure. Participants were instructed to either give a response (“Go”) or withhold a response (“NoGo”) upon presentation of each stimulus, according to the instructions that they received at the beginning of each block and then again as a reminder at the end of the practice trials.

Figure 1.

Schematic representation of the GNAT procedure.

There were four GNAT conditions: Lombard or positive (“press if a Lombard word or a positive picture”), Lombard or negative (“press if a Lombard word or a negative picture”), Italian or positive (“press if an Italian word or a positive picture”), and Italian or negative (“press if an Italian word or a negative picture”). These conditions were further split into eight based on the status of the words involved in each word-picture sequence: four conditions where the preceding word required a response (Go Word) and four conditions where the preceding required no response (NoGo word). Block order was rotated and counterbalanced across participants.

Results

GNAT Analysis

Following established practice in attitudinal applications of the GNAT which require different modalities for target category and evaluative items (e.g., Detandt et al., 2017; Wagner-Altendorf et al., 2021), we analyzed participants’ reaction times and error rates elicited by pictures (positive vs. negative picture). A 2 × 2 repeated measures ANOVA (Language: Lombard or Italian × Valence: positive or negative) revealed a medium main effect for language (F(1, 43) = 4.056, p = .050, ηp² = .086) and a very large language × valence interaction (F(1, 43) = 41.073, p < .001, ηp² = .489; Figure 2. Full results are given in Supplemental Materials, Table 2).

Figure 2.

Boxplots and violin plots of RTs in response to positive (+) and negative (−) pictures for GNAT conditions by Language and Valence. Dots depict individual average RTs, and grey lines show the correspondence between valence conditions for the same individuals. RML = regional/minority language (here Lombard) and Maj = majority language (here Italian).

Post-hoc paired t-tests with Bonferroni adjustment indicate faster reaction times for Lombard in the negative than the positive condition (mean difference = 26 ms, 95% CI [13.4, 38.3], p < .001, Cohen's d = 0.63 indicating a medium effect) and for Italian in the positive than the negative condition (mean difference = 28 ms, 95% CI [−38.6, −16.8], p < .001, Cohen's d = 0.77 indicating a medium effect).

Error rates in general were low: 2.2% misses and 3.9% false alarms. A 4 × 2 repeated measures ANOVA (GNAT condition: Italian or negative, Italian or positive, Lombard or negative, Lombard or positive × error type: miss or false alarm) with Greenhouse-Geisser correction revealed a large effect of GNAT condition (F(2.295, 98.673) = 9.241, p < .001, ηp² = .177) and a large effect of error type (F(1, 43) = 9.420, p = .004, ηp² = .180), with the Lombard or negative condition attracting fewer misses than both the Italian or negative condition (mean difference = 1.364, 95% CI [−2.487, −.240], p = .010) and the Lombard or positive condition (mean difference = 2.114, 95% CI [−3.908, −.319], p = .013). For false alarms, the Italian or negative condition elicited more than the Lombard or negative condition (mean difference = 2.455, 95% CI [.188, 4.721], p = .027). All other comparisons were not statistically significant (p ≥ .090).

Go/NoGo Split Analysis

A 2 × 2 × 2 repeated measures ANOVA (language: Lombard or Italian × Valence: positive or negative × GNAT condition: Go or NoGo) revealed a medium size main effect of GNAT condition (F(1, 43) = 4.591, p = .038, ηp² = .096). A medium size interaction was also found between Language and GNAT condition (F(1, 43) = 4.299, p = .044, ηp² = .091), and a very large size interaction between language, valence and GNAT condition was found (F(1, 43) = 44.809, p < .001, ηp² = .510). Full results are given in Supplemental Materials, Table 3.

Post-hoc comparisons with Bonferroni adjustment revealed that, when presented after a Go word (i.e., a word that required a response), positive pictures resulted in faster reaction times compared to negative pictures when the words were in Italian (mean difference = 30 ms, 95% CI [−41, −19], p < .001, Cohen's d = 0.82 indicating a large effect), while the reverse obtained for pictures following Go words in Lombard, with faster reaction times in response to negative pictures compared to positive pictures (mean difference = 23 ms, 95% CI [−8, −39], p = .004, Cohen's d = 0.46 indicating a small effect; Figure 3). Further, positive pictures resulted in faster reaction times when following Italian words than when following Lombard words (mean difference = 21 ms, 95% CI [−6, −36], p = .007, Cohen's d = 0.42 indicating a small effect), while the reverse obtained for negative pictures, where faster reaction times were recorded following Lombard words (mean difference = 33 ms, 95% CI [−21, −44], p < .001, Cohen's d = 0.83 indicating a large effect).

Figure 3.

Boxplots and violin plots of RTs for positive and negative pictures following a word in Lombard, the RML (i.e., regional/minority language) or Italian, the Maj (i.e., majority language). Left: Following a word that required a response (language in parentheses is the Go language), Right: Following a word that did not require a response. Signs + and − denote positive and negative pictures respectively. Grey lines show the correspondence between valence conditions for the same individuals.

In NoGo trials, the reverse pattern was found (Figure 3, plot on the right): positive pictures resulted in faster reaction times compared to negative pictures when following Lombard words (i.e., when Italian was the Go language, mean difference = 26 ms, 95% CI [−39, −12], p < .001, Cohen's d = 0.44 indicating a small effect), while negative pictures resulted in faster reaction times compared to positive pictures when following Italian words (i.e., when Lombard was the Go language, mean difference = 31 ms, 95% CI [−44, −17], p < .001, Cohen's d = 0.69 indicating a medium effect). Further, negative pictures were responded to faster when following Italian words than when following Lombard words (mean difference = 38 ms, 95% CI [−51, −26], p < .001, Cohen's d = 0.96 indicating a large effect), while positive pictures elicited faster reaction times when following Lombard words (mean difference = 18 ms, 95% CI [−32, −4], p = .014, Cohen's d = 0.38 indicating a small effect).

Discussion

An attitudinal bias against Lombard and in favor of Italian emerged in several analyses: GNAT responses, GNAT error rates, and split Go/NoGo analysis. Results from the split analysis are fully consistent with those of the GNAT component, since negative pictures elicited faster responses when preceded by Lombard words, and positive pictures elicited faster responses when preceded by Italian words. This overall bias against Lombard is in line with findings from previous studies that reported negative attitudes towards Lombard, including an auditory IAT (Brasca et al., 2024; Tamburelli et al., 2025), as well as reflecting the negative attitudes typically associated with lower sociolinguistic status (see Kircher & Zipp, 2022 for an overview).

Beyond mirroring GNAT trends, the Go/NoGo split analysis showed greater sensitivity to attitudinal associations, as evidenced by the large effect sizes. Further, the split analysis offers separate insights into the NoGo trials, which elicited a fully inverted pattern. This inverted pattern can be understood when we consider that, following a NoGo trial, participants were in an experimental block where it was the other language that required a Go response. It is thus explicable why the trends found for Go trials should reverse in the NoGo trial analysis: if a positive picture was responded to faster than a negative picture after a Go word in Italian, it follows that a positive picture could also be responded to faster than a negative picture after a NoGo word in Lombard, since in that block the Go words were in Italian. By the same logic, since NoGo-Italian trials occurred in the same block as Go-Lombard trials, negative pictures were responded to faster after a NoGo word in Italian.

These reversal effects further characterize the attitudinal split between Italian and Lombard. As GNAT participants are only told which language and valence require a Go response, Italian is necessarily the Go language when Lombard is the NoGo language, and vice versa. This appears to generate an attitudinal contrast effect, whereby negative attitudes towards Lombard lead to quicker RTs for positive pictures after a NoGo Lombard word, while slower RTs are triggered by positive pictures after a NoGo Italian word. This aligns almost perfectly with the GNAT effect, as having Lombard as the Go language—and thus requiring participants to refrain from responding to Italian—triggers an advantage for negative and disadvantage for positive pictures.

Study 2: Moselle Franconian–German

Method

The same method and design were used for this study as in Study 1, combining an auditory modification of the GNAT (Nosek & Banaji, 2001) with a restriction on randomization in order to control for spurious priming effects.

Participants

Forty Moselle Franconian–German bilinguals (25 female, 15 male, 3 left-handed, age range 22–36, M_age = 29.1, SD = 3.88) took part in this study. Selection criteria required participants to be native speakers of both languages from the Eifel region of East Belgium.

Materials

Materials were developed following the same procedures as for Study 1, relying on the NAPS (Marchewka et al., 2014) for pictures and running a series of norming procedures to validate the stimuli and ensure that the speaker was perceived as comparatively authentic in both languages. Relevant details of the norming procedures are outlined in Supplemental Materials.

Design

The attitude objects’ target category was language (Moselle Franconian vs. German) represented by 40 words of neutral valence per language (n = 80 auditory stimuli, 40 words × 2 languages; see Table 4, Supplemental Materials). The attribute category was emotional valence (positive vs. negative), for a total of 80 visual stimuli (40 pictures × 2 valence poles).

Procedure

The procedure was the same as for Study 1. There were four GNAT conditions: Moselle Franconian or positive (“press if a Moselle Franconian word or a positive picture”), Moselle Franconian or negative (“press if a Moselle Franconian word or a negative picture”), German or positive (“press if a German word or a positive picture”) and German or negative (“press if a German word or a negative picture”). These were further split into eight conditions based on the status of the words involved in each word-picture sequence: four conditions where the preceding word required a response (Go Word) and four conditions where the preceding word required no response (NoGo word). Block order was rotated and counterbalanced across participants.

Results

GNAT Analysis

A 2 × 2 repeated measures ANOVA (language: Moselle Franconian or German × Valence: positive or negative) revealed a medium main effect for language (F(1, 45) = 4.418, p = .041, ηp² = .089) and a large language × valence interaction F(1, 45) = 11.964, p = .001, ηp² = .210 (full results in Supplemental Materials, Table 5). Post-hoc paired t-tests with Bonferroni adjustment indicate faster reaction times for Moselle Franconian in the negative condition (mean difference = 14 ms, 95% CI [5.0, 23.5], p = .002, Cohen's d = 0.46 indicating a small effect) and for German in the positive condition (mean difference = 12 ms, 95% CI [−22.4, −0.9], p = .017, Cohen's d = 0.32 indicating a small effect; Figure 4).

Figure 4.

Boxplots and violin plots of RTs in response to positive (+) and negative (−) pictures for GNAT conditions by Language and Valence. Dots depict individual average RTs and grey lines show the correspondence between valence conditions for the same individuals. RML = regional/minority language (here Moselle Franconian) and Maj = majority language (here German).

Error rates in general were low: 1.8% misses and 3.6% false alarms. A 4 × 2 repeated measures ANOVA (GNAT condition: German or negative, German or positive, Moselle Franconian or negative, Moselle Franconian or positive × error type: miss or false alarm) with Greenhouse-Geisser correction revealed a significant effect for GNAT condition (F(2.150, 96.763) = 3.268, p = .039, ηp2 = .068) and error type (F(1, 45) = 45.262, p < .001, ηp² = .501), with the Moselle Franconian or positive condition attracting more misses than the Moselle Franconian or negative condition (mean difference = 1.239 95% CI [−2.338, −.140], p = .019). All other comparisons were not statistically significant (p ≥ .092).

Go/NoGo Split Analysis

A 2 × 2 × 2 repeated measures ANOVA (language: Moselle Franconian or German × Valence: positive or negative × GNAT condition: Go or NoGo) revealed main effects for language (F(1, 45) = 35.954, p < .001, ηp² = .444), and GNAT condition (F(1, 45) = 23.520, p < .001, ηp² = .343). Significant interactions were also found between language and valence (F(1, 45) = 14.145, p < .001, ηp² = .239), language and GNAT condition (F(1, 45) = 4.610, p = .037, ηp² = .093), and between language, valence, and GNAT condition (F(1, 45) = 11.564, p < .001, ηp² = .204) (full results in Supplemental Materials, Table 6).

Post-hoc comparisons with Bonferroni adjustment revealed that, when presented after a Go word (Figure 5, plot on the left), positive pictures resulted in faster reaction times compared to negative pictures when following German (mean difference = 21 ms, 95% CI [−33.0, −9.0], p < .001, Cohen's d = 0.53 indicating a medium effect), while faster reaction times were recorded for negative pictures when following Moselle Franconian (mean difference = 21 ms, 95% CI [−11.0, −31.0], p < .001, Cohen's d = 0.63 indicating a medium effect). Further, positive pictures resulted in faster reaction times when following German words than when following Moselle Franconian words (mean difference = 29 ms, 95% CI [−18, −40], p < .001, Cohen's d = 0.77 indicating a medium effect), while the reverse obtained for negative pictures, where faster reaction times were recorded following Moselle Franconian words (mean difference = 14 ms, 95% CI [−2, −25], p = .018, Cohen's d = 0.36 indicating a small effect).

Figure 5.

Boxplots and violin plots of RTs for positive and negative pictures following a word in Moselle Franconian, the RML (i.e., regional/minority language) or German, the Maj (i.e., majority language). Left: Following a word that required a response (language in parentheses is the Go language), Right: Following a word that did not require a response. Signs + and − denote positive and negative pictures respectively. Grey lines show the correspondence between valence conditions for the same individuals.

However, in NoGo trials (Figure 5, plot on the right) we found no interaction between Language and Valence, with German priming faster reaction times regardless of picture valence: mean difference on negative pictures = 25 ms (95% CI [−36.0, −13.0], p < .001, Cohen's d = 0.65 indicating a medium effect), mean difference on positive pictures = 15 ms (95% CI [−28.0, −3.0], p = .018, Cohen's d = 0.36 indicating a small effect).

Discussion

GNAT results indicate that Moselle Franconian–German bilinguals display more positive attitudes towards their majority language than their RML. These results tally with those from the split Go/NoGo analysis, with positive pictures attracting faster reaction times when following German words, while negative pictures attracted faster reaction times when following Moselle Franconian words. This is consistent with previous findings from an auditory IAT study (Vari & Tamburelli, 2020), while also reflecting typical attitudinal preferences for languages with higher sociolinguistic status (e.g., Kircher & Zipp, 2022). Prima facie these results may appear to echo those from the Lombard–Italian study. However, results from the NoGo trials tell a different story: while Belgian participants responded faster after a German word than after a Moselle Franconian word, no difference was found in terms of picture valence, and therefore no “reversal” of responses between Go and NoGo trials, unlike what we saw for the Lombard–Italian participants. This indicates that Belgian participants treat Go and NoGo contexts considerably differently, which only emerges when separating responses according to the status of the words involved in each word-picture sequence. It therefore appears that the split Go/NoGo analysis afforded by our design reveals an additional layer in the Belgian data, namely that when participants are instructed to withdraw responses for one of their two languages, attitude effects vanish, and no effect of valence and no interaction with language is found.

Study 3: Welsh–English

Method

The same method and design were used for this study as in Studies 1 and 2.

Participants

Welsh–English bilinguals (25 female, 16 Male, 5 left-handed, age range 24–37, M_age = 29.6, SD = 3.92) from the heartland regions of Gwynedd and Ynys Môn took part in this study. Selection criteria required participants to be native speakers of both languages. This was possible due to the fact that there is good availability of bilingual speakers among young adults in North-West Wales.

Materials

Materials were developed following the same procedures as for Studies 1 and 2, relying on the NAPS (Marchewka et al., 2014) for pictures and running a series of norming procedures to validate the stimuli. Relevant details of the norming procedures are outlined in Supplemental Materials.

Design

The attitude objects’ target category was language (Welsh vs. English) represented by 40 words of neutral valence per language (n = 80 auditory stimuli, 40 words × 2 languages; see Table 7, Supplemental Materials). The attribute category was emotional valence (positive vs. negative), for a total of 80 visual stimuli (40 pictures × 2 valence poles).

Procedure

The procedure was the same as for studies 1 and 2. The conditions were also the same, the only difference being the languages in question: Welsh and English.

Results

GNAT Analysis

A 2 × 2 repeated measures ANOVA (language: Welsh or English × Valence: positive or negative) revealed no main effects (language: F(1, 40) = 2.101, p = .155, ηp² = .050; valence: F(1, 40) = .371, p = .546, ηp² = .009) and no interaction (language × valence: F(1, 40) = 6.032, p = .089, ηp² = .070; Figure 6. Full results are given in Supplemental Materials, Table 8).

Figure 6.

Boxplots and violin plots of RTs in response to positive (+) and negative (−) pictures for GNAT conditions by Language and Valence. Dots depict individual average RTs and grey lines show the correspondence between valence conditions for the same individuals. RML = regional/minority language (here Welsh) and Maj = majority language (here English).

Error rates in general were low: 1.94% misses and 4.79% false alarms. A 4 × 2 repeated measures ANOVA (GNAT condition: English or negative, English or positive, Welsh or negative, Welsh or positive × error type: miss or false alarm) with Greenhouse-Geisser correction revealed a significant effect for error type (F(1, 40) = 36.611, p < .001, ηp² = .478), with more false alarms than misses overall.

Go/NoGo Split Analysis

A 2 × 2 × 2 repeated measures ANOVA (language: Welsh or English × Valence: positive or negative × GNAT condition: Go or NoGo) revealed a main effect for language (F(1, 39) = 55.727, p < .001, ηp² = .588) and a significant interaction between language and valence (F(1, 39) = 32.968, p < .001, ηp² = .458; full results are given in Supplemental Materials, Table 9).

Post-hoc comparisons with Bonferroni adjustment revealed that, when presented after Go words, positive pictures resulted in faster reaction times when following English than when following Welsh (mean difference = 16 ms, 95% CI [−1.0, −31.0], p = .034, Cohen's d = 0.37, indicating a small effect), while no statistically significant difference was found for picture valence within languages: mean difference for Welsh = 1 ms (95% CI [14.0, −12.0], p = .898), mean difference for English = 1 ms (95% CI [19.0, −18.0], p = .945). The plot on the left in Figure 7 provides a summary of these results.

Figure 7.

Boxplots and violin plots of RTs for positive and negative pictures following a word in Welsh, the RML (i.e., regional/minority language) or English, the Maj (i.e., majority language). Left: Following a word that required a response (language in parentheses is the Go language), Right: Following a word that did not require a response. Signs + and − denote positive and negative pictures respectively. Grey lines show the correspondence between valence conditions for the same individuals.

In NoGo trials (Figure 7, plot on the right), negative pictures resulted in faster reaction times compared to positive pictures when following Welsh words (i.e., when English was the Go language. Mean difference = 28 ms, 95% CI [−45, −10], p = .002, Cohen's d = 0.50, indicating a medium effect), while positive pictures resulted in faster reaction times compared to negative pictures when following English words (i.e., when Welsh was the Go Language. Mean difference = 22 ms, 95% CI [−37, −7], p = .006, Cohen's d = 0.39, indicating a small effect). In addition, faster reaction times were recorded following English words compared to Welsh words on positive pictures = 50 ms (95% CI [−64.0, −35.0], p < .001, Cohen's d = 1.02, indicating a large effect), while no statistically significant difference was found on negative pictures = 0 ms (95% CI [−18.0, 17.0], p = .975).

Discussion

Here the GNAT failed to detect any significant associations, consistent with previous studies which found no implicit preference for either Welsh or English using an auditory IAT paradigm (Gruffydd et al., 2025). In contrast, the split Go/NoGo analysis revealed a mixed pattern, with a slight positive association (small effect size) for English compared to Welsh (i.e., only across—not within—languages) in the Go trial analysis, consistent with overall mixed findings in cases of RMLs that enjoy relatively high levels of socio-political recognition (cf. Ianos et al., 2023 for Catalan). However, the pattern of results in the NoGo analysis tended to be the opposite of what should have been expected if participants had held negative attitudes towards Welsh. While participants were quicker to respond to positive pictures following English words, in the NoGo condition, withholding a response to an English word resulted in lower RTs than withholding a response to a Welsh word. This suggests a stronger reluctance to actively disregard a Welsh word compared to an English word, in line with attentional inhibition effects found in the broader literature on inhibitory control, where participants’ negative emotion has been found to be proportional to the degree of attentional inhibition required by a stimulus (Inoue & Sato, 2017). A similar effect is evident within each language: positive pictures elicited faster RTs than negative pictures when disregarding English words and, symmetrically, negative pictures elicited faster RTs when disregarding Welsh words. This association only emerged when looking at RTs for pictures following NoGo word trials, which suggests a profound dissociation between Go and NoGo patterns. In the overall discussion below, we contend that associations revealed by the NoGo analyses are possibly more fundamentally unintentional and attitudinal in nature than those that emerge from Go analyses.

Overall Discussion

We investigated evaluative associations as a manifestation of attitudes towards RMLs within three sociolinguistically distinct bilingual communities: Lombard–Italian, Moselle Franconian–German and Welsh–English. We presented a novel implementation of the GNAT (Nosek & Banaji, 2001) which incorporates auditory stimuli and visual cues whilst controlling for spurious affective priming effects. In the original GNAT framework, sensitivity is defined over the combined signal set comprising stimuli from target categories and evaluative attributes. Therefore, responses to evaluative stimuli are not ancillary but form part of the core signal structure. Our measurements targeted evaluative-stimulus trials to bypass the variance associated with temporal processing of spoken stimuli while allowing us to control word–picture sequencing precisely across and within trials. The observed effects should therefore be understood as reflecting one component of the GNAT signal structure rather than a different construct.

Study 1 revealed pronounced negative associations with Lombard in several analyses, and a complete reversal between Go and NoGo conditions which could only be disentangled by controlling for priming effects between target categories and evaluative attributes. This shows the effectiveness of the GNAT overall, with the split between responses being dependent on which language constituted the Go condition.

In Study 2, while similar negative associations for Moselle Franconian were uncovered in the GNAT data, results from the split Go/NoGo analysis revealed a different picture: The preference for German emerged only when German words required a Go response. This may be interpreted as evidence that attitudes towards Moselle Franconian are overall less hostile than towards Lombard, even though it is perceived as less positive relatively to German. This qualitative difference between Studies 1 and 2 also raises the possibility that controlling stimulus sequence may result in more fine-grained data than what is afforded by a standard GNAT design, disentangling two factors that contribute to the GNAT effect: positive effects of target category (languages in our case) when requiring a response, and negative effects of target category when withholding a response. Indeed, while the overall GNAT results showed a comparable picture for Italian and Belgian participants, the split Go/NoGo analysis suggests that not all negative attitudes are equally negative. In the Lombard case, the strength of the negative attitude is such that it results in a full inversion of the valence effects between Go and NoGo conditions. Hence, when participants are negatively biased towards a language, they can show opposing behaviors depending on whether or not they are instructed to respond to that language. Recall that, when in the NoGo condition for one language, participants are necessarily instructed to respond to the other language, which means that we would expect a full reversal of the pattern across Go and NoGo trials if both contributed equally to the overall GNAT effect. This is what we observed for the Lombard–Italian cohort. When participants had to withhold responses on Italian words (and therefore respond to Lombard words), they responded faster to negative pictures, leading to an overarching effect that overrode any contribution Italian words might have made in the specific word-picture sequences. It therefore appears that when bias against a language is strong, it emerges irrespective of whether we instruct participants to respond to that language.

In the case of Moselle Franconian and German, we found a very similar pattern of differences for Go Trials, with pairwise comparisons that looked very similar to those conducted for Lombard and Italian. If we had limited our investigation to the overall GNAT effect, we could have inferred that the Moselle Franconian and Lombard cases were very similar, with participants holding overall negative attitudes towards both RMLs. However, following NoGo words (i.e., words for which a response had to be withheld in that block), we did not see the reversal evident in the Lombard–Italian study: There was only an effect of language and no modulation by valence. This suggests that the GNAT design may be overly inclusive and may mask such differential trends created by preceding stimuli in the agglomerated data.

We believe that this difference between Go and NoGo trials is critical, for two reasons. First, following Go trials, one is more likely to observe attitude effects that are influenced by evaluative conditioning (De Houwer, 2018) and demand compliance (Gast & De Houwer, 2013), due to the immediacy between target item and evaluative stimulus, whereas after NoGo trials, association measures are likely to be freer of metacognitive evaluation. Following a Go word, it may be intuitive and perhaps more obvious for a participant what the task is trying to achieve: testing the association between a language and emotional valence. Following a NoGo word, however, association between the language of the word and the valence of the picture is less straightforward to draw, potentially lowering participants’ contingency awareness (on experimental contingency awareness in associations, see Gast & De Houwer, 2018). We propose that when the association carries over to the NoGo trials in our split analysis, we are observing a stronger, more encompassing association which transcends immediate association effects, indicating more genuinely implicit bias.

The second reason why differences between Go and NoGo trials is critical is linked to some of the criticism historically levelled at the IAT, which may reasonably be extended to the GNAT. For example, Karpinski and Hilton (2001) argued that IAT results may be contaminated by what they call “environmental associations,” namely associations people are frequently exposed to in their environment but that do not necessarily reflect their own attitudes. The same contamination might apply to the GNAT, particularly in the Go condition: when responding to the sequence German-positive, participants react more quickly because that association is environmentally entrenched, with German being overtly acclaimed in their community as the language of “high” culture. However, when the language in the immediate sequence is not the language that participants are responding to—as is the case for NoGo trials—environmental associations are less influential, and therefore we may be tapping on more genuinely implicit attitudinal associations.

Consequently, taken together, Study 1 and 2 suggest that the priming control introduced in our GNAT adaptation has significant potential to gauge attitudes freer of evaluative conditioning as well as potentially less susceptible to environmental associations, thus offering greater insights into the nature of attitudinal bias. The methodological potential of this approach is further evidenced by Study 3, where the split Go/NoGo analysis revealed a mixed pattern: a small effect in favor of English in the Go trials, contrasted by a medium negative effect for English and a large positive effect for Welsh in the NoGo trials. By design, the standard GNAT analysis could not detect any such contrast, as no information is retained about the Go/NoGo status of the items that precede each evaluative stimulus.

Overall, our results uncover an attitudinal continuum between the three bilingual communities, with Welsh and Lombard at two opposite ends and Moselle Franconian in between. The Welsh-English data stand in contrast with results from the other two studies, and particularly with the Lombard–Italian case, where results revealed negative associations with the RML as well as positive associations with the majority language, and where positive associations with the majority language were mirrored by negative associations with the RML. This starkly different pattern could indicate that Welsh has achieved a high level of positive implicit attitudes among our Northern Welsh cohort, potentially higher than English in some circumstances (as indicated by the large effect size). Though English does not elicit particularly negative attitudes, Welsh speakers are less reluctant to disregard it compared to Welsh.

This situation is probably intimately linked to the socio-political habits and levels of official recognition experienced by the three bilingual communities, as well as being in line with a tripartite model proposed by Tamburelli (2012) and with recent results from Brasca et al. (2024). This is most evident in the Belgian results, where—although participants showed different attitudes towards their languages—effects only emerged for the language that required a Go response. This indicates that Belgian participants were not engaging with the language that required no response, a behavior that may be a direct effect of the sociolinguistic situation in the Belgian Eifel, which constitutes a case of “classical diglossia” (Ferguson, 1959; Hudson, 2002): Moselle Franconian is almost exclusively the language of first socialization and informal communication (sociolinguistically “low” domains), while German is the only acceptable language in sociolinguistically “high” domains, encompassing formal contexts such as administrative, educational and media use. Neither of the two languages is necessarily associated with a socio-economically dominant group, being more strongly linked to contexts of use rather than to any specific speaker group (e.g., Brasca et al., 2024). Our results suggest that this compartmentalization of use may lead speakers to favor stronger suppression of one language while engaged with the other. This differs from Lombard–Italian speakers, whose sociolinguistic situation is one of vertical bilingualism (Brasca et al., 2024; Tamburelli, 2012), where diglossic separation is no longer the case. ³ Instead, Italian is almost exclusively the main language of first socialization, while Lombard is typically associated with lower socioeconomic and socio-intellectual groups. Crucially, this associates Lombard and Italian with different groups, rather than strictly with different contexts. ⁴

Our results suggest that these societal differences and the habits of use that characterize them lead to different behaviors in experimental settings. For Lombard–Italian speakers, whose majority language outranks the RML in all communicative domains, attitudes are clear cut: participants show systematically positive associations with Italian whenever they are instructed to respond to it, while the reverse is true of Lombard. For Moselle Franconian–German speakers, however, the diglossic compartmentalization at work in Belgium leads them to suppress one language to a greater extent when they are engaged with the other, and therefore their strength of association with positive and negative attributes varies depending on the experimental condition they find themselves in. This raises questions beyond the realm of language attitudes, specifically about the degree to which different bilingual societies may foster different levels of parallel language activation. While broader cultural context has been investigated as a possible factor in moderating levels of language activation (Berkes et al., 2018), with bilingual language activation being linked to sociolinguistic factors (Masullo et al., 2024), the question that arises from our results is more specific: Is the sociolinguistic setting in which bilingualism operates a factor in modulating the level of activation of each language? Our results suggest that this may be the case, with the type of sociolinguistic situation in which a cohort is immersed affecting patterns of behavioral responses in attitude measures.

Comparing Wales and Lombardy, we find sociolinguistic situations that are not (or no longer) diglossic, though in radically different ways. Wales has enacted a situation of “horizontal bilingualism” (Tamburelli, 2012), where use of both languages is encouraged, promoted and institutionally/legislatively supported (e.g., Watkin, 2016), though not entirely without challenge (e.g., May, 2000). Lombardy, on the other hand, has abandoned diglossia, with Italian becoming increasingly dominant in all communicative contexts, exemplifying the “vertical” bilingualism of Tamburelli (2012) that provides the other end of the continuum: Italian is the only institutionally supported language of Lombardy, and thus the only language in education and increasingly the only language of first socialization. In view of this, our results may be seen as reflecting sociolinguistic as well as socio-psychological effects. The all-encompassing hostility towards Lombard is the consequence of decades of negative policy combined with a benign neglect that affects most of the regional languages of Italy (Coluzzi et al., 2018). At the other end of the spectrum, the positive attitudes towards Welsh result from the hard-won battles that have incrementally raised its profile, while English also continues to elicit somewhat positive attitudes, perhaps unsurprisingly, since English is not just the other language of Wales but also an internationally powerful lingua franca.

Overall, our findings provide evidence for attitudinal effects in three different socio-linguistically structured situations: diglossia, vertical bilingualism, and horizontal bilingualism, while also showing the methodological potential of a more tightly controlled GNAT design for the study of attitudes more generally. Specifically, our design departs from canonical GNAT implementations by employing fixed stimulus sequences to control for spurious priming effects, thus increasing experimental control. A question remains, however, as to whether this structural constraint may influence the composition of GNAT effects. Future research may benefit from comparing directly sequence-controlled GNAT variants with random-sequence designs, particularly in cross-modal implementations, to further establish the psychometric consequences of such modifications.

The higher level of granularity exhibited by the split Go/NoGo analysis is particularly useful in understanding cases where attitudinal differences may be more subtle and can thus remain undetected as a result. Welsh–English bilingualism turned out to be a case in point: with speakers holding relatively positive attitudes towards both languages (e.g., Gruffydd et al., 2025; Lee, 2015; Morris, 2014), any attitudinal differences will be harder to detect. As research where GNAT data have proved inconsistent extends far beyond language attitudes—including the study of executive functions (Lazarev, 2024) and attention (Yin et al., 2016)—a split GNAT analysis can be potentially valuable in other areas of attitudinal research where a more sensitive method may be required.

Supplemental Material

sj-docx-1-jls-10.1177_0261927X261437126 - Supplemental material for Splitting Evaluative Association Measures Reveals How Sociolinguistic Context Shapes Implicit Attitudes Towards Regional and Minority Languages

Supplemental material, sj-docx-1-jls-10.1177_0261927X261437126 for Splitting Evaluative Association Measures Reveals How Sociolinguistic Context Shapes Implicit Attitudes Towards Regional and Minority Languages by Marco Tamburelli, Florian Breit, Ianto Gruffydd, Lissander Brasca and Guillaume Thierry in Journal of Language and Social Psychology