Abstract
Individuals from different marginalized groups can internalize negative social beliefs about themselves and their groups. However, a scale that allows for direct comparisons of internalized stigma across multiple groups has not yet been developed. This paper presents the development and validation of the Multiple-Group Internalized Stigma Scale (MGISS). Participants were recruited from four stigmatized groups representing the possible combinations of the stigma characteristics of mutability and concealability (i.e., Black or Indigenous, lesbian, gay or bisexual, higher body-weight, and working-class people). Study 1 developed the scale across the four groups and in English- and Spanish-speaking countries (i.e., the UK and Chile, N = 238). Study 2 replicated the results with a larger sample of emerging adults in Chile (N = 729). The MGISS demonstrated good levels of reliability and validity, with two factors: self-focused and group-focused. Levels of internalized stigma were higher among groups with mutable characteristics and were associated with higher levels of felt stigma and psychological distress. The MGISS provides a valuable tool for research on prejudice and stigma, particularly in studies involving multiple marginalized groups.
Introduction
Members of socially marginalized groups experience greater stress and poorer mental health outcomes compared to members of non-marginalized groups (Plöderl & Tremblay, 2015; Williams, 2018). Thus, it is crucial to understand the psychological mechanisms that contribute to health inequalities based on stigmatized social status. One proposed explanation for these negative mental health outcomes is internalized stigma, the psychological process whereby socially marginalized people internalize the negative beliefs others endorse about them. Indeed, several studies have found that individuals from marginalized groups who report higher levels of internalized stigma are particularly likely to experience stress and poorer mental health (Gale et al., 2020; Newcomb & Mustanski, 2010; Pearl & Puhl, 2018).
Although the body of work cited above suggests that internalized stigma is a risk factor for negative mental health outcomes across various populations, a key limitation is that the studies have tended to examine a single group in isolation using a scale specific to that group. Surprisingly, a scale that allows researchers to compare internalized stigma across groups has yet to be developed.
As a first step to address this gap in knowledge, we develop and validate a new internalized stigma measure that allows for direct comparisons across multiple stigmatized groups. As a second step, we use this scale to compare levels of internalized stigma across four social groups—including Black (in the UK) or Indigenous (in Chile), sexual minority, higher body-weight, and working-class people—using samples collected in English- and Spanish-speaking countries.
Internalized stigma across multiple groups: A methodological challenge
Internalized stigma is formed when individuals from marginalized populations endorse negative beliefs held by society about themselves or their groups (Herek et al., 2009). For example, when a bisexual man is ashamed of his sexuality and wishes he were not bisexual, he has internalized the social stigma against bisexual men and applied it to his self-concept. Similarly, higher body-weight people or working-class people are at risk of internalizing weight bias or classist beliefs, respectively.
There are theoretical and empirical reasons to think internalized stigma is a shared phenomenon across marginalized groups. Several theoretical models explaining the link between stigma and health across different groups have located internalized stigma as a stressor that stems from cultural and societal discrimination and, in turn, predicts adverse health and well-being outcomes (Frost, 2011; Major et al., 2018). Applications of these models in empirical research illustrate that the negative effects of internalized stigma on health happen across various and different stigmatized populations (Gale et al., 2020; Pearl & Puhl, 2018). However, the degree to which this effect varies across populations cannot be tested with existing group-specific measures. Understanding the shared experience of internalized stigma across groups would contribute to integrated models of stigma and health and inform future global interventions aimed at reducing internalized stigma. Despite these potential theoretical and empirical benefits of studying internalized stigma as a shared phenomenon, research on the construct has usually focused on one single group.
One explanation for the deficiency of multiple-group studies on internalized stigma is the lack of a scale that can measure internalized stigma across various populations. There are separate scales for internalized homophobia (Herek et al., 2009), internalized racism (Campón & Carter, 2015; James, 2020), internalized weight bias (Lee & Dedrick, 2016; Meadows & Higgs, 2019), and internalized classism (Mickelson & Williams, 2008), but there is no internalized stigma measure that can be used across these groups or other groups.
There has only been one attempt to unify the measurement of internalized stigma across different groups. Mak and Cheung (2010) developed the Self-Stigma Scale (SSS) to assess internalized stigma among people with different stigmatized identities that can be concealable. Although the SSS has been a useful tool for researchers, especially when studying stigmatized concealable health conditions (see Li et al., 2017; Wu et al., 2015), its focus only on concealable stigmas limits its utility for comparing multiple groups. Moreover, the authors used different studies to validate the scale for each group and did not explore its use in comparative analyses within the same study across a variety of stigmatized groups (Mak & Cheung, 2010). Therefore, research to date has not systematically studied the commonalities and differences in internalized stigma between diverse groups.
Self-focused and group-focused internalized stigma
Existing internalized stigma scales have not conceptualized the construct in comparable ways, most likely because researchers have used different scales in different groups. Particularly, previous research has not systematically distinguished that internalized stigma can be directed to the self or one’s own group (Ciaffoni et al., 2021; James, 2020). On the one hand, self-focused internalized stigma (SIS) takes the form of self-devaluing beliefs. For example, an Indigenous woman may internalize society’s negative attitudes about Indigenous people, leading her to wish she could change her race/ethnicity. On the other hand, group-focused internalized stigma (GIS) is more similar to a general prejudiced belief towards the group, community, or stigmatized characteristic. An example would be the same individual holding negative beliefs and attitudes about all Indigenous people as a group.
Although the distinction between SIS and GIS has been made when studying some groups (Ciaffoni et al., 2021; James, 2020), this distinction has not been made consistently for other groups. Further, the items used to assess SIS and GIS have varied across the scales used in different groups. Thus, it remains unclear whether SIS and GIS have dissimilar relationships with mental health outcomes across a variety of groups. Ciaffoni et al. (2021) showed that internalized sexual prejudice directed to the self was associated with negative sexual health outcomes, while internalized sexual prejudice directed to the group was associated with less participation in ingroup collective action. In the current research, we aim to develop a measure that can be used across groups that can assess both SIS and GIS to better understand their roles in explaining the consequences of stigma.
The role of stigma characteristics: Concealability and mutability
Without a common measure of internalized stigma that can be used across groups, it has not been possible to systematically examine whether some marginalized groups experience higher levels of internalized stigma than others. This research aims to address this gap by developing a unified measure. Then, we use this measure to study differences in internalized stigma across multiple minoritized groups that vary in key stigma characteristics.
One of these characteristics is concealability (Jones et al., 1984), defined by the capacity—or its expectations—to hide the features linked to stigma. For example, working-class and lesbian, gay, and bisexual (LGB) groups can be considered to have more concealable characteristics compared to people from higher body-weight and racial minority groups, such as Black or Indigenous people (Pachankis et al., 2018).
Although little is known about the role of concealability in internalized stigma, there are reasons to suspect that members of groups with more concealable stigma characteristics may experience higher levels of internalized stigma compared to members of groups with less concealable characteristics. Concealability is associated with particular forms of coping strategies when experiencing discrimination: people may feel that they need to hide their identities or avoid social situations where their identity becomes more evident (Quinn & Earnshaw, 2013). These concealment coping strategies are associated with adverse health outcomes (Thoits & Link, 2016) and, therefore, could also be related to internalizing processes. In line with this theorizing, a study (Blankenship, 2019) found that people with concealable stigmatized identities (LGB and working-class people) presented higher levels of internalized stigma compared to groups with non-concealable stigmatized identities (POC and women) who perceived the same levels of discrimination. However, in this study, the author used collective self-esteem as a measure of internalized stigma, and even when they are related constructs, they should not be understood as equal. More research is required to understand if internalized stigma is different between people with concealable and non-concealable stigmatized identities. Based on the presented theorizing and Blankenship’s (2019) study, groups with more concealable characteristics may be at greater risk of internalizing stigma relative to groups with less concealable characteristics—a hypothesis we test in the present research.
A second stigma characteristic that may account for between-group differences in internalized stigma is mutability, which can be defined as the capacity—or its expectations—that people can change the stigmatized characteristics attached to their identity or group (Jones et al., 1984). For example, individuals from working-class and higher body-weight groups are expected to be perceived as having more mutable characteristics compared to individuals from LGB and racial minority groups (Pachankis et al., 2018).
Not much is known about the role of mutability in the process of stigma internalization. However, we can theorize that people with mutable stigmatized identities or characteristics may experience higher levels of internalized stigma compared to individuals with stigmatized identities low in mutability. First, the expression of prejudice and discrimination against people with mutable stigmas might be more socially accepted and validated due to these characteristics being changeable and not permanent (Solanke, 2021). Second, experiencing discrimination may trigger self-blaming processes: like concealability, the responsibility of the experiences of stigma is directed to the self and not to their societal causes (Himmelstein et al., 2020). Although we need direct evidence to support the hypothesis, the increasing research efforts concentrating on stigmatized characteristics that are changeable or expected to change (e.g., weight-based stigma) warrant more research on the association between mutability and internalized stigma (Pearl & Puhl, 2018).
In order to examine systematic group-level differences in internalized stigma as a function of concealability and mutability, the present research focuses on members of four minoritized groups typically considered by society to vary in these stigma characteristics. With this, we can disentangle the roles of concealability and mutability at the group level to determine whether they explain systematic differences in internalized stigma.
Internalized stigma in non-English-speaking countries outside of Europe
As a secondary goal of this study, we aim to address another key limitation of previous work: Internalized stigma research has received substantially less attention in non-English speaking countries, particularly outside of Europe. The small amount of research in non-English speaking regions is also reflected in the much smaller number of existing scales for measuring internalized stigma. For example, in Spanish, there are a few recent measures for internalized homophobia (Gómez et al., 2023) and weight bias (Sarrías-Gómez & Baile, 2015) but not for internalized racism and classism. Developing a new measure in more than one language expands the scope of internalized stigma research to non-English speaking countries and creates the opportunity to conduct cross-country comparisons.
Some initial work has examined cross-country differences in stigma. For example, one study found relevant differences and similarities between France and Mexico when studying weight stigma (Rojas-Sánchez et al., 2022). Although levels of felt stigma and stigma concerns were higher in the Mexican sample, the processes linking perceived discrimination to internalized stigma and physical activity were equivalent. The authors argued that higher levels of weight stigma among the Mexican sample could be explained by globalization. Media images of slender people as a canon of beauty worldwide contrast with actual observations of diverse body sizes, the former being more pernicious and pervasive. However, this explanation is limited because it does not explain why this effect is stronger in Latin American than European countries. Further, the authors recognize the need for more research on the differences in the psychosocial processes linked to stigma across countries, especially in the Global South (Rojas-Sánchez et al., 2022).
In the present research, we develop both English- and Spanish-language versions of an internalized stigma scale and use this scale to examine differences in internalized stigma across four groups that vary in stigma concealability and mutability within two different countries: the UK and Chile. Compared to the UK and the rest of Western Europe, Chile has a late development and promulgation of laws and regulations against social discrimination (González, 2019) and is characterized by high levels of socioeconomic inequality and segregation (Fernández et al., 2016). In light of these characteristics and the practical advantage of the research team having networks in these countries, Chile and the UK were selected as suitable contexts in which to develop a new measure of internalized stigma, with both English- and Spanish-language versions. Considering the evidence suggesting that internalized stigma depends on sociocultural determinants and country differences, and, therefore, internalized stigma is higher in countries less accepting of those respective groups (Pachankis et al., 2021), we hypothesized that we might find higher levels of internalized stigma in Chile compared to the UK.
The current research
The present research addressed critical limitations of the literature on internalized stigma. Specifically, we aimed to develop and validate a new measure of internalized stigma that can be used across multiple groups, including two subscales in two different languages. We refer to this measure as the Multiple-Group Internalized Stigma Scale (MGISS). In Study 1, we recruited individuals from four marginalized groups typically considered by society to vary in the characteristics of concealability and mutability, following Pachankis et al.’s (2018) scores: working-class (high concealability and high mutability), LGB (high concealability and low mutability), higher body-weight (low concealability and high mutability), and racial minority (low concealability and low mutability) groups, in both English- and Spanish-speaking countries (UK and Chile). Although other groups could have been selected to reflect the combination of these two characteristics, the four groups in the current study were chosen because they have received significant attention in the stigma and health literature and they have existing group-specific internalized stigma scales, making them a suitable starting point for a measurement study. Using this sample, we (a) identify the factor structure of the scale, (b) assess the reliability and validity of the scale, considering its internal consistency and its concurrent and predictive validity, and (c) use this newly developed scale to examine between-group differences in internalized stigma as a function of two stigma characteristics (concealability and mutability) and country (UK vs. Chile). In Study 2, we repeat the primary analyses in a second independent sample to determine the robustness of the observed findings from Study 1.
To address the third objective (c), we include two hypotheses around stigma characteristics, which allows us to test the scale’s potential for comparing groups and increases transparency because we had initial expectations around group differences. We hypothesize that levels of internalized stigma will be higher among groups with stigmatized characteristics typically considered by society to be concealable compared to those with stigmatized characteristics considered to be less concealable. We also hypothesize that levels of internalized stigma will be higher among groups with stigmatized characteristics typically considered by society to be mutable than those with stigmatized characteristics considered less mutable.
Study 1
Method
Scale development
We aimed to develop a scale that integrated previous research and methods on internalized stigma. However, because all existing scales use group-specific references (e.g., being a Black person), we selected items from previous measures that could be easily adapted to different groups, only changing one word or phrase referring to that group when adapting items for the new scale. The scale we present in the current study includes items that can be responded to by people from different social groups with only a one-word change across them. For example, taking the item “I wish I were not a member of my race” from the POC Internalized Oppression Scale (Campón & Carter, 2015), a new item was adapted to “I wish I were not __” where the underlined word would change depending on the group membership of who is answering the item (i.e., overweight, LGB, Black/Indigenous, and lower class). The other possible changing words to use were the social category referred (i.e., body-weight, sexual orientation, skin color, and social class) and the outgroup (i.e., thin, straight, white, and upper class).
Thirty items were initially selected and included in the new measure by adapting items from measures of Weight Self-stigma (Durso & Latner, 2008, used in Lee & Dedrick, 2016); POC Internalized Oppression Scale (Campón & Carter, 2015); Internalized Sexual Stigma (Herek et al., 2009); and Internalized Stigma of Poverty (Mickelson & Williams, 2008).
Participants and procedure
Aiming to develop and test the measure in Spanish and English, and in two different cultural contexts, we conducted a cross-sectional study in Chile and the UK. We followed the general recommendation of having a sample of 200 participants for exploratory factor analysis (EFA) to calculate our sample size (de Winter et al., 2009). We recruited 238 adult participants, following a purposive design of recruiting at least 50 per group, across four groups typically considered by society to vary in the stigma characteristics of concealability and mutability: working-class (high concealability and high mutability), LGB (high concealability and low mutability), higher body-weight (low concealability and high mutability), and Black people in the UK and Indigenous people in Chile (low concealability and low mutability) groups. Regarding LGB people, both the eligibility criteria and the focus of the measurement referred to people’s sexual orientation (being gay, lesbian, or bisexual) and not to their gender identity. Similarly, regarding racial minority people, the eligibility criteria and focus of the measurement also targeted only Black people in the UK and Indigenous people in Chile, not other racial or ethnic minorities, because stigma experiences and characteristics can vary between different racial minority groups. Category membership was self-reported because people were required to self-identify in the groups, which further shaped the presentation of the measure. Participants were required to be members of only one of the groups so they could respond to just one form of the measure specific to the group they were part of. The only caveat to this condition was other racial or ethnic minority people who were not Black (in the UK) or Indigenous (in Chile) but belonged to one of the other targeted groups. For example, an Asian person who identified as LGB would have been included in the study and answered questions pertaining to sexual minority stigma. Further details of the sample characteristics can be found in Table 1.
Sample characteristics in Study 1 and Study 2.
Note. a: Participants were allowed to mark more than option. Only two marked more than one.
Participants in Study 2 were all between 18–21 years old in Chile, so there is small to no variance in country and education level.
Recruitment was conducted online using Prolific (Prolific.com, 2021) and social media platforms (Twitter, Facebook, and Reddit). Recruitment and data collection were conducted between June and December 2021. This project received full approval from the UCL Institute of Education Research Ethics Committee. Data are available upon request from the corresponding author.
Measures
Demographics
Sexual orientation was assessed using a multiple-option question including “straight,” “gay,” “lesbian,” “bisexual,” “pansexual,” “queer,” and “something not listed.” Race was asked, including “White,” “Black,” “Indigenous,” “Asian,” “Mixed,” and “something not listed” as options. Body-weight category and social class questions needed to allow participants to self-identify and create a clear distinction between groups. Therefore, based on self-perception questions like the one used in the National Health and Nutrition Examination Survey (NHANES, see Kwak et al., 2021), people were asked if they considered themselves to be overweight and to be working-class, with yes or no responses. Participants also indicated their income, educational level, and perceived socioeconomic status.
Multiple-Group Internalized Stigma Scale (MGISS)
The 30 items presented were included in the survey for all participants to complete. The survey was programmed using the same items for all participants but changing only a part of each item depending on the participant’s self-reported identity. The specific items initially used for the scale are presented in Table 2. Agreement for all items was measured using a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree).
Items and factor loadings of the Multiple-Group Internalized Stigma Scale (MGISS).
Note. {field}: sexual orientation, skin color, body-weight, social class. {group}: LGB, Black, higher body-weight, working-class. {c-group}: straight, white, thin, upper class. (R): Item with reversed score. (Item in bold): Item included in the final scale.
Existing group-specific internalized stigma scales
For testing concurrent criterion validity of the developed measures, we included four scales of internalized stigma, one for each group. These scales were also presented in a way that corresponded to the participant’s self-reported identity. People with higher body-weight completed the Weight Bias Internalization Scale (Lee & Dedrick, 2016, a Spanish version from Sarrías-Gómez & Baile, 2015), which included ten items (e.g., “I hate myself for being overweight”. Cronbach’s alpha was .93). LGB people completed the Revised Internalized Homophobia Scale (Herek et al., 2009, a Spanish version from Gómez et al., 2023), which included five items (e.g., “I wish I weren’t LGB.” Cronbach’s alpha was .73). Wording of items for LGB participants was focused specifically on people’s sexual orientation and did not apply to gender identity. UK Black and Chilean Indigenous people completed the Appropriated Racial Oppression Scale (Campón & Carter, 2015; a Spanish version did not exist, so a translation was done for this study), which included 32 items (e.g., “I feel that being a member of my racial group is a shortcoming.” Cronbach’s alpha was .94). Finally, working-class people completed the Internalized Stigma of Poverty measure (Mickelson & Williams, 2008, a translation to Spanish was also conducted for this measure), which included three items (e.g., “There have been times when I have felt ashamed because of my financial situation.” Cronbach’s alpha was .75). A 7-point Likert scale of agreement was used for all scales.
Felt stigma
This scale was initially developed by Link (1987) to study stigmatized groups associated with mental health conditions. Meyer et al. (2008) adapted this scale to be used despite the specific stigmatized group of the respondent, and therefore, it could be completed for people from multiple stigmatized groups. It uses seven items (e.g., “Most people think less of someone like me”) referring to social perceptions of “someone like me.” The instructions of the question state that “someone like you” refers to their specific social group previously reported. As with the previous measures, it was responded to using a 7-point Likert scale of agreement. Cronbach’s alpha was .87.
Psychological distress, K10
We used the Kessler-10 scale for measuring psychological distress (Kessler et al., 2002), a scale where participants are asked to state the frequency of their feelings of 10 items, including “nervous,” “hopeless,” “worthless,” among others, using a 5-point Likert scale. Cronbach’s alpha was .92.
Analysis
Factor structure
We conducted an exploratory factor analysis (EFA) on the full set of 30 items included for the development of the MGISS, including parallel analysis to determine the number of factors to extract. Then, we applied item-retaining rules (items with commonalities over .5, main factor loading over .4, and differences between cross-loading under .2; Worthington & Whittaker, 2006) and assessed for correlated residuals (Ferrando et al., 2022). In addition, we conducted a confirmatory factor analysis (CFA) on the final structure of the scale and on the original set of items with one and two factors to compare fit indexes.
Finally, we conducted tests of measurement invariance across languages and groups. For the marginalized group differences, and in line with current concerns about over-reliance on measurement invariance as a prerequisite for group comparisons (Robitzsch & Lüdtke, 2023), we consider that some levels of variance in the measurement structures can be expected, as will be discussed later.
Internal consistency reliability
Cronbach’s alpha was calculated on the final set of items for each factor. The interpretation was guided by Ponterotto and Ruckdeschel (2007).
Validity
The scale’s concurrent criterion validity was assessed through correlations (Lin & Yao, 2014) between the MGISS and the existing group-specific internalized stigma measures. Predictive validity was also assessed through correlations and regressions (Lin & Yao, 2014) between the MGISS and the felt stigma and psychological distress measures.
Group differences
Two-way analysis of variance (ANOVA) was used to compare levels of internalized stigma across groups and countries, and three-way ANOVA was used to compare them as a function of the two group stigma characteristics (concealability and mutability) and country. These analyses are very similar because they are conducted on the same data with related grouping variables: the first is on the mean values of each of the four groups disaggregated; the second is on the groups formed by the combination of high-low concealability and mutability, allowing us to test for main effects of both stigmatized characteristics. We decided to conduct both analyses because the two-way ANOVAs highlighted the use of this scale for comparing multiple groups with particular social identities and group memberships, and the three-way ANOVA focused on the stigma characteristics, directly testing our hypotheses.
R software was used to conduct parallel analysis and CFA. All other analyses were conducted using SPSS 27.
Results
Factor structure
To analyze the factorability of the scale in this sample, we calculated the KMO measure of sampling and Bartlett’s Test of Sphericity. We found a KMO measure of .95 and Bartlett’s of χ2(435) = 5153, p < .001, showing that the data were factorable, and thus we were able to continue with the factor analysis (Worthington & Whittaker, 2006). Parallel analysis pointed out that two factors should be retained. With these results, we decided to conduct an EFA constrained to a two-factor solution. See Table 2 for all items’ factor loadings.
Following the presented item-retention rules, an initial pool of 19 items was retained: 14 for the SIS subscale and 5 for the GIS subscale. The lack of more subdimensions within the SIS subscale and its larger number of items compared to the GIS subscale guided us to examine correlated residuals within the subscale. An initial CFA with the 19 items showed insufficient levels of fit (CFI = .90; RMSEA = .11; SRMR = .06) and allowed us to identify problems associated with accumulated correlated residuals, which were likely due to the close meaning that items within the SIS subscale had with each other (see Bandalos, 2021 about error due to similar items). To avoid decisions based solely on CFA and safeguard the parsimony of the scale, we shortened the SIS subscale by half, allowing us to reduce the accumulated correlated residuals and balance the number of items across the subscales. Following the same item-retention rules with EFA, we retained the seven items with the highest factor loadings within the SIS subscale.
Next, we conducted a CFA (Table 3) with the final structure of the scale (12 items with two subdimensions), and we compared this model with an alternative model of 1-factor using the same items and two alternative models of all-items version (1-factor and 2-factors following the originally hypothesized structure). The final structure of the scale reached acceptable levels of fit, and the fit of this model was better than that of the three alternative models. See Table 3 for the results of the CFA and consult the supplementary material for details of the factor structure analyses, including parallel, exploratory, and confirmatory factor analyses.
Confirmatory factor analysis.
Note. Final model: 12 items and 2 factors. Alternative model 1: 12 items and 1 factor. Alternative model 2: 30 items and 2 factors. Alternative model 3: 30 items and 1 factor.
p < .01.
The scale demonstrated acceptable to good levels of configural invariance across languages, and borderline to acceptable levels of metric, scalar, and strict invariance (see Table 4). Regarding invariance across groups, fit indices of configural invariance were under levels of acceptability but close to the threshold (see Table 5).
Measurement invariance across languages (English and Spanish): Study 1.
Note. N = 208, English = 111, Spanish = 97. MC = Model comparison reference.
p < .01.
Measurement invariance across groups (LGB, Black or Indigenous, higher body-weight, and working-class): Study 1.
Note. N = 208, LGB = 55, Black or Indigenous = 52, higher body-weight = 56, working-class = 45. Configural invariance under level of acceptance, next models not tested.
p < .01.
Reliability
We calculated Cronbach’s alpha of each subdimension and the full scale. The final 7-item SIS subdimension showed an excellent level of internal consistency (α = .95), the final 5-item GIS subscale showed a good level of internal consistency (α = .83), and the combined 12-item internalized stigma scale had an excellent level of internal consistency (α = .93). In addition, when analyzing reliability within each of the groups and country subsamples, all Cronbach’s alphas showed moderate to excellent levels of internal consistency (alphas between .70 and .95, see supplementary material for complete results).
Validity of the measure
To assess the concurrent validity of the SIS and GIS, we calculated Pearson correlations between the two scales and existing scales previously used to determine self- and group-focused internalized stigma. Because participants only identified with one of the groups, a different correlation between the previously developed group-specific scale and the new SIS and GIS was conducted for each subsample. The results of these correlations are presented in Table 6. As shown, the SIS and GIS have significant and moderate to strong associations with existing measures. In sum, we found good indicators of concurrent validity.
Pearson correlations between existing group-specific scales (rows) and new scales (columns).
p < .05. **p < .01.
We also conducted Pearson correlations between the new scales and related variables to evaluate predictive validity. In this case, each correlation was calculated with the full sample. The SIS and GIS also showed statistically significant associations of medium effect sizes with felt stigma and psychological distress, as hypothesized. These results are summarized in Table 7 and show good indicators of the new measure’s predictive validity.
Pearson correlations between new scales, felt stigma and psychological distress.
p < .01.
Following the theoretical directionality of the association between these variables, we conducted a multiple linear regression to calculate changes in levels of psychological distress based on the levels of two stigma experiences: internalized and felt stigma. We included the two subscales of internalized stigma in the first step and felt stigma in the second step of the regression model. As presented in Table 8, in Step 1, we found that SIS was significantly positively associated with psychological distress, but GIS was not. When including felt stigma in the model (ΔR2 = .05, p = .001) in Step 2, the pattern continued: SIS was significantly and positively associated with psychological distress, felt stigma was positively and significantly associated with psychological distress, and GIS’ association remained non-significant. These effects continued to be present in the model when controlling for group, age, and sex (see supplementary material for the controlled analyses). The model did not show indicators of problematic collinearity (all variance inflation factors [VIF] below 5, Kutner et al., 2004).
Regression coefficients with psychological distress as dependent variable.
Group differences
We hypothesized that stigmatized groups considered more concealable by society would present higher levels of internalized stigma compared to those considered to be less concealable. In addition, we hypothesized that groups considered to be more mutable would present higher levels of internalized stigma compared to those considered to be less mutable.
We conducted two- and three-way ANOVAs to explore group differences in levels of internalized stigma. For the two-way ANOVAs, the grouping variables were the social group (LGB, Black/Indigenous, higher body-weight, and working-class) and the country (the UK and Chile). For the three-way ANOVA, the grouping variables were concealability (high and low), mutability (high and low), and country. To avoid repeated information from the two similar ANOVAs, here we integrate both analyses. The full results of these ANOVAs can be found in the supplementary material. Figures 1 and 2 show the mean levels of SIS and GIS, respectively, by social group and country of the sample.
Mean levels of SIS by social group and country and 95% confidence intervals. Letters a, b, and c show post-hoc differences between social groups.
Mean levels of GIS by group and country and 95% confidence intervals. Letters a and b show post-hoc differences between social groups.
SIS
In the two-way ANOVA, we found a significant main effect of group explaining 76% of the variance in SIS (F [3, 208] = 216.36, p < .001, ηp2 = .76). Bonferroni post-hoc tests showed that the Black (UK)/Indigenous (Chile) (M = 1.84, SE = 0.14) and LGB groups (M = 2.34, SE = 0.13) were not statistically different from one another. Still, these groups reported significantly lower levels of SIS higher body-weight (M = 5.80, SE = 0.13) and working-class people (M = 5.14, SE = 0.14), which were also significantly different from each other.
The three-way ANOVA more clearly tested our hypotheses by mapping the groups onto the various permutations of the combination of the stigma characteristics of mutability and concealability. In this analysis, there was a significant main effect of mutability (F [1, 208] = 618.82, p < .001, ηp2 = .75): In line with our hypothesis, the high-mutability groups (higher body-weight and working-class people; M = 5.47, SE = 0.10) reported higher levels of SIS, on average, compared to the low-mutability groups (Black [UK]/Indigenous [Chile] and LGB people; M = 2.09, SE = 0.10).
Contrary to our hypothesis, there was no significant main effect of concealability (F [1, 208] = .37, p = .544, ηp2 < .01). Unexpectedly, there was a statistically significant interaction between mutability and concealability (F [1, 208] = 18.28, p < .001, ηp2 = .08). As shown in Figure 1, concealability had no significant effect in explaining differences among the groups low in mutability, but it did have a significant effect among the groups high in mutability. In direct contrast to our hypothesis for concealability, the low-concealability high-mutability group (higher body-weight people) was significantly higher in SIS compared to the high-concealability high-mutability group (working-class people).
We also found a statistically significant difference in the levels of SIS depending on the country where data were collected. People in Chile reported higher levels of SIS (M = 4.04, SE = 0.10) compared to people in the UK (M = 3.52, SE = 0.09). However, this effect by the country was smaller than the social group effect, explaining 7% of the variance (F [1, 208] = 14.53, p < .001, ηp2 = .07). There was no statistically significant interaction between social group and country (F [3, 208] = 0.44, p = .722, ηp2 = .01).
GIS
In the two-way ANOVA, there was a significant main effect of group on GIS (F [3, 201) = 24.16, p < .001, ηp2 = .27], this time explaining 27% of the variance. In line with our prediction for mutability, Bonferroni post-hoc tests showed that groups low in mutability, Black (UK)/Indigenous (Chile; M = 2.04, SE = 0.16) and LGB people (M = 1.81, SE = 0.15), did not differ significantly from one another, but reported lower levels of GIS when compared to groups high in mutability, higher body-weight (M = 3.14, SE = 0.15) and working-class people (M = 3.34, SE = 0.17), who did not significantly differ from one another. Consistent with this overall pattern, the three-way ANOVA revealed a significant main effect of mutability, explaining 26% of the variance (F [1, 208] = 70.93, p < .001, ηp2 = .26).
However, the pattern of group differences was not in line with our predictions for concealability. Indeed, the three-way ANOVA yielded no significant main effect of concealability (F [1, 208]< 0.01, p = .951, ηp2 < .01) and no significant interaction between mutability and concealability (F [1, 208] = 1.97, p = .162, ηp2 = .01).
In addition, and as before, levels of GIS were higher among the Chilean sample (M = 2.80, SE = 0.11) compared to the UK sample (M = 2.36, SE = 0.11). The effect of country was much smaller than the group effect, explaining 4% of the variance (F [1, 201] = 7.82, p = .006, ηp2 = .04). Finally, we did not find a significant interaction between social group and country (F [3, 201] = 0.19, p = .903, ηp2 < .01).
Study 2
Method
Participants and procedure
In Study 1, we developed and validated a multiple-group internalized stigma measure (i.e., the MGISS), and we then used this new scale to examine group differences in levels of internalized stigma. An important next step was determining the generalizability of Study 1’s results in a larger independent sample. In Study 2, as part of a larger study on discrimination and internalized stigma, we collected data from 729 emerging adult participants (18 to 21 years old) in Chile, which are used here for the purpose of replicating the analyses conducted in Study 1. This project was focused on emerging adulthood as a developmental context for internalized stigma, given that this period is characterized by critical changes in environment and identity (Arnett, 2000), which were deemed important for the study’s focus on variance and temporal change in social stigma (Earnshaw et al., 2022). Participants belonged to the same four groups that were recruited for Study 1. However, given that the study was conducted in Chile only, the racial minority group consisted solely of Indigenous-identified participants.
The sample size was calculated following the design of the larger longitudinal study from which these data were drawn, with a target sample of at least 600 participants. This initial sample size exceeded the needs for the CFA, calculating a ratio of 20 cases per parameter estimate (Jackson, 2003), for which we required a sample of over 500 participants. In the original design of this study, we planned to recruit 150 participants per group. We were able to meet this goal for three groups, but Indigenous people were under-sampled due to difficulties in reaching them via our recruitment methods (see Table 1). The final sample size was 729 participants. As in Study 1, all categories were self-reported. In this study, participants could be part of one or more than one of the groups; however, for the purpose of this analysis, we consider only their response to a single version of the scale following the criterion of amplifying the balance between groups. Further details of the sample characteristics can be found in Table 1.
Recruitment was conducted online using Instagram. Data collection was conducted between August and October 2022 using Qualtrics. We used a pre-registration form for recruitment to safeguard the integrity of the data and avoid responses from bots or duplicate respondents. Only after checking for eligibility criteria, a unique link to the study questionnaire was sent to each email address of registered participants. Participants read an information sheet and consented to their participation before completing the survey. This study received full approval from the University College London Institute of Education Ethics Committee and the P. Universidad Católica de Chile Social Sciences Ethics Committee. Data are available upon request from the corresponding author.
Measures
Demographics
Participants responded to the same demographics as presented in the description of Study 1. Self-identification questions regarding sexual orientation, race, social class, and body-weight were used to adapt the wording of items in the presentation of the MGISS in the same way as described in Study 1.
MGISS
Participants responded to the same items that were used in Study 1, including SIS and GIS subscales. These items stayed the same across groups, changing only one word in order to match the group of the respondent. All items were measured using a 7-point Likert scale. The 7-items SIS subscale Cronbach’s alpha was .95, the 5-items GIS subscale was .68, and the 12-item complete scale was .91.
Psychological distress
The same items from the Kessler-10 scale for psychological distress measure (Kessler et al., 2002) used in Study 1 were used for this second study. Cronbach’s alpha was .89.
Analysis
We conducted a CFA using the results from the main study, aiming to replicate the results of the factor structure found in Study 1. We also conducted measurement invariance analyses across groups. R software was used for this analysis. To replicate the analyses with the available data, we also conducted correlations and regression analyses between the presented variables and one-way and two-way ANOVAs for comparing the groups, following the same reasoning used in Study 1. These analyses were conducted using SPSS 27.
Results
We conducted CFA on the 12 items with two subdimensions identified in Study 1, and we compared it with an alternative model of 1-factor using the same items. We were able to replicate the findings presented in Study 1: the final 2-factor structure of the scale shows acceptable levels of fit and fit the data better than the alternative model with one factor (see Table 9).
Confirmatory factor analysis.
Note. Final model: 12 items, 2 factors. Alternative model 1: 12 items, 1 factor.
p < .01.
The scale held close-to-acceptable to borderline levels of configural and metric invariance across the four groups, but not acceptable levels of scalar and strict invariance (see Table 10).
Measurement invariance across groups (LGB, Black or Indigenous, higher body-weight, and working-class): Study 2.
Note. N = 728, LGB = 207, Black or Indigenous = 130, higher body-weight = 199, working-class = 192. MC = Model comparison reference. Scalar invariance under level of acceptance, strict invariance model not tested.
p < .01.
Similar to the results of Study 1, the association between SIS and GIS was positive, strong, and statistically significant, r(727) = .50, p < .001. In addition, to replicate the predictive validity analyses of the measure, we conducted a multiple linear regression analysis of SIS and GIS predicting psychological distress. In these analyses, we did not include felt stigma as a covariate in the model because this variable was not measured in this study. The results of the regression model are presented in Table 11. Similar to Study 1, when including both subscales in the model, only SIS had a significant association with psychological distress, even when controlling for group differences (see supplementary material for control analyses). The effect of SIS was smaller in this study compared to the first one.
Regression coefficients with psychological distress as DV.
Finally, we conducted one-way and two-way ANOVAs to replicate the group comparison analyses presented in Study 1. In this version, country was not included in the analysis because the study was entirely conducted in Chile. Results are presented in Figures 3 and 4, and the full ANOVA results can be found in the supplementary material.
Mean levels of SIS by social group and 95% confidence intervals. Letters a, b and c show post-hoc differences between social groups.
Mean levels of GIS by social group and 95% confidence intervals. Letters a, b and c show post-hoc differences between social groups.
In the one-way ANOVA, there was a significant main effect of group on SIS that explained 76% of the variance (F [3, 727] = 746.98, p < .001, ηp2 = .76). Bonferroni post-hoc tests show that the Indigenous (M = 2.34, SE = 0.09) and LGB groups (M = 2.18, SE = 0.82) were not statistically different from each other, but were significantly lower in SIS compared to the higher body-weight (M = 6.00, SE = 0.06) and working-class people (M = 5.38, SE = 0.06), and these latter two groups were also significantly different from each other.
The two-way ANOVA more directly tested our hypotheses by mapping the groups onto the combinations of the stigma characteristics of mutability and concealability. Replicating the results of Study 1, there was a significant main effect of mutability on SIS (F [1, 727] = 2083.97, p < .001, ηp2 = .74), such that the high-mutability groups (higher body-weight and working-class people; M = 5.67, SE = 0.05) reported higher levels of SIS, on average, compared to the low-mutability groups (Indigenous and LGB people; M = 2.26, SE = 0.06).
There was also a significant main effect of concealability on SIS explaining 4% of the variance (F [1, 727] = 27.90, p < .001, ηp2 = .04), but the pattern was in the opposite direction of our hypothesis: Groups low in concealability were found to present higher levels of SIS (M = 4.17, SE = 0.07) compared to those high in concealability (M = 3.78, SE = 0.05).
Again, replicating the results of Study 1, there was a statistically significant interaction between mutability and concealability explaining 1% of the variance (F [1, 727] = 9.32, p = .002, ηp2 = .01). Similar to the interaction pattern in Study 1, concealability had no significant effect in explaining differences among the groups low in mutability; that is, the high-concealability low-mutability group (LGB people) did not differ significantly in SIS from the low-concealability low-mutability group (Indigenous people). But concealability did have a significant effect in explaining differences among the groups high in mutability: The low-concealability high-mutability group (higher body-weight people) was significantly higher in SIS compared to the high-concealability high-mutability group (working-class people). This pattern aligns with the results of Study 1 but is in direct contrast to our initial hypothesis for concealability.
There were also significant group differences for GIS, explaining 19% of the variance (F [3, 725] = 58.26, p < .001, ηp2 = .19). Bonferroni post-hoc tests revealed a similar pattern of differences for SIS presented for this study, where mutability differences were in line with our hypothesis and concealability differences were contrary to our hypothesis.
Indigenous (M = 2.28, SE = 0.09) and LGB people (M = 2.04, SE = 0.06) were not significantly different in their levels of GIS. However, they were different and lower compared to groups high in mutability, higher body-weight (M = 3.29, SE = 0.08) and working-class people (M = 2.81, SE = 0.08), which were significantly different from each other. Indeed, the two-way ANOVA revealed a significant main effect of mutability in line with our hypothesis (F [1, 725] = 138.60, p < .001, ηp2 = .16).
The two-way ANOVA also revealed a significant main effect of concealability on GIS; contrary to our hypothesis, the groups low in concealability (M = 2.78, SE = 0.06) presented significantly higher levels of GIS compared to those high in concealability (M = 2.43, SE = 0.05; F [1, 725] = 21.69, p < .001, ηp2 = .03). Similar to Study 1, we did not find a statistically significant interaction between concealability and mutability in GIS differences (F [1, 725] = 2.47, p = .121, ηp2 < .01).
Discussion
In the present research, we developed and validated the MGISS: a 12-item internalized stigma scale that can be used to study and compare self-focused and group-focused internalized stigma across multiple marginalized groups. The scale’s psychometric properties were tested with data from LGB, racial minority (i.e., Black [in the UK] and Indigenous [in Chile]), higher body-weight, and working-class people in both English- and Spanish-speaking countries. Our results showed good to excellent levels of internal consistency of the scale and its subdimensions. We also found evidence for concurrent and predictive validity of the measure: it was strongly associated with existing internalized stigma scales, and it was associated with conceptually linked variables, such as felt stigma and psychological distress.
The final factor structure of the scale showed acceptable levels of fit in two different samples and revealed that internalized stigma can be measured through two subscales: SIS and GIS, as recent research has indicated (Ciaffoni et al., 2021; James, 2020). We found that the association between internalized stigma and psychological distress was explained by self-focused forms of internalized stigma, but not by internalized group-directed prejudice. Our results are aligned with initial analyses that have found negative mental health outcomes linked mainly to SIS over GIS among gay men (Ciaffoni et al., 2021). It is possible that negative beliefs directed at the self are more detrimental to people’s mental health because of the closeness between SIS and the devaluation of the self, in a process that can be harmful to people’s self-image and self-esteem. Although our results showed initial evidence towards a differentiated effects hypothesis, future research should continue examining SIS and GIS when testing if different outcomes are associated with these two subdimensions.
Regarding measurement invariance, we found the scale to have acceptable levels of structural equivalence between its English and Spanish versions, which is a positive indicator of the scale’s utility in different languages. In the case of group differences, we did not find evidence for complete measurement invariance; rather, we found borderline levels of configural and metric invariance in Study 2. However, we do not interpret these results as a deficiency of the scale, but as a reflection of the potential differences in the groups and the nature of the stigma attached to their identities that require further attention. Scholars have recently opposed the idea of measurement invariance as a prerequisite for using a scale across groups and conducting comparative analyses (Robitzsch & Lüdtke, 2023). In our data, close-to-acceptable levels of measurement invariance and similar reliability levels across groups are initial indicators of the equivalence of the construct. Future research should continue assessing commonalities and differences of the scale across groups.
Our research found relevant group differences in internalized stigma levels, which demonstrates the analytical potential of this new scale. First, we found support for our hypothesis that groups with stigmatized characteristics high in mutability (higher body-weight and working-class people) present higher levels of internalized stigma compared to groups low in mutability (Black (UK)/Indigenous (Chile) and LGB people). This general pattern was replicated in both studies and was observed for both the SIS and GIS sub-dimensions. The effect sizes for these group differences were large, and the effect sizes were larger for SIS compared to GIS. Following the literature on stigma characteristics (Jones et al., 1984; Pachankis et al., 2018), we interpret this as potentially reflecting the role of mutability as a risk factor for stigma internalization. It is possible that groups high in mutability not only experience more forms of stigma, but also self-blaming and desires for change may operate as maladaptive response mechanisms that locate the causes of discrimination within the self and not in society. Although we need more research to understand this phenomenon, these results have implications for what we know as mutability in stigmatized identities and how the social expectations around the possibility of change might have pernicious consequences for these populations. For example, mutability expectations could be a factor explaining the detrimental effects of sexual orientation change efforts for sexual minorities (e.g., Dehlin et al., 2015).
Second, we also found group differences in the stigma characteristic of concealability. Still, the results ran contrary to our hypothesis: groups low in concealability (Black (UK)/Indigenous (Chile) and higher body-weight people) presented higher levels of internalized stigma, on average, compared to groups high in concealability (LGB and working-class people). This main effect of concealability was statistically significant for both SIS and GIS in Study 2 but not Study 1, potentially because Study 2 had higher statistical power than Study 1; the overall size of this main effect of concealability was relatively small thus requiring more power to detect compared to the main effect of mutability. One potential explanation for this pattern of results is that groups that can conceal their identities may be able to hide their stigmatized characteristics and avoid discrimination from society (Pachankis et al., 2018), which in turn could lead to lower stigma internalization compared to groups low in concealability. The scale developed here can allow for future research on the psychological mechanisms for these unexpected results.
A particularly surprising effect was found when analyzing group differences in SIS: we found a robust interaction between group mutability and concealability that replicated in both studies. The pattern of this interaction indicated concealability was not significantly related to SIS among groups low in mutability, but concealability was significantly associated with SIS among groups high in mutability: the group with low concealability and high mutability (higher body-weight people) presented higher levels of SIS compared to the group with high concealability and high mutability (working-class people). One plausible theoretical explanation could be the role of high mutability in enhancing the effect of discrimination on stigma internalization directed toward the self. When stigma mutability is high, the potential heightened discrimination for those with less concealable identities (Pachankis et al., 2018) may be seen as caused by the self rather than society, leading to increased stigma internalization. This may explain why the low-concealability high-mutability group presented higher SIS levels compared to the high-concealability high-mutability group. However, when stigma mutability is low, the causes of discrimination may be seen within society rather than the self, thus protecting these individuals from additional internalized stigma. This may explain why the low-concealability, low-mutability group presented similar levels of SIS as the high-concealability, low-mutability group.
Regarding country-level differences in explaining differences in internalized stigma, there is evidence indicating cultural differences between the two samples in Study 1: the Chilean sample presented higher levels of internalized stigma compared to the UK sample. A potential explanation for this result might be discrimination levels being different between Chile and the UK. Chile has high social and residential segregation (Rasse Figueroa et al., 2021), and its laws against discrimination are more recent than in the UK and Western Europe (González, 2019). Levels of experienced and perceived discrimination among marginalized groups might be higher in Chile than in the UK, producing an increase in internalized stigma among these groups. Despite this, our results show that levels of internalized stigma seem to depend much more on the social group than the country.
The MGISS is the first attempt to create a measure of internalized stigma that allows for between-group comparisons, and the presented results evidence the potential utility of the scale for interdisciplinary research on stigma and mental health. However, we do not expect that the MGISS will replace research using existing single-group scales. These may still be more appropriate for within-group studies due to the ways in which items are constructed to reflect more particular experiences of stigma. Our scale may be especially useful in samples with multiple marginalized groups and its flexibility opens the door for approaches that require within and between-group comparisons (e.g., research on social explanations for health inequalities, Schwartz & Meyer, 2010).
Limitations and suggestions for future research and psychological interventions
Because the current research is an initial attempt to develop an internalized stigma measure for various groups, some limitations should be considered. Differences in the sample characteristics between Studies 1 and 2 need to be considered when interpreting the findings because they may explain some of the minor differences when replicating the results in Study 2. In Study 2, we only collected data in Chile and with a young sample, different from a more general adult sample from Chile and the UK in Study 1. Nevertheless, results are mainly consistent across two distinct samples, which is a good indicator of the robustness and validity of the measure. Additionally, our findings are limited to the groups included in the two studies presented here. Future research should examine whether the utility of the scale extends to additional groups (e.g., gender minorities or individuals with physical disabilities). Moreover, considering one of the hypotheses we presented around stigma characteristics worked contrary to our expectations, future research can benefit by continuing to explore these results by focusing on other groups that share these characteristics (e.g., people with stigmatized mental or physical health conditions varying in concealability and mutability).
A relevant element to consider in future research is the internal consistency of the GIS subscale. In these studies, SIS systematically showed excellent levels of internal consistency, while the GIS subscale’s internal consistency was good in Study 1 and minimally acceptable in Study 2. This might be explained by the disproportionally larger body of research around internalized stigma that has used a self-focused approach. On the other hand, attention to GIS as a second subscale is more recent, especially in Spanish. Future research should continue to assess the reliability of the measure and take action to improve it when needed.
The next steps in research using the MGISS include the use of longitudinal and intersectional approaches when researching internalized stigma. A call to pay greater consideration to the role of time in stigma research (Earnshaw et al., 2022) suggests that longitudinal studies with a multiple-group focus are necessary to shed light on the consequences of internalized stigma for various health outcomes. In addition, the study of internalized stigma requires intersectional approaches that consider multiple stigmatized identities or group memberships in one single person (Earnshaw et al., 2021). This scale can serve as an initial step into exploring how different expressions of the construct (for example, internalized homophobia and internalized racism) are associated with each other within individuals with multiple stigmatized identities, opening doors for future research on intersectionality.
Conclusion
Although a first step, this research shows promising indicators of reliability and validity of a new multi-dimensional internalized stigma scale, along with strong potential utility for research aiming to analyze internalized stigma across multiple groups and cultures. In doing so, the use of the MGISS has the potential to contribute to the study of internalized stigma, allowing for more systematic group comparison through the conceptual and methodological integration of literature around different marginalized populations. It also has the potential to bolster and expand psychological research and interventions with marginalized groups aimed at reducing internalized stigma and its impact on their well-being.
Supplemental Material
sj-docx-1-gpi-10.1177_13684302241312674 – Supplemental material for A personal but shared burden: Developing a multiple-group internalized stigma scale (MGISS)
Supplemental material, sj-docx-1-gpi-10.1177_13684302241312674 for A personal but shared burden: Developing a multiple-group internalized stigma scale (MGISS) by Diego Castro, Pranjal Mehta and David M. Frost in Group Processes & Intergroup Relations
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Agency for Research and Development (ANID), through the Chile Scholarship Program (Becas Chile/2020 – 72210362), and by the Society for the Psychological Study of Social Issues (SPSSI), through the Researchers in the Global South and Grants-In-Aid programs.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
