Sage Journals: Discover world-class research

Abstract

In the present study, we aimed to translate and validate two instruments designed to assess self-stigmatization: the Self-Stigma of Mental Illness Scale and the Paradox of Self-Stigma Scale, among Romanian psychiatric patients. Responses from 326 psychiatric patients (58% women) diagnosed with internalizing and externalizing disorders were collected to evaluate the instruments’ structural validity, measurement invariance, reliability, convergent and criterion validity. Confirmatory factor analyses offered support for the intended structure of both instruments and the pattern of associations with the other investigated variables further sustained their validity. Reliability was optimal, as revealed by excellent internal consistency estimates. For the first time, the invariance of these instruments was tested based on gender and psychiatric diagnosis. Our results suggest that both instruments can be confidently used by Romanian specialists, providing an effective toolkit for measuring self-stigmatization related to mental health.

Keywords

self-stigma psychiatric patients cultural adaptation psychometric properties invariance

Introduction

Internalized stigma refers to the process by which an individual incorporates the negative evaluations of others into their self-concept, later becoming part of their system of personal value. This process has been associated with a range of negative outcomes, including lower self-esteem, reduced self-efficacy, poorer quality of life, diminished social support, and poorer responses to treatment (e.g., Corrigan & Rao, 2012; Livingston & Boyd, 2010; Yanos et al., 2020).

Given its impact on patients’ health beyond the condition itself, it is recommended that psychiatrists, clinical psychologists, and psychotherapists address the phenomenon of self-stigmatization resulting from receiving a psychiatric diagnosis (Corrigan et al., 2014). Consequently, numerous instruments have been developed to measure this phenomenon among psychiatric patients. A systematic review of psychometric assessments of internalized stigma (Stevelink et al., 2012) shows that the most commonly used instruments for measuring self-stigma related to mental health issues are the Internalized Stigma of Mental Illness Scale (ISMI; Boyd et al., 2003), the Self Stigma of Mental Illness Scale (SSMIS; Corrigan et al., 2006), the Self-Stigma Scale–Short (SSS-S; Mak & Cheung, 2010), the Depression Self-Stigma Scale (DSSS; Kanter et al., 2008), and the Self-Stigma of Depression Scale (SSDS; Barney et al., 2010). Additionally, a more recent systematic review (Wei et al., 2018) reveals evidence of a significant number of past research that validated the shorter version of SSMIS, namely the Self-stigma of Mental Illness Scale-Short Form (Corrigan et al., 2012). More recently, Golay et al. (2021) developed a new instrument to measure not only stigma internalization but also the paradoxical dimension of self-stigma, referred to as righteous anger (Paradox of Self-Stigma Scale – PaSS-24; Golay et al., 2021). The concept of righteous anger is based on the theoretical insight of Corrigan and Watson (2002), who observed that some patients exposed to public stigma (as a precursor to internalized stigma) react with anger rather than internalizing it. Including this type of emotional response in a stigma measurement tool provides a much more comprehensive understanding of what psychiatric patients exposed to stigma may experience, both cognitively and emotionally.

According to the meta-analytical study of Yu et al. (2021), the more collectivistic a culture is, the stronger the correlation between experienced/perceived stigma and self-stigma. Given that Romania is still considered a collectivistic culture (David, 2015), we believe that having valid instruments is particularly important. Even though there are several robust instruments that can be used by both researchers and clinicians, in Romania, to our knowledge, the only previously used scale is the ISMI (Boyd et al., 2003), which was translated as part of a European study (Boyd et al., 2014; Brohan et al., 2011) but was not culturally validated. However, it was successfully used in consecutive studies on Romanian psychiatric samples, for example, in testing the relationship between stigmatization and the quality of life among individuals with schizophrenia (Căpățână & Miculița, 2018; Gagiu et al., 2024). The ISMI measures different aspects of self-stigma: alienation, stereotype endorsement, discrimination experience, social withdrawal, and stigma resistance, which can bring more comprehensive information about the degree of self-stigmatization. Although this is a valuable tool that should be further adapted to the Romanian culture, we opted to validate the SSMIS (Corrigan et al., 2006) and the PaSS-24 (Golay et al., 2021) instruments because they focus on the process of internalization (where a patient first becomes aware of societal stereotypes, then either accepts and incorporates them into their self-concept or resists them) rather than solely on its outcomes, such as reduced self-esteem.

We consider that capturing different facets of the self-stigma process has two major advantages. First, it enables the identification of individuals who may be more prone to internalizing stigma at an early stage, by quantifying, for example, their awareness of public stigma (e.g., through the “Aware” subscale of the SSMIS). Second, measuring the process rather than the outcome is particularly useful for designing preventive interventions that aim to interrupt or mitigate the internalization process, especially given that self-related beliefs are typically difficult to change in psychiatric patients (Hovelius et al., 2021). Thus, in the following paragraphs, we will present the two selected instruments, both developed based on Corrigan’s conceptualization of the stigma internalization process (Corrigan et al., 2006, 2011; Corrigan & Watson, 2002).

The Self Stigma of Mental Illness Scale (SSMIS; Corrigan et al., 2006) is a well-known instrument designed to give both researchers and clinicians the opportunity to operationalize the stages of stigma internalization. These stages, conceptualized in Corrigan’s social cognitive model (Corrigan et al., 2011) of self-stigma, include stereotype awareness, stereotype agreement, stereotype self-concurrence, and self-harm. Specifically, in the first stage, a person with a diagnosis becomes aware of what others may think about their condition (e.g., “People think that individuals with depression are lazy”). In the next stage, the person agrees with the public stereotype (“I also consider that people with depression are lazy”). The third step is applying the social stereotype on oneself (“Because I have depression, I am a lazy person”), and the final stage, self-harm, represents a decrease in self-esteem (“I respect myself less because I am a lazy person”). This conceptualization has strong support (e.g., Catalano et al., 2021; Corrigan et al., 2011, 2012; Göpfert et al., 2019) and is one of the most prolific models for understanding the process of internalization. Additionally, besides the original form, we want to examine the instrument’s short form. The Self Stigma of Mental Illness Scale – Short Form (SSMIS-SF; Corrigan et al., 2012) is a more concise version of the original one (20 items instead of 40), while still measuring the same constructs. The 20 items are included in the original 40-item instrument. Comparing the two forms of the same instrument could provide a wider perspective for future researchers and practitioners, helping them to choose between the two forms, considering both the psychometric properties and application time.

We also chose to adapt the Paradox of Self-Stigma Scale (PaSS-24; Golay et al., 2021) because it addresses an under-studied phenomenon, namely the paradox of self-stigma (Corrigan & Watson, 2002), which complements the aforementioned model of self-stigmatization. Instead of succumbing to negative self-attitudes, some individuals manage to respond with indifference or righteous anger. This phenomenon, characterized by a lack of concern for stigma, illustrates the self-stigma paradox. Moreover, compared to the other studies referenced, there is a clear need for more research to provide empirical evidence of the scale’s cross-cultural validity, given its novelty.

In accordance with the Test Adaptation Reporting Standards (TARES; Iliescu et al., 2024), we would like to address a potential construct bias. Both the conceptualization of righteous anger (Corrigan & Watson, 2002) and its operationalization through the PaSS-24 (Golay et al., 2021) were developed in individualistic cultures, specifically American and French, where emotions like anger are viewed as a legitimate personal response to perceived injustice, emphasizing individual rights and autonomy. In contrast, in collectivistic cultures such as Romania is still generally considered to be, anger is typically viewed through a relational lens and may even be discouraged (Mesquita & Leu, 2007). As a result, there is a risk that a measurement tool based on individualistic perspectives may fail to capture the complexities of anger experienced within collectivist contexts. However, Romania is currently undergoing a cultural transition, with trends indicating a gradual shift toward more individualistic values (David, 2015). Given this evolving context, we considered it important to take the risk of adapting and validating the PaSS-24, aiming to prepare a tool that is not only relevant for the present but also adaptable for future cultural changes.

Beyond cultural adaptation, this article brings attention for the first time to the invariance of the instruments across both gender and psychiatric diagnosis, addressing a significant gap in the literature and enabling meaningful comparisons between these categories of individuals.

To conclude, this study has two main objectives. The first is to culturally validate the Self-Stigma of Mental Illness Scale and the Paradox of Self-Stigma Scale on Romanian psychiatric patients, encouraging both researchers and clinicians to utilize these tools in their future work. The second objective is to provide, for the first time, evidence regarding the invariance of these instruments while also confirming their established psychometric properties.

Method

Ethics Approval

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2000 (World Medical Association, 2000). The University Scientific Commitee for Research and Artistic Creation granted ethical approval (27,300/11.05.2022) to the present study before its initiation.

Participants

Based on the pragmatic recommendations of White (2022), which take into account both statistical power and accessibility to participants in instrument validation studies, we conducted our data collection with 334 psychiatric patients who completed the questionnaires. Of these, 8 were excluded because of the amount of missing data or inconsistency of answers, remaining in the study 326 participants (41.4% males, M_age = 48.95, SD_age = 12.58). Inclusion criteria were to have a psychiatric diagnosis and be over 18 years of age. Most of the participants had a long-term psychiatric history (over 20 years in many cases) and were living on a minimum income (47.2%). Data collection was conducted in a Romanian psychiatric facility and a mental health clinic. Participation in the study was voluntary, and all participants signed an informed consent form before taking part. No financial or material compensation was offered. While participants’ identities were known to the researchers at the time of consent, all data were handled confidentially and securely, and responses were de-identified prior to analysis. Diagnoses were primarily based on participants’ self-reports. When they felt comfortable disclosing their diagnosis, it was confirmed by the attending clinician. In cases of comorbidities, participants in this study were grouped based on their Axis I diagnosis. The sample included both inpatients and outpatients, depending on the site of recruitment. To improve readability, all sociodemographic characteristics of participants are displayed in Table SM1 in the Supplementary Material.

Instruments

The Self Stigma of Mental Illness Scale (SSMIS; Corrigan et al., 2006) contains 40 items, with 10 items for each of the 4 stages of Corrigan’s model: awareness (“I think the public believes that most persons with mental illness are dangerous.”), acceptance (“I think most persons with mental illness are dangerous.”), applying (“Because I have a mental illness, I am dangerous”) and self-harm (“I currently respect myself less because I am dangerous”). The stereotypes that are considered by the items of the questionnaire are the blame for one’s mental condition, unpredictability, inability to recover, danger of a mentally ill person, and inability to take care of themselves. All items are rated on a 1 (strongly disagree) to 9 (strongly agree) point scale. A high score means the responder is more aware/affected by the stereotype. The shorter version of the instrument (Self-Stigma of Mental Illness Scale-Short Form - SSMIS-SF; Corrigan et al., 2012) contains 20 items (extracted from the long version), with 5 items for each of the stages of stigma internalization.

Paradox of Self Stigma Scale (PaSS-24; Golay et al., 2021) is an instrument developed in collaboration with healthcare specialists and users, designed to measure three constructs related to self-stigma, namely stereotype endorsement (“People with my condition will never have a happy life.”), righteous anger (“The stereotypes about my condition make me angry.”) and non-disclosure (“I use strategies to avoid talking about my condition.”). It contains 24 items, 8 for each of the 3 subscales, on a 1 (strongly disagree) to 5 (strongly agree) point Likert scale. A high score means the responder’s self-stigmatization/anger is more pronounced.

Internalized Stigma of Mental Illness Scale – 9 items (ISMI-9; Hammer & Toland, 2017) is a brief instrument designed to measure the degree of stigma internalization, being based on the 29-item version of ISMI, which was previously translated and used in the Romanian culture (Boyd et al., 2014). It contains 9 items, on a 1 (strongly disagree) to 4 (strongly agree) point Likert scale, two of which are reverse-scored (“In general, I am able to live life the way I want to.” and “I can have a good, fulfilling life, despite my mental illness.”). On the sample of the present study, we obtained an unexpected result regarding internal consistency, namely the Cronbach’α is better without reversing the two items mentioned above (α = .84 without reversing and α = .75 with items reversed). Because of this unusual result, we decided to exclude the two items in all future analyses. The internal consistency for the 7-item scale is α = .87 on the sample of this study.

The Satisfaction With Life Scale (SWLS; Diener et al., 1985) is a short instrument designed to measure one’s general level of satisfaction with life. All items are formulated in general terms. It contains 5 items, on a 1 (strongly disagree) to 7 (strongly agree) point Likert scale. This instrument has a good internal consistency (α = .89) on the present sample.

General Health Questionnaire-12 (GHQ-12; Goldberg, 1972) is a widely used instrument designed to measure short-term psychological functioning changes (over the last two weeks). It contains 12 items with four response options. A high score indicates greater health complaints. The instrument demonstrated excellent internal consistency in the present sample (α = .94).

Process of Translation

First, we obtained permission from the authors of the main instruments for their use and cultural adaptation. Consequently, we used a forward and backward translation design according to ITC Guidelines for Translating and Adapting Tests (Gregoire, 2018). All forward and backward translations of the instruments were done independently by two authors of this study, both of whom are clinical psychologists. They are native Romanian speakers with English as their second language (minimum C1 level) and a deep understanding of both Romanian culture and clinical context. This ensures that the translations are culturally appropriate and clinically relevant. For example, in the PaSS-24, the term “condition” would have been literally translated into Romanian as “condiție”, but this term does not carry the same health-related connotation and may be interpreted more broadly (e.g., as physical condition or social status). Therefore, the adapted version clarifies the intended meaning by explicitly referring to “mental health problems”, ensuring conceptual equivalence across languages. There was no use of artificial intelligence in the translation process. The translated versions of the instruments are available in the Supplemental Material.

Data Analysis

All the statistical analysis was conducted in R (R Core Team, 2024). Even if we excluded 8 cases from the initial sample, we were still confronted with missing data since the responses were collected in a paper-and-pencil format. One solution was to conduct the analysis using complete cases, which would constrain us to use 82.5% of the initial data. However, to avoid statistical power reduction, we used the multiple imputation method (Rubin, 2004). This method implies generating replacement values (“imputations”) for missing data and repeating this procedure for m times to result in m data sets with replaced information. We applied the recommended number of 20 imputations (Graham et al., 2007). For this purpose, we used the mice package (Van Buuren & Groothuis-Oudshoorn, 2011).

The main goal of our study was to test the equivalence of the structures of the instruments across Romanian language versions. Since the items of both scales were skewed, as revealed by statistically significant One-Sample Kolmogorov-Smirnov tests, we used confirmatory factor analysis (CFA) with the Maximum Likelihood with Robust Standard Errors (MLR) estimation method via the lavaan (Rosseel, 2012) and semTools (Jorgensen et al., 2022) packages. This is a robust estimation method used to address violations of standard normality assumptions in the data and/or ordinal data. For model fit we reported the widely used chi-square (χ2) index of fit and the following less sensitive fit indices (Nye, 2022): Comparative Fit Index (CFI ≥.90), Tucker-Lewis Index (TLI ≥.90), Root Mean Square Error of Approximation (RMSEA ≤.08), and the Standardized Root Mean Square Residual (SRMR ≤.08). For both versions of SSMI, we conducted three CFAs, testing the following models: one-factor model, three factors model, and the original four factors model. We tested the one-factor model to see if self-stigmatization can be considered a general construct. Based on the findings of Göpfert et al. (2019), which suggest an overlap between personal agreement and self-concurrence, we opted to test a three-factor model, combining the acceptance and applying stages into a single factor. Finally, these alternative models were compared to the original four factors model (4 stages process). For PaSS24, we conducted three CFAs as well: the one-factor model, the two-factor model, and the original three-factor model. The two-factor model was tested to differentiate the phenomenon (stereotype endorsement) from the responses to it (non-disclosure and righteous anger).

Consequently, by means of multi-group confirmatory factor analysis (MGCFA), we tested the invariance of the instruments across gender (male and female; there were no cases of non-binary participants) and type of diagnosis (internalizing vs. disorders). Internalizing disorders are those characterized by anxiety, depressive, and somatic symptoms, and externalizing disorders are those characterized by impulsive, disruptive conduct, substance use, and other addictive symptoms (American Psychiatric Association, 2013). So, we included bipolar and related disorders, depressive disorders, anxiety disorders (Goldberg & Fawcett, 2012), obsessive-compulsive and related disorders (Snorrason et al., 2021), and eating disorders (Mitchell et al., 2014) into internalization disorders category. We include patients with psychotic disorders, neurocognitive disorders, substance-related, addictive disorders (Krueger et al., 2005), and suicide attempts (Verona et al., 2004) into the externalizing disorders category.

To assess reliability, we estimated both instruments’ internal consistency using McDonald’s ω and Cronbach’s α coefficients, with values ≥.70 reflecting acceptable internal consistency (Nunnally, 1978).

In order to test the convergent validity of the instruments, we calculated the average variance extracted (AVE), an analysis used to measure if the amount of variance captured by a construct is based on a latent variable rather than the measurement error, considering a score above .50 as adequate (Dos Santos & Cirillo, 2021).

To further investigate the construct validity (i.e., convergent), we tested the associations between all three instruments (comprising eight subscales) designed to measure variations of the same construct – self-stigmatization.

Data Availability

To enhance the transparency of this study and facilitate its reproducibility, the data and code utilized in the analyses are publicly accessible at the following link: https://osf.io/4b5yk.

Results

Structural Validity

We aimed to test the structure of the SSMIS (long and short versions) and the PaSS-24. To achieve this, we conducted eleven confirmatory factor analyses, all of which are reported in Table 1. Additionally, Table SM2 in the Supplementary Material presents the model comparison statistics. As can be seen, the best-fitting models are the four-factor ones for both the SSMIS long form (χ² = 1751.12, df = 732, CFI = .845, TLI = .835, RMSEA = .083, SRMR = .063) and short version (χ² = 368.79, df = 162, CFI = .923, TLI = .909, RMSEA = .080, SRMR = .057), with two correlated items suggested by the modification indices function. However, the only version with acceptable fit indices is the short one. For the PaSS-24, the most suitable model is the original three-factor structure (χ² = 522.18, df = 249, CFI = .914, TLI = .904, RMSEA = .070, SRMR = .053). The standardized factor loadings ranged from .59 to .84 for SSMIS-SF and from .61 to .84 for the PaSS-24. These are illustrated in Figures SM1 and SM2 in the Supplementary Material.

Table 1

Results of the Confirmatory Factor Analysis Models for SSMIS, SSMIS-SF, and PaSS-24

Model	χ2	df	CFI	TLI	RMSEA	SRMR
1.SSMIS-1	3232.13**	740	.613	.592	.131	.115
2.SSMIS-3	2405.95**	737	.744	.729	.106	.089
3.SSMIS-4a	1880.98**	734	.826	.815	.088	.065
4.SSMIS-4b	1751.12**	732	.845	.835	.083	.063
5.SSMIS-SF-1	933.95**	170	.702	.667	.153	.103
6.SSMIS-SF-3	681.07**	167	.803	.776	.126	.080
7.SSMIS-SF-4a	510.91**	164	.868	.848	.104	.062
8.SSMIS-SF-4b	368.79**	162	.923	.909	.080	.057
9.PaSS24-1	700.01**	252	.857	.843	.090	.061
10.PaSS24-2	598.06**	251	.889	.878	.079	.056
11.PaSS24-3	522.18**	249	.914	.904	.070	.053

1, 2, 3, 4 = number of factors; a = original model; b = improved original model by correlated errors; *p < .05. **p < .01.

Invariance

As we reported in Table 2, all indices suggest that configural and metric invariance for SSMIS-SF are supported across genders (female vs. male). Regarding scalar invariance, despite the good model-fit indices, the statistically significant result of the chi-square difference test suggests that there is a lack of scalar invariance for SSMI-SF (Δχ² = 29.28, df = 16, p < .05). Regarding Pass-24, while the fit indices appear to support its configural invariance, the chi-square difference test indicates a lack of metric invariance (Δχ² = 32.88, df = 21, p < .05).

Table 2

Invariance Analysis as a Function of Participants’ Gender for SSMIA-SF and PaSS-24

Model	χ² (df)	CFI	RMSEA	SRMR	Comp. Mod.	Δχ² (Δdf)	p	ΔCFI	ΔRMSEA	ΔSRMR
SSMI-SF
1. Config	599.23 (324)*	.90	.09	.06
2. Metric	619.49 (340)*	.90	.08	.07	1 vs. 2	14.76 (16)	.54	.000	−.002	.006
3. Scalar	648.72 (356)*	.90	.08	.07	2 vs. 3	29.28 (16)	.02	−.003	−.001	.002
PaSS
1. Config	867.76 (498)*	.89	.08	.06
2. Metric	902.15 (519)*	.89	.08	.07	1 vs. 2	32.88 (21)	.04	−.002	−.001	.010

Comp. Mod. = Models’ comparisons. *p < .05. **p < .01.

As for the type of diagnosis (internalizing vs. externalizing disorders), all indices reported in Table 3 indicate that configural, metric, scalar, and strict invariance are supported for both the SSMIS and PaSS-24.

Table 3

Invariance Analysis as a Function of Diagnosis Type for SSMIS-SF and PaSS-24

Model	χ² (df)	CFI	RMSEA	SRMR	Comp.Mod.	Δχ² (Δdf)	p	ΔCFI	ΔRMSEA	ΔSRMR
SSMIS-SF
1. Config	600.58 (324)*	.89	.09	.06
2. Metric	620.63 (340)*	.89	.09	.06	1 vs. 2	15.86 (16)	.46	0	−.002	.005
3. Scalar	645.08 (356)*	.89	.09	.06	2 vs. 3	22.21 (16)	.13	−.002	−.002	.001
4. Strict	663.48 (376)*	.89	.09	.06	3 vs. 4	24.47 (20)	.22	−.003	−.001	.000
PaSS
1. Config	978.81 (498)*	.84	.09	.06
2. Metric	1011.27 (519)*	.84	.09	.07	1 vs. 2	28.39 (21)	.12	−.002	−.002	.011
3. Scalar	1033.54 (540)*	.84	.09	.08	2 vs. 3	17.00 (21)	.71	.001	−.002	.001
4. Strict	1058.06 (552)*	.84	.09	.07	3 vs. 4	27.65 (24)	.27	−.001	−.002	.001

Comp. Mod. = Models’ comparisons. *p < .05. **p < .01.

Reliability

The internal consistency coefficients for both SSMIS and PaSS-24 were adequate, with McDonald’s ω values ranging between .89 and .94 and Cronbach’s α values ranging between .87 and .92. Complete values are reported in Table SM3 in the Supplementary Material.

Construct and Criterion Validity

The convergent validity for SSMIS is acceptable for both (long/short) versions, as the average variance extracted (AVE) was greater than .50 (Awareness = .51/57, Acceptance = .60/.57, Applying = .56/.59, and Self-harm = .63/.60). These results indicate that each construct captures a sufficient proportion of variance from its indicators, supporting the reliability and validity of the short form.

For PaSS-24, convergent validity is marginal for the Stereotype Endorsement subscale (AVE = .47) but acceptable for the other two subscales: Righteous Anger = .52 and Non-Disclosure = .58.

The correlations between the various subscales of the two main instruments ranged from .42 to .84 (p < .001). Additionally, all subscales showed significant positive correlations with ISMI-9, supporting the convergent construct validity. This indicates that the scales consistently measure the same underlying construct. However, based on the original conceptualization of Righteous Anger, the moderate positive correlation between Righteous Anger and Self-Harm (r = .48, p < .001) is unexpected. All correlations are displayed in Table SM3 in the Supplementary Material.

The positive correlations between the subscales of the two main instruments and health complaints, alongside the absence or negative correlations between these subscales and life satisfaction, are a good indicator for the criterion validity of the instruments. Higher levels of internalized stigma are associated with more health issues, which is in line with the theoretical assumption that internalized stigma can negatively affect health. Moreover, higher levels of self-stigma are linked to lower life satisfaction, aligning with the expectation that stigma internalization can negatively impact overall well-being. All these findings are further discussed.

Discussion

The main objective of this study was to culturally adapt, for Romanian psychiatric patients, two instruments designed to measure aspects of self-stigmatization related to psychiatric diagnosis: the Self-Stigma of Mental Illness Scale (SSMIS; Corrigan et al., 2006; SSMIS-SF; Corrigan et al., 2012) and the Paradox of Self-Stigma Scale (PaSS-24; Golay et al., 2021).

Our CFA analysis suggests that the short version of Corrigan’s instrument represents an adequate reflection of the four stages of the process of stigma internalization in the Romanian culture. This discrepancy between the versions of the instrument can be a result of the accentuation of some stereotypes in Romanian society (included in the shorter form) compared with others (included in the longer version). This idea could also represent an explanation for the correlated errors that are related to the two stereotypes: the blame of a patient for his condition and the dangerousness of a psychiatric patient. Moreover, the strong association between applying and the self-harm stage suggests that our participants, who have a long history of psychiatric diagnoses, may not differentiate well between the last stages of stigma internalization. Regarding the PaSS-24, the structure proposed by the authors maintains its validity in our collectivistic cultural context.

We assessed measurement invariance for both instruments across two criteria: gender and type of diagnosis. Due to limited responses, gender was analyzed in a binary manner (male and female). Our findings indicate that the PaSS-24 is perceived and interpreted differently across genders, suggesting that the constructs it measures may not be equivalent for men and women. Additionally, the lack of scalar invariance for the SSMI-SF suggests systematic differences in item responses between genders, limiting the validity of direct mean comparisons. So, it is possible that previous studies reporting gender differences in stigma internalization (Corrigan & Watson, 2007) may have been influenced by measurement biases. Beyond potential measurement biases, these gender-related differences might also reflect theoretically meaningful variations, such as gendered differences in emotional expression (Deng et al., 2016), which may influence how individuals report internalized stigma. According to the authors, in the context of negative emotions, although men and women had similar emotional experiences, women showed stronger emotional expressivity. Although qualitative follow-up was not feasible in the current study, future investigations should further examine the sources and implications of this lack of measurement invariance to better understand how these instruments function differently across gender groups. Exploring these dimensions in future work may clarify whether measurement inequivalence reflects methodological artifacts or deeper conceptual divergences between groups. Moreover, these results also highlight that, for the Romanian versions of the instruments, direct gender comparisons are not advisable.

Regarding the type of diagnosis, we examined whether the instruments maintained their psychometric properties across internalizing disorders (e.g., depression and anxiety) and externalizing disorders (e.g., substance abuse) to test their applicability range. Our results support configural, metric, and scalar invariance for both the SSMIS and PaSS-24, indicating that these instruments are suitable for diverse psychiatric patient groups.

Convergent validity was assessed by calculating the average variance extracted (AVE). The only score below the acceptable threshold of .50 was for the stereotype endorsement subscale of the PaSS-24 (.46). However, this result is considered acceptable when accounting for the subscale’s strong internal consistency (Fornell & Larcker, 1981). Additionally, all instruments demonstrated good to excellent internal consistency, with McDonald’s omega values exceeding .90.

The positive correlations between all subscales of the two main instruments and internalized stigma as measured with ISMI-9 (Hammer & Toland, 2017) further support the construct validity. The weaker associations observed between the first two stages of stigma internalization and the ISMI-9 may reflect the distinction between different aspects of the self-stigmatization process, namely, the existence of social stereotypes (captured in the first two stages), on the one hand, and the outcome of internalization (measured by ISMI-9), on the other hand.

For criterion validity, we examined correlations between the two adapted instruments and life satisfaction (measured with SWLS; Diener et al., 1985) and mental health complaints (measured with GHQ-12; Goldberg, 1972). Concerning life satisfaction, significant associations were found only with the Acceptance subscale (positive correlation) and the Stereotype Endorsement subscale (negative correlation). The positive association between acceptance of social stereotypes and life satisfaction may be explained by the congruity effect, which suggests that greater alignment between an individual’s internal value system and the prevailing societal values is linked to personal well-being (Sagiv & Schwartz, 2000). The negative association between stereotype endorsement and life satisfaction, alongside positive associations between self-stigma aspects and mental health complaints, aligns with previous research on the adverse outcomes of stigma internalization (Fond et al., 2023; Sarraf et al., 2022), supporting the criterion validity of both instruments.

Cumulatively, this evidence supports the validity of the SSMIS and PaSS-24 in Romanian culture. While the SSMIS is designed to capture the staged process of stigma internalization, the PaSS-24 and ISMI-9 measure “normal” and “paradoxical” behaviors resulting from this process.

Nevertheless, our study has several limitations that should be acknowledged. One limitation is that adapting the SSMIS, an instrument designed to measure stages of the internalization process that theoretically occurs around the time of diagnosis, may seem somewhat forced when applied to a clinical population with long-term mental health problems. Thus, future studies could provide empirical evidence regarding this instrument by applying it to recently diagnosed patients. Another limitation of the study, related to the PaSS-24, is that the translation and adaptation of the instrument were done from the English version rather than the original French version. Therefore, there is a possibility that some nuances may have been lost, as subtle meanings and cultural references might not have been fully captured through a secondary translation. Thus, we recommend that in future studies, if possible, the adaptation of the instrument be conducted from the language in which it was originally developed.

Last but not least, the absence of normative data or data from a non-clinical population to establish normative benchmarks limits our ability to define clear cutoff scores indicating high levels of internalization. However, reporting means and standard deviations may provide valuable descriptive information about the distribution of internalization scores within the clinical sample. In the current sample, the central tendency scores for each self-stigma factor were slightly below the scale midpoint. However, the clinical significance of this finding is difficult to assess. Future psychometric research should consider developing normative data for these instruments to enhance their utility in clinical practice.

Conclusion

Based on the findings of this study, we conclude that both the Self-Stigma of Mental Illness Scale (especially the short form) and the Paradox of Self-Stigma Scale are valuable tools for understanding self-stigmatization related to psychiatric diagnoses within Romanian culture. These instruments maintain proper psychometric properties, making them suitable resources for researchers and practitioners in Romania to effectively address self-stigmatization.

Supplemental Material

Supplemental Material - Measuring Self-Stigma Among Romanian Psychiatric Patients: Cultural Adaptation and Further Psychometric Insights for the Self-Stigma of Mental Illness Scale and the Paradox of Self-Stigma Scale

Supplemental Material for Measuring Self-Stigma Among Romanian Psychiatric Patients: Cultural Adaptation and Further Psychometric Insights for the Self-Stigma of Mental Illness Scale and the Paradox of Self-Stigma Scale by Luana-Maria Alexa, Andrei Rusu, Juliette Richetin, Ioana-Maria Latu, Mugur Daniel Ciumăgeanu, Ana-Maria Radu, Camelia Maria Dindelegan, and Ciprian Ionuț Băcilă in Evaluation & the Health Professions

Footnotes

ORCID iDs

Luana-Maria Alexa

Andrei Rusu

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was supported by an internal grant from the Faculty of Sociology and Psychology, West University of Timişoara (grant no. 15636/15.03.2022).

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

AI Usage

During the preparation of this work the authors used ChatGPT as an editing tool to improve the wording and conciseness of the manuscript. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the content of the published article.

Supplemental Material

Supplemental material for this article is available online.

References

American Psychiatric Association . (2013). Diagnostic and statistical manual of mental disorders (5th ed.). American Psychiatric Publishing.

Barney

L. J.

Griffiths

K. M.

Christensen

Jorm

A. F.

(2010). The self-stigma of depression scale (SSDS): Development and psychometric evaluation of a new instrument. International Journal of Methods in Psychiatric Research, 19(4), 243–254. https://doi.org/10.1002/mpr.325

Boyd

J. E.

Adler

E. P.

Otilingam

P. G.

Peters

(2014). Internalized stigma of mental illness (ISMI) scale: A multinational review. Comprehensive Psychiatry, 55(1), 221–231. https://doi.org/10.1016/j.comppsych.2013.06.005

Boyd

J. E.

Otilingam

P. G.

Grajales

(2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121(1), 31–49. https://doi.org/10.1016/j.psychres.2003.08.008

Brohan

Gauci

Sartorius

Thornicroft

GAMIAN-Europe Study Group . (2011). Self-stigma, empowerment and perceived discrimination among people with bipolar disorder or depression in 13 European countries: The GAMIAN–europe study. Journal of Affective Disorders, 129(1-3), 56–63. https://doi.org/10.1016/j.jad.2010.09.001

Căpățînă

Micluția

(2018). Internalized stigma as a predictor of quality of life in schizophrenia. Journal of Evidence-Based Psychotherapies, 18(2), 35–53. https://doi.org/10.24193/jebp.2018.2.13

Catalano

L. T.

Brown

C. H.

Lucksted

Hack

S. M.

Drapalski

A. L.

(2021). Support for the social-cognitive model of internalized stigma in serious mental illness. Journal of Psychiatric Research, 137, 41–47. https://doi.org/10.1016/j.jpsychires.2021.02.014

Corrigan

P. W.

Druss

B. G.

Perlick

D. A.

(2014). The impact of mental illness stigma on seeking and participating in mental health care. Psychological Science in the Public Interest: A Journal of the American Psychological Society, 15(2), 37–70. https://doi.org/10.1177/1529100614531398

Corrigan

P. W.

Michaels

P. J.

Vega

Gause

Watson

A. C.

Rüsch

(2012). Self-stigma of mental illness scale-short form: Reliability and validity. Psychiatry Research, 199(1), 65–69. https://doi.org/10.1016/j.psychres.2012.04.009

10.

Corrigan

P. W.

Rafacz

Rüsch

(2011). Examining a progressive model of self-stigma and its impact on people with serious mental illness. Psychiatry Research, 189(3), 339–343. https://doi.org/10.1016/j.psychres.2011.05.024

11.

Corrigan

P. W.

Rao

(2012). On the self-stigma of mental illness: Stages, disclosure, and strategies for change. Canadian Journal of Psychiatry. Revue Canadienne De Psychiatrie, 57(8), 464–469. https://doi.org/10.1177/07067437120570080

12.

Corrigan

P. W.

Watson

A. C.

(2002). The paradox of self-stigma and mental illness. Clinical Psychology: Science and Practice, 9(1), 35–53. https://doi.org/10.1093/clipsy.9.1.35

13.

Corrigan

P. W.

Watson

A. C.

(2007). The stigma of psychiatric disorders and the gender, ethnicity, and education of the perceiver. Community Mental Health Journal, 43(5), 439–458. https://doi.org/10.1007/s10597-007-9084-9

14.

Corrigan

P. W.

Watson

A. C.

Barr

(2006). The self–stigma of mental illness: Implications for self–esteem and self–efficacy. Journal of Social and Clinical Psychology, 25(8), 875–884. https://doi.org/10.1521/jscp.2006.25.8.875

15.

David

(2015). Psihologia poporului român: Profilul psihologic al românilor într-o monografie cognitiv-experimentală [the psychology of the Romanian people: The psychological profile of romanians in a cognitive-experimental monograph]. Polirom.

16.

Deng

Chang

Yang

Huo

Zhou

(2016). Gender differences in emotional response: Inconsistency between experience and expressivity. PLoS One, 11(6), Article e0158666. https://doi.org/10.1371/journal.pone.0158666

17.

Diener

E. D.

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13

18.

Dos Santos

P. M.

Cirillo

M. A.

(2021). Construction of the average variance extracted index for construct validation in structural equation models with adaptive regressions. Communications in Statistics - Simulation and Computation, 52(4), 1–13. https://doi.org/10.1080/03610918.2021.1888122

19.

Fond

Vidal

Joseph

Etchecopar-Etchart

Solmi

Yon

D. K.

Correll

C. U.

Boyer

(2023). Self-stigma in schizophrenia: A systematic review and meta-analysis of 37 studies from 25 high-and low-to-middle income countries. Molecular Psychiatry, 28(5), 1–12. https://doi.org/10.1038/s41380-023-02003-4

20.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.1177/002224378101800104

21.

Gagiu

Dionisie

Manea

M. C.

Mazilu

D. C.

Manea

(2024). Internalised stigma, self-esteem and perceived social support as psychosocial predictors of quality of life in adult patients with schizophrenia. Journal of Clinical Medicine, 13(22), 6959. https://doi.org/10.3390/jcm13226959

22.

Golay

Moga

Devas

Staecheli

Poisat

Israël

Suter

Silva

Morandi

Ferrari

Favrod

Bonsack

(2021). Measuring the paradox of self-stigma: Psychometric properties of a brief scale. Annals of General Psychiatry, 20(1), 1–11. https://doi.org/10.1186/s12991-021-00325-7

23.

Goldberg

(1972). The detection of psychiatric illness by questionnaire: A technique for the identification and assessment of non-psychotic psychiatric illness. Oxford University Press.

24.

Goldberg

Fawcett

(2012). The importance of anxiety in both major depression and bipolar disorder. Depression and Anxiety, 29(6), 471–478. https://doi.org/10.1002/da.21939

25.

Göpfert

N. C.

Conrad von Heydendorff

Dreßing

Bailer

(2019). Applying Corrigan’s progressive model of self-stigma to people with depression. PLoS One, 14(10), e0224418. https://doi.org/10.1371/journal.pone.0224418

26.

Graham

J. W.

Olchowski

A. E.

Gilreath

T. D.

(2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science: The Official Journal of the Society for Prevention Research, 8(3), 206–213. https://doi.org/10.1007/s11121-007-0070-9

27.

Gregoire

(2018). ITC guidelines for translating and adapting tests. International Journal of Testing, 18(2), 101–134.

28.

Hammer

J. H.

Toland

M. D.

(2017). Internal structure and reliability of the internalized stigma of mental illness scale (ISMI-29) and brief versions (ISMI-10, ISMI-9) among Americans with depression. Stigma and Health, 2(3), 159–174. https://doi.org/10.1037/sah0000049

29.

Hovelius

Lindén

Bengtsson

Håkansson

(2021). Self-schema, attachment style, and treatment outcome of patients in an opiate maintenance treatment unit. Frontiers in Psychology, 12, Article 595883. https://doi.org/10.3389/fpsyg.2021.595883

30.

Iliescu

Bartram

Zeinoun

Ziegler

Elosua

Sireci

Geisinger

K. F.

Odendaal

Oliveri

M. E.

Twing

Camara

(2024). The test adaptation reporting standards (TARES): Reporting test adaptations. International Journal of Testing, 24(1), 80–102. https://doi.org/10.1080/15305058.2023.2294266

31.

Jorgensen

T. D.

Pornprasertmanit

Schoemann

A. M.

Rosseel

(2022). SemTools: Useful tools for structural equation modeling (R package version 0.5-6) [Computer software]. https://CRAN.R-project.org/package=semTools

32.

Kanter

J. W.

Rusch

L. C.

Brondino

M. J.

(2008). Depression self-stigma: A new measure and preliminary findings. The Journal of Nervous and Mental Disease, 196(9), 663–670. https://doi.org/10.1097/NMD.0b013e318183f8af

33.

Krueger

R. F.

Markon

K. E.

Patrick

C. J.

Iacono

W. G.

(2005). Externalizing psychopathology in adulthood: A dimensional-spectrum conceptualization and its implications for DSM-V. Journal of Abnormal Psychology, 114(4), 537–550. https://doi.org/10.1037/0021-843X.114.4.537

34.

Livingston

J. D.

Boyd

J. E.

(2010). Correlates and consequences of internalized stigma for people living with mental illness: A systematic review and meta-analysis. Social Science & Medicine, 71(12), 2150–2161. https://doi.org/10.1016/j.socscimed.2010.09.030

35.

Mak

W. W.

Cheung

R. Y.

(2010). Self-stigma among concealable minorities in Hong Kong: Conceptualization and unified measurement. American Journal of Orthopsychiatry, 80(2), 267–281. https://doi.org/10.1111/j.1939-0025.2010.01030.x

36.

Mesquita

Leu

(2007). The cultural psychology of emotion. In Kitayama

Cohen

(Eds.), Handbook of cultural psychology (pp. 734–759). The Guilford Press.

37.

Mitchell

K. S.

Wolf

E. J.

Reardon

A. F.

Miller

M. W.

(2014). Association of eating disorder symptoms with internalizing and externalizing dimensions of psychopathology among men and women. International Journal of Eating Disorders, 47(8), 860–869. https://doi.org/10.1002/eat.22300

38.

Nunnally

J. C.

(1978). An overview of psychological measurement. In Wolman

B. B.

(Ed.), Clinical diagnosis of mental disorders: A handbook (pp. 97–146). Springer. https://doi.org/10.1007/978-1-4684-2490-4_4

39.

Nye

C. D.

(2022). Reviewer resources: Confirmatory factor analysis. Organizational Research Methods, 26(4), 608–628. https://doi.org/10.1177/10944281221120541

40.

R Core Team . (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. [Computer software]. https://www.R-project.org/

41.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

42.

Rubin

D. B.

(2004) Multiple imputation for nonresponse in surveys (81). John Wiley & Sons.

43.

Sagiv

Schwartz

S. H.

(2000). Value priorities and subjective well-being: Direct relations and congruity effects. European Journal of Social Psychology, 30(2), 177–198. https://doi.org/10.1002/(SICI)1099-0992(200003/04)30:2<177::AID-EJSP982>3.0.CO;2-Z

44.

Sarraf

Lepage

Sauvé

(2022). The clinical and psychosocial correlates of self-stigma among people with schizophrenia spectrum disorders across cultures: A systematic review and meta-analysis. Schizophrenia Research, 248, 64–78. https://doi.org/10.1016/j.schres.2022.08.001

45.

Snorrason

Beard

Peckham

A. D.

Björgvinsson

(2021). Transdiagnostic dimensions in obsessive-compulsive and related disorders: Associations with internalizing and externalizing symptoms. Psychological Medicine, 51(10), 1657–1665. https://doi.org/10.1017/S0033291720000380

46.

Stevelink

I. C.

Voorend

C. G.

Van Brakel

W. H.

(2012). The psychometric assessment of internalised stigma instruments: A systematic review. Stigma, Research and Action, 2(2), 100–118. https://doi.org/10.1017/S0033291720000380

47.

Van Buuren

Groothuis-Oudshoorn

(2011). Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. https://doi.org/10.18637/jss.v045.i03

48.

Verona

Sachs-Ericsson

Joiner Jr

T. E.

(2004). Suicide attempts associated with externalizing psychopathology in an epidemiological sample. American Journal of Psychiatry, 161(3), 444–451. https://doi.org/10.1176/appi.ajp.161.3.444

49.

Wei

McGrath

Hayden

Kutcher

(2018). The quality of mental health literacy measurement tools evaluating the stigma of mental illness: A systematic review. Epidemiology and Psychiatric Sciences, 27(5), 433–462. https://doi.org/10.1017/S2045796017000178

50.

White

(2022). Sample size in quantitative instrument validation studies: A systematic review of articles published in scopus, 2021. Heliyon, 8(12), Article e12223. https://doi.org/10.1016/j.heliyon.2022.e12223

51.

World Medical Association . (2000). World medical association declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA, 284(23), 3043–3045. https://doi.org/10.1001/jama.284.23.3043

52.

Yanos

P. T.

DeLuca

J. S.

Roe

Lysaker

P. H.

(2020). The impact of illness identity on recovery from severe mental illness: A review of the evidence. Psychiatry Research, 288, Article 112950. https://doi.org/10.1016/j.psychres.2020.112950

53.

B. C. L.

Chio

F. H. N.

Mak

W. W.

Corrigan

P. W.

Chan

K. K.

(2021). Internalization process of stigma of people with mental illness across cultures: A meta-analytic structural equation modeling approach. Clinical Psychology Review, 87, Article 102029. https://doi.org/10.1016/j.cpr.2021.102029

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.28 MB