Measurement Invariance of the Perceived Invalidation of Emotion Scale: An Examination Across Race/Ethnicity,Gender,Sexual Orientation,and Intersection of Identities

Abstract

The Perceived Invalidation of Emotions Scale (PIES), developed to measure emotional invalidation, could aid research efforts on various internalizing disorders and minority mental health. A prerequisite for its use includes its psychometric evaluation in diverse samples; thus, the current study aimed to evaluate the psychometric properties of the PIES in a combined sample of minoritized adults (N = 876). Results supported a unidimensional structure of the PIES that was invariant across the two minoritized samples, race/ethnicity, gender, sexual orientation, and intersections of race/ethnicity and sexual orientation. A reduced 7- and 4-item PIES with improved unidimensionality and consequentially more interpretable total scores were generated using item response theory analyses. Significant correlations observed between theoretically relevant constructs of adverse mental health outcomes and the PIES above and beyond identity-based discrimination supported the construct validity of the PIES. Implications include the disproportionate amount of emotional invalidation experienced by individuals with minoritized sexual orientation, which may reflect the recent increases in discrimination faced by these individuals.

Keywords

emotional invalidation item response theory differential item functioning measurement invariance minority mental health intersectionality

Get full access to this article

View all access options for this article.

References

Aldao

Nolen-Hoeksema

Schweizer

(2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clinical Psychology Review, 30(2), 217–237.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). American Psychiatric Publishing.

Ammerman

B. A.

Burke

T. A.

Jacobucci

McClure

(2021). How we ask matters: The impact of question wording in single-item measurement of suicidal thoughts and behaviors. Preventive Medicine, 152, 106472.

Antony

M. M.

Bieling

P. J.

Cox

B. J.

Enns

M. W.

Swinson

R. P.

(1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychological Assessment, 10(2), 176–181.

Beard

Hsu

K. J.

Rifkin

L. S.

Busch

A. B.

Björgvinsson

(2016). Validation of the PHQ-9 in a psychiatric sample. Journal of Affective Disorders, 193, 267–273.

Beck

A. T.

Steer

R. A.

Ranieri

W. F.

(1988). Scale for suicide ideation: Psychometric properties of a self-report version. Journal of Clinical Psychology, 44(4), 499–505.

Baker

F. B.

(2001). The basics of item response theory (2nd ed.). ERIC Clearinghouse on Assessment and Evaluation.

Benbow

K. L.

Smith

B. L.

Tolbert

K. J.

Deska

J. C.

Kunstman

J. W.

(2022). Race, social pain minimization, and mental health. Group Processes & Intergroup Relations, 25(7), 1861–1879.

Benfer

Bardeen

J. R.

Fergus

T. A.

Rogers

T. A.

(2019). Factor structure and incremental validity of the original and modified versions of the difficulties in emotion regulation scale. Journal of Personality Assessment, 101(6), 598–608.

10.

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300.

11.

Broshek

C.E.

Kurtz

A.J.

Udupa

N.S.

Chang

C.J.

Rogers

M.L.

Feinstein

B.A.

Selby

E.A.

Gratz

K.L.

Tull

M.T.

Joiner

T.E.

Jeon

M.E.

(under review). Psychometric evaluation of the Heterosexist Harassment, Rejection, and Discrimination Scale: An Item Response Theory approach.

12.

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.

13.

Bentler

P. M.

(1995). EQS structural equations program manual (Vol. 6). Multivariate Software.

14.

Bovin

M. J.

Marx

B. P.

Weathers

F. W.

Gallagher

M. W.

Rodriguez

Schnurr

P. P.

Keane

T. M.

(2016). Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders–fifth edition (PCL-5) in veterans. Psychological Assessment, 28(11), 1379–1391.

15.

Braden

Anderson

Redondo

Watford

Emley

Ferrell

(2020). Emotion regulation mediates relationships between perceived childhood invalidation, emotional reactivity, and emotional eating. Journal of Health Psychology, 26(14), 2937–2949.

16.

Brandão

Brites

Hipólito

Nunes

(2022). Perceived emotional invalidation, emotion regulation, depression, and attachment in adults: A moderated-mediation analysis. Current Psychology. Advance online publication.

17.

Brandão

Brites

Hipólito

Nunes

(2023). Perceived emotional invalidation, emotion regulation, depression, and attachment in adults: A moderated-mediation analysis. Current Psychology, 42(18), 15773–15781.

18.

Brooks

V. R.

(1981). Minority stress and lesbian women. Lexington Books. https://cir.nii.ac.jp/crid/1130282271372632320

19.

Brosseau-Liard

P. E.

Savalei

(2014). Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49(5), 460–470.

20.

Brosseau-Liard

P. E.

Savalei

(2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47(6), 904–930.

21.

Cardona

N. D.

Madigan

R. J.

Sauer-Zavala

(2022). How minority stress becomes traumatic invalidation: An emotion-focused conceptualization of minority stress in sexual and gender minority people. Clinical Psychology: Science and Practice, 29(2), 185–195.

22.

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1–29.

23.

Chawla

Ostafin

(2007). Experiential avoidance as a functional dimensional approach to psychopathology: An empirical review. Journal of Clinical Psychology, 63(9), 871–890.

24.

Chu

Klein

K. M.

Buchman-Schmitt

J. M.

Hom

M. A.

Hagan

C. R.

Joiner

T. E.

(2015). Routinized assessment of suicide risk in clinical practice: An empirically informed update. Journal of Clinical Psychology, 71(12), 1186–1200.

25.

Clark

K. A.

Salway

McConocha

E. M.

Pachankis

J. E.

(2022). How do sexual and gender minority people acquire the capability for suicide? Voices from survivors of near-fatal suicide attempts. SSM-Qualitative Research in Health, 2, 100044.

26.

Clark

Coleman

A. P.

Novak

J. D.

(2004). Brief report: Initial psychometric properties of the everyday discrimination scale in Black adolescents. Journal of Adolescence, 27(3), 363–368.

27.

Cloitre

Schmidt

J. A.

(2022). STAIR narrative therapy. In Schnyder

Cloitre

(Eds.), Evidence based treatments for trauma-related psychological disorders: A practical guide for clinicians (pp. 307–328). Springer.

28.

Deska

J. C.

Kunstman

Lloyd

E. P.

Almaraz

S. M.

Bernstein

M. J.

Gonzales

J. P.

Hugenberg

(2020). Race-based biases in judgments of social pain. Journal of Experimental Social Psychology, 88, 103964.

29.

Dueber

(2021). Bifactor indices calculator (R package version 0.2.2). https://github.com/ddueber/BifactorIndicesCalculator

30.

Feinstein

B. A.

Franco

Henderson

Collins

L. K.

Davari

(2019). A qualitative examination of bisexual+ identity invalidation and its consequences for wellbeing, identity, and relationships. Journal of Bisexuality, 19(4), 461–482.

31.

Franssens

Abrahams

Brenning

Van Leeuwen

De Clercq

(2021). Unraveling prospective reciprocal effects between parental invalidation and pre-adolescents’ borderline traits: Between- and within-family associations and differences with common psychopathology-parenting transactions. Research on Child and Adolescent Psychopathology, 49(10), 1387–1401. https://doi.org/10.1007/s10802-021-00825-w

32.

Fredrick

E. G.

Mann

A. K.

Brooks

B. D.

Hirsch

J. K.

(2022). Anticipated to enacted: Structural stigma against sexual and gender minorities following the 2016 presidential election. Sexuality Research & Social Policy, 19(1), 345–354.

33.

Fresco

D. M.

Mennin

D. S.

Heimberg

R. G.

Turk

C. L.

(2003). Using the Penn State Worry Questionnaire to identify individuals with generalized anxiety disorder: A receiver operating characteristic analysis. Journal of Behavior Therapy and Experimental Psychiatry, 34(3–4), 283–291.

34.

Gratz

K. L.

Roemer

(2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 26, 41–54.

35.

Hambleton

R. K.

Jones

R. W.

(1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47.

36.

Harrell

F. E.

(2022). Hmisc: Harrell miscellaneous (4.7-0) [R package]. https://CRAN.R-project.org/package=Hmisc

37.

Harris

J. C.

Linder

(2018). The racialized experiences of students of color in higher education and student affairs graduate preparation programs. Journal of College Student Development, 59(2), 141–158.

38.

Haslam

Arcelus

Farrow

Meyer

(2012). Attitudes towards emotional expression mediate the relationship between childhood invalidation and adult eating concern. European Eating Disorders Review, 20(6), 510–514.

39.

Henry

J. D.

Crawford

J. R.

(2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 44(2), 227–239.

40.

Herr

N. R.

Jones

A. C.

Cohn

D. M.

Weber

D. M.

(2015). The impact of validation and invalidation on aggression in individuals with emotion regulation difficulties. Personality Disorders: Theory, Research, and Treatment, 6(4), 310–314.

41.

Hong

P. Y.

Ilardi

S. S.

Lishner

D. A.

(2011). The aftermath of trauma: The impact of perceived and anticipated invalidation of childhood sexual abuse on borderline symptomatology. Psychological Trauma: Theory, Research, Practice, and Policy, 3, 360–368. https://doi.org/10.1037/a0021261

42.

Hope

N. H.

Chapman

A. L.

(2019). Difficulties regulating emotions mediates the associations of parental psychological control and emotion invalidation with borderline personality features. Personality Disorders, 10(3), 267–274. https://doi.org/10.1037/per0000316

43.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

44.

Huxley

Seaton

D. C.

Grenyer

B. F. S.

(2021). Remembered childhood invalidation as a predictor of narcissism, personality functioning, and wellbeing. Personality and Individual Differences, 175, 110686. https://doi.org/10.1016/j.paid.2021.110686

45.

Huynh

V. W.

(2012). Ethnic microaggressions and the depressive and somatic symptoms of Latino and Asian American adolescents. Journal of Youth and Adolescence, 41, 831–846.

46.

Jeon

M. E.

Goncearenco

Rogers

M. L.

Campione

M. T.

Lamis

D. A.

Kaslow

N. J.

Joiner

T. E

. (under review). Psychometric properties of the Beck Scale for suicidal ideation in a diverse sample.

47.

Jeon

M. E.

Udupa

N. S.

Potter

M. R.

Robison

Robertson

Rogers

M. L.

Joiner

T. E.

(2024). Measurement invariance of the Depressive Symptom Inventory–Suicidality Scale across race, ethnicity, sexual orientation, and plurality of minoritized identities. Psychological Assessment, 36(4), 303–310.

48.

Johnson

S. U.

Ulvenes

P. G.

Øktedalen

Hoffart

(2019). Psychometric properties of the general anxiety disorder 7-item (GAD-7) scale in a heterogeneous psychiatric sample. Frontiers in Psychology, 10, 1713.

49.

Joiner

T. E.

Jr. Pfaff

J. J.

Acres

J. G.

(2002). A brief screening tool for suicidal symptoms in adolescents and young adults in general health settings: Reliability and validity data from the Australian National General Practice Youth Suicide Prevention Project. Behaviour Research and Therapy, 40(4), 471–481.

50.

Joiner

T. E.

Jr. Walker

R. L.

Jobes

D. A.

(2004). Scientizing and routinizing the assessment of suicidality in outpatient practice. Professional Psychology: Research and Practice, 30(5), 447–453. https://doi.org/10.1037//0735-7028.30.5.447

51.

Kang

Chen

T. T.

(2008). Performance of the generalized S-X2 item fit index for polytomous IRT models. Journal of Educational Measurement, 45(4), 391–406.

52.

Kroenke

Spitzer

R. L.

Williams

J. B.

(2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613.

53.

Kroenke

Strine

T. W.

Spitzer

R. L.

Williams

J. B.

Berry

J. T.

Mokdad

A. H.

(2009). The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders, 114(1–3), 163–173.

54.

Lambie

J. A.

Lindberg

(2016). The role of maternal emotional validation and invalidation on children’s emotional awareness. Merrill-Palmer Quarterly (1982), 62(2), 129–157.

55.

Linehan

(1993). Cognitive-behavioral treatment of borderline personality disorder. The Guilford Press.

56.

Lovibond

P. F.

Lovibond

S. H.

(1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33(3), 335–343.

57.

Lovibond

S. H.

Lovibond

P. F.

(1995). Manual for the Depression Anxiety Stress Scales. The Psychology Foundation of Australia.

58.

Maydeu-Olivares

Joe

(2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328.

59.

Meade

A. W.

(2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95(4), 728–743.

60.

Meyer

I. H.

(2003). Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: Conceptual issues and research evidence. Psychological Bulletin, 129(5), 674–697.

61.

Mohammadkhani

Attar

Akbari

(2023). The linkage between negative affectivity with emotional distress in college student: The mediator and moderator role of difficulty in emotion regulation, repetitive negative thinking, and emotional invalidation. Current Psychology, 42(8), 6216–6228.

62.

Mountford

Corstorphine

Tomlinson

Waller

(2007). Development of a measure to assess invalidating childhood environments in the eating disorders. Eating Behaviors, 8(1), 48–58.

63.

Munro

Travers

Woodford

M. R.

(2019). Overlooked and invisible: Everyday experiences of microaggressions for LGBTQ adolescents. Journal of Homosexuality, 66(10), 1439–1471.

64.

Museus

S. D.

Sariñana

S. A. L.

Yee

A. L.

Robinson

T. E.

(2016). A qualitative analysis of multiracial students’ experiences with prejudice and discrimination in college. Journal of College Student Development, 57(6), 680–697.

65.

Nadal

K. L.

(2011). The Racial and Ethnic Microaggressions Scale (REMS): Construction, reliability, and validity. Journal of Counseling Psychology, 58(4), 470–480.

66.

Nadal

K. L.

Wong

Griffin

K. E.

Davidoff

Sriken

(2014). The adverse impact of racial microaggressions on college students’ self-esteem. Journal of College Student Development, 55(5), 461–474.

67.

Orlando

Thissen

(2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64.

68.

Proujansky

R. A.

Pachankis

J. E.

(2014). Toward formulating evidence-based principles of LGB-affirmative psychotherapy. Pragmatic Case Studies in Psychotherapy, 10(2), 117–131.

69.

R Core Team. (2021). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/

70.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140.

71.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373.

72.

Roberts

A. L.

Austin

S. B.

Corliss

H. L.

Vandermorris

A. K.

Koenen

K. C.

(2010). Pervasive trauma exposure among US sexual orientation minority adults and risk of posttraumatic stress disorder. American Journal of Public Health, 100(12), 2433–2441. https://doi.org/10.2105/AJPH.2009.168971

73.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150.

74.

Rosseel

Oberski

Byrnes

Vanbrabant

Savalei

Merkle

Hallquist

Rhemtulla

Katsikatsou

Barendse

(2017). Package “lavaan.” https://lavaan.ugent.be

75.

Samejima

(2016). Graded response models. In van der Linden

W. J.

(Ed.), Handbook of item response theory (pp. 123–136). Chapman and Hall/CRC.

76.

Savalei

(2018). On the computation of the RMSEA and CFI from the mean-and-variance corrected test statistic with nonnormal data in SEM. Multivariate Behavioral Research, 53(3), 419–429. https://doi.org/10.1080/00273171.2018.1455142

77.

Schreiber

R. E.

Veilleux

J. C.

(2022). Perceived invalidation of emotion uniquely predicts affective distress: Implications for the role of interpersonal factors in emotional experience. Personality and Individual Differences, 184, 111191. https://doi.org/10.1016/j.paid.2021.111191

78.

Shenk

C. E.

Fruzzetti

A. E.

(2011). The impact of validating and invalidating responses on emotional reactivity. Journal of Social and Clinical Psychology, 30(2), 163–183.

79.

Shi

Maydeu-Olivares

Rosseel

(2020). Assessing fit in ordinal factor analysis models: SRMR vs. RMSEA. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 1–15.

80.

Sijtsma

Debets

Molenaar

I. W.

(1990). Mokken scale analysis for polychotomous items: Theory, a computer program and an empirical application. Quality and Quantity, 24, 173–188.

81.

Singh

R. S.

Bhambhani

Skinta

M. D.

Torres-Harding

S. R.

(2021). Measurement of intersectional microaggressions: Conceptual barriers and recommendations. Perspectives on Psychological Science, 16(5), 956–971.

82.

Spitzer

R. L.

Kroenke

Williams

J. B.

Löwe

(2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097.

83.

Steiger

J. H.

(1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173–180.

84.

Stucky

B. D.

Edelen

M. O.

(2015). Using hierarchical IRT models to create unidimensional measures from multidimensional data. In Reise

S. P.

Revicki

D. A.

(Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183–206). Routledge/Taylor & Francis Group.

85.

Stucky

B. D.

Gottfredson

N. C.

Panter

A. T.

Daye

C. E.

Allen

W. R.

Wightman

L. F.

(2011). An item factor analysis and item response theory-based revision of the Everyday Discrimination Scale. Cultural Diversity and Ethnic Minority Psychology, 17(2), 175–185.

86.

Sturrock

Mellor

(2014). Perceived emotional invalidation and borderline personality disorder features: A test of theory. Personality and Mental Health, 8(2), 128–142.

87.

Sue

D. W.

(2010). Microaggressions and marginality: Manifestation, dynamics, and impact. Wiley.

88.

Sue

D. W.

Bucceri

Lin

A. I.

Nadal

K. L.

Torino

G. C.

(2007). Racial microaggressions and the Asian American experience. Cultural Diversity and Ethnic Minority Psychology, 13(1), 72–81.

89.

Swann

Minshew

Newcomb

M. E.

Mustanski

(2016). Validation of the Sexual Orientation Microaggression Inventory in two diverse samples of LGBTQ youth. Archives of Sexual Behavior, 45, 1289–1298.

90.

Szymanski

D. M.

(2006). Does internalized heterosexism moderate the link between heterosexist events and lesbians’ psychological distress? Sex Roles, 54(3), 227–234.

91.

Thoma

B. C.

Rezeppa

T. L.

Choukas-Bradley

Salk

R. H.

Marshal

M. P.

(2021). Disparities in childhood abuse between transgender and cisgender adolescents. Pediatrics, 148(2), e2020016907.

92.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10.

93.

Van der Ark

L. A

. (2007). Mokken scale analysis in R. Journal of Statistical Software, 20, 1–19.

94.

Waller

Corstorphine

Mountford

(2007). The role of emotional abuse in the eating disorders: Implications for treatment. Eating Disorders, 15(4), 317–331.

95.

Weathers

F. W.

Blake

D. D.

Schnurr

P. P.

Kaloupek

D. G.

Marx

B. P.

Keane

T. M.

(2013). The life events checklist for DSM-5 (LEC-5). National Center for PTSD.

96.

Weathers

F. W.

Litz

B. T.

Keane

T. M.

Palmieri

P. A.

Marx

B. P.

Schnurr

P. P.

(2013). The PTSD checklist for DSM-5 (PCL-5). National Center for PTSD. https://www.ptsd.va.gov

97.

Westphal

Leahy

R. L.

Pala

A. N.

Wupperman

(2016). Self-compassion and emotional invalidation mediate the effects of parental indifference on psychopathology. Psychiatry Research, 242, 186–191.

98.

Wortmann

J. H.

Jordan

A. H.

Weathers

F. W.

Resick

P. A.

Dondanville

K. A.

Hall-Clark

Foa

E. B.

Young-McCaughan

Yarvis

J. S.

Hembree

E. A.

(2016). Psychometric analysis of the PTSD Checklist-5 (PCL-5) among treatment-seeking military service members. Psychological Assessment, 28(11), 1392–1403.

99.

Yap

M. B. H.

Allen

N. B.

Ladouceur

C. D.

(2008). Maternal socialization of positive affect: The impact of invalidation on adolescent emotion regulation and depressive symptomatology. Child Development, 79(5), 1415–1431.

100.

Yen

W. M.

(1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145.

101.

Zielinski

M. J.

Veilleux

J. C.

(2018). The Perceived Invalidation of Emotion Scale (PIES): Development and psychometric properties of a novel measure of current emotion invalidation. Psychological Assessment, 30(11), 1454–1467.

102.

Zielinski

M. J.

Veilleux

J. C.

Fradley

M. F.

Skinner

K. D.

(2023). Perceived emotion invalidation predicts daily affect and stressors. Anxiety, Stress, & Coping, 36(2), 214–228.