Item Response Theory Analysis and Differential Item Functioning of the Social Appearance Anxiety Scale

Abstract

This study examined the item- and scale-level functioning of the Social Appearance Anxiety Scale (SAAS) as well as differential functioning by gender using an item response theory (IRT) analysis. SAAS data collected from 840 college students were analyzed. A graded response model was used to analyze the 16 items comprising the SAAS. The measure was found to be unidimensional in its factor structure, and every item demonstrated high to very high ability to differentiate respondents varying in levels of the underlying trait (i.e., appearance concerns). In addition, we found evidence of differential item functioning (DIF) by gender for four items, corresponding to small effect sizes. Two of these items were related to internal experiences of appearance concerns (e.g., nervousness and discomfort when a flaw is noticed by others) that were more likely to be endorsed by women, and two of the items were related to external evaluative experiences related to appearance (e.g., missing opportunities and life being more difficult) that were more likely to be endorsed by men. Overall, the IRT and DIF results suggest that the SAAS effectively identifies appearance concerns among individuals with low to very high appearance concerns.

Keywords

social appearance anxiety scale appearance concerns item response theory differential item functioning

Get full access to this article

View all access options for this article.

References

American Psychiatric Association. (2022). Diagnostic and statistical manual of mental disorders (5th ed., text rev.). https://doi.org/10.1176/appi.books.9780890425787

Asher

Asnaani

Aderka

I. M.

(2017). Gender differences in social anxiety disorder: A review. Clinical Psychology Review, 56, 1–12.

Baker

F. B.

(2001). The basics of item response theory (2nd ed.). ERIC Clearinghouse on Assessment and Evaluation.

Bean

G. J.

Bowen

N. K.

(2021). Item response theory and confirmatory factor analysis: Complementary approaches for scale development. Journal of Evidence-Based Social Work, 18(6), 597–618.

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. https://doi.org/10.1037//0033-2909.107.2.238

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. Guilford Press.

Browne

M. W.

Cudeck

(1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230–258. https://doi.org/10.1177/0049124192021002005

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Clark

P. C.

LaHuis

D. M.

(2012). An examination of power and type I errors for two differential item functioning indices using the graded response model. Organizational Research Methods, 15(2), 229–246.

10.

Clausen

Rosenvinge

J. H.

Friborg

Rokkedal

(2011). Validating the Eating Disorder Inventory-3 (EDI-3): A comparison between 561 female eating disorders patients and 878 females from the general population. Journal of Psychopathology and Behavioral Assessment, 33(1), 101–110. https://doi.org/10.1007/s10862-010-9207-4

11.

Connor

K. M.

Davidson

J. R.

Churchill

L. E.

Sherwood

Weisler

R. H.

Foa

(2000). Psychometric properties of the Social Phobia Inventory (SPIN): New self-rating scale. The British Journal of Psychiatry, 176(4), 379–386.

12.

Doğan

(2011). An investigation of the psychometric properties of the social appearance anxiety scale in an adolescent sample. Elementary Education Online, 10(1), 12–19.

13.

Espelage

D. L.

Mazzeo

S. E.

Aggen

S. H.

Quittner

A. L.

Sherman

Thompson

(2003). Examining the construct validity of the Eating Disorder Inventory. Psychological Assessment, 15(1), 71–80.

14.

French

B. F.

Maller

S. J.

(2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67, 373–393.

15.

Garner

D. M.

Olmstead

M. P.

Polivy

(1983). Development and validation of a multidimensional eating disorder inventory for anorexia nervosa and bulimia. International Journal of Eating Disorders, 2(2), 15–34.

16.

Gerrard

Galli

Santurri

Franklin

(2021). Examining body dissatisfaction in college men through the exploration of appearance anxiety and internalization of the mesomorphic ideal. Journal of American College Health, 69(5), 560–566.

17.

Gomez

Vance

(2008). Parent ratings of ADHD symptoms: Differential symptom functioning across Malaysian Malay and Chinese children. Journal of Abnormal Child Psychology, 36, 955–967. https://doi.org/10.1007/s10802-008-9226-8

18.

Gordon

A. R.

Moore

L. B.

Guss

(2021). Eating disorders among transgender and gender non-binary people. In Nagata

J. M.

Brown

T. A.

Murray

S. B.

Lavender

J. M.

(Eds.), Eating disorders in boys and men (pp. 265–281). Springer.

19.

Hambleton

R. K.

Jones

R. W.

(1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12, 38–47.

20.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory (Vol. 2). SAGE.

21.

Hart

T. A.

Flora

D. B.

Palyo

S. A.

Fresco

D. M.

Holle

Heimberg

R. G.

(2008). Development and examination of the Social Appearance Anxiety Scale. Assessment, 15(1), 48–59.

22.

Hart

T. A.

Rotondi

N. K.

Souleymanov

Brennan

D. J.

(2015). Psychometric properties of the Social Appearance Anxiety Scale among Canadian gay and bisexual men of color. Psychology of Sexual Orientation and Gender Diversity, 2(4), 470–481. https://doi.org/10.1037/sgd0000140

23.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

24.

Jankowski

G. S.

Diedrichs

P. C.

Atkinson

M. J.

Fawkner

Gough

Halliwell

(2017). A pilot controlled trial of a cognitive dissonance-based body dissatisfaction intervention with young British men. Body Image, 23, 93–102.

25.

Jones

B. A.

Haycraft

Murjan

Arcelus

(2018). Body dissatisfaction and disordered eating in trans people: A systematic review of the literature. Gender dysphoria and gender incongruence, 28, 81–94.

26.

Jöreskog

K. G.

Goldberger

A. S.

(1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of American Statistical Association, 70, 631–639. https://doi.org/10.2307/2285946

27.

Keating

Stephens

Thomas

Castle

D. J.

Rossell

S. L.

(2016). Gender differences in weight-related and non-weight-related appearance concerns in a community sample. Australian Journal of Psychology, 68(1), 11–19.

28.

Klemchuk

H. P.

Hutchinson

C. B.

Frank

R. I.

(1990). Body dissatisfaction and eating-related problems on the college campus: Usefulness of the Eating Disorder Inventory with a nonclinical population. Journal of Counseling Psychology, 37(3), 297–305.

29.

Levinson

C. A.

Rodebaugh

T. L.

(2011). Validation of the Social Appearance Anxiety Scale: Factor, convergent, and divergent validity. Assessment, 18(3), 350–356.

30.

Linardon

Phillipou

Castle

Newton

Harrison

Cistullo

L. L.

Griffiths

Hindle

Brennan

(2018). Feeling fat in eating disorders: Testing the unique relationships between feeling fat and measures of disordered eating in anorexia nervosa and bulimia nervosa. Body Image, 25, 163–167.

31.

Malcolm

Pikoos

T. D.

Castle

D. J.

Rossell

S. L.

(2021). An update on gender differences in major symptom phenomenology among adults with body dysmorphic disorder. Psychiatry Research, 295, Article 113619.

32.

Masters

G. N.

(1988). Item discrimination: When more is worse. Journal of Educational Measurement, 25(1), 15–29.

33.

McNeish

Hancock

G. R.

(2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43–52.

34.

Meade

A. W.

(2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 94, 728–743.

35.

Moscovitch

D. A.

Rowa

Paulitzki

J. R.

Ierullo

M. D.

Chiang

Antony

M. M.

McCabe

R. E.

(2013). Self-portrayal concerns and their relation to safety behaviors and negative affect in social anxiety disorder. Behaviour Research and Therapy, 51(8), 476–486.

36.

Orfi

Ahmadkaraji

Farahani

Watson

Elhami Athar

(2023). Concern about appearance on social media: Psychometric evaluation and item response theory analysis with Iranian University students. International Journal of Human–Computer Interaction, 40(22), 7704–7719. https://doi.org/10.1080/10447318.2023.2267919

37.

Osório

F. L.

Crippa

J. A. S.

Loureiro

S. R.

(2010). Evaluation of the psychometric properties of the Social Phobia Inventory in university students. Comprehensive Psychiatry, 51(6), 630–640.

38.

Patel

T. A.

Cougle

J. R.

(2024a). A pilot open trial of a text message safety behavior fading intervention for appearance concerns among women. Behavior Therapy. Advance online publication. https://doi.org/10.1016/j.beth.2024.05.010

39.

Patel

T. A.

Cougle

J. R.

(2024b). An experimental examination of appearance-related safety behaviors in a clinical sample of women. Journal of Psychopathology and Clinical Science, 133(5), 368–377. https://dx.doi.org/10.1037/abn0000926

40.

Patel

T. A.

Cougle

J. R.

(2024c). Safety behavior reduction for appearance concerns: A randomized controlled trial of a smartphone-based intervention. Journal of Consulting and Clinical Psychology. Advance online publication. https://doi.org/10.1037/ccp0000920

41.

Patel

T. A.

Summers

B. J.

Wilver

N. L.

Cougle

J. R.

(2023). Reliability and validity of the self-report version of the Yale-Brown Obsessive-Compulsive Scale modified for Body Dysmorphic Disorder (BDD-YBOCS). Assessment, 30, 1935–1946.

42.

Penfield

R. D.

(2014). An NCME instructional module on polytomous item response theory models. Educational Measurement: Issues and Practice, 33, 36–48.

43.

Phillips

K. A.

Hollander

Rasmussen

S. A.

Aronowitz

B. R.

(1997). A severity rating scale for body dysmorphic disorder: Development, reliability, and validity of a modified version of the Yale-Brown Obsessive Compulsive Scale. Psychopharmacology Bulletin, 33(1), 17.

44.

Pliner

Chaiken

Flett

G. L.

(1990). Gender differences in concern with body weight and physical appearance over the life span. Personality and Social Psychology Bulletin, 16(2), 263–273.

45.

R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

46.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

47.

Samejima

(2016). Graded response models. In Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of item response theory (Vol. 1, pp. 123–136). Chapman & Hall.

48.

Schuster

Negy

Tantleff-Dunn

(2013). The effects of appearance-related commentary on body dissatisfaction, eating pathology, and body change behaviors in men. Psychology of Men & Masculinity, 14(1), 76–87. https://doi.org/10.1037/a0025625

49.

Striegel-Moore

R. H.

Rosselli

Perrin

DeBar

Wilson

G. T.

May

Kraemer

H. C.

(2009). Gender difference in the prevalence of eating disorder symptoms. The International Journal of Eating Disorders, 42(5), 471–474. https://doi.org/10.1002/eat.20625

50.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. https://doi.org/10.1007/BF02291170

51.

Veale

(2002). Over-valued ideas: A conceptual analysis. Behaviour Research and Therapy, 40, 383–400. https://doi.org/10.1016/S0005-7967(01)00016-X

52.

Wilver

N. L.

Summers

B. J.

Cougle

J. R.

(2020). Effects of safety behavior fading on appearance concerns and related symptoms. Journal of Consulting and Clinical Psychology, 88(1), 65–74.

53.

Zamantakis

Lackey

(2022). Dying to be (a) gendered: An exploratory content analysis of trans/nonbinary people’s experiences with eating disorders. Sociological Inquiry, 92, 870–893.