U.S. military populations experience a high level of mental health concerns, including post-traumatic stress disorder, clinical depression, and suicide, when compared with their civilian counterparts, and tend to access mental health services at a lower rate. Military health scholars have noted that stigma against mental health help-seeking has multiple sources, including professional, personal, and social components, though these components are rarely separated in examining why military service members avoid clinical help. Valid measurement of these factors is necessary to examine the heart of rising clinical needs. The current study replicates and extends prior work applying a bifactor model to the Military Stigma Scale (MSS). In a sample of n = 1,832 Army National Guard members, a bifactor model presented acceptable fit, though invariance testing by rank and education indicates disparate experiences with military service as deviating influences. Specifically, Private Stigma was significantly lower in higher paygrade service members and those with a college degree, while Public Stigma was higher. Results call into question the theoretical viability of a bifactor model of the MSS, especially in the evaluation of Expected Common Variance and specific factor reliability.
ApplegarthD. M.WoodD. S.BryanA. O.BryanC. J. (2019). Examining help-seeking among National Guard service members. Military Behavioral Health, 7(2), 198–205. https://doi.org/10.1080/21635781.2018.1526143
2.
AriasD.SaxenaS.VerguetS. (2022). Quantifying the global burden of mental disorders and their economic value. Eclinicalmedicine, 54, Article 101675. https://doi.org/10.1016/j.eclinm.2022.101675
3.
BeauducelA.HerzbergP. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
BenutoL. T.CasasJ.GonzalezF.NewlandsR. (2020). The behavioral model of health: Education, behavioral health factors, and stigma as predictors of help-seeking attitudes. Community Mental Health Journal, 56(7), 1275–1283. https://doi.org/10.1007/s10597-020-00601-y
6.
BestJ. R.MillerP. H.NaglieriJ. A. (2011). Relations between executive function and academic achievement from ages 5 to 17 in a large, representative national sample. Learning and Individual Differences, 21(4), 327–336. https://doi.org/10.1016/j.lindif.2011.01.007
7.
BlaisR. K. (2016). Preliminary evidence of differences between stigma source versus type with individual functioning variables in National Guard/Reserve troops. Military Psychology, 28(3), 162–173. https://doi.org/10.1037/mil0000119
8.
BlaisR. K.RenshawK. D. (2013). Stigma and demographic correlates of help-seeking intentions in returning service members. Journal of Traumatic Stress, 26(1), 77–85. https://doi.org/10.1002/jts.21772
9.
BlaisR. K.RenshawK. D. (2014). Self-stigma fully mediates the association of anticipated enacted stigma and help-seeking intentions in National Guard Service Members. Military Psychology, 26(2), 114–119. https://doi.org/10.1037/mil0000036
10.
BollenK. A.BarbK. H. (1981). Pearson’s R and coarsely categorized measures. American Sociological Review, 46(2), 232. https://doi.org/10.2307/2094981
BrittT. W.SiposM. L.KlinefelterZ.AdlerA. B. (2020). Determinants of mental and physical health treatment-seeking among military personnel. The British Journal of Psychiatry, 217(2), 420–426. https://doi.org/10.1192/bjp.2019.155
13.
BrittT. W.WrightK. M.MooreD. (2012). Leadership as a predictor of stigma and practical barriers toward receiving mental health treatment: A multilevel approach. Psychological Services, 9(1), 26–37. https://doi.org/10.1037/a0026412
14.
BrownM. C.CreelA. H.EngelC. C.HerrellR. K.HogeC. W. (2011). Factors associated with interest in receiving help for mental health problems in combat veterans returning from deployment to Iraq. Journal of Nervous & Mental Disease, 199(10), 797–801. https://doi.org/10.1097/NMD.0b013e31822fc9bf
15.
BrownN. B.BruceS. E. (2016). Stigma, career worry, and mental illness symptomatology: Factors influencing treatment-seeking for Operation Enduring Freedom and Operation Iraqi Freedom soldiers and veterans. Psychological Trauma: Theory, Research, Practice, and Policy, 8(3), 276–283. https://doi.org/10.1037/tra0000082
16.
BrownT. A. (2015). Confirmatory factor analysis for applied research (Second edition). The Guilford Press.
17.
BryanC. J.JenningsK. W.JobesD. A.BradleyJ. C. (2012). Understanding and preventing military suicide. Archives of Suicide Research, 16(2), 95–110. https://doi.org/10.1080/13811118.2012.667321
18.
ChenF. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
19.
ClaryK. L.PenaS.SmithD. C. (2023). Masculinity and stigma among emerging adult military members and veterans: Implications for encouraging help-seeking. Current Psychology, 42(6), 4422–4438. https://doi.org/10.1007/s12144-021-01768-7
20.
ClementS.SchaumanO.GrahamT.MaggioniF.Evans-LackoS.BezborodovsN.MorganC.RüschN.BrownJ. S. L.ThornicroftG. (2015). What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychological Medicine, 45(1), 11–27. https://doi.org/10.1017/S0033291714000129
21.
CohenK. A.Stiles-ShieldsC.WinquistN.LattieE. G. (2021). Traditional and nontraditional mental healthcare services: Usage and preferences among adolescents and younger adults. The Journal of Behavioral Health Services & Research, 48(4), 537–553. https://doi.org/10.1007/s11414-020-09746-w
22.
ColemanS. J.StevelinkS. A. M.HatchS. L.DennyJ. A.GreenbergN. (2017). Stigma-related barriers and facilitators to help seeking for mental health issues in the Armed Forces: A systematic review and thematic synthesis of qualitative literature. Psychological Medicine, 47(11), 1880–1892. https://doi.org/10.1017/S0033291717000356
23.
CornishM. A.BrennerR. E.VogelD. L.WadeN. G. (2019). Evaluation of an online help seeking stigma-reduction intervention for military personnel. Stigma and Health, 4(4), 480–486. https://doi.org/10.1037/sah0000167
CramerR. J.MontanaroE.VanSickleM.CacaceS.ZabelskiS.SmithE. L.FranksM.GroverS.CunninghamC. A. (2022). A psychometric assessment of the Military Suicide Attitudes Questionnaire (MSAQ). Psychiatry Research, 317, Article 114849. https://doi.org/10.1016/j.psychres.2022.114849
26.
DrussB.G.RosenheckR.A. (1997). Use of medical services by veterans with mental disorders. Psychosomatics, 38, 451–458.
27.
FikretogluD.GuayS.PedlarD.BrunetA. (2008). Twelve month use of mental health services in a nationally representative, active military sample. Medical Care, 46(2), 217–223. https://doi.org/10.1097/MLR.0b013e31815b979a
28.
FlakeJ. K.FriedE. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
29.
GignacG. E. (2016). The higher-order model imposes a proportionality constraint: That is why the bifactor model tends to fit better. Intelligence, 55, 57–68. https://doi.org/10.1016/j.intell.2016.01.006
30.
GoffmanE. (1986). Stigma: Notes on the management of spoiled identity. Simon & Schuster Inc.
31.
GormanL. A.BlowA. J.AmesB. D.ReedP. L. (2011). National guard families after combat: Mental health, use of mental health services, and perceived treatment barriers. Psychiatric Services, 62(1), 28–34. https://doi.org/10.1176/ps.62.1.pss6201_0028
32.
Greene-ShortridgeT. M.BrittT. W.CastroC. A. (2007). The stigma of mental health problems in the military. Military Medicine, 172(2), 157–161. https://doi.org/10.7205/MILMED.172.2.157
33.
GriffithJ.WestC. (2013). Master resilience training and its relationship to individual well-being and stress buffering among Army National Guard soldiers. The Journal of Behavioral Health Services & Research, 40(2), 140–155. https://doi.org/10.1007/s11414-013-9320-8
34.
HammerJ. H.VogelD. L.Heimerdinger-EdwardsS. R. (2013). Men’s help seeking: Examination of differences across community size, education, and income. Psychology of Men & Masculinity, 14(1), 65–75. https://doi.org/10.1037/a0026813
35.
HazellC. M.FixsenA.BerryC. (2022). Is it time to change the approach of mental health stigma campaigns? An experimental investigation of the effect of campaign wording on stigma and help-seeking intentions. PLOS ONE, 17(8), Article e0273254. https://doi.org/10.1371/journal.pone.0273254
36.
HoT. E.HesseC. M.OsbornM. M.SchneiderK. G.SmischneyT. M.CarlisleB. L.BenedaJ. G.SchwerinM. J.ShechterO. G. (2018). Mental health and help-seeking in the U.S. military: Survey and focus group findings (Technical report PERSEREC-TR-18-10, pp. 1–55). Office of People Analytics. https://apps.dtic.mil/sti/pdfs/AD1059321.pdf
37.
HomM. A.StanleyI. H.SchneiderM. E.JoinerT. E. (2017). A systematic review of help-seeking and mental health service utilization among military service members. Clinical Psychology Review, 53, 59–78. https://doi.org/10.1016/j.cpr.2017.01.008
38.
HoopsickR. A.VestB. M.HomishD. L.HomishG. G. (2024). United States Army Reserve/National Guard soldiers’ healthcare experiences, attitudes, and preferences: Differences based on deployment status. Psychology, Health & Medicine, 29, 1195–1207. https://doi.org/10.1080/13548506.2024.2303409
39.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
40.
KintzleS.LeeJ. J.RamirezA. (2023). Understanding the experience and mental health challenges of National Guard and Reserve service members. In WarnerC. H.CastroC. A. (Eds.), Veteran and military mental health: A psychiatrist’s role (1st ed., pp. 99–112). American Psychiatric Association.
41.
KomiyaN.GoodG. E.SherrodN. B. (2000). Emotional openness as a predictor of college students’ attitudes toward seeking psychological help. Journal of Counseling Psychology, 47(1), 138–143. https://doi.org/10.1037/0022-0167.47.1.138
42.
LaiJ.-S.TeresiJ.GershonR. (2005). Procedures for the analysis of differential item functioning (DIF) for small sample sizes. Evaluation & the Health Professions, 28(3), 283–294. https://doi.org/10.1177/0163278705278276
43.
LiC.-H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7
44.
MarshH. W.GuoJ.ParkerP. D.NagengastB.AsparouhovT.MuthénB.DickeT. (2018). What to do when scalar invariance fails: The extended alignment method for multi-group factor analysis comparison of latent means across many groups. Psychological Methods, 23(3), 524–545. https://doi.org/10.1037/met0000113
45.
MarshH. W.MorinA. J. S.ParkerP. D.KaurG. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10(1), 85–110. https://doi.org/10.1146/annurev-clinpsy-032813-153700
46.
McNeishD.WolfM. G. (2023). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61–88. https://doi.org/10.1037/met0000425
47.
MomenN.StrychaczC. P.ViirreE. (2012). Perceived stigma and barriers to mental health care in Marines attending the Combat Operational Stress Control Program. Military Medicine, 177(10), 1143–1148. https://doi.org/10.7205/MILMED-D-11-00286
NovakL. A.LaCroixJ. M.PereraK. U.StiversM.SchveyN. A.GoodieJ. L.OlsenC.SbroccoT.GoldstonD. B.SoumoffA.WeaverJ.Ghahramanlou-HollowayM. (2023). Help-seeking among psychiatrically hospitalized military personnel at risk for suicide. Suicide and Life-threatening Behavior, 53(1), 75–88. https://doi.org/10.1111/sltb.12923
50.
OjedaV. D.BergstresserS. M. (2008). Gender, race-ethnicity, and psychosocial barriers to mental health care: An examination of perceptions and attitudes among adults reporting unmet need. Journal of Health and Social Behavior, 49(3), 317–334. https://doi.org/10.1177/002214650804900306
51.
PakK.FerreiraK. E.Ghahramanlou-HollowayM. (2019). Suicide postvention for the United States military: Literature review, conceptual model, and recommendations. Archives of Suicide Research, 23(2), 179–202. https://doi.org/10.1080/13811118.2018.1428704
52.
PanterA. T.TanakaJ. S.WellensT. R. (1992). The psychometrics of order effects. In SchwarzN.SudmanS. (Eds.), Context effects in social and psychological research (pp. 249–264). Springer. https://doi.org/10.1007/978-1-4612-2848-6_17
53.
PescosolidoB. A.Halpern-MannersA.LuoL.PerryB. (2021). Trends in public stigma of mental illness in the U.S., 1996–2018. Journal of the American Medical Association Network Open, 4(12), Article e2140202. https://doi.org/10.1001/jamanetworkopen.2021.40202
54.
RaykovT.DimitrovD. M.MarcoulidesG. A.HarrisonM. (2019). On the connections between item response theory and classical test theory: A note on true score evaluation for polytomous items via item response modeling. Educational and Psychological Measurement, 79(6), 1198–1209. https://doi.org/10.1177/0013164417745949
55.
RaykovT.DimitrovD. M.MarcoulidesG. A.LiT.MenoldN. (2018). Examining measurement invariance and differential item functioning with discrete latent construct indicators: A note on a multiple testing procedure. Educational and Psychological Measurement, 78(2), 343–352. https://doi.org/10.1177/0013164416670984
56.
RaykovT.HuberC.MarcoulidesG. A.PusicM.MenoldN. (2021). Evaluation of response probabilities along studied latent dimensions: A polytomous item extension. Measurement: Interdisciplinary Research and Perspectives, 19(3), 179–185. https://doi.org/10.1080/15366367.2021.1923318
ReiseS. P.MooreT. M.HavilandM. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477
59.
ReivichK. J.SeligmanM. E. P.McBrideS. (2011). Master resilience training in the U.S. Army. American Psychologist, 66(1), 25–34. https://doi.org/10.1037/a0021897
60.
RhemtullaM.Brosseau-LiardP. É.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
61.
RobitzschA. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5, Article 589965. https://doi.org/10.3389/feduc.2020.589965
62.
RodriguezA.ReiseS. P.HavilandM. G. (2016). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237. https://doi.org/10.1080/00223891.2015.1089249
63.
SatorraA.BentlerP. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75(2), 243–248. https://doi.org/10.1007/s11336-009-9135-y
64.
ShamamboL. J.HenryT. L. (2022). Rethinking the use of the word “Caucasian” in clinical language and curricula: A trainee’s call to action. Journal of General Internal Medicine, 37(7), 1780–1782. https://doi.org/10.1007/s11606-022-07431-6
65.
SharpM.-L.FearN. T.RonaR. J.WesselyS.GreenbergN.JonesN.GoodwinL. (2015). Stigma as a barrier to seeking health care among military personnel with mental health problems. Epidemiologic Reviews, 37(1), 144–162. https://doi.org/10.1093/epirev/mxu012
66.
SimmsL. J.ZelaznyK.WilliamsT. F.BernsteinL. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557–566. https://doi.org/10.1037/pas0000648
67.
SkoppN. A.BushN. E.VogelD. L.WadeN. G.SirotinA. P.McCannR. A.Metzger-AbamukongM. J. (2012). Development and initial testing of a measure of public and self-stigma in the military. Journal of Clinical Psychology, 68(9), 1036–1047. https://doi.org/10.1002/jclp.21889
68.
StegerM. F.FrazierP.OishiS.KalerM. (2006). The meaning in life questionnaire: Assessing the presence of and search for meaning in life. Journal of Counseling Psychology, 53(1), 80–93. https://doi.org/10.1037/0022-0167.53.1.80
69.
TrautmannS.RehmJ.WittchenH. (2016). The economic costs of mental disorders: Do our societies react appropriately to the burden of mental disorders?EMBO Reports, 17(9), 1245–1249. https://doi.org/10.15252/embr.201642951
70.
United States Government Accountability Office. (2019). Behavioral health: Research on health care costs of untreated conditions in limited (Report to congressional requesters GAO-19-274, pp. 1–45). U.S. Government Accountability Office. https://www.gao.gov/assets/gao-19-274.pdf
71.
U.S. Department of Defense. (2022a). Annual report on suicide in the military: Calendar year 2022 (Annual report ARMS-CY22, pp. 1–60). Secretary of Defense for Personnel and Readiness.
Van De SchootR.LugtigP.HoxJ. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9(4), 486–492. https://doi.org/10.1080/17405629.2012.686740
74.
VidalesC. A.SmolenskiD. J.SkoppN. A.VogelD.WadeN.SheppardS.SpeedK.HoodK.CartwrightP. (2024). Assessing the dimensionality and construct validity of the military stigma scale across current service members. Military Psychology, 36(1), 49–57. https://doi.org/10.1080/08995605.2021.1997501
75.
VogelD. L.BitmanR. L.HammerJ. H.WadeN. G. (2013). Is stigma internalized? The longitudinal impact of public stigma on self-stigma. Journal of Counseling Psychology, 60(2), 311–316. https://doi.org/10.1037/a0031889
76.
WangJ.L.FiockG.AdairC.LaiD. (2007). Gender specific correlates of stigma toward depression in a Canadian general population sample. Journal of Affective Disorders, 103(1-3). 91–97. https://doi.org/10.1016/j.jad.2007.01.010
YuB. C. L.ChioF. H. N.ChanK. K. Y.MakW. W. S.ZhangG.VogelD.LaiM. H. C. (2023). Associations between public and self-stigma of help-seeking with help-seeking attitudes and intention: A meta-analytic structural equation modeling approach. Journal of Counseling Psychology, 73, 90–103. https://doi.org/10.1037/cou0000637
79.
ZinbargR. E.YovelI.RevelleW.McDonaldR. P. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ωh. Applied Psychological Measurement, 30(2), 121–144. https://doi.org/10.1177/0146621605278814
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
