Beyond Mono-Method Symptom Assessment: Multimethod Extension of the Trilevel Model of Anxiety and Depression

Abstract

The wide array of symptoms of and high comorbidity rates between unipolar depressive and anxiety-related disorders have raised questions about the relations among their symptoms. Factor analysis examines these relations, yet factor-analytic symptom-measurement models have relied on a single method of measurement and do not always replicate. We conceptually replicate and extend the trilevel model of anxiety and depression symptoms to encompass interviewer-rated symptoms. In the trilevel model, symptom-specific items load on three levels of factors: a narrow (or disorder-specific) factor, an intermediate-breadth factor, and a general-distress factor. The trilevel model fit well in this sample and fit better than comparison models that eliminated one of the three levels. Extension analysis successfully integrated interviewer-rating variables into the trilevel model—particularly for the broad and narrow levels. These results provide some support for the trilevel model and the use of interviewer ratings of symptoms. Research and treatment implications are discussed.

Keywords

emotional disorders factor analysis extension analysis dimensional psychopathology interviewer-rated multimethod open data preregistration

Get full access to this article

View all access options for this article.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).

Anvari

Lakens

(2018). The replicability crisis and public trust in psychological science. Comprehensive Results in Social Psychology, 3(3), 266–286. https://doi.org/10.1080/23743603.2019.1684822

Bollen

K. A.

(1996). An alternative two stage least squares (2SLS) estimator for latent variable equations. Psychometrika, 61(1), 109–121.

Bonifay

Cai

(2017). On the complexity of item response theory models. Multivariate Behavioral Research, 52(4), 465–484.

Bonifay

Lane

Reise

(2016). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5, 184–186.

Brown

T. A.

(2003). Confirmatory factor analysis of the Penn State Worry Questionnaire: Multiple factors or method effects? Behaviour Research and Therapy, 41, 1411–1426. https://doi.org/10.1016/S0005-7967(03)00059-7

Brown

T. A.

Antony

M. M.

Barlow

D. H.

(1992). Psychometric properties of the Penn State Worry Questionnaire in a clinical anxiety-related disorders sample. Behaviour Research and Therapy, 30, 33–37.

Brown

T. A.

Campbell

L. A.

Lehman

C. L.

Grisham

J. R.

Mancill

R. B.

(2001). Current and lifetime comorbidity of the DSM-IV anxiety and mood disorders in a large clinical sample. Journal of Abnormal Psychology, 110(4), 585–599. https://doi.org/10.1037/0021-843X.110.4.585

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.

10.

Carver

C. S.

White

T. L.

(1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: The BIS/BAS scales. Journal of Personality and Social Psychology, 67(2), 319–333.

11.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504.

12.

Chen

F. F.

West

S. G.

Sousa

K. H.

(2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189–225.

13.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

14.

Coles

N. A.

March

D. S.

Marmolejo-Ramos

Larsen

J. T.

Arinze

N. C.

Ndukaihe

I. L.

Willis

M. L.

Foroni

Reggev

Mokady

Forscher

P. S.

Hunter

J. F.

Kaminski

Yüvrük

Kapucu

Nagy

Hajdu

Tejada

Freitag

R. M. K.

. . . Liuzza

M. T.

(2022). A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration. Nature Human Behaviour, 6(12), 1731–1742. https://doi.org/10.1038/s41562-022-01458-9

15.

Craske

M. G.

Brown

T. A.

Barlow

D. H.

(1991). Behavioral treatment of panic disorder: A two-year follow-up. Behavior Therapy, 22, 289–304.

16.

Di Nardo

P. A.

Barlow

D. H

. (1988). Anxiety-related disorders interview schedule-revised (ADIS-R). Phobia and Anxiety-Related Disorders Clinic, State University of New York.

17.

Doty

D. H.

Glick

W. H.

(1998). Common methods bias: Does common methods variance really bias results? Organizational Research Methods, 1(4), 374–406.

18.

Earp

B. D.

Trafimow

(2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, Article 138286. https://doi.org/10.3389/fpsyg.2015.00621

19.

Ebersole

C. R.

Atherton

O. E.

Belanger

A. L.

Skulborstad

H. M.

Allen

J. M.

Banks

J. B.

Baranski

Bernstein

M. J.

Bonfiglio

D. B. V.

Boucher

Brown

E. R.

Budiman

N. I.

Cairo

A. H.

Capaldi

C. A.

Chartier

C. R.

Chung

J. M.

Cicero

D. C.

Coleman

J. A.

Conway

J. G.

. . . Nosek

B. A.

(2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012

20.

Eid

Diener

(2006). Introduction: The need for multimethod measurement in psychology. In Eid

Diener

(Eds.), Handbook of multimethod measurement in psychology (pp. 3–8). American Psychological Association. https://doi.org/10.1037/11383-001

21.

Eid

Lischetzke

Nussbeck

F. W.

Trierweiler

L. I.

(2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8(1), 38–60. https://doi.org/10.1037/1082-989X.8.1.38

22.

First

M. B.

Williams

J. B. W.

Karg

R. S.

Spitzer

R. L.

(2014). Structured clinical interview for DSM-5 disorders–research version (SCID-5-RV). American Psychiatric Association.

23.

Foa

E. B.

Huppert

J. D.

Leiberg

Langner

Kichic

Hajcak

Salkovskis

P. M.

(2002). The Obsessive-Compulsive Inventory: Development and validation of a short version. Psychological Assessment, 14, 485–496. https://doi.org/10.1037/1040-3590.14.4.485

24.

Geer

J. H.

(1965). The development of a scale to measure fear. Behaviour Research and Therapy, 3(1), 45–53.

25.

Giner-Sorolla

(2019). From crisis of evidence to a “crisis” of relevance? Incentive-based answers for social psychology’s perennial relevance worries. European Review of Social Psychology, 30(1), 1–38. https://doi.org/10.1080/10463283.2018.1542902

26.

Gorsuch

R. L.

(1997). New procedures of extension analysis in exploratory factor analysis. Educational and Psychological Measurement, 57, 725–740.

27.

Hoshino

Bentler

P. M.

(2011). Bias in factor score regression and a simple solution (UCLA Statistics Preprint No. 621). https://escholarship.org/content/qt45h3t3t2/qt45h3t3t2_noSplash_e22030f9de09f0043a5f09a2a328acad.pdf?t=lrh0g4

28.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

29.

Huppert

J. D.

Walther

M. R.

Hajcak

Yadin

Foa

E. B.

Simpson

H. B.

Liebowitz

M. R.

(2007). The OCI-R: Validation of the subscales in a clinical sample. Journal of Anxiety-Related Disorders, 21, 394–406. https://doi.org/10.1016/j.janxdis.2006.05.006

30.

Ioannidis

J. P. A.

(2005). Why most published research findingsare false. PLOS Medicine, 2, Article e124. https://doi.org/10.1371/journal.pmed.0020124

31.

Kelley

N. J.

Kramer

A. M.

Young

K. S.

Echiverri-Cohen

A. M.

Chat

I. K. Y.

Bookheimer

S. Y.

Nusslock

Craske

M. G.

Zinbarg

R. E.

(2019). Evidence for a general factor of behavioral activation system sensitivity. Journal of Research in Personality, 79, 30–39.

32.

Klein

R. A.

Cook

C. L.

Ebersole

C. R.

Vitiello

Nosek

B. A.

Hilgard

Ahn

P. H.

Brady

A. J.

Chartier

C. R.

Christopherson

C. D.

Clay

Collisson

Crawford

J. T.

Cromar

Gardiner

Gosnell

C. L.

Grahe

Hall

Howard

. . . Ratliff

K. A.

(2022). Many Labs 4: Failure to replicate mortality salience effect with and without original author involvement. Collabra: Psychology, 8(1), Article 35271. https://doi.org/10.1525/collabra.35271

33.

Klein

R. A.

Ratliff

Vianello

Adams

R. B.

Jr. Bahník

Bernstein

M. J.

Brandt

M. J.

Ijzerman

Bocian

Brooks

Brumbaugh

C. C.

Cemalcilar

Chandler

Cheong

Davis

W. E.

Devos

Eisner

Frankowska

Furrow

. . . Nosek

B. A.

(2014). Data from investigating variation in replicability: A “many labs” replication project. Journal of Open Psychology Data, 2(1), Article e4. https://doi.org/10.5334/jopd.ad

34.

Klein

R. A.

Vianello

Hasselman

Adams

B. G.

Adams

R. B.

Jr. Alper

Aveyard

Axt

J. R.

Babalola

M. T.

Bahník

Š.

Batra

Berkics

Bernstein

M. J.

Berry

D. R.

Bialobrzeska

Binan

E. D.

Bocian

Brandt

M. J.

Busching

. . . Sowden

(2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490.

35.

Kotov

Krueger

R. F.

Watson

Achenbach

T. M.

Althoff

R. R.

Bagby

R. M.

Brown

T. A.

Carpenter

W. T.

Caspi

Clark

L. A.

Eaton

N. R.

Forbes

M. K.

Forbush

K. T.

Goldberg

Hasin

Hyman

S. E.

Ivanova

M. Y.

Lynam

D. R.

Markon

. . .Zimmerman

(2017). The Hierarchical Taxonomy of Psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of Abnormal Psychology, 126(4), 454–477.

36.

Krueger

R. F.

(1999). The structure of common mental disorders. Archives of General Psychiatry, 56(10), 921–926.

37.

Marsh

H. W.

Hau

K.-T.

Wen

(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11(3), 320–341.

38.

Mattick

R. P.

Clarke

J. C.

(1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36(4), 455–470.

39.

McDonald

R. P.

(1978). A simple comprehensive model for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 3, 59–71.

40.

Meyer

T. J.

Miller

M. L.

Metzger

R. L.

Borkovec

T. D.

(1990). Development and validation of the Penn State Worry Questionnaire. Behaviour Research and Therapy, 28(6), 487–495.

41.

Mineka

Watson

Clark

L. A.

(1998). Comorbidity of anxiety and unipolar mood disorders. Annual Review of Psychology, 49, 377–412.

42.

Mosier

(1938). A note on Dwyer: The determination of the factor loadings of a given test. Psychometrika, 3, 297–299.

43.

Murray

A. L.

Johnson

(2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41(5), 407–422.

44.

Muthén

L. K.

(2008). Temporal invariance [Msg 6]. http://www.statmodel.com/discussion/messages/9/3012.html?1207672749

45.

Muthén

L. K.

Muthén

B. O.

(1998–2018). Mplus user’s guide (8th ed.). Muthén & Muthén.

46.

Naragon-Gainey

Prenoveau

J. M.

Brown

T. A.

Zinbarg

R. E.

(2016). A comparison and integration of structural models of depression and anxiety in a clinical sample: Support for and validation of the tri-level model. Journal of Abnormal Psychology, 125(7), 853–867. https://doi.org/10.1037/abn0000197

47.

Pashler

Wagenmakers

E.-J.

(2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.

48.

Prenoveau

J. M.

Zinbarg

R. E.

Craske

M. G.

Mineka

Griffith

J. W.

Epstein

A. M.

(2010). Testing a hierarchical model of anxiety and depression in adolescents: A tri-level model. Journal of Anxiety-Related Disorders, 24, 334–344. https://doi.org/10.1016/j.janxdis.2010.01.006

49.

Putnick

D. L.

Bornstein

M. H.

(2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90.

50.

Rapee

R. M.

Craske

M. G.

Barlow

D. H.

(1994). Assessment instrument for panic disorder that includes fear of sensation-producing activities: The Albany Panic and Phobia Questionnaire. Anxiety, 1(3), 114–122.

51.

Reise

S. P.

Kim

D. S.

Mansolf

Widaman

K. F.

(2016). Is the bifactor model a better model or is it just better at modeling implausible responses? Application of iteratively reweighted least squares to the Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51(6), 818–838.

52.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton-Mifflin.

53.

Snyder

Young

Hankin

(2017). Strong homotypic continuity in common psychopathology, internalizing and externalizing factors over time in adolescents. Clinical Psychological Science, 5(1), 98–110. https://doi.org/10.1177/2167702616651076

54.

Tackett

J. L.

Brandes

C. M.

King

Markon

(2018). Psychology’s replication crisis and clinical psychological science. Annual Review of Clinical Psychology, 15, 579–604. https://doi.org/10.1146/annurev-clinpsy-050718-095710

55.

Tackett

J. L.

Lilienfeld

S. O.

Patrick

C. J.

Johnson

S. L.

Krueger

R. F.

Miller

J. D.

Oltmanns

T. F.

Shrout

P. E.

(2017). It’s time to broaden the replicability conversation: Thoughts for and from clinical psychological science. Perspectives on Psychological Science, 12, 742–756. https://doi.org/10.1177/1745691617690042

56.

Vermani

Marcus

Katzman

M. A.

(2011). Rates of detection of mood and anxiety-related disorders in primary care: A descriptive, cross-sectional study. The Primary Care Companion to CNS Disorders, 13(2), Article PCC.10m01013. https://doi.org/10.4088/PCC.10m01013

57.

Watson

(2005). Rethinking the mood and anxiety-related disorders: A quantitative hierarchical model for DSM-V. Journal of Abnormal Psychology, 114(5), 522–536. https://doi.org/10.1037/0021-843X.114.4.522

58.

Watson

Clark

L. A.

Weber

Assenheimer

J. S.

Strauss

M. E.

McCormick

R. A.

(1995). Testing a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult and patient samples. Journal of Abnormal Psychology, 104(1), 15–25.

59.

Watts

A. L.

Makol

B. A.

Palumbo

I. M.

De Los Reyes

Olino

T. M.

Latzman

R. D.

DeYoung

C. G.

Wood

P. K.

Sher

K. J.

(2022). How robust is the p factor? Using multitrait-multimethod modeling to inform the meaning of general factors of youth psychopathology. Clinical Psychological Science, 10(4), 640–661. https://doi.org/10.1177/21677026211055170

60.

Yuan

K. H.

Chan

(2002). Fitting structural equation models using estimating equations: A model segregation approach. British Journal of Mathematical and Statistical Psychology, 55(1), 41–62.

61.

Zimmerman

Coryell

(1987). The Inventory to Diagnose Depression (IDD): A self-report scale to diagnose major depressive disorder. Journal of Consulting and Clinical Psychology, 55(1), 55–59.

62.

Zinbarg

R. E.

Barlow

D. H.

(1996). The structure of anxiety and the DSM-III-R anxiety-related disorders: A hierarchical model. Journal of Abnormal Psychology, 105, 181–193.

63.

Zinbarg

R. E.

Craske

Mineka

Griffith

Sutton

Rose

R. D.

Nazarian

Mor

Waters

A. M.

(2010). The Northwestern-UCLA Youth Emotion Project: Associations of cognitive vulnerabilities, neuroticism and gender with past diagnoses of emotional disorders in adolescents. Behaviour Research and Therapy, 48, 347–358.

64.

Zinbarg

R. E.

Schmidt

Feinstein

Williams

Murillo

Echiverri-Cohen

Enders

Craske

Nusslock

(2023). Personality predicts pre-COVID-19 trajectories of transdiagnostic anxiety and depression symptoms. Journal of Psychopathology and Clinical Science, 132(6), 645–656. https://doi.org/10.1037/abn0000803

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB