In psychiatry, severity of mental health conditions and their change over time are usually measured via sum scores of items on psychometric scales. However, inferences from such scores can be biased if psychometric properties such as unidimensionality and temporal measurement invariance for instruments are not met. Here, we aimed to evaluate these properties for common measures of depression (Patient Health Questionnaire–9) and anxiety (Generalized Anxiety Disorder Assessment–7) in a large clinical sample (N = 22,362) undergoing psychotherapy. In addition, we tested consistency in dimensionality results across different methods (parallel analysis, factor analysis, explained common variance, the partial credit model, and the Mokken model). Results showed that while both Patient Health Questionnaire–9 and Generalized Anxiety Disorder Assessment–7 are multidimensional instruments with highly correlated factors, there is justification for sum scores as measures of severity. Temporal measurement invariance across 10 therapy sessions was evaluated. Strict temporal measurement invariance was established in both scales, allowing researchers to compare sum scores as severity measures across time.
BagbyR. M.RyderA. G.SchullerD. R.MarshallM. B. (2004). The Hamilton Depression Rating Scale: Has the gold standard become a lead weight?American Journal of Psychiatry, 161(12), 2163-2177. https://doi.org/10.1176/appi.ajp.161.12.2163
2.
BartoloA.MonteiroS.PereiraA. (2017). Factor structure and construct validity of the Generalized Anxiety Disorder 7-item (GAD-7) among Portuguese college students. Cadernos de Saúde Pública, 33(9), Article e00212716. https://doi.org/10.1590/0102-311X00212716
3.
BeardC.BjorgvinssonT. (2014). Beyond generalized anxiety disorder: Psychometric properties of the GAD-7 in a heterogeneous psychiatric sample. Journal of Anxiety Disorders, 28(6), 547-552. https://doi.org/10.1016/j.janxdis.2014.06.002
BentlerP. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137-143. https://doi.org/10.1007/s11336-008-9100-1
6.
CameronI. M.CrawfordJ. R.LawtonK.ReidI. C. (2008). Psychometric comparison of PHQ-9 and HADS for measuring depression severity in primary care. British Journal of General Practice, 58(546), 32-36. https://doi.org/10.3399/bjgp08X263794
7.
ClarkD. M. (2018). Realizing the mass public benefit of evidence-based psychological therapies: The IAPT program. Annual Review of Clinical Psychology, 14, 159-183. https://doi.org/10.1146/annurev-clinpsy-050817-084833
8.
DowneyL.HaydukL. A.CurtisJ. R.EngelbergR. A. (2016). Measuring depression-severity in critically ill patients’ families with the Patient Health Questionnaire (PHQ): Tests for unidimensionality and longitudinal measurement invariance, with implications for CONSORT. Journal of Pain Symptom Management, 51(5), 938-946. https://doi.org/10.1016/j.jpainsymman.2015.12.303
9.
ElhaiJ. D.ContractorA. A.TamburrinoM.FineT. H.PrescottM. R.ShirleyE.ChanP. K.SlembarskiR.LiberzonI.GaleaS.CalabreseJ. R. (2012). The factor structure of major depression symptoms: A test of four competing models using the Patient Health Questionnaire-9. Psychiatry Research, 199(3), 169-173. https://doi.org/10.1016/j.psychres.2012.05.018
10.
FriedE. I. (2017). Moving forward: How depression heterogeneity hinders progress in treatment and research. Expert Review of Neurotherapeutics, 17(5), 423-425. https://doi.org/10.1080/14737175.2017.1307737
11.
FriedE. I.van BorkuloC. D.EpskampS.SchoeversR. A.TuerlinckxF.BorsboomD. (2016). Measuring depression over time . . . Or not? Lack of unidimensionality and longitudinal measurement invariance in four common rating scales of depression. Psychological Assessment, 28(11), 1354-1367. https://doi.org/10.1037/pas0000275
12.
GillespieM.TenvergertE. M.KingmaJ. (1987). Using Mokken scale analysis to develop unidimensional scales. Quality and Quantity, 21(4), 393-408. https://doi.org/10.1007/bf00172565
13.
GlasC. A. W.VerhelstN. D. (1995). Testing the Rasch model. In FischerG.MolenaarI. W. (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 69-95). Springer.
14.
Gonzalez-BlanchC.MedranoL. A.Munoz-NavarroR.Ruiz-RodriguezP.MorianaJ. A.LimoneroJ. T.SchmitzF.Cano-VindelA., & PsicAP Research Group. (2018). Factor structure and measurement invariance across various demographic groups and over time for the PHQ-9 in primary care patients in Spain. PLOS ONE, 13(2), Article e0193356. https://doi.org/10.1371/journal.pone.0193356
15.
GuoB.Kaylor-HughesC.GarlandA.NixonN.SweeneyT.SimpsonS.DalgleisheT.RamanaR.YangM.MorrissR. (2017). Factor structure and longitudinal measurement invariance of PHQ-9 for specialist mental health care patients with persistent major depressive disorder: Exploratory structural equation modelling. Journal of Affective Disorders, 219(September), 1-8. https://doi.org/10.1016/j.jad.2017.05.020
16.
HamiltonM. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry, 23(1), 56-62. https://doi.org/10.1136/jnnp.23.1.56
HeeneM.KyngdonA.SckopkeP. (2016). Detecting violations of unidimensionality by order-restricted inference methods. Frontiers in Applied Mathematics and Statistics, 2(3). https://doi.org/10.3389/fams.2016.00003
19.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. https://doi.org/10.1007/bf02289447
20.
HuL.BentlerM. P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
21.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233-255. https://doi.org/10.1207/S15328007SEM0902_5
22.
ChilcotJ.RaynerL.LeeW.PriceA.GoodwinL.MonroeB.SykesN.HansfordP.HotopfM. (2013). The factor structure of the PHQ-9 in palliative care. Journal of Psychosomatic Research, 75(1), 60-64. https://doi.org/10.1016/j.jpsychores.2012.12.012
23.
JohnsonS. U.UlvenesP. G.ØktedalenT.HoffartA. (2019). Psychometric properties of the General Anxiety Disorder 7-Item (GAD-7) Scale in a heterogeneous psychiatric sample. Frontiers in Psychology, 10, Article 1713. https://doi.org/10.3389/fpsyg.2019.01713
24.
KapurS.PhillipsA. G.InselT. R. (2012). Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it?Molecular Psychiatry, 17(12), 1174-1179. https://doi.org/10.1038/mp.2012.105
25.
KertzS.Bigda-PeytonJ.BjorgvinssonT. (2013). Validity of the Generalized Anxiety Disorder-7 scale in an acute psychiatric sample. Clinical Psychology & Psychotherapy, 20(5), 456-464. https://doi.org/10.1002/cpp.1802
26.
KeumB. T.MillerM. J.InkelasK. K. (2018). Testing the factor structure and measurement invariance of the PHQ-9 across racially diverse U.S. college students. Psychological Assessment, 30(8), 1096-1106. https://doi.org/10.1037/pas0000550
27.
KnightC.RussoD.StochlJ.CroudaceT.FowlerD.GreyN.ReeveN.JonesP. B.PerezJ. (2020). Prevalence of and recovery from common mental disorder including psychotic experiences in the UK primary care Improving Access to Psychological Therapies (IAPT) Programme. Journal of Affective Disorders, 272(July), 84-90. https://doi.org/10.1016/j.jad.2020.04.015
28.
KrauseJ. S.ReedK. S.McArdleJ. J. (2010). Factor structure and predictive validity of somatic and nonsomatic symptoms from the Patient Health Questionnaire-9: A longitudinal study after spinal cord injury. Archives of Physical Medicine and Rehabilitation, 91(8), 1218-1224. https://doi.org/10.1016/j.apmr.2010.04.015
29.
KroenkeK.SpitzerR. L.WilliamsJ. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606-613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x
30.
LaiK.GreenS. B. (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree. Multivariate Behavioral Research, 51(2-3), 220-239. https://doi.org/10.1080/00273171.2015.1134306
31.
LeeB.KimY. E. (2019). The psychometric properties of the Generalized Anxiety Disorder scale (GAD-7) among Korean university students. Psychiatry and Clinical Psychopharmacology, 29(4), 864-871. https://doi.org/10.1080/24750573.2019.1691320
32.
LevisB.BenedettiA.ThombsB. D. (2019). Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: Individual participant data meta-analysis. British Medical Journal, 365, Article l1476. https://doi.org/10.1136/bmj.l1476
33.
LinacreJ. M. (2002). What do infit and outfit, mean-square and standardized mean?Rasch Measurement Transactions, 16(2), 878. https://www.rasch.org/rmt/rmt162f.htm
34.
LoevingerJ. (1947). A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 61(4), 1-49. https://doi.org/10.1037/h0093565
35.
LoweB.DeckerO.MullerS.BrahlerE.SchellbergD.HerzogW.HerzbergP. Y. (2008). Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population. Medical Care, 46(3), 266-274. https://doi.org/10.1097/MLR.0b013e318160d093
36.
MairP.HatzingerR. (2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20(9), 1-20. https://doi.org/10.18637/jss.v020.i09
37.
ManeaL.GilbodyS.McMillanD. (2012). Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): A meta-analysis. CMAJ, 184(3), E191-196. https://doi.org/10.1503/cmaj.110829
38.
MaroufizadehS.Omani-SamaniR.Almasi-HashianiA.AminiP.SepidarkishM. (2019). The reliability and validity of the Patient Health Questionnaire-9 (PHQ-9) and PHQ-2 in patients with infertility. Reproductive Health, 16(1), 137-137. https://doi.org/10.1186/s12978-019-0802-x
MastersG. N.WrightB. D. (1997). The partial credit model. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 101-121). Springer.
41.
McDonaldR. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
MewtonL.HobbsM. J.SunderlandM.NewbyJ.AndrewsG. (2014). Reductions in the internalising construct following internet-delivered treatment for anxiety and depression in primary care. Behaviour Research and Therapy, 63(December), 132-138. https://doi.org/10.1016/j.brat.2014.10.001
44.
MillsapR. E. (2011). Statistical approaches to measurement invariance. Taylor & Francis Group.
45.
MokkenR. J. (1971). A theory and procedure of scale analysis. Mouton.
Naragon-GaineyK.GallagherM. W.BrownT. A. (2014). A longitudinal examination of psychosocial impairment across the anxiety disorders. Psychological Medicine, 44(8), 1691-1700. https://doi.org/10.1017/S0033291713001967
PrataD.MechelliA.KapurS. (2014). Clinically meaningful biomarkers for psychosis: A systematic and quantitative review. Neuroscience & Biobehavioral Reviews, 45(September), 134-141. https://doi.org/10.1016/j.neubiorev.2014.05.010
52.
ReiseS. P.MooreT. M.HavilandM. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544-559. https://doi.org/10.1080/00223891.2010.496477
RichardsonE. J.RichardsJ. S. (2008). Factor structure of the PHQ-9 screen for depression across time since injury among persons with spinal cord injury. Rehabilitation Psychology, 53(2), 243-249. https://doi.org/10.1037/0090-5550.53.2.243
55.
ShaferA. B. (2006). Meta-analysis of the factor structures of four depression questionnaires: Beck, CES-D, Hamilton, and Zung. Journal of Clinical Psychology, 62(1), 123-146. https://doi.org/10.1002/jclp.20213
56.
SijtsmaK.MolenaarI. W. (2002). Introduction to nonparametric item response theory (Vol. 5). Sage.
57.
SmithG. T.McCarthyD. M.ZapolskiT. C. (2009). On the value of homogeneous constructs for construct validation, theory testing, and the description of psychopathology. Psychological Assessment, 21(3), 272-284. https://doi.org/10.1037/a0016699
58.
SousaT. V.ViveirosV.ChaiM. V.VicenteF. L.JesusG.CarnotM. J.GordaA. C.FerreiraP. L. (2015). Reliability and validity of the Portuguese version of the Generalized Anxiety Disorder (GAD-7) scale. Health and Quality of Life Outcomes, 13(1), Article 50. https://doi.org/10.1186/s12955-015-0244-2
59.
SpitzerR. L.KroenkeK.WilliamsJ. B. W.LoweB. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092-1097. https://doi.org/10.1001/archinte.166.10.1092
60.
StochlJ.JonesP. B.CroudaceT. J. (2012). Mokken scale analysis of mental health and well-being questionnaire item responses: A non-parametric IRT method in empirical research for applied health researchers. BMC Medical Research Methodology, 12(1), Article 74. https://doi.org/10.1186/1471-2288-12-74
61.
Ten BergeJ. M. F.SočanG. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613-625. https://doi.org/10.1007/bf02289858
62.
TitovN.DearB. F.McMillanD.AndersonT.ZouJ.SunderlandM. (2011). Psychometric comparison of the PHQ-9 and BDI-II for measuring response during treatment of depression. Cognitive Behaviour Therapy, 40(2), 126-136. https://doi.org/10.1080/16506073.2010.550059
63.
van der ArkL. A. (2012). New developments in Mokken Scale analysis in R. Journal of Statistical Software, 48(5). https://doi.org/10.18637/jss.v048.i05
64.
VenkatasubramanianG.KeshavanM. S. (2016). Biomarkers in psychiatry: A critique. Annals of Neurosciences, 23(1), 3-5. https://doi.org/10.1159/000443549
65.
WrightB. D.LinacreJ. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370. https://www.rasch.org/rmt/rmt83b.htm
66.
WrightB. D.MastersG. N. (1990). Computation of OUTFIT and INFIT statistics. Rasch Measurement Transactions, 3(4), 84-85. https://www.rasch.org/rmt/rmt34e.htm
67.
ZwitserR. J.MarisG. (2016). Ordering individuals with sum scores: The introduction of the nonparametric rasch model. Psychometrika, 81(1), 39-59. doi:10.1007/s11336-015-9481-x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
