Highly efficient assessments that better account for comorbidity between mood and anxiety disorders (internalizing) are required to identify individuals who are most at risk of psychopathology in the community. The current study examined the efficiency and validity associated with a multidimensional computerized adaptive test (CAT) to measure broad and specific levels of internalizing psychopathology. The sample comprised 3,175 respondents to an online survey. Items from five banks (generalized anxiety, depression, obsessive–compulsive disorder, panic disorder, social anxiety disorder) were jointly calibrated using a bifactor item response theory model. Simulations indicated that an adaptive algorithm could accurately (rs ≥ 0.90) estimate general internalizing and specific disorder scores using on average 44 items in comparison with the full 133-item bank (67% reduction in items). Scores on the CAT demonstrate convergent and divergent validity with previously validated short severity scales and could significantly differentiate cases of DSM-5 disorder. As such, the CAT validly measures both broad and specific constructs of internalizing disorders in a manner similar to the full item bank and a static brief form but with greater gains in efficiency and, therefore, a reduced degree of respondent burden.
AchenbachT. M. (1966). The classification of children’s psychiatric symptoms: A factor-analytic study. Psychological Monographs: General and Applied, 80(7), 1-37. doi:10.1037/h0093906
2.
AndrewsG.AndersonT. M. M.SladeT.SunderlandM. (2008). Classification of anxiety and depressive disorders: Problems and solutions. Depression and Anxiety, 25, 274-281. doi:10.1002/da.20489
3.
BatterhamP. J.MackinnonA. J.ChristensenH. (2015). The Panic Disorder Screener (PADIS): Development of an accurate and brief population screening tool. Psychiatry Research, 228, 72-76. doi:10.1016/j.psychres.2015.04.016
4.
BatterhamP. J.MackinnonA. J.ChristensenH. (2016). Community-based validation of the Social Phobia Screener (SOPHS). Assessment. Advance online publication. doi:10.1177/1073191116636448
5.
BatterhamP. J.SunderlandM.CarragherN.CalearA. L. (2016). Development and community-based validation of eight item banks to assess mental health. Psychiatry Research, 243, 452-463. doi:10.1016/j.psychres.2016.07.011
6.
BellochA.MorilloC.García-SorianoG. (2007). Obsessive themes, evaluative appraisals and thought control strategies: Testing the autogenous-reactive model of obsessions. International Journal of Clinical and Health Psychology, 7, 5-20. doi:10.1037/0021-843X.110.4.585
7.
BjornerJ. B.ChangC.-H.ThissenD.ReeveB. B. (2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16(Suppl. 1), 95-108. doi:10.1007/s11136-007-9168-6
8.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459. doi:10.1007/BF02293801
9.
BrownT. A.BarlowD. H. (2009). A proposal for a dimensional classification system based on the shared features of the DSM-IV anxiety and mood disorders: Implications for assessment and treatment. Psychological Assessment, 21, 256-271. doi:10.1037/a0016608
10.
BurnhamK. P.AndersonD. R. (2004). Multimodal inference: Understanding AIC and BIC in model selection. Sociological Methods and Research, 33, 261-304.
11.
CarragherN.KruegerR. F.EatonN. R.SladeT. (2015, March4). Disorders without borders: Current and future directions in the meta-structure of mental disorders. Social Psychiatry and Psychiatric Epidemiology, 50, 339-350. doi:10.1007/s00127-014-1004-z
12.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. doi:10.18637/jss.v048.i06
13.
ChalmersR. P. (2016). Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications. Journal of Statistical Software, 71(5), 1-38.
14.
ChoiS. W.ReiseS. P.PilkonisP. A.HaysR. D.CellaD. (2010). Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research, 19, 125-136. doi:10.1007/s11136-009-9560-5
15.
EatonN. R.KruegerR. F.MarkonK. E.KeyesK. M.SkodolA. E.WallM.. . . GrantB. F. (2013). The structure and predictive validity of the internalizing disorders. Journal of Abnormal Psychology, 122, 86-92.
16.
EatonN. R.Rodriguez-SeijasC.CarragherN.KruegerR. F. (2015, February7). Transdiagnostic factors of psychopathology and substance use disorders: A review. Social Psychiatry and Psychiatric Epidemiology, 50, 171-182. doi:10.1007/s00127-014-1001-2
17.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
18.
FirstM. B. (2005). Clinical utility: A prerequisite for the adoption of a dimensional approach in DSM. Journal of Abnormal Psychology, 114, 560-564.
19.
FliegeH.BeckerJ.WalterO. B.BjornerJ. B.KlappB. F.RoseM. (2005). Development of a computer-adaptive test for depression (D-CAT). Quality of Life Research, 14, 2277-2291. doi:10.1007/s11136-005-6651-9
20.
Fusar-PoliP.NelsonB.ValmaggiaL.YungA. R.McGuireP. K. (2014). Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: Impact on psychopathology and transition to psychosis. Schizophrenia Bulletin, 40, 120-131. doi:10.1093/schbul/sbs136
21.
GardnerW. (2014). Screening for mental health problems: Does it work?Journal of Adolescent Health, 55, 1-2. doi:10.1016/j.jadohealth.2014.04.019
22.
GardnerW.KelleherK. J.PajerK. A. (2002). Multidimensional adaptive testing for mental health problems in primary care. Medical Care, 40, 812-823.
23.
GardnerW.ShearK.KelleherK. J.PajerK. AMammenO.BuysseD.FrankE. (2004). Computerized adaptive measurement of depression: A simulation study. BMC Psychiatry, 4, 13. doi:10.1186/1471-244X-4-13
GibbonsR.WeissD. J.FrankE.KupferD. (2016). Computerized adaptive diagnosis and testing of mental health disorders. Annual Review of Clinical Psychology, 12, 83-104. doi:10.1146/annurev-clinpsy-021815-093634
27.
GibbonsR.WeissD. J.KupferD. J.FrankE.FagioliniA.GrochocinskiV. J.. . . ImmekusJ. C. (2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368. doi:10.1176/appi.ps.59.4.361
28.
GibbonsR.WeissD. J.PilkonisP. A.FrankE.MooreT.KimJ. B.KupferD. J. (2012). The CAT-DI: Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104-1112. doi:10.1001/archgenpsychiatry.2012.14
29.
GibbonsR.WeissD.PilkonisP.FrankE.MooreT.KimJ.KupferD. (2014). Development of the CAT-ANX: A computerized adaptive test for anxiety. American Journal of Psychiatry, 171, 187-194. doi:10.1176/appi.ajp.2013.13020178
30.
HancockG. R.MuellerR. O. (2001). Rethinking construct reliability within latent variable systems. In CudeckR.du ToitS.SorbomD. (Eds.), Structural equation modeling: Present and future—A Festschrift in honor of Karl Joreskog (pp. 195-216). Lincolnwood, IL: Scientific Software.
31.
KesslerR. C.AngermeyerM.AnthonyJ. C.De GraafR.DemyttenaereK.GasquetI.. . . HaroJ. M. (2007). Lifetime prevalence and age-of-onset distributions of mental disorders in the World Health Organization’s World Mental Health Survey Initiative. World Psychiatry, 6, 168-176.
32.
KirisciL.TarterR.ReynoldsM.RidenourT.StoneC.VanyukovM. (2012). Computer adaptive testing of liability to addiction: Identifying individuals at risk. Drug and Alcohol Dependence, 123(Suppl. 1), S79-S86. doi:10.1016/j.drugalcdep.2012.01.016
33.
KroenkeK.SpitzerR. L.WilliamsJ. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606-613.
34.
KroenkeK.SpitzerR. L.WilliamsJ. B. W.MonahanP. O.LoweB. (2007). Anxiety disorder in primary care: Prevalence, impairment, comorbidity, and detection. Annals of Internal Medicine, 146, 317-325. Retrieved from http://www.annals.org/content/146/5/317.full.pdf
35.
KruegerR. F. (1999). The structure of common mental disorders. Archives of General Psychiatry, 56, 921-926.
36.
KruegerR. F.MarkonK. E. (2006). Reinterpreting comorbidity: A model-based approach to understanding and classifying psychopathology. Annual Reviews in Clinical Psychology, 2, 111-133.
37.
KruegerR. F.MarkonK. E.PatrickC. J.BenningS. D.KramerM. D. (2007). Linking antisocial behavior, substance use, and personality: An integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116, 645-666. doi:10.1037/0021-843X.116.4.645
38.
LöweB.DeckerO.MüllerS.BrählerE.SchellbergD.HerzogW.HerzbergP. Y. (2008). Validation and Standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the General Population. Medical Care, 46, 266-274. doi:10.1097/MLR.0b013e318160d093
39.
MarkonK. E. (2010). How things fall apart: Understanding the nature of internalizing through its relationship with impairment. Journal of Abnormal Psychology, 119, 447-458. doi:10.1037/a0019707
40.
MarkonK. E.ChmielewskiM.MillerC. J. (2011). The reliability and validity of discrete and continuous measures of psychopathology: A quantitative review. Psychological Bulletin, 137, 856-879.
41.
MartinA.RiefW.KlaibergA.BraehlerE. (2006). Validity of the Brief Patient Health Questionnaire Mood Scale (PHQ-9) in the general population. General Hospital Psychiatry, 28, 71-77. doi:10.1016/j.genhosppsych.2005.07.003
42.
Maydeu-OlivaresA. (2015). Evaluating the fit of IRT models. In ReiseS. P.RevickiD. A. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 111-127). New York, NY: Routledge.
43.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305-328. doi:10.1080/00273171.2014.911075
44.
McEvoyP. M.GroveR.SladeT. (2011). Epidemiology of anxiety disorder in the Australian general population: Findings of the 2007 Australian National Survey of Mental Health and Wellbeing. Australian and New Zealand Journal of Psychiatry, 45, 957-967.
45.
MerikangasK. R.AmesM.CuiL.StangP. E.UstunT. B.Von KorffM.KesslerR. C. (2007). The impact of comorbidity of mental and physical conditions on role disability in the US adult household population. Archives of General Psychiatry, 64, 1180-1188.
46.
MewtonL.HobbsM. J.SunderlandM.NewbyJ.AndrewsG. (2014). Reductions in the internalising construct following internet-delivered treatment for anxiety and depression in primary care. Behaviour Research and Therapy, 63, 132-138. doi:10.1016/j.brat.2014.10.001
47.
PilkonisP. AChoiS. W.ReiseS. P.StoverA. M.RileyW. T.CellaD. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): Depression, anxiety, and anger. Assessment, 18, 263-283. doi:10.1177/1073191111411667.
48.
ReckaseM. D. (2009). Multidimensional item response theory. New York, NY: Springer.
49.
ReeveB. B.HaysR. D.BjornerJ. B.CookK. F.CraneP. K.TeresiJ. A., . . . on behalf of the PROMISE Cooperative Group. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5), S22-S31. Retrieved from http://journals.lww.com/lww-medicalcare/Fulltext/2007/05001/Psychometric_Evaluation_and_Calibration_of.4.aspx
50.
ReiseS. P.HensonJ. M. (2000). Computerization and adaptive administration of the NEO PI-R. Assessment, 7, 347-364. doi:10.1177/107319110000700404
51.
RileyB. B.ConradK. J.DennisM. L. (2007). Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: The GAIN Substance Problem Scale. Journal of Applied Measurement, 8, 48-64.
52.
RodriguezA.ReiseS. P.HavilandM. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137-150. doi:10.1037/met0000045
53.
Rodriguez-SeijasC.StohlM.HasinD. S.EatonN. R. (2015). Transdiagnostic factors and mediation of the relationship between perceived racial discrimination and mental disorders. JAMA Psychiatry, 72, 706-713. doi:10.1001/jamapsychiatry.2015.0148
54.
ŞahinA.WeissD. J. (2015). Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing. doi:10.12738/estp.2015.6.0102
55.
SamejimaF. (1997). Graded response model. In Handbook of modern item response theory (pp. 85-100). New York, NY: Springer. doi:10.1007/978-1-4757-2691-6_5
56.
ScottK. M.BruffaertsR.TsangA.OrmelJ.AlonsoJ.AngermeyerM. C.. . . Von KorffM. (2007). Depression-anxiety relationships with chronic physical conditions: Results from the World Mental Health surveys. Journal of Affective Disorders, 103, 113-120. doi:10.1016/j.jad.2007.01.015
57.
SegallD. O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331-354. doi:10.1007/BF02294343
58.
SeoD. G.WeissD. J. (2015). Best design for multidimensional computerized adaptive testing with the bifactor model. Educational and Psychological Measurement, 75, 954-978. doi:10.1177/0013164415575147
59.
SheehanD. VLecrubierY.SheehanK. H.AmorimP.JanavsJ.WeillerE.. . . DunbarG. C. (1998). The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry, 59(Suppl. 20), 22-33.
60.
SladeT. (2007). The descriptive epidemiology of internalizing and externalizing psychiatric dimensions. Social Psychiatry and Psychiatric Epidemiology, 42, 554-560.
61.
SladeT.JohnstonA.Oakley BrowneM. A.AndrewsG.WhitefordH. (2009). 2007 National Survey of Mental Health and Wellbeing: Methods and key findings. Australian and New Zealand Journal of Psychiatry, 43, 594-605.
62.
SmitsN.CuijpersP.van StratenA. (2011). Applying computerized adaptive testing to the CES-D scale: A simulation study. Psychiatry Research, 188, 147-155. doi:10.1016/j.psychres.2010.12.001
63.
SouthS. C.KruegerR. F.IaconoW. G. (2011). Understanding general and specific connections between psychopathology and marital distress: A model based approach. Journal of Abnormal Psychology, 120, 935-947. doi:10.1037/a0025417
64.
SpitzerR. L.KroenkeK.WilliamsJ. B. W.LoweB. (2006). A brief measure for assessing generalized anxiety disorder: the GAD-7. Archives of Internal Medicine, 166, 1092-1097.
65.
StarrL. R.ConwayC. C.HammenC. L.BrennanP. A. (2014). Transdiagnostic and disorder-specific models of intergenerational transmission of internalizing pathology. Psychological Medicine, 44, 161-172. doi:10.1017/S003329171300055X
66.
StochlJ.BöhnkeJ. R.PickettK. E.CroudaceT. J. (2016). An evaluation of computerized adaptive testing for general psychological distress: Combining GHQ-12 and Affectometer-2 in an item bank for public mental health research. BMC Medical Research Methodology, 16, 58. doi:10.1186/s12874-016-0158-7
67.
SunderlandM.CarragherN.ChapmanC.MillsK.TeessonM.LockwoodE.. . . SladeT. (2016). The shared and specific relationships between exposure to potentially traumatic events and transdiagnostic dimensions of psychopathology. Journal of Anxiety Disorders, 38, 102-109. doi:10.1016/j.janxdis.2016.02.001
68.
SunderlandM.SladeT. (2015). The relationship between internalizing psychopathology and suicidality, treatment seeking, and disability in the Australian population. Journal of Affective Disorders, 171, 6-12. doi:10.1016/j.jad.2014.09.012
69.
SunderlandM.SladeT.KruegerR. F. (2015). Examining the shared and unique relationships among substance use and mental disorders. Psychological Medicine, 45, 1103-1113. doi:10.1017/S0033291714002219
70.
TeessonM.SladeT.MillsK. (2009). Comorbidity in Australia: Findings of the 2007 National Survey of Mental Health and Wellbeing. Australian and New Zealand Journal of Psychiatry, 43, 606-614. doi:10.1080/00048670902970908
71.
ThompsonN.WeissD. J. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research & Evaluation, 16(1). Retrieved from http://pareonline.net/pdf/v16n1.pdf
72.
TolandM. D.SulisI.GiambonaF.PorcuM.CampbellJ. M. (2016). Introduction to bifactor polytomous item response theory analysis. Journal of School Psychology, 60, 41-63. doi:10.1016/j.jsp.2016.11.001
73.
UherR.HeymanI.MortimoreC.FramptonI.GoodmanR. (2007). Screening young people for obsessive-compulsive disorder. British Journal of Psychiatry, 191, 353-354. doi:10.1192/bjp.bp.106.034967
74.
WalterO. B.BeckerJ.BjornerJ. B.FliegeH.KlappB. F.RoseM. (2007). Development and evaluation of a computer adaptive test for “Anxiety” (Anxiety-CAT). Quality of Life Research, 16, 143-155. doi:10.1007/s11136-007-9191-7
75.
WatsonD.O’HaraM. W.Naragon-GaineyK.KoffelE.ChmielewskiM.KotovR.. . . RuggeroC. J. (2012). Development and validation of new anxiety and bipolar symptom scales for an expanded version of the IDAS (the IDAS-II). Assessment, 19, 399-420. doi:10.1177/1073191112449857
76.
WhitefordH. A.DegenhardtL.RehmJ.BaxterA. J.FerrariA. J.ErskineH. E.. . . VosT. (2013). Global burden of disease attributable to mental and substance use disorders: Findings from the Global Burden of Disease Study 2010. Lancet, 382, 1575-1586.
77.
ZbozinekT. D.RoseR. D.Wolitzky-TaylorK. B.SherbourneC.SullivanG.SteinM. B.. . . CraskeM. G. (2012). Diagnostic overlap of generalized anxiety disorder and major depressive disorder in a primary care sample. Depression and Anxiety, 29, 1065-1071. doi:10.1002/da.22026
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
