Restricted accessResearch articleFirst published online 2021-5
Machine Learning to Classify Suicidal Thoughts and Behaviors: Implementation Within the Common Data Elements Used by the Military Suicide Research Consortium
Suicide rates among military-connected populations have increased over the past 15 years. Meta-analytic studies indicate prediction of suicide outcomes is lacking. Machine-learning approaches have been promoted to enhance classification models for suicide-related outcomes. In the present study, we compared the performance of three primary machine-learning approaches (i.e., elastic net, random forests, stacked ensembles) and a traditional statistical approach, generalized linear modeling (i.e., logistic regression), to classify suicide thoughts and behaviors using data from the Military Suicide Research Consortium’s Common Data Elements (CDE; n = 5,977–6,058 across outcomes). Models were informed by (a) selected items from the CDE or (b) factor scores based on exploratory and confirmatory factor analyses on the selected CDE items. Results indicated similar classification performance across models and sets of features. In this study, we suggest the need for robust evidence before adopting more complex classification models and identify measures that are particularly relevant in classifying suicide-related outcomes.
AgrestiA. (2013). Categorical data analysis (3rd ed.). Hoboken, NJ: John Wiley & Sons.
2.
AllenN. B.NelsonB. W.BrentD.AuerbachR. P. (2019). Short-term prediction of suicidal thoughts and behaviors in adolescents: Can recent developments in technology and computational science provide a breakthrough?Journal of Affective Disorders, 250, 163–169.
BachynskiK. E.Canham-ChervakM.BlackS. A.DadaE. O.MillikanA. M.JonesB. H. (2012). Mental health risk factors for suicides in the US Army, 2007–8. Injury Prevention, 18, 405–412.
5.
BaggeC. L.LittlefieldA. K.GlennC. R. (2017). Trajectories of affective response as warning signs for suicide attempts: An examination of the 48 hours prior to a recent suicide attempt. Clinical Psychological Science, 5, 259–271.
6.
BastienC. H.VallièresA.MorinC. M. (2001). Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Medicine, 2, 297–307.
7.
BeckA. T.WeissmanA.LesterD.TrexlerL. (1974). The measurement of pessimism: The hopelessness scale. Journal of Consulting and Clinical Psychology, 42, 861–865.
8.
BelsherB. E.SmolenskiD. J.PruittL. D.BushN. E.BeechE. H.WorkmanD. E.. . . SkoppN. A. (2019). Prediction models for suicide attempts and deaths: A systematic review and simulation. JAMA Psychiatry, 76, 642–651.
BreimanL. (2001). Random forests. Machine Learning, 45, 5–32.
11.
BreimanL.FriedmanJ. H.OlshenR. A.StoneC. J. (1984). Classification and regression trees. Belmont, CA: Wadsworth.
12.
BrowneM. W.CudeckR. (1993). Alternative ways of assessing model fit. In BollenK. A.LongJ. S. (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: SAGE.
13.
BurkeT. A.AmmermanB. A.JacobucciR. (2019). The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review. Journal of Affective Disorders, 245, 869–884.
14.
BushK.KivlahanD. R.McDonellM. B.FihnS. D.BradleyK. A. (1998). The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Archives of Internal Medicine, 158, 1789–1795.
ChanM. K.BhattiH.MeaderN.StocktonS.EvansJ.O’ConnorR. C.. . . KendallT. (2016). Predicting suicide following self-harm: Systematic review of risk factors and risk scales. The British Journal of Psychiatry, 209, 277–283.
17.
CzyzE. K.HorwitzA. G.ArangoA.KingC. A. (2019). Short-term change and prediction of suicidal ideation among adolescents: A daily diary study following psychiatric hospitalization. Journal of Child Psychology and Psychiatry, 60, 732–741.
18.
DevliegerI.RosseelY. (2017). Factor score path analysis: An alternative for SEM?Methodology, 13, 31–38.
19.
DwyerD. B.FalkaiP.KoutsoulerisN. (2018). Machine learning approaches for clinical psychology and psychiatry. Annual Review of Clinical Psychology, 14, 91–118.
20.
FranklinJ. C.RibeiroJ. D.FoxK. R.BentleyK. H.KleimanE. M.HuangX.. . . NockM. K. (2017). Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological Bulletin, 143, 187–232.
21.
FriedmanJ.HastieT.TibshiraniR. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1). Retrieved from https://www.jstatsoft.org/article/view/v033i01
22.
GibbonsR. D.HurK.MannJ. J. (2017). Suicide rates and the declining psychiatric hospital bed capacity in the United States. JAMA Psychiatry, 74, 849–850.
23.
GraumanK.LeibeB. (2011). Visual object recognition. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(2), 1–181.
24.
GustafsonD. H.GreistJ. H.StaussF. F.ErdmanH.LaughrenT. (1977). A probabilistic system for identifying suicide attemptors. Computers and Biomedical Research, 10, 83–89.
25.
HammondK. W.LaundryR. J.O’LearyT. M.JonesW. P. (2013). Use of text search to effectively identify lifetime prevalence of suicide attempts among veterans. In 2013 46th Hawaii International Conference on System Sciences (pp. 2676–2683). New York, NY: Institute of Electrical and Electronics Engineers. doi:10.1109/HICSS.2013.586
26.
HandD. J.TillR. J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45, 171–186.
27.
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning (2nd ed.). New York, NY: Springer.
28.
HayesT.UsamiS. (2020). Factor score regression in the presence of correlated unique factors. Educational and Psychological Measurement, 80, 5–40.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.
32.
HuangX.RibeiroJ. D.FranklinJ. C. (2020). The differences between suicide ideators and suicide attempters: Simple, complicated, or complex?Journal of Consulting and Clinical Psychology, 88, 554–569.
33.
JacobucciR.GrimmK. J. (2020). Machine learning and psychological research: The unexplored effect of measurement. Perspectives on Psychological Science, 15, 809–816.
34.
JacobucciR.LittlefieldA. K.MillnerA. J.KleimanE. M.SteinleyD. (2020). Pairing machine learning and clinical psychology: How you evaluate predictive performance matters. PsyArXiv. Retrieved from https://doi.org/10.31234/osf.io/2yber
35.
JoinerT. (2007). Why people die by suicide. Cambridge, MA: Harvard University Press.
36.
KesslerR. C.SteinM. B.PetukhovaM. V.BlieseP.BossarteR. M.BrometE. J.. . . BellA. M. (2017). Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Molecular Psychiatry, 22, 544–551.
37.
KesslerR. C.WarnerC. H.IvanyC.PetukhovaM. V.RoseS.BrometE. J.. . . FullertonC. S. (2015). Predicting suicides after psychiatric hospitalization in US Army soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry, 72, 49–57.
38.
KlineR. B. (2005). Principles and practice of structural equation modeling. New York, NY: Guilford.
39.
KlonskyE. D. (2020). The role of theory for understanding and preventing suicide (but not predicting it): A commentary on Hjelmeland and Knizek. Death Studies, 44, 459–462.
40.
KlonskyE. D.MayA. M. (2015). The three-step theory (3ST): A new theory of suicide rooted in the “ideation-to-action” framework. International Journal of Cognitive Therapy, 8, 114–129.
41.
KuhnM. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1–26.
42.
KuhnM.JohnsonK. (2013). Applied predictive modeling (Vol. 26). New York, NY: Springer.
43.
LargeM.SharmaS.CannonE.RyanC.NielssenO. (2011). Risk factors for suicide within a year of discharge from psychiatric hospital: A systematic meta-analysis. Australian and New Zealand Journal of Psychiatry, 45, 619–628.
44.
LeDellE.GillN.AielloS.FuA.CandelA.ClickC.. . . MalohlavaM. (2019). h2o: R Interface for ‘H2O’ (Version 3.26.0.10) [Computer software]. Retrieved from https://cran.r-project.org/package=h2o
45.
LiawA. (2018). randomForest: Breiman and Cutler’s random forests for classification and regression (Version 4.6.14) [Computer software]. Retrieved from https://CRAN.R-project.org/package=randomForest
46.
LyonJ. (2017). New data on suicide risk among military veterans. JAMA, 318, 1531. doi:10.1001/jama.2017.15982
47.
MeehlP. E. (1973). Why I do not attend case conferences. In MeehlP. E. (Ed.), Psychodiagnosis: Selected papers (pp. 225–302). Minneapolis: University of Minnesota Press.
48.
MeehlP. E. (1996). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence (New preface). Lanham, MD: Rowan & Littlefield/Jason Aronson. (Original work published 1954)
49.
MerkleE. C.ShafferV. A. (2011). Binary recursive partitioning: Background, methods, and application to psychology. British Journal of Mathematical and Statistical Psychology, 64, 161–181.
50.
MetalskyG. I.JoinerT. E. (1997). The hopelessness depression symptom questionnaire. Cognitive Therapy and Research, 21, 359–384.
51.
MorganJ. N.SonquistJ. A. (1963). Problems in the analysis of survey data, and a proposal. Journal of the American Statistical Association, 58, 415–434.
52.
MuthénL. K.MuthénB. O. (2017). Mplus user’s guide (8th ed.). Los Angeles, CA: Author.
53.
NaghaviM.AbajobirA. A.AbbafatiC.AbbasK. M.Abd-AllahF.AberaS. F.. . . AhmadiA. (2017). Global, regional, and national age-sex specific mortality for 264 causes of death, 1980–2016: A systematic analysis for the Global Burden of Disease Study 2016. The Lancet, 390, 1151–1210. doi:10.1016/S0140-6736(17)32152-9
NockM. K.SteinM. B.HeeringaS. G.UrsanoR. J.ColpeL. J.FullertonC. S.. . . ZaslavskyA. M. (2014). Prevalence and correlates of suicidal behavior among soldiers: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry, 71, 514–522.
56.
O’ConnorR. C.KirtleyO. J. (2018). The integrated motivational–volitional model of suicidal behaviour. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1754), Article 20170268. doi:10.1098/rstb.2017.0268
57.
OsmanA.BaggeC. L.GutierrezP. M.KonickL. C.KopperB. A.BarriosF. X. (2001). The Suicidal Behaviors Questionnaire-Revised (SBQ-R): Validation with clinical and nonclinical samples. Assessment, 8, 443–454.
58.
PatonM. B.LargeM. M.RyanC. J. (2014). Debate: Clinical risk categorisation is valuable in the prevention of suicide and severe violence–no. Australasian Psychiatry, 22, 10–12.
R Core Team (2019). R: A language and environment for statistical computing (Version 3.6.1) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.
61.
RibeiroJ. D.HuangX.FoxK. R.WalshC. G.LinthicumK. P. (2019). Predicting imminent suicidal thoughts and nonfatal attempts: The role of complexity. Clinical Psychological Science, 7, 941–957.
62.
RingerF. B.SoberayK. A.RogersM. L.HaganC. R.ChuC.SchneiderM.. . . GutierrezP. M. (2018). Initial validation of brief measures of suicide risk factors: Common data elements used by the Military Suicide Research Consortium. Psychological Assessment, 30, 767–778.
63.
RobinX.TurckN.HainardA.TibertiN.LisacekF.SanchezJ. C.MüllerM. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), Article 77. doi:10.1186/1471-2105-12-77
64.
RogersM. L.Kelliher-RabonJ.HaganC. R.HirschJ. K.JoinerT. E. (2017). Negative emotions in veterans relate to suicide risk through feelings of perceived burdensomeness and thwarted belongingness. Journal of Affective Disorders, 208, 15–21.
65.
SalganikM. J.LundbergI.KindelA. T.AhearnC. E.Al-GhoneimK.AlmaatouqA.. . . DattaD. (2020). Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, USA, 117, 8398–8403.
66.
SiddawayA. P.QuinlivanL.KapurN.O’ConnorR. C.de BeursD. (2020). Cautions, concerns, and future directions for using machine learning in relation to mental health problems and clinical and forensic risks: A brief comment on “Model complexity improves the prediction of nonsuicidal self-injury” (Fox et al., 2019). Journal of Consulting and Clinical Psychology, 88, 384–387.
67.
StanleyI. H.Buchman-SchmittJ. M.ChuC.RogersM. L.GaiA. R.WagnerR. K.. . . JoinerT. E. (2019). The Military Suicide Research Consortium common data elements: An examination of measurement invariance across current service members and veterans. Assessment, 26, 963–975.
68.
TantithamthavornC.McIntoshS.HassanA. E.MatsumotoK. (2016). An empirical comparison of model validation techniques for defect prediction models. IEEE Transactions on Software Engineering, 43, 1–18.
69.
TaylorS.ZvolenskyM. J.CoxB. J.DeaconB.HeimbergR. G.LedleyD. R.. . . ColesM. (2007). Robust dimensions of anxiety sensitivity: Development and initial validation of the Anxiety Sensitivity Index-3. Psychological Assessment, 19, 176–188.
70.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58, 267–288.
71.
TorousJ.WalkerR. (2019). Leveraging digital health and machine learning toward reducing suicide—From panacea to practical tool. JAMA Psychiatry, 76, 999–1000.
Van der LaanM. J.PolleyE. C.HubbardA. E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6, 1544–6115.
74.
Van OrdenK. A.CukrowiczK. C.WitteT. K.JoinerT. E.Jr. (2012). Thwarted belongingness and perceived burdensomeness: Construct validity and psychometric properties of the Interpersonal Needs Questionnaire. Psychological Assessment, 24, 197–215.
75.
VenablesW. N.RipleyB. D. (2002). Modern applied statistics with S (4th ed.). New York, NY: Springer.
76.
WalshC. G.RibeiroJ. D.FranklinJ. C. (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science, 5, 457–469.
77.
WeathersF.HuskaJ.KeaneT. (1991). The PTSD checklist military version (PCL-M). Boston, MA: National Center for PTSD.
78.
WeathersF. W.LitzB. T.HermanD.HuskaJ.KeaneT. (1994). The PTSD checklist-civilian version (PCL-C). Boston, MA: National Center for PTSD.
WolpertD. H. (1992). Stacked generalization. Neural Networks, 5, 241–259.
81.
WolpertD. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8, 1341–1390.
82.
ZouH.HastieT. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society B: Statistical Methodology, 67, 301–320.
83.
ZuromskiK. L.BerneckerS. L.GutierrezP. M.JoinerT. E.KingA. J.LiuH.. . . SteinM. B. (2019). Assessment of a risk index for suicide attempts among US Army Soldiers with suicide ideation: Analysis of data from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Network Open, 2(3), e190766–e190766. doi:10.1001/jamanetworkopen.2019.0766
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
