Abstract
Objective:
Established risk assessment tools are often inaccurate at predicting future suicide risk. We therefore investigated whether clinicians are able to predict individuals’ suicide risk with greater accuracy.
Method:
We used the SAFE Database, which included consecutive adult (age ≥18 years) presentations (N = 3818) over a 22-month period to the 2 tertiary care hospitals in Manitoba, Canada. Medical professionals assessed each individual and recorded his or her predicted risk for future suicide attempt (SA) on a 0-10 scale—the clinician prediction scale. The SAD PERSONS scale was completed as a comparison. SAs within 6 months, assessed using the Columbia Classification Algorithm for Suicide Assessment, were the primary outcome measure. Receiver operating characteristic curve and logistic regression analyses were conducted to determine the accuracy of both scales to predict SAs, and the scales were compared with z scores. Clinician prediction scale performance was stratified based on level of training.
Results:
Clinicians were able to predict future SAs with significantly greater accuracy (area under the curve [AUC] = 0.73; 95% CI, 0.68 to 0.77; P < 0.001) compared with the SAD PERSONS scale (z = 3.79, P < 0.001). Both scales nonetheless showed positive predictive value of less than 7%. Analyses by level of training showed that junior psychiatric residents and non–psychiatric residents did not accurately predict SAs, whereas senior psychiatric residents and staff psychiatrists demonstrated greater accuracy (AUC = 0.76 and 0.78, respectively).
Conclusions:
Clinicians are able to predict future attempts with fewer false positives than a conventional risk assessment scale, and this skill appears related to training level. Predicting future suicidal behaviour remains very challenging.
Clinical Implications
Consultation with a staff psychiatrist and/or senior psychiatry residents may be beneficial for junior trainees to increase the accuracy of risk assessment for future suicide attempt.
Medical staff should not solely rely on assessment tools when making treatment or disposition decisions (i.e., discharge versus admission).
Limitations
This study was unable to examine completed suicide. Suicide attempts may not always lead to eventual death by suicide. These 2 phenomena may or may not be predicted by the same set of risk factors.
Clinicians’ prediction scores may have been confounded by a variety of factors, including other elements of the assessment (i.e., the SAD PERSONS scale), the treatment plan, and availability of resources.
Approximately 0.01% of people in the Canadian general population die by suicide. 1 The rate of death within 1 year among emergency department (ED) patients presenting with self-harm ranges between 1% and 2%, and 4% die within 5 years. 2,3 The risk of death by suicide is especially high for psychiatric patients presenting to the ED with a suicide attempt (SA). 4 Thus, more accurate prediction of who will present to the ED with an SA may reduce eventual death by suicide and thus makes the ED an important point of assessment and intervention. However, tools that are available to clinicians for predicting suicide risk have little supporting clinical evidence. 5 The United Kingdom National Institute for Health and Care Excellence (NICE) encourages clinician assessment of risk among patients but opposes the use of assessment tools to make disposition decisions (i.e., discharge versus admission). 6 Although recent studies have shown promise with the use of certain novel paradigms such as the implicit association test in evaluating risk, 4 the usual practice of suicide risk assessment in the ED continues to involve clinician-patient interaction. A clinician’s impression of risk may be more feasible to obtain in busy hospital settings and is therefore important to examine as a potential predictive tool.
Clinicians in mental health are frequently tasked with suicide risk assessment in the ED. Disposition decisions are routinely made based on a clinician’s impression of the case, often arrived at using a nonstandardized approach influenced by patient factors, clinical experience, and other considerations. These ultimately contribute to an impression of the patient’s future risk of self-harm that in turn affects disposition. The objective of this study was to measure whether clinician impression accurately predicts future SAs within 6 months, using a simple 0-10 scale. This same scale was used in a previous study that compared the accuracy of a clinician’s prediction of future SAs with other prediction methods. 7 In that study, a clinician’s impression of risk did not predict SAs within 6 months. In addition to examining whether a clinician’s impression predicts future SAs, we sought to determine whether level of training influences the prediction ability of a clinician.
Methods
Ethical Approval
The study was approved by the Research Ethics Board of the University of Manitoba.
Setting and Study Population
Data came from the SAFE Database Study (Suicide Assessment Form in Emergency Psychiatry), a large multisite study examining risk factors for suicide. Details of the SAFE study have been previously described. 8 The study population included consecutive adult referrals to psychiatric services in the EDs of the 2 largest tertiary care hospitals in Manitoba, Canada (Figure 1). The study period was February 26, 2011, to December 31, 2012. Clinicians included physicians and trainees working in the ED psychiatry service. Level of clinician training was recorded as medical student, junior psychiatry resident (postgraduate year 1-2), senior psychiatry resident (postgraduate year 3-5), off-service resident (primarily residents in emergency medicine), staff psychiatrists, and other mental healthcare personnel (primarily physician assistants).
Flowchart of sample studied.
Baseline Measures
Clinician Prediction Scale
Clinician prediction of future SAs was assessed using the clinician prediction scale by Nock et al., 6 an 11-point scale (0-10) that clinicians used to rate their impression of the likelihood of a patient attempting suicide within the next 6 months (0 being “no likelihood” and 10 being “very high likelihood”).
SAD PERSONS Scale
This scale is a mnemonic for 10 potential suicide risk factors, including sex (male), age (>45 years), depression or hopelessness, previous attempts or psychiatric care, ethanol or substance abuse, rational thinking loss (psychosis), social supports lacking, organized plan or serious attempt, no spouse, and sickness (chronic pain or physical illness). Each item was scored as 1 if present and 0 if absent. 9 SAD PERSONS was used as a comparator assessment tool given its common use in hospital EDs. 10
Outcome Measure
An ED presentation with SA within 6 months of baseline presentation was the main outcome measure for this study. The Columbia Classification Algorithm for Suicide Assessment (C-CASA) was used to define an SA presentation. The C-CASA is a standardized scale that categorizes suicidal behaviour in 8 mutually exclusive categories. 11 This instrument differentiates SAs with intent to die from other self-harm behaviours without intent to die. Patients were considered to meet eligibility criteria if they had at least 1 ED visit between February 26, 2011, and July 1, 2012, and were followed up for 6 months after their visit. The 6-month timeframe is a convention used in several recent suicide risk assessment studies. 7,12
Statistical Analysis
Analyses were performed using SPSS 22.0 and STATA. 13,14 Multiple presenters with missing information regarding suicidal behaviour on subsequent presentations were excluded from further analyses. For people with multiple presentations, all ED presentations within 6 months prior to their SA were included as “suicide attempt within 6 months” (Figure 1).
Receiver operating characteristic (ROC) curve analyses were conducted to determine the optimum cut-off point for predicting future SA risk using the clinician prediction scale and the SAD PERSONS scale. An optimal cut-off point was determined for each scale based on the score that was associated with a sensitivity value of approximately 0.80. Area under the curve (AUC), as well as sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were calculated based on the cut-off points for each scale. Binary logistic regression analyses were conducted to examine the association between the ROC-derived cut-off points and future SAs. Separate ROC analyses were then conducted for the clinician prediction scale based on level of training, examining the same metrics. Finally, using the entire sample, we compared the predictive ability of the clinician prediction scale directly with the SAD PERSONS scale using z scores. To account for potential correlations between the 2 scales, bootstrapping was used.
Results
The study sample is described in Figure 1. Over the course of the study, there were 3818 referrals to psychiatric services in the ED that were identifiable with a correct personal health identification number. After removing initial referrals who presented after July 1, 2012, and who did not re-present to the ED with an SA, we were left with 2792 baseline referrals with enough known information for further analyses. Of the baseline sample, 136 of all presentations predicted a future SA within 6 months.
The ROC analysis for the clinician prediction scale indicated that with an optimum cut-off point of 2, 13 individuals who made a future SA were classified as having low risk, while 88 of these individuals were classified as high risk. When all training levels were combined, the scale had an AUC = 0.73 (95% CI, 0.68 to 0.77; P < 0.001) (Table 1). Information about the level of training of the clinician who completed the assessment was available for 2414 of the SAFE records. Analyses by level of training showed that the clinician prediction scores of junior and off-service residents did not predict SAs. Conversely, the scores of medical students, senior residents, and staff psychiatrists were significantly correlated with future SA, with the latter 2 groups approaching good accuracy. High risk scores on the SAD PERSONS scale were correlated with future SA, but with low accuracy, AUC = 0.62 (95% CI, 0.57 to 0.67; P < 0.001). A comparison between the clinician prediction scale and the SAD PERSONS scale revealed that the clinician prediction scale’s AUC value was significantly higher than that of the SAD PERSONS, with a z value of 3.79 (P < 0.001) based on a 1-tailed test. PPVs were low for both scales.
Receiver operating characteristic analysis for the assessment of future suicide attempt within 6 months.
a P < 0.001.
b P < 0.05.
c P < 0.01.
PGY = postgraduate year.
Conclusion
The primary finding of this study is that a clinician’s impression of risk predicts future SAs with greater accuracy than an established risk assessment scale and with an improvement of approximately 10% in specificity, which is important since it may help to save available resources. This is interesting as this impression was measured using a simple 0-10 scale that relied solely on a clinician’s opinion rather than a conventional risk assessment tool based on a scored list of risk factors. Secondary findings include a gradient of prediction based on level of psychiatric training and the superiority of clinician prediction over the commonly used SAD PERSONS scale. Despite the significant discrimination of the clinician prediction scale, and its good sensitivity, it possesses a very low PPV, which suggests that clinicians were not able to predict with great accuracy which individuals would make an SA within 6 months. This is common among suicide risk assessment scales, as low incidence events will always challenge the PPV of a test even when sensitivity and specificity are high. 15 Limitations of this study include the inability to measure suicide as an outcome and the reliance on people returning to the study hospital with an SA. There are likely people who attempted suicide and either did not present for care or went to another site. An additional limitation is that treatment may be correlated both with scale scores and the outcome of SAs, such that people perceived at higher risk receive more aggressive treatment, which then potentially influences their risk of subsequent attempt. Finally, the potential of correlated error associated with including multiple presentations by the same individual was also not assessed and remains a limitation. This study highlights the importance of clinician assessment in risk prediction. It emphasizes that clinicians formulate risk based on factors that are not always captured in risk assessment tools and that additional training and experience help refine that skill set.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Preparation of this article was supported by research grants from the Manitoba Health Research Council (Dr. Bolton), Manitoba Health Research Council Chair Award (Dr. Sareen), Canadian Institutes of Health Research New Investigator Awards (Dr. Bolton no. 113589; Dr. Sareen no. 152348), the Social Sciences and Humanities Research Council Joseph-Armand Bombardier Canada Graduate Scholarship (Ms. Wang), the University of Manitoba Graduate Fellowship (Ms. Bhaskaran), and the Manitoba Health Research Council Graduate Studentship (Ms. Bolton). The funding sources had no role in the design and conduct of the study; no role in the collection, management, analysis, and interpretation of data; and no role in the preparation, review, and approval of the manuscript.