Sage Journals: Discover world-class research

Abstract

Objective:

Suicidal ideation and suicidal behaviour are both regarded as important risk factors for suicide, but it is usually believed that suicidal ideation is less strongly linked to suicide than suicidal behaviours. In this study, we assessed and compared the strengths of the associations that suicidal ideation and suicidal behaviour have with later suicide using meta-analysis of primary studies reporting both these independent variables and the dependent variable of suicide.

Methods:

A total of 51 English language publications describing cohort or controlled studies that reported on both the association between suicidal ideation and suicide and the association between suicidal behaviours and suicide were located using searches for titles in PubMed containing variants of the word suicide (suicid*). Suicides were considered to include reported suicides and open verdicts from mortality registers. The strengths of the two associations were examined in separate random effects meta-analyses and were then compared using mixed effects meta-regression. Subgroups were examined according to study characteristics including the definitions of suicidal ideation or behaviour used, setting (psychiatric or non-psychiatric), diagnostic mix of the study population, study design (cohort or control) and study quality.

Results:

Suicidal ideation (odds ratio = 3.11, 95% confidence interval = 2.51, 3.86) and suicidal behaviours (odds ratio = 4.09, 95% confidence interval = 3.05, 5.49) were both significantly associated with suicide but there was no significant difference in the strengths of association (p = 0.14). Nor were there significant differences in the strengths of the two associations in multiple subgroup analyses.

Conclusion:

Suicidal ideation and suicidal behaviour are both moderately associated with suicide. Existing data cannot conclusively demonstrate that suicidal behaviours are more strongly associated with suicide than suicidal ideation. Clinicians should not strongly prioritise suicidal behaviour over suicidal ideation when considering suicide risk.

Keywords

Suicide suicidal behaviour suicidal ideation prediction suicide risk

Introduction

Previous suicidal behaviour is often regarded as a key risk factor and warning sign for later suicide (Berman, 2018; Bostwick et al., 2016; Gunnell and Lewis, 2005; Smith et al., 2013). A World Health Organization (WHO, 2018) fact-sheet states that ‘a prior suicide attempt is the single most important risk factor’ for suicide. The salience of the association between suicidal behaviour and suicide is also emphasised by leading psychological theories of suicide, such as Joiner’s interpersonal theory of suicide, which suggests that in order for completed suicide to occur, an individual must have both suicidal ideation and an acquired capability to engage in suicidal behaviour. An acquired capability to engage in suicidal behaviour may override the individual’s innate fear of death and tendency towards perseveration of life and can develop as a result of exposure to trauma and violence, non-suicidal self-harming behaviours, substance use and access to means, all of which may increase an individual’s capability to carry out an act that would result in death (Van Orden et al., 2010). Importantly, Joiner’s theory suggests that suicidal ideation and behaviour are separate dimensions, rather than the same construct with differing degrees of severity. Although some studies have reported very strong associations between suicidal behaviour and suicide (Beck et al., 1999; Didham et al., 2006; Hyman et al., 2012), the strength of the association between suicidal behaviours and suicide was called into questioned by a recent meta-analysis by Ribeiro et al. (2016). Ribeiro et al. (2016) found a pooled odds ratio (OR) of suicide associated with suicide attempt of 2.03 (95% confidence interval [CI] = 1.61, 2.57) among a sample of 21 studies, consistent with a weak statistical association that they described as providing ‘marginal improvement in diagnostic accuracy above chance’. Ribeiro et al. (2016) also examined a separate sample of six studies finding that suicidal ideation was associated with suicide with an OR of 1.95 (95% CI = 1.31, 2.90) suggesting that suicidal ideation and behaviour are associated with suicide with a very similar effect size. Ribeiro et al.’s (2016) meta-analysis had the important strength of examining longitudinal studies that were free from possible hindsight and outcome bias in the assessment of suicidal ideation and behaviour. However, the inclusion of un-matched data sets and the exclusion of a large number of controlled studies left open some possibility that suicidal behaviour and suicidal ideation had a similar pooled OR by chance in their study, and that a larger study of matched data for the two independent variables would demonstrate that suicidal behaviour was more strongly associated with suicide than suicidal ideation.

Knowledge of the relative strengths of the associations between suicidal ideation and suicide and suicidal behaviours and suicide is of interest for both theoretical and practical reasons. First, if an acquired capability for suicidal behaviour is a necessary precursor for suicide, then it would be expected that suicidal behaviour would be more strongly associated with suicide than suicidal ideation because suicidal ideation may occur without an acquired capability. Second, people presenting with suicidal behaviour in clinical practice sometimes deny suicidal ideation, and people with suicidal ideation may have a history of suicide attempts (Berman, 2018; Smith et al., 2013). In both situations, the relative weight given to suicidal ideation and suicidal behaviour by clinicians might be important.

The aim of this study was to assess the strengths of the associations between the two independent variables (suicidal ideation and suicidal behaviour) and the dependent variable of suicide in a sample of primary studies reporting both associations. The null hypothesis was that the effect size of the associations between suicidal ideation and suicide and suicidal behaviours and suicide would not be statistically different with an alpha of 0.05 in the main or in any subgroup analyses.

Methods

We extended a previous meta-analysis of the association between suicidal ideation and suicide (McHugh et al., 2019) to include studies also reporting on suicidal behaviour. The extended meta-analysis was registered with PROSPERO (133007) and conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009).

Searches

Preliminary searches of Medline, Embase and PsychINFO from inception to January 2017 looking for papers that examined the association between suicidal ideation and suicide using the word ‘suicide’ as a key word or in the title resulted in an excessive number of results (McHugh et al., 2019), while placing limits using relevant terms such as ‘thoughts’, ‘ideation’ or ‘mortality’ missed relevant papers that were known to the authors from earlier meta-analyses (Chapman et al., 2015; Large et al., 2011b, 2018). In contrast, searches of PubMed using variants of the word suicide in the title reliably located papers known to the authors and those found in searches of multiple databases in earlier studies (Chapman et al., 2015; Hubers et al., 2018). Therefore, and in order to obtain a large and representative sample of relevant studies, one author (M.M.L.) examined the titles of English language publications with an accompanying abstract that contained variants of the single term ‘suicide’ (suicid*) in their title and were published in PubMed from inception to 18 July 2018. The reference list of relevant review articles (Chapman et al., 2015; Franklin et al., 2017; Hubers et al., 2018; Large et al., 2011a, 2011b; McHugh et al., 2019; Ribeiro et al., 2016) was hand searched. Two authors (M.M.L. and C.M.) winnowed the resulting abstracts and full-text publications (see Figure 1, Supplementary Material).

Inclusion and exclusion criteria

Full-text papers were initially examined for cohort or controlled studies that reported the effect size data (ORs, counts or other measures of effect size discrimination) quantifying the association between suicidal ideation and suicide. The resulting papers were then examined for similar effect size data about the association between suicidal behaviours and suicide. Papers that only reported on the association between suicidal ideation and suicide were excluded in order to examine the two independent variables in matched primary study populations and to avoid any possibility of systematic bias associated with an association between the study population and the choice of either suicidal ideation or suicidal behaviours. Hence, we included papers that reported on (1) the expression or non-expression of suicidal ideation, (2) the presence or absence of non-lethal suicidal behaviour and (3) relevant effect size data or counts (true positives, false positives, false negatives and true negatives) related to suicide. We included studies of patients who had received psychiatric care and people recruited from non-psychiatric settings as defined in an earlier meta-analysis of the association between suicidal ideation and suicide (Hubers et al., 2018; McHugh et al., 2019).

We excluded studies in which suicidal ideation and behaviour were assessed by psychological autopsy method, studies in which the non-suicidal controls were deceased and studies of patients with severe medical illnesses such as malignancies or human immunodeficiency virus.

Data extraction and definitions of suicide and suicidal ideation

Two authors (M.M.L. and C.M.) independently extracted the effect size data and the moderator variables. The OR was the preferred data format, followed by raw counts and other effect size data. A third author (A.C.) re-examined the data points. The discrepant points were re-examined by two authors until there was consensus. We accepted the definitions of suicidal behaviour (such as self-harm, suicide attempt, suicidal behaviour) and suicidal ideation (suicidal ideas/ideation/thoughts/threats) as used in the primary research. Data points regarding ‘suicide plans’ were excluded because suicide planning may be considered both ideation and behaviour, in particular if steps have been taken to enact a plan. When studies reported suicidal ideation and suicidal behaviour on a spectrum that assumed that all those with suicidal behaviour had suicidal ideation, estimates for the association between suicidal ideation and suicide were made using those without suicidal ideation or behaviours as the reference category. In these studies, estimates of the association between suicidal behaviour and suicide used those with no suicidal behaviour (including those with suicidal ideation) as the reference category. Where suicidal ideation and suicidal behaviours were recorded as separate categories, we analysed both variables separately using those without suicidal ideation and suicidal behaviours, respectively, as the reference group. In regarding suicidal ideation and suicidal behaviours as separate categories, we were aware that many patients with suicidal behaviour in the primary studies would have had suicidal ideation and some patients with suicidal ideation would have had suicidal behaviour. When papers reported more than one data set for suicidal behaviour or suicidal ideation (e.g. a past history of suicide attempts and recent presentation with suicidal behaviour), we chose most contemporaneous assessment of the two independent variables.

Moderator variables

We collected five moderator variables: (1) study population (whether the study consisted of patients from non-psychiatric settings such as primary care or military settings adopting the nomenclature of earlier research) (Hubers et al., 2018) and samples of psychiatric patients defined by having a schizophrenia spectrum psychosis, a mood disorder or mixed or other psychiatric diagnoses; (2) duration of follow-up between the assessment of the independent variables and possible suicide; (3) whether the study used a cohort of control design; (4) whether the studies reported suicidal ideation and behaviour dimensionally (thus excluding those with suicidal behaviour from the suicidal ideation group) or reported suicidal ideation and suicidal behaviours as independent categories; (5) according to a strength of reporting scale.

Assessment of strength of reporting

Two researchers (M.M.L. and C.M.) independently assessed the reporting strength (and hence the risk of bias of each individual study) of the methods used in the primary research using a scale (scored 0–4) derived from the Strengthening the Reporting of Observational Studies in Epidemiology statement checklist (Von Elm et al., 2007) and used in a related study (McHugh et al., 2019). The term strength of reporting was used instead of study quality to acknowledge that the primary studies were not generally conducted to assess the association between suicidal ideation and suicidal behaviour and suicide and could be regarded as being of adequate quality for their research purpose. One point was allocated according to each of the following criteria: (1) use of a structured method to assess suicidal ideation, (2) collection of data about suicidal ideation and behaviour in a method that was blinded to the patient’s suicide (either by a blinding method or by electronic recording at the point of assessment), (3) ascertainment of suicide using a mortality database and (4) inclusion of open verdicts as suicide. These items were added to produce a 5-point scale (0–4), with a score of 2 or more being considered high quality.

Data synthesis

Random effects meta-analysis was chosen a priori for all estimates because of the diversity of study populations and differences in the methods used in the primary research. As a first analytic step, the two independent variables of suicidal ideation and suicidal behaviour and the dependent variable of suicide were examined in two separate meta-analyses. The effect size data in the primary research data and pooled effect size were expressed in OR. Between-study heterogeneity in effect sizes was examined using I², Q value statistics. The possibility of publication bias was assessed using a funnel plot and Egger’s regression (Egger et al., 1997). Subgroups were examined in both the estimates of suicidal ideation and suicidal behaviours using a mixed model that assumed a random effect within subgroups and a fixed effect between subgroups. The study null hypothesis was examined in the second analytic step in which the statistical difference between the strength of the associations between suicidal ideation and suicidal behaviour and the null hypothesis was compared using a mixed effects meta-regression of all the studies and within subgroups. For the purpose of the meta-regression, data reporting the association between suicidal ideation and suicide were coded as ‘0’ and used as the reference variable and data reporting with suicidal behaviour were coded as ‘1’ such that a positive coefficient indicated a stronger association between suicidal ideation than suicidal behaviours. Comprehensive Meta-Analysis (CMA; Version 3; Biostat, Englewood, NJ, USA) was used in all analysis.

Results

Systematic searching located 51 publications where both the associations between suicidal behaviour and suicide and the associations between suicidal ideation and suicide were reported (see Table 1). The studies reported on 7,415,193 individuals (mean per study = 146,396, standard deviation [SD] = 859,898), among whom there were 3921 suicides (mean per study = 77, SD = 150). The mean length of follow-up was 5.5 years. These included 8 studies from non-psychiatric populations and 43 from psychiatric populations including 9 studies of people with schizophrenia, 9 studies of people with affective disorder and 25 studies of people with mixed or other psychiatric diagnoses. There were 30 case–control studies and 21 cohort studies. In all, 20 were classified as of lower quality and 31 of higher quality according to our quality scale (see Table 1, Supplementary Material).

Table 1.

Studies reporting on both the association between suicide ideation and suicide and suicide behaviour and suicide.

Author (year)	Subjects	Measure of suicidal ideation	Measure of suicidal behaviour	Measure of suicide	Suicidal behaviourOdds ratio (95% CI)	Suicidal ideationOdds ratio (95% CI)
Allebeck et al. (1987)	32 former inpatients with schizophrenia and 64 matched controls	Clinical documentation in the medical records	Suicide attempt	Suicides and open verdicts obtained from a mortality register	4.9 (1.9, 12.3)	2.3 (1.0, 5.7)
Al-Sayegh et al. (2015)	13 suicides among a cohort of 6293 members of the general population	Structured interview for suicidal thoughts and behaviours	Suicide attempt	Suicides and open verdicts obtained from a mortality register	3.1 (0.7, 14.3)	1.5 (0.3, 7.0)
Appleby et al. (1999)	149 discharged mental health patients and 149 matched controls	Clinical documentation in the medical records	Self-harm	Suicides and open verdicts obtained from a mortality register	2.9 (1.6, 5.3)	1.8 (1.1, 2.9)
Baader-Matthei et al. (2004)	32 mental health patients and 32 matched controls	Clinical documentation in the medical records	Suicidal behaviour	Mortality register	0.2 (0.1, 0.6)	3.8 (1.2, 11.8)
Beck et al. (1999)	30 suicides among a cohort of 3701 mental health outpatients	Scale for suicide ideation	Suicide attempt	Mortality register	11.6 (5.5, 24.6)	5.4 (2.6, 11.2)
Beisser and Blanchette (1961)	71 inpatient suicides and 71 matched controls	Suicide threats documented in the medical records	Suicide attempt	Clinically defined	7.9 (3.5, 17.8)	5.6 (2.5, 12.4)
Berg (2010)	3 suicides among 20 patients who had received electro-convulsive therapy	Clinical documentation in the medical records	Suicide attempt	Mortality register	1.1 (0.1, 14.7)	1.6 (0.1, 23.0)
Bertelsen et al. (2007)	7 suicides among a cohort of 547 people with first episode psychosis	Clinical Assessment in Neuropsychiatry instrument	Suicide attempt	Mortality register	0.7 (0.1, 5.9)	0.6 (0.1, 4.9)
Bickley et al. (2013)	100 recently discharged mental health patients and 100 matched controls	Retrospectively assessed by treating clinicians	Self-harm	Suicides and open verdicts obtained from a mortality register	4.1 (2.1, 7.9)	2.1 (0.9, 5.3)
Bradvik and Berglund (1993)	89 melancholically depressed former inpatients and 89 matched controls	Clinical documentation in the medical records	Suicide attempt	Mortality register	2.3 (1.1, 4.6)	1.3 (0.6, 2.6)
Cheng et al. (1990)	55 people with schizophrenia and 52 matched controls	Clinical documentation in the medical records	Suicide attempt	Clinically defined	3.7 (1.6, 8.6)	8.9 (1.1, 73.1)
Coryell and Young (2005)	33 suicides among a cohort of 782 patients with a depressive disorder	Schedule for Affective Disorder	Suicide attempt	Mortality register	4.8 (2.3, 9.8)	1.3 (0.4, 4.4)
Coryell et al. (2016)	7 suicides among a cohort of 1748 patients with bipolar disorder	Schedule for Affective Disorder	Suicide attempt	Clinically defined	2.8 (0.52, 15.0)	0.6 (0.1, 3.2)
Coryell et al. (2016)	12 suicides among a cohort of 288 patients with bipolar disorder	Schedule for Affective Disorder	Suicide attempt	Clinically defined	2.3 (1.2, 4.4)	2.2 (0.7, 7.0)
De Hert et al. (2001)	63 formerly inpatients with schizophrenia and 63 matched controls	Suicidal threats documented in the medical records	Suicide attempt	Clinically defined	4.8 (1.8, 12.7)	5.1 (2.1, 12.6)
Didham et al. (2006)	221 suicides among primary care patients and 663 matched controls	Clinical documentation in the medical records	Self-harm	Mortality register	40.5 (9.6, 170.3)	16.5 (3.7, 74.4)
Dingman and McGlashan (1986)	38 suicides among a cohort of 451 former inpatients	Suicidal threat documented in the medical records	Suicide attempt	Clinically defined	4.5 (2.2, 9.0)	3.0 (1.3, 6.6)
Dong et al. (2005)	92 inpatient suicides and 92 matched controls	Clinical documentation in the medical records	Self-harm	Suicides and open verdicts obtained from a mortality register	3.9 (2.0, 7.5)	4.7 (2.3, 9.4)
Drake et al. (1986)	15 suicides among a cohort of 104 former inpatients with schizophrenia	Clinical documentation in the medical records	Suicide attempt	Clinically defined	2.8 (0.8, 9.5)	8.3 (1.8, 39.1)
Farberow et al. (1966)	218 mental health inpatient suicides and 220 inpatient controls	Suicidal threats documented in the medical records	Suicide attempt	Clinically defined	22.7 (7.0, 74.2)	7.0 (4.1, 12.1)
Fawcett et al. (1990)	25 suicides among 954 patients with major affective disorder	Schedule for Affective Disorder	Suicide attempt	Clinically defined	1.2 (1.0, 1.5)	1.3 (1.0, 1.6)
Fernando and Storm (1984)	22 suicides by current or former mental health inpatients and 22 matched controls	Clinical documentation in the medical records	Deliberate self-harm	Clinically defined	7.1 (1.3, 38.8)	6.4 (1.7, 24.8)
Fosse et al. (2017)	36 former mental health inpatients and 120 matched controls	Suicide Risk Check List	Suicide attempt	Mortality register	2.3 (1.2, 4.5)	3.0 (1.5, 5.8)
Fruehwald et al. (2004)	220 suicides of prisoners and 440 matched controls	Suicidal threats documented in custodial records	Suicide attempt	Mortality register	8.1 (5.3, 12.3)	14.9 (8.2, 27.1)
Green et al. (2015)	43 suicides among a cohort of 5200 mental health outpatients	Beck depression inventory suicide	Suicide attempt	Mortality register	10.2 (5.5, 18.8)	5.1 (2.6, 10.1)
Hoyer et al. (2009)	135 current and former mental health inpatients and 135 matched controls	Clinical documentation in the medical records	Suicide attempt	Mortality register	3.7 (2.0, 7.0)	1.5 (0.9, 2.4)
Hunt et al. (2007)	222 current mental health inpatients and 222 controls	Retrospectively assessed by treating clinicians	Deliberate self-harm	Suicides and open verdicts obtained from a mortality register	3.3 (1.8, 6.2)	1.8 (0.9, 3.4)
Hunt et al. (2013)	107 recently admitted mental health patients and 107 controls	Retrospectively assessed by treating clinicians	Self-harm	Suicides and open verdicts obtained from a mortality register	3.5 (1.4, 8.7)	1.6 (0.7, 3.6)
Hunt et al. (2009)	238 recently discharged mental health patients and 238 controls	Retrospectively assessed by treating clinicians	Self-harm	Suicides and open verdicts obtained from a mortality register	2.9 (1.4, 6.1)	3.5 (1.4, 8.7)
Hyman et al. (2012)	24 suicides among a cohort of 4,070,167 military personnel	Systematically collected department of defence data	Suicide attempt	Mortality register	49.4 (33.4, 73.0)	10.5 (6.9, 15.8)
Kessler et al. (2015)	68 suicides among a cohort of 316,686 Veterans who were former inpatients	Systematically collected department of defence data	Suicide attempt	Mortality register	2.9 (1.7, 4.9)	2.4 (1.3, 4.6)
Khang et al. (2010)	13 suicides among a cohort of 5414 members of the general population	National health survey	Suicide attempt	Mortality register	4.1 (0.2, 69.7)	2.7 (0.9, 8.0)
Khanra et al. (2016)	10 current mental health inpatients and 50 matched controls	Clinical documentation in the medical records	Suicide attempt	Clinically defined	13.5 (2.8, 64.7)	6.7 (1.1, 40.1)
Kim et al. (2012)	324 depressed veterans and 312 matched controls	Clinical documentation in the medical records	Suicide attempt	Mortality register	17.9 (5.5, 58.2)	3.3 (2.2, 5.0)
Kleiman and Liu (2014)	25 suicides among a cohort of 19,989 members of the general population	National health survey	Suicide attempt	Mortality register	4.0 (0.9, 17.1)	2.8 (1.0, 8.2)
Kjelsberg et al. (1994)	35 former adolescent mental health inpatients and 35 matched controls	Clinical documentation in the medical records	Suicide attempt	Mortality register	1.0 (0.3, 2.9)	3.0 (1.0, 8.7)
Li et al. (2008)	64 inpatients with schizophrenia and 64 matched controls	Clinical documentation in the medical records	Suicide attempt	Clinically defined	5.2 (1.5, 18.0)	3.3 (1.2, 9.2)
Lopez-Morinigo et al. (2016)	71 people with schizophrenia and 355 controls	Semi-structured risk assessment	Suicide attempt	Suicides and open verdicts obtained from a mortality register	4.4 (2.0, 9.7)	3.6 (1.4, 9.1)
Lukaschek et al. (2014)	101 former mental health inpatients and 101 matched controls	Clinical documentation in the medical records	Suicide attempt	Railway suicides obtained using a mortality register	2.7 (1.5, 5.0)	7.4 (2.7, 20.3)
Park et al. (2017)	96 suicides among a cohort of 8403 recently discharged mental health patients	Clinical documentation in the medical records	Suicide attempt	Mortality register	2.3 (1.5, 3.6)	2.7 (1.7, 4.2)
Powell et al. (2000)	112 current mental health inpatients and 112 controls	Clinical documentation in the medical records	Self-harm	Suicides and open verdicts obtained from a mortality register	8.7 (4.1, 18.3)	5.6 (2.4, 13.5)
Sani et al. (2011)	96 former mental health inpatients and 192 matched controls	Clinical documentation in the medical records	Suicide attempt	Clinically defined	3.5 (2.0, 5.9)	2.1 (1.2, 3.7)
Shah and Ganesvaran (1997)	60 current or recently discharged suicides and 60 matched controls	Clinical documentation in the medical records	Suicide attempt	Clinically defined	2.3 (0.9, 5.6)	2.3 (1.0, 5.2)
Sinclair et al. (2005)	127 current or recently discharged patients with depression and 195 matched controls	Clinical documentation in the medical records	Suicide attempt	Suicides and open verdicts obtained from a mortality register	2.4 (1.2, 6.6)	2.0 (1.2, 3.4)
Sinclair et al. (2004)	51 current or recently discharged patients with schizophrenia and 82 matched controls	Clinical documentation in the medical records	Suicide attempt	Suicides and open verdicts obtained from a mortality register	8.1 (1.8, 36.9)	2.8 (1.2, 6.6)
Spiessl et al. (2002)	30 suicides among a cohort of 21,062 mental health inpatients	Clinical documentation in the medical records	Suicide attempt	Clinically defined	5.6 (2.3, 13.7)	1.0 (0.4, 2.1)
Stephens et al. (1999)	28 suicides among a cohort of 1212 inpatients with schizophrenia	Clinical documentation in the medical records	Suicide attempt	Clinically defined	5.9 (2.7, 13.1)	5.5 (2.6, 11.8)
Thong et al. (2008)	123 mental health patients and 123 matched controls	Clinical documentation in the medical records	Deliberate self-harm	Mortality register	6.5 (3.7, 11.4)	8.0 (3.4, 18.7)
Weiser et al. (2016)	54 suicides among a cohort of 89,049 young men receiving primary mental health care	Clinical documentation in the medical records	Suicide attempts	Mortality register	1.9 (0.8, 4.5)	1.4 (0.6, 3.1)
Werbeloff et al. (2016)	8 suicides among a birth cohort of 4914	Item on a research questionnaire	Suicide attempts	Mortality register	7.0 (0.9, 57.7)	1.6 (.2, 13.4)
Yim et al. (2004)	73 recently discharged mental health inpatients and 73 matched controls	Clinical documentation in the medical records	Suicide attempts	Mortality register	3.1 (1.5, 6.5)	3.4 (1.5, 7.9)

CI: confidence interval.

Suicidal ideation

Suicidal ideation was associated with suicide with a median OR of 2.90 and a study range of 0.55 to 16.5 with a pooled OR of 3.11 (see Figure 1). Studies conducted in non-psychiatric settings had the strongest pooled association between suicidal ideation and suicide, while studies of patients with affective disorder had a weaker association (see Table 2). The strength of the association between suicidal ideation and suicide was stronger in a small subgroup of studies that reported on people who were described as making a suicidal ‘threat’ (four studies; OR = 7.82, 95% CI = 4.80, 12.47), than 47 studies that reported on patients with suicidal ideation (OR = 2.83, 95% CI = 2.30, 3.49; Q = 14.1, df = 1, p = 0.001). The association between suicidal ideation and suicide was not associated with the mean duration of follow-up (coefficient = −0.02, 95% CI = −0.06, 0.01, Z = −1.34, p = 0.18). There was no evidence of publication bias on examination of the funnel plot (see Figure 2, Supplementary Material) or by Egger’s regression (intercept = 1.03, df = 49, p = 0.09).

Figure 1.

Forest plot of studies reporting the association between suicidal ideation, behaviour and suicide.

Table 2.

Main results and subgroup analysis of the association between suicidal ideation and suicide.

	Number of studies	Point estimate	Lower limit	Upper limit	I ²	Q value	df	p value
Main	51	3.11	2.51	3.86	75.4
Non-psychiatric	8	4.41	2.08	9.35	81.4	15.3	3.00	0.002
Affective disorder	9	1.69	1.23	2.33	55.2
Schizophrenia	9	3.76	2.67	5.93	0.00
Mixed diagnosis	25	3.25	2.61	4.04	52.8
Controlled studies	30	3.32	2.62	4.20	65.2	0.73	1.00	0.39
Cohort studies	21	2.70	1.77	4.07	82.4
Lower quality	20	4.50	3.23	6.26	64.9	7.54	1.00	0.006
Higher quality	31	2.51	1.94	3.23	74.2

Suicidal behaviour

Suicidal behaviour was associated with suicide with median OR of 3.87 and study range of 0.18 to 49.4 with a pooled OR of 4.09 (see Figure 2). Studies conducted in non-psychiatric settings had the strongest pooled association between suicidal behaviour and suicide, while studies of patients with affective disorder had a weaker association (see Table 3). The was no difference in the strength of the association between suicidal behaviour and suicide between the 39 studies that reported on patients with suicide attempts (OR = 4.39, 95% CI = 3.08, 6.24) and 12 studies that reported on patients with self-harm (OR = 3.25, 95% CI = 1.95, 5.40; Q = 0.91, df = 1, p = 0.34). The association between suicidal behaviour and suicide was not influenced by the mean duration of follow-up (coefficient = −0.01, 95% CI = −0.06, 0.04, Z = −0.47, p = 0.64). There was no evidence of publication bias on examination of the funnel plot (see Figure 3, Supplementary Material) or by Egger’s regression (intercept = 0.99, df = 49, p = 0.26).

Figure 2.

Forest plot of studies reporting association between suicidal behaviour and suicide.

Table 3.

Main results and subgroup analysis of the association between suicidal behaviour and suicide.

	Number of studies	Point estimate	Lower limit	Upper limit	I ²	Q value	df	p value
Main	51	4.09	3.05	5.50	87.5
Non-psychiatric	8	8.18	2.88	23.3	90.9	3.91	3.00	0.27
Affective disorder	9	2.89	1.74	4.81	79.9
Schizophrenia	9	4.45	2.17	6.23	0.0
Mixed diagnosis	25	3.74	2.79	5.02	75.6
Controlled studies	30	3.78	2.91	4.90	70.9	0.23	1.00	0.63
Cohort studies	21	4.44	2.40	8.24	93.4
Lower quality	20	4.60	3.08	6.87	76.5	0.43	1.00	0.51
Higher quality	31	3.80	2.55	5.66	90.3

Comparison of the strength of the association between suicidal ideation and suicidal behaviours and later suicide

Suicidal behaviours were more strongly associated with suicide than suicidal ideation (OR = 4.06 vs 3.02) but this did not reach significance as assessed with meta-regression (see Table 4). Among six studies that allowed us to exclude people with prior suicidal behaviour from the suicidal ideation group, there was a trend towards a weaker association between suicidal ideation and suicide (six studies; OR = 2.57, 95% CI = 1.63, 4.03) than the association between suicidal behaviour and suicide (six studies; OR = 5.28, 95% CI = 2.72, 10.2; Q = 3.11, df = 1, p = 0.08). There was no difference between the strength of the associations between suicidal ideation and suicide and suicidal behaviour and suicide in subgroups of studies as defined by study setting (non-psychiatric or psychiatric), diagnostic mix of study population, study design (cohort or control) or study quality.

Table 4.

Main result and subgroup meta-regression of suicide behaviour using suicide ideation as the reference variable.

	Number of studies	Coefficient	Standard error	Lower limit	Upper limit	Z value	p value
Main	51	0.27	0.19	−0.09	0.64	1.49	0.14
Non-psychiatric	8	0.66	0.66	−0.63	1.95	1.00	0.31
Affective disorder	9	0.52	0.30	−0.08	1.12	1.71	0.09
Schizophrenia	9	0.17	0.25	−0.31	0.65	0.69	0.49
Mixed diagnosis	25	0.13	0.19	−0.24	0.49	0.68	0.49
Controlled studies	30	0.12	0.18	−0.23	0.47	0.69	0.49
Cohort studies	21	0.53	0.38	−0.22	1.28	1.38	0.16
Lower quality	20	0.01	0.27	−0.51	0.53	0.05	0.95
Higher quality	31	0.43	0.24	−0.04	0.90	1.78	0.08

Discussion

Both the independent variables of suicidal ideation and suicidal behaviour are associated with suicide with an effect size that can be qualitatively assessed to be moderate (Rosenthal, 1995) and is similar in the strength of association as that found between male gender and suicide in most countries (Large, 2018). In the main analysis and in each of the subgroup analyses, suicidal behaviour was more strongly associated with suicide than suicidal ideation but this difference did not reach statistical significance and we were not able to reject the null hypothesis. There was a trend towards statistical significance in the subset of six studies in which those with suicidal behaviour were excluded from the suicidal ideation group. Our results should not be interpreted to suggest that future studies will not find a significant difference in the strength of the association between suicide and either suicidal ideation or suicidal behaviour. The consistency of non-significantly stronger association between suicidal behaviour in all subgroups does suggest that suicidal behaviour might be eventually shown to be more strongly associated with suicide than suicidal ideation as more data become available. However, our data suggest that any difference in statistical strength is likely to be small and of dubious clinical significance.

This study also provided an opportunity to assess whether measurement of dimensions of suicidal thoughts and behaviour impacts the strength of associations that are found between these distinct, but overlapping constructs, and later suicide. As mentioned, our previous meta-analysis (McHugh et al., 2019) assessed the strength of association between suicidal ideation and later suicide. Of the 71 studies in that meta-analysis, 51 measured suicidal ideation and behaviour separately and were thus included in this meta-analysis. The remaining 22 studies were not included as they did not include a measure of suicidal behaviour. While in the previous meta-analysis we found a pooled OR for suicidal ideation of 3.41, the current meta-analysis found an OR of 3.11. This indicates that studies that only report on suicidal ideation and not suicidal behaviour may tend to report a slightly higher OR. This may also be true for studies that report on suicidal behaviour only; however, we were not able to compare that data as the prior review focused on ideation only.

The main weakness of this study flowed from the conceptualisation of suicidal ideation and suicidal behaviours as categorically independent entities in all but six studies. In reality, it is likely that most patients classified as having suicidal behaviours had suicidal ideation and that many patients with suicidal ideation had a history of suicidal behaviour. This overlap between the two variables would lessen the extent of any difference in the strength of the two associations in 45 of the 51 included studies and in our meta-analytic result. Specifically, the inclusion of patients with suicidal behaviour in the suicidal ideation group might increase the observed association between suicidal ideation and suicide, lessening the possibility of finding a difference in the extent of the associations between these two variables and suicide. Despite the above caveat, we believe that the result of our study is somewhat surprising to the extent that we found that the objective sign of suicidal behaviour was not very much more strongly associated with suicide than the subjective symptom of suicidal ideation and that this difference did not reach statistical significance despite a highly powered meta-analysis. Prior to conducting this study, we held the opinion that suicidal behaviours were likely to be more strongly associated with suicide than suicidal ideation and formulated our hypothesis accordingly. We believed a priori that suicidal ideation would be less strongly associated with suicidal behaviours because they can be regarded as a communication of a persons’ inner world and level of distress rather than behaviour per se. Further suicidal ideation is much more common than suicidal behaviours (Johnston et al., 2009) and can be conceptualised as being dynamic and fluctuating over time, lessening the likely strength of any association. In contrast, suicidal behaviour is more static and objective and therefore a potentially reliable marker of suicide in line with a large body of research that suggests that actuarial prediction usually outperforms clinical prediction (Dawes et al., 1989). Moreover, one might expect suicidal behaviour to be more strongly associated with suicide because it is more similar to suicide, with both involving self-destructive behaviour and an acquired capability for self-harm (Van Orden et al., 2010). After this study, we modified our opinions about the relative strengths of the association between suicidal ideation and suicidal behaviours and considered that our results might be because people may be more reliable in their reporting of suicidal ideation than might generally be believed (Berman, 2018) and/or that the recording of suicidal behaviour is either less reliable (Eikelenboom et al., 2014) or less accurate (Hart et al., 2013) than one might imagine.

Conclusion

The results of this study are clinically relevant in that it is not rare for patients to present with suicidal behaviour yet deny current suicidal ideation while other patients will present with suicidal ideation in the absence of suicidal behaviour. This study suggests that, to the extent to which suicidal ideation and behaviours are conceptualised as risk factors, patients with either of these phenomena are more or less equal but only modestly increased risk of suicide. Patients with suicidal ideation only should not be dismissed as not having suicidal behaviour and there is no strong reason why people with suicidal behaviours should receive more restrictive mental health care than those with suicidal ideation only. The study also has some implications for future primary studies. Studies that report suicidal ideation and suicidal behaviours as both dimensional and categorical risk factors could more fully discriminate between the two risk factors. Highly powered primary studies and future meta-analyses might further clarify the relative strengths of these two common and important clinical aspects of suicidality.

While we urge clinicians to make very careful assessments of both past suicidal behaviour and of suicidal ideation, our results suggest that presently there are no strong grounds to put particular emphasis on suicidal behaviour over suicidal ideation.

Supplemental Material

Suppl._Material – Supplemental material for Is suicidal behaviour a stronger predictor of later suicide than suicidal ideation? A systematic review and meta-analysis

Supplemental material, Suppl._Material for Is suicidal behaviour a stronger predictor of later suicide than suicidal ideation? A systematic review and meta-analysis by Matthew Large, Amy Corderoy and Catherine McHugh in Australian & New Zealand Journal of Psychiatry

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: M.M.L. received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. C.M. received a New Investigator Grant from the Royal Australia and New Zealand College of Psychiatry.

ORCID iDs

Matthew Large

Catherine McHugh

Supplemental Material

Supplemental material for this article is available online.

References

Al-Sayegh

Lowry

Polur

, et al. (2015) Suicide history and mortality: A follow-up of a national cohort in the United States. Archives of Suicide Research 19: 35–47.

Allebeck

Varla

Kristjansson

, et al. (1987) Risk factors for suicide among patients with schizophrenia. Acta Psychiatrica Scandinavica 76: 414–419.

Appleby

Dennehy

Thomas

, et al. (1999) Aftercare and clinical characteristics of people with mental illness who commit suicide: A case-control study. The Lancet 353: 1397–1400.

Baader-Matthei

Richter

Mundt

(2004) Suicides of psychiatric hospitalized patients and their risk factors. A case-control study. Revista chilena de neuro-psiquiatría 42: 293–316.

Beck

Brown

Steer

, et al. (1999) Suicide ideation at its worst point: A predictor of eventual suicide in psychiatric outpatients. Suicide and Life-Threatening Behavior 29: 1–9.

Beisser

Blanchette

(1961) A study of suicides in a mental hospital. Diseases of the Nervous System 22: 365–369.

Berg

(2010) Death by suicide long after electroconvulsive therapy. Is the sense of coherence test of Antonovsky a predictor of mortality from depression? Mental Illness 2: e3.

Berman

(2018) Risk factors proximate to suicide and suicide risk assessment in the context of denied suicide ideation. Suicide and Life-Threatening Behavior 48: 340–352.

Bertelsen

Jeppesen

Petersen

, et al. (2007) Suicidal behaviour and mortality in first-episode psychosis: The OPUS trial. The British Journal of Psychiatry S51: S140–S146.

10.

Bickley

Hunt

Windfuhr

, et al. (2013) Suicide within two weeks of discharge from psychiatric inpatient care: A case-control study. Psychiatric Services 64: 653–659.

11.

Bostwick

Pabbati

Geske

, et al. (2016) Suicide attempt as a risk factor for completed suicide: Even more lethal than we knew. The American Journal of Psychiatry 173: 1094–1100.

12.

Bradvik

Berglund

(1993) Risk factors for suicide in melancholia. A case-record evaluation of 89 suicides and their controls. Acta Psychiatrica Scandinavica 87: 306–311.

13.

Chapman

Mullin

Ryan

, et al. (2015) Meta-analysis of the association between suicidal ideation and later suicide among patients with either a schizophrenia spectrum psychosis or a mood disorder. Acta Psychiatrica Scandinavica 131: 162–173.

14.

Cheng

Leung

, et al. (1990) Risk factors of suicide among schizophrenics. Acta Psychiatrica Scandinavica 81: 220–224.

15.

Coryell

Young

(2005) Clinical predictors of suicide in primary major depressive disorder. The Journal of Clinical Psychiatry 66: 412–417.

16.

Coryell

Kriener

Butcher

, et al. (2016) Risk factors for suicide in bipolar I disorder in two prospectively studied cohorts. Journal of Affective Disorders 190: 1–5.

17.

Dawes

Faust

Meehl

(1989) Clinical versus actuarial judgment. Science 243: 1668–1674.

18.

De Hert

McKenzie

Peuskens

(2001) Risk factors for suicide in young people suffering from schizophrenia: A long-term follow-up study. Schizophrenia Research 47: 127–134.

19.

Didham

Dovey

Reith

(2006) Characteristics of general practitioner consultations prior to suicide: A nested case-control study in New Zealand. The New Zealand Medical Journal 119: U2358.

20.

Dingman

McGlashan

(1986) Discriminating characteristics of suicides. Chestnut Lodge follow-up sample including patients with affective disorder, schizophrenia and schizoaffective disorder. Acta Psychiatrica Scandinavica 74: 91–97.

21.

Dong

Kan

(2005) A case-control study of 92 cases of in-patient suicides. Journal of Affective Disorders 87: 91–99.

22.

Drake

Gates

Cotton

(1986) Suicide among schizophrenics: A comparison of attempters and completed suicides. The British Journal of Psychiatry 149: 784–787.

23.

Egger

Davey Smith

Schneider

, et al. (1997) Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 315: 629–634.

24.

Eikelenboom

Smit

Beekman

, et al. (2014) Reporting suicide attempts: Consistency and its determinants in a large mental health study. International Journal of Methods in Psychiatric Research 23: 257–266.

25.

Farberow

Shneidman

Neuringer

(1966) Case history and hospitalization factors in suicides of neuropsychiatric hospital patients. The Journal of Nervous and Mental Disease 142: 32–44.

26.

Fawcett

Scheftner

Fogg

, et al. (1990) Time-related predictors of suicide in major affective disorder. The American Journal of Psychiatry 147: 1189–1194.

27.

Fernando

Storm

(1984) Suicide among psychiatric patients of a district general hospital. Psychological Medicine 14: 661–672.

28.

Fosse

Ryberg

Carlsson

, et al. (2017) Predictors of suicide in the patient population admitted to a locked-door psychiatric acute ward. PLoS ONE 12: e0173958.

29.

Franklin

Ribeiro

Fox

, et al. (2017) Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological Bulletin 143: 187–232.

30.

Fruehwald

Matschnig

Koenig

, et al. (2004) Suicide in custody: Case-control study. The British Journal of Psychiatry 185: 494–498.

31.

Green

Brown

Jager-Hyman

, et al. (2015) The predictive validity of the beck depression inventory suicide item. The Journal of Clinical Psychiatry 76: 1683–1686.

32.

Gunnell

Lewis

(2005) Studying suicide from the life course perspective: Implications for prevention. The British Journal of Psychiatry 187: 206–208.

33.

Hart

Musci

Ialongo

, et al. (2013) Demographic and clinical characteristics of consistent and inconsistent longitudinal reporters of lifetime suicide attempts in adolescence through young adulthood. Depression and Anxiety 30: 997–1004.

34.

Hoyer

Licht

Mortensen

(2009) Risk factors of suicide in inpatients and recently discharged patients with affective disorders. A case-control study. European Psychiatry 24: 317–321.

35.

Hubers

Moaddine

Peersmann

, et al. (2018) Suicidal ideation and subsequent completed suicide in both psychiatric and non-psychiatric populations: A meta-analysis. Epidemiology and Psychiatric Sciences 27: 186–198.

36.

Hunt

Bickley

Windfuhr

, et al. (2013) Suicide in recently admitted psychiatric in-patients: A case-control study. Journal of Affective Disorders 144: 123–128.

37.

Hunt

Kapur

Webb

, et al. (2007) Suicide in current psychiatric in-patients: A case-control study The National Confidential Inquiry into Suicide and Homicide. Psychological Medicine 37: 831–837.

38.

Hunt

Kapur

Webb

, et al. (2009) Suicide in recently discharged psychiatric patients: A case-control study. Psychological Medicine 39: 443–449.

39.

Hyman

Ireland

Frost

, et al. (2012) Suicide incidence and risk factors in an active duty U.S. military population. American Journal of Public Health 102(Suppl. 1): S138–S146.

40.

Johnston

Pirkis

Burgess

(2009) Suicidal thoughts and behaviours among Australian adults: Findings from the 2007 National Survey of Mental Health and Wellbeing. Australian and New Zealand Journal of Psychiatry 43: 635–643.

41.

Kessler

Warner

Ivany

, et al. (2015) Predicting suicides after psychiatric hospitalization in U.S. army soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 72: 49–57.

42.

Khang

Kim

Cho

(2010) Relationships of suicide ideation with cause-specific mortality in a longitudinal study of South Koreans. Suicide and Life-Threatening Behavior 40: 465–475.

43.

Khanra

Mahintamani

Bose

, et al. (2016) Inpatient suicide in a psychiatric hospital: A nested case-control study. Indian Journal of Psychological Medicine 38: 571–576.

44.

Kim

Smith

Ganoczy

, et al. (2012) Predictors of suicide in patient charts among patients with depression in the Veterans Health Administration health system: Importance of prescription drug and alcohol abuse. The Journal of Clinical Psychiatry 73: e1269–e1275.

45.

Kjelsberg

Neegaard

Dahl

(1994) Suicide in adolescent psychiatric inpatients: Incidence and predictive factors. Acta Psychiatrica Scandinavica 89: 235–241.

46.

Kleiman

Liu

(2014) Prospective prediction of suicide in a nationally representative sample: Religious service attendance as a protective factor. The British Journal of Psychiatry 204: 262–266.

47.

Large

Myles

, et al. (2018) Suicide risk assessment among psychiatric inpatients: A systematic review and meta-analysis of high-risk categories. Psychological Medicine 48: 1119–1127.

48.

Large

Sharma

Cannon

, et al. (2011a) Risk factors for suicide within a year of discharge from psychiatric hospital: A systematic meta-analysis. Australian and New Zealand Journal of Psychiatry 45: 619–628.

49.

Large

Smith

Sharma

, et al. (2011b) Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric in-patients. Acta Psychiatrica Scandinavica 124: 18–29.

50.

Large

(2018) The role of prediction in suicide prevention. Dialogues in Clinical Neuroscience 20: 197–205.

51.

Ran

Hao

, et al. (2008) Inpatient suicide in a Chinese psychiatric hospital. Suicide and Life-Threatening Behavior 38: 449–455.

52.

Lopez-Morinigo

Ayesa-Arriola

Torres-Romano

, et al. (2016) Risk assessment and suicide by patients with schizophrenia in secondary mental healthcare: A case-control study. BMJ Open 6: e011929.

53.

Lukaschek

Baumert

Krawitz

, et al. (2014) Determinants of completed railway suicides by psychiatric in-patients: Case-control study. The British Journal of Psychiatry 205: 398–406.

54.

McHugh

Corderoy

Ryan

, et al. (2019) Association between suicidal ideation and suicide: Meta-analyses of odds ratios, sensitivity, specificity and positive predictive value. BJPsych Open 5(2): e24.

55.

Moher

Liberati

Tetzlaff

, et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine 6: e1000097.

56.

Park

Hong

Jon

, et al. (2017) Past suicidal ideation as an independent risk factor for suicide behaviours in patients with depression. International Journal of Psychiatry in Clinical Practice 21: 24–28.

57.

Powell

Geddes

Deeks

, et al. (2000) Suicide in psychiatric hospital in-patients. Risk factors and their predictive power. The British Journal of Psychiatry 176: 266–272.

58.

Ribeiro

Franklin

Fox

, et al. (2016) Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: A meta-analysis of longitudinal studies. Psychological Medicine 46: 225–236.

59.

Rosenthal

(1995) Qualitative descriptors of strength of association and effect size. Journal of Social Science Research 21: 37–59.

60.

Sani

Tondo

Koukopoulos

, et al. (2011) Suicide in a large population of former psychiatric inpatients. Psychiatry and Clinical Neurosciences 65: 286–295.

61.

Shah

Ganesvaran

(1997) Inpatient suicides in an Australian mental hospital. Australian and New Zealand Journal of Psychiatry 31: 291–298.

62.

Sinclair

Harriss

Baldwin

, et al. (2005) Suicide in depressive disorders: A retrospective case-control study of 127 suicides. Journal of Affective Disorders 87: 107–113.

63.

Sinclair

Mullee

King

, et al. (2004) Suicide in schizophrenia: A retrospective case-control study of 51 suicides. Schizophrenia Bulletin 30: 803–811.

64.

Smith

Kim

Ganoczy

, et al. (2013) Suicide risk assessment received prior to suicide death by Veterans Health Administration patients with a history of depression. The Journal of Clinical Psychiatry 74: 226–232.

65.

Spiessl

Hubner-Liebermann

Cording

(2002) Suicidal behaviour of psychiatric in-patients. Acta Psychiatrica Scandinavica 106: 134–138.

66.

Stephens

Richard

McHugh

(1999) Suicide in patients hospitalized for schizophrenia: 1913-1940. The Journal of Nervous and Mental Disease 187: 10–14.

67.

Thong

Chan

, et al. (2008) Suicide in psychiatric patients: Case-control study in Singapore. Australian and New Zealand Journal of Psychiatry 42: 509–519.

68.

Van Orden

Witte

Cukrowicz

, et al. (2010) The interpersonal theory of suicide. Psychological Review 117: 575–600.

69.

Von Elm

Altman

Egger

, et al. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. The Lancet 370: 1453–1457.

70.

Weiser

Goldberg

Werbeloff

, et al. (2016) Risk of completed suicide in 89,049 young males assessed by a mental health professional. European Neuropsychopharmacology 26: 341–349.

71.

Werbeloff

Dohrenwend

Levav

, et al. (2016) Demographic, behavioral, and psychiatric risk factors for suicide. Crisis 37: 104–111.

72.

World Health Organization (WHO) (2018) Suicide. Available at: www.who.int/news-room/fact-sheets/detail/suicide (accessed 16 December 2019).

73.

Yim

Yip

, et al. (2004) Suicide after discharge from psychiatric inpatient care: A case-control study in Hong Kong. Australian and New Zealand Journal of Psychiatry 38: 65–72.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB