An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide

Abstract

Background:

Suicide rates continue to rise across the United States, galvanizing the need for increased suicide prevention and intervention efforts. The Zero Suicide (ZS) model was developed in response to this need and highlights four key clinical functions of high-quality health care for patients at risk of suicide. The goal of this quality improvement study was to understand how six large health care systems operationalized practices to support these functions—identification, engagement, treatment and care transitions.

Methods:

Using a key informant interview guide and data collection template, researchers who were embedded in each health care system cataloged and summarized current and future practices supporting ZS, including, (1) the function addressed; (2) a description of practice intent and mechanism of intervention; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented and/or measured. Normalization process theory (NPT), an implementation evaluation framework, was applied to help understand how ZS had been operationalized in routine clinical practices and, specifically, what ZS practices were described by key informants (coherence), the current state of norms/conventions supporting these practices (cognitive participation), how health care teams performed these practices (collective action), and whether/how practices were measured when they occurred (reflexive monitoring).

Results:

The most well-defined and consistently measured ZS practices (current and future) focused on the identification of patients at high risk of suicide. Stakeholders also described numerous engagement and treatment practices, and some practices intended to support care transitions. However, few engagement and transition practices were systematically measured, and few treatment practices were designed specifically for patients at risk of suicide.

Conclusions:

The findings from this study will support large-scale evaluation of the effectiveness of ZS implementation and inform recommendations for implementation of high-quality suicide-related care in health care systems nationwide.

Plain Language Summary

Many individuals see a health care provider prior to death by suicide, therefore health care organizations have an important role to play in suicide prevention. The Zero Suicide model is designed to address four key functions of high-quality care for patients at risk of suicide: (1) identification of suicide risk via routine screening/assessment practices, (2) engagement of patients at risk in care, (3) effective treatment, and (4) care transition support, particularly after hospitalizations for suicide attempts. Researchers embedded in six large health care systems, together caring for nearly 11.5 million patients, are evaluating the effectiveness of the Zero Suicide model for suicide prevention. This evaluation focused on understanding how these systems had implemented clinical practices supporting Zero Suicide. Researchers collected qualitative data from providers, administrators, and support staff in each system who were responsible for implementation of practices supporting Zero Suicide. Normalization process theory, an implementation evaluation framework, was applied following data collection to: (A) help researchers catalog all Zero Suicide practices described, (B) describe the norms/conventions supporting these practices, (C) describe how health care teams were performing these practices, and (D) describe how practices were being measured. The findings from this evaluation will be vital for measuring the effectiveness of different Zero Suicide practices. This work will also provide a blueprint to help health care leaders, providers, and other stakeholders “normalize” new and existing suicide prevention practices in their own organizations.

Keywords

Suicide prevention implementation science normalization process theory health services research Zero Suicide

Introduction

Suicide rates have continued to rise across the United States for the past two decades, thus galvanizing a need for increased prevention and intervention efforts across settings (Caine, 2013; Hogan & Grumet, 2016). Health care systems have a key role to play in suicide prevention, because many individuals see a health care provider prior to death by suicide—nearly a third in the week prior to suicide (Ahmedani et al., 2019), half in the month prior, and the vast majority in the year prior (Ahmedani et al., 2014; Luoma et al., 2002). The Zero Suicide (ZS) model is a quality improvement initiative for safer and more effective suicide care in health care settings that came out of a “call to action” developed by the Office of the US Surgeon General and the National Action Alliance for Suicide Prevention (U.S. Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention, 2012). The ZS model is not a formal validated treatment protocol with prescribed interventions, but rather a framework of evidence-based practices and implementation strategies designed to transform organizational culture (Hogan & Grumet, 2016). The ZS model strategy defines four essential clinical functions of high-quality care for patients at risk of suicide: identification of high-risk patients, engagement and care management, effective treatment, and supportive care transitions (see Table 1 for Terms and Definitions) (Brodsky et al., 2018; Education Development Center, 2020). Each of these functions is supported by the application of a procedure or practice intended to support a specific function (Jolles et al., 2019). Identifying high-risk patients involves screening and risk assessment, for example, using the nine-item Patient Health Questionnaire (PHQ-9)] (Kroenke et al., 2001), and the Columbia Suicide Severity Rating Scale (C-SSRS) (Posner et al., 2011). Engagement practices may include collaborative crisis response and safety planning interventions to help patients identify coping strategies and other resources during suicidal crises (Bryan et al., 2018; Stanley & Brown, 2012). Effective treatments may include evidence-based therapies targeting intensity and/or frequency of suicidal thoughts and behaviors (Brown & Jager-Hyman, 2014). Supportive care transitions ensure that patients receive uninterrupted care as they move between care settings, particularly from inpatient to outpatient settings following suicide attempt (Chung et al., 2017).

Table 1.

Terms and definitions.

Term	Definition
Determinant	A factor that enables or hinders the clinical practice from achieving the desired effect.
Function	The purpose of a practice. The core 4 clinical functions of high-quality suicide care defined in the ZS Model include identification, engagement, treatment and transition.
Mechanism	Process or event through which a clinical practice operates to affect outcomes.
Practice	The application of a procedure intended to support a specific function (aka “form following function”). This evaluation focuses on clinical practices intended to support the ZS Model.
ZS Model	A framework designed to support system-wide, organizational commitment to high-quality suicide care in healthcare.

ZS: Zero Suicide.

Although the ZS model has great potential to improve the way health care systems care for patients at risk of suicide, and has been widely adopted, there is limited evidence of the effectiveness of a collection of linked, mutually supporting clinical practices for prevention of suicide attempts (fatal and non-fatal). For example, the identification function is supported by research demonstrating that disclosure of suicidal thoughts on the PHQ-9 depression assessment is a strong predictor of subsequent suicide attempts (Louzon et al., 2016; Simon et al., 2013, 2016), but the link between identification of patients experiencing suicidal thoughts and the ability of health care systems to intervene effectively and help prevent suicide attempts is unclear. Similarly, though randomized clinical trials have demonstrated efficacy of some psychotherapeutic interventions for reducing suicide attempts, little is known about the effectiveness of these practices for suicide prevention in real-world settings or among people who have not volunteered to participate in research (Brown & Jager-Hyman, 2014). Moreover, though prior research has demonstrated that systemic, bundled suicide prevention practices are more effective than any single practice (M. J. Coffey et al., 2015; Knox et al., 2003; Martin et al., 2016; While et al., 2012), it is unclear which specific practices, or combination of practices, supporting different ZS functions are most effective (Ahmedani & Vannoy, 2014).

Early examples of implementation of clinical practices related to the ZS model have been encouraging, such as the substantial reductions seen in annual suicide rates following implementation of the Perfect Depression Care program at the Henry Ford Health System (C. E. Coffey et al., 2013; Hampton, 2010). Nevertheless, more rigorous study is needed to demonstrate how specific ZS clinical practices are associated with important patient outcomes, specifically suicide attempt and mortality. Therefore, researchers in six large health care systems are investigating the effectiveness of different clinical practices (and combinations of practices) for preventing suicide attempts and deaths (Yarborough et al., 2019). At the time of this evaluation, these six health care systems had all implemented varying combinations of suicide prevention practices and were routinely collaborating via a formal suicide prevention learning collaborative (Bruschke & Flores, 2020). Therefore, in support of the learning collaborative and the planned investigation of ZS effectiveness (Figure 1), this research team sought to build a blueprint of the practices supporting the four key clinical functions (identify, engage, treat, transition) of the ZS model across health care systems.

Figure 1.

Goals of the planned Zero Suicide evaluation.

Thus, this evaluation focused on understanding how these six health care systems had implemented (or planned to implement) clinical practices supporting the four key clinical functions highlighted in the ZS Model, in support of a broader ZS evaluation. Building a thorough working knowledge of practice variation across systems would be essential for measuring and comparing the effectiveness of different ZS practices (Figure 1). Therefore, we used normalization process theory (NPT), an implementation evaluation framework (McEvoy et al., 2014; Nilsen, 2015), to help explain which individual and interrelated practices had been implemented and how those practices had been operationalized across different health care systems to support ZS. NPT was selected because the framework focuses on the organic process of how complex interventions become normalized in practice, shaped by four determinants—coherence (how the intervention is understood), cognitive participation (how users engage in the intervention), collective action (how the intervention is performed), and reflexive monitoring (how the intervention is monitored over time) (May et al., 2009; May & Finch, 2009). Specifically, these four NPT determinants were applied following key informant data collection to help explain how different clinical practices supporting ZS had become normalized across organizations and care settings.

Methods

Participating health care systems

Six large health care systems, together caring for nearly 11.5 million members in California, Colorado, Michigan, Oregon, and Washington, are implementing clinical practices supporting the ZS model and agreed to participate in the implementation evaluation. All participating health care systems provide both comprehensive health care and insurance coverage to a defined patient population. Each system is described here and additional details are presented in Table 2, using data compiled from 10/2018-9/2019 (Henry Ford Health System, 2019; Kaiser Permanente, 2019). The participating Kaiser Permanente (KP) health care systems in California together served approximately 9 million patient members, followed by Henry Ford Health System in Michigan serving 1.2 million, and KP health care systems in Oregon, Washington and Colorado similarly serving between 600,000–700,00 members each. The Institutional Review Board at each system approved its participation in the ZS evaluation.

Table 2.

Characteristics of participating health care systems and patient-populations (10/1/2018-9/30/2019).

System characteristics	KPSC	KPNC	KPNW	KPCO	HFHS	KPWA
Members	4.6 million	4.4 million	620,000	640,000	1.2 million	700,000
Outpatient Medical Centers/Clinics	231	252	59	34	50	34
Hospitals	15	21	2	0	6	0
Patient characteristics
Female	51.7%	51.6%	52.1%	52.9%	57.4%	53.9%
Age
0–19	23.4%	22.2%	20.9%	19.9%	18.1%	15.9%
20–39	27.4%	27.1%	25.8%	23.6%	21.5%	25.7%
40–64	33.4%	34.3%	34.7%	35.9%	36.6%	39.9%
65+	15.9%	16.4%	18.6%	20.6%	23.8%	18.4%
Insurance
Medicaid	10.4%	7.3%	9.2%	6.8%	0.4%	0.6%
Medicare	13.6%	19.6%	19.6%	21.7%	23.5%	19.6%
Socioeconomic status
Neighborhood income < 25 K	0.8%	1.2%	0.4%	0.6%	6.7%	1.2%
Neighborhood education < 25% college	53.8%	37.6%	39.3%	25.4%	52.9%	38.9%
Race/Ethnicity
Asian	10.9%	20.1%	6.4%	3.5%	4.6%	7.4%
Black/African American	8.2%	6.9%	3.4%	4.2%	28.2%	3.4%
Hispanic/Lantinx	40.9%	21.1%	8.5%	15.9%	2.3%	4.0%
Hawaiian/Pacific Islander	0.8%	1.0%	0.9%	0.3%	0.1%	0.9%
American Indian/Alaska Native	0.3%	0.5%	0.8%	0.6%	0.4%	1.0%
Multiple/Other	0.0%	0.0%	0.4%	3.1%	1.8%	1.0%
Unknown	7.0%	7.1%	8.1%	12.4%	8.7%	34.1%
White	31.9%	43.4%	71.6%	59.9%	54.0%	48.2%
Any Mental Health Diagnosis in 2018	16.2%	14.4%	22.3%	20.3%	16.8%	20.4%

KP: Kaiser Permanente; KPSC: KP Southern California; KPNC: KP Northern California; KPNW: KP Northwest (Oregon/Southern Washington); KPCO: KP Colorado; HFHS: Henry Ford Health System; KPWA: KP Washington.

The characteristics of the different health care system patient populations vary by sociodemographic characteristics associated with suicide, including age, race/ethnicity, income, and education (Nock et al., 2008; Rehkopf & Buka, 2006). For example, at the time of this evaluation, Henry Ford provided care to the oldest patient population and had the largest proportion insured by Medicare (23.5%), while KP Southern California served the youngest population and had the largest proportion insured by Medicaid (10.4%). Henry Ford served the largest proportion of Black/African American patients (28.2%), KP Southern California the largest proportion of Hispanic/Latinx patients (40.9%), and KP Northern California the largest proportion of Asian patients (20.1%).

Suicide prevention learning collaborative

In 2016, Kaiser Permanente Care Management Institute (CMI) , convened a Suicide Prevention Learning Collaborative workgroup, which included clinical, operational, and patient safety stakeholders, as well as peer advisors (KP health plan members and advocates with lived experience from national mental health advocacy organizations), and researchers, including those from all six health care systems partici-pating in the ZS evaluation. The charter of this workgroup included cross-regional collaboration on implementation of best-practices for suicide prevention (Bruschke & Flores, 2020). With oversight from operational mental health leadership, CMI works with members from the suicide prevention learning collaborative to facilitate development new electronic health record (EHR)-based tools, trainings and metrics to support key ZS practices which build from the work described in this evaluation.

Key informant data collection

A key informant meeting guide (see Supplement A) was developed to assist researchers embedded in each health care system (JER, GES, JMB, AB, BHY, KJC, SAS, JW, BKA) to collaborate with key informants and document clinical practices the systems had implemented (or planned to implement) across the different clinical functions of ZS. Prior to data collection, embedded researchers all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems, so data collection was designed to enhance that knowledge. Key informants included local leaders and staff with knowledge of implementation of suicide prevention related processes and protocols in the various clinical settings. A structured data collection template (Miles et al., 2014) was designed to help the researchers catalog details about current and planned suicide prevention practices across care settings (Supplement B). The data collection template did not explicitly assess the domains of the ZS Model focused on implementation strategies (training, leadership, and continuous quality improvement) (Education Development Center, 2020), but these domains were implicitly addressed in questions about how the practices had/or would be implemented. The template was originally developed and tested at KP Colorado and subsequently tested at two other health care systems, with the finalized version made available for widespread use in March 2018. Templates were populated and reviewed iteratively by the embedded researcher and team members at each participating health care system. In parallel, templates were also uploaded to a private web-based file storage platform available to Health care System Research Network members (Health Care Systems Research Network, 2019) and each reviewed in detail by members of the full study team online during routine all-site team meetings (December 2018 through April 2019). Following these presentations, some researchers made additional edits and updates to their templates to reflect changing ZS practices across their systems and uploaded current versions to the shared private website. In August 2019, a health services research (JER) used the most recent version of the template from each health care system to summarize ZS practices across health care systems, including (1) the function addressed; (2) a brief description of the practice intent and mechanism of intervention (e.g., EHR-based clinical decision support tools), standard workflows/processes and health care system policies; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented/measured.

Application of NPT

NPT was applied following key informant data collection to help understand how health care systems had implemented different clinical practices, and combinations of practices supporting the ZS model (May et al., 2009; May & Finch, 2009). NPT assumes the way that practices become routinely embedded and integrated into their social contexts—how they become normalized—is not a structured sequence of events, but instead is an organic process shaped by four determinants: coherence, cognitive participation, collective action, and reflexive monitoring (May & Finch, 2009; Nilsen, 2015). To support this evaluation, we first developed a working interpretation of the NPT determinants for application to ZS implementation, guided by two prior studies evaluating implementation of depression care in primary care (Franx et al., 2012; Gunn et al., 2010). Specifically, like Gunn et al. (2010), we developed working definitions for how each of the NPT determinants would apply to health care system implementation of practices (and combinations of practices) supporting the ZS model (Table 3). Specifically, we used NPT to catalog and connect the descriptions of current and future clinical practices to ZS implementation (i.e., creating coherence) across the health care systems, and to facilitate the specification of the workflows/processes (cognitive participation; collective action) and approaches to measuring these workflows/processes (reflexive monitoring). This approach involved using NPT pragmatically to define specific products resulting from our application of NPT to the key informant data (Table 3) to support the goals of the broader ZS evaluation (Figure 1). Specifically, to describe coherence, a health services research (JER), first used the templated key informant data to create a thematic network (Attride-Stirling, 2001) (i.e., affinity diagram) of the existing practices supporting the four key clinical functions of ZS across all six participating health care systems. Embedded researchers from all participating health care systems, responsible for data collection, participated in several rounds of iterative review and refinement to ensure ZS practices were comprehensively and accurately represented in the final version of the thematic network. Second, we summarized the current state of practices supporting ZS across health care systems and created a list of tools (e.g., screeners, assessments, templates) supporting those practices to describe cognitive participation and collective action. Third, to describe reflexive monitoring we built a working knowledge of whether and how ZS practices were being measured and how different practices and combinations of practices could be measured over time. This work also involved conceptually mapping a suicide care continuum, similar to treatment cascades used to evaluate care gaps for patients living with HIV (Kay et al., 2016).

Table 3.

Normalization process theory (NPT) determinants and application to Zero Suicide (ZS) evaluation.

NPT determinant	Definition (May & Finch, 2009)	Application to ZS Evaluation	Product
Coherence	Do people know what the work is?	What clinical practices support the four clinical functions of ZS?	Thematic network
Cognitive Participation	Do people join in to the work?	What norms/conventions support ZS practices?	Summary of current workflows supporting ZS practices
Collective Action Skillset Workability Contextual Integration Interactional Workability Relational Integration	How do people do the work?	What tools are used to support ZS practices? How?	Summary of clinical decision support tools used across healthcare systems
Reflexive Monitoring	How do we know the work is happening?	How should we measure ZS practices?	Care continuum model

NPT: normalization process theory; ZS: Zero Suicide. z

Results

Key informant data collection and application of NPT

Embedded health care system researchers had all begun populating the implementation templates in collaboration with local key informants by June 2018. An average of 14 key informants per health care system provided data to help populate each template (range 7–26). Key informants included providers, administrators, EHR programmers, and project managers (often known to embedded researchers) who were responsible for supporting implementation of practices supporting ZS in different ways. All health care systems had participated in virtual team review and updated their templates at least once before the designated health services researcher (JER) begin reviewing template data in August 2019. Prior to application of NPT, the health services researcher organized the practices summarized in the templates by function. In some cases, the function was changed (e.g., “treat” to “engage”) to be consistent with the ZS framework definitions, in collaboration with a ZS expert (JGG).

Each product associated with application of NPT to the templated data (Table 3) was created consecutively and is described in detail below, including (1) the catalog of practices understood to support ZS (coherence); (2) the summary of the current state of norms/conventions supporting these practices and how health care teams performed these practices (cognitive participation and collective action); and (3) a list of tools available to measure practices across systems and a model describing combinations of practices available to measure the continuum of suicide care across settings (reflexive monitoring).

Coherence

In Figure 2, we present the thematic network of clinical practices key informants understood to support ZS (coherence), derived from the implementation templates completed by an embedded researcher at each site. The number of current/planned clinical practices described by key informants varied across health care systems (average per health system = 32, range 19–55), as did the level of detail in the description of each practice (Supplement B). All templates included fairly detailed descriptions of practices supporting the identify function (i.e., standard workflows and tools), but practices supporting the engage, treat and transition functions were often less clearly defined. Moreover, often the clinical, operational, and patient safety stakeholders did not explicitly reference the ZS model when they described the current state of health care system practices supporting suicide prevention. Some templates also included descriptions of practices that were not specifically designed to support suicide prevention, but rather mental health and addiction care more generally. For example, substance use disorder screening/assessment, social support, care coordination/outreach, and types of psychotherapy. Relatedly, some of these practices had been in place at most health care systems more than 5 years prior to this evaluation (like depression severity assessment with the PHQ-9 and different types of psychotherapy), while others had been implemented more recently (suicide severity assessment with the C-SSRS).

Figure 2.

Zero Suicide (ZS)-related practices across all participating health care systems cataloged by the clinical function of ZS.

Cognitive participation and collective action

Identification of high-risk patients

Practices described by key informants used to support the identification function of ZS included standard clinical decision support tools, implemented in the EHR, to support suicide risk screening followed by more comprehensive risk assessment. At the time of this evaluation, all health care systems monitor depression symptom severity, including suicidal ideation, among adult patients receiving care in the outpatient mental health specialty setting. All systems also had screening/assessment practices in place for substance use disorders, although the norms/conventions supporting these screening practices varied. Workflow variation across care settings was common and the health care systems had different screening and assessment approaches (e.g., varying criteria for deciding which patients received the PHQ-9). In the primary care setting, population-based screening and suicide risk assessment were less common; screening and assessment were more often done at the discretion of primary care providers and many systems were engaged in planning future implementation of different approaches for more universal screening/assessment. Some systems were also in the process of planning implementation of suicide risk prediction algorithms (Simon et al., 2018) for purposes of enhancing suicide risk assessment and engagement in suicide-related care.

Engagement and care management

Practices designed to support the engagement function of the ZS model consistently included references to development of a safety plan or crisis response plan (Bryan et al., 2018; Stanley & Brown, 2012), which usually included lethal means assessment (i.e., discussions about prescription medications, firearms, and planning how to limit access). Researchers cataloged varying safety planning workflows and documentation practices across sites and providers (Yarborough et al., 2019); however, a shift toward practice standardization was underway. CMI was facilitating the standardization of safety planning by promoting use of a common EHR-based template across health care systems (i.e., KP regions). Key informants also described integrating social support, referral practices, and care coordination programs to engage patients in care for suicidality. The norms/conventions supporting these practices, however, were often not well-defined.

Effective treatment

Embedded health system researchers documented a variety of practices across health care systems that addressed the treatment function of ZS. These included pharmacotherapy; different types of evidence-based psychotherapy available for depression and other mental health disorders, including psychoeducational group-based therapy and cognitive behavioral therapy (CBT) (Hofmann et al., 2012), electroconvulsive therapy (Pagnin et al., 2004; UK ECT Review Group, 2003); and psychiatric hospitalization, which is the current standard of care for patients at high risk of suicide (Brown & Jager-Hyman, 2014). Researchers also cataloged evidence-based psychotherapies being used in some health care systems that were designed specifically for patients at high risk of suicide, including CBT for suicide prevention (CBT-SP) (Stanley et al., 2009) and dialectical behavior therapy (DBT) (Linehan et al., 2006). Generally, mental health specialty providers in all systems were responsible for treatment of patients at risk of suicide, but the specific treatments providers were using were unknown in some health care systems and difficult to discern in others (e.g., differentiating CBT from CBT-SP, and DBT groups from other group-based therapies). Moreover, it was not possible to know exactly what parts of specific therapies patients were receiving (e.g., specific DBT/CBT skills/strategies) (Bryan, 2019; Koerner, 2013).

Supportive care transitions

The practice consistently documented across health care systems designed to support the transition function of ZS was follow-up care after discharge from inpatient psychiatric settings. However, follow-up practices after discharge from the emergency department (ED) setting (without inpatient admission) were varied. Other practices designed to support care transitions included intensive case management programs typically designed to support patients with severe and persistent mental illnesses in outpatient settings (Dieterich et al., 2017). Finally, key informants at two health care systems reported sending caring messages (Motto, 1976; Motto & Bostrom, 2001) following inpatient discharge which addressed the ZS transition function, but the norms/conventions supporting this practices were not well-defined.

Reflexive monitoring

The data from key informants were also used to conceptualize how different practices and combinations of practices could be measured over time to support development of standard process measures and evaluation of ZS effectiveness (Figure 1). First, we summarized the common screening tools/assessments and health care utilization codes that could be extracted from EHRs across systems (Table 4). Specifically, at the time of this evaluation all health care systems were using the PHQ-9 to measure depression severity (Kroenke et al., 2001) and most participating health care systems also implemented a standardized tool for suicide risk assessment (most commonly the C-SSRS) (Madan et al., 2016; Posner et al., 2011) in their mental health and addiction medicine settings. No systems were using the same tools for drug use disorder screening/assessment, but several systems were using a common screening tool for alcohol use disorders (AUDIT-C) (Bradley et al., 2007; Bush et al., 1998). All systems were also in various stages of implementing new EHR tools designed to support consistent documentation of the key components of a safety plan (Bottomley, 2019; Stanley & Brown, 2012). Providers in all systems also had routine procedures for documenting treatment in patient EHRs via diagnosis and procedure codes (i.e., ICD and CPT) associated with health care encounters (CMS.gov Centers for Medicare & Medicaid Services, 2019; World Health Organization, 1992). Finally, follow-up within 7-day and 30-day post-discharge from inpatient psychiatric settings was consistently measured across all health care systems in accordance with national quality metrics measuring (National Committee for Quality Assurance [NCQA], 2019).

Table 4.

ZS practices measured with common tools across health care systems.

ZS function	Practice	Healthcare system						Measured with common tools across systems^a
ZS function	Practice	1	2	3	4	5	6	No	Yes	How?
Identification	Depression/Suicide Risk Screening	X	X	X	X	X	X		X	PHQ-2, PHQ-9Q9
	Depression Severity Assessment	X	X	X	X	X	X		X	PHQ-9
	Suicide Risk Assessment	X	X	X	X	X	X		X	C-SSRS
	Suicide Risk Prediction Algorithm						X	X
	Substance Use Screening/Assessment	X	X	X	X	X	X		X	AUDIT-C
Engagement	Collaborative Safety Planning	X	X	X	X	X	X		X	^bSafety Plan
	Social Support Programs		X					X
	Referrals/“Warm” hand-offs	X	X	X	X	X	X	X
	Care Coordination/Outreach Programs	X	X	X	X	X	X	X
	Care Management Programs	X	X	X	X	X	X	X
Treatment	Safety Plan updates/monitoring	X	X	X	X	X	X		X	^bSafety Plan
	Psychotherapy	X	X	X	X	X	X		X	ICD-10
	Pharmacotherapy	X	X	X	X	X	X		X	NDC Code
	Electroconvulsive Therapy	X	X	X	X	X	X		X	CPT Code
	Psychiatric Hospitalization	X	X	X	X	X	X		X	ICD-10
Transition	Follow-up Post Hospitalization	X	X	X	X	X	X		X	FUH HEDIS Measure
	Follow-up Post ED Discharge	X	X	X	X	X	X		X	^cEncounter
	Caring Message		X		X			X
	Intensive Case Management (Out-patient)	X		X		X		X

ZS: Zero Suicide; PHQ: Patient Health Questionnaire; C-SSRS: Columbia Suicide Severity Rating Scale; AUDIT-C: Alcohol Use Identification Test Consumption; ICD-10: 10th revision of the International Statistical Classification of Diseases; NDC: National Drug Code; CPT: Current Procedural Terminology; FUH: Follow-Up After Hospitalization for Mental Illness; HEDIS: Healthcare Effectiveness Data and Information Set; ED: emergency department.

Defined as more than 1 health care system presently or in the process of being implemented.

Discrete data element(s) implemented to capture EHR documentation.

Healthcare encounter (in-person, phone, etc.) EHR documentation.

Next, we defined a conceptual care continuum (i.e., treatment cascade) (Figure 3) to measure combinations of measurable practices supporting the ZS model. In this continuum, primary care patients enter the ZS care pathway at the point of population-based depression screening. Patients receiving mental health specialty care enter the care pathway via routine depression and/or suicide risk severity assessment. Patients seen in emergency or urgent care settings for mental health-related concerns also enter at the point of depression/suicide risk severity assessment. Patients identified as being at risk of suicide via screening/assessment may be engaged in collaborative safety planning and receive one or more types of treatment (e.g., pharmacotherapy, psychotherapy). Finally, patients who received care for suicidality in an inpatient or ED setting receive proactive outreach following discharge.

Figure 3.

Suicide risk care continuum.

Discussion

This evaluation used NPT (McEvoy et al., 2014; Nilsen, 2015) to build a fundamental understanding of how six health care systems had implemented ZS practices to support four key clinical functions of high-quality care for patients at risk of suicide, and lays the foundation for health systems nationwide to understand the impact of implementing these practices. Our innovative application of NPT to data from key informants enabled us to comprehensively catalog practices understood to support ZS (coherence); summarize the current state of norms/conventions supporting these practices (cognitive participation) and how health care teams performed these practices (collective action); and define how we know ZS practices are happening and how to measure when they occur (reflexive monitoring). Embedded researchers who all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems strengthened data collection and interpretation. The most well-defined practices focused on the identification function—specifically, the use of common screening/assessment tools (Kroenke et al., 2001; Posner et al., 2011) to identify patients at high risk of suicide attempt. Many potential treatment practice options were also defined, but few treatment practices identified were designed specifically for patients at risk of suicide and it was difficult to differentiate whether and how providers were using various therapy options. In addition, engagement and transition practices were often inconsistently used and measured within health care settings; but health care systems were consistently measuring health care encounters documentation following psychiatric hospitalization (NCQA, 2019) and a shift toward standardized safety planning was underway (Bruschke & Flores, 2020; Stanley & Brown, 2012).

This working knowledge of ZS practice variation will be used to support efforts to continue to improve suicide care across health care systems nationwide (Bruschke & Flores, 2020) and large-scale evaluation of the effectiveness of ZS practices, both alone and as part of a continuum of care for patients at risk of suicide. Specifically, defining an approach for measuring different practices (and combinations of practices) over time via development of standard process metrics (i.e., reflexive monitoring) have enabled participating health care systems, and will enable others nationwide, to undertake their own appraisal processes in support of ZS implementation. CMI worked in collaboration with mental health leaders to develop a standardized set of metrics to track implementation of ZS practices and suicide prevention outcomes (Bruschke & Flores, 2020). These metrics will enable clinical, operational, and patient safety stakeholders to measure whether identification and engagement practices are occurring when expected (i.e., continuous quality improvement) within their own health care systems. The working knowledge of the variation of ZS practice implementation across health care systems will also allow researchers to compare the effectiveness of different approaches (i.e., workflows) used for identification, engagement, and transitioning patients at high risk of suicide. This evaluation will also support treatment metric development; however, as this work demonstrated, the investigation of the real-world effectiveness of specific psychotherapies for suicide prevention may be limited until we are able to more accurately define ways to measure variation in these treatment practices.

Future directions

This project can provide a framework to support ongoing ZS implementation and evaluation in health care systems across the country. Specifically, this evaluation suggests important considerations for integrating or normalizing new practices to support the ZS model. For example, as suicide risk prediction algorithms (Kessler et al., 2017; Simon et al., 2018) are implemented to improve identification and engagement of patients at risk (U.S. Department of Veterans Affairs, 2017), it will be important to consider how to support this practice within the context of existing norms/conventions, workflows, and tools. This evaluation also underscored the potential utility of using standardized definitions and tools to measure existing engagement and transition practices, such as care coordination/outreach and caring message programs. Caring contact programs in particular have potential to be a simple and effective ZS practice, scalable across different types of health care systems (Carter et al., 2005, 2007; Comtois et al., 2019; Hassanian-Moghaddam et al., 2011; Luxton et al., 2012; Motto, 1976; Motto & Bostrom, 2001).

The work presented here will also provide a vital foundation for understanding potential adaptation of practices supporting ZS during environmental disruption and evaluating those changes. For example, in response to the coronavirus disease 2019 (COVID-19) pandemic many US health care organizations rapidly converted to providing many services, including mental health and addiction medicine care, virtually (Wosik et al., 2020). The processes used to identify patients at risk of suicide via virtual screening/assessment practices may be different than prior in-person practices. Moreover, reduced reliance on emergency and inpatient psychiatric settings during the COVID-19 response may also impact suicide rates and our understanding of the effectiveness of mental health care provided in these settings (Loch, 2014; Rabinowitz et al., 1994). When the effects of COVID-19 on suicide become clearer (Reger et al., 2020), understanding how ZS implementation adapted and changed during this time across these health care organizations will be vital for understanding which practices may have helped mitigate potential unintended negative effects of social isolation on suicide outcomes (Ahmed et al., 2020; Fatke et al., 2020; Vieira et al., 2020).

Limitations

This evaluation did not explicitly address the implementation strategy domains of the ZS model—training, leadership, and continuous quality improvement (Education Development Center, 2020). These domains are critical to ensuring that practices supporting the four key clinical functions of ZS we evaluated result in high-quality suicide care. For example, providing training in specific evidence-based interventions provides clinicians with valuable skills and enhances fidelity to the intervention. The presence of a completed safety plan in an EHR does not necessarily mean that the safety plan was completed collaboratively in a way that was meaningful to the patient. Training, leadership, and continuous improvement are implementation strategies critical for ensuring consistent delivery of high-quality care and will be important considerations for the future evaluation of the effectiveness of different ZS practices (and combinations of practices).

This evaluation also did not take variations in practice maturity into account or consider all the ways that clinical ZS practices interact with related interventions in health care systems. For example, screening and assessment for substance use disorders was identified as a ZS identification practice, which makes sense given the strong association between suicide and substance use disorders (Espinet et al., 2019; Wilcox et al., 2004), particularly alcohol use disorders (Bagge et al., 2013; Caetano et al., 2013; Cherpitel et al., 2004; Lejoyeux et al., 2008; Powell et al., 2001; Richards et al., 2020b). However, we did not consider whether and how to measure engagement in care for alcohol and drug use disorders (NCQA, 2020), which could also be useful mechanism for improving suicide-related care (Richards et al., 2020a, 2020b) or addressing suicidal ideation and risky patterns of alcohol use together for purposes of suicide prevention (Kalk et al., 2019).

Last, health care systems participating in this evaluation provide both integrated, comprehensive health care and insurance coverage to a defined member/patient population. This enabled us to define a conceptual care continuum for patients at risk of suicide across care settings (e.g., primary care/mental health specialty, inpatient/outpatient), but this care continuum may not be generalizable to organizations without the same responsibility to provide comprehensive health care to a defined population of members/patients across care settings. Health care organizations may also not have embedded researchers to assist with monitoring ZS implementation progress. However, many organizations have stakeholders invested in suicide prevention (like the clinical, operational, and patient stakeholders who participate in the suicide prevention collaborative within our health care systems) who can utilize the products of this evaluation (e.g., practice descriptions, measurement tools) to inform their own quality improvement processes.

Conclusion

This novel evaluation used NPT in combination with data from key informants to improve our understanding of clinical practices supporting the ZS model and inform recommendations for implementation of high-quality care for patients at risk of suicide in health care systems nationwide. Specifically, NPT was applied to create a catalog and description of practices supporting the four key clinical functions of ZS and model how to measure specific practices. These schemas will be used to inform a large-scale evaluation of the effectiveness of different ZS practices (and combinations of practices) and provide a blueprint to support suicide prevention practice implementation across health care systems, service settings, and patient populations.

Supplemental Material

sj-pdf-1-irp-10.1177_26334895211011769 – Supplemental material for An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide

Supplemental material, sj-pdf-1-irp-10.1177_26334895211011769 for An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide by Julie E Richards, Gregory E Simon, Jennifer M Boggs, Rinad Beidas, Bobbi Jo H Yarborough, Karen J Coleman, Stacy A Sterling, Arne Beck, Jean P Flores, Cambria Bruschke, Julie Goldstein Grumet, Christine C Stewart, Michael Schoenbaum, Joslyn Westphal and Brian K Ahmedani in Implementation Research and Practice

Supplemental Material

sj-pdf-2-irp-10.1177_26334895211011769 – Supplemental material for An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide

Supplemental material, sj-pdf-2-irp-10.1177_26334895211011769 for An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide by Julie E Richards, Gregory E Simon, Jennifer M Boggs, Rinad Beidas, Bobbi Jo H Yarborough, Karen J Coleman, Stacy A Sterling, Arne Beck, Jean P Flores, Cambria Bruschke, Julie Goldstein Grumet, Christine C Stewart, Michael Schoenbaum, Joslyn Westphal and Brian K Ahmedani in Implementation Research and Practice

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors JER, GES, JMB, BHY, KJC, SAS, AB, JPF, CB, CCS, MS, JW, and BKA report no conflicts of interest. Author RB receives royalties from Oxford University Press and has provided consulting for the Camden Coalition of Health care Providers. She is an Associate Editor at Implementation Research & Practice. Author JGG is the Director of the Zero Suicide Institute. The Zero Suicide Institute provides consultation and training on the Zero Suicide framework.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This evaluation was supported by the National Institute of Mental Health U01 MH114087 and U19 MH092201. Portions of these findings were presented as a poster at the 2019 12th Annual Conference on the Science of Dissemination and Implementation in Health, Arlington, VA, United States.

ORCID iD

Julie E Richards

Supplemental material

Supplemental material for this article is available online.

References

Ahmed

M. Z.

Ahmed

Aibao

Hanbin

Siyu

Ahmad

(2020). Epidemic of COVID-19 in China and associated psychological problems. Asian Journal of Psychiatry, 51, 102092. https://doi.org/10.1016/j.ajp.2020.102092

Ahmedani

B. K.

Simon

G. E.

Stewart

Beck

Waitzfelder

B. E.

Rossom

Lynch

Owen-Smith

Hunkeler

E. M.

Whiteside

Operskalski

B. H.

Coffey

M. J.

Solberg

L. I.

(2014). Health care contacts in the year before suicide death. Journal of General Internal Medicine, 29(6), 870–877. https://doi.org/10.1007/s11606-014-2767-3

Ahmedani

B. K.

Vannoy

(2014). National pathways for suicide prevention and health services research. American Journal of Preventive Medicine, 47(3, Suppl. 2), S222–S228. https://doi.org/10.1016/j.amepre.2014.05.038

Ahmedani

B. K.

Westphal

Autio

Elsiss

Peterson

E. L.

Beck

Waitzfelder

B. E.

Rossom

R. C.

Owen-Smith

A. A.

Lynch

C. Y.

Frank

Prabhakar

Braciszewski

J. M.

Miller-Matero

L. R.

Yeh

H. H.

Doshi

Waring

S. C.

Simon

G. E.

(2019). Variation in patterns of health care before suicide: A population case-control study. Preventive Medicine, 127, 105796. https://doi.org/10.1016/j.ypmed.2019.105796

Attride-Stirling

(2001). Thematic networks: An analytic tool for qualitative research. Qualitative Research, 1(3), 385–405.

Bagge

C. L.

Lee

H. J.

Schumacher

J. A.

Gratz

K. L.

Krull

J. L.

Holloman

Jr. (2013). Alcohol as an acute risk factor for recent suicide attempts: A case-crossover analysis. Journal of Studies on Alcohol and Drugs, 74(4), 552–558. http://www.ncbi.nlm.nih.gov/pubmed/23739018

Bottomley

(2019). Identifying suicide on the front lines of health. Permanente Medicine. https://permanente.org/identifying-suicide-on-the-front-lines-of-health/

Bradley

K. A.

DeBenedetti

A. F.

Volk

R. J.

Williams

E. C.

Frank

Kivlahan

D. R.

(2007). AUDIT-C as a brief screen for alcohol misuse in primary care. Alcoholism: Clinical and Experimental Research, 31(7), 1208–1217. https://doi.org/10.1111/j.1530-0277.2007.00403.x

Brodsky

B. S.

Spruch-Feiner

Stanley

(2018). The Zero Suicide model: Applying evidence-based suicide prevention practices to clinical care. Frontiers in Psychiatry, 9, Article 33. https://doi.org/10.3389/fpsyt.2018.00033

10.

Brown

G. K.

Jager-Hyman

(2014). Evidence-based psychotherapies for suicide prevention: Future directions. American Journal of Preventive Medicine, 47(3, Suppl. 2), S186–S194. https://doi.org/10.1016/j.amepre.2014.06.008

11.

Bruschke

Flores

(2020). Zero Suicide research: Kaiser Permanente. https://zerosuicide.edc.org/evidence/outcome-story/kaiser-permanente

12.

Bryan

C. J.

(2019). Cognitive behavioral therapy for suicide prevention (CBT-SP): Implications for meeting standard of care expectations with suicidal patients. Behavioral Sciences & the Law, 37(3), 247–258. https://doi.org/10.1002/bsl.2411

13.

Bryan

C. J.

May

A. M.

Rozek

D. C.

Williams

S. R.

Clemans

T. A.

Mintz

Leeson

Burch

T. S.

(2018). Use of crisis management interventions among suicidal patients: Results of a randomized controlled trial. Depression and Anxiety, 35(7), 619–628. https://doi.org/10.1002/da.22753

14.

Bush

Kivlahan

D. R.

McDonell

M. B.

Fihn

S. D.

Bradley

K. A.

(1998). The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol use disorders identification test. Archives of Internal Medicine, 158(16), 1789–1795. http://www.ncbi.nlm.nih.gov/pubmed/9738608

15.

Caetano

Kaplan

M. S.

Huguet

McFarland

B. H.

Conner

Giesbrecht

Nolte

K. B.

(2013). Acute alcohol intoxication and suicide among United States ethnic/racial groups: Findings from the national violent death reporting system. Alcoholism: Clinical and Experimental Research, 37(5), 839–846. https://doi.org/10.1111/acer.12038

16.

Caine

E. D.

(2013). Forging an agenda for suicide prevention in the United States. American Journal of Public Health, 103(5), 822–829. https://doi.org/10.2105/AJPH.2012.301078

17.

Carter

G. L.

Clover

Whyte

I. M.

Dawson

A. H.

D’Este

(2005). Postcards from the EDge project: Randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning. British Medical Journal, 331(7520), 805. https://doi.org/bmj.38579.455266.E0

18.

Carter

G. L.

Clover

Whyte

I. M.

Dawson

A. H.

D’Este

(2007). Postcards from the EDge: 24-month outcomes of a randomised controlled trial for hospital-treated self-poisoning. The British Journal of Psychiatry, 191, 548–553. https://doi.org/10.1192/bjp.bp.107.038406

19.

Cherpitel

C. J.

Borges

G. L.

Wilcox

H. C.

(2004). Acute alcohol use and suicidal behavior: A review of the literature. Alcoholism: Clinical and Experimental Research, 28(Suppl. 5), 18S–28S. https://doi.org/00000374-200405001-00004

20.

Chung

D. T.

Ryan

C. J.

Hadzi-Pavlovic

Singh

S. P.

Stanton

Large

M. M.

(2017). Suicide rates after discharge from psychiatric facilities: A systematic review and meta-analysis. JAMA Psychiatry, 74(7), 694–702. https://doi.org/10.1001/jamapsychiatry.2017.1044

21.

CMS.gov Centers for Medicare & Medicaid Services. (2019). ICD-10-CM, ICD-10-PCS, CPT, and HCPCS code sets. https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/MLN-Publications-Items/ICN900943.html

22.

Coffey

C. E.

Coffey

M. J.

Ahmedani

B. K.

(2013). An update on perfect depression care. Psychiatric Services, 64(4), 396. https://doi.org/10.1176/appi.PS.640422

23.

Coffey

M. J.

Coffey

C. E.

Ahmedani

B. K.

(2015). Suicide in a health maintenance organization population. JAMA Psychiatry, 72(3), 294–296. https://doi.org/10.1001/jamapsychiatry.2014.2440

24.

Comtois

K. A.

Kerbrat

A. H.

DeCou

C. R.

Atkins

D. C.

Majeres

J. J.

Baker

J. C.

Ries

R. K.

(2019). Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention: A randomized clinical trial. JAMA Psychiatry, 76(5), 474–483. https://doi.org/10.1001/jamapsychiatry.2018.4530

25.

Dieterich

Irving

Bergman

Khokhar

Park

Marshall

(2017). Intensive case management for severe mental illness. Cochrane Database of Systematic Reviews, 1, CD007906.

26.

Education Development Center. (2020). Zero Suicide. http://zerosuicide.edc.org/

27.

Espinet

Corrin

Baliunas

Quilty

Zawertailo

Rizvi

S. J.

deRuiter

Bonato

De Luca

Kennedy

Selby

(2019). Predisposing and protective factors influencing suicide ideation, attempt, and death in patients accessing substance use treatment: A systematic review and meta-analysis protocol. Systematic Reviews, 8(1), 115. https://doi.org/10.1186/s13643-019-1028-2

28.

Fatke

Holzle

Frank

Forstl

(2020). Psychische Probleme in der Pandemie—Beobachtungen wahrend der COVID-19-Krise [COVID-19 crisis: Early observations on a pandemic’s psychiatric problems]. Deutsche Medizinische Wochenschrift, 145(10), 675–681. https://doi.org/10.1055/a-1147-2889

29.

Franx

Oud

de Lange

Wensing

Grol

(2012). Implementing a stepped-care approach in primary care: Results of a qualitative study. Implementation Science, 7, 8. https://doi.org/10.1186/1748-5908-7-8

30.

Gunn

J. M.

Palmer

V. J.

Dowrick

C. F.

Herrman

H. E.

Griffiths

F. E.

Kokanovic

Blashki

G. A.

Hegarty

K. L.

Johnson

C. L.

Potiriadis

May

C. R

. (2010). Embedding effective depression care: Using theory for primary care organisational and systems change. Implementation Science, 5, 62. https://doi.org/10.1186/1748-5908-5-62

31.

Hampton

(2010). Depression care effort brings dramatic drop in large HMO population’s suicide rate. Journal of the American Medical Association, 303(19), 1903–1905. https://doi.org/10.1001/jama.2010.595

32.

Hassanian-Moghaddam

Sarjami

Kolahi

A. A.

Carter

G. L.

(2011). Postcards in Persia: Randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-poisoning. The British Journal of Psychiatry, 198(4), 309–316. https://doi.org/10.1192/bjp.bp.109.067199

33.

Health Care Systems Research Network. (2019). Who we are. http://www.hcsrn.org/en/About/

34.

Henry Ford Health System. (2019). 2019 fact sheet. https://www.henryford.com/newsroom/facts#numbers

35.

Hofmann

S. G.

Asnaani

Vonk

I. J.

Sawyer

A. T.

Fang

(2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36(5), 427–440. https://doi.org/10.1007/s10608-012-9476-1

36.

Hogan

M. F.

Grumet

J. G.

(2016). Suicide prevention: An emerging priority for health care. Health Affairs, 35(6), 1084–1090. https://doi.org/10.1377/hlthaff.2015.1672

37.

Jolles

M. P.

Lengnick-Hall

Mittman

B. S.

(2019). Correction to: Core functions and forms of complex health interventions: A patient-centered medical home illustration. Journal of General Internal Medicine, 34(6), 1032–1038. https://doi.org/10.1007/s11606-019-04885-z

38.

Kaiser Permanente. (2019). Kaiser Permanente fast facts. https://about.kaiserpermanente.org/who-we-are/fast-facts

39.

Kalk

N. J.

Kelleher

M. J.

Curtis

Morley

K. I.

(2019). Addressing substance misuse: A missed opportunity in suicide prevention. Addiction, 114(3), 387–388. https://doi.org/10.1111/add.14463

40.

Kay

E. S.

Batey

D. S.

Mugavero

M. J.

(2016). The HIV treatment cascade and care continuum: Updates, goals, and recommendations for the future. AIDS Research and Therapy, 13, 35. https://doi.org/10.1186/s12981-016-0120-0

41.

Kessler

R. C.

Hwang

Hoffmire

C. A.

McCarthy

J. F.

Petukhova

M. V.

Rosellini

A. J.

Sampson

N. A.

Schneider

A. L.

Bradley

P. A.

Katz

I. R.

Thompson

Bossarte

R. M.

(2017). Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration. International Journal of Methods in Psychiatric Research, 26(3), Article e1575. https://doi.org/10.1002/mpr.1575

42.

Knox

K. L.

Litts

D. A.

Talcott

G. W.

Feig

J. C.

Caine

E. D.

(2003). Risk of suicide and related adverse outcomes after exposure to a suicide prevention programme in the US Air Force: Cohort study. British Medical Journal, 327(7428), 1376. https://doi.org/10.1136/bmj.327.7428.1376

43.

Koerner

(2013). What must you know and do to get good outcomes with DBT? Behavior Therapy, 44(4), 568–579. https://doi.org/10.1016/j.beth.2013.03.005

44.

Kroenke

Spitzer

R. L.

Williams

J. B.

(2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

45.

Lejoyeux

Huet

Claudon

Fichelle

Casalino

Lequen

(2008). Characteristics of suicide attempts preceded by alcohol consumption. Archives of Suicide Research, 12(1), 30–38. https://doi.org/10.1080/13811110701800699

46.

Linehan

M. M.

Comtois

K. A.

Murray

A. M.

Brown

M. Z.

Gallop

R. J.

Heard

H. L.

Korslund

K. E.

Tutek

D. A.

Reynolds

S. K.

Lindenboim

(2006). Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Archives of General Psychiatry, 63(7), 757–766. https://doi.org/10.1001/archpsyc.63.7.757

47.

Loch

A. A.

(2014). Discharged from a mental health admission ward: Is it safe to go home? A review on the negative outcomes of psychiatric hospitalization. Psychology Research and Behavior Management, 7, 137–145. https://doi.org/10.2147/PRBM.S35061

48.

Louzon

S. A.

Bossarte

McCarthy

J. F.

Katz

I. R.

(2016). Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients? Psychiatric Services, 67(5), 517–522. https://doi.org/10.1176/appi.ps.201500149

49.

Luoma

J. B.

Martin

C. E.

Pearson

J. L.

(2002). Contact with mental health and primary care providers before suicide: A review of the evidence. American Journal of Psychiatry, 159(6), 909–916. https://doi.org/10.1176/appi.ajp.159.6.909

50.

Luxton

D. D.

Kinn

J. T.

June

J. D.

Pierre

L. W.

Reger

M. A.

Gahm

G. A.

(2012). Caring Letters Project: A military suicide-prevention pilot program. Crisis, 33(1), 5–12. https://doi.org/10.1027/0227-5910/a000093

51.

Madan

Frueh

B. C.

Allen

J. G.

Ellis

T. E.

Rufino

K. A.

Oldham

J. M.

Fowler

J. C.

(2016). Psychometric reevaluation of the Columbia-Suicide Severity Rating Scale: Findings from a prospective, inpatient cohort of severely mentally ill adults. The Journal of Clinical Psychiatry, 77(7), e867–e873. https://doi.org/10.4088/JCP.15m10069

52.

Martin

Swannell

Milner

Gullestrup

(2016). Mates in construction suicide prevention program: A five year review. Journal of Community Medicine & Health Education, 6, 465.

53.

May

C. R.

Finch

(2009). Implementing, embedding, and integrating practices: An outline of normalization process theory. Sociology, 43(3), 535–554.

54.

May

C. R.

Mair

Finch

MacFarlane

Dowrick

Treweek

Rapley

Ballini

Ong

B. N.

Rogers

Murray

Elwyn

Legare

Gunn

Montori

V. M.

(2009). Development of a theory of implementation and integration: Normalization process theory. Implementation Science, 4, 29. https://doi.org/10.1186/1748-5908-4-29

55.

McEvoy

Ballini

Maltoni

O’Donnell

C. A.

Mair

F. S.

Macfarlane

(2014). A qualitative systematic review of studies using the normalization process theory to research implementation processes. Implementation Science, 9, 2. https://doi.org/10.1186/1748-5908-9-2

56.

Miles

M. B.

Huberman

A. M.

Saldana

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). SAGE.

57.

Motto

J. A.

(1976). Suicide prevention for high-risk persons who refuse treatment. Suicide and Life-Threatening Behavior, 6(4), 223–230. https://www.ncbi.nlm.nih.gov/pubmed/1023455

58.

Motto

J. A.

Bostrom

A. G.

(2001). A randomized controlled trial of postcrisis suicide prevention. Psychiatric Services, 52(6), 828–833. https://doi.org/10.1176/appi.ps.52.6.828

59.

National Committee for Quality Assurance. (2019). Follow-up after hospitalization for mental illness (FUH). https://www.ncqa.org/hedis/measures/follow-up-after-hospitalization-for-mental-illness/

60.

National Committee for Quality Assurance. (2020). Initiation and engagement of alcohol and other drug abuse or dependence treatment (IET). https://www.ncqa.org/hedis/measures/initiation-and-engagement-of-alcohol-and-other-drug-abuse-or-dependence-treatment/

61.

Nilsen

(2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53. https://doi.org/10.1186/s13012-015-0242-0

62.

Nock

M. K.

Borges

Bromet

E. J.

Cha

C. B.

Kessler

R. C.

Lee

(2008). Suicide and suicidal behavior. Epidemiologic Reviews, 30, 133–154. https://doi.org/10.1093/epirev/mxn002

63.

Pagnin

de Queiroz

Pini

Cassano

G. B.

(2004). Efficacy of ECT in depression: A meta-analytic review. The Journal of ECT, 20(1), 13–20. https://doi.org/10.1097/00124509-200403000-00004

64.

Posner

Brown

G. K.

Stanley

Brent

D. A.

Yershova

K. V.

Oquendo

M. A.

Currier

G. W.

Melvin

G. A.

Greenhill

Shen

Mann

J. J.

(2011). The Columbia-Suicide Severity Rating Scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults. American Journal of Psychiatry, 168(12), 1266–1277. https://doi.org/10.1176/appi.ajp.2011.10111704

65.

Powell

K. E.

Kresnow

M. J.

Mercy

J. A.

Potter

L. B.

Swann

A. C.

Frankowski

R. F.

Lee

R. K.

Bayer

T. L.

(2001). Alcohol consumption and nearly lethal suicide attempts. Suicide and Life-Threatening Behavior, 32(Suppl. 1), 30–41. http://www.ncbi.nlm.nih.gov/pubmed/11924693

66.

Rabinowitz

Modai

Inbar-Saban

(1994). Understanding who improves after psychiatric hospitalization. Acta Psychiatrica Scandinavica, 89(3), 152–158. https://doi.org/10.1111/j.1600-0447.1994.tb08085.x

67.

Reger

M. A.

Stanley

I. H.

Joiner

T. E.

(2020). Suicide mortality and coronavirus disease 2019—A perfect storm? JAMA Psychiatry, 77(11), 1093–1094. https://doi.org/10.1001/jamapsychiatry.2020.1060

68.

Rehkopf

D. H.

Buka

S. L.

(2006). The association between suicide and the socio-economic characteristics of geographical areas: A systematic review. Psychological Medicine, 36(2), 145–157. https://doi.org/10.1017/S003329170500588X

69.

Richards

J. E.

Shortreed

S. M.

Simon

G. E.

Penfold

R. B.

Glass

J. E.

Ziebell

Williams

E. C.

(2020a). Association between patterns of alcohol use and short-term risk of suicide attempt among patients with and without reported suicidal ideation. Journal of Addiction Medicine, 14(5), e160–e169. https://doi.org/10.1097/ADM.0000000000000637

70.

Richards

J. E.

Shortreed

S. M.

Simon

G. E.

Penfold

R. B.

Glass

J. E.

Ziebell

Williams

E. C.

(2020b). Short-term risk of suicide attempt associated with patterns of patient-reported alcohol use determined by routine AUDIT-C among adults receiving mental healthcare. General Hospital Psychiatry, 62, 79–86. https://doi.org/10.1016/j.genhosppsych.2019.12.002

71.

Simon

G. E.

Coleman

K. J.

Rossom

R. C.

Beck

Oliver

Johnson

Whiteside

Operskalski

Penfold

R. B.

Shortreed

S. M.

Rutter

(2016). Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. The Journal of Clinical Psychiatry, 77(2), 221–227. https://doi.org/10.4088/JCP.15m09776

72.

Simon

G. E.

Johnson

Lawrence

J. M.

Rossom

R. C.

Ahmedani

Lynch

F. L.

Beck

Waitzfelder

Ziebell

Penfold

R. B.

Shortreed

S. M.

(2018). Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. American Journal of Psychiatry, 175(10), 951–960. https://doi.org/10.1176/appi.ajp.2018.17101167

73.

Simon

G. E.

Rutter

C. M.

Peterson

Oliver

Whiteside

Operskalski

Ludman

E. J.

(2013). Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatric Services, 64(12), 1195–1202. https://doi.org/10.1176/appi.ps.201200587

74.

Stanley

Brown

Brent

D. A.

Wells

Poling

Curry

Kennard

B. D.

Wagner

Cwik

M. F.

Klomek

A. B.

Goldstein

Vitiello

Barnett

Daniel

Hughes

(2009). Cognitive-behavioral therapy for suicide prevention (CBT-SP): Treatment model, feasibility, and acceptability. Journal of the American Academy of Child & Adolescent Psychiatry, 48(10), 1005–1013. https://doi.org/10.1097/CHI.0b013e3181b5dbfe

75.

Stanley

Brown

G. K.

(2012). Safety planning intervention: A brief intervention to mitigate suicide risk. Cognitive and Behavioral Practice, 19(2), 256–264.

76.

UK ECT Review Group. (2003). Efficacy and safety of electroconvulsive therapy in depressive disorders: A systematic review and meta-analysis. The Lancet, 361(9360), 799–808. https://doi.org/10.1016/S0140-6736(03)12705-5

77.

U.S. Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention. (2012). 2012 national strategy for suicide prevention: Goals and objectives for action.

78.

U.S. Department of Veterans Affairs. (2017). VA REACH VET initiative helps save veterans lives: Program signals when more help is needed for at-risk Veterans. https://www.va.gov/opa/pressrel/pressrelease.cfm?id=2878

79.

Vieira

P. R.

Garcia

L. P.

Maciel

E. L. N.

(2020). Isolamento social e o aumento da violencia domestica: o que isso nos revela? [The increase in domestic violence during the social isolation: What does it reveals?]. Revista Brasileira de Epidemiologia, 23, Article e200033. https://doi.org/10.1590/1980-549720200033

80.

While

Bickley

Roscoe

Windfuhr

Rahman

Shaw

Appleby

Kapur

(2012). Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: A cross-sectional and before-and-after observational study. The Lancet, 379(9820), 1005–1012. https://doi.org/10.1016/S0140-6736(11)61712-1

81.

Wilcox

H. C.

Conner

K. R.

Caine

E. D.

(2004). Association of alcohol and drug use disorders and completed suicide: An empirical review of cohort studies. Drug and Alcohol Dependence, 76(Suppl.), S11–S19. https://doi.org/10.1016/j.drugalcdep.2004.08.003

82.

World Health Organization. (1992). The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines.

83.

Wosik

Fudim

Cameron

Gellad

Z. F.

Cho

Phinney

Curtis

Roman

Poon

E. G.

Ferranti

Katz

J. N.

Tcheng

(2020). Telehealth transformation: COVID-19 and the rise of virtual care. Journal of the American Medical Informatics Association, 27(6), 957–962. https://doi.org/10.1093/jamia/ocaa067

84.

Yarborough

B. J. H.

Ahmedani

B. K.

Boggs

J. M.

Beck

Coleman

K. J.

Sterling

Schoenbaum

Goldstein-Grumet

Simon

G. E.

(2019). Challenges of population-based measurement of suicide prevention activities across multiple health systems. eGEMS, 7(1), 13. https://doi.org/10.5334/egems.277

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB

0.09 MB