Abstract
Objective:
To determine whether the release of the first season of the Netflix series ‘13 Reasons Why’ was associated with changes in emergency department presentations for self-harm.
Methods:
Healthcare utilization databases were used to identify emergency department and outpatient presentations according to age and sex for residents of Ontario, Canada. Data from 2007 to 2018 were used in autoregressive integrated moving average models for time series forecasting with a pre-specified hypothesis that rates of emergency department presentations for self-harm would increase in the 3-month period following the release of 13 Reasons Why (1 April 2017 to 30 June 2017). Chi-square and t tests were used to identify demographic and health service use differences between those presenting to emergency department with self-harm during this epoch compared to a control period (1 April 2016 to 30 June 2016).
Results:
There was a significant estimated excess of 75 self-harm-related emergency department visits (+6.4%) in the 3 months after 13 Reasons Why above what was predicted by the autoregressive integrated moving average model (standard error = 32.4; p = 0.02); adolescents aged 10–19 years had 60 excess visits (standard error = 30.7; p = 0.048), whereas adults demonstrated no significant change. Sex-stratified analyses demonstrated that these findings were largely driven by significant increases in females. There were no differences in demographic or health service use characteristics between those who presented to emergency department with self-harm in April to June 2017 vs April to June 2016.
Conclusions:
This study demonstrated a significant increase in self-harm emergency department visits associated with the release of 13 Reasons Why. It adds to previously published mortality, survey and helpline data collectively demonstrating negative mental health outcomes associated with 13 Reasons Why.
Introduction
The first season of the Netflix show ‘13 Reasons Why’ (13RW), which included a lengthy depiction of the suicide of a teenager, Hannah Baker, was released on 31 March 2017 (Yorkey, 2017) and was widely criticized by mental health experts for violating numerous recommendations for responsible portrayals of suicide (Arendt et al., 2017; Bridge et al., 2020; Feuer and Havens, 2017; Hong et al., 2019; Niederkrotenthaler et al., 2019; Rosa et al., 2019; Sinyor et al., 2019). The final episode, which depicts the character’s suicide—she lies in a bathtub cutting her wrists with a razor blade and exsanguinates—was a focus of particular concern. Several studies have already investigated the impact of 13RW, Season 1, on suicide (Arendt et al., 2017; Ayers et al., 2017; Bridge et al., 2020; Feuer and Havens, 2017; Hong et al., 2019; Niederkrotenthaler et al., 2019; Rosa et al., 2019; Sinyor et al., 2019). Most importantly, the show was associated with increases in youth suicides of approximately 15% in the United States (Niederkrotenthaler et al., 2019) and 18% in Canada (Sinyor et al., 2019). These findings reinforce prior work demonstrating that 13RW was associated with an increase in Google searches such as ‘how to commit suicide’ (Ayers et al., 2017), online survey data showing worsening mood in one in four adolescents after they viewed the show (Rosa et al., 2019) and a survey of 87 adolescents presenting to a US emergency department (ED) in a suicidal crisis, of whom roughly one quarter believed that the show had increased their suicide risk (Hong et al., 2019).
The mortality data and those from clinical convenience/survey samples are already compelling and highly suggestive of harm to adolescents resulting from the release of 13RW. Self-harm (self-injurious behavior with or without suicidal intent) that does not result in death is, in itself, an important negative health outcome and is a key predictor of future suicide attempt and death (Mars et al., 2019). In the general population, there are an estimated 20 non-fatal suicide attempts for every one that results in death (Fazel and Runeson, 2020) and the ratio may be as high as 200 to 1 in adolescents (McKean et al., 2018). Furthermore, in Season 1, a secondary character in the show, Skye, is revealed to have cuts on her wrist and describes self-harm by cutting as ‘what you do instead of killing yourself’ (Yorkey, 2017; episode 13). Although not as well characterized as suicide contagion, depictions of self-harm do also appear to lead to copycat behavior (Jarvi et al., 2013; Khasawneh et al., 2020). Given that both the Hannah and Skye characters model self-harm to adolescents, we would expect to also see non-fatal imitative behavior at the population level. Despite this, there are very limited data on self-harm presentations to EDs following the release of Season 1 of 13RW.
The relative dearth of studies in this area represents an important gap in the literature examining the 13RW phenomenon, given that increases in such presentations, if observed, would represent highly relevant negative health outcomes that would reinforce previous findings on suicide deaths. One study did find an increase in suicide attempt/self-harm-related hospitalizations at a single children’s hospital in the United States (Cooper et al., 2018). Another preliminary analysis of ED presentations for self-harm in the United States after 13RW indicated a possible increase in such behavior (Feuer and Havens, 2017). This survey of 14 pediatric ED sites indicated that 95% reported an increase in visits and 40% observed ‘copycat’ gestures or attempts in the 30 days following release of the first season of 13RW (Feuer and Havens, 2017). However, this study examined only preliminary data from a sample of sites that responded to the survey and such effects have yet to be investigated systematically. The current study aims by examining changes in healthcare utilization in youth (adolescents: aged 10–19 years and young adults: aged 20–29 years) as well as a middle-aged adult comparator group (aged 30–45 years) in Ontario, Canada, in the months following the initial release of 13RW. The a priori primary hypothesis was that we would observe increases in adolescent ED visits for self-harm post-13RW, above and beyond any pre-existing trends, and that the magnitude of increases would be largest in adolescents (aged 10–19 years). A secondary objective was to examine rates of other health service use associated with the initial release of 13RW, including all adolescent mental health and addiction (MHA)-related ED visits and outpatient physician visits. Outpatient visits were included to ascertain whether the outpatient sector was observing (and responding to) any increase in demand observed in ED settings associated with the release of 13RW.
Methods
Study design
This study took place in Ontario, Canada. Ontario residents, like all Canadian residents, receive universal health coverage that covers most hospital and physician-provided services including all services considered medically necessary. Health administrative data generated from the delivery of universal healthcare are routinely collected in Ontario. This is a retrospective time series analysis measuring monthly ED visits for self-harm (primary outcome), all MHA-related ED visits, and all MHA-related outpatient physician visits for Ontario residents between 2007 and 2018. The age groups of greatest interest were adolescents aged 10–19 years, the target audience of the show. Young adults aged 20–29 years and those aged 30–45 years were also included for comparison, and to help characterize any potential influence of 13RW (or lack thereof) on adults who may have also been exposed to the show. Patients were excluded if they had an invalid health card number, were missing key demographic information (age or sex), or were not an Ontario resident for the duration of the study period. This study measured monthly ED visit rates; in general, approximately 9% of the initial sample was excluded in each month, with the most common reason being that the individual was not an Ontario resident (approximately 7.5% per month).
Data sources
We used the following data sources maintained at the Institute for Clinical Evaluative Sciences (ICES), a research institute in Toronto, Ontario, with access to Ontario health administrative data: the National Ambulatory Care Reporting System (NACRS), which contains information related to ED visits, including whether or not someone had a visit related to self-harm, as well as outpatient visits; the Ontario Health Insurance Plan (OHIP) Claims Database, which contains information related to physician visits; and the Registered Persons Database, which contains information on patient demographics (age, sex, neighborhood income from postal code and Census files). These datasets were linked using unique encoded identifiers and analyzed at ICES. The use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board (Government of Ontario, 2004).
Outcomes
Our primary outcome was self-harm-related ED visits. We used a validated method of identifying self-harm in ED data (Bethell and Rhodes, 2019). Self-harm was identified using the following International Classification of Diseases, 10th Revision (ICD-10) codes: X60-X84 (intentional self-injury by self-poisoning; hanging, strangulation and suffocation; drowning and submersion; firearm or explosive material; smoke, fire and flames; steam, hot vapors and hot objects; sharp object; blunt object; jumping or fall from a high place; jumping or lying before moving object; crashing of motor vehicle; other specified or unspecified means), Y10-Y19 (poisoning where there is insufficient information to determine if the cause was accident, self-harm or assault) and Y28 (injury with a sharp object with undetermined intent). We also examined trends in all MHA-related ED visits, defined as ED visits with ICD-10 codes F06-F99 (encompassing the entire ICD section on mental and behavioral disorders) and self-harm ED visits not associated with a mental health diagnosis. Finally, we examined trends in MHA-related outpatient physician visits to measure any change in ambulatory visit trends in response to 13RW. All outpatient visits to psychiatrists were considered MHA-related outpatient visits. Outpatient visits to primary care physicians or pediatricians were captured using a validated algorithm that categorizes outpatient physician visits as MHA- vs non-MHA-related (Steele et al., 2004).
Covariates
We measured demographics, including sex, age (as a continuous variable, and categorized as 10–19, 20–29 and 30–45 years), neighborhood income quintile and rural residence, comparing individuals with self-harm-related ED visits in the 3 months immediately following the release of 13RW (1 April 2017 to 30 June 2017) to a control group of individuals who had self-harm-related ED visits in the same 3 months 1 year prior to release (1 April 2016 to 30 June 2016). We used the Rurality Index of Ontario (RIO) score developed by the Ontario Ministry of Health and Long-Term Care and the Ontario Medical Association to measure rurality (Kralj, 2009). While the primary analysis focused on patient volumes, this additional analysis examined patient characteristics with controls included to test whether the cohort of people presenting with self-harm-related ED visits in 2017 were comparable to those who had a self-harm-related ED visit in 2016. We also measured health service utilization among individuals with a self-harm-related ED visit, including all-cause ED visits and outpatient visits. We compared these measures of utilization to the self-harm-related ED visit populations from the same two 3-month time periods in 2016 and 2017.
Statistical analysis
For descriptive analyses, we compared individuals who had self-harm-related ED visits between 1 April 2017 to 30 June 2017 (immediately following the release of 13RW) to individuals who had self-harm-related ED visits between 1 April 2016 and 30 June 2016, the same time period 1 year prior. We used t tests to compare continuous variables and chi-square tests for categorical variables. Statistical differences were compared using standardized differences due to the large sample size, with a standardized difference greater than 0.1 suggesting statistical difference (Austin, 2009; Yang and Dalton, 2012). We tested autoregressive integrated moving average (ARIMA) models for time series forecasting and intervention analysis of our monthly series data (April 2007–March 2018) (Austin, 2009; Yaffee and McGee, 2000; Yang and Dalton, 2012). ARIMA models were fitted to the data based on analysis of the pre-intervention period (before show release). Models with the lowest Bayesian information criterion value, and non-significant Ljung–Box Q statistics were subsequently fitted to the entire time series. Given that social media interest peaked between 1 April 2017 and 30 June 2017 (Niederkrotenthaler et al., 2019), we used a 3-month pulse model to specifically test the effect of the release of 13RW, with the intervention date set at 1 April 2017. In addition to the primary outcome of total self-harm-related ED visits, all analyses were stratified by sex. Note that the primary analysis was conducted for all forms of self-harm but an additional exploratory analysis was conducted examining only presentations for injury with a sharp object, given that this was the suicide method in the show.
Sensitivity analysis
In March 2017, the month leading up to the release of 13RW, there was a substantial amount of marketing and social media exposure about the show that could have had an impact on self-harm-related ED visits prior to the 31 March 2017 release. To account for this, we conducted a sensitivity analysis by re-running the time series analyses removing the visits for the month of March 2017.
Results
In the 3 months immediately following release of 13RW, time series estimates showed a statistically significant increase in the number of self-harm-related ED visits spanning the 3 months immediately following the release of 13RW and continuing for a total of 5 months (Figure 1). In the 3 months following the release, across all ages, there were 75 more self-harm-related ED visits compared to the expected numbers in the ARIMA model (standard error [SE] = 32.4; p = 0.02), a 6.4% increase (Supplemental Table 1). Self-harm-related ED visits by age are shown in Figure 2. After stratifying by age, adolescents had 60 excess visits (SE = 30.7; p = 0.048), whereas other age groups demonstrated no significant change. Self-harm-related ED visits by sex are shown in Supplemental Figure 1. When stratified by sex, only the female model retained significance, with 85 more self-harm-related ED visits compared to the ARIMA forecast (SE = 26.8; p = 0.002).
Self-harm-related emergency department visits in Ontario comparing model-based forecasts to actual rates following release of 13RW.
Self-harm-related emergency department visits in Ontario by age, May 2007 to March 2018.
Over the 3-month span, there were 3577 ED visits for self-harm, compared to 2944 in the same time frame 1 year previously. Increases were observed for all age groups with the largest numerical and proportionate increase in adolescents (10–19 years: +290 cases, +25.4%; 20–29 years: +169 cases, +17%; 30–45 years: +174 cases, +21.2%; see Figure 3). There were no differences between the two groups with respect to demographic or health service use characteristics (Table 1).
Total mental health and addiction-related (a) emergency department and (b) outpatient visits in Ontario by sex, May 2007 to March 2018.
Characteristics and health service utilization of individuals with self-harm-related emergency department visits in a 3-month period 1 year prior to and immediately following release of 13RW.
13RW: 13 Reasons Why; IQR: interquartile range; ED: emergency department.
Similar trends were observed in overall MHA-related ED visits during the 3-month window post-13RW release (Figure 3[a]). The ARIMA model estimated 486 (SE = 213; p = 0.02) excess MHA-related ED visits in the 3 months following the release of 13RW, a 3.1% increase. However, ARIMA models found that this increase in visits was significant only for those within the 10–19 age group (263 excess visits; SE = 114.5, p = 0.023). When stratifying by both age and sex, females aged 10–19 years had a significant finding (243 excess visits; SE = 77.5; p = 0.002) but males did not (24 excess visits; SE = 45.8; p = 0.59). Figure 3(b) and Supplemental Figure 2 show monthly trends in MHA-related outpatient visits by age and by sex, respectively. There was no observed increase in outpatient visits immediately following the release of 13RW, and the ARIMA models revealed no significant difference in monthly MHA-related outpatient visits following the release of 13RW. There were also no significant differences in any of the above analyses when they were restricted to the sharp objects method.
For the sensitivity analysis, all models were reconstructed by alternatively fitting ARIMA models up until March 2017, and then assessing the same 3-month period (April to June 2017, with March omitted). Sensitivity models found the same patterns of estimates. Further, to assess potential issues due to non-normality, a robustness analysis was applied by square root transformation. Models for total self-harm, total MHA ED, female self-harm and female MHA ED remained below the p = 0.05 significance threshold.
Discussion
To our knowledge, this is the first study to systematically investigate healthcare utilization patterns across an entire region following the release of 13RW. In Ontario, Canada, there was a significant excess in both ED visits that were self-harm-related and MHA-related ED visits overall in the 3 months following the release of 13RW; the self-harm-related increases were driven by excess visits in adolescents. Sex-stratified analyses demonstrated that these excess visits were primarily driven by increases in visits by females. Intervention analysis by ARIMA modeling noted significant increases in both total visits during this period, and in visits by females. The fact that proportional increases were largest in the 10- to 19-year age group with no significant differences in self-harm presentations in adults is highly consistent with what would be expected if imitation effects were occurring, given that adolescents were both the subject and target audience of the series; the findings also support the directionality of prior evidence about suicide deaths in this demographic group (Niederkrotenthaler et al., 2019). We did not, however, find any indication that the series was associated with an increase in presentations for self-harm using the method depicted (cutting/sharp objects). This aligns with findings from prior research examining the potential impact of 13RW on suicide rates which, for example, showed an increase in hanging suicides in adolescents in the United States (Niederkrotenthaler et al., 2019). This may owe to the fact that cutting is a method with a relatively lower risk of morbidity and mortality and thus elevated rates of cutting in the community, if they occur, may be less likely to be reflected in emergency presentations and/or deaths.
Although this study is unable to prove causation, these results are consistent with the notion that the series spurred imitative behavior in vulnerable viewers, especially when considered in the context of online survey data (Rosa et al., 2019), preliminary US healthcare utilization data (Hong et al., 2019) and mortality findings (Bridge et al., 2020; Niederkrotenthaler et al., 2019; Sinyor et al., 2019). The congruence of these outcomes is noteworthy. The results also suggest that harms associated with 13RW extend to a much larger group of people experiencing mental distress and engaging in self-harm behavior. Given the confluence of US and Canadian data available to date, it also suggests that such harms were international in scope.
One of the foundational ideas expressed by the creators and marketers of 13RW is that it was intended to engage people on the topic of suicide and that exposure to the show could help those who are struggling with mental illness and suicidal thoughts. The 13RW Discussion Guide notes that ‘13 Reasons Why is meant to provoke important conversations’ (Netflix, 2018). If the show had had that effect, one would have expected to observe increases in MHA-related primary care and psychiatric outpatient visits; however, neither was observed. US crisis line data demonstrated fewer calls in the period immediately after the release of Season 1 (Thompson et al., 2019). Taken together, these findings suggest that the show was associated with increased mental health and self-harm crises, necessitating ED visits, rather than facilitating non-acute care or crisis line contacts. The results are also notable from a healthcare delivery standpoint. Even if mental health crises were induced by 13RW with no additional de novo help seeking, we should still have expected an increase in outpatient MHA visits. While some of the people accounting for excess presentations to the ED might have benefited from emergency care and required no further follow-up, it would be expected that a substantial proportion could have benefited from further mental health assessment and care. Yet the data examined here provide no evidence that this occurred. These results suggest that more work needs to be done in Ontario to ensure that the mental healthcare system is responsive to fluctuating mental health needs of the population.
This study has a number of limitations. It is a natural experiment in which we can neither prove exposure among the people who presented to the ED nor prove that other concurrent events and societal factors did not cause a portion of or the entirety of the increase in ED visits. Our ability even to estimate potential exposure was undermined by the fact that Netflix does not release measures of viewership either overall or by age. Given that the series was presented, at least in part, as a public health intervention, it raises a question about whether platforms have a responsibility in such instances to release data that may assist researchers in quantifying impact. This issue deserves further consideration going forward. This study also only examined a single Canadian province and the degree of its generalizability is unclear. Furthermore, it only examined data on physician services as these are collected systematically in Ontario and was not able to examine non-physician healthcare visits (e.g. psychologist or social worker visits), which are especially relevant in the younger age groups, or to identify episodes of self-harm that did not come to the attention of healthcare services. Despite these limitations, it is important to reiterate that if significant increases in suicide deaths are truly attributable to 13RW, we would expect to observe parallel increase in proxy measures such as self-harm-related ED visits and that is precisely what we observed.
Conclusion
This study demonstrated increases in self-harm-related and MHA-related ED visits in Ontario, Canada, associated with the release of 13RW. It adds to now considerable evidence suggesting that the show caused harms and underscores the need for further engagement with entertainment media creators and providers to disseminate best practices for safe depictions of suicide (World Health Organization, 2019).
Supplemental Material
sj-docx-1-anp-10.1177_00048674211065999 – Supplemental material for Emergency department visits for self-harm in adolescents after release of the Netflix series ‘13 Reasons Why’
Supplemental material, sj-docx-1-anp-10.1177_00048674211065999 for Emergency department visits for self-harm in adolescents after release of the Netflix series ‘13 Reasons Why’ by Mark Sinyor, Emilie Mallia, Claire de Oliveira, Ayal Schaffer, Thomas Niederkrotenthaler, Juveria Zaheer, Rachel Mitchell, David Rudoler and Paul Kurdyak in Australian & New Zealand Journal of Psychiatry
Footnotes
Authors’ Note
Ms. Mallia and Dr. Kurdyak had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by Academic Scholars Awards from the Departments of Psychiatry at the University of Toronto and Sunnybrook Health Sciences Centre.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
