Abstract
Objectives:
To improve rapid detection and response to spikes in suspected nonfatal opioid overdoses in the community, the Rhode Island Department of Health (RIDOH) sought to modify its case definition to align with newly released guidance from the Council of State and Territorial Epidemiologists (CSTE).
Methods:
RIDOH used CSTE guidance to create 2 alternative criteria (CSTE-guided and RIDOH-modified CSTE) to replace one of RIDOH’s original criteria. We gathered emergency medical services (EMS) records from 4 months in 2022 to determine whether they met any criteria, and we reviewed them to determine the accuracy of identifying an opioid overdose. We calculated the positive predictive value for each criterion individually and as a complete case definition (RIDOH original definition, CSTE-guided definition, and RIDOH-modified CSTE definition).
Results:
The RIDOH-modified CSTE case definition was adopted because it had a positive predictive value of 91.5% in identifying nonfatal opioid overdose–related EMS incidents, similar to the original definition (91.3%) and higher than the CSTE-guided definition (82.2%). The RIDOH-modified CSTE case definition identified 624 additional nonfatal opioid overdoses in EMS data from 2018 through 2022, a 7% increase from the original definition.
Conclusions:
The RIDOH-modified CSTE case definition increased RIDOH’s ability to detect suspected nonfatal opioid overdoses while maintaining the accuracy of the case definition. With the new case definition, RIDOH aims to improve allocation of resources to communities experiencing increased overdose burden.
As the number of fatal opioid overdoses continues to rise in the United States, 1 the rapid detection and response to spikes in suspected nonfatal opioid overdoses have become increasingly essential to guide prevention activities. In the United States, emergency medical services (EMS) data are stored in the National Emergency Medical Services Information System (NEMSIS; https://nemsis.org). NEMSIS is a syndromic surveillance system used by EMS personnel and health departments nationwide to identify outbreaks and monitor changes in EMS runs. EMS agencies are required to enter incident data into NEMSIS within 2 hours of an incident. Because NEMSIS was not created to identify patient conditions and outcomes, case definitions must be created to accurately identify nonfatal opioid overdose–related EMS runs from the NEMSIS database. To begin the process of extracting suspected nonfatal opioid overdose data, local and state health departments must use coded and text elements in the incident reports to identify potential nonfatal opioid overdoses. The Rhode Island NEMSIS interface is called the Rhode Island EMS Information System (RI-EMSIS), and it enables the Rhode Island Department of Health (RIDOH) to collect, validate, and monitor all EMS agency documentation for the state in a single convenient portal.
In 2019, the RIDOH Substance Use Epidemiology Program used Enhanced State Opioid Overdose Surveillance funding and RI-EMSIS data to create a case definition to identify and track suspected nonfatal opioid overdose–related EMS runs. RIDOH has been using RI-EMSIS data to monitor overdose trends, identify communities disproportionately affected, respond to spikes in activity, and inform prevention activities. 2 In 2020, the Council of State and Territorial Epidemiologists (CSTE) convened an EMS Advisory Group call to create standard guidance for the development of a suspected nonfatal opioid overdose case definition. Rhode Island, among other states, collaborated to develop the EMS case definition with the CSTE advisory group. The advisory group identified key variable components and addressed data validation concerns, which led to the release of the new CSTE standard guidance in 2022. This standard guidance can be used to identify suspected nonfatal opioid overdoses in NEMSIS data by examining various coded elements and the free-text field in the patient narrative report. The CSTE guidance identifies suspected nonfatal opioid overdoses when an EMS run meets any of 4 criteria: (1) the provider’s primary impression or provider’s secondary impression is opioid overdose related, OR (2) the primary symptom or other associated symptoms is opioid overdose related, OR (3) medication administered is naloxone and response to medication administered is improved, OR (4) the patient care report narrative contains (a) at least 1 opioid-related keyword and (b) at least 2 overdose-related keywords. 3 Based on this new standard guidance, in 2023 RIDOH updated its existing criteria for suspected nonfatal opioid overdose–related EMS case definition to align with CSTE guidance.
The objective of this study was to formally evaluate each component of the revised RIDOH definition and to assess the overall accuracy of identifying suspected nonfatal opioid overdoses.
Methods
Case Definition for Suspected Nonfatal Opioid Overdose
The RIDOH original case definition identified a suspected nonfatal opioid overdose when the EMS record met any of the following 5 criteria: (1) the primary or secondary impression of the patient was opioid related and naloxone was given, OR (2) the primary or secondary impression of the patient was opioid related and the narrative report mentioned naloxone and a specific overdose-related term, OR (3) naloxone was given and the patient condition improved, OR (4) the narrative report mentioned naloxone and a specific unresponsive term, OR (5) naloxone was given to the patient prior to EMS arrival (Rhode Island–specific variable). 2 After comparing the RIDOH case definition with the CSTE standard guidance, we deemed RIDOH criteria 1, 2, 3, and 5 to be aligned with CSTE criteria 1, 2, and 3, but we determined that RIDOH’s criterion 4 definition might be enhanced by using CSTE recommendations on the patient care report narrative search. In this study, we evaluated (1) the positive predictive value (PPV) of each criterion in our existing case definition (criteria 1, 2, 3, and 5) that will not change after considering CSTE guidance; (2) 3 options for criterion 4 (criteria 4a, 4b, and 4c); and (3) the overall definitions for the original RIDOH case definition (criteria 1, 2, 3, 4a, and 5), the CSTE-guided definition (criteria 1, 2, 3, 4b, and 5), and the RIDOH-modified CSTE definition (criteria 1, 2, 3, 4c, and 5). As CSTE suggests, all definitions exclude nonemergent ambulance encounters to minimize duplicative records and fatal patient dispositions that may have insufficient evidence regarding cause of death.
Assessing CSTE’s Keywords for Criterion 4
To assess the effect of incorporating CSTE’s overdose-related keyword terms into the narrative report search, we analyzed records whose narrative report met CSTE’s criterion 4 guidance and contained at least 2 overdose-related keywords and 1 opioid-related keyword (Table 1). Preliminary review found that this criterion 4 guidance was too broad: it identified a large quantity of records (170 records in June 2022) with low accuracy (68% true opioid overdoses), and many records were inaccurately flagged by describing the same symptom with different terms (eg, unconscious, unresponsive, altered mental status) or were included in a list of denied/not-observed patient symptoms (eg, patient does not have altered mental status, patient denied shortness of breath). To address these issues, we organized the list of CSTE overdose-related keywords into 5 categories—altered mental status, pupil constriction, decreased respirations, ingestion, and overdose—and we required each record to include at least 2 overdose keywords from differing categories. We also created comprehensive exclusion criteria to avoid falsely including overdose symptoms that the patient denied experiencing or EMS personnel did not observe.
Differences in EMS narrative report definitions of opioid overdose case criterion 4 in a study conducted to evaluate definitions for the purposes of identifying suspected nonfatal opioid overdose–related incidents in Rhode Island, 2022
Abbreviations: CSTE, Council of State and Territorial Epidemiologists; EMS, emergency medical services; RIDOH, Rhode Island Department of Health.
After categorizing keywords, we modified the narrative report search criteria to require at least 2 overdose-related symptom categories. These modified criteria identified 114 records in June 2022, where 78% were true opioid overdoses. The categorization of symptoms and exclusion criteria ensured the sensitivity of identifying patients with multiple overdose symptoms. In our database of opioid overdose–related EMS runs, we reviewed 200 narrative reports that met the criteria for each symptom category. Through manual review, we double-checked that the categorical keywords were appropriately identified; we added alternative expressions of keywords to the inclusion criteria (eg, barely/shallow breathing); and we created exclusion criteria consisting of keywords that frequently led to false positives (eg, no/did not have difficulty breathing). Our review highlighted the need to expand the suggested CSTE overdose-related keyword list to include additional acronyms (eg, c/o SOB = complains of shortness of breath), regional layperson terms (eg, shooting up), and typos to ensure that our code could accurately identify the unique reporting styles of Rhode Island EMS personnel in the free-text narrative report.
After finalizing the Rhode Island–specific inclusion and exclusion criteria for each category of overdose symptoms, we repeated the review steps until the accuracy of the search for the categorized terms was roughly ≥95%. We initiated a similar process for CSTE’s 50 opioid-related keywords, including naloxone, using keywords in their unmodified format; however, an exclusion criterion was created to exclude narrative reports that mentioned that Narcan was not required (eg, Narcan not indicated or administered). We incorporated these vetted key terms to create the CSTE-guided definition (4b) and the RIDOH-modified CSTE definition (4c).
Creating Options for Criterion 4 Narrative Report
To assess whether criterion 4 could be improved by infusing all CSTE guidance, we created and assessed the PPV for 3 criterion 4 case definitions. First, we kept the original Rhode Island case definition defined previously (4a) to use as a comparison, which identified cases mentioning naloxone and a specific overdose-related term. Second, we incorporated the Rhode Island–categorized list of CSTE overdose symptoms, opioid-related keywords, and exclusion criteria to create criterion 4b, at least 1 opioid-related keyword (including the opioid reversal medication naloxone), and at least 2 overdose-related symptom categories. The validation of criterion 4b found numerous false positives due to the opioid-related keyword criteria; a common pattern was the inclusion of an opioid keyword in the narrative report for patients who were not acutely overdosing but rather had an allergy to opioids, used opioids unrelated to their present illness, or had EMS-administered opioids. When the narrative report included the keyword naloxone, as opposed to any opioid keyword, the case was more likely an acute overdose than something else. Identifying naloxone as the primary opioid keyword that routinely found true opioid overdoses led to the creation of 4c, replacing any opioid term from inclusion criteria and instead requiring naloxone with at least 2 overdose category symptoms. The CSTE-guided definition (4b) and RIDOH-modified CSTE definition (4c) were not developed a priori; they were developed during the manual keyword review of EMS cases and discussion with subject matter experts.
Assessing the Accuracy of EMS Run Records of a True Opioid Overdose
To determine whether each criterion was flagging true suspected nonfatal opioid overdoses, we initiated a tiered review using input from the chief of EMS (J.R.). First, we read each EMS record and marked a record as an opioid overdose if the narrative report mentioned that the patient was experiencing overdose symptoms (ie, loss of consciousness, respiratory depression, pale or blue skin discoloration, or constricted pupils) AND one of the following options: (a) the patient was overdosing, or (b) drug paraphernalia found at scene/on person, or (c) patient/bystander mentioned patient took opioids, or (d) patient has a medical history of opioid use, or (e) Narcan/naloxone administered and patient condition improved. Second, if the EMS record did not meet the narrative report parameters, we identified a case as an opioid overdose if the primary or secondary field impression was opioid related and naloxone was administered by EMS. Third, if a record did not meet the aforementioned 2 parameters, we identified a case as an opioid overdose if naloxone was administered by EMS and the patient’s condition improved. We considered cases that did not meet any of the 3 parameters false positives.
We gathered data on all cases occurring in March 2022 (n = 248), June 2022 (n = 212), September 2022 (n = 245), and December 2022 (n = 205) that were flagged by any of the 7 overdose criteria (1, 2, 3, 4a, 4b, 4c, or 5). Each epidemiologist (M.R., M.B.) was assigned 2 months of EMS data to assess whether records reflected a true opioid overdose by meeting any of the 3 opioid overdose parameters. To determine interrater reliability, we used a random number generator function to identify 15% of each month’s cases (n = 137 total) to be double reviewed.
Statistical Analysis
We calculated the PPV of each criterion individually (1, 2, 3, 4a, 4b, 4c, and 5) and as composite case definitions—RIDOH’s original case definition (criteria 1, 2, 3, 4a, and 5), the CSTE-guided case definition (criteria 1, 2, 3, 4b, and 5), and the RIDOH-modified CSTE case definition (criteria 1, 2, 3, 4c, and 5)—to determine which case definition most accurately identified the EMS records of patients who had nonfatal opioid overdoses. Each EMS record could meet multiple criteria; therefore, it was important to assess each criterion by itself and as part of the complete case definition.
We divided the number of true overdoses by the sum of the number of true overdoses (true positive) and the number of cases that were incorrectly categorized as an overdose (false positive).
We calculated a κ score 4 to estimate the interrater reliability (agreement) of each narrative report reviewer using the new definition of an opioid overdose case.
Results
The PPV for individual criteria 1, 2, 3, and 5 ranged from 97.5% to 100.0%, and each criterion identified 28 to 200 overdose events (Table 2). Criteria 4a, 4b, and 4c each identified 282 to 467 overdose events and had lower PPVs (69.2%-85.1%).
Performance of updated opioid overdose case definition among EMS runs classified as opioid overdose, by criteria, Rhode Island, 2022 a
Abbreviations: CSTE, Council of State and Territorial Epidemiologists; EMS, emergency medical services; PPV, positive predictive value; RIDOH, Rhode Island Department of Health.
Data were collected from EMS records from 4 months in 2022.
Criteria 1, 2, 3, 4a (naloxone + specific overdose-related term), and 5.
Criteria 1, 2, 3, 4b (1 opioid-related keyword + 2 overdose-related category keywords), and 5.
Criteria 1, 2, 3, 4c (naloxone + 2 overdose-related category keywords), and 5.
Overall, of the 646 cases that were identified by using the original RIDOH case definition (criteria 1, 2, 3, 4a, and 5), 590 records were found to be true opioid overdoses (PPV = 91.3%). Of the 838 cases that were identified by using the CSTE-guided case definition (criteria 1, 2, 3, 4b, and 5), 689 records were found to be true opioid overdoses (PPV = 82.2%). Finally, for the 693 cases that were identified by using the RIDOH-modified CSTE case definition (criteria 1, 2, 3, 4c, and 5), 634 records were found to be true opioid overdoses (PPV = 91.5%).
In the second review of randomly selected cases in each month (n = 137), the reviewers agreed on 93.4% (n = 128; 106 yes + 22 no) of the cases, with 77.4% (n = 106) of the cases accurately flagged as opioid overdoses. The interrater reliability between reviewers for the opioid overdose case definition was moderate (κ = 0.77; 95% CI, 0.63-0.91) (Table 3).
Interrater reliability for EMS runs classified as opioid overdoses for each reviewer as compared with the other in a study conducted to evaluate definitions for the purposes of identifying suspected nonfatal opioid overdose–related incidents in Rhode Island, 2022 a
Abbreviation: EMS, emergency medical services.
Data were collected from EMS records from 4 months in 2022.
Expected agreement = 0.69; actual agreement = 0.93; κ agreement = 0.77 (95% CI, 0.60-0.91).
Discussion
After examining the PPV of 3 nonfatal suspected opioid overdose case definitions, we found the new RIDOH-modified CSTE case definition (criteria 1, 2, 3, 4c, and 5) to be the best at identifying true nonfatal opioid overdoses (PPV = 91.5%) when compared with RIDOH’s original case definition (PPV = 91.3%; criteria 1, 2, 3, 4a, and 5) and the CSTE-guided case definition (PPV = 82.2%; criteria 1, 2, 3, 4b, and 5). Although the CSTE-guided definition detected the highest number of overall overdoses (n = 689) relative to the RIDOH-modified CSTE (n = 634) and the original (n = 590) RIDOH case definition, RIDOH adopted the RIDOH-modified CSTE case definition as the new EMS case definition to avoid the numerous false positives generated by the CSTE-guided definition. When RIDOH applied the newly adopted definition to historical data, from January 1, 2018, through December 31, 2022, it found that the RIDOH-modified case definition identified 7% more (n = 9454) EMS-attended suspected nonfatal opioid overdoses than what the original case definition found (n = 8832). While the PPV of the RIDOH-modified CSTE case definition is similar to the original definition, the ability to detect 7% more cases can improve RIDOH’s ability to determine the incidence of nonfatal suspected opioid overdoes in the community and guide prevention efforts.
The key to obtaining the high PPV and the larger case count with the RIDOH-modified CSTE case definition is criterion 4c: adding the CSTE-suggested list of descriptive overdose-related key terms, categorizing them, and ensuring that narrative reports contain at least 2 symptom categories and naloxone. Aligning with prior work, the requirement of naloxone in the narrative report helped eliminate cases in which an opioid may have been mentioned but was not the cause of the acute medical presentation. 5 Because naloxone is generally given if a bystander or first responder suspects that the patient is experiencing an overdose, using the naloxone term was more helpful in the identification of true opioid overdoses than searching for opioid-related terms, which have a broad range of applications in EMS response and use in the community. We found that the unmodified CSTE case definition (4b), which indicates searching the narrative report for 2 uncategorized overdose symptoms and at least 1 opioid term, accidentally flagged records in which the narrative mentioned patient allergy to opioids, EMS-administered opioids, or patients who take prescribed or illicit opioids unrelated to the medical emergency at hand. Additionally, modifications to the CSTE-guided definition—which included requiring the naloxone term only (not any opioid term), creating overdose-related categories versus keywords, and adding regionally informed terms—further reduced the false-positive rate and would be recommended for other jurisdictions implementing the CSTE guidance.
The original RIDOH nonfatal opioid overdose case definition was created and evaluated for accuracy in 2019 by using 2018 EMS data; however, we felt that it was important to ensure that the case definition remained accurate by using 2022 data and incorporating the most up-to-date national guidance. By performing this re-evaluation, we have been able to increase the precision and number of nonfatal cases detected, which will better guide partner outreach and prevention activities to prevent overdoses. Despite the accuracy of the definition, it is important to acknowledge that nationally and in Rhode Island, 9-1-1 is not called for roughly 50% of overdoses for various reasons (eg, fear of being stigmatized, negative interactions with law enforcement, feeling equipped to respond to the overdose with naloxone), and this percentage could be even higher in areas with strict law enforcement, larger racial and ethnic minority populations, or increased naloxone distribution.6 -8 For these reasons, we are focused on obtaining a high PPV rather than flagging all possible overdoses with a lower PPV.
In RIDOH’s overdose prevention response, EMS data are used to create hot spot maps for targeted geographic interventions through peer outreach and communication campaigns. Additionally, RIDOH has begun mapping fatal and nonfatal hot spots side by side, to identify additional areas of the state that should be targeted for outreach where 9-1-1 may not be routinely called. However, because nonfatal opioid overdose data are reported in real time, in contrast to fatal overdose data, which can take several months to be confirmed, preventive response relies heavily on the accuracy and timeliness of the nonfatal opioid overdose data. Looking forward, we plan to build on the accuracy of the RIDOH-modified CSTE case definition by linking EMS and emergency department data to discover the true number of overdoses and find cases missed by using EMS data alone.
A few states use variations of the CSTE opioid overdose case definition to better fit state syndromic surveillance system capabilities; however, the Florida Department of Health’s Enhanced State Opioid Overdose Surveillance (ESOOS) program and the RIDOH-modified CSTE case definition feature many of the same components to identify suspected nonfatal opioid overdose–related EMS incidents. 9 The most recent Florida Department of Health publication that assessed the opioid-involved overdose case definition reported 45% sensitivity, 99% specificity, and 45% PPV. 9 The Florida and Rhode Island definitions differ in 3 main ways. First, the RIDOH-modified CSTE case definition expands on the narrative report search by adding to the list of unresponsive and respiratory distress terms and adding keywords on other overdose symptoms (eg, pupil constriction) and related terms (eg, ingestion, overdose), whereas the Florida ESOOS definition does not. Second, the Florida ESOOS case definition includes specific opioid-related key terms that the RIDOH case definition chose to exclude to avoid including false positives. Lastly, the Florida ESOOS case definition uses key terms found in the chief complaint, whereas the RIDOH definition does not use this field. 10
Limitations
This study had several limitations. First, manual record review is time intensive, although it is not required once case definition validation is complete. Second, because of the emergent nature of overdoses, it is possible that the first responder reported insufficient documentation; therefore, reviewers would not be able to flag such records as an opioid overdose. Although all EMS personnel are formally trained in reporting and documentation standards, staff members at the same agency and staff members at different agencies vary in reporting practices. Third, we were able to identify nonfatal opioid overdoses only, so the way that we created and validated definitions may not be replicable for other types of drug surveillance.
Conclusions
RIDOH’s new opioid overdose case definition will be used for reporting data on suspected nonfatal opioid overdoses in Rhode Island. Overall, the critical analysis of the RIDOH and CSTE opioid overdose case definitions improved the accuracy and precision of RIDOH’s nonfatal data reporting and will likely improve outreach in response to overdose spikes and the overall incidence of overdoses in the state. We hope that this work can help guide public health jurisdictions implementing CSTE guidance to identify nonfatal opioid overdoses by using EMS data.
Footnotes
Acknowledgements
McClaren Rodriguez and Melissa Basta contributed equally to this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: M.R. was supported by the Centers for Disease Control and Prevention (CDC) of the US Department of Health and Human Services (HHS) as part of a financial assistance award totaling $9 087 646, with 100% funding by CDC/HHS. The contents are those of the authors and do not necessarily represent the official views of, or an endorsement by, CDC/HHS or the US government.
