Abstract
Objectives
We assessed the differences between the first version of the Centers for Disease Control and Prevention (CDC) opioid surveillance definition for suspected nonfatal opioid overdoses (hereinafter, CDC definition) and the North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT) surveillance definition to determine whether the North Carolina definition should include additional International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes and/or chief complaint keywords.
Methods
Two independent reviewers retrospectively reviewed data on North Carolina emergency department (ED) visits generated by components of the CDC definition not included in the NC DETECT definition from January 1 through July 31, 2018. Clinical reviewers identified false positives as any ED visit in which available evidence supported an alternative explanation for patient presentation deemed more likely than an opioid overdose. After individual assessment, reviewers reconciled disagreements.
Results
We identified 2296 ED visits under the CDC definition that were not identified under the NC DETECT definition during the study period. False-positive rates ranged from 2.6% to 41.4% for codes and keywords uniquely identifying ≥10 ED visits. Based on uniquely identifying ≥10 ED visits and a false-positive rate ≤10.0%, 4 of 16 ICD-10-CM codes evaluated were identified for NC DETECT definition inclusion. Only 2 of 25 keywords evaluated, “OD” and “overdose,” met inclusion criteria to be considered a meaningful addition to the NC DETECT definition.
Practice Implications
Quantitative and qualitative trends in coding and keyword use identified in this analysis may prove helpful for future evaluations of surveillance definitions.
Keywords
Syndromic surveillance is a key surveillance component of public health entities at all levels. Syndromic surveillance uses near–real-time electronic prediagnostic and diagnostic health-related data to detect, characterize, and monitor events of potential public health importance. It has diverse applications for improving disease detection, providing real-time insights into the health of communities, and informing public health action across a range of hazards. 1 Syndromic surveillance of nonfatal opioid overdoses is a national priority in monitoring the opioid overdose epidemic in the United States. The Centers for Disease Control and Prevention’s (CDC’s) Enhanced State Opioid Overdose Surveillance (ESOOS) program began funding North Carolina to collect timely data on fatal and nonfatal opioid overdoses in 2017. 2 As a part of the ESOOS program, CDC developed and distributed guidelines on how to identify nonfatal overdoses nationally, using emergency department (ED) syndromic surveillance data shared with CDC by most states and the District of Columbia. 3 In this article, we refer to the initial version of CDC’s nonfatal opioid overdose syndrome definition as CDC Opioid V.1. Although syndromic surveillance of nonfatal opioid overdoses in North Carolina is funded through CDC, the North Carolina Division of Public Health (NC DPH) uses a locally developed opioid overdose syndrome definition in the North Carolina Disease Event Tracking and Epidemiologic Tool (NC DETECT), North Carolina’s syndromic surveillance system. 4,5
The national (CDC Opioid V.1) and state (NC DETECT) opioid overdose syndrome definitions differ slightly because of how they are used, to what data they are applied, and how their results are communicated to the public. NC DPH uses the NC DETECT definition to provide publicly available monthly updates at the state level and for select counties that meet count thresholds (≥5) that allow for public sharing. 6 Because NC DPH shares raw ED visit counts with the public, the definition used is based solely on poisoning diagnosis codes. In addition, North Carolina receives ≥1 diagnosis code for approximately 90% of ED visits statewide, which reduces reliance on keyword searches to identify potential overdoses. 7 In comparison, the CDC definition must accommodate variability in diagnosis code completeness across all states that participate in its overdose surveillance activities and includes both diagnosis codes and keywords in its suspected opioid overdose definition. CDC reports findings as trends by region and state and does not report absolute rates or count data. 8
The purpose of this study was to assess differences between the CDC definition and the NC DETECT definition for nonfatal opioid overdoses in an effort to improve state-based syndromic surveillance and further inform national syndromic surveillance efforts. We examined data on encounters determined to be false positives among those ED visits captured by the components of the CDC Opioid V.1 nonfatal opioid overdose syndrome definition but not the NC DETECT nonfatal opioid overdose syndrome definition. Because the NC DETECT definition is conservatively written (a subset of opioid poisoning codes only, all intents except adverse effects and underdosing), we wanted to study the CDC definition in our data to determine whether to expand the NC DETECT definition to include additional International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes and/or chief complaint keywords. Our results will inform modifications to existing syndrome definitions and/or new approaches for opioid-related surveillance in North Carolina. The North Carolina Injury and Violence Prevention Branch, part of the North Carolina Department of Health and Human Services, provides monthly reports with counts of ED visits for opioid overdose, whereas CDC presents quarterly opioid overdose data as a percentage change. Any changes to the NC DETECT opioid overdose definition used for reporting monthly counts, therefore, must minimize the number of false positives to the greatest extent possible. Conducting rapid syndromic surveillance with a minimum number of false positives is important at the state and local level to inform the distribution of public health resources and personnel for combating the opioid overdose epidemic.
Materials and Methods
We implemented the CDC Opioid V.1 definition in NC DETECT as closely as possible to the CDC implementation for national syndromic surveillance. Applying the national definition directly to NC DETECT data was necessary to facilitate accurate comparison, because the numbers generated by implementation of CDC Opioid V.1 in NC DETECT do not exactly match the numbers generated within CDC’s National Syndromic Surveillance Program’s BioSense Platform. Data preprocessing procedures and business rules differ for each system, and NC DETECT uses a different technique for querying the data.
Definition Components
The CDC Opioid V.1 definition included ICD-10-CM diagnosis codes and chief complaint keywords meeting any of 4 components of the definition (Table 1). 3 The first component of the definition flagged an overdose if the chief complaint contained the terms “Narcan” or “naloxone.” The second component of the definition identified combinations of chief complaint keywords; overdoses were flagged if they included 1 of a group of overdose-related symptomatic or thematic keywords, such as “altered mental status” or “overdose,” along with a second keyword naming a type of opioid, such as “heroin” or “codeine.” These keywords included abbreviations and common misspellings. For this investigation, the second component of the definition was reported by overdose symptomatic or thematic keywords rather than by opioid keyword, because these terms can describe many medical conditions other than opioid overdose. The third component of the definition identified ICD-10-CM diagnosis codes. Some, but not all, of these codes overlapped with the NC DETECT definition. Codes not included in the NC DETECT syndrome definition include T40.6 codes and F11 codes, pertaining to opioid use/abuse/dependence with intoxication. 9 Although the CDC Opioid V.1 definition includes International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes and Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) codes (a system of standardized terminology used in clinical documentation), we did not include these search terms in our analysis. We excluded them because North Carolina hospitals do not send SNOMED CT codes to NC DETECT and ICD-9-CM codes were not valid codes during the study period; that is, they were no longer used by hospitals after September 30, 2015. The fourth component of the CDC Opioid V.1 definition identified ED visits with 1 overdose-related symptomatic or thematic chief complaint keyword, such as “altered mental status” or “overdose,” along with 1 of a set of F11 ICD-10-CM diagnosis codes not included in the third component of the definition (pertaining to opioid use/abuse/dependence without intoxication). Finally, the definition excluded ED visits identified by keyword only when certain negation terms (eg, “denies heroin,” mention of withdrawal symptoms, or requests for detoxification) were present.
Abbreviations: CDC, Centers for Disease Control and Prevention; ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; NC DETECT, North Carolina Disease Event Tracking and Epidemiologic Tool; SNOMED CT, Systematized Nomenclature of Medicine Clinical Terms.
aThe initial version of CDC’s nonfatal opioid overdose syndrome definition was developed for the Enhanced State Opioid Overdose Surveillance program in 2017.
bSNOMED CT codes were not included in this study because they are not provided to NC DETECT.
cNegation terms such as “denies heroin” were analyzed separately and discussed in the Results section.
dICD-9-CM codes, although a part of the CDC Opioid V.1 definition, were not valid codes in our data during the study period. Thus, only the ICD-10-CM codes are shown.
eCodes that are also included in the NC DETECT opioid overdose syndrome definition.
Identification of False Positives
In contrast, the NC DETECT nonfatal opioid overdose syndrome definition used in our study is composed solely of ICD-10-CM codes and includes all ICD-10-CM codes in the third component of the CDC Opioid V.1 definition, with the exception of the T40.6 and F11 opioid use/abuse/dependence with intoxication codes. To better understand the effect of the additional components within the CDC Opioid V1 definition, we applied the CDC Opioid V.1 definition to all ED visits from a 7-month period in the NC DETECT data (January 2018–July 2018) and determined which of the 4 criteria were met by each visit (Table 2). Two independent, medically trained graduate research assistants (D.B. and C.W.) clinically reviewed the observations (N = 2296) identified as a suspected opioid overdose by the CDC Opioid V.1 definition, while excluding ED visits also captured by the NC DETECT nonfatal opioid overdose syndrome definition. The objective of this secondary analysis was to identify false positives—observations misidentified as opioid overdoses. Reviewers were instructed to be conservative in their identification of false positives, identifying a false positive only if evidence was available to support an alternative explanation for the patient presentation that the reviewers deemed more likely than an overdose. Clinical reviewers (D.B. and C.W.) used information from the chief complaint, triage note, blood pressure readings, nursing comments, ED disposition (eg, admitted, discharged), and disposition diagnosis codes during this analysis. Once individual analyses were complete, the 2 independent reviewers met to reconcile disagreements. The independent reviewers discussed differences until they reached consensus, conservatively identifying false positives only if both reviewers agreed. After independent medical record review and reconciliation between the reviewers, quantitative and qualitative results were summarized.
Abbreviations: CDC, Centers for Disease Control and Prevention; ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification.
aFalse positives designated by concordant decision following independent medical record review by 2 clinical reviewers.
bData source: North Carolina Disease Event Tracking and Epidemiologic Collection Tool.
cRepresents the total number of ED visits meeting criterion specified in the first column; ED visits may meet >1 criterion and, as such, will add to more than the total. Percentages given in rows with a keyword or ICD-10-CM criterion are of total ED visits and will add to >100 because ED visits may meet >1 criterion. Percentages given in rows with false positives are the proportion of the keyword or ICD-10-CM criterion in the previous row that are false positives.
dRepresents the number of ED visits identified only by the criterion specified in the first column. Percentage is of total uniquely identified ED visits. Percentages given in rows with false positives are the proportion of the uniquely identified keyword or ICD-10-CM criterion in the previous row that are false positives.
eIndicates a criterion with a false-positive rate >10% and/or a criterion uniquely identifying <10 ED visits.
fData source: National Center for Health Statistics.9
gT40.6 codes include T40.601A, T40.604A, T40.691A, and T40.694A.
hF11.1 codes include F11.12, F11.120, F11.121, F11.122, and F11.129.
iF11.2 codes include F11.22, F11.220, F11.221, F11.222, and F11.229.
jF11.9 codes include F11.92, F11.920, F11.921, F11.922, and F11.929.
CDC released a second version of its nonfatal opioid overdose surveillance definition (Opioid V.2) in October 2019, after we completed our initial analyses. To further confirm the validity of our findings, we compared the V.1 and V.2 definitions and examined the additional negation terms included in the CDC Opioid V.2 definition to determine how many unique ED visits the definition identified and whether those visits were false positives.
We conducted analyses using Stata SE version 15.1 (StataCorp) and Microsoft Excel 365 (Microsoft Corp). Because this project was a quality improvement project for public health surveillance, institutional review board approval was not required.
Results
Of the 2296 ED visits reviewed, the reviewers initially disagreed on 312 (13.6%) observations. After discussing disagreements, reviewers reached a consensus on all visits. Results were summarized in several ways. The first column of each results table reports the total number of ED visits meeting the opioid overdose criteria in question, regardless of whether that visit also met additional criteria. For example, an ED visit with a chief complaint that included the word “Narcan” and a diagnosis code of T40.601A would be flagged based on both the chief complaint, under the first component of the CDC Opioid V.1 definition, and the ICD-10-CM diagnosis code included in the third component of the definition. Each table also includes the number of “unique” ED visits identified by the criteria—this designation signifies visits that, were it not for that specific criteria being included in the definition, would not have been identified as opioid overdoses. We found substantial differences in false-positive rates for the additional ICD-10-CM codes and keywords included in the CDC Opioid V.1 definition, ranging from 2.6% to 41.4% for codes and terms uniquely identifying ≥10 ED visits (Tables 2 -4).
Abbreviations: CDC, Centers for Disease Control and Prevention; ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification.
aFalse positives designated by concordant decision following independent medical record review by 2 clinical reviewers.
bSee component 3 in Table 1.
cData source: North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).
dData source: National Center for Health Statistics. 9
eRepresents the total number of ED visits meeting criterion specified in the first column. Percentages given in rows with an ICD-10-CM criterion are of total ED visits in the definition component, and the percentage of the total number of ED visits for the component are from the total sample (N = 2296). Percentages given in rows with false positives are the proportion of the ICD-10-CM criterion in the previous row that are false positives. ED visits may meet >1 criterion; as such, counts will add to more than the total across all definition components and percentages will add to >100 across all definition components.
fRepresents the number of ED visits identified only by the criterion specified in the first column. Percentages given in rows with an ICD-10-CM criterion are of total uniquely identified ED visits in definition component, and the percentage of the total number of unique ED visits for the component are from the total sample (N = 2152). Percentages given in rows with false positives are the proportion of the uniquely identified ICD-10-CM criterion in the previous row that are false positives.
Abbreviations: AMS, altered mental state; CDC, Centers for Disease Control and Prevention; ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; LOC, loss of consciousness; OD, overdose; SOB/SHOB, shortness of breath.
aFalse positives designated by concordant decision following independent medical record review by 2 clinical reviewers.
bSee component 2 in Table 1.
cData source: North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).
dRepresents the total number of ED visits meeting criterion specified in the first column. Percentages given in rows with a keyword combination criterion are a percentage of the total ED visits in definition component, and percentage of the total number of ED visits for the component are from the total sample (N = 2296). Percentages given in rows with false positives are the proportion of the keyword combination criterion in the previous row that are false positives. ED visits may meet >1 criterion, so counts will add to more than the total across all definition components, and percentages will add to >100 across all definition components.
eRepresents the number of ED visits identified only by a keyword combination criterion, including the one specified in the first column. ED visits may meet >1 keyword combination criterion and will add to more than total. Percentage of each criterion is of the total ED visits identified solely by a keyword combination. Percentage of total is out of the total sample (N = 2296). Percentages given in rows with false positives are the proportion of the keyword criterion in the previous row that are false positives.
fRepresents the number of ED visits identified only by the criterion specified in the first column. Percentages given in rows with a keyword combination criterion are of total uniquely identified ED visits in definition component, and the percentage of the total number of unique ED visits for the component are from the total sample (N = 2152). Percentages given in rows with false positives are the proportion of the uniquely identified keyword combination criterion in the previous row that are false positives.
Diagnoses made using the T40.6 ICD-10-CM code accounted for 692 of 2152 (32.2%) ED visits uniquely identified by the CDC Opioid V.1 definition, with an overall false-positive rate of 7.1% (49 of 692). The increase in identification of nonfatal opioid overdose visits resulted primarily from the addition of ICD-10-CM codes T40.601A and T40.604A (Table 3). This increase represents a sizeable proportion of ED visits not previously identified with the NC DETECT definition, with a false-positive rate meeting our criteria for future inclusion. Two F11 diagnoses codes (F11.129: opioid abuse with intoxication, unspecified; and F11.929: opioid use unspecified with intoxication, unspecified) also met criteria for inclusion, although most F11 diagnoses codes did not. False-positive rates were high for “Narcan” and “naloxone” keyword-identified unique ED visits (20.0%) and for the definition component that combined an overdose keyword and an opioid use/abuse/dependence without intoxication F11 diagnosis code (25.2% overall). Most keyword combinations (13 of 25 overdose symptomatic or thematic keywords) failed to uniquely identify any ED visits. Only “overdose” and “OD” met criteria for inclusion; together, they accounted for 53 of 77 (68.8%) ED visits uniquely identified by a keyword combination (Table 4).
We also identified common qualitative trends in coding and keyword use that may prove helpful for further analysis of nonfatal opioid overdose surveillance definitions by NC DETECT or others doing work in this area of syndromic surveillance (Box). Many ED visits identified as false positives involved patients with documented opioid use and one of the following medical conditions, which provided an alternative explanation for their clinical presentation: chronic obstructive pulmonary disease, hypoglycemia, or sepsis. In contrast, most ED visits correctly identified as opioid overdoses contained an opioid overdose ICD-10-CM code and/or the keywords “OD” or “overdose.”
Box. Summary of qualitative findings from clinician review of identified opioid overdose–related ED visits, North Carolina, January–July 2018
Abbreviations: COPD, chronic obstructive pulmonary disease; ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; SOB, shortness of breath.
We separately analyzed chief complaint negation terms to determine whether our findings would be substantially altered by omitting negation terms from the NC DETECT definition. None of the false positives we identified in our analysis included opioid overdose denials in the chief complaint (using the words “denies,” “deny,” “not,” and “no”). We also searched for all additional 78 CDC Opioid V.2 definition chief complaint negation terms in our full sample of 2296 ED visits and identified only 12 visits; 7 were marked as false positives through our review, and 5 were coded as potential overdoses. Of the 5 potential overdoses identified, 1 included “Narcan” or “naloxone” in the chief complaint, 3 met criteria for the chief complaint combination designation, 3 included ICD-10-CM codes specified in the definition, and 1 met the chief complaint keyword and F11 ICD-10-CM code combination criteria (visits could meet >1 definition criteria). Of the keyword combination visits, 2 were identified by “OD” or “overdose.” Of the ICD-10-CM code visits, 2 were identified by the code T40.601. The remainder fell into categories with false-positive rates >10% or that uniquely identified <10 ED visits.
Discussion
Syndromic surveillance is an essential part of understanding and combating the opioid overdose epidemic, but the strength of a syndromic surveillance system is largely in the syndrome definitions it uses. Although the use of ICD-10-CM codes is the most reliable approach for identifying nonfatal opioid overdoses, challenges remain in developing a national definition that can be used by all jurisdictions participating in the National Syndromic Surveillance Program. Not all jurisdictions receive diagnostic codes reliably for all of their ED visits for reasons including, but not limited to, data use agreements, data transmission business rules, and/or lack of mandates. Investigations such as ours allow not only the comparison of state and national definitions but also the rigorous implementation of a national definition in state-level data, generating further insight into how both definitions can be improved for greater external validity and dissemination to other states. Both the selected nonfatal opioid overdose syndrome definition and the data source chosen for routine syndromic surveillance can affect the generated results; investigators need a detailed understanding of how best to optimize their identification of true opioid overdoses and minimize the number of false positives, based on the intricacies of their own data. For example, Kentucky conducted a similar study evaluating heroin and other opioid poisoning surveillance definitions relying on ICD-10-CM definitions. Unlike our study, researchers in Kentucky did not include chief complaint keywords in their definitions, but they had access to detailed medical records for their clinician medical record review and found some false-positive trends that were similar to our analysis. 10 Although these investigations are a start, additional investigations are needed to continuously improve syndrome definitions.
Limitations
This study had several limitations, primarily because of the inherent limitations of syndromic surveillance data. First, we based our assessment of whether identified ED visits were false positives on subjective evaluation of available data for each visit, focusing primarily on the chief complaint, triage note, initial blood pressure reading, nursing comments, ED disposition, and disposition diagnosis codes. We did not have access to data on inpatient diagnoses or laboratory results for admitted patients. Furthermore, triage notes often provided valuable historical, contextual, and behavioral information, but they were not available for nearly half the visits (n = 1112, 48.4%). Second, although the use of 2 clinically trained reviewers independently evaluating visits before meeting to reconcile disagreements, in addition to our conservative definition of false positives, likely reduced the chance of bias arising from differential availability of triage notes, visits without triage notes may have been either more or less likely to be designated as false positives.
Third, many criteria for the CDC Opioid V.1 definition identified <10 visits during the 7-month study period, making accurate analysis of false-positive rates impossible. Eliminating these criteria identifying <10 ED visits may introduce bias if certain areas of the state with a small number of overdoses preferentially use one of these criteria; however, on the whole, the criteria identifying <10 unique ED visits accounted for <2.5% of the 2296 opioid overdoses reviewed statewide. A potential risk of bias is also a concern for facilities differentially using more common Opioid V.1 definition criteria (identifying ≥10 unique ED visits) with false-positive rates >10%, which are also not recommended for inclusion in NC DETECT’s nonfatal opioid overdose syndrome definition. Fourth, we did not evaluate ED visits currently identified under NC DETECT’s definition criteria to ascertain false-positive rates. As such, we could not directly compare the validity of the new criteria with the criteria already in use.
Fifth, when considering the results of this analysis, it is important to remember that we defined false-positives conservatively. If coders could not rule out an opioid overdose given evidence in the available NC DETECT data elements, that observation was coded as an overdose. This method relied on evidence in support of or in opposition to an opioid overdose identification being readily available in the medical record. Given the need for this documentation, the presented false-positive rates are likely an underestimate of the true number of false positives.
Finally, since we conducted this investigation, CDC has released an updated syndrome definition for the identification of nonfatal overdoses. This CDC Opioid V.2 definition added more drug names and 78 additional negation terms that may be listed in the chief complaint field. We analyzed the negation terms separately and found they identified only 12 of 2296 ED visits. Given the small number of false-positive visits identified by these additional negation terms and the lack of a prevalent pattern in them, we believe our findings are unchanged under the CDC Opioid V.2 definition. However, we did not complete a full analysis of the Opioid V.2 definition.
Practice Implications
Recommendations for changes to the NC DETECT syndrome definition were based on the number of ED visits uniquely identified by a given criterion and the false-positive rates of those visits. We determined that criteria identifying <10 unique ED visits would have unstable false-positive rates because of small numbers; as such, we could not recommend their inclusion in the NC DETECT definition. Of those definitional criteria identifying ≥10 unique ED visits, we recommended keywords and diagnosis codes that had a false-positive rate of ≤10% for inclusion in the NC DETECT nonfatal opioid overdose syndrome definition.
Based on our findings, we recommend the following changes to the NC DETECT nonfatal opioid overdose syndrome definition: (1) the addition of 4 ICD-10-CM codes (T40.601A, T40.604A, F11.129, and F11.929) and (2) the addition of 2 keywords (“od” and “overdose”). These recommendations expand the current definition used by NC DETECT to include additional codes and keywords while adhering to a conservative approach to false positives. We suspect the false-positive rates we obtained are an underestimate of the actual false-positive rates, and analysis of count data, in particular, requires greater specificity than trend data. This expanded definition will allow for the analysis of count or trend data with minimal negative effect on specificity. Future analysis should evaluate facility-level use of keywords and codes targeted for elimination because of sparse use or high false-positive rates. If smaller facilities have unusual documentation of probable overdose ED visits, the elimination of some of these keywords and codes may have an outsized effect on results for those facilities. Continued analyses can help ensure syndromic surveillance data are useful to inform direct public health action.
Since we completed this analysis, NC DPH has begun including T40.6 diagnosis codes in its opioid overdose syndrome definition. Future analyses should be used to evaluate other definitions including those for heroin and stimulant overdoses that are monitored as part of CDC’s Overdose Data to Action cooperative agreement. 11
Footnotes
Authors’ Note
The authors Danielle M. Brathwaite and Catherine S. Wolff contributed equally to the work.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding was provided by the Centers for Disease Control and Prevention, National Center for Injury Prevention and Control Enhanced State Opioid Overdose Surveillance agreement to the North Carolina Division of Public Health (agreement #5NU17CE924902).
