Abstract
Accurately measuring the incidence of major postoperative complications is essential for funding and reimbursement of healthcare providers, for internal and external benchmarking of hospital performance and for valid and reliable public reporting of outcomes. Actual or surrogate outcomes data are typically obtained by one of three methods: clinical quality registries, clinical audit, or administrative data. In 2017 a perioperative registry was developed at the Alfred Hospital and mapped to administrative and clinical data. This study investigated the statistical agreement between administrative data (International Statistical Classification of Diseases and Related Health Problems (10th edition) Australian Modification codes) and clinical audit by anaesthetists in identifying major postoperative complications. The study population included 482 high-risk surgical patients referred to the Alfred Hospital anaesthesia postoperative service over two years. Clinical audit was conducted to determine the presence of major complications and these data were compared to administrative data. The main outcome was statistical agreement between the two methods, as defined by Cohen’s kappa statistic. Substantial agreement was observed for five major complications, moderate agreement for three, fair agreement for six and poor agreement for two. Sensitivity and positive predictive value ranged from 0 to 100%. Specificity was above 90% for all complications. There was important variation in inter-rater agreement. For four of the five complications with substantial agreement between administrative data and clinical audit, sensitivity was only moderate (61.5%–75%). Using International Statistical Classification of Diseases and Related Health Problems (10th edition) Australian Modification codes to identify postoperative complications at our hospital has high specificity but is likely to underestimate the incidence compared to clinical audit. Further, retrospective clinical audit itself is not a highly reliable method of identifying complications. We believe a perioperative clinical quality registry is necessary to validly and reliably measure major postoperative complications in Australia for benchmarking of hospital performance and before public reporting of outcomes should be considered.
Keywords
Introduction
Accurately measuring the incidence of major postoperative complications is essential for funding and reimbursement of healthcare providers, for internal and external benchmarking of hospital performance and for valid and reliable public reporting of outcomes. Actual or surrogate outcomes data are typically obtained by one of three methods: clinical quality registries, clinical audit, or administrative data. National or geographically defined clinical quality registries may have full or partial coverage of the target population.1,2 Clinical audit is commonly undertaken at the institutional, local or individual clinician level to define, measure and record outcomes. Administrative data are appealing as a surrogate for clinical outcomes because they are universally collected, impose no additional cost to obtain, and use national or international coding systems that facilitate comparisons within and between institutions. 3
Surgical care accounted for 23% of hospital separations in Australia in 2016–17. 4 In the absence of a national or state-based perioperative clinical quality registry or audit programme, administrative data are sometimes used as surrogates for major postoperative complications in Australia. The Australian hospital system data adopt the International Statistical Classification of Diseases and Related Health Problems (ICD) coding system for both diagnostic and hospital funding purposes. According to the World Health Organization, the ICD is ‘the foundation for the identification of health trends and statistics globally, and the international standard for reporting diseases and health conditions. It is the diagnostic classification standard for all clinical and research purposes’. 3 Using hospital coder assigned ICD (10th edition) Australian Modification (ICD-10-AM) codes as a proxy for clinician- or researcher-collected outcome data has face validity because ICD-10-AM codes are assigned based on physicians’ documentation in the medical record and should produce an accurate account of all major complications. This suggestion of face validity is reinforced by compulsory regular monitoring of certain hospital-acquired complications by Australian public and private hospitals based on administrative data.
There are, however, several unique aspects to the ICD-10-AM codes that suggest construct validity, content validity and reliability in this context require further exploration. These include a different threshold for the coding of comorbidities compared to the international version of the codes, variation in coding protocols between different Australian states and territories and over time, and the use of coding data for hospital-funding purposes. 5 Although originally designed for clinical and research purposes, ICD-10-AM definitions can be quite broad in comparison to the highly specific definitions usually applied for the purpose of perioperative outcomes audit or research, potentially overestimating the incidence of complications. Conversely, ICD-10-AM codes can only be applied based on the appearance of exact words or phrases observed in the medical record during screening by human eye and cannot be inferred by the coder from clinical events beyond the scope of their professional training, 5 potentially underestimating the incidence of complications.
Previous studies have found highly variable agreement between administrative datasets and validated perioperative clinical outcomes in the United States, Canada and the United Kingdom.6–10 No studies in this area were identified from Australia, where the literature on administrative data focuses primarily on the accuracy of original coding data,11,12 coding comparison studies 13 and population-based estimates or surveillance of clinical outcomes based on administrative data alone.14–17 This study investigated the statistical agreement between administrative data (ICD-10-AM codes) and clinical audit by anaesthetists in identifying major postoperative complications in a cohort of high-risk surgical patients.
Materials and methods
The primary objective was to identify and assess ICD-10-AM codes as an indicator of postoperative morbidity in a cohort of high-risk surgical patients. Given that in Australia, hospital-acquired complications are reported and benchmarked nationally based on ICD-10-AM coding 18 and coding data are used for hospital reimbursement, it was anticipated there would be near-perfect agreement between ICD-10-AM codes and major postoperative complications determined by clinical audit by anaesthetists (kappa 0.81–1.00), and near perfect inter-rater agreement between anaesthetists (kappa 0.81–1.00) for each postoperative complication. 19
In 2017 a perioperative registry was designed and developed using REDCap electronic data capture tools at the Alfred Hospital, a tertiary referral hospital in Melbourne, Australia.20,21 Data fields were mapped to multiple clinical and administrative hospital datasets on the hospital electronic medical record. This included using hospital coder-assigned ICD-10-AM codes to identify patient comorbidities and in-hospital complications.
The study population included 482 high-risk surgical patients referred to the Alfred Hospital anaesthesia postoperative service during a two-year period from 1 September 2015 to 30 August 2017. Patients were referred to this service when, following a post-operative intensive care review in recovery, they were deemed not to require a critical care bed, or when they had two or more clinical risk factors (Figure 1). All patients were 16 years or older and had undergone elective or emergency non-cardiac surgery under general or regional anaesthesia and stayed at least one night in hospital. Exclusion criteria included patients under 16 years of age and patients not managed by the anaesthesia postoperative service including those who underwent cardiac or obstetric surgery and any patient admitted directly to critical care after surgery.
Referral criteria to anaesthesia postoperative service.
The Core Outcomes Measures in Perioperative and Anaesthetic Care – Standardised Endpoints for Perioperative Medicine (COMPAC-StEP) group is developing standardised endpoints for use in research in anaesthesia and perioperative medicine. 22 A set of clinical outcomes of interest was selected a priori from these standardised endpoints. A proposed list of ICD-10-AM codes corresponding to these outcomes was manually constructed. The proposed list was compared to actual ICD-10-AM codes assigned by hospital coders for a series of patients previously referred to the anaesthesia postoperative service. Any missing ICD-10 codes were added to arrive at a final list of ICD-10-AM codes corresponding to the selected clinical outcomes (Supplementary Table 1).
Patient data including the medical record number and date of surgery were exported from the anaesthesia postoperative service database and imported into the study database. The investigators undertook clinical chart review of the electronic medical records to identify pre-determined COMPAC-StEP or American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP®) major complications. The results were recorded in the study database and this determination became the clinical reference standard. A unique identifier was then used to import matched data from the perioperative registry into a concealed arm of the study database.
Clinical chart review was conducted by eight anaesthetists: four consultants, three provisional fellows and one registrar with prior research experience. Each anaesthetist could access only the records assigned to them in the study database. The anaesthetists were provided with a data dictionary (Supplementary Table 2) and undertook brief individual training prior to commencement. As the consensus definitions for clinical endpoints in research in perioperative medicine were still under development at the time of the study, 22 the most recent commonly applied research definition was used for each morbidity endpoint. For morbidity endpoints without a commonly applied research definition, a clinically relevant study definition was created by consensus between the lead investigators.
Both arms of study data were collected and managed using a REDCap database hosted at the Alfred Hospital. To assess inter-rater reliability the anaesthetists were grouped into four pairs and each anaesthetist assessed 10 records of their counterpart such that each pair assessed 20 records in common. Inter-rater reliability was assessed for each morbidity endpoint for each pair. Anaesthetists (except JR) were blinded to the pairing allocations, to the knowledge that there was one consultant anaesthetist in each pair and to which medical records were co-reviewed. The anaesthetists did not have access to the arm of the database containing registry data, including ICD-10-AM codes, and data linkage to the perioperative registry did not occur until after the clinical chart review was complete.
Approval for the study was granted by the Alfred Health Human Research Ethics Committee (approval 161/17), with a waiver of patient consent because the study was limited to a retrospective review of routinely collected data in the electronic medical record.
The main outcome was the statistical agreement between ICD-10-AM codes and the occurrence of major postoperative complications determined by an anaesthetist reviewing the medical record. 19 Using the Cohen’s kappa statistic, levels of agreement were defined as near perfect (0.81–1.0); substantial (0.61–0.8); moderate (0.41–0.6); fair (0.21–0.4); and poor (0–0.2). 23
Descriptive statistics were used to describe baseline characteristics and clinical outcomes in the study population. Cohen’s kappa statistic was used to assess the level of agreement between the two methods of determining each postoperative complication. Confidence intervals (CI) were computed by bootstrapping and a test of statistical significance was performed. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were calculated for the ICD-10-AM diagnosis of each complication endpoint using the anaesthetists’ diagnosis as the clinical reference standard. Inter-rater agreement was assessed within each of the four pairs of anaesthetists by Cohen’s kappa, where there was at least one incidence of a complication diagnosed by one member of any pair. A mean kappa statistic was calculated for each complication by averaging the kappa values for each pair of investigators, because there were the same number of records in each pair. Statistical analysis was performed using SPSS version 25.0.0.1 (IBM Corporation, Armonk, NY, USA, 2017). The study is reported in accordance with Benchimol and colleagues’ criteria for reporting validation studies of administrative data, adapted from the Standards for Reporting of Diagnostic Accuracy framework.24,25
Results
Of 510 patients referred to the anaesthesia postoperative service during the study period, 28 were found to not meet inclusion criteria during the medical record review process or were unable to be matched with corresponding ICD-10-AM codes. The remaining 482 patients were included in the analysis (Figure 2). Medical record review was conducted between September 2017 and February 2018.
Study population.
The median age was 74 years, with 200 (42%) being female (Table 1). A total of 212 patients (44%) underwent emergency surgery, with the remaining 270 (56%) undergoing scheduled elective surgery. The majority of patients had a high American Society of Anesthesiologists Physical Status (ASA PS), with 318 (66%) being ASA PS 3, 90 (19%) ASA PS 4 and two (0.4%) ASA PS 5. The most common surgical specialties represented were orthopaedic (n = 138, 29%), colorectal (n = 75, 16%), upper gastrointestinal (n = 72, 15%), vascular (n = 52, 11%) and neurosurgery (n = 37, 8%). Patient characteristics are described in detail in Table 1.
Patient characteristics.
ASA: American Society of Anesthesiologists; IQR: interquartile range; PS: Physical Status.
The incidence of complications identified by anaesthetist chart review is presented in Table 2. Thirteen patients (2.7%) died within 30 days of surgery. The most common complications were constipation (24.3%), delirium/confusion (17.6%), pneumonia/aspiration pneumonitis (10.9%) and acute kidney injury (10.6%).
In-hospital major complications and 30-day mortality.
SIRS: systemic inflammatory response syndrome; TIA: transient ischaemic attack; CPR: cardiopulmonary resuscitation.
aNational Surgical Quality Improvement Program (NSQIP) major complications. 9
Substantial agreement was observed between investigators and coding data for five major postoperative complications: ileus and obstruction (kappa 0.69, 95% CI 0.53–0.81), myocardial infarction (kappa 0.69, 95% CI 0.40–0.88), cardiac arrest requiring cardiopulmonary resuscitation (CPR) (kappa 0.67, 95% CI 0.00–1.00), pneumonia/aspiration pneumonitis (kappa 0.66, 95% CI 0.54–0.76) and stroke or transient ischaemic attack (TIA) (kappa 0.62, 95% CI 0.22–0.87), with p-values <0.001. Moderate agreement was observed between investigators and coding data for delirium/confusion (kappa 0.57, 95% CI 0.47–0.66), acute renal failure (kappa 0.53, 95% CI 0.41–0.63) and surgical site infection (kappa 0.42, 95% CI 0.28–0.56), with p-values <0.001. Fair agreement was observed for arrhythmia, heart failure, deep venous thrombosis, major wound disruption, vascular graft failure and sepsis/septic shock/severe inflammatory response syndrome (SIRS). Poor agreement was observed for constipation and hypothermia. Cohen’s kappa was unable to be calculated for pulmonary embolus because there were no events in the cohort. Estimates of diagnostic accuracy for all outcomes are shown in Table 3 and visually in Figure 3.
Agreement between anaesthetists and International Classification of Disease (10th edition) Australian Modification (ICD-10-AM) codes.
PPV: positive predictive value; NPV: negative predictive value; SIRS: systemic inflammatory response syndrome; TIA: transient ischaemic attack; CPR: cardiopulmonary resuscitation; NE: not estimable; CI: confidence intervals.
aNational Surgical Quality Improvement Program (NSQIP) major complications. 9
bAgreement could not be calculated for pulmonary embolism because no cases were diagnosed by anaesthetists.
Agreement between anaesthetists and International Classification of Disease (10th edition) Australian Modification (ICD-10-AM) codes. SIRS: severe inflammatory response syndrome; TIA: transient ischaemic attack; CPR: cardiopulmonary resuscitation; NSQIP: National Surgical Quality Improvement Program.
Sensitivity ranged from 100% for cardiac arrest requiring CPR to 0% for hypothermia. Sensitivity was highest for cardiac arrest requiring CPR (100%), pneumonia/aspiration pneumonitis (75.0%), acute renal failure (70.0%), ileus and obstruction (67.7%) and delirium/confusion (66.7%). Specificity was above 90% for all conditions and above 99% for myocardial infarction, stroke or TIA, cardiac arrest requiring CPR, pulmonary embolism, deep venous thrombosis, vascular graft failure and hypothermia. Sensitivity and specificity of coding data compared with the clinical reference standard are shown in Table 3. PPVs ranged from 80% (myocardial infarction) to 0% (hypothermia) and NPVs were above 90% for all except two conditions (Table 3).
Within pairs of anaesthetists, kappa values ranged from below zero to 1.00 with wide CIs throughout, indicating the full range of possible inter-rater agreement from poor to almost perfect. Averaged kappa values indicated near perfect agreement for ileus and obstruction and substantial agreement for delirium/confusion, arrhythmia, stroke or TIA, deep venous thrombosis and major wound disruption. Remaining conditions showed moderate agreement (weighted kappa 0.41–0.60, three conditions) or fair agreement (weighted kappa 0.21–0.40, two conditions), with one condition (sepsis/septic shock/SIRS) showing poor agreement (weighted kappa 0.00–0.20). Results are shown in Supplementary Table 3. In Figure 4, averaged kappa values for inter-rater reliability are superimposed on the image showing agreement between anaesthetists and ICD-10 codes.
Agreement between anaesthetists and International Classification of Disease (10th edition) Australian Modification (ICD-10-AM) codes. SIRS: severe inflammatory response syndrome; TIA: transient ischaemic attack; CPR: cardiopulmonary resuscitation; NSQIP: National Surgical Quality Improvement Program.
Discussion
Neither administrative data nor clinical audit were highly reliable methods of identifying major postoperative complications in our study. Near-perfect diagnostic agreement between ICD-10-AM codes and anaesthetist chart review did not occur for any postoperative complications. Although substantial agreement was observed for five conditions (myocardial infarction, stroke or TIA, pneumonia/aspiration pneumonitis, cardiac arrest requiring CPR, ileus and obstruction), the lower limit of their CIs included moderate, fair or poor agreement. Agreement for all other conditions was moderate to fair at best. Surprisingly, inter-anaesthetist agreement also failed to reach the anticipated near-perfect agreement. This reinforces the concept that neither administrative data nor retrospective clinical audit are a substitute for meticulous, prospectively collected outcomes data by trained experts in accordance with a protocol. 26
These results suggest using ICD-10-AM codes to identify postoperative complications at our hospital underestimates the incidence of complications compared to diagnosis by retrospective clinical audit and that retrospective clinical audit itself is not a highly reliable method. The sensitivity and PPV of ICD-10-AM compared with the clinical reference standard were modest at best. The wide range of sensitivities observed suggests significant variation in the degree of under-diagnosis between different complications. While inter-anaesthetist agreement was higher than agreement between anaesthetists and ICD-10 codes, near-perfect inter-rater reliability between pairs of anaesthetists was only present for one condition, raising the possibility that under-diagnosis also occurred during retrospective clinical audit. In contrast to the modest sensitivity observed, specificity was above 90% for all conditions and above 99% for seven conditions. NPV was similarly high, making a high false positive rate unlikely. The moderate sensitivity (61.5%–75%) of four of the five major complications that did show substantial agreement between administrative data and clinical audit limits their utility for internal or external benchmarking purposes if the variable reliability observed in our data occurs within and between hospitals over time. The sole major complication with both substantial agreement and high sensitivity (100%) was cardiac arrest requiring CPR, an event extremely rare in the perioperative setting (incidence 0.2% in this cohort).
A limitation of this study is the relatively small sample size and event rate, which may have resulted in lower kappa values and wider confidence limits than would have occurred in a larger sample. For example, there was only one case of cardiac arrest requiring CPR, resulting in the widest possible confidence limits (0.0–1.0). Cohen’s kappa can only be calculated in instances where at least one reviewer diagnosed at least one complication. Due to the small sample size for inter-rater reliability (20 records for each pair of reviewers), the kappa was calculated for less than half the possible combinations of reviewers and complications.
These results are consistent with the international literature, which demonstrates variable and inconsistent under-diagnosis of major perioperative complications depending on the outcome being measured. In 2011 a comparison was conducted to compare adverse events in 7606 surgical patients at the Mayo Clinic. 6 Events were identified by administrative data from the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ PSI), using a series of algorithms based on ICD-9-CM (International Classification of Diseases, Ninth Revision, Clinical Modification) codes, and prospectively collected ACS NSQIP data. The administrative data identified adverse events among 3.5% of patients, compared to NSQIP data’s 7.5% event rate and had variable but predominantly modest sensitivity and PPVs. This analysis was repeated in 4583 surgical patients at the Cleveland Clinic in 2012 with similar results and the authors highlighting the limitations of using administrative data for secondary purposes without standardising definitions, data collection and management. 7
At the University of Calgary in 2013, agreement was examined between AHRQ PSI–defined complications and ICD-10-CM codes, using full medical chart review as the reference standard among 334 surgical patients. PPVs ranged from 12.5% for sepsis through to 89.5% for venous thromboembolism. 8 At the same institution in 2016, agreement between coding and clinical record review for surgical site infection among 162 elective primary total hip or knee arthroplasty patients found administrative data had a sensitivity of 90% and specificity of 99%. 9 In 2015 in the United Kingdom, agreement between NSQIP and Hospital Episode Statistics data on 1323 surgical patients ranged from poor to moderate (kappa 0.00 to 0.56) for nine different postoperative complications. 10 The Australian coding literature does not provide reassuring evidence that hospital administrative data are consistently coded to begin with, reporting that 13%–61% of original coding diagnoses required recoding and significant interstate variation in the coding of important secondary diagnoses.11,13 Multiple studies have nevertheless attempted to estimate the incidence of in-hospital perioperative complications from Australian administrative data.16,27,28
The relative importance of different summary measures of diagnostic accuracy—sensitivity, specificity PPV and NPV—depends on the intended use of the data. 29 A high sensitivity is important when the goal is to identify all instances of a condition in the population, such as with a screening test. Diagnostic tests used in clinical practice or in a clinical trial require both high sensitivity and specificity. Although administrative data are likely to underestimate the true incidence of major postoperative complications, when used for the purpose of hospital funding and reimbursement the trade off in time and cost savings may be reasonable. Acknowledging the limitations of administrative data, we believe our perioperative registry mapped to administrative data will be useful for identifying risk factors, hypothesis generation and sample size calculation for future research. For the purpose of benchmarking hospital performance and public reporting of hospital outcomes, more reliable data will be necessary. After this study was undertaken, our institution commenced recording comorbidities and in-hospital diagnoses in the electronic medical record using SNOMED CT codes, 30 which may provide a more reliable method of measuring major postoperative complications and presents an opportunity for further research both at our institution and at other Australian hospitals using this method.
In Australia there are 2.7 million hospital admissions involving surgery each year and one in four people who are hospital inpatients experience an avoidable complication.31,32 Our study has highlighted the limitations of both administrative data and retrospective clinical audit in identifying major postoperative complications, leading us to conclude that the most accurate method will be a clinical quality registry with prospectively collected baseline and outcomes data, using explicit definitions, specifically trained staff and systematic data-quality controls. 33 Although registries are resource-intensive, the return on investment in clinical quality registries in Australia is between two and seven dollars for every dollar spent, creating health economic savings that can be used to fund additional episodes of healthcare or research. 34 We believe a perioperative clinical quality registry is necessary to validly and reliably measure major postoperative complications in Australia for benchmarking of hospital performance and before public reporting of outcomes should be considered.
Supplemental Material
sj-pdf-1-aic-10.1177_0310057X20905606 - Supplemental material for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications
Supplemental material, sj-pdf-1-aic-10.1177_0310057X20905606 for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications by Jennifer R Reilly, Mark A Shulman, Annie M Gilbert, Bismi Jomon, Robin J Thompson, Jonathon J Nicholson, Justin A Burke, Daragh N Lehane, Chen-Mai Liaw, Adam J Mahoney, Peter A Stark, Lise Hales and Paul S Myles in Anaesthesia and Intensive Care
Supplemental Material
sj-pdf-2-aic-10.1177_0310057X20905606 - Supplemental material for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications
Supplemental material, sj-pdf-2-aic-10.1177_0310057X20905606 for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications by Jennifer R Reilly, Mark A Shulman, Annie M Gilbert, Bismi Jomon, Robin J Thompson, Jonathon J Nicholson, Justin A Burke, Daragh N Lehane, Chen-Mai Liaw, Adam J Mahoney, Peter A Stark, Lise Hales and Paul S Myles in Anaesthesia and Intensive Care
Supplemental Material
sj-pdf-3-aic-10.1177_0310057X20905606 - Supplemental material for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications
Supplemental material, sj-pdf-3-aic-10.1177_0310057X20905606 for Towards a national perioperative clinical quality registry: the diagnostic accuracy of administrative data in identifying major postoperative complications by Jennifer R Reilly, Mark A Shulman, Annie M Gilbert, Bismi Jomon, Robin J Thompson, Jonathon J Nicholson, Justin A Burke, Daragh N Lehane, Chen-Mai Liaw, Adam J Mahoney, Peter A Stark, Lise Hales and Paul S Myles in Anaesthesia and Intensive Care
Footnotes
Acknowledgements
None.
Author contributions
JR, MS and PS designed the study and drafted the manuscript. All authors provided input into the manuscript. JR, MS, PS, AG, BJ and RT designed and developed the perioperative registry. JR, MS and LH determined the complications and corresponding ICD-10-AM codes. JR wrote the study protocol, obtained ethics approval and designed and developed the study database. DL piloted the study protocol and database. JR, MS, JN, JB, DL, CML, AM and PS undertook clinical chart review. JR and PM undertook data analysis. AG, BJ and RT mapped the perioperative registry to the study database, tested and refined the mapping and uploaded matched registry data to the study database.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by an Australian and New Zealand College of Anaesthetists Project Grant (MS), an Australian National Health and Medical Research Council Practitioner Fellowship (PM), and an Australian Government Research Training Program Scholarship (JR).
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
