Are We Confident of Across-Hospital Mortality Comparisons?

Hospital mortality rates have lately been proposed for greater prominence in measuring the quality of inpatient care. The Medicare Payment Advisory Commission (MedPAC) has advocated using hospital mortality measures, both as part of their proposed redesign of the Merit-based Incentive Payment System in which mortality rates after hospital stays are compared by physician ¹ and in redesigning/consolidating hospital pay-for-performance programs by eliminating hospital complication measures and more heavily weighting hospital mortality rates. ² The Government Accountability Office agrees with the need to emphasize the importance of mortality in quality measurement, ³ while many health policy researchers concur that the current system of quality measurement undervalues mortality outcomes. ⁴

Although pressure may be mounting to do more with mortality, we should take a step back and consider the following questions:

1. How “accurate” or reproducible are comparative mortality rates (across hospitals), and how actionable is the rate being calculated?

2. How effective is the risk adjustment in standardizing in-hospital deaths—is the calculation of expected deaths reasonable, accurate, and appropriate?

3. Is mortality information presented in such a way as to be usable by clinicians?

Issues

Studies have estimated that between 4% and 8% of hospital deaths are actually preventable.^5,6 If we extrapolate from these estimates, then we can predict that Medicare admissions for heart failure (with an observed 30-day mortality rate of 11.6% ⁷ ) have an underlying 0.97% “preventable” mortality rate at the upper level of 8%. If these estimates hold, then a hospital with 500 heart failure admissions may expect 58 deaths, 5 of which may be preventable. Taking the typical Poisson distribution for hospital mortality, we can expect the standard deviation of 58 deaths to be 7.6 and thereby larger than the number of preventable deaths. This is often termed a weak “signal-to-noise ratio” but essentially means that, except in extreme circumstances where preventable deaths are rampant, we are not measuring relative performance but rather random results.

Therefore, risk adjustment is essential to ensure that comparisons are accurate and fair. Multiple factors contribute to variation in mortality rates. Factors influencing mortality rates include variation in the reason for and severity of the condition that provoked the hospitalization, the status or prior progression of the underlying condition when patients are admitted because of acute deterioration of an underlying chronic condition, variation in comorbidity (or the number of additional organ systems affected by chronic conditions), and variation in patients’ average physiologic reserve, or functional status (exercise capacity, pulmonary function, cardiac function). Less obviously, but of at least equal importance, is the extent to which the underlying condition is amenable to treatment.

The last aspect, the amenability of the condition to treatment, is particularly important if we wish to get to actionable (preventable) mortality rates. The ability (and desirability) to prevent mortality also requires consideration of how we treat admissions for palliative care and those patients with do not resuscitate orders. Coding for these types of admission is variable across hospitals,and their inclusion can have a substantial impact on comparisons. ⁸ Current Medicare risk-standardized mortality rates (RSMR) do not adjust for the coding of palliative care; instead,

If, as a result, a hospital’s RSMR is higher than expected, the hospital may choose to share the reason publicly and engage its community in a discussion of the hospital’s role in end of life care. ⁹

So How Close Are We to Doing This With Precision?

Surgical admissions, for which mortality might typically be an unexpected and unwarranted outcome, seem well placed for across-hospital comparisons. Risk adjustment is required, not only to separate out higher risk cases (trauma, coronary procedures for acute ischemia, revascularization for peripheral vascular acute occlusion) from elective surgeries but also to adjust for the burden of underlying comorbid illness. With modest exclusions (eg, the exclusion for preoperative cardiogenic shock in the New York State cardiac data system ¹⁰ ), preventable and observed mortality rates for surgical admissions will converge ⁵ to a much greater extent than for medical admissions. Providers will be both able and most likely willing to respond to information that compares them to peer providers with similar caseloads.

Medical admissions, both for acute medical events (eg, pneumonia, stroke) and acute decompensation of chronic illnesses (eg, heart failure, decompensation of chronic lung disease), create more complexity for across-hospital comparison. For these admissions the inability to identify “preventable” mortality from “any” mortality will confound results while the risk-adjustment model likely will require routine data elements not readily available to date (eg, stroke scale, ejection fraction). This does not mean that hospitals cannot monitor their own and their physicians’ performance, but simply that drawing inference from across-hospital comparisons is inexact with these limitations.

A separate consideration is whether, particularly for patients with chronic conditions being managed by care teams operating outside the hospital, mortality rates for medical admissions should more appropriately be viewed as part of longer term care management with the specified care team assessed over a longer period (eg, a year), not just subsequent to a hospital admission.

Delineation between medical and surgical admissions has not been surfaced in current proposals looking to make mortality measures “more reliable.” The approach, championed by MedPAC, is to include many different admission types within a single mortality measure. Although appealing in terms of increasing the sample size, there should be concern that performance in treating one case type offsets that of another. This is particularly troubling when blending medical and surgical admissions without regard to their underlying relative preventability.

Companion Reforms

Things can be done in addition to model design to make mortality measures both more meaningful and more actionable.

First, there is insufficient information provided to hospitals to allow self-assessment and ongoing monitoring of mortality. Mortality rates are standardized within a fixed period post admission or post procedure, which introduces a period of time post discharge for which the hospital lacks knowledge. Payers, therefore, should be required to notify hospitals of patient outcomes (such as mortality) evaluated within a fixed period post discharge. A simple notification to a hospital of patient status 30 or 90 days after discharge or transfer would enable hospitals to compile their own mortality rates. Additionally, risk-adjustment models may utilize variables that are unknown to the hospital. These data should be released to the hospital such that patient risk can be better understood and replicated.

Second, measurement should extend to as many payers as possible. Minimally, comparisons should not be restricted to Medicare Fee for Service as the basis of current Medicare hospital measures, but include Medicare Advantage. By incorporating all admissions we begin to address sample size issues and focus on a more complete picture of potential performance issues.

Third, we should consider adjusting results for variation in socioeconomic status (SES)—or at least measure if SES has an impact on our refined measure. By including a post-discharge period within the mortality measure, we include a period for which the patient is managed in the community and is thereby reliant on the adequacy of community resources and social support.

And for the Mortality Measures Themselves . . .

For now, we should limit any pay for performance adjustment for comparative mortality rates to surgical admissions with the inclusion of a modest number of exclusions. Medical admissions require more careful handling than that which has presently been exhibited, both in defining what constitutes a preventable death and the structure of the underlying risk-adjustment model, before we seek to draw strong inference from across hospital comparisons. As part of that evolution, we need to assess whether hospitals offer the proper focus for our efforts to lower mortality rates or whether mortality measures for at least some medical conditions need to move beyond an event focus and into population health.