Reliability assessment of a hospital quality measure based on rates of adverse outcomes on nursing units

Abstract

The purpose of this study was to develop methods for assessing the reliability of scores on a widely disseminated hospital quality measure based on nursing unit fall rates. Poisson regression interactive multilevel modeling was adapted to account for clustering of units within hospitals. Three signal-noise reliability measures were computed. Squared correlations between the hospital score and true hospital fall rate averaged 0.52 ± 0.18 for total falls (0.68 ± 0.18 for injurious falls). Reliabilities on the other two measures averaged at least 0.70 but varied widely across hospitals. Parametric bootstrap data reflecting within-unit noise in falls were generated to evaluate percentile-ranked hospital scores as estimators of true hospital fall rate ranks. Spearman correlations between bootstrap hospital scores and true fall rates averaged 0.81 ± 0.01 (0.79 ± 0.01). Bias was negligible, but ranked hospital scores were imprecise, varying across bootstrap samples with average SD 11.8 (14.9) percentiles. Across bootstrap samples, hospital-measure scores fell in the same decile as the true fall rate in about 30% of cases. Findings underscore the importance of thoroughly assessing reliability of quality measurements before deciding how they will be used. Both the hospital measure and the reliability methods described can be adapted to other contexts involving clustered rates of adverse patient outcomes.

Keywords

Health care quality quality measurement reliability

Get full access to this article

View all access options for this article.

References

Staggs

Gajewski

. Bayesian and frequentist approaches to assessing reliability and precision of health-care provider quality measures. Stat Meth Med Res 2015. Epub ahead print 17 March.

Paddock

. Statistical benchmarks for health care provider performance assessment: a comparison of standard approaches to a hierarchical Bayesian histogram-based method. Health Serv Res 2014; 49: 1056–1013.

Staggs

Mion

Shorr

. Consistent differences in medical unit fall rates: Implications for research and practice. J Am Geriatr Soc 2015; 63: 983–987.

Christiansen

Morris

. Hierarchical Poisson regression modeling. J Am Stat Assoc 1997; 92: 618–632.

Adams JL. The reliability of provider profiling: A tutorial. Santa Monica, RAND Corporation, 2009.

Adams

Mehrotra

Thomas

et al.

Physician cost profiling–reliability and risk of misclassification. N Engl J Med 2010; 362: 1014–1024.

Centers for Medicare & Medicaid Services. Memorandum: Results of reliability analysis from Mathematica Policy Research. Baltimore, MD, U.S. Department of Health & Human Services, 2012, http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/hospital-value-based-purchasing/Downloads/HVBP_Measure_Reliability-.pdf (accessed 15 July 2015).

Dunton

Staggs

. Unit-level time trends in inpatient fall rates of US hospitals. Med Care 2012; 50: 801–807.

Waters

Daniels

Bazzoli

et al.

Impact of Medicare’s nonpayment for hospital acquired conditions: Lessons for future policy?

JAMA Intern Med 2015; 175: 347–354.