There is an increasing use of statistical methods, such as funnel plots, to identify poorly performing healthcare providers. Funnel plots comprise the construction of control limits around a benchmark and providers with outcomes falling outside the limits are investigated as potential outliers. The benchmark is usually estimated from observed data but uncertainty in this estimate is usually ignored when constructing control limits. In this paper, the use of funnel plots in the presence of uncertainty in the value of the benchmark is reviewed for outcomes from a Binomial distribution. Two methods to derive the control limits are shown: (i) prediction intervals; (ii) tolerance intervals. Tolerance intervals formally include the uncertainty in the value of the benchmark while prediction intervals do not. The probability properties of 95% control limits derived using each method were investigated through hypothesised scenarios. Neither prediction intervals nor tolerance intervals produce funnel plot control limits that satisfy the nominal probability characteristics when there is uncertainty in the value of the benchmark. This is not necessarily to say that funnel plots have no role to play in healthcare, but that without the development of intervals satisfying the nominal probability characteristics they must be interpreted with care.
BlackN. Assessing the quality of hospitals: hospital standardised mortality ratios should be abandoned. Br Med J2010; 340: 933–934.
7.
LilfordRPronovostP. Using hospital mortality rates to judge hospital performance: a bad idea that just won't go away. Br Med J2010; 340: 955–957.
8.
HendersonGMeadGVan DijkeMet al.Use of statistical process control charts in stroke medicine to determine if clinical evidence and changes in service delivery were associated with improvements in the quality of care. Qual Saf Health Care2008; 17: 301–306.
9.
BenneyanJLLoydRPlsekP. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care2003; 12: 458–464.
10.
SpiegelhalterDSherlaw-JohnsonCBardsleyMet al.Statistical methods for healthcare regulation: rating, screening and surveillance. J R Stat Soc Ser A2012; 175: 1–25.
11.
SpiegelhalterD. Funnel plots for comparing institutional performance. Stat Med2005; 24: 1185–1202.
12.
MohammedADeeksJ. In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann Thorac Surg2008; 86: 348–355.
13.
National Clinical Audit Advisory Group. Detection and management of outliers, London: Department of Health, 2011.
EggerMDavey SmithGSchneiderMet al.Bias in meta-analysis detected by a simple, graphical test. BMJ1997; 315: 629–629.
16.
ShewhartW. Economic control of quality of manufactured product, New York: D. Van Nostrand Company, Inc, 1931.
17.
PearsonE. The application of statistical methods to industrial standardization and quality control, London: British Standards Institute, 1935.
18.
Neave H and Wheeler D. Shewhart’s charts and the probability approach. In: Ninth annual conference of the British Deming Association, London, 1996.
19.
SeatonSManktelowB. The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio. BMC Med Res Methodol2012; 12: 98–98.
20.
ManktelowBSeatonS. Specifying the probability characteristics of funnel plot control limits: an investigation of three approaches. PLoS One2012; 7: e45723–e45723.
MikuleckyM. Confidence and tolerance intervals – a tool for biomedical data analysis aimed at clear evidence. Am J Cardiol2004; 13: 211–215.
23.
NeymanJ. On the problem of confidence intervals. Ann Math Stat1935; 6: 111–116.
24.
HahnGChandraR. Tolerance intervals for Poisson and binomial variables. J Qual Technol1981; 13: 100–111.
25.
AgrestiACoullB. Approximate is better than “exact” for interval estimation of binomial proportions. Am Stat1998; 52: 119–126.
26.
New Jersey Department of Health and Senior Services. Cardiac surgery in New Jersey 2004, Trenton, NJ: New Jersey Department of Health and Senior Services, 2007.
27.
MayerEBottleARaoCet al.Funnel plots and their emerging application in surgery. Ann Surg2009; 249: 376–383.
28.
KirkhamJBouamraO. The use of statistical process control for monitoring institutional performance in trauma care. J Trauma2008; 65: 1494–1501.
JarmanBGaultSAlvesBet al.Explaining differences in English hospital death rates using routinely collected data. BMJ1999; 318: 1515–1520.
31.
GirlingAHoferTWuJet al.Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf2012; 21: 1052–1056.
32.
MohammedMDeeksJGirlingAet al.Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ2009; 338: b780–b780.
33.
PenfoldRDeanSFlemonsWet al.Do hospital standardized mortality ratios measure patient safety? HSMRs in the Winnipeg regional health authority. Healthcare Pap2008; 8: 8–24.
34.
ScottIBrandCPhelpsGet al.Using hospital standardised mortality ratios to assess quality of care – proceed with extreme caution. Med J Aust2011; 194: 645–648.
35.
AzizOFinkDHobbsLet al.Hospital mortality under surgical care. Ann R Coll Surg Engl2011; 93: 193–200.
36.
ShojaniaKForsterA. Hospital mortality: when failure is not a good measure of success. Can Med Assoc J2008; 179: 153–157.
