Abstract
Quality measurement plays a significant role in the modern quality movement. Considerable attention has been devoted to the development of quality indicators to monitor health care performance and support quality improvement in service of quality agendas.1–6
The rapid growth of quality indicators has presented challenges in practice. With so many indicators, it is not always clear which indicators to direct attention toward. Performance improvement in some indicators may not necessarily translate into a meaningful improvement in a clinical outcome. 7 Additionally, the high volume of indicator reporting can incur considerable administrative burden on health care providers. 8
The development of quality indicators typically begins with identifying candidate indicators through a literature review and the development of a conceptual model for the area of interest, followed by specification of indicator definitions and collection of evidence supporting the indicators. Quality indicators are often selected to monitor various dimensions of quality,9,10 using consensus-based methods such as modified Delphi or Nominal Group processes. However, health outcome and cost implications of quality indicators, which are key objectives in most quality agendas, are often not given serious consideration. Health economics provides a useful framework for decision makers to evaluate alternative ways of providing health care by relating health benefits with the costs incurred in their production. The economic framework can be extended to the task of selecting quality indicators to explicitly consider how population health outcomes and costs vary with performance. 11 This approach may have particular utility in major chronic diseases, which are areas of interest to policy makers given their large burdens of disease. 12
In this article, we explore the potential utility of applying a health economic framework to prioritize a set of disease-specific quality indicators by the expected variations in population health outcomes and costs, using smoking cessation in chronic obstructive pulmonary disease (COPD) as a proof-of-concept example. Smoking cessation is an important element in COPD management with the largest potential to influence the natural history of the disease. 13
Methods
We developed a simulation model to represent smoking cessation practices among persons with COPD in Ontario, Canada. The model was used to simulate changes in performance (monitored by quality indicators) and the expected change in smoking abstinence. Health outcomes and costs, representing health care resource utilization, varied as a function of smoking abstinence. The joint variation in health and costs associated with indicator performance was represented by the incremental net health benefit (INHB). INHB is a metric that expresses health gains minus their costs in units of health by converting costs into health gains foregone due to the resources that are consumed. 14 Health outcome was denominated in terms of quality-adjusted life years (QALYs), a composite measure of quantity and quality of life. 15
Though documentation of quality indicators are not interventions, they represent the rate at which practices are performed or outcomes are achieved. Hereafter, we refer to changes to quality indicators to imply changes in associated performance. In this example, we explore the potential relationship between performance and INHB by simulating maximum/optimal changes in indicator performance.
Model Structure
A microsimulation, health state transition model was developed, using TreeAge Pro 2018, 16 based on data from the existing literature to predict outcomes of improved quality indicators for smoking cessation in COPD. All estimated parameters were converted to a 3-month cycle length probability. A half-cycle correction was applied using the cycle tree method. 17 A provincial government payer perspective for Ontario, Canada, was adopted, and a lifetime horizon was employed in the analysis. A discount rate of 1.5% was applied to costs and benefits, and costs were expressed in 2016 Canadian dollars. 18
The model was conceptualized into five health states: stable, current smoker; stable, former smoker; stable, never smoker; exacerbation; and death (Figure 1). Depending on the current health state, individuals could quit smoking or relapse. Never-smokers were assumed to remain nonsmokers throughout the duration of the simulation but were subject to smoking status assessments.
Health state transition model.
Quality Indicators
A comprehensive set of quality indicators for smoking cessation were identified from a systematic review of quality indicators for COPD. 19 Indicator performances were represented as the probabilities of persons receiving clinical practices (represented by process indicators) or experiencing an outcome (represented by outcome indicators). One process indicator, assessing a patient’s secondhand smoke exposure, was considered beyond a health care practitioner’s reasonable capacity to influence directly and was excluded. In addition, the quality indicators for smoking cessation encouragement and smoking cessation support were considered similar enough to smoking cessation counseling and smoking cessation intervention, respectively, to combine as single indicators. Out of 11 smoking cessation indicators, 8 quality indicators for smoking cessation in COPD were included in the final model.
Process indicators were represented by clinical practices occurring within health states (Figure 2). The model structure explicitly incorporated the dependence of the quality indicators with each other. A smoker must have received a smoking status assessment before receiving encouragement to quit or a readiness to quit assessment. An assumption that the patient must be known to be ready to quit before receiving a referral to a smoking cessation intervention (modelled as nicotine replacement therapy plus counselling for this example) was used. Encouragement to quit was only conditional on smoking status assessment. If the patient experienced a severe exacerbation resulting in a hospital admission, they must have received a smoking status assessment before receiving an inpatient smoking cessation intervention. Smokers could make an unassisted attempt to quit at any time. Assessments of smoking status and readiness to quit conferred benefits by identifying eligible patients for smoking cessation interventions. Encouragement to quit and smoking cessation interventions conferred benefits by increasing the probability of a successful quit attempt.20,21 Additional details on the structure of the quality indicators in the model are provided in the supplementary materials.
Flow diagram of process indicators in the model. Process indicators influence the occurrence of other process indicators, intermediate events, and the abstinence rate.
Whether a patient would receive a clinical practice associated with a process indicator was determined stochastically. For this example, the model employed a simple assumption that patients visited the same provider and that individual provider performance was consistent. Thus, if a practice was applied to a patient it would always be applied if the patient was eligible.
Two outcome indicators, annual quit rate and quit rate six months after discharge from a hospital admission, were modelled by changing the probability of a successful quit attempt such that a smoker would successfully quit by the end of the year or within 6 months after discharge. Any patient who quit smoking was still subject to a probability of relapse.
Data
Natural History of COPD
Demographic data for age, sex, and smoking status were obtained from studies of Ontario COPD patients to produce cohort representative of the Ontario COPD population.22,23 Other model inputs were derived from studies in published literature. A list of model parameters and their data sources are provided in the supplementary materials.
Disease progression was modelled through the decline in lung function, represented by the patient’s forced expiratory volume in 1 second (FEV1). 24 Disease severity was defined using the percentage of the patient’s predicted normal FEV1 and the Global Initiative for Chronic Obstructive Lung Disease (GOLD) classification. 13 Starting FEV1 values were sampled to mimic proportions of GOLD stages observed in a Canadian cohort. 25 The probabilities of exacerbations were modelled according to disease severity. 26 Severe exacerbations were defined as exacerbations resulting in a hospitalization. 27 Mortality was modelled as a function of age, sex, disease severity, and smoking status and implicitly represents both COPD and non-COPD effects of smoking cessation. 28
Smoking Cessation
The effect of smoking cessation was modelled as a transition from the current smoker to the former smoker health state, where disease progression reflects the trajectory of former smokers. Encouragement to quit was modelled as brief advice and was modelled separately from the smoking cessation intervention indicator. 20 Outpatient smoking cessation interventions were modelled as nicotine replacement therapy (NRT) plus counselling, with placebo arms from NRT trials used for baseline abstinence rates. 21 Other smoking cessation interventions were not included for simplicity and demonstrative purposes. Inpatient smoking cessation interventions were also modelled as NRT plus counselling, based on the Ottawa Model for Smoking Cessation. 29 Durations of effect for all smoking cessation interventions were modelled as 6 months, which corresponded to the treatment regimen of NRT and the follow-up times from studies of inpatient smoking cessation and brief advice.20,21,29 There was no limit to the number of times a patient could receive any smoking cessation practice.
The probability of relapse was derived from the cumulative probability of not remaining abstinent using declining sustained abstinence rates over 5 years published from the Lung Health Study.21,30 Individuals who were continuously abstinent for 5 years were assumed to have permanently quit smoking. 31 The probabilities of quitting or relapse were independent of previous quit attempts or durations of abstinence.
Quality Indicators
There was limited information on the clinical performance of the selected quality indicators for Ontario. Thus, baseline performance levels were obtained from published survey data or published studies of clinical practice patterns of select hospitals in Ontario.29,32–34 For the indicator monitoring the proportion of patients whose smoking status is assessed at admission for an exacerbation, a baseline performance level could not be obtained. In this case, the performance level was set to match a related indicator, the proportion of smokers who receive a smoking cessation intervention at admission for an exacerbation.
Quality of Life
The impact of COPD on quality of life was proportional to disease severity. Health utilities were applied according to disease severity. 35 Exacerbations had a negative impact on quality of life with a QALY penalty applied for each exacerbation occurrence, also according to severity.
Costs
Costs were modelled as health resource utilization and included the costs of performing smoking cessation practices monitored by the indicators, maintenance costs of usual care, exacerbation costs, and a cost of death. All costs were converted to 2016 Canadian dollars using the Statistics Canada Consumer Price Index. 36
Costs related to quality indicator performance included physician billing amounts for performing clinical practices. 37 Costs incurred from smoking cessation practices were subtracted from the annual maintenance costs to avoid double counting. Costs related to implementation of quality indicators or quality improvement interventions were not included in this exercise.
The cost of the outpatient smoking cessation intervention was estimated from a typical regimen of nicotine gum and up to 6 months of treatment. 38 Two follow-up counselling sessions were assumed to accompany the application of NRT. Costs of a smoking cessation intervention at hospital admission were based on an economic evaluation of the Ottawa Model for Smoking Cessation (OMSC) in Ontario. 39 The published costs of the OMSC intervention implicitly included the assessment of smoking status at admission.
Annual maintenance costs of usual care and exacerbation costs were stratified by severity and obtained from an economic evaluation of COPD interventions in Ontario, Canada. 38 The cost of death was modelled as the cost in the final month of life. 40
Analysis
Variations in lifetime QALYs and costs associated with each indicator were first estimated by comparing the simulated baseline performance of each indicator with its highest/optimal theoretical value (100%), while holding all other indicators at their baseline performances. Second, variations in QALYs and costs were estimated across the entire theoretical performance range of each indicator (0% to 100%). Finally, performances of closely related indicators were varied in combination.
Incremental net health benefit was calculated as the change in lifetime QALYs minus the change in lifetime costs, represented as QALYs, over the variation in indicator performance (INHB = ΔQALY −ΔCost/Willingness to pay for additional QALY). To convert costs to QALYs, the value of an additional QALY was set at $50,000, a value within the common threshold range of $20,000 to $100,000. 41
This study was not conducted with any external funding source.
Results
Table 1 shows the starting characteristics of the COPD patient cohort. The cohort was mild-moderate in severity with approximately half being current smokers. Also shown are the starting quality indicator performance levels. Variations in outcomes associated with changes in performance for each quality indicator are outlined in Table 2. Outcomes associated with individual indicators reflect variations associated with changes in performance of the individual indicator while all other indicators remain at their baseline performances. Additionally, variations in multiple process indicators are displayed.
Starting Characteristics of COPD Patients and Quality Indicator Performance Levels in COPD Simulation Model
COPD, chronic obstructive pulmonary disease; FEV1, forced expiratory volume in 1 second; GOLD, Global Initiative for Chronic Obstructive Lung Disease.
Variations in Life Expectancy, Quality-Adjusted Life Years, Costs, and Net Health Benefit Associated With Variations in Indicator Performance
CAD, Canadian dollar; INHB, incremental net health benefit; NHB, net health benefit; QALY, quality-adjusted life year.
At baseline performance levels, the model showed an average life expectancy of 15.69 years and a quality-adjusted life expectancy of 11.87 QALYs. The maximum potential variation in health benefits, from varying performance on all process indicators to their highest levels, were an additional 0.85 life-years and 0.60 QALYs. Performances of assessing smoking status (QI 1), assessing readiness to quit (QI 3), and smoking cessation intervention (QI 4) indicators were associated with the largest variations in life years and QALYs among process indicators. The same indicators were also associated with the largest variations in lifetime costs. Both outcome indicators were associated with much greater variations in health benefits and costs than the process indicators, though both started from a much lower proportion of the maximum performance level. Of the two outcome indicators, improvement in the annual quit rate was associated with a near twofold greater increase in life years and QALYs, and a greater than twofold increase in lifetime costs than the quit rate after discharge from a hospital admission. The largest INHBs were associated with the same process and outcome indicators with only modest INHBs associated with process indicators. With an estimated prevalence of 11.8% among the 14 million residents in Ontario, Canada, 42 the potential variations in population INHB project as high as 132,160 QALYs among process indicators and 2.86 million QALYs among outcome indicators.
Performance variations among multiple indicators showed the potential effect of correlated indicator performance with most of the INHB represented by indicators monitoring outpatient practices. Of the outpatient indicators, combined performance variations to the indicators monitoring smoking cessation intervention, smoking status assessment, and readiness to quit assessment accounted for most of the INHB (0.47 QALYs), representing 81% of the maximum possible INHB. Variations to just two of those indicators resulted in a sizeable decrease in the proportions of the maximum INHBs (34% and 36%).
Figure 3 shows the total variations in expected outcomes for each quality indicator as they move between their extreme values (0% to 100%). Among process indicators, smoking cessation intervention showed the largest variation in health outcomes across its performance range, while encouragement to quit represented the smallest variation. Among outcome indicators, variation in the annual quit rate was associated with a much larger variation in outcomes, compared with variation in quit rate after discharge from a hospital admission.
Total variation in life years, quality-adjusted life years, costs, and net health benefit associated with smoking cessation quality indicators. Incremental outcomes between performance levels of 0% to 100%. Bars are positioned around outcomes at baseline performance levels (vertical line). Maximum range represents combined changes to all process indicators and does not include explicit changes to outcome indicators.
Discussion
Using smoking cessation in COPD as an example, we showed how this concept could be applied to prioritize quality indicators based on how much information they can provide about variations in population health and costs. Under the health economic framework in our example, we would recommend monitoring smoking status assessments, readiness to quit assessments, and outpatient smoking cessation interventions. The health economic model provided a means to relate the smoking cessation practices underlying each indicator with variations in QALYs and costs through their effect on smoking abstinence rates. Increased abstinence resulted in a slower progression of disease and lower mortality rate, producing in a larger quality-adjusted life expectancy. Performance of smoking cessation practices affected abstinence primarily by identifying more patients eligible for a smoking cessation intervention (i.e., assessing smokers and ready-to-quit smokers) or by directly increasing the smoking cessation intervention rate. Variations in costs were also driven by abstinence. Former smokers had lower annual health care costs, but a longer life expectancy also produced greater lifetime costs. Analyses of both individual and grouped indicator variations suggest that prioritizing indicators monitoring assessment of smoking status, assessment of readiness to quit, and smoking cessation interventions would capture most of the variations in QALYs and costs in the population.
While process indicators provide information about provider performance, outcome indicators provide information about patients. Like process indicators, outcome indicators can be prioritized by their relationship to population health and costs. The large disparity in QALYs and costs between the two outcome indicators in this study suggests that the annual quit rate is a more informative indicator of how health and costs are expected to vary in the patient population.
It is important to highlight that the quality indicators do not produce health benefits. Rather, the indicators produce information. The INHB associated with an indicator represents how much variation in health, net the resources that will be consumed, could be detected by a change in its performance. It may be tempting to treat this approach as a means of prioritizing quality improvement by cost-effectiveness. However, this requires information on the effectiveness, costs, and implementation rates of appropriate quality improvement interventions 11 and is likely to be difficult to obtain for many candidate indicators. However, this framework may be useful for thinking about indicators as a sentinel tool to assist providers and policy makers in deciding where to respond as performance improves or regresses. It is also important to note that the model results were deterministic and were produced under considerable computational burden.
Some questions remain unanswered in this study. First, it is unclear if costs should be part of a prioritization framework or how to incorporate them. Costs may be difficult to interpret and prone to misinterpretation. Although improved performance in smoking cessation indicators was associated with greater lifetime costs, this does not imply that improving performance is undesirable as the health benefits also needs to be considered. The INHB incorporates the joint variation in health and costs but may also be difficult to interpret with indicator performance. Moreover, a health economic framework lends greater weight to dimensions of effectiveness and cost, which may come at the expense of other dimensions of social value that are important to society and health care quality. In decisions involving drug coverage and other health technologies, economic evaluations may comprise only part of the decision input. 43 It may, therefore, be prudent to consider population health and costs as one of several important dimensions of quality. Second, a modelling approach is a time-consuming process. This study describes a proof-of-concept model for a small subset of COPD indicators. Scaling this framework to a large number of indicators across multiple diseases or contexts of care would represent a considerable undertaking. In COPD alone, more than 100 potential quality indicators have been identified in the literature and there may be similar numbers of indicators in other major chronic diseases. 19 We also note the considerable computational burden in producing our model output and restricted the analysis to deterministic simulations. A probabilistic simulation could result in some indicators appearing less important due to parameter uncertainty. Policy makers will also need to consider the technical capacity required to support the adoption of this approach and how to incorporate it into current prioritization processes. Third, there are technical challenges associated with availability of evidence concerning performance variations. This will likely be a recurring challenge as many candidate indicators will not be reported in populations easily generalizable to the target population, if at all.
Expert and stakeholder panels are commonly used to prioritize quality indicators for implementation, using consensus methods. 6 Such methods are useful as they can consider multiple attributes of quality from various perspectives in a relatively short timeframe. Using economic evaluation to prioritize quality indicators based on expected health and costs is a powerful way to identify measurements that closely reflect changes in population health. This may be particularly useful as a supplement to the current consensus-based methods that may be better suited to assessing other dimensions of quality such as access, feasibility, and person-centeredness. 9 Moreover, the use of a health economic framework should not preclude the broader consideration of these other dimensions. In principle, we see this as a step forward in the methodology, but further research is still needed to better understand how to incorporate costs and how this framework fits with current prioritization methods.
Supplemental Material
appendix1_online_supp – Supplemental material for Using a Health Economic Framework to Prioritize Quality Indicators: An Example With Smoking Cessation in Chronic Obstructive Pulmonary Disease
Supplemental material, appendix1_online_supp for Using a Health Economic Framework to Prioritize Quality Indicators: An Example With Smoking Cessation in Chronic Obstructive Pulmonary Disease by Austin Nam, David M. J. Naimark, Matthew B. Stanbrook and Murray D. Krahn in MDM Policy & Practice
Supplemental Material
appendix2_online_supp – Supplemental material for Using a Health Economic Framework to Prioritize Quality Indicators: An Example With Smoking Cessation in Chronic Obstructive Pulmonary Disease
Supplemental material, appendix2_online_supp for Using a Health Economic Framework to Prioritize Quality Indicators: An Example With Smoking Cessation in Chronic Obstructive Pulmonary Disease by Austin Nam, David M. J. Naimark, Matthew B. Stanbrook and Murray D. Krahn in MDM Policy & Practice
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Authors’ Note
This article was prepared at THETA at University Health Network and the Institute for Health Policy, Management and Evaluation at the University of Toronto (this study was unfunded). Earlier versions of this article appeared as poster presentations at the 2017 Canadian Agency for Drugs and Technologies in Health Symposium and the 39th Annual North American Meeting for the Society for Medical Decision Making.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
