Abstract
Objectives:
Performance indicators are a popular mechanism for measuring the quality of healthcare to facilitate both quality improvement and systems management. Few studies make comparative assessments of different countries’ performance indicator frameworks. This study identifies and compares frameworks and performance indicators used in selected Organisation for Economic Co-operation and Development health systems to measure and report on the performance of healthcare organisations and local health systems. Countries involved are Australia, Canada, Denmark, England, the Netherlands, New Zealand, Scotland and the United States.
Methods:
Identification of comparable international indicators and analyses of their characteristics and of their broader national frameworks and contexts were undertaken. Two dimensions of indicators – that they are nationally consistent (used across the country rather than just regionally) and locally relevant (measured and reported publicly at a local level, for example, a health service) – were deemed important.
Results:
The most commonly used domains in performance frameworks were safety, effectiveness and access. The search found 401 indicators that fulfilled the ‘nationally consistent and locally relevant’ criteria. Of these, 45 indicators are reported in more than one country. Cardiovascular, surgery and mental health were the most frequently reported disease groups.
Conclusion:
These comparative data inform researchers and policymakers internationally when designing health performance frameworks and indicator sets.
Keywords
Introduction
For more than two decades, regulators, policymakers, researchers and clinicians have endeavoured to improve the quality of healthcare by designing and applying indicators of performance. There are national and international incentives for rating the performance of health systems. The World Health Organisation (WHO) 1 and others2,3 have attempted to rank health systems for the insights gained from global comparisons, while consumers have an interest in selecting the best provider for treatment for their particular condition and knowing that their taxes are being spent wisely. 4 To meet these multiple demands, performance indicators (‘measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality, and hence change of quality, of care provided’) and performance frameworks (‘conceptual frameworks that set out the rationale and design principles for an indicator set’)5,6 are typically designed to routinely monitor aspects of healthcare performance such as effectiveness, efficiency, safety and quality. 7 The quest for a single composite indicator of quality, prevalent in the early days of indicator development, has largely been abandoned in favour of multidimensional frameworks. 1 Indicator sets commonly contain a combination of structure, process and outcome assessments. 8
The Organisation for Economic Co-operation and Development (OECD) publishes 60 internationally comparable indicators of healthcare quality. 9 These are useful and influential. However, many countries, even those with advanced data systems, have difficulty linking practice performance to outcomes because of limitations in data availability and poor capabilities to link data. Notwithstanding these kinds of shortcomings, it is useful to assess the frameworks and performance indicators in a sample of countries for the insights this provides.
Some health systems have moved faster than others in adopting performance indicators as tools for quality improvement and have made details of their indicators and systems for applying them publicly available at national, regional or institutional levels. We selected eight prominent health systems for review and assessment based on purposively selecting a sample that were exemplars in using indicators and making their data and performance systems available: Australia, Canada, Denmark, England, the Netherlands, New Zealand, Scotland and the United States. At the time of our review, all had made progress in selecting or applying indicators to measure or stimulate improved performance and most had developed a framework for conceptualising performance improvement or indicator use.
This research aims to identify and analyse indicators and their frameworks which report on the performance of healthcare organisations and local health areas. This will provide comparative cases and information on progress for the benefit of regulators, policymakers and researchers within those countries and elsewhere, but is of particular use to policymakers interested in constructing future frameworks.
Methods
We searched for relevant performance indicators and their domains across the eight countries. Following this, we analysed performance indicators that were nationally consistent (used across the country rather than just regionally), locally relevant (measured and reported publicly at a local level, for example, a health service) and measured patient-level metrics. We conducted our study in four stages.
Stage 1: identify comparable nations using performance indicators to monitor and report on healthcare
To make comparison relevant, all selected countries are OECD members and have been classified by the World Bank as high income. 10 Data on the rates of health expenditure and life expectancy for 19 countries were obtained from Australia’s Health 2010 11 and from OECD reports 9 (including Australia, Austria, Canada, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Japan, the Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, the United Kingdom and the United States). 12 After screening by the research team, the eight countries we noted above were selected for detailed review on the basis that each had made substantial progress in using indicators and developing performance frameworks and had made their indicators and performance frameworks widely available.
Stage 2: finding performance indicators
We conducted our Internet search of performance indicator systems in the eight comparator countries in May 2013. The OECD and Departments or Ministries of Health and associated government health organisations in each country were searched. A scoping table detailing the indicators by country was developed. Indicators that were collected consistently on a national scale were included, but could be relevant and useful to local quality improvement efforts. The purpose of the table was to compile an initial ‘long-list’ of available indicators and then to identify a ‘short-list’ of those reported in multiple countries.
Stage 3: detailed review of selected performance indicators
The performance indicators were subjected to a detailed assessment and were classified according to whether they applied to community/hospital/population, country of origin and clinical specialty.
Stage 4: country-specific frameworks
The health system performance frameworks for each country were reviewed together with their accompanying online and published documentation. Domains within the performance frameworks were compared.
Results
Performance indicator frameworks
A summary of each country’s approach to performance indicator use is shown in Tables 1 and 2. Most of the eight countries have an overarching framework for the selection and reporting of indicators which establish the broader aims for their implementation activity and play a large role in indicator selection and use. The number and focus of frameworks varied greatly between the eight countries, but typically included reference to both monitoring and improving quality and efficiency of the healthcare system. There appears to be considerable overlap between the definitions of many of the domains such as effectiveness and appropriateness. Indicators are sometimes also used to promote consumer choice at a regional or local level.
Summary of the eight countries: demographic characteristics and health performance frameworks.
DBH: District Health Boards; GDP: gross domestic product; NA: Nott applicabel; NHS: National Health Service; QAO; Quality and Outcomes; PAF: Performance and Accountability Framework; PHO: Primary Health Organisations; ROGS: Report on Government Services; QOF: Quality and Outcomes Framework; CCG: Clinical Commissioning Group; QOI: Quality Outcome Indicator; QSM: quality and safety marker; CIHI: Canadian Institute for Health Information.
Rank refers to CIA World Factbook, 15 country compared to the rest of the world.
Figures for UK in total, not England or Scotland specifically.
Domains of performance indicators by country.
PAF: Performance and Accountability Framework; ROGS: Report on Government Services; NHS: National Health Service; OECD: Organisation for Economic Co-operation and Development.
In Australia, the National Health Performance Authority (NHPA) 16 was established under the Australian National Health Reform Act 2011 17 as an independent portfolio agency to monitor and report on healthcare system performance; it has since been merged with the Australian Institute of Health and Welfare. NHPA commenced operations in 2012. As part of its Strategic Plan 2012–2015, 16 NHPA is required to regularly review its Performance and Accountability Framework (PAF) to ensure it remains relevant and continues to address the needs of the Australian public for high-quality healthcare. The PAF consists of 48 national indicators: 31 indicators for Medicare Locals now called Primary Health Care Networks; (geographically based primary care co-ordinating agencies) and 17 indicators for performance of Local Hospital Networks and hospitals 18 (see Table 1).
The Canadian framework has two main goals: to determine (1) the health of Canadians and (2) how well the health system performs and operates according to the published principles of providing reports that respect Canadians’ privacy and are also consistent, relevant, flexible, integrated, user-friendly and accessible.19,20 The indicator framework is conceptualised in terms of the provision of high-quality comparative information across four dimensions. Within these, eight domains of health system performance are defined.19,21
The Danish do not have a formal framework, but one is currently being developed. Instead, as a proxy framework, the Danish Institute for Quality and Accreditation in Healthcare (IKAS) manages the Danish Healthcare Quality Program (DDKM) as a national accreditation and standard–based programme. At the time of the study, this provides advanced indicators and applies them throughout the country. These standards are overseen by the International Society for Quality in Healthcare (ISQua). 22 The Danish National Indicator Project (DNIP) merged with the Clinical Quality Development Programme (RKKP) in 2010. 8 Although Denmark lacks a formal framework, the DNIP manual outlines the thinking behind its clinical indicators and planned future indicators (1) to improve prevention, diagnostics, treatment and rehabilitation; (2) to provide documentation for setting priorities and (3) to create an information resource for patients and consumers.
An example of a framework which operates at multiple geographical levels is that used in England’s National Health Service (NHS). This comprises three performance frameworks: the NHS Outcomes Framework, which focuses on performance and accountability; the Clinical Commissioning Group (CCG) Outcomes Indicator Set aimed at helping the CCG in planning and benchmarking and providing information to consumers and the Quality and Outcomes Framework (QOF) which is a voluntary pay for performance programme for general practice in England. 23
The Dutch framework, by comparison, is relatively streamlined and more consumer-focussed. Representatives of the Netherlands’ Ministry of Health, Welfare and Sport collaborated with academic researchers to develop the conceptual framework after reviewing the strategic goals of the healthcare system, information needs of policymakers and studying existing theory and international experiences. 24 The resulting framework divides healthcare into four specific community needs: (1) staying healthy, (2) getting better, (3) living independently with a chronic illness and (4) end-of-life care. 25
New Zealand has included an atlas of healthcare variation as one of its four health system performance monitoring mechanisms,26–29 and other countries such as the United States and the United Kingdom utilise an atlas, too. The atlas is organised according to clinical areas: maternity, demography, cardiovascular disease, gout, polypharmacy and surgical procedures. 29 In 2013, the NZ Health Quality and Safety Commission’s campaign Open for Better Care was commenced to measure whether planned changes in practice occur and whether they have resulted in reduced costs and harms. 28
Scotland conceptualised its Quality Measurement Framework on three levels to structure and coordinate the range of measurements that are taken across NHS Scotland. For monitoring long-term progress, there are 12 Quality Outcome Indicators (QOIs) which do not have specific targets; short-term priority areas are focussed on by the ‘HEAT’ targets: Health improvement for the people of Scotland (H), Efficiency and governance improvements (E), Access to services (A) and Treatment appropriate to individuals (T); and the third category includes all other national and local reporting requirements.
In the United States, three identifiable entities report on healthcare performance. One reports nationally (the US Department of Health and Human Services’ Agency for Healthcare Research and Quality (AHRQ)), one internationally (The Commonwealth Fund) and one locally (Hospital Compare). While there is no single integrated framework, the AHRQ measures health system performance across four dimensions 13 and the Commonwealth Fund aims to be a catalyst for change by identifying promising practices to help the United States create a high-performing healthcare system. 14 The Commonwealth Fund spans four dimensions of health system performance: access, including insurance and affordability of care; prevention and treatment, including quality of ambulatory, acute, post-acute and palliative care; avoidable hospital use and cost, such as care that could have been avoided if the patient received appropriate care initially; and indicators assessing the extent to which people can enjoy long and healthy lives. 15
Of the 11 frameworks published in five countries and the OECD, the most frequently used (self-reported) domains were effectiveness (eight), access and safety (seven each) and efficiency (five; Table 2). There are likely to be considerable overlap between the definitions of some of the domains such as effectiveness and appropriateness. For example, the OECD considers these two domains as separate while the Australian framework considers appropriateness as a subset of effectiveness.6,18 Because of this, and hierarchical relationships between domains within some frameworks, it is difficult to report the number of indicators used against each domain for each country.
Indicator choice
The search in eight countries found 401 indicators that fulfilled the ‘nationally consistent and locally relevant’ criteria we applied. Of these, 45 indicators are reported in more than one country. Table 3 contains a breakdown of indicators by country.
Nationally consistent and locally relevant indicators by country.
The search yielded 219 community-level, 231 hospital-level and 37 population-level indicators. Some indicators were classified into more than one category (Table 4).
Number of international indicators by domain and community, hospital and population.
We classified the indicators, where possible, into major disease groups (Table 5). Cardiovascular, surgery and mental health were the most frequently reported disease groups. Indicators tend to be more specifically linked to a clinical condition or disease group in some countries such as Denmark. 30
Nationally consistent and locally relevant indicators by disease group.
Denominator = 401.
Review processes
Regular review of the performance framework and indicators is conducted in most of the eight countries by government or government-funded, arm’s-length bodies. For example, the Canadian framework has continually developed and evolved since its inception, as a result of collaboration from a dedicated group including the Canadian Institute for Health Information (CIHI), Statistics Canada (SC), Health Canada (HC) and representatives from other stakeholder groups. 19 Similarly, the National Institute for Health and Clinical Excellence (NICE) has a key role in indicator development in England. NICE is responsible for managing the development process of clinical and public health indicators for the QOF and the CCG indicator sets. 31 NICE also recommends whether the existing indicators should continue to be part of the frameworks. NICE has developed guides,32,33 which set out in detail the processes involved in managing the development of indicators. Thus, indicators tend to be developed in a relatively open and transparent way, with input from individuals and stakeholder organisations. Of course, this statement masks the contested and political components of indicator development and use, which does not figure explicitly in policy documentation, academic articles or this review and is mostly country-specific. 34
Reporting
The timing and mechanism of reporting on indicators were not consistent between countries, nor were they always internally consistent. This can be seen in the reporting on Canada’s health system performance, where various indicators are reported via multiple channels. There were 101 performance indicators listed on the SC website. 35 The CIHI also has indicators listed under the Canadian Hospital Reporting Project (CHRP). While some of the indicators are the same as those listed by SC, there are some additional hospital performance indicators (21 in total). 36 Additionally, the Government of Canada 37 has a candidate list of 70 indicators that were approved for use by Federal, Provincial and Territorial Ministers of Health in 2004. However, it is difficult to gauge how many indicators are in use, because only certain indicators are selected for inclusion in the annual reports and there appear to be various degrees of overlap. For example, the Health Indicators 2013 report 20 provides results for 88 indicators, 13 of which were additional indicators chosen to be included to measure performance by socioeconomic status at provincial and national levels. 20 Although this appears confusing from an external perspective, variable reporting may be more effective in some instances, as CIHI addresses reporting needs by acknowledging different audiences and tailoring reporting for their requirements.
Meanwhile, in the Netherlands, a report detailing results for all 125 indicators is published every 2 years. 38 From 2011, the information was updated via a website twice a year and from 2014, the report will be published every 4 years. 38 The indicators are reported at the national level, not locally. Indicators reported locally (at regional levels) occur via the Dutch Hospital Database. There are two dedicated websites that provide consumers of healthcare with information about the quality of a service and provide ratings for their service. 39 These are Independer – www.independer.nl – and Kiesbeter – www.kiesbeter.nl or ‘Choose Better’.
Similarly, in Denmark, clinical indicators are reported, and a structured audit process is initiated every 6 months by audit groups of clinicians at national, regional and local levels to explain the risk-adjusted results and to plan improvements. After the audit process is complete, the data are released publicly, including comments on the results from the audit groups. 40 Reports on many of the indicators are available on the www.sundhed.dk website.
Discussion
In this study, the eight countries selected for review were using indicators and had implemented a performance framework, several over more than a decade. The progress they have made, and choices taken in selecting and using indicators, might be of value for other health systems in contemplating the development of their indicators or frameworks, or modifying their performance mechanisms. A key finding was the widespread support for implementing a healthcare system performance framework. The importance of a logical, universally acceptable and viable conceptual framework to underpin development of a national performance indicator set is also emphasised in the literature.41,42 A conceptual framework sets out the rationale and design principles for the indicator set and links it to the wider health system context. It seeks to answer the question ‘performance of what – and to what ends?’ 6 Reasons given for developing such a framework are as follows: (1) to define the scope and dimensions of measurement;24,6 (2) to help align the performance measurement system with other policy priorities and financial exigencies; 43 (3) to provide a clearly defined vision to encourage buy-in by clinicians and consumers24,43 and (4) by involving potential end-users of the data in the design of the framework, to ensure its future usefulness. 44 A conceptual framework encompassing multiple domains and with balanced representation across structure, process and outcome indicators is considered to be a key element of health reform over time. 45
Although we presented the self-reported domains by country, consistency of definitions between countries and the level of semantic overlap was not tested; however, these are likely to be substantial. For example in the Australian PAF, the domain appropriateness is subordinate to, or sub-class of, the domain effectiveness. 18 In Canada, these two domains are not grouped, but are classified as separate concepts. Definitions for these domains are often not explicit in the policy documents. 20 Definitional consistency between countries should be the subject of more research or efforts to internationally standardise.
Although there is a substantial literature dealing with the design, properties and scientific soundness of individual indicators, there is considerably less attention given to how indicators are used in practice and the impact they may have on the behaviour of health professionals, or on the quality of care. While there is no answer to questions such as how many indicators, which domains should be targeted or what should be the right mix of indicators, there is a fundamental debate centred on whether the purpose of performance indicators is accountability or quality improvement.41–43,46 Internationally, there is a split between those countries which emphasise public reporting and accountability (e.g. the UK NHS’s ‘star-ratings’ system of 2001–2005) 47 and those that use results for non-publicised feedback to organisations to stimulate improvement.
It is broadly agreed that monitoring performance imposes an inherent pressure on healthcare organisations or services to improve practice. 48 However, the extent to which this is accomplished is disputed and under-researched. The paucity of research examining the links between indicators and improvement may be due to the difficulty in attributing change to any particular policy initiative or intervention. 49
The literature supports the use of performance indicators, suggesting that their impact is more likely to be on provider rather than consumer behaviour.43,50 However, there is a general call for more good quality studies on impact.50–52
England and Canada do the most extensive research and development work to select indicators. The role of NICE in England exemplifies a thorough and considered approach to continued indicator development. The process of review, seen especially in England, Canada and Australia, is critical to the continued development of performance indicators and their use.
Conclusion
A large amount of comparative information about international performance indicators is now available. 53 We examined the systems in use in eight countries. Assessing commonalities and differences between indicator specification and application in comparable health systems may be of value to regulators, policymakers, researchers and clinicians and forms a foundation for further research into the practical impact of indicators on the quality of healthcare.
Footnotes
Acknowledgements
The authors would like to thank their colleagues Dr Diane Watson, Bernie Harrison, Jessica Stewart, Dr Julia Langton, Jason Boyd, Jessica Herkes and Kristiana Ludlow for their support to the project.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
This work was supported, in part, by the National Health Performance Authority, Australia (but no one including the Authority’s officers had a role in the study, its conduct, results, interpretation or publication of this article) and the National Health and Medical Research Council (APP1054146).