Assessing Performance in Health Care Using International Surveys: Are Patient and Clinician Perspectives Complementary or Substitutive

Abstract

Background:

Over the last decade, international surveys of patients and clinicians have been used to compare health care across countries. Findings from these surveys have been extensively used to create aggregate scores and rankings.

Objective:

To assess the concordance of survey responses provided by patients and clinicians.

Methods:

Analysis of 16 pairs of questions that focused on coordination, organizational factors, and patient-centered competencies from the Commonwealth Fund International Health Policy Survey of older adults (2014) and of primary care physicians (2015). Concordance was assessed by comparing absolute rates and relative rankings.

Results:

In absolute terms, patients and clinicians gave differing responses for questions about coordination of care (patients were more positive) and provision of after-hours care (patients were less positive). In relative terms, country rankings were positively correlated for 5 of 16 question pairs (Spearman ρ > .6 and P < .05).

Conclusion:

Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients’ and clinicians’ perspectives before combining them to create aggregate assessments of performance.

Keywords

health-care performance concordance international surveys rankings

Background

Health-care systems are complex and multifaceted. This complexity means that the measurement of health-care performance should incorporate a wide array of information. At the same time, quality in health-care is a contested construct, shaped by expectations, experiences, context, and priorities of different stakeholders and so meaningful assessment should also encapsulate different perspectives (1,2). Considering the extent of concordance in perspectives from different stakeholder groups may reveal elements of performance that are unwaveringly viewed in the same way (either positively or negatively), both within and across health-care systems. It can also reveal aspects of performance that are seen very differently from a patient and from a provider perspective—with this difference in perspective replicated in all health-care systems. Finally, it can reveal elements of performance that are shaped by differences in health system context, resulting in little concordance both between stakeholders and across jurisdictions.

Patient surveys are a common feature in most developed health-care systems and international surveys occupy an important space in performance assessment efforts and policy discussions in many countries (3 –5). Patients provide a unique view of health care as they are often the only constant presence across episodes of care, treatments, and clinicians and are best placed to observe the extent to which care is patient centered and integrated (6). Increasingly, provider or clinician perspectives are also used in quality assessment and performance reporting (7 –9). Clinicians are well placed to report on issues such as coordination and integration of care through technology, communication, and teamwork across teams (10 –12). Clinicians are also able to assess the culture of safety in their organization; to describe how they work and the tools they have available to enable them to deliver care to patients; and to reflect on coordination with, and information flow from, other clinicians.

The use of multisource feedback is increasingly used to provide information for revalidation and other performance applications in health care (13). Both patients and clinicians are seen as key informants, and both groups serve as an input to policy making (14,15). This is especially important in the context of wide recognition that care should be patient-centered and also promote good experience in delivering health care for clinicians (16,17). Patient and clinician surveys have been used to assess health care performance in terms of the provision of accessible and appropriate care and with regard to the outcomes of care, across and within countries, combining their perspectives using qualitative or quantitative aggregation techniques (3 –5,18). However, studies that compare perspectives have shown that physicians are sometimes limited in their ability to accurately self-assess (19). This means that a range of perspectives are often required to fully explore issues of performance.

While many assessments use scoring and ranking approaches to compare performance and create aggregate measure, there are concerns about the validity of many scoring and ranking methods. The Commonwealth Fund’s work is an example where patient and provider surveys are used to create aggregate ranking scores to compare countries (3 –5).

The idea of “concordance” of patient and clinician perspectives has been considered in alternative ways in research. Some studies have assessed concordance with regard to demographic characteristics of patients and clinicians such as gender, language, or ethnicity (20). Others have explored the alignment of patient and clinician perspectives with regard to shared decision-making (21), attitudes about respect (22), and medication adherence (23). Here, we are interested in concordance as the level of agreement between patients and clinicians at a geographic level with regard to satisfaction with or experience of care (24).

The extent to which the views of patients and clinicians agree, or are concordant, remains to be assessed. This study aims to assess how patient-reported and clinician-reported measures on similar questions provide concordant assessment of performance within 11 countries with developed health care systems across Europe, North America, and Australasia. In addition, it aims to assess the impact of using patient-reported or clinician-reported measures to compare countries with regard to performance in health care. Furthermore, this article proposes an approach to investigate concordance between patient and clinician perspectives regarding particular aspects of care, such as coordination or availability of urgent appointments, for measures reported across countries. This approach may inform both measurement efforts and the interpretation of measures combining patient and clinician perspectives.

Methods

This study is exploratory in nature and uses international surveys of patients and primary care clinicians to assess concordance of assessment. We consider survey concordance in 2 ways: absolute and relative. Absolute concordance refers to the similarity—within a jurisdiction, in the percentage of patients and of providers who selected a particular response option (usually the most positive response option). For example, within a jurisdiction, if the percentage of patients who said their regular doctor spends enough time with them is similar to the percentage of providers who said they were satisfied with the amount of time they have to spend per patient—that is deemed as high absolute concordance.

Relative concordance places the comparison between patient and provider responses in a broader context. It considers whether the extent of concordance is consistent across jurisdictions. For example, looking across all of the participating jurisdictions in the survey—if both patients and providers from a particular jurisdiction gave the highest proportion of positive responses (ie, the jurisdiction was ranked 1 in both surveys), that would be deemed to high relative concordance (regardless of whether the absolute % responses were similar or highly disparate in that jurisdiction).

Data Source and Variables

We used the Commonwealth Fund International Health Policy Surveys of older adults (2014) and primary care physicians (2015). Surveys were conducted by telephone in 11 countries—Australia, Canada, France, Germany, the Netherlands, New Zealand, Norway, Sweden, Switzerland, the United Kingdom, and the United States. The number of older adult respondents ranged across countries from 928 to 7206, and the number of primary care clinicians’ respondents ranged from 502 to 2905 (see Appendix A). Analyses were weighted to reflect the distribution of adults aged 55 years and older and of practicing primary care physicians in each country (11,12).

To identify question pairs that addressed the same aspects of care, 2 researchers independently matched survey questions across the patient and provider surveys. Disagreements in matching were resolved by discussion within the broader research group. In total, 16 pairs of questions were identified (although there were slight differences in phrasing for some question pairs), covering topics such as availability and timeliness of care, coordination of care, and overall views on the health system (Appendix B). A performance dimension, such as accessibility, appropriateness, efficiency or sustainability, was assigned to each question pair based on existing performance reports (5,17). Matching response categories for each question were selected for analysis. In the case of one question regarding overall health system views, both the most and least positive categories were used in this analysis. Appendix C provides descriptive statistics for each pair of questions by country.

Assessment of Concordance

The differences between patient and clinician responses were calculated to provide an estimate of the absolute concordance. Differences were calculated both within countries and based on the average value across countries. The relative concordance between country rankings was assessed for each pair of measures using Spearman rank-order correlation test. Results with P < .05 results were noted as being concordant. Based on the correlation coefficient, ranks were considered to be moderately concordant (>0.4) or highly concordant (>0.6) (21). The averages of each of the concordance measures (absolute and rank differences) were used to summarize results across countries. Within each country, the average concordance values across the 16 pairs of questions were calculated for each of the measures. Finally, to explore possible relationships between the differences in perspectives and the sample information, a rank correlation between the concordance measures with both sample size and response rates was calculated.

Results

Descriptive results and measures of concordance between levels and rankings for each of the 16 pairs of questions are summarized in Table 1. The absolute differences provide an indication of whether patients or clinicians tended to be more positive. Patients responded more positively than clinicians to questions regarding coordination. For example, a minority of patients said their records were unavailable or they had care coordination issues, whereas a majority of clinicians said these issues had occurred in the preceding month. Patients were more likely to say their regular place of care seemed informed about the hospital care they received, while clinicians were less positive about receiving notification from hospitals. In contrast, clinicians responded more positively than patients about the availability of after-hours care. Differences in average patient and clinician responses were small (<10 percentage points) for measures about overall system views, discussing end-of-life care, provision of written plans for chronic conditions, and the time spent in consultation with clinicians.

Table 1.

Descriptive Statistics Including Percentage, Average Across Countries by Patient or Provider Perspective, and Absolute and Relative Concordance Measures Between Perspectives by Question Pairs.^a

Performance Dimension	Survey of Adults Aged 55 Years and Over (2014)			Survey of Primary Care Physicians (2015)		Concordance
	Questions	Average of Countries (%)	Range	Questions	Average of Countries (%)	Range	Difference in Averages (Percentage Points)	Average Rank Difference	Spearman Rank-Order Test
	Questions	Average of Countries (%)	Range	Questions	Average of Countries (%)	Range	Difference in Averages (Percentage Points)	Average Rank Difference	Coefficient	P Value
Accessibility	Health-care professional makes contact for chronic condition	22	36.9	Have staff who contact patient to monitor chronic condition	34	37.9	12	1.6	0.83	.00^b
Accessibility	Availability of same or next day appointments	66	39.3	Almost all patients able to get same or next day appointments	47	44.4	19	1.6	0.72	.03^b
Accessibility	Waited 2 months or longer for specialist appointment	15	25	Patients experience long waits for specialist appointments	47	61.5	32	1.8	0.7	.00^b
Accessibility	Skipped care due to cost	9	20.4	Patient had difficulty paying for medical expenses	24	58.2	15	1.8	0.68	.01^b
Accessibility	Very easy to get after-hours primary care	24	32.7	Practice has after-hours arrangements for patients	75	54.6	51	1.5	0.61	.01^b
Appropriateness	Discussed with family, friend, health-care professional about treatment	44	58.9	Had conversations about treatment wishes with older/sicker patients	43	45.4	1	2.6	0.51	.08
Sustainability	Health system is working well, only minor changes	46	40.6	Health system is working well, only minor changes	39	51.2	7	2.6	0.51	.13
Appropriateness	Received written plan for management of chronic condition	35	37.2	Patients with conditions given written plan to manage care	30	41.5	5	2.6	0.47	.83
Appropriateness	Medical staff seemed informed about care in hospital	86	25.6	Received notification about patient’s care in hospital	34	62.0	52	2.9	0.45	.35
Appropriateness	Received a list of medications	58	49.4	Practice can generate list of patient’s medications	72	60.3	14	2.2	0.45	.75
Sustainability	Health system needs a complete rebuild	11	29.7	Health system needs a complete rebuild	6	13.6	5	2.7	0.21	.31
Efficiency	A test repeated because results unavailable	7	11.0	A patient’s test was repeated because results unavailable	30	30.4	23	3.6	0.10	.11
Accessibility	GP always spent enough time	65	36.4	Satisfied with time you have to spend per patient	57	48.5	8	3.6	0.08	.51
Appropriateness	Experienced care coordination problem	20	25.7	Patients had care coordination problems	51	50.3	31	4	–0.07	.08
Appropriateness	Medical staff seemed informed about care in hospital	86	25.6	Received notification about patient’ care in ED	33	61.7	53	2.9	0.32	.13
Efficiency	Test results or medical records not available at the time of visit	9	13.2	A patient’s medical record or relevant clinical information not available at the time of visit	65	23.6	56	4.2	–0.29	.8

Abbreviations: ED, emergency department; GP, general practitioner.

^aThe dark shading indicates the lowest levels of concordance, lighter shading denotes a moderate level of concordance, and no shading indicates a high level of concordance or a significant concordance of rankings (P < .05). For absolute concordance and rank difference measures, shading is based on tertiles. For the rank-order coefficient, the most concordant measure pairs have coefficient >0.6 and P < .05, the moderately concordant pairs are those where 0.4 < coefficient < 0.6, and anything between (−0.4 and 0.4) is noted as not concordant in this analysis.

^b P < .05.

The Spearman rank correlation analysis shows a strong concordance (coefficient >.6 and significant P < .05) between rankings based on patients and clinician responses for 5 questions. For these questions, all of which were related to the dimension of accessibility, the average difference in rankings was less than 2 places. However, when looked at in terms of the percentage differences for all 5 questions, there was moderate and low agreement with differences of over 10 percentage points. For the only question that was the same in both surveys, there was strong concordance based on the levels and moderate concordance based on the rankings in terms of positive patient and clinician views seeing health system as working well (Figure 1).

Figure 1.

Example of moderate relative concordance and high absolute concordance: overall health system views.

According to the relative concordance using ranking, there was low concordance for 6 of 16 questions, covering 3 performance dimensions (efficiency, accessibility, and appropriateness). Within this subset of 4 questions, the average difference in rank was approximately 3 or more places between perspectives. The least concordant question pair addressed patients’ medical records, tests, or clinical information being unavailable at the time of their visit, with an average rank difference between perspectives of 4 places (Figure 2).

Figure 2.

Example of low relative and low absolute concordance: medical records or tests not available.

How Do Perspectives Vary by Country?

The differences between patient and clinician perspectives are summarized across the 16 question pairs in Table 2. In terms of absolute differences between patient and clinician perspectives, the Netherlands had the smallest average (absolute) difference of <20 percentage points and Norway had the largest average difference of 30 percentage points. Within countries, the average difference in rankings between questions ranged from 1.9 in Sweden to 3.3 places in the United Kingdom. Countries with larger sample sizes, such as Sweden and Canada, tended to have smaller differences in ranking and smaller correction factors. There were strong negative Spearman rank correlation coefficients between the number of respondents in the patient survey and the rank differences (ρ = −.7).

Table 2.

Average Absolute Difference and Rank Difference Across 16 Pairs of Questions, by Country.^a

	Average Difference Between Patient and Provider Responses (Percentage Points)	Average Rank Difference
Sweden	29	1.9
United States	24	2.1
Canada	29	2.3
Australia	29	2.6
Switzerland	26	2.6
France	28	2.7
New Zealand	25	2.8
Germany	29	2.9
Netherlands	19	2.9
Norway	30	3.1
United Kingdom	22	3.3
Spearman rank-order coefficient between concordance measures and survey respondent/response rates
2014 respondents	0.33	−0.70
2014 response rate	0.02	−0.28
2015 respondents	0.16	−0.48
2015 response rate	−0.05	0.04

^a For full survey details on the number of respondents and response rate, see Appendix A.

Discussion

This study examined the concordance of measures from patient and clinician perspectives used in international surveys and compared a range of methods to assess concordance. Overall, concordance was stronger for certain aspects of health care, such as access. In addition, results for concordance in ranking differ from those for concordance in absolute levels, and in many cases, the magnitude of differences is important.

Our academic and gray literature review did not find any studies that looked at approaches to interpret concordance of measures from patient and clinician perspectives used in international performance comparisons. However, there are examples of geographic correlation in satisfaction of patients and clinicians in the United States and the United Kingdom (25). Further, one country-level study in Ukraine did consider population and physician views about several measures of quality of health care (26). It reported that both groups felt the health system should be reformed, care for common chronic diseases was poor, and affordability was a problem. However, their perspectives differed on quality of care where household respondents saw it as a priority and physicians did not, and results were consistent with our findings of moderate concordance with regard to affordability of care and overall views about performance. As far as we are aware, ours is the first study that has looked at concordance of performance rankings and perspectives from 2 international surveys.

Complementary or Substitutive Perspectives?

Our findings may help to better understand the value of measuring aspects of care from different perspectives, the need to consider both rankings and absolute levels, and the fact that the extent to which perspectives are concordant may be sensitive to context.

Our study identified strong concordance for some pairs of performance measures, regardless of the method used to assess concordance. Questions where perspectives aligned well or moderately well in terms of both levels and rankings suggest performance measurement based on either perspective would have similar conclusions and could be substitutive. In other words, we may be able to aggregate the patient and provider perspectives to compare performance despite differences in expectations between patients and providers, or we may not need to ask both patients and providers about this aspect of care.

For other pairs of performance measures, the assessment of concordance was sensitive to the method of concordance assessment used. For some questions (eg, on after-hours care), there was little difference in rankings while the concordance based on absolute levels was low. For others, absolute concordance was good but rankings varied much more (eg, time spent in consultations), especially when the actual range of results was small between countries. These may point to interesting divergence of perspectives within countries and aspects where assessment may benefit from both perspectives, with patients and clinicians providing complementary assessments of performance.

In situation of strong concordance, the 2 perspectives can be combined. However, we could potentially rely on the use of only one of the perspectives to understand performance. In case of low concordance, the aggregation of perspective risks drawing an inaccurate conclusion and the measurement of both perspectives provide complementary understanding.

Why Might Patients’ and Clinicians’ Responses Vary

From a purely applied perspective, both patients and clinicians provide valuable insights on their own. From a comparative perspective, however, additional insights may come when we look at the concordance of these patient-based and clinician-based assessments across health-care systems. Varying expectations, differences in actual exposures to aspects of health-care delivery, and temporality of experiences of health care and its reporting through surveys may be factors that affect the levels of concordance across patients’ and clinicians’ assessments. Patients’ responses may not show the same time lag between “real experience” and responses at the time of surveys. For example, clinicians are experiencing health care every day, while most patients only experience health care on an infrequent basis. Therefore, changes in performance may actually be picked up more quickly by clinicians compared to most patients. Interestingly, our assessment showed that despite moderate to high concordance in terms of overall views, the United Kingdom ranked second in terms of whether the system was working well based on the perspective of adults aged 55 years and over, while it ranked ninth from a clinician perspective on the same question 1 year later. This could show how expectations toward health care could be influenced at a different pace depending on the group providing their views about health care. Our results suggest that caution is required in the construction of composite measures and aggregation of measures—particularly those that combine different stakeholder perspectives.

This analysis also points to areas where survey data should be used with caution, or performance measures selected more intentionally, from only one perspective. Patients or clinicians may be better placed to report on some of the measures. For instance, patients can report whether they were treated with respect, and doctors can report whether they generally received appropriate information from a colleague following their patients’ referral. Where there is a lack of concordance across methods of assessment, such as the case for reporting of the availability of medical records at the time of patient visits, the conclusions of performance assessment depend on the perspective used. For example, clinicians see many patients and have more chance to encounter issues with availability of records, whereas a patient may see only 1 clinician during a time frame, and lack of availability of records may be a rarer occurrence from their perspective. This could also explain differences with regard to the assessment of coordination of care, since providers and patients will have a very different actual experience of the flow of services. Interestingly, access seems less associated with discordance. Ultimately, patients are the witness of their entire experience with various providers, while providers have an experience shaped by the totality of their roster of patients. Moves toward patient-centered care and a stronger focus on key aspects that relate to coordination of care, communication, and team-based approaches may result in an increased concordance of experience of care between providers and patients.

Limitations

There are several limitations that need to be considered. First, we acknowledge the challenges in comparing the average levels of responses from 2 groups of respondents, using different survey questions with varying response formats (eg, satisfaction vs reporting of experience), over 2 time periods, based on unlinked data sets. However, while not identical, the paired questions ask about the same constructs or elements of care. Assessing the same pair across jurisdictions allows us to see the relationship in terms of patient experience and provider satisfaction. We acknowledge that we have not established whether the questions used in this study actually measure the same constructs, for all raters, as no testing of measurement invariance of multisource feedback has been performed.

Correlational analysis based on 11 countries and the use of country averages and arbitrary cutoffs are also limitations. Further, despite being used repeatedly in several international surveys, the survey measures have not been thoroughly validated. Despite these methodological issues, some clear and consistent themes emerged by triangulating perspectives and methods (absolute and relative rankings).

Survey methods may also affect differences in rankings between patient and clinician perspectives within countries. Countries with larger numbers of respondents such as Sweden and Canada have smaller differences between rankings. This may be due in part to the smaller margins of error or less variability in estimates from larger samples.

Conclusion

Comparing patient and clinician views across jurisdictions provides new insight into assessing performance using survey data. Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients’ and clinicians’ perspectives before combining them to create aggregate assessments of performance.

This was a simple, mainly descriptive analysis with potential for greater development. Conceptually, it considers survey data in 4 “planes”—patient perspectives, clinician perspectives, jurisdictional context, and time. Given that heterogeneity—lack of concordance is hard to interpret, however, high concordance may provide more guidance. High concordance could be the reflection of fundamental features of health care that are viewed in the same way by patients and providers and across different contexts and jurisdictions. Future research could more purposefully assess the concordance of perspectives between patients and providers using a prospective approach.

These results reinforce conclusions drawn from the use of many survey measures in international comparisons and show a robustness of survey data as complementary information in performance assessment alongside more established administrative and emerging patient outcome measures. The strong concordance in rankings for accessibility measures indicates that from either perspective, country comparisons of performance would result in similar conclusions about relative performance in access to care. However, there is more work to be done to develop survey questions into performance measures. Measures that are not concordant may point to lessons to be learned about the sensitivity of rankings to the survey or perspective. Patterns across countries showed potential differences that may result from the different timing of surveys in the context of change at the system level. To understand differences in perspectives at the country level, results need to be analyzed separately by measure and with a more in-depth assessment of the health-care policy environment in the specific country.

Footnotes

Appendix A. Sample Size and Response Rate by Countries,2014 and 2015 International Health Policy Surveys

	2014 Survey of Adults 55+		2015 Survey of Primary Care Physicians
	Number of Respondents	Response Rate	Number of Respondents	Response Rate
Australia	3310	31%	747	25%
Canada	5269	28%	2284	32%
France	1500	29%	502	8%
Germany	928	26%	559	19%
Netherlands	1000	25%	618	41%
New Zealand	750	27%	503	28%
Norway	1000	16%	864	44%
Sweden	7206	23%	2905	47%
Switzerland	1812	60%	1065	39%
United Kingdom	1000	23%	1001	39%
United States	1755	24%	1001	31%

Appendix B. Pairs of Measures Selected for Analysis from the International Health Policy Survey of Older Adults (2014) and of Primary Care Providers (2015)

Appendix B.

Dimension	Older Adults Survey Questions (2014)	Primary Care Physician Survey Questions (2015)
Accessibility	Last time you were sick or needed medical attention, how quickly could you get an appointment to see a doctor or a nurse? (Same or next day)	What proportion of your patients who request a same- or next-day appointment can get one? (Almost all [more than 80%])
	After you were advised to see or decided to see a specialist, how long did you have to wait for an appointment? (2 months or longer)	How often do you think your patients experience the following? Experience long waiting times to see a specialist. (Often)
	How easy or difficult is it to get medical care in the evenings, on weekends, or holidays without going to the hospital emergency department? (Very easy)	Does your practice have an arrangement where patients can see a doctor or nurse if needed when the practice is closed without going to the hospital ED? (Yes)
	When you need care or treatment, how often does your regular doctor or medical staff you see spend enough time with you? (Always)	Please indicate how satisfied you are with the following aspects of your medical practice. The time you have to spend per patient. (Satisfied)
	During the past 12 months, was there a time when you skipped care (treatment, visit or prescription) due to cost? (Yes to any of the 3).	How often do you think your patients experience the following? Have difficulty paying for medications that they have to pay for themselves or other out-of-pocket costs. (Often)
	Between doctor visits, is there a health-care professional who contacts you to see how things are going with your condition? (Yes)	Do you and/or other personnel who work with you provide care in any of the following ways? Contact patients between visits to monitor their condition. (Yes, frequently)
Appropriateness	Any coordination problem (test results not available, conflicting information, or unnecessary test). (Yes to any of the 3)	During the past month, did the following occur with any of your patients? A patient experienced problems because care was not well coordinated across multiple sites or providers. (Yes)
	In the event you become very ill or injured and you cannot make decisions for yourself, have you had a discussion with family, friend, or a health professional about what treatment you want or do not want? (Yes)	Do you have conversations with older or sicker patients about the health-care treatment they want or do not want in the event they become very ill, injured, or cannot make decisions for themselves? (Yes, routinely)
	In the past 12 months, has a health-care professional given you a written list of all your prescribed medications? (Yes)	Can your practice generate a list of all medications taken by an individual patient (including those that may be prescribed by other doctors? (Yes)
	During the past year…has any health-care professional you see for your condition given you a written plan to help you manage your own care? (Yes)	Are your patients with chronic conditions given written instructions about how to manage their own care at home…? (Yes, routinely)
	After you left the hospital, did the doctors or staff at the place where you usually get medical care seem informed and up-to-date about the care you received in the hospital? (Yes)	When your patients go to the ED, how often do you receive? Notification your patient has been seen in the ED. (Always)
		When your patients go to the hospital, how often do you receive? Notification your patient has been seen. (Always)
Efficiency	Now thinking about the past 2 years, when receiving care for a medical problem, was there ever a time when test results or medical records were not available at the time of your scheduled medical care appointment? (Yes)	During the past month, did the following occur with any of your patients? A patient’s medical record or other relevant clinical information was not available at the time of the patient’s scheduled visit. (Yes)
Efficiency	Was there ever a time in the past 2 years when doctors ordered a medical test that you felt was unnecessary because it had already been done? (Yes)	During the past month, did the following occur with any of your patients? Tests or procedures had to be repeated because results were unavailable. (Yes)
Sustainability	Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (On the whole, the system works pretty well, and only minor changes are necessary to make it work better)	Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (On the whole, the system works pretty well, and only minor changes are necessary to make it work better)
Sustainability	Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (Our health-care system has so much wrong with it that we need to completely rebuild it)	Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (Our health-care system has so much wrong with it that we need to completely rebuild it)

Abbreviations: ED, emergency department; GP, general physician.

Appendix C. Percentage of Adults Aged 65+ (2014) and Providers (2015)Who Provided the Selected Response Option,by Country

Patient Experience	Australia	Canada	France	Germany	Netherlands	New Zealand	Norway	Sweden	Switzerland	United Kingdom	United States	GP Experience	Australia	Canada	France	Germany	Netherlands	New Zealand	Norway	Sweden	Switzerland	United Kingdom	United States
Medical record not available at the time of visit	6	13	3	9	8	7	7	9	7	8	16	A patient’s medical record not available at the time of visit	74	61	58	54	59	77	76	62	59	69	62
Experienced care coordination problem	20	25	6	30	14	16	20	20	23	18	32	Patients had care coordination problems	64	51	29	57	47	64	40	54	33	79	46
GP always spent enough time	67	60	47	72	83	74	57	50	80	60	65	Satisfied with time you have to spend per patient	75	67	65	55	45	59	67	41	68	26	55
A test repeated because results unavailable	8	7	2	11	4	4	5	4	12	8	13	A patient’s test was repeated because results unavailable	39	28	26	26	22	28	34	35	18	48	28
Health system needs a complete rebuild	6	10	3	12	20	6	7	11	5	5	32	Health system needs a complete rebuild	2	3	13	12	2	1	1	11	2	6	14
Medical staff seemed informed about care in hospital	87	85	86	93	93	87	88	68	89	82	88	Received notification about patient’s care in ED	18	33	22	20	68	57	32	7	31	49	32
Medical staff seemed informed about care in hospital	87	85	86	93	93	87	88	68	89	82	88	Received notification about patient’s care in hospital	18	29	33	28	70	49	38	8	29	37	31
Received a list of medications	63	67	28	47	66	66	44	77	38	75	65	Practice can generate list of patient’s medications	78	56	35	69	93	86	82	70	48	96	74
Received written plan for management of chronic condition	40	37	42	25	22	45	23	24	22	59	50	Patients with conditions given written plan to manage care	40	18	20	33	41	28	14	10	25	52	46
Discussed treatment wishes at end of life	53	61	14	65	43	40	21	31	53	35	73	Had conversations about treatment wishes with older/sicker patients	40	44	36	50	59	34	22	24	48	67	48
Health system is working well, only minor changes	51	35	41	46	44	49	55	44	62	56	22	Health system is working well, only minor changes	48	36	29	27	51	57	68	19	54	22	16
Very easy to get after-hours primary care	20	13	13	28	39	39	23	6	28	35	21	Practice has after-hours arrangements for patients	78	48	73	87	95	92	81	75	69	89	40
Skipped care due to cost	11	12	4	7	8	12	5	4	9	5	25	Patient had difficulty paying for medical expenses	25	31	17	13	53	30	3	6	9	12	61
Waited 2 months or longer for specialist appointment	16	28	13	17	11	21	25	20	3	14	3	Patients experience long waits for specialist appointments	57	70	65	62	11	66	48	56	9	41	34
Availability of same or next day appointments	67	44	83	81	76	81	53	54	68	63	54	Almost all patients able to get same or next day appointments	46	28	61	65	52	56	40	20	54	38	53
Health-care professional makes contact to check on chronic condition	22	15	22	12	21	30	13	17	10	47	330	Have staff who contact patient to monitor chronic condition	27	28	31	38	39	49	16	35	19	54	39

Abbreviations: ED, emergency department; GP, general physician.

Authors’ Note

This study is a secondary analysis of anonymous surveys and therefore did go through an ethical review process. The study was conducted as part of the work of the Bureau of Health Information and the Agency for Clinical Innovation and respected the ethical conduct policies of these public organizations.

Acknowledgments

The authors would like to acknowledge the support of the staff of the Bureau of Health Information.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jean-Frederic Levesque, MD, PhD

References

Leatherman

Sutherland

. Designing national quality reforms: a framework for action. Int J Qual Health Care. 2007;19(6):334–40. Epub October 18, 2007.

Papanicolas

Cylus

Smith

. An analysis of survey data from eleven countries finds that ‘satisfaction’ with health system performance means many things. Health Aff (Millwood). 2013;32(4):734–42. doi:10.1377/hlthaff.2012.1338.

Davis

Schoen

Schoenbaum

Doty

Holmgren

Shea

, et al. Mirror on the Wall: An International Update on the Comparative Performance of American Health Care. New York, NY: The Commonwealth Fund; 2007.

Davis

Schoen

Stremikis

. Mirror, Mirror 2017: International Comparison Reflects Flaws and Opportunities for Better U.S. Health Care. New York, NY: The Commonwealth Fund; 2017.

Davis

Stremikis

Squires

Schoen

. Mirror Mirror on the Wall: How the Performance of the US Health Care System Compares Internationally. New York, NY: The Commonwealth Fund; 2014.

Browne

Roseman

Shaller

Edgman-Levitan

. Analysis & commentary measuring patient experience as a strategy for improving primary care. Health Aff (Millwood). 2010;29(5):921–5. doi:10.1377/hlthaff.2010.0238.

Bodenheimer

Sinsky

. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6. doi:10.1370/afm.1713.

Braithwaite

Hibbert

Blakely

Plumb

Hannaford

Long

, et al. Health system frameworks and performance indicators in eight countries: a comparative international analysis. SAGE Open Med. 2017;5:2050312116686516. doi:10.1177/2050312116686516. eCollection 2017.

Carinci

Van Gool

Mainz

Veillard

Pichora

Januel

, et al. Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care. 2015;27(2):137–46. doi:10.1093/intqhc/mzv004. Epub March 10, 2015.

10.

Cooley

McAllister

. Building medical homes: improvement strategies in primary care for children with special health care needs. Pediatrics. 2004;113(suppl 5):1499–506.

11.

Osborn

Moulds

Schneider

Doty

Squires

Sarnak

. Primary care physicians in ten countries report challenges caring for patients with complex health needs. Health Aff (Millwood). 2015;34(12):2104–12. doi:10.1377/hlthaff.2015.1018.

12.

Osborn

Moulds

Squires

Doty

Anderson

. International survey of older adults finds shortcomings in access, coordination, and patient-centered care. Health Aff (Millwood). 2014;33(12):2247–55. doi:10.1377/hlthaff.2014.0947. Epub November 19, 2014.

13.

Campbell

Roberts

Wright

Hill

Greco

Taylor

, et al. Factors associated with variability in the assessment of UK doctors’ professionalism: analysis of survey results. BMJ 2011;343:d6212. doi:10.1136/bmj.d6212.

14.

Confederation NHS. Feeling Better? Improving Patient Experience in Hospital. London, England: NHS Confederation; 2010.

15.

DeCourcy

West

Barron

. The national adult inpatient survey conducted in the english national health service from 2002 to 2009: how have the data been used and what do we know as a result? BMC Health Serv Res. 2012;12:71. doi:10.1186/1472-6963-12-71.

16.

Berwick

Nolan

Whittington

. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759–69. doi:10.1377/hlthaff.27.3.759.

17.

OECD. Health at a Glance 2015. Paris, France: OECD Indicators; 2015.

18.

Bureau of Health Information. Healthcare in Focus 2015. How Does NSW Compare. 2016. Retrieved from Sydney. Available from: http://www.bhi.nsw.gov.au/BHI_reports/healthcare_in_focus. Retrieved on February 6, 2019.

19.

Davis

Mazmanian

Fordis

Van Harrison

Thorpe

Perrier

. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–102. doi:10.1001/jama.296.9.1094.

20.

Street

O’Malley

Cooper

Haidet

. Understanding concordance in patient-physician relationships: personal and ethnic dimensions of shared identity. Ann Fam Med. 2008;6(3):198–205. doi:10.1370/afm.821.

21.

Coran

Koropeckyj-Cox

Arnold

. Are physicians and patients in agreement? Exploring dyadic concordance. Health Educ Behav. 2013;40(5):603–11. doi:10.1177/1090198112473102. Epub January 23, 2013.

22.

Beach

Roter

Wang

Duggan

Cooper

. Are physicians’ attitudes of respect accurately perceived by patients and associated with more positive communication behaviors? Patient Educ Couns. 2006;62(3):347–54. Epub July 21, 2006.

23.

Chakrabarti

. What’s in a name? Compliance, adherence and concordance in chronic psychiatric disorders. World J Psychiatry. 2014;4(2):30–36. doi:10.5498/wjp.v4.i2.30.

24.

DeVoe

Fryer

Jr Straub

McCann

Fairbrother

. Congruent satisfaction: is there geographic correlation between patient and physician satisfaction? Med Care. 2007;45(1):88–94.

25.

Aiken

Sloane

Ball

Bruyneel

Rafferty

Griffiths

. Patient satisfaction with hospital care and nurses in England: an observational study. BMJ Open. 2018;8(1):e019189. doi:10.1136/bmjopen-2017-019189.

26.

Luck

Peabody

DeMaria

Alvarado

Menon

. Patient and provider perspectives on quality and health system effectiveness in a transition economy: evidence from Ukraine. Soc Sci Med. 2014;114:57–65. doi:10.1016/j.socscimed.2014.05.034. Epub May 21, 2014.