Abstract
Patient-reported experience measures (PREMs) are widely used for quality improvement, benchmarking, and physician compensation, yet few large-scale studies have examined interventions to improve scores. We examine the association of an annual, mandatory, peer-led continuing professional development (CPD) program at a large, multispecialty group practice aimed at enhancing patient–clinician communication. This quality improvement study analyzed ambulatory pre-post patient experience surveys from January 2018 to October 2022 for top box ratings for four PREMs focusing on provider communication for “explains”, “listens” to them, “overall provider rating,” and would “recommend” the practice were modeled using generalized estimating equations, adjusting for patient and provider characteristics. Over 1 million surveys per year were analyzed and the likelihood of top-box response in all four domains increased post implementation in 2019- 2022 compared to 2018 with all adjusted odds ratios > 1.0 and p ≤ 0.001. Our findings suggest CPD programs may improve PREMs in a large healthcare system, across all communication-related domains. However, there could be limits and a plateau to how much a CPD intervention can improve PREMs.
Keywords
Introduction
The nexus of a healthcare organization’s long term success is based on continual improvement of the quality of patient outcomes as well as quality of patients’ experiences. 1 Many healthcare systems use patient-reported experience measures (PREMs), such as National Research Corporation [NRC] surveys, to inform reporting, benchmarking, quality improvement initiatives and in some cases determine provider compensation.2,3 Most of the PREMs have questions targeted at provider communication. However, there is limited data on how differences in provider communication affect overall patient experience.4-8
There are several continuing professional development (CPD) courses to improve patient-provider communication; including standardized patients, 9 peer-to-peer provider coaching,10,11 in-person sessions, and virtual sessions. 7 Unfortunately, to date the findings on the effects of CPD interventions on patient experience scores have been at least limited, if not mixed.12-15
In 2018 the Palo Alto Medical Foundation (PAMF) developed an annual, mandatory, interactive, peer-led communication development program for providers called “Clinician PAMF CARES,” derived from the acronym PCARES: Prepare, Connect, Actively Listen, Reassure, Explain, Sincerely Appreciate. Here we describe our experience developing and implementing this program and addressing the following research question: What is the impact or association of Clinician PAMF CARES with PREMs?
As part of our program evaluation, we conducted retrospective analyses of ambulatory patient experience surveys or PREMs before (i.e. January-December 2018) and after (i.e. January 2019- October 2022) Clinican PAMF CARES to determine if annual provider training improved the patient experience scores on four domains focused on provider communication and overall practice ‘recommend’. In this study, we tested a hypothesis that CPD program – here, Clinician PAMF CARES – focused on improving communication can increase positive patient experience over time. We further investigated if patients of the providers who were Clinician PAMF CARES program facilitators are more likely to report positive experiences than those cared for by non-facilitators.
Methods
We conducted a retrospective observational review of PREMs data obtained from National Research Corporation (NRC) for ambulatory patients that were cared for by providers at PAMF from January 2018-October 2022. PAMF is a large, multispecialty ambulatory group practice in Northern California serving over 1 million patients annually. Patient and provider information was extracted from the electronic health records. The unit of analysis is the patient survey. The analysis compared PREMs before (January–December 2018) and after (January 2019–October 2022) implementation of the Clinician PAMF CARES program. We only examined 1 year of surveys before the intervention given that there was a change in survey vendors by the healthcare system and there were differences in the survey questions and administration. This study was determined as quality improvement (QI) by the Sutter Health Institutional Review Board and followed the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines.
About Clinician PAMF CARES
Initially, this program in 2018 was implemented as an in-person, 90-minute session with small groups of up to 12 providers, led by 2 provider facilitators, allowing for peer-to-peer teaching, discussion, and role-playing practice in a safe space, addressing how they might respond to common clinical communication challenges encountered in clinics and hospitals. Based on prior feedback from attendees, the program was modified in 2019 to eliminate the role playing “in the hot seat” and replaced by voluntary participation in a facilitator led discussion to reduce discomfort and increase engagement. In 2020, due to the COVID-19 pandemic, the program pivoted to virtual 90-minute sessions on Zoom (Zoom Video Communications, Inc., San Jose, CA), keeping small group sizes of no more than 12 clinician participants led by 2 clinician facilitators. Facilitators were volunteer providers and providers in leadership roles. Facilitators received approximately 4 hours of training, 2 hours in didactic; and 2 hours of active practice through role-playing/teaching a live class on how to facilitate. Each facilitator aimed to facilitate at least 5 or more courses each year.
The curriculum for that year was developed from a combination of well-known practices and learnings or reflections from patient experience champions and communication coaches. Examples of the topics that were presented over the years included “Requests for Unnecessary Tests or Medications,” “Giving Bad News,” “Communication on Perception of an Error,” “Potpourri of Challenging Patient Scenarios,” and “Written Communication”. The main framework for communication strategy was based on PCARES: prepare, connect, actively listen, reassure, explain and sincerely appreciate.
Each session began with facilitators laying ground rules for the attendees, such as maintaining privacy and confidentiality surrounding the experiences, both about patients and clinicians. The sessions continued with introductions and a warm-up activity to engage attendees, and to set an expectation of active participation. The facilitator prompted the group to share and reflect on their experiences throughout the session based on a hypothetical case and the topic selected for that year. During the initial discussion, the group was asked to share a time that they or their family members experienced a specific situation as a patient and reflect on that experience. This was aimed at helping the group to experience empathy. During the follow-up discussion the facilitator then asked participants to reflect on a similar circumstance, but where the attendee was the clinician. Throughout the session facilitators asked prompting questions and guided the discussion using techniques, such as summarization (e.g., “what I am hearing you say is [. . .]”) and/or validating the emotions expressed by the participants (e.g., “I can see how [. . .] could be difficult”). Facilitators highlighted pertinent patient phrases in communication to further focus on empathy, as well as introduced conflict resolution techniques.
At the end of each session, each participant was asked to summarize and share take-home points. After the session, the participants received a succinct handout summarizing the key points. Annual participation is mandatory and is tracked by the organization and with an annual participation rate of 98-99%. From 2021 onwards, we also included an invitation for a brief online survey via email, with questions about the facilitators, the content, and experience with the format. Response options were on a 5-point satisfaction Likert-like scale.
Outcome Measures
Patient Experience Survey Questions, Response Categories, and Top Box Coding for 2018-2022
In these multi-year analyses, we obtained provider demographics from Clarity, human resource and web services data: age, sex, race/ethnicity, language, and provider type. We constructed year-specific facilitator measures at the survey level. Providers were classified as being facilitators if they participated as facilitators during the same year of the patient visit/survey. We obtained demographics from the electronic health record (EHR) of patients who completed a patient experience survey during 2018-2022: age, race/ethnicity, sex, and language. Additionally, patient-provider concordance measures were assessed. These included gender concordance and racial concordance, defined as the provider and patient sharing the same recorded gender and race/ethnicity, respectively, and language concordance, defined as provider being able to speak the patient’s language at a competency level of conversational, medical, or higher. We had a data broker who linked all of the various data sources-survey, demographics-into a secure database that was then aggregated and analyzed by our statisticians.
Analytical Methods
We summarized the patient and provider characteristics at the survey level and computed the yearly top-box response rates for each of the four domains on “explain”, “listen”, “overall rating” and “recommend”. We used Generalized Estimating Equation (GEE) to analyze the association of the trend (year of visit), provider (facilitator vs. non-facilitator) with the probability of a top-box response. We chose GEE as it provides population-averaged estimates which align better with our objective of identifying factors associated with patient responses to inform system-level patient experience improvements. GEE is robust to misspecification of the working correlation structure and provides consistent estimates without having to make restrictive assumptions about the provider-level random effects. This robustness is important given the large and complex nature of the survey data and is also computationally efficient and stable for analyses at this scale. The GEE models accounted for the correlation of surveys from the same surveyed provider and were further adjusted by patient and additional provider characteristics described above. 22% of providers did not report their age, so provider age was not included in the GEE models. The working correlation structure minimizing the Quasi Information Criterion (QIC) was selected. All statistical analyses were performed using R version 4.4.1 and SAS V9.4 (Cary, NC).
Results
Patient and Provider Characteristics for all 4 Domains From 2018-2022
1n (%); Median (IQR).
Among responding patients, 61% were female, 65–67% were White, the median age was 66–67 years, and 96% reported English as their primary language. Surveyed providers were majority male (54–55%), 61–62% White, median age 49 years, and predominantly English-only speakers (83–84%) (details provided in Supplement 1). 77% of the providers with patient survey responses in 2022 were in the sample in 2018.
We found that during January 2019- October 2022, the top-box response rates increased from the 2018 rate in all 4 domains of PREMs during the post PAMF CARES period, though there were some fluctuations with “Recommend” (Figure 1). After adjusting for patient and provider characteristics and accounting for clustering within the provider, compared to 2018, the GEE model results showed that the likelihood of top-box response in all four domains increased significantly from 2019-2022 (Figure 2). Both “Explain” and “Listen” had a large increase from below 80% in 2018 to 90% or above in 2021 and 2022. “Overall Rating” also had a large increase, improving steadily year by year from 83% in 2018 to 90% in 2022. The “Recommend” score had the least improvement, with a medium increase from 84% in 2018 to 87% in 2020 and then plateauing afterward. The largest relative gain was in the “Listen” category with a 216% increase in the odds of top-box response in 2022 (Figure 2, additional details in Supplement 2). Top box responses to 4 PREM questions from all PAMF physicians January 2018-October 2022 Selected generalized estimation equation results
We also found that facilitators were more likely to receive a top-box response than non-facilitator providers in all domains, i.e. for “Recommend:” odd ratio [OR] 1.15 p=0.006; for “Overall rating:” OR 1.24, p<.001; for “Explain:” OR 1.57, p<.001; for “Listen:” OR 1.63, p< .001. Female providers were more likely to receive top-box response than male providers (“Recommend:” OR 1.06, p=0.007; “Overall rating:” OR 1.10, p<.001; “Explain:” OR 1.11, p<.001; “Listen:” OR 1.14, p < .001). Interestingly, the likelihood of top-box response was lower among female patients compared with male patients (“Recommend:” OR 0.97, p<.001; “Overall rating:” OR 0.94, p<.001; “Explain:” OR 0.90, p<.001; “Listen:” OR 0.86, p<.001).
Gender discordance between patient and provider was positively associated with slightly higher odds for top-box response for “Recommend” (OR 1.04, p<.001) and “Overall rating” (OR 1.03, p<.001) but lower odds for “Explain” (OR 0.97, p = 0.001). Patient-provider racial discordance was associated with lower odds of top-box response in all four domains (“Recommend:” OR 0.96, p=0.002; “Overall rating:” OR 0.96, p=0.005; “Explain:” OR 0.93, p<.001; “Listen:” OR 0.94, p < .001). Patient-provider language discordance was associated with lower likelihood of top-box response in two domains, “Recommend” (OR 0.92, p=0.003) and “Overall rating” (OR 0.93, p=0.04). Additionally, non-English-speaking patients were less likely to give top-box ratings across all domains (“Recommend:” OR 0.88, p<.001; “Overall rating:” OR 0.92, p=0.02; “Explain:” OR 0.56, p<.001; “Listen:” OR 0.56, p<.001). (Table 2) (more details in Supplement 2).
Lastly, our study evaluated provider surveys about their experience with Clinician PAMF CARES that we started collecting in 2021. Providers who reported “very satisfied” increased from 2021 to 2022 for the facilitator (89% to 91%), session content (70% to 76%), and ease of using the virtual format (83% to 88.0%) (Supplemental Table 4).
Discussion
Clinician PAMF CARESs developed each year for providers, guided and led by providers. The program creates a safe space for providers to share their experiences and challenges across specialties, thereby fostering a culture of psychological safety and collegiality, as well as increasing credibility that is important for continuing professional development (CPD). 20
Our findings suggest that Clinician PAMF CARES may be an effective, sustainable and generalizable intervention 4 that are associated with improvement in PREMs in a large healthcare system, as shown by the year over year increase in our patient experience scores in domains focused primarily on provider communication related to explaining information and listening, but has less effect on the broader construct of rating the provider and minimal influence on a patient’s recommendation of the practice. Our findings add to the limited literature on large-scale, sustained improvements in PREMs through communication-focused CPD. There are several important findings, implications and strengths to this study. It is feasible to successfully implement an interdisciplinary CPD session for providers in a large healthcare organization with high level of participation and satisfaction from providers. Factors that might contribute to success include the development of relevant material, reiterations of the format and content based on participant feedback and leveraging stakeholder engagement. Also, sustained improvement in patient experience scores might be due to clinicians continuing to receive an annual Clinician PAMF CARES session, potentially acting as a ‘booster session.’ This may provide a more consistent and persistent coaching effect 10 that falls within the suggested period of 6-12 months to maintain behaviors and scores. 21 Future research might consider what the optimal frequency of training is and if there may be added benefit from additional refreshers or trainings.
Our regression results suggest the importance of patient-clinician concordance for race/ethnicity, gender, and language. Similar to other studies, our results further contribute to existing evidence supporting language concordance may play a role in increased satisfaction of patient care with patient-clinician language concordant care22-24 as well as differences in patient responses that were correlated with patient race/ethnicity.25-27 These findings suggest that communication improvement efforts must explicitly address equity and there is a need for targeted strategies to ensure that gains are shared across all patient groups including diverse ethnicities and non-English speaking patients, as the effect of a peer-to-peer physician coaching program was found to differ for PREMS of Spanish vs English-preferring patients. 15
The strengths of this study include its large sample size, multi-year follow-up, system-wide scope, and robust statistical modeling. The inclusion of both patient and clinician perspectives enhances the relevance of the findings to real-world practice. However, there are also several limitations to this study. The main limitation is that our results are from a single ambulatory organization and our findings may not be generalizable. As an observational quality improvement evaluation, causal inferences cannot be made, and unmeasured confounders may have influenced the PREM results, such as factors outside of clinician control, including wait times in the clinic, facility environment, parking, provider access, interaction with staff, etc. Future studies could consider alternative prospective designs, such as stepped-wedge or quasi-experimental approaches, to strengthen causal inference while remaining compatible with real-world quality improvement settings.
Another limitation is that the patient population was predominantly English-speaking and White, so results may not generalize to more linguistically or ethnically diverse settings. The COVID19 pandemic occurred during the evaluation period, which resulted in more patient telehealth visits and that the clinician training being conducted virtually. Given these changes, we would have expected to see a global decrease in patient experience scores, yet, there was not one at our organization. This may suggest that the communication techniques discussed in these classes could be applicable to any visit modality. The potential differences between telehealth and in-person visits warrants future investigation, as well as further understanding the impact of facilitators previous communication skills versus their role of leading these sessions.
Our findings show that there may be a ceiling effect of these interventions over time as reflected by plateauing of the scores, particularly when scores are approaching 90 and above, in 2022 for all domains.10,28 Lastly, the clinician’s experience survey response rates for PAMF CARES were approximately 30%, raising the possibility of response bias. Also, the data set was limited to 2021 and 2022 only as we did not have surveys prior to these years and may not reflect earlier years of 2018-2020.
Care should be taken to both report and strive toward improvements on an aggregated level, rather than an individual level. For example, using such scores for provider compensation and/or promotion decisions may disincentivize providers from caring for a diverse patient population and/or accelerating job dissatisfaction. 29 Additionally, fear of such use of scores might negatively affect recruiting, mentoring, and retention of medical students, residents, and practicing providers, especially those who identify as members of underrepresented minority groups. 30
Future research should explore targeted interventions to address the lower PREM scores among non-English-speaking patients and in racially discordant encounters. More studies could examine whether improvements in PREMs translate into downstream effects, such as increased patient adherence, improved health outcomes, or reductions in complaints. Another area would be to understand how skills and communication styles of facilitators may change as a result of leading these sessions.
Conclusion
Clinician PAMF CARES, a peer-led, interactive communication development program, was associated with sustained, system-wide improvements in patient-reported experience measures, related primarily to explaining information, listening, and overall provider ratings, over four years in a large multi-specialty group practice.
Supplemental Material
Supplemental Material - Impact of a Physician Communication Development Program on Patient Experience Scores
Supplemental Material for Impact of a Physician Communication Development Program on Patient Experience Scores by Cheryl D. Stults, Su-Ying Liang, Yian Guo, Diana Nguyen, Ridhima Nerlekar, Tony Chen and Minal Bhanushali in Journal of Patient Experience.
Supplemental Material
Supplemental Material - Impact of a Physician Communication Development Program on Patient Experience Scores
Supplemental Material for Impact of a Physician Communication Development Program on Patient Experience Scores by Cheryl D. Stults, Su-Ying Liang, Yian Guo, Diana Nguyen, Ridhima Nerlekar, Tony Chen and Minal Bhanushali in Journal of Patient Experience.
Supplemental Material
Supplemental Material - Impact of a Physician Communication Development Program on Patient Experience Scores
Supplemental Material for Impact of a Physician Communication Development Program on Patient Experience Scores by Cheryl D. Stults, Su-Ying Liang, Yian Guo, Diana Nguyen, Ridhima Nerlekar, Tony Chen and Minal Bhanushali in Journal of Patient Experience.
Supplemental Material
Supplemental Material - Impact of a Physician Communication Development Program on Patient Experience Scores
Supplemental Material for Impact of a Physician Communication Development Program on Patient Experience Scores by Cheryl D. Stults, Su-Ying Liang, Yian Guo, Diana Nguyen, Ridhima Nerlekar, Tony Chen and Minal Bhanushali in Journal of Patient Experience.
Footnotes
Acknowledgements
We would like to thank Adam Sukhija-Cohen, PhD and Jeffrey M Hoyt, PhD for their assistance in editing the manuscript. A previous version of this work was shared in an oral presentation at the April 2024 Health Care Systems Research Network (HCSRN) Conference in Milwaukee, Wisconsin.
Ethical Considerations
This evaluation was reviewed by the Sutter Health Institutional Review Board and determined to be quality improvement. Implied consent was obtained for completing surveys by both patients and clinicians.
Author Contributions
Concept and design: Stults, Liang, Bhanushali; Acquisition, analysis, or interpretation of data: All authors; Drafting of the manuscript: All authors; Statistical analysis: Guo, Nerlekar; Obtained funding: Stults; Administrative, technical, or material support: All authors; Supervision: Stults, Liang
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Palo Alto Medical Foundation Philanthropy funded the evaluation. The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
The data underlying this article cannot be shared publicly due to privacy of individuals since data may contain identifiable participant information.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
