Sage Journals: Discover world-class research

Abstract

We aim to incorporate physician insights into the development of a performance dashboard based on a novel patient experience survey system at a large academic health system. A multidisciplinary team of physicians, researchers, and designers met regularly to develop the dashboard informed by Clinical Performance Feedback Intervention Theory. Semistructured qualitative interviews with frontline physicians underwent combined inductive-deductive thematic analysis to inform dashboard improvements. A total of 20 interviews were conducted April-July 2023 with 9 neurologists and 6 primary care physicians in 2 phases. Physician preferences converged along design, content, and administration features, which were incorporated into the dashboard and accepted by physicians during phase 2. Important themes included establishment of survey credibility, preference for qualitative over quantitative data, and associated incentive structure. Dashboard design required an intuitive data overview with features such as hover-over insights that allowed deeper exploration. Physicians valued patient comments over quantitative data, prompting further redesign of the dashboard to prioritize qualitative insights, contrasting with the national emphasis on quantitative benchmarks.

Keywords

patient experience performance feedback physician wellbeing quality improvement coaching

Introduction

Patient satisfaction is an important health services outcome and is also correlated to other health outcomes. Satisfied patients are more likely to adhere to treatment plans, exhibit greater efficacy in self-care, experience better health outcomes, and are less likely to change providers or pursue malpractice claims.^1-4

In the early 2000s, the implementation of standardized patient-facing surveys aimed to enable interinstitutional comparison and enhance patient satisfaction over time.¹ Despite their widespread adoption, however, these surveys have been subject to significant criticism. They often fail to capture the nuanced complexities of clinical care, focus on elements beyond physician control such as room cleanliness, and may inadvertently pressure physicians to provide inappropriate care.⁵

Other critiques of patient-facing surveys include poor actionability of data, the focus on individual episodes rather than overarching experiences of care, inappropriate comparisons, a ceiling effect whereby small outliers cause outsized effects, burdensome survey length resulting in low response rates, and timeliness of data analysis to inform improvement.^6-8 Moreover, they are susceptible to racial, gender, and age biases.^9-11

Feedback on individual and team performance is a standard management tool in healthcare,^1,12 but traditional instruments for measuring and presenting patient satisfaction are often met with resistance from physicians and may exacerbate physician discontent.^13-15 Physicians’ disfavor is rooted in concerns over data presentation, including peer comparison, “likelihood to recommend,” “top-box scores,” and the inclusion of clinical aspects perceived to be beyond their control.^8,12,16

User-centered dashboard design, which integrates end-user perspectives in an iterative process, offers promise in enhancing effectiveness and physician acceptance of performance dashboards. Successful implementation benefits from early stakeholder engagement and dynamic visualization of the data.^17-20 Recent studies, such as Khanbai et al,²¹ demonstrate the effectiveness of co-design in developing patient satisfaction dashboards that garner increased acceptance among physicians.

Given objections to both the content and the presentation of patient satisfaction measures, some health systems are developing in-house measurement systems tailored to their specific needs. We aim to incorporate primary care and specialist physician insights into the development of a performance dashboard based on a novel patient experience survey system at a large academic health system.

Methods

A large academic health system previously captured patient satisfaction scores using a standardized program (Press Ganey, South Bend, IN, USA). Given a desire for increased flexibility and actionability of data, the need to capture longitudinal care over time, suboptimal physician experience and other needs, they embarked on a multiyear journey starting in 2019 to develop an internal patient satisfaction measurement system with the support of quantitative survey experts.⁶ Survey developers conducted patient and physician interviews to develop and validate both the content and the presentation of the novel patient experience survey; this article focuses on the development of the physician-facing performance dashboard.

A multidisciplinary team of quality and wellbeing physician leaders from the Department of Neurology, health service researchers from the Evaluation Sciences Unit,²² and designers from the health care's Office of Patient Experience met regularly to support the design of the dashboard based on user feedback through a multiphase approach. A product developer and digital designer crafted an initial dashboard mockup, drawing insights from the team's examination of clinician feedback best practices based on the Clinical Performance Feedback Intervention Theory (CP-FIT).²³ First published in 2019 based on a systematic review and meta-synthesis of qualitative research, CP-FIT outlines a set of feedback variables (eg, display), recipient variables (eg, health professional characteristic), and context variables (eg, team characteristics) that operate via certain mechanisms (eg, social influence) to drive a multipart feedback cycle. The authorship team first individually and then collectively reviewed feedback best practices based on the CP-FIT interpretation of existing literature.²³ These were applied to the initial design of the dashboard in a series of iterative meetings between the authorship team and the dashboard designer and product developer.

Given the national emphasis on quantified likelihood to recommend “top-box” scores and guidance from the organization's leadership, these scores were included in all iterations of the dashboard. The “top-box” score is defined as the percentage of patients who scored the physician 5 out of 5 possible points on a particular measure. “Likelihood to recommend” indicates how likely patients would be to recommend the physician to a friend or family member.” A“top-box” score of 30% for “likelihood to recommend,” therefore, indicates that 30% of patients gave the highest score, “5,” to the question “How likely would you be to recommend this doctor to a friend or family member.”

Attending faculty physicians with a practice in Primary Care or Urgent Care or neurologists with an ambulatory outpatient practice were recruited initially through a pre-existing directory.¹⁶ The remainder of interviewees were recruited via purposive sampling to represent a diversity of subspecialties, genders, years of practice, and races/ethnicities. These specialties were chosen given high prevalence of physician burnout in our healthcare system and past experience with this type of feedback.^24-26 While participants may have self-selected to participate based on strong opinions on patient feedback mechanisms, the insights provided were seen as important to inform future implementation of the dashboard, particularly to address preemptively those aspects that participants identified as harmful.

All potential interviewees were contacted by email, first by a department leader or physician researcher (authors CG or SV) and then by a qualitative researcher (CB). All interviews were conducted via video conferencing (Zoom, San Jose, CA). Participation was voluntary and not incentivized. Each participant gave informed consent prior to the interview. This evaluation was reviewed by our Institutional Review Board and did not meet the definition of human subject research (protocol ID #60537).

User feedback was collected in 2 phases following the Plan Do Study Act quality improvement framework in which improvements are evaluated in an iterative cycle.^27,28

In phase 1, semistructured interactive interviews focused on the dashboard's general appearance, appropriateness of highlighted information, anticipated impact on behavior, incentivization structure, clinical leader review, and improvement recommendations (see Supplemental Appendix A). Interviews utilized “think aloud” techniques, adapted to prompt participants during long silences and to probe for specific topics.²⁹ Participants were asked to verbally express all thoughts, reactions, questions, and actions as they viewed the prototype dashboard via screen-sharing functionality.

The prototype dashboard was updated for phase 2 by a product designer based on the team's synthesis of phase 1 feedback. Phase 2 focused on technical aspects, incentive to use, and improvement recommendations for the dashboard and future deployment. Participants directly interacted with the dashboard through the ability to remotely take over the interviewer's screen via the video conferencing application during interviews.

Interviews were conducted from April to July 2023. Round 1 interviews ranged from 29 to 54 min (mean 39 min). Round 2 interviews ranged from 18 to 34 min (mean 25 min). We targeted 5 participants from both specialties (neurology and primary care) per round of interviews to sufficiently identify problems and understand user experience of the dashboard prototype; these methods are consistent with usability research methods.³⁰ The total number of interviews (20) is consistent with practices in qualitative research to identify themes such as incentive to use and impact on physician behavior.³¹

Interviews were audio-recorded, transcribed verbatim (Rev, San Francisco, CA), and analyzed through a multistep, inductive approach to identify key improvement recommendations.

Specifically, a qualitative researcher (CB) analyzed interview transcripts using rapid thematic analysis to provide timely, actionable feedback to dashboard designer.³² We used the Stanford Lightning Report method,³³ a version of rapid thematic analysis which has been used previously to successfully inform in-progress implementation and was used here to inform in-progress design.³⁴ Interview results were captured in a matrix where each interview entailed 3 rows (patient comments, quantitative data, and impact on behavior) and 3 columns Plus (“what works”), Delta (“what needs to be changed”), and Insight (participant or evaluator insights, ideas, and recommendations).³³ This preliminary analysis was verified by a random review of 3 of the 10 interviews in each of the 2 phases by a second researcher (SV).

Following the analysis of each phase, all authors reviewed the findings to independently identify dashboard improvement recommendations and key themes. These insights were integrated through consensus discussions, and final recommendations were shared with the designer to inform subsequent revisions.

Results

A total of 20 interviews were conducted in 2 phases comprised of 9 unique neurologists and 6 unique primary care physician perspectives. During phase 1, 6 neurologists and 5 primary care physicians agreed to participate out of a total of 13 and 8 who were asked, respectively. During phase 2, 6 neurologists, 3 of whom participated in phase 1, and 3 primary care physicians, 2 of whom participated in phase 1, agreed to participate out of 10 and 6 who were asked, respectively (see Supplemental Appendix B).

A visual representation of the dashboard was presented in static form during phase 1 (Figure 1) and in dynamic, interactive form during phase 2 (Figure 2). Sample patient comments were drawn from real-world data unrelated to the individual physician participant.

Figure 1.

Patient satisfaction performance feedback dashboard phase 1.

Figure 2.

Patient satisfaction performance feedback dashboard phase 2.

Physicians were open to engaging with patient experience data via a dashboard as a part of their job responsibilities, as long as the data were relevant, valid, and actionable:

“…if I felt like this data is not accurate, I think that would dissuade me from checking in. Because if you think about it, what is the value for me? I think the biggest value is, ‘do I have a blind spot?’” (R2 Participant 2, Neuro)

Specific feedback collected during phases 1 and 2 are summarized in Table 1, including the degree of consensus, exemplary quotations, and actions taken to improve the dashboard between phases following evaluation and implementation team discussions.

Table 1.

Key Takeaways of a Patient Satisfaction Performance Feedback From 2 Phases Based on Interviews With Practicing Neurologists and Primary Care Physicians.

	Phase 1			Phase 2
Feature	Participant recommendation	Degree of consensus & exemplary quotation	Action taken following discussion with leadership and survey teams	Participant recommendation	Degree of consensus & exemplary quotation	Action taken following discussion with leadership and survey teams
Overall layout	Unintuitive design	Majority: “On first glance, without instruction, it would be hard for me to figure out what these [performance lines] mean.” (R1 Participant 4, Neuro)	Added labeled x-axis; updated colors; aligned corresponding charts horizontally; added n-size; updated short-hand question names	Dashboard design is acceptable; a demonstration video may be helpful	Majority: “I think everything looks really great, and I like all of the options.” (R2 Participant 4, Neuro) Minority: “You've done a really good job to make it pretty self-explanatory, but [a demo video would] save someone like five minutes."(R2 Participant 2, Neuro)	Consensus that design should be so intuitive that a demonstration video is not required; under ongoing consideration if this will be needed after assessing engagement following deployment
Top-box score	Use of top box decreases engagement and negatively impacts physician wellbeing	Mixed: “If you care about them [top box score], and you're less than perfect, then you feel absolutely awful.” (R1 Participant 7, PC)	Consensus for now this industry-wide standard must remain	[Unchanged] Use of top box decreases engagement and negatively impacts physician wellbeing	Majority: “Oh, anything that's not top box gets zero. I don't like that.” (R2 Participant 1, PC)	No change though under ongoing consideration
Survey items displayed	Information overload versus desire to see all data; curiosity as to why some survey items are displayed over others	Mixed: “It seems overwhelming.” (R1 Participant 4, Neuro) “It's good to see all questions that are asked.” (R1 Participant 7, PC)	Consensus on the need to prevent information overload; displayed survey items can be changed dynamically based on leadership preference	Not specifically brought up by participants in phase 2	None	No change
Link to training	Access to training resources appreciated	Majority: “I think maybe that addresses the one concern I had about being able to act on these things. So that if I see something that I don't like here, I can click on a link …it can say, ‘Hey, you can make your comprehension score better.’” (R1 Participant 4. Neuro)	No change	Less conspicuous link to optional training	Minority: “I wonder if it might be off on the side, not in your face… ‘if you want any further training, or if you're looking for training, here are some resources that might be helpful.” (R2 Participant 1, PC)	Minor design changes
Comparison to other physicians in the same clinical division	Remove division comparison	Mixed: “It's not the right way to measure someone or someone's performance.” (R1 Participant 1, Neuro) “…If you removed it, then you have no way to gauge what that information means.” (R1 Participant 3, Neuro)	Removed division comparison	Not specifically brought up by participants in phase 2	None	No change
Hover-over feature sharing additional information	Hover-over would add important clarity for both quantitative data and patient comments	Majority: “I always like that feature. That's good to have.” (R1 Participant 10, Neuro) “It is also nice that it had an n size, so that [I know] if it's only 2 people who were interviewed versus 120.” (R1 Participant 7, PC)	Added hover-over features to display entire patient comment and sample size (denominator)	Hover-over feature provides clarity but one participant found this feature distracting	Minority: “The one thing that gets a little bit, I don't want to say annoying or gets too busy…as soon as I move my mouse, this pops up… I have to keep moving my mouse around….it seems to be almost too obtrusive…” (R2 Physician 7, Neuro)	No change
Emphasis on patient comments	Patient comments most valuable aspect of dashboard	Majority: “So, I feel like that feedback [via patient comments] can be very useful, because that's something specific that the patients are looking for. And if you see a common trend, then that's a behavior you can act upon.” (R1 P2, Neuro)	Consensus that patient comments should be emphasized over quantitative measures; patient comments tab moved to left as the default landing tab	Not specifically brought up by participants in phase 2	None	No change
Link to clinical encounter	Link patient comments to specific clinical encounter in the electronic health record	Majority: “But if I could link it to [encounter]…all the answers from this one person, that would be more helpful.” (R1 Participant 5, Neuro)	Consensus that patient confidentiality should take priority over providing physicians additional context; no change	Not specifically brought up by participants in phase 2	None	No change
Background information on the survey	Framing language overlooked given small text and location at the bottom of page; persistent desire to understand how survey was developed and what it entails	Majority: “I think where the numbers come from, some background, who are these sent to? What's the average response rate….Why is it done? Kind of like basic information. Otherwise, you just get hit with the data right away and you don't really have a background for where it's coming from.” (R1 Participant 7, PC)	Framing language removed and replaced with “Hover to view additional information”, which describes survey in more depth	Ongoing desire to understand how survey was developed and what it entails	Majority: “I'd love to know more about this instrument…How was it designed? Is it validated? How does it compare to Press Ganey? What was the thought process in terms of switching to the survey as opposed to keeping with Press Ganey?” (R2 Participant 5, PC)	Consensus to develop a link to an internal website page that describes survey in more depth; in process
Engagement prompt	Reminders are required; access should be easy	Majority: “One thing you could imagine doing is that it could send an automatic email, saying, ‘Hey, your report for the past month is ready.’… I think for me, that would be very key. If I don't get prompted by it, then I probably won't look at it.” (R1 Participant 4, Neuro)	Consensus that department level email with personalized 1-click link is needed, which will allow for the tracking of user engagement	[Unchanged] Reminders are required; access should be easy	Majority: “If it was put in front of my face in an email, I would do that [engage] more often.” (R2 Participant 1, PC) “Needs to be easy, quick.” (R2 Participant 10, PC)	No change
Engagement incentive	Engagement-based incentive preferred over performance-based incentive	Majority: “Awesome [engagement-based incentive]” (R1 Participant 8, PC) “I would totally understand if people would be worried about a bait and switch, meaning initially it's about engagement and then it's about performance.” (R1 Participant 9, PC)	Engagement-based versus performance-based incentive under ongoing consideration	Not specifically brought up by participants in phase 2	None	No change
Clinical leader review	Review of data by a clinical leader anticipated and accepted; mixed perspective as to who should review	Mixed: “And I can imagine this being one added component to what we talk about [during an annual review]. I think that would make sense.” (R1 Participant 3, Neuro) “I don't think it should be your division chief. I think it should be someone that is not part of your department…” (R1 Participant 2, Neuro)	Under ongoing consideration; division chiefs to be provided with performance data and will have discretion over how data is discussed with physicians	Not specifically brought up by participants in phase 2	None	No change

In phase 1 (left side of Table 1), most physicians agreed on several features: overall layout, link to training, hover-over feature, emphasis on patient comments, link to clinical encounter, background information, engagement prompt and engagement incentive. Perspectives were mixed on other features including top-box score, number of survey items displayed, division comparison, and clinical leader review.

Significant changes were made to the dashboard when either a majority of perspectives aligned in the same direction (eg, the importance of qualitative patient comments over quantitative findings) or perspectives were mixed (eg, how colors were used to display quantitative data). In the latter case where disagreement occurred, the evaluation team relied on the dashboard designer to use her expertise to make favorable design changes.

By phase 2 (right side of Table 1), there was greater agreement across these features. The 5 returning participants from phase 1 expressed satisfaction with the changes that had been made. A minority of physicians still wanted to see other improvements following phase 2. These included an accompanying demonstration video, a less conspicuous link to additional training, and less-distracting hover-over features.

Three topics inspired considerable discussion among physicians and subsequently, the evaluation and survey implementation teams, as they sought to improve the dashboard: (1) imparting survey credibility; (2) qualitative over quantitative data, and (3) the associated incentive structure.

Imparting Background Information to Support Survey Credibility

Most physicians wanted to know more about the development of the dashboard:

“I'm cautiously optimistic that the institution is moving away from [legacy survey system], given…deficiencies. I think the big gap in my knowledge is what is this [novel] instrument? What is the evidence behind this instrument?” (R2 Participant 5, PC)

Leading up to phase 1, the evaluation team anticipated these needs based on the prior site-specific evaluation.¹⁶ Background information was included at the bottom of the phase 1 dashboard image to: (1) emphasize the shared goal of improving the patient experience; (2) establish the relevance, validity, actionability of survey data; (3) acknowledge unavoidable biases in survey instruments; and (4) establish a process to remove patient responses if they demonstrated abusive and/or drug-seeking behavior. Interviews during phase 1 revealed the provision of the background information was ineffective because physicians could not readily digest it as presented; it therefore did not impact their user experience.

Successfully imparting survey background information therefore posed a challenge, as perspectives were mixed on how thoroughly users wanted to engage with the tool:

“I think the five-question view is useful for someone who just wants to know if they're doing okay and then if they have any extra time to dig into the details. I think all of these data are useful.” (R1 Participant 9, PC)

“It seems overwhelming.” (R1 Participant 4, Neuro)

Discussions between the evaluation and survey implementation teams led to a layered approach for phase 2: hover-over features and links to click were added that shared more information only if the user was interested in learning more. While it was not technically feasible to implement all planned features prior to phase 2 interviews, these interviews revealed the layered approach was more acceptable to most participants.

Qualitative Data Preferred Over Quantitative Data

All physicians were ultimately most interested in the patient comments, which they felt were more actionable and beneficial to their practice than quantitative data. A few felt the quantitative data could serve as a red flag that more investigation into the qualitative was needed:

I'm not that terribly interested in the numbers… Outlier numbers correlated or substantiated by actual comments that were behavioral, I think would be interesting for myself and for coaching others. (R1 Participant 9, PC)

These comments ultimately led to a content and design shift that prioritized the qualitative over the quantitative. Specifically, patient comments became the default landing page in phase 2, which physicians accepted.

Engagement-Based Incentive Preferred Over Performance-Based Incentives

Physicians nearly universally preferred engagement-based financial incentives over performance-based incentives, given concerns regarding the underlying validity of the data. Only one individual expressed doubt and thought a performance incentive was still needed: “I think maybe that addresses the one concern I had about being able to act on these things…You have to have some kind of incentive to do it.…[otherwise] you're going to lose a lot of people” (R1 Participant 4, Neuro).

A few distrusted the sustainability of an engagement-based incentive from an administrative perspective: “We've been burned in the past” (R1 Participant 5, Neurology). Thus, despite favorable perspectives of the proposed financial incentive structure, the topic remained an area of concern.

Limitations to Integrating Physician Recommendations

The teams were not able to accommodate every recommendation made during phase 1 due to administrative and technical challenges.

Top-box score: Perspectives were nearly universally negative on the inclusion of the top-box score. Whereas some physicians advocated for its removal, a few found ways to move past it: “I feel like so much of it is out of my control that I can't really get worked up about the score.” (R1 Participant 6, PC). The evaluation team was unable to secure the removal of top-box scores given the preferences of health system leaders.

Clinical context: Several physicians wanted a link from a patient comment that would go directly to the unique clinical encounter from which the comments were drawn. They felt the clinical context was necessary to integrate patient recommendations. While this was technically feasible, it would reveal the patient identity to the physician. Administrative leadership has access to this information to facilitate appropriate follow-up with patients, but physician access would be a significant departure from the status quo. After an exploration of the issue with leadership, it was felt that that protecting anonymity of comments took precedence over providing physicians with clinical context to protect patient trust in both the health system and the survey system.

Ability to flag inappropriate clinical encounters: Physicians desired a process in which they could flag inappropriate clinical encounters (eg inappropriate request for opioids) such that they would not be punished for any unfavorable ratings resulting from a patient's disagreement with appropriate care. While this feature could not be integrated prior to Phase 2 interviews, the health system is exploring inclusion of this feature for survey deployment.

Engagement exclusively through email: Finally, physicians universally requested that access to the dashboard be made as simple as possible, and several wanted to receive the information directly to their email or electronic health record inboxes. While a static image of the dashboard within the email could depict the relevant information in a simple format, it would not facilitate tracking engagement metrics. Thus, the decision was made to work towards a periodic email that prompted engagement through a single click.

Discussion

We evaluated primary care and neurologist physicians’ perspectives of a novel patient experience performance feedback dashboard at a large academic medical center through an iterative, user-centered design approach. Overall, physicians were receptive to the dashboard, especially after several of their suggestions were incorporated. However, mixed perspectives and administrative constraints prevented the integration of all recommendations.

A key theme that emerged was the importance of establishing trust in the validity and actionability of the novel survey system. While many participants expressed a desire for more contextual information on the survey's development, validation, and proposed implementation, others felt such information as a component of the tool would overwhelm users.

A significant design challenge was providing a clear overview of the data while allowing users to intuitively explore deeper insights through hover-over features (eg, to display sample size) and links to additional information (eg, a webpage the describes survey development). The survey implementation team recognized the importance of a thoughtful system wide roll-out, as busy physicians were likely to quickly assess the credibility of the new survey system and engage accordingly.

Where physicians preferred to spend their limited attention was on patient comments.¹⁶ This feedback led to a fundamental shift in the dashboard design, which put the primary focus on qualitative data rather than quantitative data. This preference echoes former work^16,35 and runs counter to the historical and ongoing national trend of comparing and incentivizing providers based on quantitative benchmarks—the utility of which remains under scrutiny.⁸ The shift in the focus of the dashboard based on physician engagement in the design process echoes the findings of other recent studies.^36,37

The rise of artificial intelligence (AI) gives us an opportunity to rethink our assessment of patient experience in this regard. AI can quickly summarize large swaths of narrative patient experience data, akin to current AI-based summarization of customer product reviews.^35,38 Further, such data has also been shown to correlate to existing quantitative measures,³⁹ suggesting that with further validation, open-ended prompts could supersede our predominant reliance on quantitative data; future validation should include assessment of potential bias and ethical issues. Finally, patient comment data may be further transformed to provide AI-based coaching to clinicians, a method which has shown early promise in the surgical setting.⁴⁰ Future research can explore practical implementation and physician acceptance of these technologies. The findings of this research may inform broader healthcare policy related to patient experience data and might lead to greater engagement in continuous improvement among physicians.

This study focused on the physician experience of the dashboard among 2 physician specialties; additional physician, patient and administrator and experience research is needed to take into account broader end-user perspective to fully optimize the novel survey system. Further, a self-selection bias of individuals with an interest in quality and/or wellbeing improvement may skew perspectives more favorably, in the case of likelihood to engage in the novel survey system, or unfavorably, such as overestimating the negative potential impact of the novel survey system on provider wellbeing. A purposive sampling sought to mitigate this bias, through a wider sampling of user data and perspectives following implementation are needed.

Conclusion

Neurologists and primary care physicians in a large academic health system accepted a performance feedback dashboard based on a novel patient experience survey following content and design changes. Early incorporation of physician stakeholder feedback in dissemination of performance metrics is feasible and may increase adoption, and should be considered as a best practice when developing new dashboards that relate to physician performance. Concerns remained regarding top-box score, consideration of the clinical context of patient ratings, and incentive structure.

Supplemental Material

sj-docx-1-jpx-10.1177_23743735251341724 - Supplemental material for Towards a Novel Patient Experience Survey System: Incorporating Physician Perspectives into Performance Feedback Dashboard Design

Supplemental material, sj-docx-1-jpx-10.1177_23743735251341724 for Towards a Novel Patient Experience Survey System: Incorporating Physician Perspectives into Performance Feedback Dashboard Design by Stacie Vilendrer, Cassandra Bragdon, Rebecca K Miller-Kuhlmann, Nirali Vora, Carl A Gold and Marcy Winget in Journal of Patient Experience

Footnotes

Acknowledgements

The authors would like to thank Alpa Vyas, Mysti Smith-Bentley, Joshua Frost, and Siying Wang for their spirit of collaboration and improvement and for their generous support; and Dr. Frank Longo, Dr. Paul Fisher, Dr. Antonio Omuro, and Dr. Eva Weinlander for empowering improvement efforts.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was generously supported through an internal grant from the Stanford Health Care Patient Experience Office and in-kind support from the Stanford Department of Neurology and Neurological Sciences..

Ethics Statement

This study was reviewed by our organizations Institutional Review Board (IRB) and it was classified as not human subjects.

Statement of Human and Animal Rights

This article does not contain any studies with human or animal subjects.

Informed Consent

Verbal informed consent was obtained from the interviewees for their anonymized information to be published in this article.

ORCID iD

Cassandra Bragdon

Marcy Winget

Supplemental Material

Supplemental material for this article is available online.

References

Richard Bolton Siegrist

. Patient satisfaction: history, myths, and misperceptions. AMA J Ethics. 2013;15(11):982-987. doi:https://doi.org/10.1001/virtualmentor.2013.15.11.mhst1-1311

Why Improve Patient Experience? Agency for Healthcare Research and Quality. 2020. Accessed October 10, 2023. https://www.ahrq.gov/cahps/quality-improvement/improvement-guide/2-why-improve/index.html

Glickman

Boulding

Manary

, et al. Patient satisfaction and its relationship with clinical quality and inpatient mortality in acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(2):188-195. doi:https://doi.org/10.1161/CIRCOUTCOMES.109.900597

Manary

Boulding

Staelin

Glickman

. The patient experience and health outcomes. N Engl J Med. 2013;368(3):201-203. doi:https://doi.org/10.1056/NEJMp1211775

Darnall

Schatman

. Autonomy vs paternalism in the emergency department: the potential deleterious impact of patient satisfaction surveys. Pain Med. 2013;14(7):968. doi:https://doi.org/10.1111/pme.12170

Vyas

Allen

Brown

, et al. Measuring what matters: a proposal for reframing how we evaluate and improve experience in healthcare. Patient Exp J. 2022;9(1):5-11. doi:https://doi.org/10.35680/2372-0247.1696

North

Tulledge-Scheitel

. Patient satisfaction with providers: do patient surveys give enough information to help providers improve specific behaviors. Health Serv Res Manag Epidemiol. 2019;6(11):233339281988528. doi:https://doi.org/10.1177/2333392819885284

Richman

Schulman

. Are patient satisfaction instruments harming both patients and physicians? JAMA. 2022;328(22):2209. doi:https://doi.org/10.1001/jama.2022.21677

Sotto-Santiago

Slaven

Rohr-Kirchgraber

. (Dis)Incentivizing patient satisfaction metrics: the unintended consequences of institutional bias. Health Equity. 2019;3(1):13-18. doi:https://doi.org/10.1089/heq.2018.0065

10.

Hall

Gulbrandsen

Dahl

. Physician gender, physician patient-centered behavior, and patient satisfaction: a study in three practice settings within a hospital. Patient Educ Couns. 2014;95(3):313-318. doi:https://doi.org/10.1016/j.pec.2014.03.015

11.

Chen

Zou

Shuster

. Relationship between patient satisfaction and physician characteristics. J Patient Exp. 2017;4(4):177-184. doi:https://doi.org/10.1177/2374373517714453

12.

Rosenthal

Shortell

Shah

, et al. Physician practices in accountable care organizations are more likely to collect and use physician performance information, yet base only a small proportion of compensation on performance data. Health Serv Res. 2019;54(6):1214-1222. doi:https://doi.org/10.1111/1475-6773.13238

13.

Schneider

Ehsanian

Schmidt

, et al. The effect of patient satisfaction scores on physician job satisfaction and burnout. Future Sci OA. 2020;7(1):FSO657. doi:https://doi.org/10.2144/fsoa-2020-0136

14.

Aleksandra

Rabago

Miller

Impact of patient satisfaction ratings on physicians and clinical care.

Patient Prefer Adherence. 2014;8(4):437-446. doi:https://doi.org/10.2147/PPA.S59077

15.

Vilendrer

Kling

SMR

Wang

, et al. How feedback is given matters: a cross-sectional survey of patient satisfaction feedback delivery and physician well-being. Mayo Clin Proc. 2021;96(10):2615-2627. doi:https://doi.org/10.1016/j.mayocp.2021.03.039

16.

Vilendrer

Levoy

Miller-Kuhlmann

, et al. Physician perceptions of performance feedback and impact on personal well-being: a qualitative exploration of patient satisfaction feedback in neurology. Jt Comm J Qual Patient Saf. 2023;49(3):138-148. doi:https://doi.org/10.1016/j.jcjq.2022.12.003

17.

Dowding

Randell

Gardner

, et al. Dashboards for improving patient care: review of the literature. Int J Med Inf. 2015;84(2):87-100. doi:https://doi.org/10.1016/j.ijmedinf.2014.10.001

18.

Dowding

Merrill

Russell

. Using feedback intervention theory to guide clinical dashboard design. AMIA Annu Symp Proc. 2018;2018:395-403.

19.

Lagha

Burningham

Sauer

, et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform. 2020;11(4):528-34. doi:https://doi.org/10.1055/s-0040-1714693

20.

Kemp

Santana

Quan

. Interactive data visualization of patient experience and inpatient datasets using tableau desktop. Int J Popul Data Sci. 2018;3(4):042-042. doi:https://doi.org/10.23889/ijpds.v3i4.632

21.

Khanbhai

Symons

Flott

, et al. Enriching the value of patient experience feedback: web-based dashboard development using co-design and heuristic evaluation. JMIR Hum Factors. 2022;9(1):e27887. doi:https://doi.org/10.2196/27887

22.

Vilendrer

Saliba-Gustafsson

Asch

, et al. Evaluating clinician-led quality improvement initiatives: a system-wide embedded research partnership at Stanford medicine. Learn Health Syst. 2022;6(4):1-13. doi:https://doi.org/10.1002/lrh2.10335

23.

Brown

Gude

Blakeman

, et al. Clinical performance feedback intervention theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14(1):40. doi:https://doi.org/10.1186/s13012-019-0883-5

24.

Berg. At Stanford, physician burnout costs at least $7.75 million a year. American Medical Association. November 17, 2017. Accessed January 25, 2024. https://www.ama-assn.org/practice-management/physician-health/stanford-physician-burnout-costs-least-775-million-year

25.

Shanafelt

West

Sinsky

, et al. Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017. Mayo Clin Proc. 2019;94(9):1681-1694. doi:https://doi.org/10.1016/j.mayocp.2018.10.023

26.

Busis

Shanafelt

Keran

, et al. Burnout, career satisfaction, and well-being among US neurologists in 2016. Neurology. 2017;88(8):797-808. doi:https://doi.org/10.1212/WNL.0000000000003640

27.

Burningham

Lagha

Duford-Hutchinson

, et al. Developing the VA geriatric scholars programs’ clinical dashboards using the PDSA framework for quality improvement. Appl Clin Inform. 2022;13(4):961-70. doi:https://doi.org/10.1055/s-0042-1757553

28.

Knudsen

Laursen

HVB

Johnsen

Bartels

Ehlers

Mainz

. Can quality improvement improve the quality of care? A systematic review of reported effects and methodological rigor in plan-do-study-act projects. BMC Health Serv Res. 2019;19:683. doi:https://doi.org/10.1186/s12913-019-4482-6.

29.

Jaspers

MWM

Steen

Bos

C van den

Geenen

The think aloud method: a guide to user interface design. Int J Med Inf. 2004;73(11):781-795. doi:https://doi.org/10.1016/j.ijmedinf.2004.08.003

30.

Turner

C.W.

Lewis

J.R.

Nielsen

Determining usability test sample size. In: International Encyclopedia of Ergonomics and Human Factors. 3. 2nd ed. CRC Press; 2006.

31.

Creswell John

David

. Research design: Qualitative, quantitative, and mixed methods approaches. 5th ed. Sage; 2017.

32.

Hamilton

Finley

. Qualitative methods in implementation research: an introduction. Psychiatry Res. 2019;280(10):112516. doi:https://doi.org/10.1016/j.psychres.2019.112516

33.

Brown-Johnson

Safaeinili

Zionts

, et al. The Stanford lightning report method: a comparison of rapid qualitative synthesis results across four implementation evaluations. Learn Health Syst. 2020;4(2):e10210. doi:https://doi.org/10.1002/lrh2.10210

34.

Brown-Johnson

Chan

Winget

, et al. Primary Care 2.0: design of a transformational team-based practice model to meet the quadruple aim. Am J Med Qual. 2019;34(4):339-347. doi:https://doi.org/10.1177/1062860618802365

35.

Khanbhai

Anyadi

Symons

Flott

Darzi

Mayer

. Applying natural language processing and machine learning techniques to patient experience feedback: a systematic review. BMJ Health Care Inform. 2021;28(1):e100262. doi:https://doi.org/10.1136/bmjhci-2020-100262

36.

Patel

Pierce

Jones

, et al. Using participatory design to engage physicians in the development of a provider-level performance dashboard and feedback system. Jt Comm J Qual Patient Saf. 2022;48(3):165-172. doi:https://doi.org/10.1016/j.jcjq.2021.10.003

37.

Shuldiner

Kiran

Agarwal

, et al. Developing an audit and feedback dashboard for family physicians: user-centered design process. JMIR Hum Factors. 2023;10(1):e47718. doi:https://doi.org/10.2196/47718

38.

Van Buchem

Neve

Kant

IMJ

Steyerberg

Boosman

Hensen

. Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM). BMC Med Inform Decis Mak. 2022;22(1):183. doi:https://doi.org/10.1186/s12911-022-01923-5

39.

Huppertz

Otto

. Predicting HCAHPS scores from hospitals’ social media pages: a sentiment analysis. Health Care Manage Rev. 2018;43(4):359-367. doi:https://doi.org/10.1097/HMR.0000000000000154

40.

Kiyasseh

Laca

Haque

, et al. A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons. Commun Med. 2023;3(1):42. doi:https://doi.org/10.1038/s43856-023-00263-3

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB