Implementing online patient feedback in a ‘special measures’ acute hospital: A case study using Normalisation Process Theory

Abstract

Objective

Online patient feedback is becoming increasingly prevalent on an international scale. However, limited research has explored how healthcare organisations implement such feedback. This research sought to explore how an acute hospital, recently placed into ‘special measures’ by a regulatory body implemented online feedback to support its improvement journey.

Methods

Semi-structured interviews were conducted with eleven key stakeholders involved in the implementation and/or use of online patient feedback. Data was analysed using deductive thematic analysis with Normalisation Process Theory used as the analytical framework. Research findings are translated into the Engage, Support and Promote (ESP) model, a model of rapid feedback adoption.

Results

Participants viewed the implementation of online feedback as an opportunity to learn, change and improve. Factors found to facilitate implementation were often linked to engagement, support and promotion. Although less frequently described, barriers to implementation included staff anxieties about time pressures, moderation processes and responding responsibilities. Such anxieties were often addressed by activities including the provision of evidence based responder training. Overall, staff were overwhelmingly positive about the value of online feedback with 24 impacts identified at an individual and organisational level, including the ability to boost staff morale, resilience and pride.

Conclusions

The rapid implementation of online patient feedback can be achieved in a ‘special measures’ organisation. However, the difficulties of implementing such feedback should not be underestimated. In order to embed online feedback, staff members need to be engaged and feel supported, with opportunities to provide, respond and invite patient feedback frequently promoted to both patients and staff members.

Keywords

Patient feedback online patient feedback qualitative studies normalisation process theory patient experience care opinion

Introduction

Patient feedback is considered integral to patient safety and quality improvement.^1–4 On an international scale, patients are increasingly sharing their healthcare experiences online using publicly available websites such as Care Opinion, IWantGreatCare and NHS.UK (formerly NHS Choices).^4–7 However, despite their increasing use and frequent association with patient-centred care, quality improvement and patient autonomy,^1,2,8,9 limited research has explored the motivation, attitudes, potential impacts and perceived barriers or enablers to embedding online feedback in a healthcare organisation.⁴

As recently suggested by Powell et al.,¹⁰ the implementation of online feedback has been slower to materialise in health and social care settings in comparison to other industries such as business, hospitality and travel. While the functionality of existing feedback platforms used in health and care settings share many similarities with those used in business and hospitality such as TripAdvisor, the implementation of such platforms between these settings largely differs given their variance in focus and implementation context. For example, there are relational differences between a voluntary hospitality and mandated healthcare related experience; differences in commercial interests and motivations i.e. a paid for service vs a ‘non-repeatable,’ or free at the point of access service; resource access and availability and variation in the complexity, or number of organisations involved. Understanding the context in which online feedback is being implemented in is therefore vitally important.

The Point of Care Foundation recently suggested that online feedback continues to remain an untapped hub of patient experience,¹¹ with healthcare organisations and professionals rarely prepared for its use, or adoption.^4,10,12 Furthermore, Boylan et al.’s recent scoping review and consultation with stakeholders identified, six gaps in existing literature including how organisations use online patient feedback and staff attitudes towards such feedback, particularly nursing staff.¹ Pertinent to the context of this research, Boylan’s scoping review did not identify any studies that considered the processes involved in the use, or implementation of online feedback in healthcare organisations.¹ As a result, guidance on how “best to harness and use online patient feedback to improve care quality” ^(p.2)⁵ remains limited, identifying a further gap in existing knowledge and understanding. Finally, following the analysis of free text survey responses from 1001 primary and secondary care doctors in the UK, Turk et al. recently concluded that further efforts should focus on exploring the attitudes of health-care professionals towards online patient feedback given their significant influence on its implementation in health-care settings.⁴

This research therefore sought to address identified gaps in existing literature by exploring the implementation of online patient feedback in an acute hospital recently placed into ‘special measures’ by England’s healthcare regulator, the Care Quality Commission (CQC) (please see the section below that gives a further description of the study setting and implementation process). Using Normalization Process Theory (NPT)¹³ as a theoretical framework, this research sought to address the following research questions, how do staff:

Understand online patient feedback?

Engage with online patient feedback?

Work with online patient feedback for quality improvement purposes and what factors support or inhibit its use for such purposes?

Appraise the value of online patient feedback?

This research concludes by sharing practical guidance on how to rapidly adopt online patient feedback recognising the need for such information.^5,12 In the absence of a single agreeable term, the word ‘patient’ is used to be inclusive of service-users, customers, clients, consumers, carers and/or family members, although the important distinctions between these terms are acknowledged.^14–16

Case study setting and implementation description

This research was conducted in a secondary care acute NHS Trust that serves a population of around 550,000 people, with approximately 750 beds across three sites employing around 5000 staff. The Trust was rated as ‘inadequate’ by England’s healthcare regulator, the Care Quality Commission in October 2017 and put into ‘special measures’.

At the time of initiating the research (May 2019), the Trust had had a full subscription with the online feedback provider Care Opinion for just over a year providing an opportune moment to evaluate its use, impact and implementation.

Care Opinion works on the premise that (i) patients share their story, (ii) the story is sent to relevant staff members to facilitate learning, (iii) patients receive a response and (iv) the original patient story may lead to a beneficial change. On publication, staff members in subscribing organizations who have opted into alerts are made aware of the story. Other relevant organizations are also contacted by Care Opinion. A responder may indicate in their response that they have made a change as a result of the feedback received. This claim is made by the responder, not Care Opinion. A self‐reported change is then visually shown on the website. It is up to individual or organisational discretion who responds. There is no guarantee that patients will get a response. All stories and subsequent responses published on Care Opinion are publicly available, providing real‐time feedback with the intention of providing cost‐effective, measurable and transparent improvements.

Approach to implementation

Six months before the Trust was put into special measures, their new head of Patient and Family Experience had moved its two Care Opinion subscriptions (allocated to all NHS organisation across the UK) from the Trust’s communication team to the Patient and Family Experience Team. This put oversight of stories and responses into the hands of colleagues who were experienced in listening and responding to feedback about care received in the Trust.

As the Trust went into ‘special measures’, its focus on improvement was prioritised. A business case for a full subscription to Care Opinion was proposed and supported. The Chief Nurse and new head of Patient and Family Experience championed the platform as it was strongly believed that the platform would be an enabling factor in the Trust’s improvement journey: not only enabling the Trust to listen and learn from care experiences and stories, but also galvanise the workforce’s engagement in a program of work focused on improving its safety culture.

On subscribing, the executive team of the Trust became listeners to Care Opinion stories, as the Patient and Family Experience Team led the rollout. The rollout focused on teams and departments already receiving high levels of Care Opinion feedback e.g. the Emergency Department and paediatrics. Later, more targeted team engagement was adopted to respond to growing numbers of stories as the platform was promoted to all patients.

Methods

Design

This research used a qualitative case study design. Case studies have been identified as a valuable and sufficient method for studying complex, under-researched phenomena and evaluating programmes or interventions in-depth.^17,18 Given the limited amount of existing research into this area, a single case study design can be justified. Furthermore, this research was informed by Normalisation Process Theory (NPT) and its focus on individual and collective action, i.e. who, when and what people do.¹³ NPT is a theoretical model comprised of four components: i) coherence, or sense making work, how do people make sense of the intervention?; ii) cognitive participation, the relational work that people do to support a new intervention, how do people engage with the intervention?; iii) collective action, the operational work, what do people do to make an intervention work? and finally, iv) reflexive monitoring, the appraisal work, how do people perceive the intervention once it has been in place for a while? NPT is often used to evaluate systems or interventions that are particularly complex in nature,¹⁹ such as online patent feedback.

Recruitment and sampling

Individuals involved in the implementation and/or use of online patient feedback across a variety of settings within the Trust were purposefully recruited on a voluntary basis. This was achieved through the sharing of an advertisement poster by a hospital staff member to a list of individuals known to have been involved (whether positively or critically) in the implementation and/or use of online patient feedback. The sample list intentionally reflected a diversity of experience, perspectives and opinions from frontline clinical, management and administration staff to ensure a variety of different roles and experiences related to the implementation process were explored. While dissemination by a staff member may encourage those with a similar viewpoint to take part, the advertisement was distributed to all potentially relevant participants (i.e. those who had been involved in the implementation and/or use of online feedback) providing an equal opportunity for involvement. The researcher followed up expressions of interest to coordinate the semi-structured interviews as described below. Recruitment was conducted over six weeks to maximise engagement opportunities. Staff members at Care Opinion directly involved in facilitating the implementation and/or subscription process at the organisation were also invited to take part to explore a previously under researched perspective, that of the feedback platform provider. Participants must have been involved in the implementation and/or use of online patient feedback in order to be involved due to the previously defined scope of this research.

Semi-structured interviews

Informed by the four domains of NPT,¹³ topic guides and relevant prompts for the semi-structured interviews were piloted and developed in collaboration with a member of the Patient and Family Experience Team and Chief Executive of Care Opinion during a short planning meeting. This was to ensure sufficient coverage and sensitivity of the topic area given their relevant knowledge and expertise in providing an online patient feedback platform and desire to learn about staff experiences, whether positive or critical regarding its implementation. For clarity, the Patient and Family Experience Team was not informed about which members of staff participated in the interviews or not. During the interview schedule creation, the staff members involved did not ask for any suggested questions to be removed. The resulting interview schedule explored stakeholder understanding, motivations, experiences, perceptions and evaluations of online feedback.

The researcher (RB) conducted interviews by phone (May-July 2019) to minimise any potential disruption to existing work schedules and commitments. All interviews were audio recorded, conducted with written informed consent, anonymised and transcribed verbatim by the researcher. The researcher did not have any prior relationship to the participants. Potential participants were provided with brief information about the interviewer including her position at the University, reasons and interest in doing the research. No repeat interviews were conducted or field notes taken during the interviews.

Analysis and synthesis

Data was analysed using deductive thematic analysis as outlined by Braun and Clarke.²⁰ NPT was used as the analytical framework with each of the four components acting as individual core themes. Analytical rigour was ensured by the independent coding of each transcript and the scrutinising and discussion of coding amongst two of the listed authors (RB and FU). No coding disagreements were encountered. Analysis was conducted after each interview and a view on data saturation considered. Data saturation was defined for the purposes of this research as the point at which no new generic themes or variations of a given them emerged.^21,22 NVivo 11 software was used to facilitate data analysis.²³

The reporting of this qualitative study follows the consolidated criteria for reporting qualitative studies (COREQ) checklist.²⁴ A report of the findings was shared with participants prior to this publication.

Results

Eleven interviews were conducted with a variety of staff members including three Clinical managers, three members of the Patient and Family Experience Team, one Ward Nurse, one Ward Clerk, one Non-clinical staff member, and two Care Opinion staff members. Research findings are presented in order of the four NPT constructs previously outlined above. Table 1 below provides a summary of the NPT domains and related themes identified in this research.

Table 1.

NPT domains and associate themes identified in this research.

NPT domain	Related themes identified
Coherence – how do people understand online patient feedback?	Motivations and perceived purpose
	Differences between existing and online feedback methods
Cognitive participation – how do people engage with online feedback?	Transference of responsibility
	“Selling” online feedback
	Developing an active network across the Trust
	Reconceptualising patient feedback as an integral part of every job role
Collective action – how do staff work with online patient feedback for quality improvement purposes and what factors inhibit or facilitate its use?	Inhibitors
	Facilitators
Reflexive monitoring – how do people appraise online patient feedback?	Critical appraisals
	Positive appraisals

NPT: normalization process theory.

Coherence – How do people understand online patient feedback? What do they see as its perceived purpose?

Motivations and perceived purpose

Participants described a range of motivations for implementing online feedback. Those most commonly described included the opportunity to:

“Learn, change and improve” (Patient and Family Experience Team member, PFET hereafter, participant 1)

“To actually hear patient voices” from “source” (PFET, participant 8)

Streamline existing, often archaic methods of patient feedback -“we only wanted to really focus on one website to respond to all of our feedback and Care Opinion was a much more attractive website to use” (PFET, participant 1)

Adhere to organisational strategy and culture - “part of our Patient and Family Experience strategy in the organisation has been really shifting from retrospective opinion and old experiences, where change and improvement were limited to more real time feedback and Care Opinion was attractive to me and others because of that reason” (Clinical manager, participant 3)

Demonstrate change - “it was introduced to me as a platform in which patients and relatives and any other service-user can give anonymous feedback where they can receive a response and be made aware of any changes that have been made as a result” (Ward clerk, participant 4)

Improve quality - “we saw it as an opportunity to look at the services we provide and whether or not we could offer any improvements” (Non-clinical staff member, participant 9)

Celebrate and motivate - “if it’s a positive story, that’s just an opportunity to celebrate” (Clinical manager, participant 3)

One participant referred to online patient feedback, particularly constructive stories “as a gift” (PFET, participant 8), identifying a further understanding of online feedback.

Identified differences to existing feedback methods

Other ways of understanding online feedback by participants included acknowledged differences between online and other existing patient feedback methods. For example, some participants differentiated between “retrospective opinion and experiences” often provided by paper-based surveys, and more “real time feedback” provided by online methods. Some staff described off-line feedback as “more formal” (Non-clinical staff member, participant 9) in comparison to that provided by online methods. Concerns of anonymity appeared key to this distinction. For example: “I think Care Opinion is a slightly more informal way of commenting on services, you can be anonymous, whereas with the Patient Experience team, you’re actually sitting with someone and making a formal complaint” (Non-clinical staff member, participant 9).

Similarly:

If someone is really unhappy then of course we direct them to our complaints team and it would go through the formal process … I think the fact that it can be anonymous [online feedback] they don’t have to provide their details gives them a bit more reassurance than perhaps if they were raising something as a complaint. They [patients] quite often have the impression that it might jeopardise their future care. I think that thought stops people sometimes from giving their feedback. (PFET, participant 2)

The seemingly informal nature of providing and receiving patient feedback online was often favourably described. For example, “that’s the best bit about the job, anything that gives you feedback in a real, personal way is always much better than anything formal” (Clinical manager, participant 10).

One participant shared their hope of Care Opinion reducing “not the formal complaints, but our informal complaints, so more of our queries, concerns” (PFET, participant 2). This notion was reflected by other participants who suggested that “sometimes instead of making a complaint, it might be negative feedback, but people still want to share it. They [patients] just don’t want to make a complaint so, it’s a useful channel for that as well” (PFET, participant 8). However, while the informality of patient feedback was often favourably described, staff members repeatedly stressed the importance of taking online feedback as “seriously” as complaints or other, more formally recognised methods of patient feedback collection - “give it [online feedback] as much authority as your complaints, your Friends and Family Test, treat it exactly the same” (PFET, participant 2).

Cognitive participation, how do people engage with online feedback?

Participants described a number of decisions made to facilitate staff engagement with online feedback.

Transference of responsibility

The decision to move the responsibility of online feedback from the communications team to the Patient and Family Experience Team was repeatedly described as beneficial by participants. When asked about the rationale behind this decision, one staff member replied:

The rationale behind that was because the communications team have a huge remit that they look after … they were only having the opportunity to look at Care Opinion on an ad hoc basis, mainly when we were receiving negative feedback. It gave us the opportunity to look at the positive feedback as well. Because, that was being read, but not really being shared with anyone, and we thought that was really important (PFET, participant 2)

‘Selling’ online feedback

Other decisions made to facilitate staff engagement included the ‘selling’ of online patient feedback to colleagues and senior staff members. Methods identified by participants included:

Emphasising the meaning, value and importance of online feedback - “The way we sold it is, we went out to different ward Sister meetings for example and we showed them, we said you know this is the stuff that has been said about your team and wouldn’t it be great rather than me responding as a manager, that you as ward Sister respond? How much more meaningful is that going to be and how much more meaningful for your staff is that going to be if you get to share that with them rather than another email from the patient experience team?” (PFET, participant 1)

Highlighting the ability to share online feedback more freely - “We really pushed the fact that Care Opinion is in the public eye already so you can put posts up, you can share it far and wide and I think that kind of sold it” (PFET, participant 8)

Emphasising the ability to directly respond to online feedback and receive targeted alerts - “I think that was another selling point, they liked the idea that they could get alerts when someone said something about their area, they like the idea that they could go and search for stories about their area as well” (PFET, participant 1). Many participants drew on their own personal experiences of receiving a response to facilitate such engagement. For example, “as an individual, I kind of think if I was taking the time to do a TripAdvisor review and the manager or the person from that service has taken the time to respond to me I am reassured that my feedback has been read. I feel valued as a customer, and so that was my individual thinking of well if we can do that for all of our patients and relatives that's only going to be a good thing” (PFET, participant 1)

Many of the ‘selling points’ identified by participants are not possible with more archaic, often paper-based forms of feedback including the Friends and Family Test.

Developing an active network across the trust

In addition to ‘selling’ online feedback, developing an active network of people, or champions, across the organisation also appeared key in facilitating staff engagement. As one participant reported “staff engagement is key” (PFET, participant 1). In order to achieve this, members of the patient and family experience team “started going out to different care groups and the service areas promoting” (PFET, participant 1) online feedback and “talking about it to staff” (Clinical manager, participant 3). This was described as an intentional decision by the patient experience team with its importance repeatedly reiterated by participants.

Reconceptualising patient feedback as an integral part of every role

Other approaches to engagement included the reconsideration of patient feedback as an integral part of every job role, regardless of seniority or specialty. Many interviewees already viewed patient feedback as ‘part of my job’ (Non-clinical staff member, participant 9) or “every day work” (Clinical staff member, participant 11), although this may reflect the volunteer nature of this research.

How do staff work with online patient feedback for quality improvement purposes and what factors inhibit or facilitate its use?

For the third construct of NPT, collective action, staff described factors that both supported and inhibited the use of online feedback for quality improvement activities.

Inhibitors

Staff anxiety. Beginning with staff anxiety, participants described initial concerns around time pressures, responding responsibilities and external moderation. Such anxieties appeared to diminish once people had begun to engage with online feedback or had the opportunity to speak to other staff members:

When it came from us [patient experience team], it was a bit like, ‘oh, well you’re admin, you don’t really understand the clinical pressures’ and things like that. But now they’re sort of talking to each other and they’re realising that actually it doesn’t take a huge amount of time …We’re seeing more and more staff being happy to be involved (PFET, participant 8)

Time. The organisation also tried to alleviate time concerns by providing examples of people already working with online feedback in particularly busy departments. For example:

A barrier with our staff is that they think it’s going to take a long time to reply, and they’re going to be flooded with stories, but in reality, it’s not. So we always try to use examples of people that are using it really well in their area. We've got a clinical matron in ED, who feels like she's going to be the most busy person in the hospital, but she uses it all the time, she's constantly on their responding straight away (PFET, participant 8)

Responding. Other techniques used by the organisation to reduce staff anxiety included the provision of evidence-based responder training. As one participant noted, staff “were quite concerned about responding … That’s why we set up our responder training to give them some guidance and confidence” (PFET, participant 2). For clarity, the responder training referred to is supported by the PFET and consists of a simple workbook that uses the previously published co-produced patient feedback response framework.¹⁶ The workbook contains various reading materials and two activities including reading an online patient feedback case study example; reviewing its associated response with reference to the published patient feedback response framework; reviewing another patient feedback case study example and constructing a response using the framework which is then submitted and reviewed by the PFET, with individual feedback and coaching support provided. The training typically takes 60-90 minutes.

The provision of evidence-based training appeared key in facilitating staff confidence, instilling a sense of pride and achievement while helping to maintain quality and consistency in provided responses. For example: “We've actually provided response training … giving them some proper guidance, proper training, so they have the confidence and they're not going to be weary of it” (PFET, participant 2). “There was training we went through. I had a little certificate as well which was lovely. I also had a photo which they put on the hospital twitter site” (Clinical staff member, participant 11). “It was really good, I found that I was assured….” (Clinical manager, participant 5). “We don’t just give everybody the ability to respond, they go through a small training programme to be able to respond, so we've got a standard about compassionate responding… I think that's how we sort of keep a consistency.” (Clinical manager, participant 3).

The provision of responder training appeared to be a unique feature of this organisation as identified by one of the staff members at Care Opinion:

The other thing that not everybody does, is they've provided sort of explicit and identified dedicated training to get people to be a responder on Care Opinion, they ask people to receive the training, and when they've finished it, they give them a certificate so that they can become a responder, and I think again, that gives some people some confidence that they know what they're doing, and some sense of pride in their involvement…we use that as a suggestion for other people now. They kind of formalised that procedure and they almost added a bit of prestige to it (Care Opinion staff member, participant 6)

External moderation. The final area of concern described by participants related to feedback moderation from another external online feedback platform, NHS Choices. It is important to clarify that Care Opinion has the facility to pull through stories from NHS Choices but works on the assumption that these stories have been sufficiently moderated by NHS Choices. Two participants described an event where data had been pulled through from NHS Choices containing a staff member’s name:

We had a story recently where a staff member was named negatively and it had come through. It had past their [NHS Choices] moderation somehow with the staff name left in there and it has then been posted. Care Opinion were fantastic, as soon as we phoned them, they edited it, it was gone and obviously it wasn’t their fault because it was moderated by another platform, But I think it was just a bit of, we’ve worked so hard to build this reputation, ‘oh now what we are going to do …’ but things are in place to make sure it doesn’t happen again which has been really good for our staff (PFET, participant 2)

Such findings reiterate the importance of moderation processes and subsequent assurances.

Facilitators

Conversely, participants also described a total of 16 facilitators for working with online patient feedback. These are presented in order of frequency in Table 2. Facilitators regularly described included engaging as many relevant people as possible (“Get as many people onto it… the people who are front facing” Clinical manager, participant 10), providing responder training (“… proper guidance, proper training so they have the confidence and they’re not going to be weary of it” PFET, participant 2) and getting stakeholder buy in, particularly at a senior level (Get your Board on Board” PFET, participant 2). Notions of engagement, support and promotion underpinned the majority of facilitators described by participants as later discussed.

Table 2.

Facilitators described by participants ordered according to frequency.

Facilitator	Verbatim examples taken from transcripts
Engage as many (relevant) people as possible	“Get as many people onto it, give access to your specialist nurses, the people who are front facing, your ward sisters, your deputies on the ward, so they can see this feedback and reply to patients … because that's a big benefit and something that I think it's just really good practice” (Clinical manager, participant 10)
Provide responder training	“We've actually provided response training … giving them some proper guidance, proper training, so they have the confidence and they're not going to be weary of it” (PFET, participant 2)
Get stakeholder buy in, particularly at senior level	“Get your board on board, make sure you’ve got buy in from quite a high level because otherwise, everyone is so busy, it will just get overtaken by something else that is more of a priority for the Trust at the time” (PFET, participant 2)
Provide support
Patient experience team	“We always say, the first time, if you do receive a bit of feedback that is mixed or negative, and you’re not sure what to say, give one of our team a call and we will come and sit with you and do it with you, you feel like you’ve got a bit of a safety net if you like” (PFET, participant 2)
Staff support services	“I know that she's [patient experience team member] really supportive, and I know that I'm comfortable to ask her questions if I'm stuck. I think as long as you have the right training and the right support, it’s absolutely fine…I didn't have any concerns or worries about it … they’ve been absolutely brilliant, it's making sure you've got someone there to ask questions I think” (Clinical staff member, participant 11)
Staff support services	“Also putting things in place, like through occupational health, so if anything is causing staff anxiety or stress, they have people to speak to, so I think it’s really about us as a trust letting them know who they can go to if they are affected by anything that is mentioned online” (PFET, participant 2)
Showcase feedback
Social media	“We are tweeting most days, we have our [hastag’s] #TransformationTuesday, and #FeedbackFriday. It’s a really good way of being able to promote it and we do get quite a lot of comeback on that which is good…We share snippets of stories, we normally do it in a pictorial sort of way, so it’s a quick quote and then we invite staff to access the link to the full story. That’s really well received, especially when you’re tagging the appropriate people, they can then share it with their teams” (PFET, Participant 8)
Know how you’re doing boards	“In all of our wards and departments, we have ‘know how you're doing boards’, which tell people, staff and the public how this ward is doing, and it has their performance metrics in there. We put a couple for Care Opinion in there, and every ward has a space on their ‘know how you're doing’ board to display the latest bit of feedback. So, if they've had no feedback, there’s a slide in their for people to say, ‘tell us your story’” (Clinical manager, participant 3)
On wards and departments	“We’ve had feedback from people to say we’ve been able to put it up around the department and that’s across the disciplines so medical staff, therapy, nursing and its specific to their area” (Clinical manager, participant 10)
Staff rooms	“Quite often you’re going into the nurses office and they’ve got the quotes printed out and put on their wall, so they’re hearing what people are saying which is really, really lovely” (PFET, participant 8)
Advertise/use promotional materials
Promotional materials	“Get Care Opinion to send you all of the materials they can, they have posters, lovely business cards, lots of things that you can really put around the trust to really raise awareness for patients and staff, really use those materials” (PFET, participant 1)
Business cards	“We give all our staff the business cards” (PFET, participant 8)
Posters	“Putting posters up everywhere” (PFET, participant 1)
QR codes	“It has a QR code so people can scan that and it would take them…straight to the platform” (Clinical manager, participant 3)
Hospital website	“The hospital website now hosts the last five comments on Care Opinion” (Clinical manager, participant 3)
Waiting room electronic displays	“We’ve got the slides now throughout our trust in our waiting area” (PFET, participant 2)
Provide an offline option	“There was a story teller who spoke about his nan not being able to leave feedback and she couldn't quite grasp the concept of doing it online, and that really highlighted to us that we had nowhere for people to write [feedback for Care Opinion]. We [now] have, the free post cards where you can write and send them in, we've now got them in public areas, and invite that feedback, so that story has been really, really, really useful to us, and we've been able to make a really quick change. We get quite a few posted that way now” (PFET, participant 8)
Use league tables	“I think one of the things that has been important to us…is the league table about organisations adoption of Care Opinion and how we’re doing. We’ve enjoyed in this first year climbing that table of more listeners and responders. I think that’s been useful in the organisation as well, we’ve been able to say look, we are into the top 25, we’ve now broken through the top 15. It’s a good way of bringing people on and engaging people with the platform, because we do want to be actively listening to patients. It’s a positive way to be able to go to board and say, ‘we are an active listening organisation’, ‘We’re third in the country that have got the most people listening through this platform’” (Clinical manager, participant 3)
Share online feedback at staff inductions and pre-registration trainee meetings	“We now go to pre-registration trainee meetings, so people that are coming in to train as nurses and we’re sort of capturing at the point of entering the trust as well, to think that actually Care Opinion is there, and people can share their story that way” (PFET, participant 8); “It’s about sharing things like Care Opinion in induction” (PFET, participant 2)
Don’t be afraid to say sorry	“Don’t be afraid to say sorry about someone’s experiences, because I think a lot of us working I the NHS we feel if we say sorry, we’re admitting fault in some way and people get worried about that. I think it’s just having a bit of a shift on that … I’m saying sorry because of your experience, and that sorry can go a long way” (PFET, participant 8)
Target specific teams	“We’re picking teams off specifically to use it” (Clinical manager, participant 3)
Collaborate with external organisations	“We’re really pleased that we’re able to collaborate with our local HealthWatch. As an organisation they often respond to experience feedback and are happy to give it back to us. But, it was often quite old and historical experience and it made us really struggle to respond to it because it wasn't that ‘real-time’ … We worked really hard with them to make them understand that we want ‘real-time’ feedback … They're now out capturing experiences around the county” (Clinical manager, participant 3)
Be proactive	“It’s a positive, proactive strategy to involve staff, it’s been very active to involve as many people as possible” (Care Opinion staff member, participant 6)
Push for relevant responders	“We definitely want to make sure that we have staff responding from every area, so it’s actually so important when service-users leave their feedback, when the response comes from someone in the [patient experience] team. I think it means so much more to them” (PFET, participant 2)
Sign up staff to alerts	“They now get alerts directly so any stories that are posted where positive or negative they’ll receive an alert and that’s part of their training, they sign up to share that with their staff” (PFET, participant 2)
Identify staff benefits	“We say we’ll train you up. It will be really good for your development’, and ‘it’s something that you can take forward for your ward” (PFET, participant 1)
Cover areas without a responder	“Any services that we get feedback for, that we haven't got a responder for within the service, our team monitor it all the time, so we will copy and paste the story, copy and paste our response saying thank you very much and we're going to share it with the team, and we will email that to the relevant manager for that department and ask that they share it with the relevant staff” (PFET, participant 2)

PFET: patient and family experience team member.

Reflexive monitoring, how do people appraise online patient feedback?

Finally, we explored how staff appraised the value of online patient feedback (reflexive monitoring), with findings ordered according to critical and positive appraisals.

Critical appraisals

Two critical appraisals of online feedback were reported by participants. One participant referred to the potential difficulty of identifying relevant areas of concern in a feedback entry:

Sometimes you have to sift through something that somebody is expressing … sometimes it’s just a comment that can encompass a lot of services, they do it in a capture all. You have to be able to read through it and sometimes determine what that person was really unhappy about (Non-clinical staff member, participant 9)

The other critical appraisal described by participants related to the service provision of another feedback provider (NHS choices) and their moderation process as previously mentioned. These were the only critical appraisals provided by participants.

Positive appraisals

Participants were positive about the value of online feedback. Many participants referred to Care Opinion as “a powerful platform” that affected people on both an individual and organisational level. For example: “It gives you a sense of pride, you can recognise that actually the work that you are doing within your department is working” (Clinical manager, participant 5).

Just the personal experience, I’ve tried to read as many of the posts as I can, it was really powerful to do that … it’s something that makes me feel really proud that we have got a very capable workforce taking this [online feedback] … it’s powerful for me (Clinical manager, participant 3)

Staff resilience, morale and pride through celebration and motivation

One of the perceived benefits of online feedback most commonly described by participants included its ability to improve staff morale, resilience and pride through motivation and celebration. One participant described this benefit at length:

I really love it, I do, because for me working in the Emergency department, you see a lot of terrible things happen, you see a lot of sick patients, a lot of illness, and really poorly patients. The staff here have to, on a daily basis, see things that you wouldn't want anyone to see. So for me, it’s such a refreshing change because we predominantly get positive responses and feedback. It's just so nice to be able to cling on to that and share the feedback to the staff, those really nice, positive things. I think it helps improve morale. It helps people’s resilience as well, to know there are actually some really grateful people out there and you know what, you are doing a good job and sometimes you just need that bit of a reminder. And when I see comments like that, that helps me to deal with the everyday situation. Generally, it makes people feel appreciated in what is a really hard, difficult working environment, its morale boosting … for me personally, it's that little glimmer of hope that you hold onto amongst a multitude of horrible things … (Clinical manager, participant 5)

Furthermore, in addition to quality improvement opportunities that may have otherwise remained unknown, online feedback was also seen as a way to “address the balance” by incorporating online feedback into forms of celebration including Learning from Excellence and employee of the month awards, recognising that individuals and healthcare organisations are typically “very quick to say you haven't done this or you don't do that, but we're not very quick to say guess what, you did a really great job yesterday” (Non-clinical staff member, participant 9).

All positive appraisals (n = 24) of online patient feedback described by participants are listed in Table 3 in order of frequency.

Table 3.

Positive appraisals of online feedback by participants, ordered according to frequency.

Positive appraisal	Verbatim examples taken from transcripts
Ability to publicly demonstrate change	“I think the ‘changes made’ part of Care Opinion, where you can highlight when a change has been made, it's really nice and visual. I think before, patients, never got to see that, they don't really know what change their feedback has had. Whereas that is really visual and clear, you know as a direct result of your feedback, we've made a really good change here.” (PFET, participant 2)
Accessibility and ease of use	“Whereas this time maybe two years ago, feedback was kind of getting lost, and staff weren't hearing it. Feedback is really out there and accessible to all our staff and Care Opinion has been a really pinnacle point in us being able to do that … it’s very easy to use” (PFET, participant 8) “It’s very accessible for patients and for us … we find it really simple to use as a site, we like the way it works and the way it is, it’s really accessible” (PFET, participant 2)
Boosting staff morale, resilience and pride	“We use it for Learning from Excellence … anytime a member of staff is named personally and thanked, we feed that through the Learning From Excellence teams, and the Learning From Excellence team contact that member of staff personally with a card to say, this is what a patient said about you on Care Opinion, and that’s been really positive” (PFET, participant 1) “Sometimes we get feedback that picks out individuals or also a service. So it's really nice to pick up that because we also run what we call the [named award], and that's where we nominate individuals each month for going above and beyond, or doing a really great job, we have them as employee of the month. Its evidence, if you like, to support individuals who are doing a really great, great job, and if a service is doing really, really well. I copy and paste out those comments, ‘just to let you know such and such on this ward, doing a great job, let them know about it,’ because we don't do that very often, we're very quick to say you haven't or you don't, but we're not very quick to say guess what, you did a really great job yesterday.” (Non-clinical staff member, participant 9) “I was really proud to see how our staff responded compassionately to people that had told their story … and I followed up with emails thanking them” (Clinical manager, participant 3)
Care Opinion team support and ethos	“Just generally, their support, they’re so good the team at Care Opinion, when we have any queries or questions, we email them, we phone them, they’re so quick at responding and if they can’t answer you there and then, they’ll get back to you, and that’s fab. And obviously they do their webinars and things are really helpful as well” (PFET, participant 2) “I think when we were shown it, the ethos of the organisation came from the right place which is why it feels like the right thing to do …. It’s not ticking a box, it’s very patient focused” (Clinical manager, participant 10)
Receiving and sharing positive feedback	“I think the nice thing for me, is you get more compliments than issues. Because I deal with quite a lot of complaints in the department, it makes a real nice change. It gives you a sense of pride, you know in that you can recognise, actually the work that you are doing within your department is working” (Clinical manager, participant 5)
Getting real time feedback and sharing feedback in near real time	“Real time listening … Real time feedback, you are able to influence staff, change minds and behaviours quite quickly because things are more real” (Clinical manager, participant 3)
Public facing	“It’s all open and in the public” (Clinical manager, participant 3)
Opportunity to provide further information	“We’re able to offer additional information when somebody has pointed out something” (Non-clinical staff member, participant 9)
Reporting and evidencing	“We always use the visualisation reports, I think its brilliant … we get asked for information a lot … it's made a really big difference to how I can report feedback and how I can report changes that are made” (PFET, participant 1)
Ability to be a compassionate organisation	“It's just a way to really illustrate up and down, ward to board, how we are committing to putting patients first. Listen to their story. Responding compassionately to them. I think for me, it's a real outward illustration of how a compassionate organisation is present in the community by demonstrating that there's a compassionate response, they're not just words, there are some real staff” (Clinical staff member, participant 3)
Prompts dialogue	“It prompts dialogue … it does prompt that ‘What can we do’ kind of conversation” (Non-clinical staff member, participant 9)
Confidence and reassurance of moderation	“I'm really assured of the moderation that they do, and the passion of the staff that work with them, and the seriousness of which they take every story” (PFET, participant 1)
Preference for stories (qualitative data) over ratings	“It focused more on the story than the four-star rating [as provided by NHS choices] It’s really valuable … it’s a lot more qualitative than Friends and Family Test. Care Opinion very much says ‘tell us your story” (PFET, participant 1)
Ability to influence staff attitudes and behaviours	“You are able to influence staff, and change minds and behaviours quite quickly” (Clinical manager, participant 3)
Provision of anonymity	“So for us it's very much, we see it as a really valuable platform to receive feedback, especially reaching patients that wouldn't normally provide us with their feedback because it's anonymous. I think the fact that it can be anonymous and they don't have to provide their details, it gives them a bit of reassurance that perhaps if they were raising something as a complaint. They quite often have the impression that it might jeopardise their future care, obviously it never would, but I think just that thought stops people sometimes from giving their feedback, so the face it can be anonymous, I think it's definitely a huge one for us” (PFET, participant 2)
Diversity of feedback provider	“They don't necessarily have to be the patient, they can be a service user, a visitor, it can be anybody” (PFET, participant 2)
Ability to directly respond	“Also, it was nice that you could reply directly to somebody” (Clinical manager, participant 10)
Alternative to submitting a complaint	“There's a lot of people who want to praise, which is brilliant, there's also a lot of people who don't want to make a complaint, but want to know that someone has learnt from what their experience was. I think that's a really good way to be able to do that, anonymously as well, if they want to” (PFET, participant 1)
Honesty and openness of feedback	“Care Opinion invites you to sort of be honest, and say what your experience was” (PFET, participant 1) “It's nice to hear a patient perspective that isn't necessarily forced” (PFET, participant 8)
Provision of rich information	“It offers very rich information” (PFET, participant 8)
Visibility	“It gives visibility because every executive board member is an active listener. They get their weekly digest, and they understand what the top stories are, whether good or bad, they’re able to look at those. It’s a good way for teams to give visibility of their services to the board too” (Clinical manager, participant 3)
Less structured than other forms of patient experience	“We see it as a really valuable platform … our Friends and Family Test is anonymous, and it's very structured. I think people feel like they can only talk about certain things. Whereas [Care Opinion], they can literally provide feedback on absolutely anything they want” (PFET, participant 2)
Actively listening	“We're starting to more actively listen, and actively engage with people” (Clinical manager, participant 3)
Receive feedback from patients that might otherwise not give it	“It’s a really valuable platform to receive feedback, especially reaching patients that wouldn't normally provide us with their feedback” (PFET, participant 2)

PFET: patient and family experience team member.

Finally, to facilitate the translation of research findings, The Engage, Support and Promote Model (ESP Model) (Figure 1) was created by the authors to facilitate the rapid adoption of online feedback based on the research findings presented.

Discussion

This study addressed identified gaps in existing literature by exploring staff perceptions of how an acute hospital, recently placed into ‘special measures’ embedded online feedback to support its improvement journey.^1,5,12 Using NPT as a theoretical framework, this research provides unique insight into the motivations, understanding and methods used to implement online feedback, culminating in the creation of practical guidance and support (ESP model).

Participants often understood the implementation of online feedback as an opportunity to learn, change and improve, echoing the shift in organisational strategy and culture. The ethos of never wasting a story to generate good is recognised in existing literature, as is the power of storytelling in organisational culture change.²⁵ A difference in perceived ‘formality’ between online and ‘off-line’ feedback was often reported by participants, with the provision of enhanced anonymity considered key to this distinction. In contrast to existing literature including that recently provided by Turk et al. who concluded that participating doctors viewed anonymity as a negative feature of online patient feedback due to its inability to verify feedback content and suggestion that such anonymity encouraged negative or malicious comments,⁴ participants in this research appeared to value the provision of anonymity and the potential increase in feedback responses and authenticity as a result.^11,26,27 This disparity in findings may reflect the voluntary nature of this research and level of experience amongst some participants as recent research conducted by Locock et al., suggests individuals who have been responding to online feedback for longer may be more confident in such practice.²⁷ However, the provision of responder training may also have alleviated anonymity concerns and provided necessary support about how best to work with such information. Further research into this area would be beneficial. While the informality of online feedback was often favourably described, participants repeatedly stressed the importance of taking online feedback as “seriously” as other, more formally recognised methods of patient feedback.

Figure 1.

The Engage, Support and Promote Model for rapid adoption of online feedback platforms.

Factors found to facilitate implementation were often linked to engagement, support and promotion. Actions taken by the organisation to facilitate feedback implementation included moving its responsibility from communications, to the Patient and Family Experience Team; ‘selling’ online feedback to staff members, particularly senior staff members by emphasising its ability to facilitate patient safety and quality of care; developing an active network of individuals across the organisation to facilitate wider staff engagement and reconceptualising patient feedback as an integral part of every job role, irrespective of speciality or seniority.

Factors identified as inhibitory included staff anxieties, external moderation processes and responding capabilities. Such concerns were typically reduced through the provision of sufficient support including evidence-based responder training,¹⁶ clear explanations of the moderation process and demonstrations of busy healthcare professionals already working with online patient feedback. The importance of supporting and empowering staff members during the implementation and continued use of online feedback appears integral to implementation processes and sustainability.^27–30

With the exception of two critical appraisals, participants were overwhelmingly positive about the perceived value of online patient feedback, specifically Care Opinion. This was often attributed to its ability to publicly demonstrate change, its ease of use, and ability to enhance staff morale, resilience and pride through the sharing of positive feedback in near real-time. The ability to celebrate, motivate and enhance staff resilience through online feedback appears to be an underreported benefit in existing literature. Further exploration of this potential impact would be beneficial, particularly given the increasing workload pressures and changes in healthcare delivery healthcare professionals are currently facing in response to Covid-19.

Finally, participants often saw the public nature of online feedback as a strength, enabling the hospital to publicly respond, demonstrate change and create compassionate responses. The implementation of online patient feedback came at a time when reputational management was key for the hospital. Interestingly, in comparison to existing literature,^4,31–33 discussions around the necessity of patient feedback ‘representation’ or ‘validity’ were not described by participants. Alternatively, patient feedback was repeatedly conceptualised as a ‘gift’, whether critical, or positive in nature with each feedback entry providing a potential opportunity to learn, change and develop. This difference in conceptualisation raises important questions of why we collect patient feedback in the way that we do, how we define ‘valid’ feedback, and attribute ‘value’ to the multitude of feedback tools currently available and for whom. As recently suggested by Sheard et al., the collection of patient feedback is at risk of becoming it’s “own self-perpetuating industry” ^(p.46),³⁰ with opportunities to achieve the desired change reported in this research often limited. Ensuring methods of patient feedback collection are accessible and valuable to all stakeholders is essential, as is critical consideration of the competing pressures placed on healthcare professionals and organisations pertaining to national and local policy initiatives. As reported in other areas of healthcare,³⁴ the multitude of often mandatory patient feedback surveys and related targets may be inhibiting the implementation of more acceptable methods, although further research into this area is required.

Implications for practice & policy

Mirroring the ESP model, the implications of this research are as follows. Firstly, the importance of online feedback in facilitating patient safety and quality improvement should not be underestimated. Patients, are increasingly sharing their healthcare experiences online, perhaps even more so given the global pandemic and migration towards digital care services. Secondly, the purpose, value and motivations for both collecting and using specific patient feedback tools should be critically considered and regularly reviewed. Critical consideration of how feedback is defined is urgently required. Thirdly, staff members must feel supported and empowered in receiving, responding to and inviting patient feedback.²⁷ Reconceptualising patient feedback as an integral component of all job roles irrespective of specialty or seniority appears to be essential, as does the provision of relevant training.^27,30 Furthermore, innovative ways to promote online patient feedback should be explored to continually facilitate feedback awareness and engagement. Finally, the ability of online feedback to act as a freely available resource to celebrate and motivate staff members should be widely encouraged and promoted.

Strengths and limitations

Strengths of this research include its use of a theoretical framework (NPT)¹³ to explore how an acute hospital recently assessed as inadequate by the CQC implemented online patient feedback. The study provides important insight into how people understand online patient feedback, the relational work undertaken to facilitate feedback engagement, factors that support and inhibit its implementation and staff appraisals of its value. However, the limitations of this study must also be acknowledged. This research draws on a purposive volunteer sample in one healthcare organisations. Staff members who chose not to participate may have differing views that could be helpful to explore in future research. Research findings may not therefore be generalizable to other settings. Comparisons between individuals with varying levels of online patient feedback experience across a variety of staff roles including clinical, non-clinical and IT or communications may also be beneficial. This research also explored patient feedback in a secondary care setting only. Further exploration of online feedback implementation in other settings including primary care may be beneficial. Finally, this research looked at staff perspectives only; exploration of patient perceptions is also required.

Future research

Implications for future research include the: examination of how, if at all, the provision of responder training affects staff engagement and confidence given emerging evidence on the importance, yet variability of responding quality;^16,35 exploration of how, if at all, online feedback may facilitate staff morale and resilience in comparison to existing methods; exploration of how organisational cultures both within and between healthcare services may support or inhibit patient feedback use and acceptance;¹⁰ further testing and exploration of the ESP model including a comparison of implementation processes in other healthcare settings including those not recently placed into special measures or those with a longer history of online feedback and exploration of whether the rapid digitisation of healthcare services and removal of more informal measures of patient experience such as gift giving in light of Covid-19 has affected the uptake and content of online feedback from both a patient and professional perspective. Future research may also build on findings from this case study by examining the processes involved in the longer term embedding of online feedback and further consider how the ‘real time’, or temporal aspect of online patient feedback systems may differentiate from other forms of feedback and the implications this may have for their adoption, use and embedding.

Conclusion

Implementing online feedback in a ‘special measures’ organisation is possible. However, while the organisation achieved rapid progress, the difficulties of implementing online patient feedback should not be underestimated. In order to embed online feedback, staff members need to be supported and empowered, with opportunities to provide, respond and invite patient feedback opportunities frequently promoted. As described by one participant, online patient feedback can “make people feel appreciated in what is a really hard, difficult working environment, its morale boosting … it's that little glimmer of hope that you hold onto amongst a multitude of horrible things …” (Clinical manager, participant 5). Research findings and its accompanying model could be used to help inform local practice and meaningful implementation of online feedback more broadly.

Footnotes

Acknowledgements

We would like to thank all participants for taking the time to be involved and share their experiences; patients, carers and family members for kindly sharing their experiences of care and Care Opinion staff members for their help and assistance with this research.

Contributorship

RB and FU researched the literature and conceived the study. RB was involved in protocol development, gaining ethical approval, recruitment and data analysis. RB wrote the first draft of the manuscript. RB and FU developed the ESP Model. All authors reviewed and edited the manuscript and approved the final version.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: FU is a National Institute for Health Research (NIHR) Senior Nurse and Midwife Research Leader. The views expressed in this article are those of the authors and not necessarily those of the NIHR, or the Department of Health and Social Care.

Ethical approval

The ethics committee of Faculty Research Ethics and Integrity Committee at the University of Plymouth approved this study (ref./19-1055).

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Guarantor

RB.

Peer review

Lauren Ramsey, University of Leeds, Bradford Institute for Health Research and Mike Bracher, University of Southampton have reviewed this manuscript.

ORCID iDs

Rebecca Baines

RayB Jones

References

Boylan

A-M

Williams

Powell

Online patient feedback: a scoping review and stakeholder consultation to guide health policy. J Health Serv Res Policy 2020; 25: 122–129.

Sheard

Marsh

O'Hara

, et al. The patient feedback response framework – understanding why UK hospital staff find it difficult to make improvements based on patient feedback: a qualitative study. Soc Sci Med 2017; 178: 19–27.

Griffiths

Leaver

MP.

Wisdom of patients: predicting the quality of care using aggregated patient feedback. BMJ Qual Saf 2018; 27: 110–118.

Turk

Fleming

Powell

, et al. Exploring UK doctors’ attitudes towards online patient feedback: thematic analysis of survey data. Digit Health 2020; 6: 2055207620908148.

Dudhwala

Boylan

A-M

Williams

, et al. What counts as online patient feedback, and for whom?. London: SAGE Publications, 2017.

Emmert

Meier

Heider

, et al. What do patients say about their physicians? An analysis of 3000 narrative comments posted on a German physician rating website. Health Policy 2014; 118: 66–73.

Atkinson

Current status of online rating of Australian doctors. Aust J Prim Health 2014; 20: 222–223.

Bardach

Asteria-Peñaloza

Boscardin

, et al. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Qual Saf 2013; 22: 194.

Gondek

Edbrooke-Childs

Fink

, et al. Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: a systematic review. Admin Policy Ment Health 2016; 43: 325–343.

10.

Powell

Atherton

Williams

, et al. Using online patient feedback to improve NHS services: the INQUIRE multimethod study. Health Serv Deliv Res 2019; 7: 1–150.

11.

Point of Care Foundation. Online feedback, 2019, https://www.pointofcarefoundation.org.uk/resource/using-patient-experience-for-improvement/gathering-data/online-feedback/ (accessed 15 September 2019).

12.

Atherton

Fleming

Williams

, et al. Online patient feedback: a cross-sectional survey of the attitudes and experiences of United Kingdom health care professionals. J Health Serv Res Policy 2019; 24: 235–244.

13.

Murray

Treweek

Pope

, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med 2010; 8: 63.

14.

Baines

Regan de Bere

Optimizing patient and public involvement (PPI): identifying its “essential” and “desirable” principles using a systematic review and modified Delphi methodology. Health Expect 2018; 21: 327–335.

15.

Tritter

JQ.

Revolution or evolution: the challenges of conceptualizing patient and public involvement in a consumerist world. Health Expect 2009; 12: 275–287.

16.

Baines

Donovan

Regan de Bere

, et al. Responding effectively to adult mental health patient feedback in an online environment: a coproduced framework. Health Expect 2018; 21: 887–898.

17.

Baxter

Jack

Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep 2008; 13: 544–559.

18.

Flyvbjerg

Five misunderstandings about case-study research. Qual Inq 2006; 12: 219–245.

19.

Carter

Davey

Wright

, et al. Capturing patient experience: a qualitative study of implementing real-time feedback in primary care. Br J Gen Pract 2016; 66: e786–e793.

20.

Braun

Clarke

Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

21.

Eliacin

Salyers

Kukla

, et al. Factors influencing patients’ preferences and perceived involvement in shared decision-making in mental health care. J Ment Health 2015; 24: 24–28.

22.

Staniszewska

Boardman

Gunn

, et al. The Warwick patient experiences framework: patient-based evidence in clinical guidelines. Int J Qual Health Care 2014; 26: 151–157.

23.

NVivo Qualitative Data Analysis Software. Doncaster. QSR International Pty Ltd. Version 12. Available at: https://www.qsrinternational.com (accessed 04 April 2020).

24.

Tong

Sainsbury

Craig

Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357.

25.

Reissner

Pagan

Storytelling in management practice: dynamics and implications. Abingdon-on-Thames: Routledge, 2013.

26.

Patel

Cain

Neailey

, et al. Exploring patients’ views toward giving web-based feedback and ratings to general practitioners in England: a qualitative descriptive study. J Med Internet Res 2016; 18: e217.

27.

Locock

Skea

Alexander

, et al. Anonymity, veracity and power in online patient feedback: a quantitative and qualitative analysis of staff responses to patient comments on the ‘care opinion’ platform in Scotland. Digit Health 2020; 6: 1–13.

28.

Locock

Montgomery

Parkin

, et al. How do frontline staff use patient experience data for service improvement? Findings from an ethnographic case study evaluation. J Health Serv Res Policy 2020; 25: 151–161.

29.

Locock

Graham

King

, et al. Understanding how front-line staff use patient experience data for service improvement: an exploratory case study evaluation. Health Services and Delivery Research 2020; 8: 13.

30.

Sheard

Peacock

Marsh

, et al. What's the problem with patient experience feedback? A macro and micro understanding, based on findings from a three-site UK qualitative study. Health Expect 2019; 22: 46–53.

31.

Campbell

Narayanan

Burford

, et al. Validation of a multi-source feedback tool for use in general practice. Educ Prim Care 2010; 21: 165–179.

32.

Asprey

Campbell

Newbould

, et al. Challenges to the credibility of patient feedback in primary healthcare settings: a qualitative study. Br J Gen Pract 2013; 63: e200–e208.

33.

Donnon

Ansari

Alawi

, et al. The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review. Acad Med 2014; 89: 511–516.

34.

Clarke

Godfrey

Hawkins

, et al. Implementing a training intervention to support caregivers after stroke: a process evaluation examining the initiation and embedding of programme change. Implement Sci 2013; 8: 96.

35.

Ramsey

Sheard

Lawton

, et al. How do healthcare staff respond to patient experience feedback online? A typology of responses published on care opinion. Patient Experience J 2019; 6: 42–50.