Abstract
Aims and Method:
To discuss the methods involved in gathering real-time feedback (RTF) by a London Home Treatment Team. We hypothesized that RTF would lead to changes in service provision and improvements in patient and carer feedback. Patients were invited to provide RTF on discharge. Quantitative and qualitative data were collected and analyzed centrally by the Trust before being disseminated to the team and changes made based upon the results. Quantitative feedback on the team’s performance in the first 6 months of RTF use was compared against data from the 6 months prior to March 2015 using 2-tailed Z tests.
Results:
There were significant improvements in feedback around the team visiting at the agreed times (P = .0069) and patients feeling that they had been involved in treatment decisions (P = .0371).
Clinical Implications:
Real-time feedback is a potentially valuable method for obtaining patient feedback and can result in service improvements if used appropriately.
Keywords
Introduction
Over the past several years, there has been increasing emphasis on obtaining the views of service users to help monitor and improve mental health services. National Institute of Clinical Excellence (NICE) released a quality standard (QS14) (1) including a statement on this specific topic (Quality Statement 5) advising that service users should feel confident that mental health services are monitoring and acting upon their views of the care they have received. There is also an increasing focus on mental health services engaging carers more effectively and involving them in care planning (2).
In 2013, real-time feedback (RTF) was introduced to the South West London and St George’s Mental Health NHS Trust as a means of gathering data from patients and carers about their experience of the care they had received. This enabled feedback data to be obtained contemporaneously during an episode of care or at the point of discharge or transfer to another service. Initially, feedback could be left online but now different feedback methods, including tablets and free-standing consoles have been used. Initially, RTF was piloted on inpatient psychiatric wards in the Trust. The next phase was to implement RTF in community settings across the 5 Trust Home Treatment Teams (HTTs), seeing them as akin to “virtual wards” in the community, supporting people in crisis who otherwise would likely require inpatient care. This paper examines the experience of the Merton HTT in South West London of implementing and receiving RTF and the changes in scores obtained over time with impacts on service delivery.
Aims and Hypothesis
We hypothesized that:
implementing an RTF system would be feasible in a HTT setting with patients in crisis; obtaining RTF would enable the Merton HTT to respond to service user and carer feedback, leading to improvements in service delivery and quality; and this would be demonstrated by obtaining further patient and carer feedback over time, as part of a “virtuous circle.”
Methods
A process was devised of implementing RTF into the Trust HTTs. At the last planned visit from the HTT, RTF would be sought from all patients and carers if available. This would be obtained using a handheld electronic “smart tablet” device, and 1 tablet per HTT was deployed. All patients and carers were offered the option of completing the same questionnaire online. These RTF devices were preloaded with an RTF questionnaire, adapted from the national patient survey (3). This questionnaire is from a national standard, although slightly amended into a patient and carer version for the use in a HTT setting, as shown in Table 1. It is similar to the measure used by Hubbeling and Bertram (4).
Patient and Carer Questionnaire Items and Percentage of Positive Responses in the First and Last 6 Months of RTF Use.
Abbreviations: HTT, Home Treatment Team; RTF, real-time feedback.
aSignificant result (P < .05).
Setting
Real-time feedback was sought from patients and carers receiving support from Merton HTT. This team serves a South West London borough (Merton) with an estimated population of 219 600 (5). This covers a diverse community in terms of ethnicity and socioeconomic status across Mitcham, Morden, and Wimbledon. Merton HTT receives around 60 to 80 referrals per month and has a typical caseload of 30 to 40 patients at a time. It is an accredited service under the Royal College of Psychiatrists Home Treatment Accreditation Scheme (6).
Patients were invited to provide RTF on discharge from the Merton HTT. This was gathered via a tablet computer carried by team members. Patients could also provide feedback via an online questionnaire after being provided with a website link by team members. The feedback surveys available were identical. Real-time feedback was gathered if it was possible to do so. The questionnaire consisted of a 5-point Likert scale and a free text box inviting additional comments without any specific suggestions.
Quantitative and qualitative data were collected and analyzed centrally by the Trust before being disseminated to the team. The data were presented to the team at regular monthly business meetings and discussed. The feedback was displayed on a visual team dashboard of performance measures that were reviewed on a daily basis by the team manager. Areas for improvement were identified by the team consultant and manager, and changes were made to improve working practices.
Quantitative feedback on the team’s performance in the first 6 months of RTF use was compared against data from the 6 months prior to March 2015 using 2-tailed Z tests. At the time of data analysis, RTF had been in use for around 18 months.
Results
The percentage of positive responses to each item in the RTF questionnaire for patients and carers are shown in Figures 1 and 2, respectively. Items for carers are labeled as such. There were significant improvements in feedback around the team visiting at the agreed times (P = .0069) and patients feeling that they had been involved in treatment decisions (P = .0371). These items are labeled with asterisks.
Bar chart of patient responses. *Significant result (P < .05).
Bar chart of carer responses. *Significant result (P < .05).
Other changes did not reach significance. Table 1 gives the details on the number of responses, the percentage of those that were positive and the P values for each comparison of the proportion of positive responses between the first and last 6 months of the data collection period for patients and carers, respectively.
The qualitative data received were generally of limited quality, consisting mainly of brief positive comments without elaboration. As such, it has not been included in this article.
Discussion
The results appear to suggest that implementing the use of RTF in a HTT is an effective feasible process. The feedback data collected led to significant improvements in the experience of patients with more reporting that they (1) felt involved in decision-making and that (2) visits took place when they had been arranged. Figure 1 indicates that there was a possible general trend toward subjective improvement in the care provided by the HTT, although, only 2 results reached statistical significance.
The feedback received had been discussed within the team, and to improve the patient experience, staff members were encouraged to provide clearer information to patient regarding visit times, particularly if there was potential uncertainty. We also ensured that patients had our contact details from the beginning of their period of care with the team and placed more emphasis on the completion of collaborative crisis plans at an earlier stage. These specific changes appear to have led to a measurable improvement in the relevant items of the feedback questionnaire. Specific changes to clinical practice, rather than simply being aware of feedback, would appear to be important in enabling service improvement and change. The main challenge met by the team was determining how to make specific changes to service provision in a manner which would be sustainable within the usual workload and service structure. As with many HTTs, the team has a high volume of work, and any change to usual activities is only likely to be successful if it does not unduly burden the staff and is realistic to achieve. We also felt it was beneficial if staff could see a direct link between the change and an improvement for patients, particularly if this could be demonstrated via improved feedback.
There was evidence of some differences in response rates to questions within the survey. It was not necessary for patients to answer all questions to submit to a response; this was deliberate as we wished to reduce any barriers to providing feedback and some patients could, potentially, be disinclined to complete the entire questionnaire. The number of carer responses was also limited due to some patients not having social support or carers, or if they did, not always being present during reviews. Although data were not collected regarding the frequency with which the feedback was provided via the portable devices compared to the online questionnaire, it did appear that most was obtained through the tablet computers. This may be due to it being more convenient for patients and carers to use a tablet handed to them in the context of an appointment, rather than find time later to logon to a new website or complete and post a paper questionnaire. Although it is possible that the presence of staff members may have impacted on the feedback given, patients and carer were informed that their answers were entirely anonymized and this could have mitigated the risk of them providing more positive responses. Unfortunately, data were not collected regarding response rates or the reasons given by patients and carers for not wishing to provide the feedback.
The main limitations of this evaluation are in terms of the limited sample size, particularly for carers, and that the patient and carer survey questionnaires used are not validated, although based on a national previous survey (2). It is unfortunate that a larger RTF sample was not obtained as this may have produced further significant results. The survey item of “The support available was clear—Carer” was very close to significance and a larger sample may have revealed this as a statistically significant improvement from a carer perspective. The sample size was limited by the tablets used not always being functional, the willingness of patients or carers to provide feedback, the limited number of patients who had carers involved and HTT staff remembering to ask about the feedback and take the RTF device on the last home visit. More reliable IT equipment, preparation of patients and carers about requesting RTF, and increased emphasis upon obtaining feedback could address some of these issues and provide a larger sample and more robust results. Real-time feedback could also be requested at other time points in the patients’ care, for example, the RTF device being taken at the second to last visit to improve discussion and collection of data. It would be very useful if a larger evaluation of this sort could be performed. It would also be beneficial if a future evaluation could collect data on response rates and the demographic characteristics of patients and carers to determine whether these impacts upon the likelihood of feedback being provided and the rating received. These data were unfortunately not collected by this survey and so it is possible that the changes seen are due to changes in characteristics of the respondents, rather than changes in the way the service was delivered.
Conclusion
It is unclear by what mechanism the positive changes observed have occurred. Although the results obtained were monitored by the team consultant psychiatrist and team manager and fed back to the team, it would be useful if the methods used to effect these changes could be investigated. They could then be replicated elsewhere or improved upon to provide further benefits. A more detailed service evaluation could identity those processes that led to improved service provision and better experiences of care for patients and carers. Real-time feedback has now been expanded to all HTTs across the Trust and the number of devices per team increased from 1 to 3 to mitigate the risk of technical problems and to increase the rate of data capture and so the generalizability of feedback obtained.
Footnotes
Authors’ Note
The data used in this paper were part of routine service audit, and evaluation of ethical approval was not sought for publication. This is consistent with local practice.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.