Six years of Project ECHO: Implementation and evaluation of a hospice-led multiprofessional education programme

Abstract

Objectives: This article reports on the implementation and evaluation of an established technology-enabled collaborative learning programme (Project ECHO) at an independent UK hospice in the North of England over a 6-year period. Methods: An independent audit of collated, anonymised data from the programme is used to report attendance patterns and session evaluations. Results: The results show a gradual increase in attendances, programmes, sessions and hours of education, coupled with consistently positive evaluation reports. Conclusion: This supports existing evidence that Project ECHO is an effective method of delivering remote healthcare education, demonstrating impact on the first three levels of Moore’s education framework; participation, satisfaction and learning. Future expansion in terms of geography and topics covered is proposed, alongside enhanced evaluation methods to demonstrate impact at the higher levels of Moore’s framework.

Introduction

Project ECHO

In the early 2000s, there was an increasing health deprivation gap between urban and rural areas in the United States of America. ¹ Those individuals who needed expert care often had to travel hundreds of miles to specialist centres, and increasing waiting lists meant many patients were deteriorating and dying before being seen. ² Sanjeev Arora, a hepatologist, hypothesised that if rural physicians were trained in more specialist care, patients would be able to access this support quicker, and closer to home. ³ Consequently, Project Extension for Community Health Outcomes (Project ECHO) was created. Project ECHO made use of remote video-conferencing software long before this was common practice and integrated a structured approach to learning around this platform. Methods were based on social cognitive learning theory, situated learning theory, and community of practice theory.

ECHO was structured as a hub and spoke model, originally to help manage Hepatitis C infection in rural New Mexico. ³ The intention, outlined by the founder, was to provide “telementoring” of non-specialist clinicians, as opposed to telemedicine (where the expert clinician is responsible for the patient) or simple didactic teaching. Physicians received an initial day’s training at the central hub followed by weekly two-hour telehealth sessions consisting of a short lecture and case studies brought by attendees. Community providers became proficient in managing Hepatitis C infections leading to similar outcomes for their patients than those seen in the specialist centre.^2,3 Project ECHO therefore defines both the use of technology and the method by which the learning is structured.

Following on from this success, Project ECHO methodology was tested in other areas and conditions in New Mexico and the United States with similar outcomes.^4–6 The impact was considerable, leading to the passing of a bill in the US Senate recommending that barriers to the use of technology-enabled collaborative learning be reduced and opportunities and recommendations made regarding such models in medical education. ⁷ The current structure of a programme is similar to the original, and comprises a series of 1–2 h sessions, beginning with an induction in which the curriculum is agreed, and followed by a series of topic-specific sessions. Each of these sessions includes a didactic teaching followed by 1–2 anonymised case discussions supported by one or more experts in the subject and a trained ECHO model facilitator. Data collection is built into the model and evaluation follows the framework set out by Moore and colleagues, ⁸ which maps impact of medical education according to seven levels (Figure 1). The only data collection mandated by the ECHO institute is participation. This numerical information is entered onto a central database run by the ECHO institute. Other data collection is voluntary, although hubs are encouraged to collect data on satisfaction, confidence and competence.

Figure 1.

Moore’s evaluation framework. ⁸

In contrast with the large rural areas in the United States, the United Kingdom is approximately eight times more densely populated with much shorter distances to large hospitals. Despite this, there are still many hard-to-reach areas for healthcare education, especially within palliative and end-of-life care.^9–11

St Luke's ECHO programme

The St Luke's Project ECHO programme began as a response to a need to enhance support to care homes in the Sheffield area. St Luke's Hospice had a dedicated community specialist nurse supporting care homes, but it was recognised that there was scope to enhance this support. Following discussion with colleagues at Hospice UK (HUK) and Highland Hospice, the programme lead (SK) attended training and a pilot programme was set up in 2017 with the existing clinical team.

Over time, a combination of inquiries from educators and clinical services, and a need to grow the outreach to local care homes, resulted in the expansion of the programme, and a dedicated leadership fellow and administrator were appointed. It was recognised that a combination of local clinical support and ECHO expertise was ideal for ongoing education. This, coupled with the positive evaluations of the existing work, ¹² led to the decision of the programme lead to work with HUK in developing St Luke's as a ‘Superhub,’ becoming the third in the UK alongside HUK ¹³ and Highland Hospice. ¹⁴ Superhubs are designated by the ECHO institute as centres of expertise, acting as both ECHO delivery centres and providers of training for other units to become hubs. From this point, the St Luke's ECHO team was well placed to grow a network of ECHO hubs.

Further development in St Luke's enabled progression from a Superhub poised to train others, to one that would also deliver a range of ECHOs across subjects and health and social care boundaries. This was made possible by the team focusing on a service development approach and implementation that was inspired by the early work of Eva Serhal and colleagues. ¹⁵ As opportunities for collaboration and connection rapidly increased it enabled the St Luke's team to have a coherent, service improvement-based approach to the planning, delivery and evaluation of a range of different partner organisations and subject areas.

Areas for further development were (1) to grow the ECHO programme to areas beyond local specialist palliative care, including Internal Medicine Training (IMT) ¹⁶ and paramedic training ¹⁷ and (2) to scale to support in a wider area. These changes have progressed and, in some areas, have been specifically beneficial by allowing ongoing educational training to professionals unable to attend central units during the pandemic. ¹⁶

In the early stages of St Luke's ECHO programme, the team provided IT hardware support to care homes such as cameras and Wi-fi connection devices. One impact of the COVID-19 pandemic has led to increased use of remote technology, meaning implementing ECHO as an attendee or a hub is now relatively straightforward. The infrastructure supporting St Luke's ECHO programme has grown iteratively, however, as the ambitions expanded. The ECHO team comprises a manager, leadership fellow, medical lead, administrators and clinical support staff. A dedicated centre provides work space for the ECHO team and two protected rooms for delivery and support of programmes.

Local, national, and international ECHO recognition, alongside the persistent efforts of the strategic ECHO lead, persuading commissioners of the many advantages of the project, has led to a substantial increase in ECHO delivery which currently sits at 13 programmes and 10 team members. As part of their Superhub requirement, St Luke's have also trained 15 other organisations to be ECHO hubs, including three others in South Yorkshire, ensuring that ECHO delivery is equitable throughout the South Yorkshire Integrated Care Board.

This paper reports an evaluation of a large-scale programme of education which has grown over 6 years. We describe the degree of education and support provided across the local, regional and national networks, with particular focus on the number of programmes, sessions and attendances. No other existing studies report Project ECHO activity on the same scale. This is useful for planning further services and studies and estimating workload involved in developing a similar programme.

Methods

Project ECHO integrates assessment into the delivery programme; a guiding principle is using data to monitor outcomes to increase impact. ¹⁸ Attendees agree to submit pre- and post- session self-rated confidence measures, to rate the quality of the programme, and for attendance data to be recorded. A GDPR consent form is completed by attendees to allow evaluation of anonymised data.

Data is collected through an online form and stored on a central database at the Superhub site. Pre- and post- data is linked using a pseudonym. Data is stored and analysed using Microsoft Excel. Additional aggregated data regarding attendances is submitted to the ECHO institute. Specific programmes have additional outcomes for analysis; this paper focuses on common data collected across the majority of programmes.

To provide internal validation, an initial 10 polling questions were developed by the ECHO institute to evaluate ECHO delivery and were used for the first 4 years of Project ECHO at St Luke's. These were reviewed and adapted based on an internal service evaluation and published literature ¹⁹ in order to standardise data collection and further evaluate effectiveness, resulting in a standard set of eight questions introduced in March 2020. However, it should be noted that not all programmes include these standard questions, for reasons including scope/purpose of the individual programme, logistics or attendee profile (e.g. for the Bereavement programme which involved members of the public).

In collaboration with The University of Sheffield, we used collated anonymised data to report attendance patterns and session evaluation over the 6-year period since the implementation of Project ECHO in Sheffield (14th February 2017 – 31st March 2023). This involved an independent audit of ECHO activity conducted by a researcher from The University of Sheffield embedded within the St Luke's Hospice Project ECHO team, as part of an evaluation of Project ECHO in the region commissioned by the South Yorkshire and Bassetlaw Integrated Care Board (SYB ICB). Ethics approval for the evaluation of the programme was given by the Research Ethics Committee in the School of Medicine and Population Health at the University of Sheffield (application reference no. 050553).

Results

Since the commencement of Project ECHO at St Luke's Hospice there have been 87 programmes incorporating 550 ECHO sessions. This adds up to 762 h of education delivered and 11,515 individual attendances (see Tables 1–7 in Appendix 1).

Over the 6-year period, Figure 2 reports numbers of sessions and total hours of training delivered, and Figure 3 reports the number of attendances. In these reports, an “attendance” represents an individual attending a single session. These figures demonstrate the consistent increase in ECHO delivery across this 6-year period. Attendance during the period 2020–2021 is an outlier to this due to the COVID pandemic.

Figure 2.

St Luke's Project ECHO number of sessions delivered and education hours each year.

Figure 3.

St Luke's Project ECHO total attendance each year.

Figure 4 shows average attendances in each session across ECHO programmes, limited to programmes containing between three and five sessions (as there are fewer programmes containing six or more sessions). This demonstrates that there is a steady decline in attendance over the course of a programme.

Figure 4.

Average attendances across programmes (3 to 5 sessions in length; N = 24).

A number of programmes include a standard set of eight evaluation questions that are shared with attendees at the end of each session. Figure 5 shows the cumulative session evaluation data for 15 different ECHO programmes over the last 2 years. This aggregated data demonstrates that Project ECHO encourages a safe learning environment where attendees have the opportunity to ask questions. Most attendees felt that the sessions met their expectations and that the education and case studies increased their knowledge. Over three quarters of individuals reported that they would change their practice following something they had learnt in the session indicating that Project ECHO is making a difference to patient care.

Figure 5.

Cumulative programme evaluation data at St Luke's Hospice 2021–2023 (n = 15 programmes).

Discussion

This evaluation reports on the implementation of Project ECHO at an independent hospice over a 6-year period. During this time period, there was a gradual increase in all outcomes measured, including unique attendances, number of programmes delivered, number of unique sessions and hours of education. This is consistent with ECHO as an effective method of delivering healthcare education remotely. Where evaluations were returned, findings are consistently positive, supporting ECHO as a method that is well-received, enhances self-confidence and impacts on practice. Infrastructure grew iteratively with the expanding interest and ambitions of the project, leading to allocation of dedicated staff, physical space, and resources to support the programme.

Whilst attendance figures increased over time as the programme developed; a peak in 2020/2021 can be explained by the impact of the COVID-19 pandemic. Although all face-to-face training ceased, Project ECHO was already set up to provide education remotely. This allowed continued delivery of some existing palliative care training, ¹⁶ and provided opportunities to disseminate the latest pandemic knowledge to primary care and care homes in Sheffield with substantial attendances (see Table 5 in Appendix 1); an opportunity noted beyond Sheffield. ²⁰

Although St Luke's have had over 11,000 individual attendances, attendance throughout a given programme is not consistent, and reduces over time. The only exception to this was the IMT ECHO, which rewards attendance and is mandated by training bodies. The reason for attendance drop off is not known; an evaluation by Diffin and colleagues ²¹ on Project ECHO attendance in Northern Ireland found that low attendances were associated with staff shortages and workplace demands. This was echoed internationally by Salvador and colleagues. ²²

Comparison with existing literature

Project ECHO references Moore’s framework for medical education, ⁸ which defines the impact of training as moving through seven levels from participation to community health. By demonstrating increased self-reported confidence and impact on practice, this programme has shown impact on the first three levels; participation, satisfaction and learning. This is consistent with other studies reporting Project ECHO,^23–25 including detailed evaluations of individual programmes by this centre.^12,16,17

As part of a service evaluation project for St Luke's, we conducted a rapid systematic review of UK-specific evidence relating to the delivery of medical education to health and social care professionals in the UK via Project ECHO. 13 studies conducted in Scotland, Northern Ireland, and England were included; the majority of which related to qualitative studies of Project ECHO and the delivery of palliative care education. Qualitative synthesis of the included studies highlights the impacts of Project ECHO, including: accessibility and flexibility, community of learning, knowledge and skills, changes in professional practice, and importantly - impact on patients. We also identified key positives of Project ECHO, such as the peer support and democratisation of knowledge which the approach fosters, as well as reducing barriers to training and education and the capability to deliver a wide range of multidisciplinary topics. However, there are still challenges to overcome in order to improve and capitalise on the successes – short term funding and lack of infrastructure can be a barrier, and for optimum impact there needs to be adequate managerial, administrative, and IT support.

Strengths and limitations

This paper reports a large-scale programme of education which has grown over time; no other existing studies report Project ECHO activity on the same scale. This is useful for planning further services and studies and estimating workload involved in developing a similar programme. We also report some consistent data across multiple ECHOs, demonstrating that the benefits are not specific to given subjects, but are replicated across programmes.

This evaluation does, however, use retrospective data, originally collected for attendance purposes and then scaled to incorporate evaluations. As a result, detailed data and data on impact is not collected across the full 6-year period, or consistently between programmes, and should therefore be interpreted with caution. Furthermore, future work and development of hubs would benefit from ensuring that there is a consistent core of questions measuring general impact, confidence and satisfaction, which would allow benchmarking and measuring change over time.

Implications for practice

Unlike delivery in the United States or Northern Ireland, St Luke's Hospice does not receive central funding for ECHO delivery therefore each programme has had to be commissioned separately. The implications of this are two-fold. First, smaller one-off programmes tend to be commissioned due to the uncertainty of NHS funding streams. Second, St Luke's programme evaluation has been collecting evidence for evaluation from the beginning due to the necessity to consistently demonstrate effectiveness; conversely where funding is centralised, ECHO networks can build engagement and develop their network to promote sustainability. ²⁶

It is also important to highlight that the delivery of ECHO here and in other Superhubs has been contingent on strong leadership, championed by a programme lead who works in a number of clinical and non-clinical domains, and who was able to build a team to support the vision. Furthermore, successful implementation of a programme of this scale requires significant organisational buy-in, in terms of technological support, staff time and resource.

Implications for further research

One area which is important to address is the reduction in attendance over time in a given ECHO programme. Where attendance is incentivised, attendance is higher; for example, in the IMT ECHO, attendees also receive a day in lieu following attendance in the full programme (9 h) and attendance allows trainees to meet mandated training objectives. A recent programme providing training to professionals supporting people with learning disabilities has included routine contact with non-attenders to explore reasons for absence. Future evaluations would be useful in further understanding reasons for and patterns of non-attendance and methods for incentivisation.

As noted above, the impact of ECHO can be mapped to Moore’s framework, ⁸ but evidence related to the higher levels (competence, performance, patient health, population health) are not readily demonstrated here. Future research should consider these higher levels; this necessitates moving beyond the routine integrated evaluation data and requires a combination of prospective qualitative and quantitative research, following up attendees and inclusion of wider outcomes specific to the programme. This level of research is, however, costly, requiring dedicated grant funding. Examples of such research include a proposed 2-arm randomised controlled trial by Godino et al. ²⁷ to evaluate Long-COVID management using ECHO. Outcomes for this will be patient surveys, physical assessments and ‘Fitbit’ activity as well as clinician surveys and an evaluation on programme sustainability. Another mixed-methods evaluation of ECHO is proposed by Calo et al. ²⁸ Their use of implementation outcomes, implementation effectiveness and patient outcomes in HPV vaccination aims to demonstrate ECHO as effective at level 6 (patient health) on Moore’s evaluation framework. One study taking this further is Rattay et al. ²⁹ who performed a cost effectiveness analysis in ECHO for Hepatitis C. They demonstrated that Project ECHO increased survival rates and quality adjusted life years indicating a population health impact.

Conclusion

This evaluation reports on the implementation of Project ECHO at an independent hospice in the North of England, and the delivery of training across the region. We demonstrate significant outreach in terms of attendances and sessions delivered, with self-reported evaluations showing further positive impact on attendees. Project ECHO has the potential to expand further in terms of geography and topics covered, and future developments should consider enhancing the evaluations to include further evidence of impact.

Supplemental Material

Supplemental Material - Six years of project ECHO: Implementation and evaluation of a hospice-led multiprofessional education programme

Supplemental Material for Six years of project ECHO: Implementation and evaluation of a hospice-led multiprofessional education programme by Phil Joddrell, Jane Manson, Sam Kyeremateng, Gerlinde Pilkington, Steven Ariss, Kinga Lowrie, Laura McTague and Paul M Taylor in Health Informatics Journal.