Positive impact of a telemedicine education program on practicing health care workers during the COVID-19 pandemic in Ontario,Canada: A mixed methods study of an Extension for Community Healthcare Outcomes (ECHO) program

Abstract

Introduction

In addition to shifting and expanding clinical responsibilities, rapidly evolving information and guidelines during the COVID-19 pandemic has made it difficult for health care workers (HCW) to synthesise and translate COVID-19 information into practice. This study evaluated whether a COVID-19-specific telemedicine education program (ECHO COVID) would impact health care workers’ self-efficacy and satisfaction in the management of patients with COVID-19.

Methods

A prospective mixed methods parallel-design study was conducted among ECHO COVID participants using pre-post questionnaires and a focus group discussion. Questionnaire results were examined for changes in health care workers’ self-efficacy and satisfaction. Focus group discussion data were analysed to explore health care workers’ experience in ECHO COVID and the context of their practice during the COVID-19 pandemic.

Results

239 health care workers registered in ECHO COVID and 114 (47.7%) completed questionnaires and attended at least one ECHO COVID session. Median self-efficacy scores increased from 5 (IQR 4–6) to 6 (IQR 6–6) (p < 0.0001), independent of profession, years in practice, age group, or practice environment. Participants were highly satisfied with ECHO COVID sessions with a median score of 4 (IQR 4–5). Focus group discussion data indicated that health care workers gained knowledge through ECHO COVID and revealed facilitators for ECHO COVID program success, including the transition to virtual care, the practicability of knowledge provided, and a ‘perspective from the trenches.’

Discussion

This study demonstrated that a telemedicine education program aimed to support health care workers in managing patients with COVID-19 had a positive impact on health care workers’ self-efficacy and satisfaction. This impact was specifically mediated by the ECHO COVID program.

Keywords

telemedicine,COVID-19 primary care mixed methods continuing medical education ECHO (Extension for Community Healthcare Outcomes)

Introduction

The impact of the COVID-19 pandemic on health care service and delivery has been profound.^1–4 In addition to navigating the transition to virtual care,^5–7 staffing and clinic changes,^8–12 and vaccine distribution challenges,¹³ health care workers (HCWs) have had to learn about the novel SARS-CoV-2 virus. Many HCWs worldwide are experiencing increased psychological distress^14–18 and burnout.^19–21 In Canada, the burden of COVID-19 has put strain on the health care system.^22–25 Furthermore, HCWs have difficulty synthesising and applying evidence-based COVID-19 guidelines and research into practice due to the rapidly evolving changes in guidelines, volume, and quality of information.^26–28

Project Extension for Community Healthcare Outcomes (ECHO) model is a virtual, telemedicine education model that provides longitudinal support and addresses the emerging needs of HCWs.²⁹ ECHO programs have been successfully implemented internationally, providing rapid access to evidence-based research, policy, and practice information.^30–33 In April 2020, University Health Network (UHN) leveraged the pre-existing ECHO model as a model of care to support assessment and management of COVID-19 patients. ‘ECHO Ontario: Managing COVID-19 Patients in the Community’ (ECHO for the remainder of this paper) was launched in July 2020. The goal of the program was to disseminate best practices regarding COVID-19 as they emerged and to increase HCWs’ self-efficacy and knowledge of COVID-19.

The objectives of this study are: (1) to examine the impact of a telementoring education program on HCWs’ self-efficacy and satisfaction and (2) to explore HCWs’ experience in the program and context of practice during the COVID-19 pandemic.

Methods

Study setting

The ‘ECHO Ontario: Managing COVID-19 Patients in the Community’ program was represented by an interprofessional ‘hub’ team, including medicine (family medicine, infectious diseases, internal medicine, respirology, critical care, palliative care, and physiatry), occupational therapy, naturopathy, pharmacy, physiotherapy, social work, and information specialists. The target audience was any practicing HCWs in Ontario.

Each ECHO session ran on Tuesdays for one hour. Weekly sessions included a didactic lecture and a patient presentation.³⁴ All sessions were recorded and available for review after sessions. Participants attended at no-cost and received Continuing Medical Education (CME) or Continuing Professional Development credits for their attendance.

A needs assessment was conducted to develop the didactic curriculum, based on available resources from the international ECHO (MetaECHO) community.³⁵ The final curricula included: classification of COVID-19, pharmacotherapy, management of special populations, community resources, respiratory health, etc. (Appendix 1). Patient lived experiences as didactic sessions were also incorporated in each cycle, which offered a unique perspective on care.

Study design

A prospective mixed methods parallel-design study was conducted to evaluate the impact of ECHO on HCWs.³⁶ A pre-post questionnaire was administered through Survey Monkey®. A focus group discussion (FGD) was conducted to explore the context of practice during the COVID-19 pandemic and the impact of ECHO on their practice. Data were collected from both parts of the study concurrently and analysed.

Study participants

Participants were recruited from June 2020 to March 2021. Participant eligibility for this study included (1) all HCWs (family medicine and specialist physicians, physician assistants, nurse practitioners, registered nurses, registered practical nurses, pharmacists, and other allied health professionals), (2) practicing in Ontario and (3) attended at least one ECHO session. Exclusion criteria included all HCWs practicing outside of Ontario, Canada or those not actively in practice.

Data sources: Pre-post questionnaires

The Pre-Post questionnaires were self-reported and contained sections on self-efficacy and acceptability and satisfaction. Demographics and practice characteristics were collected.

Self-efficacy is defined as an individuals’ confidence in their ability to perform a particular task.³⁷ In addition to knowledge, self-efficacy has been demonstrated to be an important mediator for behavioural change among practicing HCWs.^38,39 Participants reported self-efficacy using a 17-item scale, adapted from previous ECHO programs. Each item in this section was assessed on a 7-point Likert scale which ranged from 1 = ‘strongly disagree’, to 7 = ‘strongly agree’. Responses of ‘not applicable’ for any item were excluded. Self-efficacy items included ability to diagnose COVID-19, to identify which tests to order, and to manage special patient populations.

Acceptability and satisfaction with ECHO was measured using an 11-item scale. This section used a 5-point Likert scale where 1 = ‘strongly disagree’ and 5 = ‘strongly agree’, with a ‘not applicable’ option. Items included statements like ‘I believe that the quality of care I provide my patient has improved considerably as a result of my involvement in project ECHO’ and ‘Project ECHO has reduced variations in care’.

Data sources: focus group discussion

One FGD was conducted in November 2020 with participants from the first cycle. The group consisted of five study participants with one facilitator (QJZ) and one note taker. The FGD was conducted and recorded over Zoom. The recording was transcribed verbatim.

Purposive sampling was used to recruit participants who had completed the pre-ECHO questionnaire. This sampling strategy was employed in order to represent a broad cross section of participants, including professions, practice types, and years in practice.

The FGD guide was developed with the research team, containing open-ended questions (Appendix 2). Two broad topics of inquiry included 1) HCWs’ experience in ECHO and the impact on their practice, and 2) the context of practice during the COVID-19 pandemic. All FGD participants received a $50 honorarium for their time.

Data analysis

Statistical analysis

Statistical analyses were performed using GraphPad Prism 9.0 (GraphPad Software, Inc., San Diego, CA). For participant characteristics, self-efficacy, and satisfaction responses, continuous and categorical variables were used including mean ± standard deviation, median (interquartile range [IQR]), frequencies (n), and percentages, as applicable. Data distribution was evaluated visually and with the use of Kolmogorov-Smirnov and Pearson omnibus normality tests. Statistical significance between groups was assessed using t-tests (parametric and non-parametric) and chi-square for categorical variables. The changes in self-efficacy scores were evaluated using paired t-tests. Those with missing data for a particular parameter or question were excluded. A p-value of <0.05 was considered significant for all analyses.

Qualitative data analysis

The qualitative descriptive method was employed to synthesise and analyse the focus group transcript. This analysis method was used because it allowed us to produce a comprehensive summary of the data in plain language and to stay close with the original meaning of the data.^40,41 Two members of the study team (QJZ, SN) reviewed the questionnaire data and transcript to develop a coding framework of recurrent themes. The themes were then presented to a small working group (DR, SN, AF). Through iterative discussion during the analysis phase, all themes were discussed and discrepancies were resolved verbally.

Qualitative content analysis was conducted inductively to identify emergent themes from the data and develop a framework for coding.^42,43 Codes identified the topics in the data pertinent to the research objectives. Data were then grouped into broader themes. All qualitative data was analysed using NVivo (NVivo 11 for Windows, Version 11.0.0.300 by QSR International).

Ethics approval

This study received an exemption by the UHN Research Ethics Board and was approved through the UHN Quality Improvement stream (QI ID#20-0039).

Results

Between July 2020 and February 2021, ECHO offered 24 sessions (twelve per cycle). 239 HCWs registered for the program and of those, 114 (47.7%) attended at least one ECHO session and completed at least one of the pre-post questionnaires. 50 (43.9%) completed both pre-post questionnaires and were grouped for subgroup analyses. Completion of the pre-ECHO questionnaire took 4 min and 19 s on average, while completion of the post-ECHO questionnaire took an average time of 4 min and 38 s. Comparing HCWs who registered but did not attend any sessions and HCWs who registered and attended at least one ECHO session with completion of questionnaires, differences were observed between groups for years in practice (p = 0.0009), age group (p = 0.0006), and number of patients with COVID-19 (p = 0.0023) (Appendix 3).

Participant characteristics

Of the 114 HCWs who were included in this study, 99 participants were female (86.8%), 42 identified as a nurse practitioner (36.8%), and nearly half reported to be in their first ten years of practice (49, 43.0%) (Table 1). 42 ECHO participants reported practicing in a family health team (36.8%), while 82 worked in an urban setting (71.9%), and 45 (39.5%) reported no patients with COVID-19 in their practice. Participants in the total cohort had a median attendance of 3 ECHO sessions (IQR 2–6). The participants in the total cohort attended fewer sessions than those in the subgroup with a median of 3 [2–6] and 4 [3–9] sessions, respectively (p = 0.014). No other statistical differences between participant characteristics in the total cohort and subgroup were observed.

Table 1.

Baseline cohort characteristics of participants.

	Total cohort (n = 114)	Subgroup^a (n = 50)	p-value
Profession			0.94
NP	42 (36.8%)	26 (52.0%)
MD	34 (29.8%)	14 (28.0%)
Other	38 (33.3%)	10 (20.0%)
Female	99 (86.8%)	44 (88%)	0.84
Years in practice			0.70
0–9	49 (43.0%)	21 (42.0%)
10–19	30 (26.3%)	14 (28.0%)
≥20	21 (18.4%)	13 (26.0%)
Unknown	14 (12.2%)	2 (4.0%)
Age group			0.51
20–39	47 (41.2%)	18 (36.0%)
40–59	40 (35.1%)	20 (40.0%)
≥60	15 (13.2%)	10 (20.0%)
Unknown	12 (10.5%)	2 (4.0%)
Type of practice ^b			0.95
Family health team	42 (36.8%)	20 (40.0%)
Hospital	19 (16.7%)	10 (20.0%)
Independent practice	13 (11.4%)	6 (12.0%)
Community health center	10 (8.8%)	7 (14.0%)
Long-term care	8 (7.0%)	4 (8.0%)
Family health organization	6 (5.3%)	4 (8.0%)
Other	16 (14.0%)	13 (26.0%)
Two or more groups selected	11 (9.6%)	4 (8.0%)
Attendance			0.17
Attended one cycle	91 (79.8%)	35 (70.0%)
Attended both cycles	23 (20.2%)	15 (30.0%)
Total attendance, median (IQR)	3 (2–6)	4 (3–9)	0.014
Type of practice environment			0.53
Urban	82 (71.9%)	35 (70.0%)
Rural	27 (23.7%)	14 (28.0%)
Not applicable	5 (4.4%)	1 (2.0%)
Number of patients with COVID-19 in practice			0.97
0	45 (39.5%)	23 (46.0%)
1–9	17 (14.9%)	7 (14.0%)
≥10	11 (9.6%)	5 (10.0%)
Not applicable	7 (6.1%)	2 (4.0%)
Unknown	34 (29.8%)	13 (26.0%)

Note: NP: nurse practitioner; MD: physician; IQR: interquartile range.

Participants who completed both pre- and post-ECHO self-efficacy questionnaire.

Will not total to 100% as participants were given the option to select more than one type of practice.

Quantitative findings

Self-Efficacy

The median self-efficacy score for the total cohort pre-ECHO was 5, ‘somewhat agree’, (IQR 4–6) and the post-ECHO score was 6 ‘agree’ (IQR 6–6) (Figure 1(A)). HCWs reported lowest self-efficacy pre-ECHO in managing their pregnant patients with COVID-19 and their patients with substance use disorder with COVID-19 (Appendix 4). In the subgroup, median scores for all items increased post-ECHO (Figure 1(B)), with the sum of all items increasing from 4.7 (IQR 4.1–5.5) pre-ECHO to 6.0 (IQR 5.5–6.4) post-ECHO (p < 0.0001) (Appendix 5).

Figure 1.

(A) percentage distribution of median pre-ECHO and post-ECHO self-efficacy scores across all 17 items for total cohort (n = 114), (B) percentage distribution of median pre-ECHO and post-ECHO self-efficacy scores across all 17 items for subgroup (n = 50).

There was significant improvement in self-efficacy scores for all 17 items independent of underlying participant characteristics, with the exception of male sex (p = 0.094) (Table 2). Similarly, ECHO participants who had been in practice for 10 or more years were found to have higher pre-ECHO self-efficacy scores compared to those with 0–9 years in practice (p = 0.0209), but scores increased in both groups to similar values post-ECHO.

Table 2.

Comparison of participant characteristics and pre-post ECHO self-efficacy median scores, subgroup (n = 50).

	Self-efficacy score (Median, IQR)		Median difference Post versus Pre (95% CI)	p-value
	Pre-ECHO	Post-ECHO	Median difference Post versus Pre (95% CI)	p-value
Profession
NP	4.7 [4–5.4]	6 [5.3–6.4]	1.3 (0.6–1.7)	<0.0001
MD	4.8 [4.1–5.8]	6 [5.4–6.5]	1.1 (0.1–2)	0.0051
Other	4.7 [4.3–5.3]	6 [5.8–6.6]	1.4 (0.3–1.9)	0.0020
Sex
Female	4.8 [4.1–5.6]	6 [5.5–6.5]	1.3 (0.8–1.5)	<0.0001
Male	4.4 [3.7–5.5]	6 [5.5–6.3]	1.6 (−0.4–4)	0.0938
Years in practice
0–9	4.2 [4–5.4]^a	6 [5.3–6.4]	1.5 (1.0–2.1)	<0.0001
≥10	4.9 [4.4–5.8]	6.1 [5.6–6.5]	0.9 (0.3–1.5)	<0.0001
Age group
20–39	4.6 [4–5.5]	6 [5.3–6.4]	1.5 (0.1–2.1)	0.0013
≥40	4.8 [4.2–5.7]	6 [5.7–6.5]	1.2 (0.7–1.7)	<0.0001
Attendance
Attended one cycle	4.6 [4–5.6]	6 [5.5–6.5]	1.4 (0.8–1.7)	<0.0001
Attended both cycles	4.8 [4.3–5.4]	6 [5.5–6.3]	1 (0.2–1.8)	0.0084
Type of practice environment
Suburban/urban	4.5 [4–5.7]	6 [5.5–6.4]	1.3 (0.7–1.7)	<0.0001
Rural	5.1 [4.2–5.5]	6.1 [5.7–6.4]	1.1 (−0.4–2.1)	0.0134
Number of patients with COVID-19 in practice
0	4.3 [4–5.5]	6 [5.3–6.3]	1.4 (0.3–1.7)	0.0007
≥1	4.8 [ 4.4–5.7]	6.2 [5.6–6.5]	1.3 (0.1–1.8)	0.0034

Note: CI: confidence interval; NP: nurse practitioner; MD: physician.

Within group difference, p = 0.0209.

Satisfaction

All satisfaction items were evaluated for each participant (Appendix 6). The median of all satisfaction scores for all ECHO attendees was 4 [IQR, 4–5], indicating that most participants ‘agreed’ or ‘strongly agreed’ with the majority of satisfaction items (Figure 2).

Figure 2.

(A) percentage distribution of median acceptability and satisfaction scores across 11 items for total cohort (n = 114), (B) percentage distribution of median post-ECHO satisfaction scores across 11 items for subgroup (n = 50).

Qualitative findings

Five participants participated in the 60-min FGD on November 24, 2020. Two were nurse practitioners, one family medicine physician, one pharmacist, and one chiropractor. They practiced at six practice settings: family health team, long-term care facility, independent private practice, aboriginal health access centre, hospital, and nurse practitioner-led clinic. Three participants (60%) practiced in urban centres in Southern Ontario, while two (40%) practiced at rural centres in Northern Ontario. Three participants (60%) identified as women, and the mean years in practice were 35.4 years (SD 11.3).

After analysing the FGD transcript, three themes were identified: (1) knowledge gained in ECHO, (2) context of practice during the COVID-19 pandemic, and (3) facilitators for ECHO program success. HCWs discussed how their unique context of practice changed during the COVID-19 pandemic (Table 3). Interestingly, virtual care increased attendance at ECHO sessions, allowing access to sessions where it may not have been feasible prior.

Table 3.

Participant quotations supporting each theme.

Theme	Representative quotation
Knowledge gained in ECHO Asymptomatic transmission Patient experience Patient dermatological changes	Most asymptomatic transmission is pre-symptomatic and they will develop symptoms, however mild. […] There are very few asymptomatic transmissions. (P_037, NP)In terms of [the patient] and her lived experience, I think that's so powerful to hear a message like that. (P_020, Pharm)Seeing some of the dermatological changes that patients potentially experience with COVID [was helpful]. We can talk about it but when you can see it, it makes a difference. (P_028, DC)
Context of practice during the COVID-19 pandemic Different work environment: changes in clinic structure: PPE, virtual care Shut down or reduction of health care services Emotional components: burnout, stress, hopelessness Relief for no COVID-19 patients, but sense of impending Increased hospitalisation of patients with chronic diseases due to delayed care	[COVID] changed the whole dynamics and everything about the facility changed for everybody […]. I could hardly recognise some of my colleagues with all our gear, even without full PPEs we’re all in masks and shields. […] A lot of staff are extremely stressed, it's more work and the end isn't in sight. […] It's a totally different work environment to anything I have worked in, in all my [37-year] career. (P_009, MD)For myself as a chiropractor, we had a complete shut down in all the services we were providing in the province. (P_028, DC)We have been lucky because we don't have a lot of cases in the area that we cover, so for us we really dodged a bullet and I feel a nice relief. But at the same time, we have not been spurred into practice because we haven't had positive cases. (P_029, NP)We are seeing an increase in chronic diseases that are ending up in the hospital. We have 120% capacity across Northern Ontario in many of our hospitals. Our larger centres are reaching out to smaller centres to take patients that are not COVID-related and if it's not COVID-related, it's related to people being very sick because they’re not getting care early on. (P_037, NP)
Facilitators for ECHO program success Virtual event facilitated ECHO attendance Practicability and relevance of knowledge Multidisciplinary aspect of sessions Clinical pearls Perspective from the trenches Connection to community of support Connection for HCWs practicing in more isolated regions	It's actually one of the advantages of the pandemic because I’m working from home more, so can schedule around the time of the ECHO. And when I was in office more, I wouldn't have had the time to do that. (P_009, MD)There's no problem getting knowledge. The hardest part is integrating it into your practice. And that is where I find [ECHO] COVID is good. […] There is such a lag between the time the information gets out there until it's actually adopted in practice. (P_029, NP)It's also nice that the folks are multidisciplinary, and we get perspective from different people. (P_009, MD)With ECHO, I don't have to spend $500.00 [on a conference] and I get usually more than one pearl out of the whole process […] I could apply a lot of those pearls on Monday morning quite easily. (P_028, DC)A lot of the education is academic ivory tower and sometimes this gives a perspective to people in the trenches who can't be exactly like what the ivory tower is. It's made more real in that way. (P_009, MD)ECHO is a great way to connect with other specialists, […]especially in northern Ontario. It really is connecting us to the more urban centres and I also think […]it's good for others to see the work that we’re doing in northern Ontario. (P_037, NP)
Take-home points:	HCWs describe the overwhelming and shifting context of primary care medical practice during the COVID-19 pandemic in Ontario, Canada. HCWs share that they have gained knowledge from attendance at ECHO COVID telemedicine education sessions. These include knowledge around asymptomatic patient transmission, visual images of patient symptom changes, and understanding more fully a patient with COVID-19's lived experience through hospital care. Many facilitators for ECHO program success were shared: ECHO COVID offered a space for connection, community, and support amongst colleagues; ECHO offered a perspective from the trenches with other stories of frontline HCWs, and that virtual care enabled attendance at ECHO and other virtual events.

Integration

Synthesising the quantitative and qualitative data together, the data collected illustrated a rich view of frontline HCWs during the COVID-19 pandemic in the province of Ontario. The FGD explored HCWs’ knowledge gained and offered facilitators to program success, which were not otherwise available through questionnaire data. Since knowledge acquisition through multiple choice testing methods could not be ascertained from questionnaires due to the evolving nature of COVID-19 information at the time of program implementation, data collected through the FGD described examples of knowledge change.

Discussion

This study demonstrated that a telemedicine education program (ECHO Ontario: Managing COVID-19 Patients in the Community) had a positive impact on HCWs in primary care. Quantitative results demonstrated that median self-efficacy scores increased significantly from 5 (IQR 4–6) to 6 (IQR 6–6) (p < 0.0001), independent of profession and other characteristics. Participants were also highly satisfied, median score of 4 out of 5 (IQR 4–5). Qualitative results revealed that knowledge was gained through session attendance and explored the context of practice in the COVID-19 pandemic in Ontario, Canada. The participants of the FGD also shared facilitators for ECHO program success, including the transition to virtual care.

As the COVID-19 pandemic has evolved, HCWs’ learning needs have evolved as well. Findings from this study are similar to others, which demonstrate that the ECHO model is well-suited for rapid and proactive implementation, features identified that increase the strategic uptake and sustainability of telehealth interventions overall.^44–46 Globally, 304 ECHO programs have launched in 35 countries focusing on COVID-19.³⁵ Across the United States, states like Missouri, Oregon, New Mexico, and Virginia rapidly developed and launched programs in response to COVID-19.^47–52 Response has been overwhelmingly positive, with 500 to over 9000 attendees and reports of high satisfaction. The World Health Organization partnered with the ECHO Institute to launch an ECHO program for the continent of Africa, where there was high demand for credible sources on COVID-19.⁵³ Finally, in Canada, ECHO Coping with COVID (ECHO-CWC) was launched to support HCWs’ mental well-being, where they found that, again, participants were highly satisfied with initial sessions (mean = 4.26/5).⁵⁴

The COVID-19 pandemic has been an inciting event that resulted in increased utilisation and innovation of virtual care.^5,55,56 Virtual care has broadened access to medical care while maintaining clinical effectiveness, though success remains to be demonstrated across all clinical domains.^57–59 Another outcome of virtual care has been increased attendance at CME events like conferences and ECHO programs. This was exemplified through this ECHO COVID program, which ran Tuesdays during clinic hours. Although attendance was high compared to other programs, findings from this study reveal that the transition to virtual care enabled participants the flexibility to attend ECHO sessions. This aligns with several studies across medical disciplines, where virtual care has enabled higher attendance at such events.^60–62 Timing and delivery platform should be considered when planning and implementing CME events.

To our knowledge, this was one of the first studies that explored knowledge gain by HCWs in a COVID-19 telemedicine education program. Knowledge change is difficult to measure among learners, especially in CME events and with a multidisciplinary audience.^63,64 Nonetheless, knowledge acquisition is a primary goal of education. Rather than miss this opportunity, a mixed methods study approach was used.^65,66 For future program evaluation with an evolving clinical evidence base, qualitative research methods to measure knowledge change can be considered.

Canada is in the midst of a fourth wave: 1.58 million people have tested positive for COVID-19 with over 27,500 deaths, individuals 12 years and older are eligible to receive a vaccine, and hospital systems in some regions of Canada remain at capacity from surging acute care hospitalisations.^25,67 This study does not capture the impact of the vaccine rollout on outcome measures. Furthermore, even though participants joined from outside of Ontario and internationally (Brazil), this data was not analysed for this study. Future research will aim to collect more qualitative data from HCWs and compare the out-of-province and international cohorts with Ontario participants. Future research will also aim to collect healthcare utilisation data through provincial administrative databases, which will investigate impact of training HCWs on clinical utilisation and costs.

As the transition begins to examining the long-term socio-economic impact of COVID, understanding new vaccines and novel therapies, and preparing for future waves, the adaptability of the ECHO COVID curricula will evolve to reflect these issues.

Limitations

There are several limitations to this study. All questionnaires are self-reported and there was no control group for comparison of self-efficacy. A pre-post assessment, however, of self-efficacy scores within a subgroup of participants was able to be collected. There were differences in age group, years in practice, and number of patients assessed with COVID-19 between HCWs who registered but did not attend any ECHO sessions compared to HCWs who registered and attended at least one ECHO session. This can potentially be attributed to the larger number of participants who answered ‘unknown’ to the questions related to the years in practice and age group categories who attended ECHO sessions. There was also a larger observed proportion of HCWs without any COVID-19 patients in their clinical practice who registered and did not attend any ECHO sessions. Thus, the lack of perceived applicability to their practice may have accounted for the lower attendance in this group. Furthermore, qualitative data was collected only from five participants, though purposive sampling strategy was employed to ensure a representative sample participated. Given the tight time frame of this study, additional qualitative data collection was not feasible. Finally, due to the rapidly evolving knowledge base regarding COVID-19, knowledge testing using multiple choice or key feature questions was not undertaken.

Conclusion

This study demonstrated that a telemedicine education program (ECHO Ontario: Managing COVID-19 Patients in the Community) had a positive impact on HCWs’ self-efficacy and satisfaction. The focus group discussion revealed that knowledge was gained through sessions and explored the context of practice for HCWs during the COVID-19 pandemic in Ontario. Participants of the focus group discussion also discussed facilitators for ECHO program success, including the transition to virtual care. Even though the COVID-19 pandemic has been overwhelming and pervasive, ECHO has provided an opportunity for HCWs to gain access to rapidly evolving information, best practices, and a space for community support.

Footnotes

Acknowledgements

The authors would like to thank all who participated in the ECHO COVID-19 program as well as Evan Kirsh, Shamini Martin, Rhonda Mostyn, Sarah Tea, and Maja Zagorac for their assistance in ensuring all sessions ran seamlessly.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ontario Ministry of Health and Long-Term Care.

ORCID iDs

Q. Jane Zhao

Ani Orchanian-Cheff

Andrea D Furlan

Appendix

Appendix 2. ECHO Ontario COVID-19 Focus Group Discussion Guide

Overview: The purpose behind this set of questions is to understand the changing nature of primary care in the face of the COVID-19 pandemic, the role of ECHO, and feedback for program evaluation.

- Welcome and thanks: finishing within 1 h [by 6:00 pm]

- Introduction of moderator and note-taker

- Some housekeeping for this focus group:

Finish by 6:00 pm

We are recording this session so we can transcribe this for program evaluation and research purposes later

No right/wrong answer; hoping that you will share your experiences regarding ECHO

Please don't share any patient information and please respect the confidentiality of what's said in the group.

If no dissent, will start with the questions

[2 min] Intro- thank you all for being here. You are all here because you’ve participated in Project ECHO: Managing COVID-19 in Primary Care. Tonight, we’re interested in exploring two main topics with you: (1) how the COVID-19 pandemic has affected your practice and work, and (2) the impact of ECHO on your clinical practice.

A. [3 min] Round of intros: name, profession, location of practice, number of years in practice

B. [15 min] We’ve all had to change the way we work in response to the COVID-19 pandemic. Responding to a rapidly changing environment, keeping up-to-date with new guidelines, and living with and in uncertainty has become a new ‘normal’ and can be overwhelming. These first set of questions we’d like to explore are around this:

1) How has your practice changed since the pandemic started?

Prompts: online consults, virtual assessments, PPE, staffing, patients with COVID

2) Have you had any patients in your practice develop COVID-19?

Prompts: management of COVID-19 patients, telemedicine consults, adhering to public health guidelines

3) What do you foresee being the biggest challenges in your clinical practice and care in the coming months?

Prompts: wave 2, lack of public health guidance or enforcement, burnout

C. [40 min] Thank you for that. Switching gears to ECHO now – ECHO is an education program aimed at bridging the gap between specialist and primary care. Our sessions are delivered virtually with a didactic and case presentation component.

4) Please tell us about your experience of participating in ECHO COVID.

Prompts: how well ECHO addressed participants’ needs, COVID compared to other ECHO programs, learning from the community

5) Tell us about your experience of presenting a patient at ECHO.

Prompts: how is your patient doing now, watching others present, selection process, preparing the presentation, giving the presentation.

6) Please tell us about the ways in which ECHO has helped you navigate the pandemic.

Prompts: COVID knowledge, fast information from knowledgeable hub members, patient navigation, transition to virtual care, dissemination to others

7) What advantages do you get from ECHO over google or Up-to-date?

Prompts: didactics, real patients, building community, interprofessional hub team, transitions in care (where to refer), the clinical librarian's resources provided in each session

8) Who do you think would benefit most from ECHO?

Prompts: patients to be presented, colleagues or other clinics to attend sessions

9) Thinking back, can you tell us what's been the best thing for you about participating in ECHO?

We have about 5 min left. Is there anything that we haven't covered that strikes you as relevant (and brief)?

Thank you. You can also email us with any additional thoughts that occur to you after this conversation. Again, thank you so much for your time and your thoughtfulness.

References

Huston

Campbell

Russell

, et al. COVID-19 and primary care in six countries. BJGP Open 2020; 4: 6. 2020/09/10. DOI: 10.3399/bjgpopen20X101128.

Gray

Sanders

. A reflection on the impact of COVID-19 on primary care in the United Kingdom. J Interprof Care 2020; 34: 672–678. 2020/09/24.

DeJong

Katz

Covinsky

. Deferral of care for serious Non-COVID-19 conditions: a hidden harm of COVID-19. JAMA Intern Med 2021; 181: 274. 2020/10/27.

Tanne

Hayasaki

Zastrow

, et al. Covid-19: how doctors and healthcare systems are tackling coronavirus worldwide. BMJ 2020; 368: m1090.

Bhatia

Chu

Pang

, et al. Virtual care use before and during the COVID-19 pandemic: a repeated cross-sectional study. CMAJ Open 2021; 9: E107–E114.

Menon

Belcher

HME

. COVID-19 Pandemic health disparities and pediatric health care—The promise of telehealth. JAMA Pediatrics 2021; 175: 345–346.

Hollander

Carr

. Virtually perfect? Telemedicine for covid-19. N Engl J Med 2020; 382: 1679–1681. 2020/03/12.

Keeley

Jimenez

Jackson

, et al. Staffing Up For The surge: expanding The New York city public hospital workforce during The COVID-19 pandemic. Health Affairs 2020; 39: 1426–1430.

Villarin

Gao

McCann

. Frontline redeployment of psychiatry residents during the COVID-19 pandemic. Psychiatric Services 2020; 71: 1207.

10.

Hourston

GJM

. The impact of despecialisation and redeployment on surgical training in the midst of the COVID-19 pandemic. Int J Surg 2020; 78: 1–2. 2020/04/18.

11.

O’Sullivan

. PPE Guidance for covid-19: be honest about resource shortages. BMJ 2020; 369: m1507.

, et al. Evaluating the national PPE guidance for NHS healthcare workers during the COVID-19 pandemic. Clin Med (Lond) 2020; 20: 242–247. 2020/05/03. DOI: 10.7861/clinmed.2020-0143.

13.

Lindeman

. Canada’s Vaccine Mess, https://www.theatlantic.com/international/archive/2021/04/canada-vaccine-rollout-problems/618516/ (2021, accessed October 1 2021).

14.

Bennett

Noble

Johnston

, et al. COVID-19 confessions: a qualitative exploration of healthcare workers experiences of working with COVID-19. BMJ Open 2020; 10: e043949. 2020/12/18.

15.

Lasalvia

Bonetto

Porru

, et al. Psychological impact of COVID-19 pandemic on healthcare workers in a highly burdened area of north-east Italy. Epidemiol Psychiatr Sci 2020; 30: e1. 2020/12/18.

16.

Firew

Sano

Lee

, et al. Protecting the front line: a cross-sectional survey analysis of the occupational factors contributing to healthcare workers’ infection and psychological distress during the COVID-19 pandemic in the USA. BMJ Open 2020; 10: e042752. 2020/10/23.

17.

Zhang

Peng

, et al. Individual perceived stress mediates psychological distress in medical workers during COVID-19 epidemic outbreak in wuhan. Neuropsychiatr Dis Treat 2020; 16: 2529–2537. 2020/11/06.

18.

Sahin

Aker

Sahin

, et al. Prevalence of Depression, Anxiety, Distress and insomnia and related factors in healthcare workers during COVID-19 pandemic in Turkey. J Community Health 2020; 45: 1168–1177. 2020/09/12.

19.

Morgantini

Naha

Wang

, et al. Factors contributing to healthcare professional burnout during the COVID-19 pandemic: a rapid turnaround global survey. PLoS One 2020; 15: e0238217. 2020/09/04.

20.

Shreffler

Petrey

Huecker

. The impact of COVID-19 on healthcare worker wellness: a scoping review. West J Emerg Med 2020; 21: 1059–1066. 2020/09/25.

21.

Hlubocky

Symington

McFarland

, et al. Impact of the COVID-19 pandemic on oncologist burnout, emotional well-being, and moral distress: considerations for the cancer organization’s response for readiness, mitigation, and resilience. JCO Oncol Pract 2021; 17: 365–374. 2021/02/09. DOI: 10.1200/OP.20.00937.

22.

Barrett

Khan

Mac

, et al. Estimation of COVID-19–induced depletion of hospital resources in ontario, Canada. Canadian Medical Association Journal 2020; 192: E640–E646.

23.

Ontario taking over 5 long-term care homes following ‘gut-wrenching’ military report, https://www.cbc.ca/news/canada/toronto/covid-19-coronavirus-ontario-update-may-27-emergency-order-1.5586256 (2020, accessed October 1 2021).

24.

Martin

. COVID-19 news today: Several provinces bring in new restrictions; Ontario reports 3,216 new cases, https://www.theglobeandmail.com/canada/article-ontario-reports-additional-3216-covid-19-cases-47-deaths-as/ (2021, accessed October 1 2021).

25.

Coronavirus disease (COVID-19): Outbreak update, https://www.canada.ca/en/public-health/services/diseases/2019-novel-coronavirus-infection.html (2021, accessed October 1 2021).

26.

Poonia

Rajasekaran

. Information overload: a method to share updates among frontline staff during the COVID-19 pandemic. Otolaryngol Head Neck Surg 2020; 163: 60–62. 2020/04/22.

27.

Rosenberg

Syed

Rezaie

. The twitter pandemic: the critical role of twitter in the dissemination of medical information and misinformation during the COVID-19 pandemic. CJEM 2020; 22: 418–421. 2020/04/07.

28.

Fan

and Smith AP. Information Overload, Wellbeing and COVID-19: a survey in China. Behavioral Sciences 2021; 11: 62.

29.

Arora

Kalishman

Dion

, et al. Partnering urban academic medical centers and rural primary care clinicians to provide complex chronic disease care. Health Aff (Millwood) 2011; 30: 1176–1184. 2011/05/21.

30.

Arora

Thornton

Murata

, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med 2011; 364: 2199–2207. 2011/06/03.

31.

Furlan

Zhao

Voth

, et al. Evaluation of an innovative tele-education intervention in chronic pain management for primary care clinicians practicing in underserved areas. J Telemed Telecare 2019; 25: 484–492. 2018/07/12.

32.

Bridges

Morris

McElroy

, et al. Utility of dermatology extension for community healthcare outcomes (ECHO) sessions in the adult and paediatric population. J Telemed Telecare 2021; 27: 376–381. 2019/09/19.

33.

McBain

Sousa

Rose

, et al. Impact of project ECHO models of medical tele-education: a systematic review. J Gen Intern Med 2019; 34: 2842–2857. 2019/09/06.

34.

Project ECHO at UHN. COVID, https://uhn.echoontario.ca/Our-Programs/COVID (accessed October 1 2021).

35.

The ECHO Institute. COVID-19 Response, https://hsc.unm.edu/echo/partner-portal/programs/covid-19-response/ (accessed October 1 2021).

36.

Tashakkori

Teddlie

. SAGE Handbook of mixed methods in social & behavioral research, Second ed. Thousand Oaks (CA): SAGE, 2010, p. 912.

37.

Bandura

. Guide for constructing self-efficacy scales. In: Pajares F and Urdan T (eds) Self-Efficacy beliefs of adolescents. Greenwich, Conn: IAP - Information Age Publications, Inc., 2006, pp. 307–319.

38.

Damschroder

Aron

Keith

, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4: 50. 2009/08/12.

39.

Rimal

. Closing the knowledge-behavior gap in health promotion: the mediating role of self-efficacy. Health Commun 2000; 12: 219–237. 2000/08/12.

40.

Sandelowski

. What’s in a name? Qualitative description revisited. Res Nurs Health 2010; 33: 77–84. 2009/12/17.

41.

Colorafi

Evans

. Qualitative descriptive methods in health science research. HERD 2016; 9: 16–25. 2016/01/23.

42.

Tong

Sainsbury

Craig

. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357. 2007/09/18.

43.

Hsieh

Shannon

. Three approaches to qualitative content analysis. Qual Health Res 2005; 15: 1277–1288. 2005/10/06.

44.

Smith

Thomas

Snoswell

, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–313. 2020/03/21.

45.

Thomas

Haydon

Mehrotra

, et al. Building on the momentum: sustaining telehealth beyond COVID-19. J Telemed Telecare 2020; September: 1–8. 2020/09/29. DOI: 10.1177/1357633X20960638.

46.

Lieneck

Garvey

Collins

, et al. Rapid telehealth implementation during the COVID-19 global pandemic: a rapid review. Healthcare (Basel) 2020; 8: 517. 2020/12/03. DOI: 10.3390/healthcare8040517.

47.

The ECHO Institute. COVID-19 Response: United States, https://hsc.unm.edu/echo/partner-portal/programs/covid-19-response/us-covid19/ (accessed October 1 2021).

48.

Edison

. Telehealth and outreach in the post COVID-19 world: the MU show-Me ECHO program as a national model. Mo Med 2020; 117: 214–215. 2020/07/09.

49.

McElroy

Day

Becevic

. The influence of telehealth for better health across communities. Prev Chronic Dis 2020; 17: E64. 2020/07/18.

50.

Steeves-Reece

Elder

Graham

, et al. Rapid deployment of a statewide COVID-19 ECHO program for frontline clinicians: early results and lessons learned. J Rural Health 2021; 37: 227–230. 2020/05/13.

51.

Katzman

Tomedi

Thornton

, et al. Innovative COVID-19 programs to rapidly serve New Mexico : project ECHO. Public Health Rep 2021; 136: 39–46. 2020/11/21.

52.

Archbald-Pannone

Harris

Albero

, et al. COVID-19 Collaborative model for an academic hospital and long-term care facilities. J Am Med Dir Assoc 2020; 21: 939–942. 2020/06/22.

53.

Talisuna

Bonkoungou

Mosha

, et al. The COVID-19 pandemic: broad partnerships for the rapid scale up of innovative virtual approaches for capacity building and credible information dissemination in Africa. Pan Afr Med J 2020; 37: 255. 2021/02/19.

54.

Sockalingam

Clarkin

Serhal

, et al. Responding to health care Professionals’ mental health needs during COVID-19 through the rapid implementation of project ECHO. J Contin Educ Health Prof 2020; 40: 211–214. 2020/07/14.

55.

Agarwal

Mukerji

Laur

, et al. Adoption, feasibility and safety of a family medicine–led remote monitoring program for patients with COVID-19: a descriptive study. CMAJ Open 2021; 9: E324–E330.

56.

Heslin

Nappi

Kelly

, et al. Rapid creation of an emergency department telehealth program during the COVID-19 pandemic. J Telemed Telecare 2020; September: 1–6. 2020/09/03. DOI: 10.1177/1357633X20952632.

57.

Snoswell

Chelberg

De Guzman

, et al. The clinical effectiveness of telehealth: a systematic review of meta-analyses from 2010 to 2019. J Telemed Telecare 2021; June: 1–16. 2021/06/30. DOI: 10.1177/1357633X211022907.

58.

Zachrison

Yan

Sequist

, et al. Patient characteristics associated with the successful transition to virtual care: lessons learned from the first million patients. J Telemed Telecare 2021; June: 1–11. 2021/06/15. DOI: 10.1177/1357633X211015547.

59.

Glazier

Green

, et al. Shifts in office and virtual primary care during the early COVID-19 pandemic in ontario, Canada. CMAJ 2021; 193: E200–E210. 2021/02/10.

60.

Kisilevsky

Margolin

Kohly

. Access, an unintended consequence of virtual continuing medical education during COVID-19: a department’s experience at the university of toronto. Can J Ophthalmol 2021; 56: e18–e19. 2020/10/19.

61.

Almarzooq

Lopes

Kochar

. Virtual learning during the COVID-19 pandemic: a disruptive technology in graduate medical education. J Am Coll Cardiol 2020; 75: 2635–2638. 2020/04/19.

62.

Shah

Diwan

Kohan

, et al. The technological impact of COVID-19 on the future of education and health care delivery. Pain Physician 2020; 23: S367–S380. 2020/09/19.

63.

Forsetlund

O'Brien

Forsen

, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2021; 9: CD003030. DOI: 10.1002/14651858.CD003030.pub3.

64.

Davis

O’Brien

Freemantle

, et al. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 1999; 282: 867–874. 1999/09/09.

65.

Fetters

Curry

Creswell

. Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013; 48: 2134–2156. 2013/11/28.

66.

Caffery

Martin-Khan

Wade

. Mixed methods for telehealth research. J Telemed Telecare 2017; 23: 764–769. 2016/09/04.

67.

Detsky

Bogoch

. COVID-19 in Canada: experience and response to waves 2 and 3. JAMA 2021; 326: 1145–1146. 2021/08/24.