Abstract
Introduction:
The Blended Quality Improvement (QI) Workshop is a programme designed to provide learners with basic tools and techniques in QI using a combination of e-learning and face-to-face classroom interactions. To adapt to the coronavirus disease 2019 (COVID-19) situation for safe social distancing since the end of January 2020, SingHealth Duke-NUS Institute for Patient Safety & Quality (IPSQ) suspended all face-to-face workshops. However, the need for effective QI training remains.
Objective:
The aim was to develop an effective virtual interactive Blended QI Workshop through videoconferencing.
Methods:
The workshop was redesigned by reviewing the curriculum and incorporating the use of interactive functions. The new workshop took place virtually via a video conferencing platform and incorporated the use of interactive polling, game, videos and case study. A total of 23 learners attended the two pilot runs that took place on 28 April 2020 and 22 May 2020. A survey assessed learners’ reactions and satisfaction with the training and preference in using video conferencing tools during disease and non-disease outbreak situations, while the quizzes assessed learners’ learning in QI knowledge. The results on the improvement in QI knowledge were compared to the original blended e-learning and face-to-face workshop (n=63) that took place before the COVID-19 pandemic.
Results:
The gain in QI knowledge from virtual and from face-to-face Blended QI Workshops was statistically significant (p<0.01, n=86). However, there was no statistically significant difference in the gain in QI knowledge with both content delivery approaches (p=0.13). All 23 learners agreed that the video conferencing platform was convenient, 20 (87.0%) felt that the platform was effective in delivering content and 18 (78.3%) were keen to use video conferencing tools for QI workshops during disease outbreak situations. Remarkably, a substantial number of learners (n=11; 47.8%) preferred the use of interactive video conferencing to supplement face-to-face classroom-based Blended QI Workshops for non-disease outbreak situations.
Conclusion:
The virtual interactive Blended QI Workshop was as effective as the face-to-face Blended QI Workshop based on learners’ views and gains in QI knowledge post workshop. The success of virtual QI workshops makes them a new norm in the post-COVID-19 environment.
Keywords
Introduction
The coronavirus disease 2019 (COVID-19) crisis triggered the need for educators to rethink instructional choices to sustain education amid a pandemic that restricts social contacts because of safe distancing measures. The SingHealth Duke-NUS Institute for Patient Safety & Quality (IPSQ) conducts quality improvement (QI) workshops to uplift and upskill staff capabilities in QI. The IPSQ Blended QI Workshop is a training programme designed to provide learners with the basic QI methodology and tools via a combination of e-learning and face-to-face classroom facilitation (Figure 1).
Curriculum of Blended Quality Improvement (QI) Workshop
The identification of Singapore’s first positive case of COVID-19 on 23 January 2020 triggered the suspension of the Blended QI Workshop in SingHealth. 1 It instantly disrupted the norm of learning in a physical classroom setting. In response to safe distancing measures, virtual delivery of workshop – a new paradigm for QI education – was explored. The IPSQ QI faculty embarked on the redesign of the workshop to substitute face-to-face learning.
Searches across the publication literature platforms did not reveal a precedent in the delivery of health-care QI training with the use of video conferencing. Hence, the curriculum and delivery format was redesigned based on insights gathered from other non-health-care QI virtual education settings and experiences, with adaption to the organisational context.
We considered teaching, cognitive and social presence to maximise the potential success of the virtual workshop. Each presence played its distinct role, and intertwined with one another to promote the success of online education. 2 Teaching presence was especially vital because of its central role in balancing cognitive and social presence. 2 It was defined as the ability of the teacher to be aware and receptive of the learners’ physical, mental and emotional state in the context to their learning environment, exhibiting self-knowledge, trust and compassion while connecting to learners to achieve meaningful and effective learning outcomes. 3 Second, cognitive presence referred to the extent of learners’ ability to construct and confirm meaning through performing cognitive tasks such as thinking, reasoning and analysing information to attain higher-order knowledge and application.2,4 Lastly, social presence was defined as the learners’ ability to participate and project their true personalities via the medium of communication used. 2
The components of teaching presence (instructional design, discourse facilitation and direct instruction) of the face-to-face Blended QI Workshop were explored for the redesign of the virtual workshop.2,4 Instructional design focused on analysis to design and develop the workshop before its implementation and subsequent evaluation. In contrast, discourse facilitation and direct instruction focused on the interaction and engagement with the learners and responding to learners’ questions and feedback promptly, motivating them to achieve higher-order learning and knowledge (cognitive presence).2–5 Considerations for collaborative learning include functionalities such as video, audio and interactive functions of the virtual platform, that helped to promote social presence for the learners. 2
This study examined the effectiveness of the redesigned health-care QI workshop using a video conferencing platform to deliver health-care QI training.
Methods
The redesign of the workshop was carried out via the essential phases (ADDIE: analysis, design, development, implementation and evaluation) of systematic instructional design to increase instructional efficiency and promote student learning. 5
In the analysis phase, the target audience, training and instructional materials were analysed to identify the learning goals and outcomes for the targeted learners. It was essential for the learning objectives from face-to-face learning to be achieved in the redesigned workshop.
Subsequently, in the design phase, the team identified the following key points to work out the curriculum and instructional strategies to accommodate distance education:
The tangible aims of the programme and the specific goals for each topic to be achieved;
The impact of the selected platform to deliver the programme without compromising learning outcome and experience; and
The technical knowledge and support required for the faculty and learners to allow smooth transition to the virtual workshop.
The IPSQ QI faculty adopted the video conferencing platform; Zoom, to substitute classroom teaching, as it enabled live viewing, interactions and group discussion functions for activity-based learning. The functionality of the platform was tested with prototypes, for example teaching notes, interactive quizzes, polls and group discussions. The video conferencing platform was selected based on its functionality to achieve the learning objectives of the workshop. In contrast, e-learning and recorded e-lectures were not selected based on the lack of interactive functionality which was required in the workshop.
Finally, in the development phase, the curriculum delivery format was transformed. The engagement tools to be used for activities during the workshop was set up, and outcome indicators to determine the effectiveness of the training were identified. During the registration phase, an email was sent to the learners one week before the commencement of the workshop. It provided access to open school e-learning materials and pre-reading materials to promote self-directed learning for learners. A pre-workshop package consisting of training handouts, activities template and an instructional Zoom guide was also sent to the learners one week before the workshop to facilitate their preparation for the training.
The duration of didactic teaching was reduced compared to the face-to-face Blended QI Workshop. This was replaced with an increased amount of interactive activities such as case study discussion, polling questions, games and videos to maintain learners’ attention span and to facilitate communication among learners and faculty via the virtual space. The first two pilot runs for the virtual interactive Blended QI Workshop were implemented and evaluated, with the class size capped at 15 learners per class.
Outcome indicators
The following outcome indicators and data were collected to study if the selected training approach could meet the learning objectives and if it was acceptable to the learners:
Virtual workshop completion rate;
Learners’ demographics and reactions to the new teaching approach via a post-workshop evaluation survey;
Learners’ gain in QI knowledge via pre- and post-workshop QI quizzes; and
Completion of learners’ QI projects after the workshop (a time frame of 6–12 months was taken into consideration for a typical project to complete).
The number of learners who stayed through the entire session of the virtual workshop, reflected by the virtual workshop completion rate, was assessed to gauge learners’ interest and attention span in the workshop. 6 Learners’ responses via the post-workshop evaluation survey were used to analyse the relevance and the strengths and weaknesses of the training. 7 In addition, the learners’ quiz scores were used to relate if the training met the learning objectives.
Workshop evaluation survey
A workshop evaluation survey (Figure 2) was designed to assess learners’ reactions to the use of video conferencing to deliver the QI curriculum and their learning preference during disease and non-disease outbreak situations. The survey consisted of closed and open-ended questions. Closed-ended questions were interpretative and based on either a five- or three-point Likert scale. Questions to assess learners’ reactions to the workshop used a five-point Likert scale that ranged from ‘strongly agree’ to ‘strongly disagree’. The remaining closed-ended questions to assess learners’ preference on the adoption of video conferencing for learning in disease and non-disease outbreak situations used a three-point Likert scale (‘as a substitute’, ‘as a supplement’ and ‘not at all’). Percentages and the mean rating of the learners’ responses were calculated for the evaluation of all closed-ended questions. Open-ended questions were asked to gather qualitative information to complement the closed-ended feedback.
Virtual interactive Blended QI Workshop evaluation survey form
QI quiz design
The pre- and post-workshop QI quizzes (Figure 3) have been administered since October 2019 when the face-to-face Blended QI Workshops were still ongoing. They consisted of the same set of 10 theory-based questions which revolved around the QI concepts and tools that the learners would acquire through self-reading and attending the workshop.
QI quiz questions (choices in bold orange are the correct answers)
Pre- and post-workshop QI quiz
Non-parametric, Wilcoxon and Mann–Whitney U-tests were used to compare between pre- and post-workshop QI quiz scores and virtual versus face-to-face workshop QI quiz scores, respectively, using IBM SPSS Statistics for Windows v22.0 (IBM Corp., Armonk, NY; Table 2). The Wilcoxon test was used to determine if the differences between the pre- and post-workshop QI quiz was of statistical significance. Comparing pre-workshop QI quiz scores between virtual and face-to-face Blended QI Workshops with the Mann–Whitney U-test determined whether there was a statistically significant difference in learners’ baseline knowledge. On the other hand, the comparison of post-workshop QI quiz (Table 2) and the gain in QI quiz score between virtual and face-to-face Blended QI workshops with the Mann–Whitney U-test (Table 3) determined whether there was a statistically significant difference in the gain of learners’ knowledge after attending the workshop.
Results
Learner demographics
Out of the 23 learners, 17 (73.9%) doctors, 4 (17.4%) nurses, 1 (4.3%) administrator and 1 (4.3%) pharmacist took part in the virtual interactive Blended QI Workshops. Based on learners’ self-assessment on their knowledge and experience with QI before the start of the virtual workshop, all 23 (100%) learners stated that they lacked QI-related experience and possessed no or some QI knowledge.
Didactic teaching and QI tools application ratio
The entire training lasted from 8:30 am to 4:30 pm (seven hours, excluding lunch and break time). Out of the nine topics in the 7-hour workshop, 2.25 hours (32.1%) were spent on didactic teaching, averaging 15 minutes per topic. The remaining time (4.75 hours; 67.9%) was spent on activities such as case study discussion, polls, videos and game to allow learners to grasp the concepts in a fun and interactive manner.
Virtual workshop completion rate
Video participation was enabled for all learners right from the start of the workshop, and all 23 learners stayed through the entire training. Therefore, the virtual workshop completion rate for the two runs of virtual interactive Blended QI Workshop was 100%.
Learners’ reaction to virtual interactive Blended QI Workshop
From the responses gathered from the workshop evaluation (Table 1), majority of the learners were satisfied with learning QI knowledge via video conferencing. They agreed that:
It was convenient to access the learning platform (n=23; 100%; mean rating: 4.48/5);
They felt engaged during the session (n=22; 95.7%; mean rating: 4.26/5);
The platform was effective in delivering content (n=20; 87.0%; mean rating: 4.13/5); and
They felt comfortable raising questions during the session (n=19; 82.6%; mean rating: 4.17/5).
Learners’ Feedback on Video Conferencing Approach (n=23)
As a side note, only one (4.3%) learner agreed that he/she felt distracted during the session in the negatively worded survey question. The scale of this negatively worded survey question was reverse coded, with a mean positive rating of 3.65/5 (Table 1).
Learners’ learning preference in disease and non-disease outbreak situations
Based on their responses, 18 (78.3%) learners were keen to use video conferencing as a learning tool to replace the physical classroom during disease outbreak situations, four (17.4%) learners preferred using it as a supplementary tool and one (4.3%) learner did not recommend it at all. In addition, the majority (n=11; 47.8%) felt that video conferencing could be a supplementary learning tool in non-disease outbreak situations, and nine (39.1%) learners agreed that it could replace the physical classroom, while only three (13.0%) learners did not recommend it.
Many learners who agreed that video conferencing could replace the physical classroom during disease and non-disease outbreak situations felt that it was a convenient and effective learning tool regardless of the situation. It helped in disease outbreak situations, allowing learners to adhere to safe distancing measures while learning. Some learners preferred interacting with fellow learners in a physical classroom to minimise distraction. However, they also agreed that video conferencing could supplement learning in non-disease outbreak situations.
Learners’ feedback and suggestions for improvement
A total of 17 suggestions for improvement were gathered from learners and were categorised. Six (35.3%) on improvements to the technical issues of the virtual platform, and three (17.6%) were on shortening training duration. The remaining points were on increasing interaction opportunities (n=2; 11.8%), having enthusiastic participants (n=2; 11.8%), removing background music during assignment practices (n=2; 11.8%), having faculty provide clearer instructions (n=1; 5.9%) and having the flexibility to allow learners to disable the video function (n=1; 5.9%).
In contrast, 34 positive comments were gathered from the learners on the areas of satisfaction of the workshop. The comments were categorised, with 14 (41.2%) being on the effective training approach and 10 (29.4%) on the convenient platform. The remaining comments were on overall satisfaction (n=4; 11.8%) and satisfaction with the training activities (n=3; 8.8%), logistic arrangement (n=2; 5.9%) and trainer effectiveness (n=1; 2.9%). Some of the learners’ comments were ‘there are multiple ways to raise questions and provide feedback’, ‘the session was interactive and engaging’ and ‘I like the online polls and games’.
Pre- and post-workshop QI quiz score comparison
All 23 learners who attended the virtual interactive Blended QI Workshops attempted both pre- and post-workshop quizzes successfully.
There was a positive increment in QI quiz scores for both virtual (62.2% to 81.7%) and face-to-face (56.8% to 82.5%) Blended QI Workshops. The difference between pre- and post-workshop QI quiz scores was statistically significant for the virtual Blended QI Workshops (p<0.01, n=23), as well as the face-to-face Blended QI Workshops (p<0.01, n=63) via Wilcoxon test (Table 2).
Statistical analysis of virtual and face-to-face pre- and post-workshop QI quiz scores
QI: quality improvement; SD: standard deviation.
There was no statistical significant difference in the pre-workshop QI quiz scores between virtual (n=23) and face-to-face (n=63) Blended QI Workshops using the Mann–Whitney U-test (p=0.076), ensuring that learners’ baseline knowledge was not statistically significant prior to attending the workshop (Table 2).
Comparison of post-workshop QI quiz scores between virtual and face-to-face Blended QI Workshops via the same statistical analysis method reflected that it was not statistically significant (p=0.464; Table 2). In addition, comparison of the gain in quiz score between virtual and face-to-face Blended QI Workshops was also not statistically significant (p=0.13; Table 3). We further concluded that the gain in quiz scores of learners attending the virtual interactive Blended QI Workshop was on par with those attending the face-to-face Blended QI Workshop.
Statistical anaylysis on the gain in post-workshop quiz score between virtual and face-to-face Blended QI Workshops
Discussion
In our global literature search, this is the first publication to date on the use of interactive video conferencing for a health-care QI workshop. Our results showed that it was effective to conduct the health-care QI workshop virtually through video conferencing. The virtual interactive Blended QI Workshop stimulated vigorous active learning, involving learners in the learning process with the problem-based case study, polling questions, videos and self-direct learning materials introduced, similar to other studies in other domains.2,4,8 Learners’ survey responses reflected that despite the distance, they were satisfied with the interactions and engagement to share their experiences and suggestions through the audio and video functions, similar to other studies in other domains. 9
The redesign of our virtual interactive Blended QI Workshop incorporated the three critical elements (teaching, social and cognitive presences) for successful online education. To promote teaching presence, we selected and exploited the full functions of the virtual platform, which included live viewing, interactions and group discussion functions for activity-based learning to maximise active interactions and engagement. The faculty was able to connect with the learners using the functions of the video conferencing platform. It helped to encourage the learners to participate in the discussion and activities, thus achieving the learning objectives. Social presence was enhanced through encouraging collaborative learning among learners through the platform. They were encouraged to share details of their QI projects with one another, generating discussion. Cognitive presence was promoted through facilitating trainers and learners to engage in active interactive discussions. Learners were encouraged to raise questions or feedback on any activities or training content via the multiple feedback channels of the platform while trainers were delivering the training. The prompt feedback loop during the workshop encouraged both learners and trainers to share experiences, enhancing the depth of learning.
The simultaneous use of images, videos with narration, animation and text with varying font size and colours to highlight key points were incorporated into our learning materials to complement teaching, cognitive and social presences. Similar to other studies in other domains, the addition of appropriate multimedia into the training content enriched teaching and improved learners’ satisfaction with the learning experience.8,10
Out of the 23 respondents, almost all learners (n=22; 95.7%) generally felt that the sessions were engaging and suggested spending even more time on videos, interactive activities and discussion. Our survey responses showed that two-way communication using the videos and audio-enabled video conferencing platform aids the learning experience. Previous meta-analyses of the effectiveness in telecourses revealed that two-way communication was the best mode of interaction among trainers and learners, and this was incorporated into our workshop. 11
Though the survey questions asked in the face-to-face and virtual interactive Blended QI Workshops were different to a large extent, we identified a single question on learners’ overall satisfaction with the face-to-face Blended QI Workshop to compare with learners’ reactions to the virtual Blended QI Workshop. The overall satisfaction rating of the face-to-face Blended QI Workshop used a four-point Likert scale, which consisted of ‘needs improvement’, ‘partially meets expectations’, ‘meets expectation’ and ‘exceeds expectation’. Out of the 63 respondents, majority of the learners (n=60, 95.2%) were satisfied with the face-to-face Blended QI Workshop overall. Though there were challenges in performing direct comparisons of the survey questions between the face-to-face and virtual Blended QI workshops, we could infer that the learners were satisfied with both the face-to-face and virtual Blended QI Workshops based on their responses in the different parts of the survey.
Earlier research in the 1990s where technology was less advanced suggested that the learning outcomes of distance learning were similar to those who attended traditional classroom.12–15
Results from our workshop evaluation survey offered compelling evidence that learners preferred using video conferencing tools for the Blended QI Workshop during disease outbreak situations. Even though a substantial number of learners (n=11; 47.8%) highlighted that video conferencing could be a supplement to blended QI physical workshops in non-disease outbreak situations, an appreciable number (n=9; 39.1%) of learners thought that video conferencing could completely substitute the physical classroom, even in non-disease outbreak situations, as it was convenient and as effective as attending a physical workshop. There was already strong demand from learners with adequate technology knowledge eager to participate in the virtual interactive Blended QI Workshop when the face-to-face workshop was suspended. With a range of readily virtual platforms available to evaluate and select, our virtual workshop was conceived in 1 month.
Besides providing two-way communication and immediate feedbacks among learners and faculty, the upgraded software and high-speed network from the current advancement in technology allowed faculty to collect data such as evaluation and quiz results easily in real-time. 2 Additional functionality of the virtual platform included multiple feedback channels, which were appreciated by more reserved learners who prefer to post questions in a subtle manner.
Challenges such as technical operating issues occurred during the workshop, temporarily disrupting teaching and learning flow. However, these issues were often resolved promptly, and learners were able to participate throughout the workshop without much disruption. The majority of learners in the virtual interactive Blended QI Workshop were from the ‘millennium generation’ – those born between 1980 and 2000. 16 They possessed high level of technology literacy. Hence, technical issues were not the main challenge that would affect learning outcomes.16,17 Moreover, the virtual platform was user-friendly, and the step-by-step Zoom instructional guide provided aid in the navigation for all learners. Learners’ learning attitude, interactive and engaging learning materials and platform, as well as the environment where the learners logged into the virtual workshop were the critical factors that influenced the learning outcomes. 17
Statistical analysis of the pre- and post-workshop QI quizzes indicated that our Blended QI Workshops delivered either face-to-face or virtually resulted in a gain in QI knowledge, meeting the learning objectives stated in the design of the curriculum. The gain in QI knowledge in learners from the virtual interactive Blended QI Workshops was on par with that of the learners from the face-to-face Blended QI Workshops. The results suggested that the different delivery approaches could be used alternatively, even after the COVID-19 pandemic has stabilised. Likely, the new norm of the QI workshop will now include a virtual form post pandemic.
As the faculty played an active role in facilitating learners’ discussion during the virtual workshop, the virtual interactive Blended QI Workshop started with a maximum class size of 15 learners to ensure the quality of teaching. However, eliminating the limitation of space and distance with the use of virtual platforms allowed an expansion in class size, enabling more learners to benefit from a single training session. 18 Therefore, the class size for our future virtual interactive Blended QI Workshops can be expanded to meet the growing demand of virtual health-care QI curriculum in both disease and non-disease outbreak situations. The ‘breakout rooms’ functionality of the teleconference platform to facilitate small group discussions can maintain training quality with the expansion of class size. Monitoring learners’ completion of QI projects and its outcomes would be a useful evaluation of their ability to apply what they have learnt in their job, bringing beneficial outcomes at an organisational level. 19
Conclusion
This study showed that the virtual interactive Blended QI Workshop can achieve a significant improvement in learners’ QI knowledge, despite the lack of a physical classroom. While it demonstrated that the virtual approach was useful for disease outbreak situations, learners felt that it could also be used in non-disease outbreak situations. The new norm of QI workshops will likely include a virtual form post pandemic.
Footnotes
Acknowledgements
We would like to thank the learners for providing their feedback to us.
Authors’ contributions
S.C., W., Y.T. and Z. researched the literature and conceived the study. D.F. and Y.T. performed data analysis, and Y.T. wrote the first draft of the manuscript. K.H. provided guidance in the development of this manuscript. All authors reviewed and edited the manuscript, and approved the final version of the manuscript.
Availability of data and materials
The data sets generated and/or analysed during the current study are available from the IPSQ shared document hosted on the SingHealth Intranet platform.
Conflict of interest
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Ethical approval
This study was exempt from SingHealth CIRB based on Category 2 – Anonymous Educational Tests, Surveys, Interviews, or Observation. SingHealth Centralised Institutional Review Board A (CIRB A), CIRB Reference number: 2020/2768.
Funding
The authors received no financial support for the research, authorship and/or publication of this article.
Informed consent
Written informed consent was obtained in the form of survey response and quiz attempts from all subjects before the study. Subjects were informed that the information in the workshop evaluation survey and QI quizzes would be used to study the workshop outcome.
