Abstract
Background
Survival Radiology (SR) is a flagship annual full-day in-person radiology workshop targeted at final year medical students in Singapore to prepare them for internship. Previous in-person editions have consistently received positive reviews from 2014 to 2019. However, the COVID-19 pandemic necessitated a rapid online pivot for its sixth edition in 2020.
Objectives
This study aims to (a) identify key success factors of a traditional in-person medical student radiology workshop, (b) describe the rapid online pivot in 2020 and (c) to identify key success factors for online educational initiatives.
Methods
Post-workshop survey responses of SR from 2014 to 2020 were evaluated. Likert-scale data were quantitatively analysed, while free-text responses were qualitatively analysed.
Results
A total of 1248 post-workshop surveys (2014–2020 workshops) and 266 free-text responses (2020 workshop) were received from 2640 participants over the years. Progressive changes that sustained or improved participant feedback for in-person SR workshops included adoption of a case-based approach, utility of ‘live’ audience response systems and incorporation of quizzes with a favourable overall feedback rating of 4.42–4.89 from 2014 to 2019. The webinar version of SR in 2020 became the best-rated edition since inception with a rating of 4.9. Qualitative analysis of feedback from SR 2020 showed that the participants preferred the webinar model, online modes of engagement and interactivity.
Conclusion
Our experience shows that it is not only possible to successfully pivot online for such workshops, but that blended educational formats utilising online engagements supplemented by in-person activities will be well-received by ‘Generation Z’ learners even after the COVID-19 pandemic.
Introduction
Survival Radiology (SR) is an annual full day in-person radiology workshop for final-year medical students from the three medical schools in Singapore. It is an interactive workshop focusing on important and urgent radiological findings to equip future interns with essential knowledge and basic interpretation skills of common radiological studies relevant for safe practice. SR had undergone five in-person iterations from 2014 to 2019, evolving from consultant-led didactic lectures to resident-led interactive case-based workshops. The iteration in 2020 involved pivoting form a large in-person gathering of about 250 students in 2019 to a totally online webinar in 2020. This study aims to (a) identify key success factors of a traditional in-person medical student radiology workshop, (b) describe the rapid online pivot in 2020 and (c) to identify key success factors for online educational initiatives.
Traditional Model of SR
The evolving model of the in-person editions of SR was based on the Kolb cycle of experiential learning by integrating real-time audience response systems (ARS) (Poll Everywhere, San Francisco, California) into case-based scenarios.1–3 The SR programme had various subspecialty-based modules, each consisting of multiple case-based scenarios. Each case scenario provided a clinical vignette accompanied by one or more radiological images. Participants’ responses were recorded via ARS. The use of real-time ARS allowed participants to attempt cases (active experimentation), enabled a simulated clinical encounter (concrete experience) with interactive options such as a multiple-choice quiz (Figure 1), open-ended question with word cloud feature (Figure 2) and image-based interactive response (Figure 3). The real-time participant responses allowed the faculty to tailor the explanations accordingly. It also gave participants a chance to reflect on the entire cohort answers (reflective observation). At the end of the case, a summary of learning points was provided, enabling the participants to conclude each case encounter (abstract conceptualization). Multiple choice question on Poll Everywhere. Open-ended question using word cloud on Poll Everywhere. Interactive response image on Poll Everywhere.
Rapid Online Pivot for SR 2020
The COVID-19 pandemic curtailed medical student training due to strict social-distancing measures.
4
In-person lectures were temporarily halted, as were many educational activities in the hospitals. In Singapore, the most restrictions were imposed during the ‘Circuit Breaker’ period from April to June 2020,
5
which coincided with the usual SR workshop. These restrictions made it unfeasible to conduct SR with the traditional in-person model. However, in response to numerous requests for radiology teaching sessions from both local and international medical students who had their medical education disrupted during this time, the organising team put aside the initial notion of cancelling the workshop. The team began exploring alternate avenues to hold SR 2020. Multiple technical and human factors which may pose as challenges to a successful implementation were considered. During this time, work-from-home arrangements and virtual meetings were just starting to become commonplace. Hospitals and educational institutions also transitioned to videoconferencing and webinars for multidisciplinary team meetings and teaching sessions.
6
A timely collaboration with the College of Radiologists, Singapore (CRS) allowed the team to tap into their experience with hosting online workshops. Together with strong faculty support, SR 2020 (Pandemic Edition) was launched within 6 weeks of its conception as an online webinar series via a cloud-based web conferencing platform (Zoom Video Communications, San Jose, California, USA) (Figure 4). Timeline of SR 2020 Pandemic Edition from its conception to implementation.
Concerns Going Online
Can People ‘Zoom’ for an Entire Day?
The phenomenon of ‘Zoom fatigue’ emerged as the world embraced video-conferencing platforms for work, education and personal communication. Zoom fatigue is the exhaustion arising from extended virtual online videoconferencing. Factors such as excessive close-up eye gaze, cognitive load, unhealthy self-evaluation and limited mobility have been touted to result in Zoom fatigue. 7
Since the traditional SR model was a full-day workshop, the organising team mitigated Zoom fatigue by dividing the workshop into smaller components over several weekend afternoons.
The pilot session was a 1-hour webinar titled ‘Lines and Tubes on radiographs’. The second and third sessions were longer 3.5-hour sessions with three modules each (brain, spinal cord and chest for session 2, followed by abdomen, spine and limbs for session 3). Each module had 45-minute for case-based scenarios, followed by 15 minutes of moderated question and answer (Q&A) segment.
A mandated 5-minute break between each module was implemented, during which engaging interfaces such as music and space-filler PowerPoint slides were presented to break the monotony.
Anticipating More ‘Live’ Questions
Current webinar capabilities include ‘live’ chat and Q&A options where queries can be submitted in real time. With this feature, the team anticipated more queries from the audience while the session was still in progress as compared to a classroom setting. 8 The team decided to employ a larger group of faculty members to answer ‘live’ ad-hoc questions. This allowed the students to quickly clarify simple concepts without disrupting the flow of the main interactive presentation.
Anticipating Technical Issues
A dry-run with the entire team was performed 1 week before the webinar to identify and resolve any technical issue, including audio-visual clarity of the presenters and presentations, and internet connectivity problems such as lag time. The short pilot session and additional mini dry-runs prior to each session served as additional checks to pick up on any last-minute technical difficulties. Team debriefing after each webinar session reviewed the overall experience and participant feedback, with an aim to improve subsequent sessions.
Online Safety
Online safety is a major challenge when using video-conferencing platforms. There have been reports of security breaches, unwanted disruption and trolling.9,10 Measures employed to secure the webinar included the use of meeting passcodes, waiting room function to admit only registered participants, requiring authentic profiles, disabling screen-sharing option for participants and keeping meeting details private. 11
Changes from the Traditional in-Person Model
Content Modification
PowerPoint Slide Optimization
The existing SR modules were case-based scenarios with short clinical vignettes and radiological images on PowerPoint (Microsoft, Washington, USA) slides. The team ensured that all content were modified to suit a virtual platform. For example, using a darker background for better visual appeal and using legible font type, size and colour. 12 The radiological images were adjusted to the appropriate contrast and size. Stack images were animated to simulate viewing a cross-sectional radiology study. Additionally, annotations and drawing tools were used to help illustrate key concepts or structures on images, enhancing understanding and engagement.
Cancellation of Quizzes
Quizzes with prizes were an integral part and highlight of prior in-person editions of SR. However, conducting the individual and team-based quizzes was technically challenging on an online platform. Decision was made to discontinue this activity.
Continued Use of Audience Response Systems
The interactive case-based lectures with audience response system (Poll Everywhere) were easily adaptable for the online platform. It has been implemented in many previous SR workshops with good audience reception.
Creation of New Roles for a ‘Webinar’ Format
Event Host as the Main Presenter
The event host was responsible for guiding the audience and faculty members through the webinar to ensure a smooth and well-timed workshop programme. The event host focused on engaging the participants in a more informal, and at times, entertaining manner, to sustain interest and engagement. The designated host was one of the workshop directors who had an overview of the webinar programme and roles of all team members.
Event Moderator as an Advocate for the Audience
The moderator was responsible for monitoring the ‘live’ chat and Q&A discussion, consolidating recurrent themes and presenting them to the speaker during the Q&A segment for each module. The designated moderator was one of the senior faculty members with 3 years of experience in conducting previous SR quizzes.
Dedicated Administrative Team
The scale of the online webinar required a dedicated administrative team to admit participants, ensure presentations were running well and manage any acute technical issues. The administrative team was a mix of personnel from previous SR workshops and new members from the CRS who had experience hosting virtual workshops.
Non-Presenting Faculty Members
Other faculty members besides the speaker were responsible for answering questions as they arose in the ‘live’ chat and Q&A function. These faculty members also highlighted technical issues to the administrative team.
Materials and Methods
Post-Course Survey
Feedback of SR from 2014 to 2020.
*p-values <.05; **p-value <.01; ***p-value <.005.
The free-text responses in the survey forms were analysed qualitatively by inductive thematic analysis, and the responses were coded manually.
13
The codes were then categorised into different themes. The sample selection is illustrated in Figure 5. Sample selection for quantitative and qualitative analyses of SR from 2014 to 2020.
Results
The number of participants (with survey response rates in parentheses) in 2014, 2015, 2017, 2018 and 2019 were 49 (83.6%), 96 (75.0%), 118 (78.0%), 215 (33.0%) and 236 (51.2%), respectively. In 2020, the number of participants increased to 1911 (44.5%). Altogether from 2014 to 2020, we had a total of 2640 participants in SR workshops. A total of 1248 post-workshop surveys submitted by participants were included in analysis. In 2020, among 850 survey responses, there were 266 free-text responses (Figure 5). The number of participants and survey responses for each year are represented in the bar diagram in (Figure 6). Participants and survey responses: Number of participants (blue) and the number of survey responses (orange) over the years are presented. The maximum capacities of the venues are provided in the brackets next to the year on the x-axis.
(I) Evaluation of in-Person Editions of SR From 2014 to 2019
Quantitative Analysis (Table 2)
The overall course rating has been consistently more than 4 with the highest in 2017 (mean = 4.89) and lowest in 2014 (mean = 4.42) and 2015 (mean = 4.53). Significant increase in rating was observed between 2015 and 2017 (p-value < 0.005).
The response to whether the course helped participants perform their jobs effectively was also consistently more than 4 with the highest in 2017 (mean = 4.81) and lowest in 2014 (mean = 4.46) and 2015 (mean = 4.56). Significant increase in rating was observed between 2015 and 2017 (p-value < 0.005).
The rating of whether the level of the course was at the right educational level was also consistently more than 4 with highest in 2017 (mean = 4.82) and lowest in 2014 (mean = 4.33) and 2015 (mean = 4.52). Significant increase in rating was observed between 2015 and 2017 (p-value < 0.005).
(II) Evaluation of SR 2020 Webinar Format
Quantitative Analysis (Table 3)
A total of 99.5% responded favourably to ‘overall quality of the course’. Favourable responses to ‘if the course helped them perform their job effectively’, ‘if the course was at the expected level of education’ and ‘if quiz platform was easy to use’ were 99.8%, 99.2% and 99%, respectively. 86% of the participants responded that the course duration was just right, and 99.7% responded that they would recommend this course to a friend.
Survey Results of SR 2020.
Qualitative Analysis
Themes, Theme Definition, Codes and Code Definition of Survey Results of SR 2020.
Q&A: question and answer; CXR: chest radiograph; AXR: abdominal radiograph.
Overall Learner Experience and Perception of the Webinar
This theme emerged from grouping of nine codes, namely, perception on general webinar experience (50%), webinar format, host and moderator, content and case-based approach, presentation style, annotating images, clarity of images, provision of notes and webinar logistics.
The webinar was generally well-liked and well-received by the participants and many felt that it was useful, enjoyable and improved their confidence in interpreting imaging skills.
‘I believe this year’s sessions has shown that online webinars can work as well as in real life classes. Thank you’.
‘More importantly this made me more confident in my interpretation skills and encouraged me to keep trying and practicing’.
The participants enjoyed both the humorous interactions with the host and the dedicated moderated Q&A segments facilitated by the moderator.
‘Host was friendly, humorous, and time management was excellent, thank you so much to the whole team’.
‘Having moderators for Q&A is excellent because it doesn’t disrupt the speaker from delivering the content’.
The case-based approach and presentation styles were well-liked. Many participants enjoyed personalised viewing of images on their own screens and suggested to continue in this format for future editions of SR. Interestingly, there were many requests for handouts without annotations to be provided before the sessions so that the participants can annotate and make additional notes in real-time.
Engagement and Interactivity of the Webinar
This theme is a combination of codes such as general interactivity on the online format, interactivity via online tools (Poll Everywhere and ‘Live’ Q&A function) and faculty engagement.
The participants felt that it was easier to ask questions via the ‘live’ Q&A and also easy to follow the answers without disrupting the ongoing lecture. Team effort by the entire faculty to answer the ‘live’ Q&A was highly appreciated. The participants found the faculty to be engaging and encouraging. The participants particularly enjoyed the audience response system.
‘Hosting the Q&A session online is beneficial as it allows all participants to see both important and miscellaneous questions accounted for by the doctors through text, which is easier for us to keep track of than verbal explanation sometimes’.
(III) Comparison of SR2020 Webinar Format With the Previous Years In-Person Editions
Both the overall course rating and the response on whether the course helped participants perform their jobs effectively were highest in 2020 (mean = 4.9) compared to the previous years (Table 2) (Figure 7). Comparison of survey results of Survival Radiology from 2014 to 2020.
The rating of whether the level of the course was at the right educational level was the same compared to 2017 (mean = 4.82) but higher than other previous years (Table 2) (Figure 7).
Discussion
Evolution of the SR format mirrored the learning journey of the organising team as it gained experience over time. The first edition in 2014 consisted primarily of consultant-led didactic sessions with some case-based content, following a more traditional radiology tutorial style. Interactive ‘live’ audience response systems and individual quizzes were incorporated in 2015 for better participant engagement. A senior resident was also invited to participate as faculty in line with efforts to nurture future educators.
In 2017, there was a major revamp of SR as the entire workshop was led by a resident team with consultants taking up a supervisory role. The senior resident who participated as faculty in 2015 took on the role of workshop director. The workshop content was converted to full case-based sessions. A new team-based quiz was added to complement the individual quiz. The top scorers of the individual quizzes from different schools were grouped into teams to compete in a ‘grand finale’ team-based quiz which was conducted in a ‘Jeopardy’ game-show format. Paralleling this major revamp, the 2017 workshop also marked the highest course rating of the in-person editions, with the biggest improvement in the course rating between 2015 and 2017. It is likely that radiology residents are more cognisant of the needs of new generation learners in view of their own recent medical student training. Younger faculty may also impart a ‘softer touch’, bridging the usual generation gap between the audience and the faculty. Incorporating quizzes likely enhanced audience engagement while at the same time helped to nurture friendly competition, making the workshop more enjoyable.
By 2017, the workshop format was standardised with only smaller changes incorporated in the subsequent editions in 2018 and 2019. The 2017 resident team also led the 2018 edition with inclusion of new resident understudies who subsequently led the 2019 edition. In 2018, the presentations were enhanced to cater to a larger venue for a bigger audience with minor improvements in content delivery. The concluding team quiz was conducted between different medical schools in contrast to 2017 where the teams were all mixed, which further created a sense of gamification and excitement. In 2019, SR underwent a curriculum revamp with greater focus on systems-based exam-relevant clinical scenarios using guidelines from the Singapore Ministry of Health (MOH) National Outcomes for Medical Schools. The 2019 iteration retained the concluding team quiz conducted between different medical schools. These progressive changes allowed SR to maintain a favourable feedback rating of more than 4 for overall course ratings from 2014 to 2019.
SR2020 clearly surpassed the previous ratings of the in-person editions including the most successful 2017 edition. Similar to the previous iterations, participants did not have to pay to attend SR2020. Previous success factors of the in-person editions such as a resident-led team, case-based scenarios and audience response systems were easily ported to SR2020. Accessibility and ease of attending the workshop from any location, personalised viewing of the screen and images and ability to interact via Poll Everywhere, ‘live’ chat and Q&A functions appealed to tech-savvy participants.14–16 Although the well-liked quizzes of previous in-person SR editions had to be discontinued in SR2020, introduction of new players such as a host, moderator and utilization of webinar tools such as ‘live’ chat and Q&A functions contributed to its success.
Poll Everywhere has been one of the main components in our workshops leveraging on audience interactivity since 2015. It was easy to tap on our experience incorporating Poll Everywhere into the presentations for the webinar version. Poll Everywhere is a type of audience response system allowing active and ‘live’ audience participation in a fun and engaged manner by using a variety of interactive features (Figures 1–3).17–19 Multiple choice question (MCQ) format may be better suited for difficult questions or new content where a prompt could help the participants to answer. 20 On the other hand, open-ended word cloud can add in a layer of challenge and motivate critical thinking for easier questions and concepts.20,21 In the context of radiology, interactive response images are more suited to directly identify abnormalities directly on radiological images. Usage of Poll Everywhere can be anonymous but at the same time allows participants to visualise answers from their peers in real time which enables reflective observation.
Leveraging on webinar capabilities of the ‘live’ chat and Q&A functions also increases audience participation. The participants may find it less intimidating and be less hesitant to put forward queries and doubts as there is a room to accommodate for more questions on an online platform. It would be beneficial to provide the participants with all the questions and answers at the end of the session for them to look through at their own pace, as it could be difficult to follow-up on all questions asked. The highlight of our webinar was the team effort and the presence of the entire faculty answering these ‘live’ questions. A total of 245 questions were answered in real time during all the three parts of the webinar over 16 h which approximated to one question being answered every 4 min. This feature of ‘live’ online panel providing real-time answers for the audience was a great enabler for a personalised experience and may not have been feasible in a traditional in-person setting. At the end of each session, moderated verbal answers or summaries of recurring theme of questions were presented by the tutor.
Unlike the in-person model, the scope of personal interaction offered on a webinar model is minimal, but there are other ways to improve personalisation and enhance participant engagement and interactivity. Whilst a moderator acts as an audience advocate sieving through the live-stream of questions to highlight recurrent themes or important queries to the tutors, an engaging host can add a touch of humour and keep the sessions lively. Both the host and moderator roles were separately defined and well-received by participants.
Irrespective of advanced functions and software available, the driving force of all interactions is the faculty itself. The time taken to building a strong and dedicated team of radiology resident educators was paramount to the success of previous as well as current editions of SR, and the importance of nurturing the next generation of educators during residency cannot be overstated. 22 For SR, some residents progressed from participant to faculty and eventually to workshop director over the years. Incremental yearly teaching experience coupled with mentoring by more experienced senior faculty allowed SR to continue to generate a strong team of motivated and enthusiastic resident faculty. This is reflected by the consistent positive feedback by participants on faculty engagement.
Summary Table for ‘Learning Points’ and Their Rationales.
IOCM: iso-osmolar contrast medium.
Limitations
The quantitative analysis is based on learner perception and how the participants reacted to the workshop experience (Kirkpatrick evaluation level one), and the level and impact of learning were not assessed. Certain factors such as humour by the host and likability of instructors are difficult to measure and have limited generalisability to other educational contexts.
Conclusion
An online pivot can be intimidating, especially if one is used to prior successful in-person engagements. However, an online format can be successful and appeal to post-millennial learners. Not only can prior interactive elements be sustained, one can even engage learners at a level beyond what is achievable during traditional in-person sessions while reaching out to a far larger audience. Image-intensive specialties (e.g. radiology, dermatology, pathology, etc.) may be uniquely suited for online versions due to the inherent advantages of viewing high-quality images on personal devices. It is likely that blended learning formats (both online and in-person, synchronous and asynchronous) will be useful ways of engaging the Generation Z’ learner in the years ahead.
Footnotes
Acknowledgments
N/A.
Author Contributions
All authors of this paper have directly participated in the planning, execution and writing of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Availability of Data
The datasets generated for the current study have been obtained from the post-course survey responses of the participants via Google forms
Ethical Approval
N/A.
Informed Consent
N/A.
Trial Registration (where applicable)
N/A
