Sage Journals: Discover world-class research

Abstract

Background:

To meet ACGME-I (Accreditation Council for Graduate Medical Education–International) training and duty hour requirements, we converted our 3-week-long pediatric emergency medicine induction program to an eLearning program.

Objectives:

The study aimed to identify areas of the eLearning program residents perceived useful and the components that helped them prepare for clinical work.

Methods:

The qualitative study took place in a tertiary pediatric emergency department. Twenty-seven residents from family medicine, emergency medicine, and pediatric medicine participated in focus group discussions to explore how they perceived the eLearning program helped prepare them for work. The interviews were audio-recorded, and transcripts were analyzed and coded into categories and themes.

Results:

Four themes emerged from the data analysis: residents’ access to the eLearning program, instructional methods, eLearning design elements, and supplementary learning. Residents valued autonomy to control their pace of learning and use online features that matched their preferred learning styles. Design features such as the use of questions and quizzes helped stimulate learning, but attention had to be paid to the order of questions in the modules and the format of the questions. Written guidelines served as a good reference for learners and face-to-face sessions accompanying the eLearning program helped reinforce knowledge and offered opportunities to interact with faculty members to clarify questions.

Conclusion:

Systematic planning focusing on access, instructional methods, and design is essential when creating eLearning programs for residency training. eLearning programs can be enhanced by the incorporation of team-based learning and having accompanying written content to reference.

Keywords

eLearning blended learning emergency medicine residency training

Introduction

When Singapore adopted the Accreditation Council for Graduate Medical Education–International (ACGME-I) training program in 2013, new residents were posted to the pediatric emergency department (ED) every 1–2 months. Before this, residents rotated to the ED every 6 months, and we offered a 3-week-long face-to-face induction program to prepare residents for the rotation. However, it was not easy to offer the same program every 2 months. In addition, duty hour restrictions and shift work at the ED made it challenging to plan face-to-face training for residents. To overcome these constraints, we developed a new eLearning induction program to address the learning needs of the new residents.

Well-designed eLearning programs allow trainees to learn at their own pace without time and location constraints and have been shown efficient and effective for graduate medical trainees to acquire knowledge, learn skills, and improve clinical practice.^1–3 Evidence supporting eLearning has mostly compared eLearning programs with traditional teaching methods, but there is a great need to ask learners which aspects of the eLearning programs are helpful and what design features help them acquire clinical knowledge that they can apply to patient care. This will help curriculum developers and faculty members to develop more effective eLearning programs.

To identify areas of the eLearning program and design features that residents perceived useful, we employed a qualitative phenomenological approach that was exploratory in nature, conducting focus group discussions (FGDs) with residents who completed the eLearning program. The manuscript is reported consistent with the Consolidated Criteria for Reporting Qualitative Research (COREQ)-32 criteria.⁴

ELearning program

Research on eLearning in healthcare education suggests a clear course overview, well-defined learning objectives, learner assessment and feedback, technology support for learners and instructors, and course evaluation.^5–9 Our program incorporated these features, and we made a conscious effort to include components that made the eLearning modules more interactive to achieve learning objectives better.^10,11 The details of the eLearning program can be found in Table 1.

Table 1.

Main components of the pediatric emergency medicine residents eLearning program.

Main Components	Aims
1. Introductory email and video	To give an overview of the program
2. Authentic case scenarios (integrated into a virtual patient’s clinical journey where relevant); self-assessment questions; use of X-rays, photos, audio and videos	To embed learners into real clinical situations using examples from actual practice
3. Immediate feedback to self-assessment questions	To learn the rationale behind the answers
4. Summaries of the key learning pearls from clinical cases and assessment	To emphasize the key learning points
5. Quizzes in multiple formats, that is, single best answer, multiple-choice questions, extended matching, short answer questions, hotspot (learners mark a specific point on the image as the correct answer), ordering (learners provide an answer by selecting the correct order of a series of items), and multiple answer questions (learners choose more than one answer)	To assess learners’ progress
6. Links to articles and guidelines	To allow learners to do further reading if needed
7. Supplementary written content in the form of clinical guidelines	To serve as a source of information that can be referenced

Course content

A panel of pediatric emergency physicians analyzed feedback from residents and faculty who had previously attended or taught the induction program, examined the clinical case mix in the department, and identified 16 core topics in pediatric emergency medicine (PEM). More important topics were divided into shorter modules, each taking 20–30 minutes for a resident to complete. At the start of this study, the eLearning modules for five topics were still under development, so recorded lectures accompanied by PowerPoint slides were used. As multiple faculty members were invited to develop the content, there were variations in content and the delivery formats. Residents were required to complete the online modules on their own before attending the face-to-face sessions led by a faculty member, to discuss and clarify learning content.

ELearning platform

The modules were developed with the support from two eLearning experts using the Sharable Content Object Reference Model (SCORM). We used Blackboard, a learning management system, to deliver the online modules.

One month before the start of the rotation, A.T. (the principal investigator (PI)) emailed upcoming residents the overview of the program and the web link and login details to the program. Administrative matters were handled by administrative staff at the ED. Our organization’s educational IT (information technology) specialists provided ongoing technical support.

Methods

Setting and participants

The study took place at the ED of a tertiary pediatric hospital in Singapore, which attends to 170,000 children annually. The department is one of two accredited residency training sites for PEM in Singapore. A total of 119 doctors attended the eLearning program from January 2017 to January 2018 and were invited to participate in FGDs. Twenty-seven consented to participate, and they did not receive any incentives other than working an hour less in the week participating in the FGD. They were from three residency programs: pediatric medicine, emergency medicine, and family medicine. All participants completed the entire eLearning program.

Data collection

From March 2017 to March 2018, we conducted seven FGDs. Each FGD lasted approximately an hour. Each FGD consisted of residents from the three different residency programs. C.D., the co-author, recruited participants and conducted the FGDs. C.D. was well versed in technology-enhanced learning and qualitative research methods and did not have a working relationship with the participants. The two authors developed the FGD questions (Supplemental Appendix 1) with A.T. as the content expert and C.D. familiar with eLearning.¹² The questions were also guided by reviewing relevant studies on similar topics to ensure these guiding questions were not answered in the existing literature. As an internal education grant supported this project, the FGD questions were revised based on the inputs from a panel of education research experts during the grant application process. We decided to use FGDs to gather information from different participants, and the discussion among the participants and the moderator offered opportunities to explore what participants were thinking, which fits well with the exploratory nature of our study.

Data analysis

FGDs were audio-recorded, and the recordings were transcribed verbatim. Data analysis was guided by the discourse analysis approach¹³ and it was done by A.T. (PEM physician, course developer, and faculty) and C.D. (medical education researcher). They coded independently and then met to discuss any discrepancies until an agreement was reached. A system of coding was developed to identify common words or phrases that recurred throughout the transcripts. Seventy-three codes were developed through two coding cycles, during which codes were renamed or subsumed when there was overlap (Table 2). We analyzed the data iteratively throughout data collection and continued until data saturation was reached. The codes were then analyzed and grouped into categories and later into themes to reflect the essence of the content. Three rounds of analysis took place as themes emerged by identifying similarities and differences between codes and categories. Clearer patterns and meaning were identified throughout the analysis before the final themes and categories emerged. Table 3 lists the categories and themes from three rounds of analysis.

Table 2.

List of codes generated from the data.

1—instructions^a
2—preparation^a
3—introduction^a
4—time^a
5—effort^a
6—access^a
7—scope^b
8—volume^c
9—duration^b
10—coping^a
11—MCQ^b
12—videos^b
13—visual auditory mismatch^c
14—knowledge^b
15—fast-forward^c
16—clarity^b

17—repetition^b
18—unnecessary/irrelevant^b
19—recorded sessions^c
20—listen^c
21—read^c
22—variety^c
23—receiving information^c
24—slides^b
25—animation^b
26—standardization^c
27—pictures^b
28—cohesiveness/complementary^d

29—technical^a
30—navigation^c
31—concise^b
32—efficient^b
33—reference^a
34—practical^b
35—scenario^b
36—concentration^c
37—VAP^b
38—interaction^c
39—quiz^b
40—immediate feedback^b
41—rationale^b
42—focused^b
43—consolidate^b
44—authentic^b

45—decision-making^b
46—sequence^c
47—summary^b
48—printout^d
49—convenient^a
50—content^b
51—useful^b
52—learning style^c
53—synergistic^d
54—approach^b
55—application^b
56—reinforces^b
57—pearls^b
58—structured^b
59—device^a
60—giving up^c
61—streamline^c

62—crowded^c
63—flowcharts^b
64—articles^d
65—congruency^d
66—write^d
67—option to choose/autonomy^c
68—administrative^a
69—image resolution^c
70—spacing^c
71—TBL^d
72—knowledge retention^d
73—guidelines^d

MCQ: multiple-choice question; VAP: voice annotated presentation; TBL: team-based learning.

Analysis of these codes eventually led to the theme Access.

Analysis of these codes eventually led to the theme Instructional Method.

Analysis of these codes eventually led to the theme Design Elements.

Analysis of these codes eventually led to the theme Supplementary Learning.

Table 3.

Themes and categories.

Four themes	Categories (Round 2)	Categories (Round 1)
Access	Timing of Enrolment	Accessing program before rotation
		Enrolment
		Coping & time to complete
	Technical Issues	Administrative issues
		Passwords
		Access on different electronic devices
		Browser compatibility
	Unlimited Access	Engaging on the move
		Engaging during free time
		Access before exams
Instructional Method	Authentic Patient Cases	Case-based scenarios
		Authentic and practical
		Decision-making
	Interactivity with Questions/ Quizzes	MCQs with interaction
		Knowledge application through quizzes
		Consolidate knowledge
	Feedback	Immediate feedback
	Feedback	Understanding rationale behind questions
	Videos and Media Links	Videos
	Videos and Media Links	Procedures
Design Elements	Purposeful Design	Sign-posting & organization
		Recorded sessions
		Unnecessary/irrelevant content
	Learner Autonomy	Navigation
		Learning style and learners’ needs
		Ability to re-visit modules
		Option to choose how they wanted to learn
	Audio	Audio-visual mismatch
		Control over audio
		Visual transcript of audio
	Spacing	Volume of content
	Spacing	Spreading out modules
	Order of Questions	Questions interspersed within module
	Order of Questions	Question before teaching slide
	Consistency in Format	Consistency and streamlining of quizzes
	Consistency in Format	Receiving information
Supplementary Learning	Blended TBL Format	Synergism of eLearning and TBL
	Blended TBL Format	TBL improved knowledge retention
	Printouts	Writing down notes
	Printouts	Printouts
	Written Guidelines	Reference to written guidelines clarified protocols

MCQ: multiple-choice question; TBL: team-based learning.

Results

Four themes emerged from data analysis: residents’ access to the eLearning program, instructional methods, eLearning design elements, and supplementary learning. Supplemental Appendix 2 includes the participants’ quotations categorized under each theme.

Theme 1: eLearning program access

Enrolling residents and granting them early access to the eLearning program were vital to allocate sufficient time to finish reviewing important content to prepare them for work in a new environment. Residents took different amounts of time to complete the online modules, with some finishing in a week, and those without prior pediatric experience, taking up to 6 weeks. Some suggested granting residents earlier access to the modules several weeks before the rotation started.

Learners valued the breadth and depth of eLearning content and appreciated unlimited access to the modules to refer to the content when they had doubts during a clinical interaction or were preparing for exams.

Two common technical issues users encountered were related to login IDs and platform/browser compatibility. Learners from different healthcare systems had to come to the department to activate the IDs, which was troublesome when they were expected to review the online modules before the rotation started. The online program did not work on tablets or mobile phones and only worked on specific web browsers. Some shared that it would be more convenient if the program worked on the phone or iPad to review the modules during their commute.

Theme 2: instructional methods

Learners shared their preferred instructional methods. First, working through authentic clinical cases in the eLearning program helped them with clinical decision-making, which they could not acquire from reading the department’s clinical guidelines. Second, questions and quizzes embedded in clinical cases put learners in the driver’s seat, which offered them opportunities to apply the knowledge to clinical cases, consolidate ideas, summarize essential learning points, reinforce clinical pearls, and become more alert to red flags. Third, immediate feedback following questions and quizzes helped learners to understand the rationale behind decision-making. Fourth, teaching videos were perceived valuable when they demonstrated procedures step by step.

Theme 3: eLearning design elements

Residents appreciated that the content was aligned with their learning needs even though they had diverse pre-existing knowledge in PEM. Features such as providing an outline and a navigation bar allowed learners to advance or re-play certain sections based on their needs.

In general, residents did not like real-time recordings of lectures as they were lengthy with unnecessary pauses, for example, during Q & A sessions. These recordings also made it difficult for them to search for specific information within the video.

Audio control was highlighted several times with some fast-forwarding or even turning the audio off. Some found that audio helped with concentration, but others found it distracting. Transcripts of the audio were requested by those who preferred to read rather than to listen. Transcripts would also help when it was inconvenient to use audio or if the audio was unclear.

Spacing the modules over a more extended period would have helped learners manage the volume of new content they had to learn. Some felt that their pace of learning slowed down after the first week as they were exhausted by clinical shifts while adjusting to a new working environment.

Although questions and quizzes were perceived valuable, learners suggested using a consistent format through different modules, for example, single best answer multiple-choice questions. Variation in the quiz formats across modules distracted some and made it difficult to concentrate on learning. The order of questions within the modules was also important. Residents preferred to start with questions at the beginning of the modules to help identify knowledge gaps and arouse their curiosity. They also valued questions at the end of modules to test how much they learned.

Theme 4: supplementary learning

ELearning modules were more effective when combined with additional learning activities and resources such as face-to-face sessions, printouts of the online presentations, and clinical guidelines. In particular, eLearning and team-based learning (TBL) sessions were complementary. The TBL sessions helped reinforce existing knowledge gained through eLearning and allowed residents to clarify doubts with senior faculty. The deep discussions among peers also helped learners apply knowledge in clinical scenarios and identify knowledge deficiencies.

Printouts of the online modules, even as a summary, cater to different learning styles. Many preferred to write down notes on printouts while reviewing online material. The availability of clinical guidelines that were referenced and emphasized in the online modules helped learners learn the guidelines and department workflows effectively for use in daily work.

Discussion

This study highlights features that are important to include in an eLearning program for residency training and identifies the understudied aspects that need to be looked into to maximize the benefit of eLearning programs.

Given the nature of self-direction in eLearning, every effort must be made to ensure that learners can get easy access to the content whenever they want, wherever they are, for however long they need, and using whatever technological devices they have access to. Most of our residents were keen to prepare for the new rotation beforehand, despite the lack of protected time to complete the modules.¹⁴ Due to the pressure to fulfill training requirements and clinical duties, residents preferred the flexibility to choose when they could access the content. The residents, especially those new to PEM, perceived this to help them adapt to their new clinical role and make them more confident at clinical encounters. Residents wanted to control their pace of learning, and many did not want to cram a large volume of content in a short period.

Despite every effort to manage technical issues with support from educational IT experts, residents still encountered technical issues. This highlights the importance of having a team handle these issues expeditiously to mitigate the impact on learning. Given the wide range of computer platforms and browsers and the limitations of the online learning management system we used, access was limited for some users due to technical issues, which reduced their satisfaction. We learned the importance of informing learners beforehand of the required specifications of the eLearning program. For example, our program functioned best with Internet Explorer and Firefox, and the browsers had to accommodate Java and JavaScript. Administrative support is critical in engaging learners from the start, for example, to make it easier to create user IDs.

Integrating real clinical cases into the eLearning program enabled residents to work through a clinical case and answer questions to make a diagnosis, decide on relevant investigations and create a plan for management and disposition. This process is known to enhance knowledge retention by the teaching and testing effect.¹⁵ They were encouraged to refer to the clinical guidelines to solve the problems, helping them to understand department protocols and learn how to apply the algorithms to clinical practice.

The eLearning program should offer residents flexibility to cater to their different learning needs. Ensuring that eLearning content adapts^16,17 to the varying needs of learners came across in several areas. They wanted flexibility within the eLearning modules to navigate the content and focus on what they did not know. They also wanted control over audio features such as fast-forwarding or muting and requested written transcripts.¹⁸ These features allow residents to customize learning based on their preference and access the content anywhere, even during their commute or in a library. The live recordings of lectures uploaded to Blackboard were poorly received as they were difficult to navigate, taking away learners’ flexibility and their ability to learn at their style and pace.

The integration of the eLearning program with the face-to-face teaching sessions helped to enhance the value of the online content. One strength of our program was the choice to use the TBL methodology to conduct the face-to-face component. Throughout the TBL sessions, well-crafted clinical scenarios were used to replicate daily work and allowed residents to engage in decision-making while referencing the department clinical guidelines. The face-to-face TBL sessions allowed them to ask questions and seek clarification with faculty, similar to what they would do during an ED shift. Seeing the program’s relevance with actual life practice kept the learners engaged and motivated to complete the eLearning modules. In addition, the process of working on clinical problems in a team facilitated by experienced faculty contains teaching presence, social presence, and cognitive presence, the three components of the Community of Inquiry framework, which aims to encourage deep and meaningful learning.¹⁹ eLearning and TBL are synergistic educational elements that can help prepare residents for clinical work by their constructivist elements, which are helpful to achieve learning outcomes.²⁰

There is still a lot to learn about the design of eLearning modules, such as how to best space out eLearning modules, what format of questions should be used, and where to embed the questions and quizzes. The location of questions in the eLearning modules affected our residents’ experience. Some learners felt it was better to start with a clinical question before receiving teaching content, compared with receiving teaching content first and then being tested with a question in the form of a clinical vignette. Starting with questions before instruction helps to activate prior knowledge, which serves as a foundation for the new knowledge,²¹ and identify learning gaps. Learning may be enhanced when learners recognize gaps in their current knowledge and experience, after which the content that follows becomes more relevant. Questions can also be embedded throughout the online modules to break down content into smaller chunks. The format used to present questions and quizzes may also affect learning. We had initially included different quiz styles as we felt that the variety would keep learners engaged. This could have added to the cognitive load of our learners, who found the variety of quiz styles distracting and preferred consistency. Variability in length, format, and quality of content may confuse learners and make them feel lost,²² and this should apply to the style and format of questions and quizzes. Our finding indicates that further research is warranted to understand the value of the sequence of questions in eLearning.

Strength and limitation

One strength of this study is that being qualitative, we could go beyond just understanding which components of the eLearning program affected learning and could explore how and why these components worked. This adds to the literature on how eLearning programs should be designed. Although residents thought that the program prepared them for work, we need more experimental studies to determine whether the knowledge learned translates into improvements in clinical practice.

Conclusion

Systematic planning focusing on access, instructional methodology, and design is essential when creating eLearning programs for residency training. ELearning programs can be enhanced by incorporating TBL, and content must be developed to meet the needs of learners, with relevance to their daily work and the system in which they work. In addition, flexibility within the program is essential to meet different learning preferences.

Supplemental Material

sj-pdf-1-hkj-10.1177_10249079211044911 – Supplemental material for An eLearning program to prepare residents for a rotation in pediatric emergency medicine: A qualitative study

Supplemental material, sj-pdf-1-hkj-10.1177_10249079211044911 for An eLearning program to prepare residents for a rotation in pediatric emergency medicine: A qualitative study by Arif Tyebally and Chaoyan Dong in Hong Kong Journal of Emergency Medicine

Footnotes

Acknowledgements

We would like to thank Ms. Daphne Chan, Mr. Bagus Gheo Pradikta, Office of SingHealth Academy, Ms. Lim Xiao Hui, KK Women’s and Children’s Hospital, for their assistance in this research. We would like to thank all the study participants.

Author contributions

A.T. and C.D. have directly participated in the planning, execution, and writing of this article; both reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a grant from the Academic Medicine Education Institute, SingHealth/Duke-NUS, Singapore.

Availability of data

The data sets generated and analyzed during the current study are available from A.T., the corresponding author.

Informed consent

Written informed consent was obtained from the participant(s) for their anonymized information to be published in this article.

Ethical approval

Ethical approval for this study was obtained from SingHealth Centralised Institutional Review Board (CIRB Ref No: 2016/3037).

ORCID iD

Chaoyan Dong

Supplemental material

Supplemental material for this article is available online.

References

Wittich

Agrawal

Cook

, et al. E-learning in graduate medical education: survey of residency program directors. BMC Med Educ 2017; 17: 114.

Cook

Levinson

Garside

, et al. Internet-based learning in the health professions: a meta-analysis. JAMA 2008; 300: 1181–1196.

Means

Toyama

Murphy

, et al. Evaluation of evidence-based practices in online learning. Structure 2009; 66, www.ed.gov/about/offices/list/opepd/ppss/reports.html

Consolidated criteria for reporting qualitative research (COREQ): a 32-item Checklist for Interviews and focus groups —PubMed, https://pubmed.ncbi.nlm.nih.gov/17872937/?from_term=tong+coreq&from_pos=1 (accessed 29 May 2020).

Cook

Ellaway

RH.

Evaluating technology-enhanced learning: a comprehensive framework. Med Teach 2015; 37(10): 961–970.

Cook

Dupras

DM.

A practical guide to developing effective web-based learning. J Gen Intern Med 2004; 19(6): 698–707.

McGee

Kanter

SL.

How we develop and sustain innovation in medical education technology: keys to success. Med Teach 2011; 33(4): 279–285.

Clark

Mayer

RE.

E-learning and the science of instruction. San Francisco: Pfeiffer, 2003.

Mayer

RE.

The Cambridge handbook of multimedia learning. New York: Cambridge University Press, 2005.

10.

Rasmussen

Belisario

Wark

, et al. Offline eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health 2014; 4: 010405.

11.

Morgulis

Kumar

Lindeman

, et al. Impact on learning of an e-learning module on leukaemia: a randomised controlled trial. BMC Med Educ 2012; 12: 36.

12.

Krueger

RACM

. Focus groups, a practical guide for applied research. 4th ed. Thousand Oaks, CA: SAGE, 2009.

13.

Hodges

Kuper

Reeves

Discourse analysis. BMJ 2008; 337: 570–572.

14.

Brooks

Pontefract

Vallance

, et al. Perceptions and impact of mandatory eLearning for foundation trainee doctors: a qualitative evaluation. PLoS One 2016; 11: e0168558.

15.

Matava

Rosen

Siu

, et al. eLearning among Canadian anesthesia residents: a survey of podcast use and content needs. BMC Med Educ 2013; 13: 59.

16.

Cook

McDonald

FS.

E-learning: is there anything special about the “e”?

Perspect Biol Med 2008; 51(1): 5–21.

17.

Chen

Paul

RJ.

Editorial: individual differences in web-based instruction-an overview. Br J Educ Technol 2003; 34: 385–392.

18.

Dong

Goh

PS.

Twelve tips for the effective use of videos in medical education. Med Teach 2015; 37(2): 140–145.

19.

Garrison

Anderson

Archer

The first decade of the community of inquiry framework: a retrospective. Internet High Educ 2010; 13: 5–9.

20.

Davidson

LK.

A 3-year experience implementing blended TBL: active instructional methods can shift student attitudes to learning. Med Teach 2011; 33(9): 750–753.

21.

Merrill

MD.

First principles of instruction: assessing and designing effective, efficient, and engaging instruction. Educ Technol Res Dev 2002; 50: 43–58.

22.

Reid

Thomson

McGlade

KJ.

Content and discontent: a qualitative exploration of obstacles to elearning engagement in medical students. BMC Med Educ 2016; 16: 188.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.16 MB