Abstract
Background:
Team-Based Learning (TBL) is an instructional strategy that embraces the concept of a flipped classroom. At our institution, the Clerkship in Pediatrics utilizes TBL sessions to deliver curricular content rather than lecture format. In 2016, matriculating medical students were given iPads for the purpose of transitioning to electronic delivery of curricular materials in the pre-clinical phase of their study. In 2017, the Clerkship in Pediatrics was tasked with investigating methods to integrate iPad use into clinical education.
Activity:
We employed the web-based platform Examplify to deliver test questions for TBL sessions over the span of 1 academic year (2018-2019). Curricular content was converted to Examplify for half of the TBL sessions and team performance for the sessions was compared to those administered traditionally. Students participating in the course were surveyed regarding their satisfaction with the 2 formats using a 15-item survey with a 4-point rating scale and open-ended questions.
Results and discussion:
Integration of technology into TBL sessions was met with mixed results. Performance on the eTBL sessions was significantly higher for 2 of the 6 sessions: the Abdominal Pain (90.2% vs 84.1%, P = .04) and the Toxicology (85.6% vs 79.4%, P = .03) sessions. A majority of students felt that the eTBL sessions facilitated multiple learning styles, promoted discussions, and nurtured different learning styles. However, students also felt that the electronic sessions were more cumbersome and difficult to navigate. Future research comparing the 2 modalities using a more comprehensive integration of media content, such as video and audio files, will further inform the success of this endeavor.
Introduction
Undergraduate medical education (UME) has traditionally focused on 2 main tenets; didactic teaching of basic science in the classroom followed by content application in the clinical setting. Today’s learners, aka “millennials” or “Gen Y” are digital natives. Typically born between 1980 and 2000, they have never known a time without technology. These learners have shorter attention spans, prefer graphics to text, and have been using web-based technology to facilitate their learning since primary school. Over the course of the past 10 years, UME programs have begun to embrace technology-based education. 1 Additionally, UME programs have been tasked with developing curricula that promote the concept of “self-directed learning,” 2 replacing lectures with interactive teaching such as problem-based group work and team-based learning. 3
Team-based learning (TBL) is an instructional strategy developed by Larry Michaelson within the world of business education in an attempt to address a problem with increased learner enrollment. TBL embraces the concept of a “flipped classroom” 3 wherein the learners come to the dedicated in-class time prepared to problem solve and work, rather than observe a didactic lecture. The emphasis of TBL is content application rather than content acquisition. Studies evaluating the effectiveness of TBL in medical education have demonstrated positive learning outcomes, increased learner engagement, communication, teamwork, and problem-solving skills. 3
Likewise, integration of medical technology into the medical learning environment is crucial to the success of an academic program. Electronic health records (EHR) have become fully embedded into the practice of medicine and web/mobile device-based decision-making tools have been instrumental in improving the quality of healthcare. 4 Telemedicine, simulation and video-based learning have also become intrinsic to medical education and the delivery of healthcare. Tablet computers such as the iPad have demonstrated added educational value in both the clinical and classroom setting.4-9 Several studies of curricula in undergraduate medical education have described improved learner engagement and increased learner retention when iPads are used in the pre-clinical environment. 4 Ease of use and portability make table computers a highly desirable tool in the clinical learning environment as well. Expansion of access affords ready availability to the EHR, prescription and order writing, radiology and laboratory review, and patient education materials. Quick access to mobile health apps, e-textbooks, and medical databases provide opportunities for real-time evidenced-based point-of-care clinical decision support. Students have also reported time management, productivity, and organizational benefits associated with iPad use in the clinical setting.8,9
In 2014, Renaissance School of Medicine at Stony Brook University launched the LEARN (Learner-focused, Experiential, Adaptive, Rigorous and Novel) curriculum, comprised of 3 phases: (1) a 18-month Phase I (Foundational Phase), (2) a 12-month Phase II (Primary Clinical Phase), and (3) a 16-month Phase III (Advanced Clinical Phase). 10 In response to increasing evidence across undergraduate medical curricula that iPads add high value to the educational experience, matriculating medical students at our institution were given iPads with a migration of all Phase I curricular materials to an electronic platform in 2016. When this class of students transitioned to the clinical learning environment (Phase II of the LEARN curriculum) in the 2018-2019 academic year, our institution sought to evaluate the educational benefit of iPad use in several core clerkships and asked clerkship directors to consider ways to leverage their use to enhance clinical learning. As TBL had already been fully integrated in the Clerkship in Pediatrics, we endeavored to deliver the session content using iPads and the web-based platform ‘Examplify’ by Examsoft (Dallas, Texas). In this study, we describe the integration of technology into clerkship TBL sessions and assessment of student performance with the alternate content delivery modality.
Methods
The Clerkship in Pediatrics at our institution utilizes a modified TBL format as a modality to deliver curricular content relevant to both clinical education and preparation for the National Board of Medical Examiners (NBME) Subject Exam in Pediatrics. Students in the clerkship are organized into teams of 3 to 4 people for the entire length of the course. Team assignments are formed by the clerkship director such that each team has an equal mix of students who had begun their clinical rotation on the inpatient and ambulatory portions of the clerkship. Before each session, students are assigned mandatory reading materials to prepare for the TBL. The session is comprised of a readiness assurance process (RAP) consisting of a short test reflective of concepts covered in the pre-course assignment as well as a group application activity. The RAP traditionally includes an Individual Readiness Assurance Test (IRAT) followed by a Group Readiness Assurance Test (GRAT), both contributing to the individual’s grade during the course. We modified the TBL structure such that only the GRAT is used in the RAP, with teams working together to answer the test questions immediately after starting the session. We did this to shorten formalized learning sessions and allow for more time in the clinical learning environment. Additionally, we did not expect students to spend too much time at home to prepare for the TBLs due to the significantly large clinical workload that is intrinsic to the Clerkship in Pediatrics. Followed by the GRAT, team questions are reviewed by a faculty facilitator to clarify uncertainties and disputes and provide added experiential teaching. The last element of the TBL session is a group application activity that requires students to work as a team using course content to tackle more challenging clinical questions. Based on student feedback to the curriculum at our institution, students prefer TBLs to be low stakes or non-graded activities. Therefore, TBLs are formative exercises in our clerkship and as such performance in these sessions does not contribute to the students’ final grade.
The LEARN Phase II curriculum is divided into quarters with each quarter split into two 6-week blocks. Approximately 130 to 140 students rotate through the Pediatrics clerkship every academic year in groups of 10 to 12 per 6-week block. All students enrolled in the clerkship participated in the TBL sessions and were eligible for enrollment in this study. Rather than randomize students to one specific TBL format, each 6-week Pediatrics clerkship block of the 2018-19 academic year experienced both formats in an alternating fashion. For example, 1 clerkship would have 3 TBL sessions (A, B, C) administered using the iPad/Examplify platform (eTBL), and 3 TBL sessions (D, E, F) administered traditionally, using paper handouts and Immediate Feedback Assessment Technique (IF-AT, Epstein Educational Enterprises, Cincinnati, OH) cards (Table 1). In the following clerkship block, sessions D, E, and F were administered using the iPad/Examplify platform (eTBL) and sessions A, B, and C were administered using paper handouts. This was done to ensure that students at similar levels of training would be able to experience both eTBLs and paper TBLs. Taking into consideration that the time of the academic year that students rotate through the Pediatrics Clerkship may impact their level of medical knowledge and confidence, we analyzed the students in the first half of Phase II (blocks 1 and 2) in 1 group and students in the second half of Phase II (blocks 3 and 4) in 1 group. The 6 TBL sessions covered topics of Abdominal Pain, Anemia, Nutrition, Asthma, Fever and Pneumonia, and Toxicology. TBL sessions were adapted from content published on MedEd Portal with the exception of the Toxicology session which was independently developed by the pediatric clerkship director.11-16
Schedule of team-based learning sessions.
In addition to the utilization of technology, the content of the eTBLs only differed from the paper TBL sessions with the inclusion of graphics into question stems and answer choices when appropriate. For example, the fever/pneumonia eTBL included radiographic images and the abdominal pain eTBL included physical exam findings, stool examinations, and plain abdominal films. By contrast, the asthma eTBL questions did not contain many images. The number of images added to each individual session is outlined in Table 2. Graphics were obtained via web searches for free images and reviewed by the clerkship director prior to their addition. Aside from the added graphics, the question vignettes and answers were otherwise identical for both paper and eTBLs.
Team-based learning sessions, topics, and inclusion details of graphs in the eTBLs.
A 3-way factorial ANOVA was conducted that included 3 independent variables; format of TBL (iPad/Examplify and Paper), topic of TBL session (6 topics), and time of the TBLs (first half of Phase II and second half of Phase II), and 1 dependent variable (percentage correct score on the RAP). This allowed us to not only evaluate whether or not any of the 3 independent variables had a main effect on the dependent variable, but also analyze the 2 way interactions (format and time; time and topic; and format and topic) as well as the 3-way interaction (format, topic, and time) among the 3 independent variables on the students’ RAP scores. In case of a significant 3-way-interaction, subsequent 2-way analyses of variance were computed to determine the group differences. Subsequently, pairwise differences were calculated, using independent samples t-tests. A given P-value <.05 is considered statistically significant.
Students also completed a 15-item post-session questionnaire using a 4-point rating scale (Strongly Agree, Agree, Disagree, Strongly Disagree) and 4 open-ended questions that asked for comments/feedback regarding: (1) how helpful was the iPad during the eTBL session, (2) differences experienced during the RAP process using the iPad compared to traditional paper, (3) problems (if any) encountered using the iPad and Examplify, and (4) why or why not they would recommend the adoption of the iPad/Examplify in other clerkship TBL sessions. Descriptive statistics are presented as frequencies and percentages. Open-ended question responses were analyzed qualitatively by the 3 authors (MC, RP, W-HL). Coding of phrases and texts was performed inductively for identification of key themes.
This study was designated as exempt and waiver of consent was obtained from the Stony Brook University IRB as there was no direct or indirect risk to study subjects (approval number 1171749-1).
Results
A total of 138 students completed the Clerkship in Pediatrics for the 2018-2019 academic year and participated in the study. A 3-way ANOVA was conducted to examine the main effects and interaction effects of the independent variables (format, topic, and time of the TBLs) on the dependent variable (RAP score). Results showed a significant main effect for time [F(1,137) = 18.41, P < .001] and topic [F(5,162) = 17.45, P < .001] but not for format [F(1,137) = 2.27, P < .135]. Overall, students in the second half of Phase II had higher RAP scores (M = 87.9, SD = 5.7) than students in the first half of Phase II (M = 83.3, SD = 8.7). Student performance on the Nutrition TBL was significantly the highest and performance on the Asthma TBL was significantly the lowest compared to performance on the other TBL topics (Figure 1).
Overall student team-based learning (TBL) readiness assurance process (RAP) mean scores by topic.
Additionally there was also a significant 2-way interaction effect between topic and format [F(5,162) = 2.81, P < .02] (Figure 2) but not for the other 2-way (time and topic; time and format) (Figure 3) or the 3-way interactions (format, topic, and time) (Table 3).
Student team-based learning (TBL) readiness assurance process (RAP) mean scores (standard deviations) by format of TBL and topic of TBL.
Student team-based learning (TBL) readiness assurance process (RAP) mean scores (standard deviations) by format of TBL and time of TBL.
Three-way analysis of variance results.
Follow-up independent samples t-test analysis of TBL RAP performance demonstrated that student percentage scores were slightly higher in the eTBL group on most topics as compared to student groups utilizing the paper TBL format though most were not statistically significant. As seen in Table 4, RAP scores were only significantly different between eTBL and paper TBL in the Abdominal Pain (90.2% vs 84.1%, P = .04) and Toxicology (85.6% vs 79.4%, P = .03) TBLs. Although not significant, students only scored lower on the RAP mean percentage in the Fever and Asthma eTBLs.
Independent samples T-test results between eTBL and paper TBL groups.
P < .05.
End of clerkship survey responses revealed mixed opinions regarding the eTBLs (Figure 4). Students felt that the eTBLs were beneficial in facilitating multiple learning activities (55.6%), nurturing learning styles (53.3%), promoting discussion (53.3%), increasing excitement to learn (51.1%), and encouraging active participation (51.1%). However, students also reported that the eTBL format hindered convenience of learning (60%), achieving learning goals (57.8%), confidence in understanding material (55.6%), enriching learning experience (53.3%), developing connection with team (53.3%), and engagement in the learning process (51.1%). Analysis of the open-ended responses indicated that students felt that the eTBL images were beneficial in understanding the case and providing a point for further discussion and the sessions were more environmentally friendly. Criticisms of the eTBL format involved difficulty in navigating the Examplify platform, specifically with logon issues and inflexibility with the program interface (Table 5). Students also cited an inability to write notes during the session, and reported that iPad use during teamwork would occasionally detract from collaboration among group members.
Student end of clerkship survey responses (%).
Themes identified from qualitative analysis of open-ended survey comments by students.
Discussion
The incorporation of iPads into medical education has been described in both the pre-clinical and clinical settings, affording real-time, and portable access to learning resources.4-9 Studies have also demonstrated the beneficial effects of iPads and mobile application integration in the clinical learning environment for formative activities such as note and order writing, radiographic interpretations, and learner assessment and feedback.17-19
Our first pass at technology integration into group learning sessions within a clinical course was met with mixed results. As with many educational interventions, a “one size fits all” approach does not equate with success among learners who thrive with varied learning styles. Traditional paper format TBLs specifically target the VARK 20 (Visual, Auditory, Reading, Kinesthetic) reading learning style. We were able to broaden our reach to include the visual learner as well by embedding graphics into the vignettes. Future development of these TBLs would consider the addition of audio and video files as well as integrative technology such as mind/concept mapping to target the auditory and kinesthetic learner.
We anticipated an overall improvement in both eTBL and paper TBL scores over the course of the academic year as students gained more clinical experience in patient care and NBME shelf exam performance. To counter this, we split the results into 2 groups, one who participated in the Clerkship in Pediatrics during the first half of Phase II, and the other who participated in the second half of Phase II. As Figure 3 has illustrated, eTBL performance was higher in the first half of the year, but by the end of Phase II, this difference was eliminated. We conclude that early in their clinical rotations, the addition of images was generally helpful in guiding students in their clinical reasoning, whereas later in the year, these clues are not as essential to TBL success.
It is interesting to note that the students had statistically significant improvements in the eTBLs specifically for the abdominal pain and toxicology sessions which differed greatly in the number of images included in the questions (Abdominal Pain eTBL had 19 pictures and Toxicology eTBL had 6 pictures). Additionally, the Asthma eTBL performed lower than the paper TBL and item analysis revealed one particularly confounding image of atopic dermatitis that had been integrated into a question. For the paper version of this session, the students were given the diagnosis of atopic dermatitis in the answer choice, but with the eTBL, they needed to consider the image and recognize it as atopic dermatitis in order to answer the question correctly. We surmise that the addition of the image to this particular question necessitated higher level processing and negatively affected overall team score on the asthma eTBL and that quality and type of images selected for inclusion likely have more impact than the number of images per session.
A limitation of our study was that we were unable to meaningfully compare performance on the NBME subject exam in Pediatrics between TBL delivery modalities due to a number of concomitant institution-wide curricular changes. Most significantly, Renaissance School of Medicine engaged in a step-wise migration of USMLE Step 1 exam administration from the end of Phase I (pre-clinical) coursework to the end of Phase II during the course of our study. With this migration, our institution noted a significant decline in the NBME subject exam scores across all clerkships taken at the beginning of Phase II, and we felt that this would make it difficult to determine if the lower NBME subject exam scores were attributed to the mode of delivery of the TBL sessions or the timing of the USMLE Step 1 exam.
In the age of growing use of technology within the arenas of undergraduate and graduate medical education, we endeavored to study an educational modality that leveraged the use of a handheld device with an online-learning platform. Though it was feasible to add technology to TBLs in the clinical course curriculum there were many learners who preferred traditional modalities. Our study used the Examplify test-taking web-based platform, which was often cited as a major point of frustration from students. These frustrations were ameliorated by the fact that the TBL scores were low-stakes, formative assessments, and would not affect their overall grade in the clerkship. While certain images may have made selecting the correct answer more difficult, the discussion that followed to clarify the question was often rich with debate; affording students the opportunity to develop clinical reasoning skills in a formative activity. Optimizing mode of delivery and improving the integration of media such as audio and videos into cases might allow for a more comprehensive and enjoyable experience.
The emergence of the novel coronavirus (SARS-CoV-2) pandemic has forced many institutions to develop means of delivering curricular content remotely.21,22 During peak surges in disease, some institutions elected to remove students from the learning environment entirely, while others continued clinical education but chose to deliver didactic and group teaching via distance learning. Development of innovative means to engage learners and cultivate relationships between learner and educator have proven challenging. Use of a web-based exam platform to deliver TBL content as we have outlined could prove useful for many educators in both the UME and GME arena. Taking the results of our study into consideration, we have decided to administer the content of the TBL sessions during the pandemic entirely using iPads and Examplify. We have continued to solicit feedback from our learners and modify the sessions as is appropriate. Although our study met with mixed results, we conclude that the positive impact on the environment and educational benefit of the inclusion of graphics into the sessions outweighs the negative. Future research will evaluate the educational impact of TBL administration entirely online in a distance-learning format. We will also want to examine student performance on the NBME Pediatrics subject exam to identify any potential differences between groups that utilized eTBL versus paper TBL.
Footnotes
Funding:
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
MBC conceived this study, participated in data collection and analysis, and the drafting and revising of this manuscript. RP participated in data collection and analysis as well as manuscript revision. W-HL participated in study conception, data analysis, manuscript drafting and revision.
Ethical Approval
This study was designated as Exempt by the Stony Brook University IRB.
Informed Consent
Waiver of consent was obtained from the Stony Brook University IRB as there was no direct or indirect risk to study subjects.
