Abstract
The coronavirus disease 2019 (COVID-19) pandemic has disrupted undergraduate medical education, including preclinical class-based courses, by requiring social distancing and essentially eliminating in-person teaching. The aim of this study was to compare student performance and satisfaction before and after implementation of remote instruction in a first-year introductory pathology course. Assessments (3 quizzes, 1 practical exam, and 1 final) were compared between courses given before (January 2020) and during (January 2021) the COVID-19 pandemic, in terms of mean scores, degree of difficulty, and item discrimination, both overall and across different question types. Students’ evaluations of the course (Likert scale-based) were also compared between the 2 years. Significantly higher mean scores were observed during remote instruction (compared to the prior, in-person year) on verbatim-repeated questions (94.9 ± 8.8 vs 89.4 ± 12.2; P = .002) and on questions incorporating a gross specimen image (88.4 ± 7.5 vs 84.4 ± 10.3; P = .007). The percentage of questions that were determined to be moderate/hard in degree of difficulty and good/very good in item discrimination remained similar between the 2 time periods. In the practical examination, students performed significantly better during remote instruction on questions without specimen images (96.5 ± 7.0 vs 91.2 ± 15.2; P = .004). Finally, course evaluation metrics improved, with students giving a higher mean rating value in each measured end point of course quality during the year of remote instruction. In conclusion, student performance and course satisfaction generally improved with remote instruction, suggesting that the changes implemented, and their consequences, should perhaps inform future curriculum improvements.
Introduction
The coronavirus disease 2019 (COVID-19) pandemic has severely disrupted both health care and higher education throughout the world, and as a result, medical education, being at the nexus between the two, has been profoundly impacted. 1,2 Social distancing measures, complete lockdowns and the overwhelming burden on health care utilization due to the disease, have led to major disruptions in the content, conduct, and evaluation of undergraduate medical courses. 3 In order to ensure continuity in training, most academic medical centers in the United States made the strategic decision to transition to an almost completely remote learning model, including online instruction and assessment of preclinical courses. Thus, the pandemic has served as a catalyst in a paradigm shift that has significantly accelerated recent trends in medical teaching, such as reducing the number and size of large lectures, providing opportunities for individualized instruction, and shifting to electronic educational media. 4,5 However, the acute chronology of these changes has also brought to the forefront existing risks and difficulties in terms of mental health and the social and economic well-being of students, with likely long-term implications. 6,7
The discipline of anatomic pathology, with its emphasis on the histomorphologic appearance of cells and tissues and the schematic representation of pathophysiology pathways in human disease is perhaps uniquely positioned for a seamless transition to remote learning entirely based on digital images. 8,9 In fact, even before the pandemic, efforts had been well underway to implement updated technology modalities, such as livestreaming, social media, and virtual microscopy, in pathology training. 10 -12 Nevertheless, there are obstacles during the employment of these learning techniques, including, but not limited to, the need for appropriate hardware and compatible devices, and the requirement for adequate connectivity and storage capacity. 13,14 Besides technical issues, distance learning faces additional challenges in the realms of engagement/commitment, structure/organization, and communication/mental well-being. 15,16
To that end, data are needed that determine the efficacy and evaluate the outcomes of purposefully and completely online medical teaching programs, but these are generally lacking. The aim of this study was to measure the effect of remote instruction on students, both in terms of their capacity for learning as well as their perception of course quality. Given the context of reacting to COVID-19, we reviewed assessment performance and course satisfaction among undergraduate medical students remotely participating in a first-year introductory pathology course and compared them in detail to their classmate cohort from the immediately preceding year, taking the same course entirely in-person. Based on our previous item-level analysis, we specifically examined student performance among various types of assessment questions and also determined item discrimination and degree of difficulty for these question categories. 17 In addition, we evaluated student performance on a team-based practical examination which had been transformed from a hands-on activity to an online conference-room format. Finally, we compared the students’ post-course evaluation of its different components regarding learning objectives, instruction methods and materials, and content quality before and after the pandemic-induced transition in mode of instruction.
Materials and Methods
The study was approved by our Institutional Review Board (#18-00479) and conducted according to the World Medical Association Declaration of Helsinki. Deidentified scores were retrospectively collected for the academic year immediately before COVID-19 (2019-2020) and the year during the pandemic (2020-2021) from the web-based course management platform Blackboard Learn (Blackboard Inc.). Student attempts were recorded for all assessments and average responses (percent correct) were calculated for each question in each of the 2 events (years). Each question was characterized across a number of categorical variables as previously described. 17 Briefly, questions were categorized in terms of: answer type (multiple choice or matching), presence of clinical vignette (if so, whether simple or complex), presence of specimen image (gross or microscopic), depth of information (simple recall or interpretation), knowledge density (first or second order), Bloom level (1-3, in increasing complexity), and, for the final exam, subject familiarity (new or repeated subject and, if so, slightly modified or verbatim). Item difficulty was graded according to published guidelines: hard if <30%, moderate (desired range) if 30% to 80%, and easy if >80%. 18 Item discrimination was calculated based on the point biserial index and graded as follows: poor if below 0.2, fair if 0.2 to 0.29, good if 0.3 to 0.39, and very good if 0.4 to 0.7. 19 Student evaluations of various course parameters were obtained using a 5-level Likert scale. Mean student scores and rating values across the various question characteristics and evaluation parameters, respectively, were compared between the 2 time periods using unpaired t test with a two-tailed P value. The Bonferroni correction for multiple univariate comparisons was applied by dividing each P value with the number of hypotheses (ie, variables) tested. Percentage of questions in terms of degree of difficulty (hard and moderate together vs easy) and item discrimination (very good and good together vs fair and poor together) were compared using Pearson χ2 tests. All statistical analyses were carried out using Statistical Package for the Social Sciences software (SPSS; Build 1.0.0.1327; copyright 2019, IBM) with P < .05 considered significant.
Course Design
General Pathology is a 4-week introductory first-year medical school course at our institution, which normally consists of 32 hours of instruction including 17 hours of lectures and 15 hours of lab-based interaction. Lectures are created in Microsoft PowerPoint (Microsoft Corporation), presented to the entire class, recorded using Echo360 (Echo360 Inc), and are available on Blackboard for future access and archiving. Labs are organized in small groups of 36 students each. Each lab includes review of gross organ specimens under the guidance of pathology residents (the small group of 36 students is divided into 4 groups of 9 students each for this part of the lab). Subsequently, students participate in computer-based exercises in teams of 3, under the direction and with the support of pathology faculty instructors. These patient vignette-based lab exercises present clinical scenarios and digital images and ask the students to answer questions reinforcing concepts introduced in the preceding lecture(s). Students complete the exercise through real-time (wiki) collaboration within their teams on Microsoft Word-based documents, available through Google Docs software (Google LLC).
During the COVID-19 pandemic, instruction at the medical school switched from in-person for the academic year 2019-2020 to remote teaching in 2020-2021. In fact, this first-year class of medical school students had not been physically on campus and had not interacted in person with one another at all for the Fall semester. The General Pathology course also switched from completely in-person instruction in January 2020 to entirely remote teaching in January 2021. For that purpose, lectures were transformed from being physically given in a large auditorium to digitally delivered (same topic by the same instructor) over video conferencing using the screen-sharing tool on Zoom software (version 5.4.7; Zoom Video Communications, Inc) and were subsequently available as Blackboard-linked Echo360 video recordings. At the same time, lab exercises transitioned to virtual events also held via Zoom video conference. Residents displayed gross specimens using a high-definition desktop document camera (Lumens PS760; Lumens Digital Optics, Inc; Figure 1A). Students subsequently completed the same lab exercises within their teams, using the “breakout room” functionality on Zoom. Instructors and assisting residents “floated” between breakout rooms to answer questions and the entire group reconvened within the main Zoom meeting room at the end of each exercise for final summation, group discussion, and concept consolidation. This implementation and execution of remote instruction was delegated to individual course directors, with substantial administrative and logistic support from the medical school’s Office of Curricular Affairs and with the expressed involvement of the Office of Student Affairs, the latter regarding issues of student safety and overall well-being.
Remote instruction using Zoom video conferencing. A, Photograph of the desktop camera used to capture video of the specimen on the stage and project the feed onto a Zoom meeting with students. In the photograph background, there are trays with additional specimens to be shown. B, Screen shot of a question from the practical examination that required review of gross specimens. A video recording of the pathology resident (inset) presenting specimen findings was embedded within the assessment question. The students had the ability to expand the video to fill the entire screen or click a link for the video to open in a separate window.
Course assessments of individual students comprise a total of 125 questions which were identical between the 2 academic years 2019-2020 and 2021-2022. Questions are distributed among 3 weekly quizzes and 1 final exam (quiz one: 15 questions, quizzes two and three: 20 questions each, and final: 70 questions). All assessment examinations are closed-book and are administered individually, remotely, and online, with each student completing an honor code attestation. Some of these assessment questions include digital photographs of gross and/or microscopic specimens that are similar or identical to images previously included in lectures or lab exercises. However, neither these images included in assessment questions nor the images that students used to study from (ie, from lectures and lab exercises) were any different between the course taught in-person (2019-2020) and the course given remotely (2020-2021).
In addition, the last lab session of the course takes the form of a practical examination, whereby students answer 38 different questions collectively as a team-based exercise, in their usual 3-member lab group teams. These practical exam questions are based on unknown surgical and autopsy patient cases. In prior years, these cases were presented to the students in person and they had the opportunity to physically examine gross specimens, before attempting to answer assessment questions on the associated online quiz. During the pandemic, this entire practical assessment was moved online, with pathology residents using the camera set-up described above to present cases in video vignettes that were then embedded within the online quiz questions (Figure 1B).
Results
The number of students enrolled in the course during in-person instruction (January 2020) was 140 and during remote instruction (January 2021) was 139, for a total of 17,500 and 17,375 graded attempts, respectively, which were scored as correct or incorrect (Table 1). Overall, the mean percentage of correct answers was not significantly different (86.1 ± 13.3 vs 85.9 ± 16.5). However, when looking at questions with different characteristics, students had significantly higher mean scores during remote instruction (2021), compared to in-person teaching (2020) on questions repeated verbatim in the final exam from prior quizzes (94.9 ± 8.8 vs 89.4 ± 12.2; P = .002 with Bonferroni correction) and, surprisingly, on questions that contained a gross specimen image (88.4 ± 7.5 vs 84.4 ± 10.3; P = .007, corrected).
Student Performance in Assessment Questions During In-Person and Remote Instruction.*
Abbreviation: SD, standard deviation
* Statistically significant P values (ie, < .05) are bold. Applying the Bonferroni correction for 23 univariate tests leaves only presence of gross image and verbatim repetition of subject in the final as significantly different between the 2 sets (P values of .007 and .002, respectively).
As expected, given that the questions did not change year-over-year, the percentages of questions with moderate or high (hard) degrees of difficulty compared to those that were considered easy were similar between the 2 years, regardless of question type or characteristic (Table 2). Similarly, capacity for item discrimination (good/very good vs fair/poor) was analogously distributed during in-person and remote instruction, across all question subtypes (Table 3).
Degree of Difficulty in Assessment Questions During In-Person and Remote Instruction.*
* There are no significant differences between the 2 years (ie, all are P > .05).
Item Discrimination of Assessment Questions During In-Person and Remote Instruction.*
* There are no significant differences between the 2 years (ie, all are P > .05).
In terms of the 38 different questions in the separate practical examination, students performed better during the year of remote instruction on matching type questions, those with simple clinical vignettes, questions testing first order knowledge, and questions assigned a Bloom taxonomy level one, and performed significantly worse on questions with an image of a gross specimen, compared to students during the prior year with in-person teaching (Table 4). However, these differences did not survive the Bonferroni correction for multiple univariate comparisons. In contrast, the improved performance of students during COVID-19 on questions lacking a specimen image with a mean score of 96.5 ± 7.0 (compared to 91.2 ± 15.2 the year before) remained statistically significant (P = .004, with Bonferroni correction).
Student Performance in the Practical Exam During In-Person and Remote Instruction.*
Abbreviation: SD, standard deviation.
* Statistically significant P values (ie, < .05) are in bold. Applying the Bonferroni correction for 19 univariate tests leaves only the absence of specimen image as significantly different between the 2 sets (P value of .004).
Finally, students’ responses during the post-course evaluation were recorded and compared between the course given remotely and the course with in-person instruction the prior year. Each year, approximately half the students in the class are randomly asked to anonymously and confidentially provide ratings based on a 5-level Likert scale (from 1-strongly disagree/poor to 5-strongly agree/excellent) on the quality of various course parameters, including organization, learning objectives, content, teaching, and instruction methods (Table 5). There was improvement in the mean rating values assigned by students during remote teaching (compared to those during in-person instruction) in every single parameter measured, however none reached statistical significance.
Comparison of Students’ Course Evaluations Following In-Person and Remote Instruction.*
Abbreviation: SD, standard deviation.
*P values for univariate comparison across each row were not statistically significant (ie, > .05).
† Rated as follows: 1 = poor, 2 = fair, 3 = good, 4 = very good, 5 = excellent.
§Rated as follows: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.
Discussion
The results of this study show that transition to remote instruction is not associated with a degradation in student performance and in fact may be accompanied by improvement, particularly on assessment questions with certain characteristics, such as those being repeated from previous exams or those that contain an image of a gross specimen and, in the practical examination, those that lacked a specimen image. However, assessment questions retained their degree of difficulty and capacity for item discrimination overall, compared to the prior year with in-person teaching. Importantly, student satisfaction with course quality improved over the previous year, as evidenced in evaluation ratings.
The rapid and comprehensive disruption caused by the COVID-19 pandemic compelled academic institutions worldwide to reevaluate and adapt their organizational models. In medical education, and specifically pathology teaching, a varying array of solutions were implemented, from mixed models incorporating flipped classrooms with limited, unidirectional, and timed in-person specimen examination to fully virtual laboratory classes using online video-conferencing tools. 20 -22 These changes reflected similar adjustments imposed on pathology resident training and upper-class student electives and mostly concerned the replacement of instruction centered on a physical microscope with the use of virtually accessed digitized slides. 23 -27 Among individual medical schools, the decision of how to modify instruction depended on factors such as content complexity, technical feasibility, time limitations, and infrastructure capacity. 28 In our institution, the decision involved an immediate transition to completely remote instruction for all preclinical courses and was partly influenced by the fact that our school’s location was at the time an epicenter of the pandemic.
This precipitous and near universal adoption of remote learning modalities also necessitates a framework for their evaluation, both for quality assurance as well as for the development of standards and best practice indicators as they have been identified in the areas of organizational capacity, educational effectiveness, and human resources. 29 We were especially interested in examining how online instructional methods affected student involvement in and understanding of the learning material as well as their perception of course quality. To measure these effects, we examined students’ performance on the course learning assessments and their subsequent evaluation of course content and delivery. While it may be difficult to compare student performance across different years, the large number of scored attempts and the fact that course content and assessments have remained relatively constant in recent years hopefully ameliorated some of the cyclical variation. In addition, we had previously observed that year-over-year divergence has been minimal in this cohort. 17 Finally, with the course being in the middle of an academic year with completely remote instruction, the students were used to the video conferencing format and thus fairly capable in its usage and comfortable with its features.
Student assessment scores in this cohort were remarkably similar to prior years, with the transition to remote instruction leading to significant increases in the mean percentage of correct answers in only 2 types of questions, those with a gross specimen image and those repeated verbatim in the final exam from prior quizzes. We hypothesize that improved performance in both of these question categories reveals an effort on the part of the students to concentrate more and invest time in their studying. Having been warned that images would be included in the assessments and that question topics may be repeated in the final exam, they may have focused their individual learning on preparing for these exact eventualities, with the resultant better scores. The partial lockdown still in effect at the time probably minimized extraneous distractions. Supporting this explanation, a recent study of a medical histopathology course found that, while student scores on team quizzes during the lab sessions declined significantly during COVID-19 compared to prior years, their individual performance on weekly quizzes and module examinations (more involved assessments occurring at a later time) was unaffected. 21 Other authors have also found increased medical student academic performance with the transition to e-learning necessitated by COVID-19, however they were not based in the United States and examined overall test scores between semesters without a detailed breakdown by course and question type. 30 Furthermore, degree of difficulty and item discrimination for these questions in our course did not differ from prior years, implying that educational goals and program objectives, vis-à-vis the function of student assessments, were still being met. 17
At the same time, the lack of in-person instruction may have been an incentive for students to specifically emphasize learning themes that may have been affected by remote instruction, such as gross pathologic appearance of disease processes, hence improving their understanding and scores on such topics. Consistent with this hypothesis, a large cross-sectional survey in the United Kingdom found a significant increase in the amount of time spent by medical students on electronic learning platforms, something that has been echoed by others. 31,32 We posit that the lab practical, being an exercise given in real time and for which students didn’t really prepare (since it was supposed to be open-book and taken collectively in teams of 3), did not pose as much of a challenge or an impetus for them to study gross images ahead of time, thus leading to lower performance in this assessment (Table 4). In addition, perhaps because of this lower performance on questions with gross images in the lab practical, students were compelled to prepare and study these types of images for the upcoming final exam, leading to a higher score in such questions during the final, summative assessment (Table 1).
An intriguing alternative explanation is that due to the remote set-up used in this course, pathology residents were displaying via camera only one gross specimen at a time (as opposed to students congregating around a table with all the organs of the day’s session simultaneously exposed). This may have allowed students to concentrate on one specimen at a time and follow the resident’s guidance in describing its physical characteristics (color, size, shape, texture, consistency, etc). On the other hand, students performed significantly better in the practical examination during COVID-19, but only on questions without a specimen image, suggesting that there are still barriers to remote instruction in terms of adequately portraying perhaps more subtle findings on gross organs.
We also found that students were overall very pleased with the course delivered remotely, which scored higher on their evaluations compared to the previous year. Even though it is a different set of students responding to these evaluation surveys and we did not specifically address remote instruction and online learning, general trends can be surmised from their narrative answers that accompanied these questionnaires (data not shown). Students appreciated the interactive, albeit virtual format of the course: they responded positively to the lab setups with team-based learning and the opportunity to interact with residents in small groups. They also welcomed the chance to work in groups privately as afforded to them in the breakout rooms, before reconvening in the main online meeting room for group discussion. From their comments, it was evident that students valued the organized, targeted, and effective teaching provided, and rated the course very highly despite the disruption caused by COVID-19. As is also being reported by others, while online teaching is favorably received, most students report that they still prefer the physical interaction of in-person instruction. 22,25,31,33 -36
Generally, reported strengths of remote learning include increased participation, adaptability, and flexibility of both content and schedule, organized and personalized delivery geared toward individualized goals, and a positive education environment that is useful and constructive. 28,31,35 Nevertheless, significant challenges to remote learning remain, including infrastructure and technical issues such as internet connectivity (speed, bandwidth, interruptions in service) and physical space requirements that may lead to confidentiality issues and family or other distractions. 28,31 For example, students have reported that online educational tools tend to decrease peer-to-peer teaching, obstruct the learning experience, and increase anxiety, outcomes that may disproportionally affect some groups of learners more than others. 21,37 In addition, the teaching of practical skills, which are essential in medical education, particularly in the clinical years, may not be as amenable to remote instruction. 38 Similarly, in the preclinical years, remote instruction faces an almost insurmountable deficit in teaching dissection and grossing. 35 Finally, impaired or reduced social connections during virtual interactions may lead to diminished observation of students’ performance in these skills and hinder proper and adequate feedback, particularly in terms of narrative assessment. 21,28,35 Going forward, with the pandemic’s grip easing, it would be important to observe and evaluate student performance and attitudes as we return to in-person instruction and this study would form the basis for continued analysis.
In conclusion, while the necessary transition to completely remote instruction has mostly proven to be a success, in terms of outcomes such as students’ assessment performance and course satisfaction, the long-term effects of this paradigm shift in medical (and all) education will need to keep being monitored and studied. Crucially, student satisfaction with course quality improved over the previous year as evidenced in evaluation ratings, suggesting that the impact of COVID-19 may not manifest itself in ways previously feared, but also instructing future attempts in curriculum development. It is difficult to imagine a future where online teaching is not a major component of medical learning but the well-being of both students and instructors, particularly in these interactions with reduced physical contact and less structured frameworks will need to remain in the forefront. 31,39,40 This is undoubtedly an opportunity for additional innovation in the delivery and assessment of medical education and, in this context, perhaps a hybrid model will be best suited to combine the efficacy and usefulness of remote instruction with the benefits and community of in-person engagement.
Footnotes
Acknowledgments
Authors would like to thank Megan Sacco, Kira Schmidt, and Jeff Cavalli for administrative and technical support during the transition to remote instruction.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
