Abstract
Background:
During the rapid onset of the pandemic, clinicians transitioned from traditional outpatient practice to virtual modalities for providing routine care to patient panels. Like training programs nationwide, telemedicine training and assessment had not been systematically incorporated into our residency. In response, a scheduled Internal Medicine (IM) Objective Structured Clinical Examination (OSCE) was adapted to a remote modality to become virtual care-focused learning experience for trainees and to provide valuable feedback to educators.
Methods:
Standardized Patients (SPs) rated residents on their communication (including information gathering, relationship development and patient education), patient activation and satisfaction, and telemedicine skills. Analyses included a comparison of domain scores for residents who participated in both the 2020 remote and 2019 in-person OSCEs, and a review of written resident comments about the virtual OSCE.
Results:
During 2020’s video visit OSCE (VOSCE), residents (n = 23) excelled at nonverbal communication but struggled with virtual physical exams and information gathering. In debrief, residents expressed substantial interest in more opportunity to practice virtual visit skills going forward. In comparing scores of the virtual care (2020) OSCE with the in-person (2019) version, the small subset of residents who participated in both assessments (n = 9) performed similarly on communication skills, patient satisfaction and activation. Patient education scores were significantly lower during the virtual care OSCE (P = .008).
Conclusion:
Our reformulated OSCE accomplished 3 goals including; (1) physically distancing residents from SPs per COVID regulations, (2) providing residents with the opportunity to practice critical virtual visit skills, and (3) alerting our educators to curricular improvement areas. Our methods are useful for other institutions and have applications to the larger medical education community.
Keywords
Introduction
The rapid onset of COVID-19 forced primary care clinicians to transition from traditional outpatient practice to telemedicine (ie, virtual video visits), both to triage potential COVID-positive patients and provide routine care. 1 Like programs nationwide, telemedicine and virtual care provision training had not been systematically incorporated into our residency. In response to this gap and as mandated by social distancing recommendations during the emerging pandemic, our scheduled Internal Medicine (IM) objective structured clinical examination (OSCE) was adapted to take place over a remote, video modality in order to provide a virtual care-emphasized learning experience. The goal of this video visit OSCE (VOSCE) was to teach skills on best practices for conducting virtual visits and to provide educational leadership with valuable information on areas for curricular improvement.
The definition of telemedicine has evolved. Telemedicine is now defined as broadly incorporating the delivery of healthcare services and education from 1 geographical space to a separate one. 2 Virtual clinical encounters, during which a clinical care provider and patient interact over a video tool, fall under the larger umbrella of telemedicine. Telemedicine communication curriculum and learner assessment has not been a big focus in medical education until recently. We developed a video telemedicine OSCE rating checklist using focus groups and iterative feedback with experienced telemedicine practitioners, along with live observation of virtual clinic encounters. 3 Skills described in our assessment tool reflect practice guidelines established by the American Telemedicine Association and used in other specialties’ OSCEs.4,5 Observation of video clinical encounters has shown that some communication behaviors, including expression of empathy and facilitation of medical history gathering, occur less often in televisits than during in-person encounters. 6 Research is needed to fully understand factors contributing to satisfaction, patient-centered care, and practice adoption during virtual visits, not to mention the systemization and dissemination of successful physician training models.
For the purposes of 2020s VOSCE, we utilized a web-based conference platform to simulate a virtual visit in order to assess a specific set of telemedicine skills. The goal of this standardized assessment was to understand resident ability to engage in virtual visit-based care provision. Further, we sought to better identify which skills are necessary for virtual visit-specific curricula in an effort to maximize opportunity for successful adoption of this burgeoning modality by our residency and health system.
Methods
Our new VOSCE included 5 stations from our traditional in-person OSCE, 4 of which utilized video conference software to simulate the remote care environment and are reported in the current study (Table 1). One VOSCE case featured a standardized patient (SP) presenting with COVID-19 symptoms and was designed to assess our learner’s telemedicine-specific skills. Residents participated in 3 additional virtual care cases; (1) educating a patient on buprenorphine maintenance, (2) counselling an elderly patient on mammogram screening, and (3) addressing a patient’s frustration with documentation in their electronic health record. Cases offered virtual visit practice in common primary care scenarios and an opportunity to assess trainees’ remote physical examination and history-taking, prescribing, and communication and patient-centered care skills.
Virtual OSCE cases with standardized patient (SP) demographics.
During the VOSCE, experts at our simulation center set up video conferencing materials in each room. Residents read the door note’s instructions, entered the exam room and signed on to the conferencing platform, signifying the start of the virtual clinical encounter. SPs, who signed on to the virtual encounter at the same time as residents, were stationed in different rooms in an effort to ensure social distancing per emerging regulation. Residents had 10 minutes to complete a virtual visit with the SP, which included taking a complete history, assessing the patient’s condition, and initiating next steps. Faculty members observed performance through the video conferencing system. Upon completion, the SP completed an extensive behaviorally-anchored checklist (3 point: not done, partly done, well done) on resident performance.
SPs rated residents on their core communication skills (including information gathering, relationship development, and patient education and counseling), patient activation and satisfaction, and our case-specific telemedicine items. Domain summary scores were calculated as mean percent rated “Well done.” Residents participated in a virtual group debrief at the end of the VOSCE and completed a post-visit questionnaire, consisting of Likert and open-ended questions on their experience and comfort in handling cases during the VOSCE. Analyses included a paired Wilcoxen signed-rank comparison of domain scores for a small subset of residents who participated in both the 2020 remote and 2019 in-person OSCEs, and a review of written resident comments about the VOSCE. Individual resident consent was not required as this project qualified as a curricular quality improvement project through the NYU Institutional Review Board (IRB).
Results
23/25 possible residents (92%) participated in 2020’s VOSCE, and 9 residents participated in both the 2019 and 2020 OSCE. Fewer than half (46%) of 2020’s residents (n = 23) performed well on the telemedicine skill domain (Table 2). Residents excelled in using nonverbal communication to enrich on-camera communication (100%), but struggled with virtual physical exams (13%), gathering information (4%), and optimizing technology (4%).
2020’s Telemedicine scores and comparison of virtual versus in-person domain scores.
In review of results from the exit survey on attitudes and self-efficacy, few residents reported high confidence in their ability to conduct virtual visits before the VOSCE (25%), but a majority (80%) felt confident or very confident in their virtual visit abilities afterwards. Among VOSCE feedback, residents commented it was “so helpful to have just-in-time training for telemedicine,” and “[the] telehealth and COVID focus was necessary.”
In comparing data from the 2019 in-person to 2020’s virtual OSCEs, residents who participated in both assessments (n = 9) demonstrated consistent performance across most of the communication domains (Table 2). Satisfaction scores were marginally lower in the VOSCE when compared to the in-person. Performance in the patient education domain was significantly lower in the VOSCE (P = .008).
Discussion
We sought to better understand our how residents approach a virtual visit and the learning impact of a remote OSCE in order to identify areas for curricular improvement. In comparing mean scores between our in-person and socially distanced VOSCE, we noted that most resident physician communication skills transfer from physical clinic visits to virtual clinical encounters. However, patient education appears to be more difficult in remote visits, drawing attention to necessary curricular focus as telemedicine adoption in primary care increases. Historically physicians have used telephone calls as a remote panel management tool, and our results demonstrate that they have the skills to communicate clinical basics remotely, but until recently have not been expected, or trained, to deliver patient-centered care on video.
Patient education, activation, and satisfaction skills are essential for patient-centered care, with research in this area suggesting that relationships exist between these skills and longer-term care and health. 7 While residents in our program excel at aspects of communication, patient education skills are often rated lower during routine OSCEs and was noted even lower in virtual visit simulation. We hypothesize that the video visit environment introduces additional barriers to incorporating patient education-specific skills such as diagram drawing/sharing, teach back with audio delay and time management. Our results leave room for future study. Further, trainees consistently struggled with core telemedicine skills like optimizing technology (sound, screen visualization) and virtual information gathering augmentation. Increasing exposure to best practices in patient-centered care while also providing opportunity to practice those skills in simulation is relevant not only to our training program, but to other institutions looking to introduce telemedicine curricula as well.
Post program survey results indicated a substantially higher degree of perceived self-efficacy in conducting virtual visits following participation in the VOSCE compared to prior to participation. Since the initial VOSCE, residents have been given additional materials on best practices for virtual visits that include training guides on video-based communication skills and YouTube tutorials covering virtual physical exams. Our VOSCE findings have helped us develop and implement a curricula on virtual care best practices which will be assessed in subsequent OSCEs. Further, resident feedback following 2020’s inaugural VOSCE indicated desire for a higher ratio of virtual clinical encounters in future simulation and assessment. We will continue to integrate telemedicine training and practice throughout the residency program in alignment with emerging standards, and prime residents for lifelong skill development as technology develops and virtual care becomes integrated in to all practices.
As the medical landscape shifted overnight, our VOSCE was implemented to empower trainees facing the pandemic. This performance-based competency assessment took on increased importance as policymakers and medical professionals accepted virtual clinical visits as a means of care provision. In addition to making residents aware of gaps in their own knowledge, results serve as a needs assessment, alerting our faculty and programmatic leadership to areas for telemedicine-specific curricular enhancement. Insight gained through the VOSCE will impact our residency training going forward. Our findings and lessons learned can provide valuable information for both clinical and education researchers leading training programs.
Footnotes
Acknowledgements
The authors would like to acknowledge the staff and standardized patients at the New York Simulation Center for the Health Sciences (NYSIM) for the assistance with OSCE operations.
Funding:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by Grant No. 1 T0BHP285770100 from the Health Resources and Services Administration, an operating division of the U.S. Department of Health and Human Services.
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
JAW and DB served as primary writers, data managers, and analysts. SZ, DS, RG, KH, and JA contributed to OSCE case development and administration during the event. Each author constributed to revision of written materials.
