Development of a Virtual Pediatric Telemedicine Training Program

Abstract

Background:

Medical students and pediatric residents are not routinely trained in urgent care telemedicine. Increased demand for virtual care following the COVID-19 pandemic is expected to remain, underscoring the need for health care organizations to develop structured training programs that meet the changing educational requirements of future pediatricians.

Materials and Methods:

The Nemours IRB reviewed this study and designated it a “Not Research” (#1905675). This mixed-methods study assessed knowledge and comfort levels with telemedicine before and after participants completed a virtual telemedicine training rotation. Seven medical students and pediatric residents completed a brief, pre- and post-rotation survey with multiple-choice, true/false, and open-ended questions. Open-ended questions were thematically analyzed.

Results:

Improvement in participant knowledge was most notable for understanding the diagnostic limitations of telemedicine. At pretest, only 28.6% of participants believed they could adequately perform a telemedicine examination; after the program, 100% reported comfort with the modality. The most common theme among the participants was the belief that telemedicine played a role in expanding health care access.

Discussion:

Most participants chose the rotation because of their curiosity about telehealth and came into the training with some general knowledge of its purpose and operation. Exposure to various teaching methods, one-on-one time with attending physicians, and exposure to telemedicine clinical guidelines resulted in positive overall experiences.

Conclusion:

Our virtual telemedicine rotation curriculum uses an iterative process based on participant feedback to provide a learning experience that is increasingly improved to meet the knowledge level of our participants. Through this quality improvement lens, we aim to maximize participant learning and motivate the development of other remote telemedicine training programs.

Introduction

As a result of the COVID-19 pandemic, telemedicine has become an integral aspect of health care delivery. Although health care systems rapidly deployed providers via this medium during the initial stages of the pandemic, a growing need for training and educational programs was evident. ¹ Before the pandemic, medical students and pediatric residents were not trained routinely in telemedicine.^2,3 Since then, medical colleges and health care systems have recognized this educational gap and started designing telemedicine programs to develop future physicians.^4–7 In 2021, the American Association of Medical Colleges (AAMC) released telehealth core competencies under the following six domains: patient safety, access and equity, communication, data collection, technology, and ethical and legal practices. ⁸ These domains offer a framework when designing educational telemedicine programs. Many programs use virtual workshops and online modules for telemedicine education.^9–12 Our pediatric telehealth teaching program is a clinical model, through which faculty, through a live multiperson platform, provide real-time assessment based on the AAMC telehealth core competencies.

Nemours Children’s Health (NCH) is a pediatric health care system with two hospitals, primary care offices, subspecialty clinics, and urgent care facilities. Centered in the Delaware Valley and Florida, NCH providers care for children across seven states. NCH has operated a pediatric urgent care telemedicine program since 2015, through which its licensed pediatricians provide acute care virtually to patients. In 2021, the NCH residency program in Orlando, Florida, the University of Central Florida (UCF) College of Medicine, and the NCH telemedicine team collaboratively developed an acute care training program aligned with the AAMC core competencies to alleviate the shortfall in medical student and resident education. The program was initiated virtually, with the telehealth technology itself being used as the platform for remotely training young pediatricians in the clinical and technical aspects pertinent to telemedicine care delivery. As a quality improvement initiative, we sought to evaluate and improve the training with each subsequent iteration of resident and student participants who completed the program. We requested feedback from the 19 participants (4 residents and 15 medical students) who completed the first iteration of the training program as we shaped the initial program curriculum. ¹³ Both residents and medical students benefited from the training, demonstrating that the program was worthwhile and supportive for early career pediatricians. ⁹ Initial feedback prompted improvements, including allotting academic time to the schedule for independent study, and all preceptors were encouraged to provide real-time feedback after a proctored visit. Participants acknowledged the benefit of the post-visit huddle and expressed a desire to review research regarding current telemedicine topics.

In October 2022, we initiated a revised, 24-item digital version of the pre-and post-rotation survey to administer to present and future participating medical students and pediatric residents. The survey was revised and expanded by the telemedicine physician team to measure the rotation of participants’ changing knowledge and beliefs on the use of telemedicine. Items were revised or developed to align with the revised training content and were inspired by the extant literature on telemedicine and its practical application. For this second iteration, we aimed to obtain participants’ knowledge and experiences with telemedicine that could further refine the training program, thus informing any need for additional emphasis or content revision. The mixed-methods study discussed here represents the learnings obtained from these second-iteration training surveys. To the extent that participants who were knowledgeable about telemedicine and comfortable with its implementation might be more inclined to utilize it in their future practice, we were particularly interested in their incoming knowledge about telemedicine, what they learned from the training, and how comfortable they were with virtual care following the rotation.

Materials and Methods

The virtual telemedicine training program, offered as a 2-week or 4-week elective, includes a detailed clinical and technical orientation, a review of current policy and licensing requirements, and teaching best practices. Orientation begins with a presentation about NCH, an overview of telemedicine, and expectations for the rotation. Training includes a remote one-on-one session with the support staff to ensure secure, efficient connections, device compatibility, equipment readiness, and application of peripheral devices. Participants learn to troubleshoot and seek assistance for patients who experience technical problems. The orientation session concludes with a fully functional and secure test visit.

Following orientation, participants receive a schedule of their virtual clinical shifts. Participants log patient visits and are encouraged to engage in as many telemedicine encounters as possible. The remote training program provides a one-on-one, preceptor-to-participant experience through the virtual platform where the patient, participant, and preceptor are in separate locations. Hands-on training offerings differ between medical students and residents based on their education level. Medical students and residents participate in observed and proctored visit types. In an observed visit, the participant observes the interaction between the preceptor and the family. In the proctored visit, the participant leads the encounter and the preceptor proctors. In a third visit type, only for residents, the participant evaluates the patient independently and invites the preceptor to discuss the wrap-up at the end of the encounter. Medical students do not evaluate patients independently. Residents and medical students receive real-time education and feedback from faculty through a pre- and post-visit huddle using a separate virtual chat. If the patient’s encounter is determined not to be suited for telemedicine, the patient is directed to schedule an in-person examination.

To begin an observed telehealth visit, the preceptor sends a secure email invitation to the participant to join the platform. This invitation includes the patient’s medical record number, enabling the participant to review the patient’s medical history, including primary care visits, subspecialty visits, hospitalizations, and emergency room visits. Participants check their audiovisual equipment to ensure their surroundings are private before connecting with the parent/guardian. Once the participant accepts the preceptor’s invitation, the visit becomes a three-way encounter with the participant, patient, and preceptor on one screen. The preceptor addresses ethical and legal requirements, confirming patient identification, guardianship, and the location at the beginning of every encounter. The preceptor also requests parent/guardian permission regarding comfort with a teaching environment. Once confirmed, the visit proceeds in which the preceptor leads and the participant observes the interaction. At the completion of the visit, the preceptor and participant use a virtual chat for a post-visit huddle; the preceptor shares the computer screen with the participant to demonstrate appropriate telehealth documentation.

Once the participant is comfortable observing a visit, the preceptor recommends a proctored visit. Proctors assess “web-side” manner, history-taking skills, physical examination techniques, professionalism, and communication skills. In a proctored visit, the preceptor transfers the patient to the participant’s waiting room. The participant then sends a secure email invitation to the preceptor to join. Once the preceptor, participant, and family are on the screen, the participant leads the visit. Participants receive intra-visit support and real-time guidance through the virtual chat feature so that the encounter flow is not interrupted, and the visit appears seamless to the family. If working with a medical student, the preceptor writes the note and sends the prescription; otherwise, the resident writes the note, and the preceptor cosigns. Independent visits performed only by residents are similar to proctored visits; however, the resident invites the preceptor to join at the end to review the history, physical, assessment, and plan.

The elective rotation also provides dedicated time toward academics and independent study. During this time, participants practice writing progress notes, review our “interesting clinical cases” file, and select a journal article relevant to telemedicine. The participant presents their article, guideline, and case presentation to the telemedicine team during the final week of the rotation.

A year after the first participant training, we began administering the digital pre- and post-rotation survey. Study data were collected and managed using the Research Electronic Data Capture tools hosted at NCH.^14,15 The confidential survey included multiple-choice, true/false, and open-ended questions (see Table 1). Responses to survey items were categorized in the following three ways for purposes of presentation: knowledge demonstration (predominantly multiple-choice, true/false responses), informational/experiential (i.e., no correct or incorrect responses), and open-ended opinion (qualitative) responses. Informational/experiential responses given as “Other” were probed for further detail. Open-ended responses were analyzed using conventional content analysis, through which the authors separately identified and categorized themes in the qualitative responses, followed by collective discussion and coding of common themes across responses to address any discrepancies. ¹⁶ Following group discussion, the authors unanimously agreed on the themes as presented.

Table 1.

Survey Items Administered to Pediatric Residents and Medical Student Participants Before and After the Virtual Training Rotation

Knowledge Demonstration:

1. How would you define the term “telemedicine”? _______________________

2. What equipment is essential for conducting a successful telemedicine visit? (Select all that apply)

A. headphones

B. hi-definition web camera

C. peripheral devices

D. broadband/reliable internet connection

3. The technical platform for telehealth must: (Select all that apply)

A. be HIPAA 1

B. meet minimum bandwidth requirements

C. incorporate use of a desktop device

D. offer high-speed upload and download capabilities

4. Studies show that most patients report being satisfied with their telemedicine experience.

A. True

B. False

5. The benefits of direct-to-consumer telemedicine to the clinician include: (Select all that apply)

A. decreased travel time to and from work

B. ability to conduct a hands-on physical examination

C. expansion of patient geographic reach

D. ability to establish continuity of care through a medical home

6. The benefits of direct-to-consumer telemedicine to the patient include: (Select all that apply)

A. improved access to internet

B. decreased travel time to and from an appointment

C. decreased absences from school/work

D. ability to be examined virtually for any illness

7. Barriers of telemedicine utilization from the perspective of the patient include: (Select all that apply)

A. patient education/knowledge on how to access a visit

B. limited access to necessary broadband technology

C. increased potential for disease transmission

D. proximity to an urgent care or emergency room facility

8. What are some diagnostic limitations of telemedicine (without use of peripheral devices)?

A. acute otitis media

B. strep pharyngitis

C. upper respiratory infection

D. urinary tract infection

9. What are some examples of federal regulations that impact pediatric telemedicine implementation? (Select all that apply)

A. 1

B. age of child

C. birthplace of child

D. originating site/location of patient

10. How has the COVID-19 pandemic changed telemedicine? (Select all that apply)

A. improved patient education

B. increased patient volume

C. increased technology requirements

D. eliminated need for provider licensing and credentialing

11. The following components are required to document a telehealth visit: (Select all that apply)

A. patient’s city

B. patient’s state

C. patient’s name

D. date of next office visit

Informational/Experiential:

12. Other than this rotation, have you had any formal education on telehealth/telemedicine?

A. yes

B. no

13. What reason(s) did you choose to select this rotation?

A. interest in telemedicine/telehealth

B. interest in pediatrics

C. this elective worked best for my schedule

D. other. Please state your reason: ______

14. Do you believe you can adequately do an exam via telemedicine? _______________

A. yes

B. no

C. unsure

Why or why not? ____________________________

15. Have you observed and/or conducted a telemedicine patient encounter?

A. I have not observed or conducted a telemedicine patient encounter

B. I have observed but not conducted a telemedicine patient encounter

C. I have conducted but not observed a telemedicine patient encounter

D. I have observed and conducted a telemedicine patient encounter

16. Have you participated in a telemedicine encounter as a patient?

A. yes

B. no

17. Rate your anticipated/actual comfort level with seeing a patient via telemedicine

A. not at all comfortable

B. somewhat comfortable

C. comfortable

D. very comfortable

E. extremely comfortable

Open-ended:

18. In your opinion, what role does telemedicine play in health care? ___________

19. What do you expect to learn/what did you learn from this rotation? ________________

20. How do you think this rotation will benefit your medical training? _________________

21. Name one thing you would like to learn more about/one thing you learned from this rotation that you will use in your practice of pediatric medicine. ____________

22. Post-training only: Name one part of the training that you thought was most effective in developing your understanding of pediatric telemedicine. _____________________

23. Post-training only: Name one part of the training that you thought was least helpful in developing your understanding of pediatric telemedicine. _____________________

24. Post-training only: What did you hope to learn about telemedicine that was not taught in the training? _____________________

¹HIPAA is the Health Insurance Portability and Accountability Act

Results

To date, three residents and four medical students participated in the training and completed the current survey before (pretest) and after (posttest) the telemedicine rotation.

Knowledge demonstration

Eleven questions were drawn from the research literature on telemedicine to develop the Knowledge Demonstration items. In completing the Knowledge Demonstration items, participants were asked to select all the correct answers that applied to a question. Table 2 presents pretest and posttest frequencies and percentages for responses to the multiple-choice, Knowledge Demonstration survey items.

Table 2.

Pretest and Posttest Frequencies and Percentages for the Ten Knowledge Demonstration Multiple-Choice Survey Items

Question	Pretest “yes” response n (%)	Posttest “yes” response n (%)
What equipment is essential for conducting a successful telemedicine visit?
a. Headphones	3 (42.9)	3 (42.9)
b. Hi-definition web camera*	5 (71.4)	4 (57.1)
c. Peripheral devices	1 (14.3)	2 (28.6)
d. Broadband/reliable internet connection*	7 (100)	7 (100)
The technical platform for telehealth must:
a. Be HIPAA compliant	6 (85.7)	7 (100)
b. Meet minimum bandwidth requirements*	7 (100)	7 (100)
c. Incorporate use of a desktop device	2 (28.6)	1 (14.3)
d. Offer high-speed upload and download capabilities*	4 (57.1)	3 (42.9)
Studies show that most patients report being satisfied with their telemedicine experience.
a. True*	7 (100)	7 (100)
b. False	0 (0)	0 (0)
The benefits of direct-to-consumer telemedicine to the clinician include:
a. Decreased travel time to and from work*	7 (100)	7 (100)
b. Ability to conduct a hands-on physical examination	0 (0)	0 (0)
c. Expansion of patient geographic reach*	6 (85.7)	7 (100)
d. Ability to establish continuity of care through a medical home*	6 (85.7)	2 (28.6)
The benefits of direct-to-consumer telemedicine to the patient include:
a. Improved access to internet	1 (14.3)	1 (14.3)
b. Decreased travel time to and from an appointment*	7 (100)	7 (100)
c. Decreased absences from school/work*	7 (100)	7 (100)
d. Ability to be examined virtually for any illness	3 (42.9)	1 (14.3)
Barriers of telemedicine utilization from the perspective of the patient include:
a. Patient education/knowledge on how to access a visit*	7 (100)	7 (100)
b. Limited access to necessary broadband technology*	7 (100)	7 (100)
c. Increased potential for disease transmission	0 (0)	0 (0)
d. Proximity to an urgent care or emergency room facility	0 (0)	0 (0)
What are some diagnostic limitations of telemedicine (without use of peripheral devices)?
a. Acute otitis media*	7 (100)	7 (100)
b. Strep pharyngitis*	5 (71.4)	7 (100)
c. Upper respiratory infection	0 (0)	0 (0)
d. Urinary tract infection*	4 (57.1)	6 (85.7)
What are some examples of federal regulations that impact pediatric telemedicine implementation?
a. HIPAA compliance*	7 (100)	7 (100)
b. Age of child	2 (28.6)	1 (14.3)
c. Birthplace of child	0 (0)	0 (0)
d. Originating site/location of patient*	7 (100)	7 (100)
How has the COVID-19 pandemic changed telemedicine?
a. Improved patient education*	2 (28.6)	6 (85.7)
b. Increased patient volume*	7 (100)	7 (100)
c. Increased technology requirements	2 (28.6)	2 (28.6)
d. Eliminated need for provider licensing and credentialing	0 (0)	0 (0)
The following components are required to document a telehealth visit:
a. Patient’s city	1 (14.3)	1 (14.3)
b. Patient’s state*	7 (100)	7 (100)
c. Patient’s name*	7 (100)	7 (100)
d. Date of next office visit	0 (0)	1 (14.3)

Correct answers are indicated with an asterisk.

HIPAA, health insurance portability and accountability act.

At pretest, all participants demonstrated basic knowledge of telemedicine at the start of the rotation, correctly responding to questions about the requirement for broadband/reliable internet connection, time efficiencies, necessary documentation, patient satisfaction, and federal telemedicine regulations. Following the rotation, pretest to posttest improvement was especially notable in participants’ understanding of the role of telemedicine in patient education (29–86% correct) and its diagnostic limitations (81–95% correct). On the open-ended Knowledge Demonstration item “How would you define the term ‘telemedicine?’” all participants correctly identified telemedicine as a remote modality for providing health care at pretest. Table 3 presents participant pretest and posttest verbatim responses to this open-ended Knowledge Demonstration item.

Table 3.

Categorized Pretest and Posttest Responses to Open-Ended Knowledge Demonstration and Informational/Experiential Questions

Knowledge Demonstration: How would you define the term “telemedicine”?	(“Visits done virtually”“Medical care provided virtually”“Virtual physician–patient encounters with the goal of increasing access to care”“Health assessment and treatment done virtually to allow for greater accessibility”“Medicine encounters that take place virtually through video”“Remotely providing care to patient virtually”“Medicine via telecommunication”)	(“Virtual mediums of providing care”“Health care delivered via a virtual setting”“Providing medical care via technology when the provider and patient are not in the same location”“Evaluation and treatment of patients virtually”“Medical care that is delivered using phone, or video/audio”“Virtual method of providing clinical care to patients”“Use of telecommunications to provide health/medical care”)
Informational/Experiential: Do you believe you can adequately do an exam via telemedicine?Why or why not?	• Yes: (“I think I can do a limited exam via telemedicine.”“While an in-person exam is superior, I do believe there is an adequate telemedicine exam that informs whether in-person examination is indicated.”)• No: (“Untrained for it”)• Unsure: (“I think it depends on the exam and patient complaint. It would be difficult to do an EENT exam or full neuro exam for example.”“ You are able to see general things such as appearance, activity level, etc., but you are unable to auscultate or palpate.”“Some components of exam may be very limited, difficult, or impossible to assess when unable to be hands-on or reliant on parent’s use of peripheral devices.”“Highly depends on the type of exam that needs to be performed.”)	• Yes: (“With camera and good lighting, you can do an exam for complaints appropriate for telemedicine.”“I think there are certainly limitations to the telemedicine physical exam, but I think it is possible to get an assessment as far as a patient’s overall clinical status, hydration state, respiratory status, etc.”“There is a lot you are able to observe over the computer. I think that some aspects of the exam are limited, but you can do enough at least to determine if the child needs to be seen in person.”“I can determine generally how sick a patient is and observe myself as well as utilize help from a parent relevant systems in the exam.”“Yes, I feel I have learned a variety of maneuvers and techniques to conduct a thorough examination.”“Better understanding of skills and limitations.”“Yes, I can adequately do an exam via telemedicine with adequate lighting, instruction, and maneuvering.”)

Informational/experiential survey items

At pretest, one participant (14.3%) reported having had prior telemedicine training; following the rotation, all participants reported having formal telemedicine training. Reasons provided at pretest for choosing the telemedicine rotation included having an interest in telemedicine/telehealth (71.4%), an interest in pediatrics (71.4%), convenience for schedule (71.4%), or other (limited infectious exposure risk, 14.3%). Regarding participants’ prior experience with telemedicine at pretest, four participants (57.1%) indicated that they had observed and conducted a telemedicine patient encounter. More participants experienced a telemedicine encounter as a patient following the rotation (71.4%) than they had before its start (42.9%).

Participants reported improvement in their comfort level of seeing a patient via telemedicine following the rotation experience. Before the rotation, five participants reported being “comfortable” or “very comfortable” seeing a patient virtually. Following the rotation, all seven participants reported being “comfortable” (14.3%), “very comfortable” (42.9%), or “extremely comfortable” (42.9%) conducting a telemedicine visit.

Based on participants’ comments on the open-ended Informational/Experiential item “Do you believe you can adequately do an exam via telemedicine? Why or why not?” at pretest, 28.6% of participants believed they could adequately perform an examination via telemedicine, 14.3% did not, and 57.1% were unsure. At posttest, all participants (100%) reported feeling comfortable conducting an examination via telemedicine. Table 3 presents participant pretest and posttest verbatim responses to this open-ended Informational/Experiential item.

Content analysis themes

There were four open-ended questions on the pretest and seven on the posttest that were subject to content analysis. Table 4 presents verbatim responses and thematic results of the content analysis.

Table 4.

Content Analysis Pretest and Posttest Themes and Responses to Open-Ended Questions

Question	Pretest theme (response)	Posttest theme (response)
In your opinion, what role does telemedicine play in health care?	• Access to care: (“I think that it increases access to care for lower acuity problems and removes barriers such as distance to office or inability to get time off work or school. I think that it has limitations when it comes to the need to conduct a physical exam or run in-office tests, and it cannot fully replace in-person visits.”“Allows for greater access and compliance for medical appointments”“Expands reach of care”)• Triage/Redirection of care: (“It can help relieve the burden of primary care offices and urgent cares by allowing some of the patients to be seen without coming in.”“I think telemedicine has an important role in triaging, as well as keeping patients out of emergency departments/urgent cares when they don't need to be seen in person.”)• Patient safety: (“A vital role in limiting transmission of diseases like COVID”)• Innovation: (“It’s the new frontier”)	• Access to care: (“Telemedicine expands the reach of health care.”“Telemedicine is great for expanding access to care. It’s a great resource for triaging patient needs and keeping patients out of the emergency department when it is unnecessary.”“Expands access to care to more rural and low-income communities. Allows for care that is more flexible to a patient’s schedule. Can help decrease load on emergency rooms and PCP offices by filtering out some less serious problems.”“It allows for easier and better access to health care in certain situations where it would otherwise be difficult to obtain.”“Increased access to care and reduced overloading of hospital systems”)• Triage/Redirection of care: (“It plays a huge role in managing nonurgent or nonemergent patients.”“A vital role in decreasing office, urgent care, and ED volumes by seeing patients that can be seen virtually”)
What do you expect to learn/what did you learn from this rotation?	• How to conduct a telemedicine visit: (“How to better conduct telemedicine visits and how it can better be utilized.”“How to effectively perform physical exams through video chat.”“How to conduct a remote interview and physical.”“How to complete a variety of successful medicine visits to assess, diagnose, and treat patients appropriately.”)• How to triage using telemedicine:“How to triage patients and determine who needs to be immediately evaluated in person.”“Sick versus not sick.”“I would like to learn more about different acute problems that prompt a patient to use telehealth. I would also like to improve my interviewing skills.”	• How to conduct a telemedicine visit: (“How to do a good physical exam virtually and gain comfort with the telemedicine platform.”“I learned how to better examine patients via telemedicine. I also learned better ways of approaching anticipatory guidance and discussing when antibiotics are/ are not needed.”“I learned a lot about the uses of telemedicine, how to conduct a physical exam over video, and improved my communication skills.”“How to conduct interview”)• Appropriate use of telemedicine: (“You can diagnose and treat most conditions over telehealth, but should refer for in-person visits specifically for concern of UTI, ear infections, and strep throat.”“I learned how much can be assessed through telemedicine and what cannot be assessed.”“Bullous impetigo requires oral antibiotics! Be mindful of the lighting when evaluating dermatologic findings, aim for natural light!”)
How do you think this rotation will benefit your medical training?	• Future telemedicine utilizers: (“I anticipate using telemedicine in the future during fellowship and beyond, so I hope to get more comfortable with the platform.”“It will be crucial for me to navigate successful telemedicine appointments as I start my career in primary care and can offer these appointments to my patients.”“Telehealth is a growing part of emergency medicine, my chosen specialty.”)• Satisfy curiosity on telemedicine methodology: (“I hope to improve my interviewing skills and gain knowledge about a system for visits I have not used much before.”“Inform perspective on health care delivery via telemedicine.”“Telemedicine training provided.”)	• Future telemedicine utilizers: (“I think it allow me to be more comfortable with it and incorporate it into my future practice.”“I think this rotation has made me much more comfortable with using telemedicine, which I hope to incorporate into my medical practice after residency.”) • Familiarity with telemedicine will be useful: (“Telemedicine is likely going to be a big part of medicine in the future, so it is good to become familiar with it this early on.”“Additional access to me as a physician for patients for follow-up care”)• Knowledge of telemedicine will make me a better physician: (“Understanding of how to carry out telemedicine appointments creates more well-rounded physicians.”“It provided me insight into the field and increased my comfort performing telemedicine examinations.”“Learning how to be comfortable conducting exams virtually allowed me to refine my exam skills and depend more on history and what I see visually.”)
Name one thing you would like to learn more/learned about that you will use in your practice of pediatric medicine?	• Appropriate use of telemedicine: (“How to best utilize telemedicine”“When to escalate care”)• Characteristics of a telemedicine visit: (“How to efficiently steer the interview”“How to do remote physical”)• Equipment and peripheral devices: (“Use of peripheral devices in the telemedicine appointment”)• General clinical knowledge: (“I would like to become more comfortable with rashes.”“I would like to learn more about the different stages of development.”)	• Clinical elements/ complaints that can be examined virtually: (“How to better identify rashes”“Guidelines for asthma exacerbation management”“Checking a toddler’s eye movements by grabbing objects to track their eyes”)• Procedural elements of virtual care: (“Gained better physical examination maneuvers and tips and tricks for the virtual physical exam”“Virtual physical exams”)• Importance of telemedicine: (“I learned how beneficial providing telehealth is to creating better access to care, and I will offer my patients more opportunities to take advantage of this option.”)• Other: (“I learned the difference between over-the-counter and prescription strength topical steroids, which I hadn’t realized before.”)
Name one part of the training that you thought was most effective in developing your understanding of pediatric telemedicine?	—	• One-on-one time with multiple attending physicians: (“Working with attendings”“I really found it helpful, especially at the beginning, when attendings would share their screens with me to show me how to move through the Epic context.”“Conversations with attendings between patients.”“Getting to work with various attendings to see different approaches in treatment and management.”)• Platform experience: (“Experience, just going to the patient visits both observing and proctored.”“The platform onboarding training!”)• Telemedicine-specific standards and guidelines: (“Reviewing specific telemedicine guidelines for how we approach different diagnoses and chief complaints.”)
Name one part of the training that you thought was least helpful in developing your understanding of telemedicine.	—	• Shadowing visits: (“Shadowing visits toward the end of the rotation”)• Night shift: (“Perhaps working the “night” shifts on telemedicine was the least impactful part of the training.”)• Nothing, all of it was impactful: (“Unsure.”• “n/a” [4 responses])
What did you hope to learn about telemedicine that was not taught in the training?	—	• Peripheral device use: (“How to use peripheral devices for an exam”“I was hoping to learn more about the supplemental instruments used in telemedicine like otoscopes, etc.”)• Interviews: (“How to conduct interview”)• Alternative forms of telemedicine: (“Would have liked to see a provider-to-provider telehealth experience [e.g., consults/referrals]”)• Nothing, all of my goals were met: (“Nothing, my goals for the rotation were met.”• “n/a” [2 responses])

Please see Table 1 for a complete list of survey questions.

On pretest, when asked what role telemedicine plays in health care, the most common theme among answers was access to care (n = 3), followed by triage/redirection (n = 2), patient safety (n = 1), and innovation (n = 1). On the posttest, response themes included access to care (n = 5) and triage (n = 2).

When asked on the pretest what the participants expected to learn from the rotation, the themes included how to conduct a telemedicine visit (n = 4) and how to triage using telemedicine (n = 3). By posttest, responses to what they had learned centered on how to conduct a telemedicine visit (n = 4) and appropriate use of telemedicine (n = 3).

Pretest responses regarding the benefit of this rotation on their medical training generated two themes, including future telemedicine utilization (n = 3) and satisfying curiosity on telemedicine methodology (n = 3). Three themes identified from posttest responses included future telemedicine utilization (n = 2), familiarity with telemedicine will be useful (n = 2), and knowledge of telemedicine will make [me] a better physician (n = 3).

The final question asked what one thing the participant would like to learn more or had learned from the rotation that could be implemented in practicing pediatric medicine. At pretest, response themes included the appropriate use of telemedicine (n = 2), characteristics of a telemedicine visit (n = 2), equipment and peripheral devices (n = 1), and general clinical knowledge (n = 2). By posttest, the response themes included clinical elements/complaints that can be examined virtually (n = 4), procedural elements of virtual care (n = 2), and importance of telemedicine (n = 1).

To gain participants’ recommendations regarding future training, three additional posttest questions asked what was most helpful and least helpful in developing an understanding of pediatric telemedicine and what participants had hoped to learn about telemedicine but were not taught in the training. Participants found one-on-one time with multiple attending physicians (n = 4), platform experience (n = 2), and learning telemedicine-specific standards and guidelines (n = 1) to be most helpful in developing their understanding of pediatric telemedicine. The least helpful aspects of the training, according to responses, were shadowing visits (n = 1) and working the night shift (n = 1); remaining participants indicated all of it was impactful (n = 5). Whereas three participants noted all their goals were met, responses from the remaining four participants were varied with three themes noted. Two participants hoped to learn more about the use of peripheral devices during the training, one hoped to learn more about conducting the interview, and one would have liked to see alternative forms of telemedicine such as “a provider-to-provider telehealth experience (e.g., consults/referrals).”

Discussion

We developed a virtual telemedicine training rotation for residents and medical students as a quality improvement initiative, aiming to build capacity among early career physicians who may choose to implement telemedicine in their future pediatric practice. We found participants came into the rotation possessing knowledge about telemedicine requirements, with survey respondents offering correct responses at pretest to questions about barriers and benefits of telemedicine utilization, necessary documentation, patient satisfaction, and knowledge about federal regulations impacting telemedicine. At pretest, all participants were aware that the COVID-19 pandemic increased patient volumes, and following the rotation, participants had learned that the pandemic improved patient education on telemedicine. The posttest improvement was most notable for understanding the diagnostic limitations of telemedicine.

Regarding practical experience and exposure to telemedicine at pretest, only one participant responded as having prior telemedicine experience. After the program, all participants responded as having formal training. At pretest, only 28.6% of participants believed they could adequately perform a telemedicine examination. Afterward, 100% had gained comfort with the modality. The gained experience and improved comfort level were important as most participants chose the rotation because of their interest and curiosity about telehealth.

Regarding effective program components, students noted that the telemedicine standards and clinical guidelines were most beneficial. Clinical guidelines were then integrated into daily teaching practices to support their use. Participants were asked to review the guidelines corresponding to the patient on the platform. For example, if a patient presented with eye redness and eye discharge, the student performed the virtual history and physical and then was asked to review the conjunctivitis guideline to best understand the assessment and plan. Blending the practical experience with the academic follow-up during the real-time post-visit huddle provided a positive overall experience for the participant.

Participants also noted that the program’s one-on-one preceptor-to-student ratio was effective. The participants felt that working with multiple attending physicians during their elective was beneficial as it exposed them to a wide variety of teaching styles. Therefore, 2 h block times for teaching per attending were implemented. This allowed participants to work alongside four different preceptors during an 8 h shift. Exposure to various teaching methods, real-time review of the clinical telemedicine guidelines, and one-on-one time with several attendings resulted in a positive overall participant experience.

Finally, in evaluating the survey responses, it was evident that some participants interpreted questions differently. For example, in seeking participants’ opinions on whether they believed they could conduct an examination via telemedicine, we received open-ended responses consistent with the information we were seeking (“I think I can do a limited exam via telemedicine”) as well as responses that were inconsistent (“You are able to see general things such as appearance, activity level, etc., but you are unable to auscultate or palpate”). Future iterations of survey questions will be reworded to better target the intended information.

The strengths of our study include a 100% response rate to the digital survey. Given that it is a confidential survey, there is a reduced risk of bias. Feedback was requested shortly after the rotation, diminishing recall bias when answering the posttest survey questions. Study limitations include the small sample size and the assessment of data from a single medical school and pediatric resident training program. Future directions for this project include making program adaptations based on participant feedback, refining the survey questions, and engaging more residents and students to participate in the rotation.

Conclusion

The COVID-19 pandemic has challenged educational institutions to expedite the incorporation of telemedicine into medical school clinical clerkships and residency training programs. Designed within the AAMC telehealth competency guideline framework, ⁸ our virtual telemedicine rotation was inspired to help address the shortfall in available pediatricians prepared to administer health care virtually. Importantly, opportunities for residents and medical students to participate in a telemedicine training rotation may foster comfort with telemedicine and increase the likelihood that they will utilize the virtual modality in their future practice. Whereas other pediatric teaching programs use online modules and virtual workshops, our telehealth teaching program is based on live, real-time clinical encounters.^17–20 Our urgent care pediatric platform is multiperson, where participants, preceptors, and patients meet from different locations in a single virtual room. At every clinical encounter, students can access and review the patient’s electronic medical record and request assistance from translators if needed. The pre-and post-visit huddle and one-on-one preceptor-to-participant teaching model enable individualized and tailored instruction based on the AAMC telehealth competencies. Our program uses participant survey feedback for continuous improvement and ongoing refinement. Telemedicine rotation curricula should be revised regularly based on participant feedback to ensure that the content appropriately challenges participants. Using a quality improvement lens to facilitate the ongoing refinement of our program, we hope to benefit the next generation of pediatricians by exposing and training them in this virtual model of care, refining and improving our teaching methods, and offering a foundation for other telemedicine training programs.