Surgical Training Using Augmented Reality-Based Optical Head-Mounted Displays: A Pilot Study

Abstract

While the educational benefits of the modern techniques such as virtual reality (VR) or augmented reality (AR) have been suggested, there is still a lack of reports on actual surgeons’ experiences. In this study, we evaluated the effectiveness of a holographic AR-based surgical training in tonsillectomy. Two otolaryngologists, 1 trainee and 1 instructor, performed 5 tonsillectomies using an AR headset (HoloLens 2, Microsoft, USA). The trainee wore the AR headset to share the surgical view through front camera while the instructor remotely accessed the device using the Microsoft Teams program and provided real-time guidance. The AR-based surgical training offered several advantages, including direct real-time guidance for the trainee and clear instructions without disturbing the surgical process. However, there were also drawbacks, such as the front camera not always matching the trainee’s view and some difficulty with focusing, depending on the depth of the oral cavity. Our study suggests that AR devices are a feasible and alternative method for surgical training. With the ability to provide clear guidance, even from a distance, this technology has the potential to revolutionize surgical training in the future.

Introduction

The development of an effective training method to produce skilled surgeons has been a long-standing concern in the field of surgery. Traditionally, the apprenticeship model, where a trainee is mentored by an experienced surgeon, has been the predominant method of surgical education and is still in use today. However, as the field of surgery has become more specialized and diverse, techniques are constantly evolving, and more efficient and effective methods of surgical education are needed. One such method is the use of virtual reality (VR) or augmented reality (AR)-based technology simulation tools.^1-3

VR is the use of computer technology to generate a simulated environment with scenes and objects that seem real, allowing users to feel immersed in their surroundings. Recent developments in haptics, display technology, and motion-sensing techniques have enabled users to experience more realistic interaction. The VR process has been used in various medical fields and is ideally used for training in hands-on procedures. It shows its effectiveness in the education of surgical procedures or other interventional procedures.^4,5

While VR is an all-surrounding artificial environment that totally obstructs the real world, AR enhances users’ real-world views by combining virtual and artificial objects. Users can obtain additional information that is difficult to obtain only from the real world by superimposing virtual information on the actual world. Furthermore, AR makes it possible to interact between users in real time through an interface. In medical educational programs, AR provides more efficient information by projecting and combining information on the real-world environment in real time, allowing users to understand physiological mechanisms or anatomical structures. ⁶ The development of displays shows a new paradigm of AR application. In the early days, a mobile device with an LCD display was mainly used. However, it could not sufficiently reflect the actual view due to its small display size. Recently, many technologies using separate wearable displays such as head-mounted displays have been developed to overcome these limitations.

Numerous studies have reported on the educational benefits of VR and AR technologies in various fields, highlighting their potential for educational applications. ⁶ However, most of these studies have focused on VR technology, which has been limited to understanding anatomy and virtually performing surgical methods.^3,7 While some recent studies have shown that AR can improve surgical efficiency, the latest display technology, such as a head-mounted display, is not widely used in the medical field.^8,9 Moreover, there have been few reports on the use of AR technology in otolaryngologic surgical fields due to the narrow and deep nature of the surgical field, which makes it challenging for assistants to observe the surgical area clearly. As a result, learning surgery in detail and guiding less-skilled surgeons can be difficult. However, we believed that these challenges could be overcome through real-time video chat and view-sharing technology via AR. Therefore, we conducted a study to evaluate the feasibility and effectiveness of AR-based surgical training in the otolaryngologic field by sharing the surgical field and using real-time guidance to overcome the limitations of traditional surgical training methods.

Methods

A commercially available AR headset was used in this study (HoloLens 2, Microsoft, USA). We applied AR headset-based surgical training to tonsillectomy, one of most common surgeries in the otolaryngology field. To teach the surgical procedure, we matched 2 otolaryngologists; one is an experienced instructor, and the other is a trainee with limited experience in the procedure. All tonsillectomies were performed by the same pair of instructor and trainee. The trainee performed the tonsillectomy while wearing the AR headset, and the instructor remotely accessed the AR headset through communication platform software (Microsoft Teams®, Microsoft, USA) from an office away from the operating room. The trainee’s surgical view while wearing the AR headset was shared with the instructor through this software as the camera is located on the front of the AR headset. The instructor gave comments to the trainee while watching the video shared through video chat. The tonsillectomies were performed following the standard procedure, with no differences in surgical technique or equipment except for the use of the AR headset.

After completing each surgery, we recorded the operation time and the instructor and trainee discussed the advantages and disadvantages of using AR headsets freely and reviewed operation video. Furthermore, we conducted outpatient follow-up observations for 2 weeks to ensure that no additional complications occurred, following the same protocol as other patients.

This study protocol was approved by Institutional review board of Hanyang University Guri Hospital (HUGH-2023-01-029) and informed consent was obtained from all patients included in the study.

Operative Results and Feedback

The trainee put on the AR headset before the surgery (Figure 1A) and tonsillectomy was prepared as usual under general anesthesia (Figure 1B). We initially intended to use the conventional headlight, but due to the bulk of the headset, it could not be worn on the head. As an alternative, we used the light attached to the loupes. After preparing for the surgery, the trainee accessed the video chat with the instructor and checked the 2-way communication system (Figure 1C). The trainee shared the surgical field of view through the communication software and could talk to the instructor on a video chat through the AR screen. During the surgery observation, the instructor taught the trainee the correct steps for the surgery to proceed by drawing or pointing to the AR surgical field. The instructor’s drawings on the screen appear on the trainee’s AR view in real time (Figure 2) (Supplemental video).

Figure 1.

The trainee with AR headset and video chat. (A) The trainee wearing the holographic AR-based optical head-mounted display headset. (B) Tonsillectomy was performed wearing the AR headset. (C) After preparing the surgery, the trainee accessed a video chat with the instructor and checked system working for 2-way communication. (Video chat window on the head up display of the AR headset.).

Figure 2.

The shared surgical view through a front camera of the headset. (A) Guiding the incision line (yellow line). (B) Pointing the part to be cut (blue arrow).

A total of 5 cases of bilateral tonsillectomy for chronic tonsillitis were performed under the AR-based training. All operations were completed by a trainee under the supervision of an instructor through AR-based training without cessation of the operation. The mean operation time was 42.2 ± 8.0 (35-56) minutes. There were no serious complications, including massive or postoperative bleeding or taste loss. After completing each surgery, the instructor and trainee reviewed the recorded operation video and discussed about the advantage and disadvantage of AR-based surgical training. Feedback from the instructor and trainee was summarized in Table 1.

Table 1.

Feedback After Operation With AR Headset.

Advantage	Disadvantage
✓ High resolution video ✓ Smooth real-time guidance without interruption ✓ Clear guidance in the actual view using hologram display ✓ Sharing the actual field of view in real time ✓ Feels more independent ✓ Able to guide regardless of the location ✓ No differences in surgery time	✓ Takes time for adaptation ✓ Heavy headset ✓ Misalignment of displaying instructions ✓ Out of focus ✓ Mismatch between the actual view and the camera

Abbreviation: AR, augmented reality.

Discussion

This study demonstrates the feasibility of AR-based surgical training in the otolaryngology field. The trainee effectively shared the surgical view remotely with the instructor and received immediate and clear feedback. Furthermore, this technique could be applied to other surgeries because tonsillectomy, which has a narrow and deep surgical field can be performed with this technique.

Although VR and AR are widely used in other industries, the medical community still primarily relies on 2D-based information sharing. Due to the ethical nature of medicine, it is challenging to freely apply new technologies. Nevertheless, the surgical application of modern techniques has been attempted through various studies. Most of the studies have been performed in spinal surgeries using the AR navigation system and a head-mounted display.^10-12 Huan Liu et al. tried an intraoperative CT image-based AR-guided method in percutaneous lumbar pedicle screw placements. ¹⁰ They reported that AR-guided percutaneous lumbar pedicle screw placements were significantly more efficient than radiograph-guided placements. Wei P. et al. assessed the clinical outcome of 40 cases of percutaneous kyphoplasty assisted with AR and VR. ¹² The study revealed that technology-assisted percutaneous kyphoplasty accurately orientated the injection point, leading to improved results and pain relief. Burstrom et al. conducted a systemic review of 28 studies of spinal surgery with AR navigation and analyzed the accuracy, radiation dose, and cost-effectiveness. ¹³ The results showed both higher accuracy and lower radiation exposure in AR-navigated spinal surgery.

However, most previous AR systems had limitations in applying AR to various surgeries. Ahmad et al. summarized the history of wearable devices with AR application in spinal surgery and discussed the feasibility of AR technology. ¹⁴ They mainly focused on head up display (HUD) technology. A HUD is a transparent screen that provides information without requiring users to shift their focus from their regular viewpoint. It is widely used in various aspects such as aircraft, automobiles, and other professional areas. Early devices required complex and bulky external monitors or navigation devices, which made it difficult to apply in real surgery. A new concept that overcomes this limitation is HUD technology that uses a hologram of a patient’s anatomy and reflects it directly on the surgical field. HUD using holograms can track the surgeon’s head and eye movement and overlay images onto the exact surgical field. They asserted that the implementation of AR in spinal surgery has resulted in improved surgical accuracy and efficiency. Beyond spinal surgery, the AR system could be widely applied to other medical fields with the advancement of HUD. Wang et al. performed a study with Microsoft HoloLens to explore the feasibility of an ultrasound telemedicine mentoring system. ¹⁵ They captured the mentor’s hand gestures and virtually displayed them in the AR space of the HoloLens. They reported that the performance of the AR setup using the HoloLens is comparable to a full telemedicine setup that consists of a full overhead view of the entire patient room captured through a pan-tilt-zoom camera near the ceiling and a second view of the patient captured from a webcam placed on the ultrasound machine by analyzing feedback from the trainees, mentor, and objective observers.

Although there have been many experimental studies on the application of AR technology in orthopedics and neurosurgery fields, it has rarely been adopted to actual surgery, and fewer cases have been used for educational purposes. Based on our knowledge, there are few reports of using AR technology in otolaryngology surgery. Chen et al. reviewed the application of AR and head-mounted display in the otology and neurology, revealing a total of 18 articles published from 2007 to 2022. ¹⁶ However, 14 of the 18 studies were limited to the use of AR only in surgical navigation. There was only one report using AR in training for residents, but still, this study was only conducted in a cadaveric laboratory, not performed on actual patients and surgery. ¹⁷ Our study shows the feasibility of AR technology in surgical view sharing and surgical training. First, the biggest advantage for learners is that they can receive clear comments on surgery in real time through video chat. Since tonsillectomy has narrow and deep surgical field, sharing the surgical view and providing guidance from the instructor through the AR headset is more convenient than guiding from beside the trainee. The trainee felt like he was performing the surgery more independently since there was no one right next to him, but at the same time, he was able to feel confident as direct comments were made during surgery. Secondly, the instructor can display his instructions clearly on the trainee’s view through AR, without interfering with the surgery. When the instructor teaches, an operation that has a narrow surgical field may stop, and it can be difficult for the trainee to see the instruction in detail, whereas in AR-based training, instructions are shared through the holographic technology without interference. Moreover, the directions that the instructor marks are fixed in the initial position regardless of the trainee’s head position because AR headset recognizes the space and head motion. This is a big advantage and point of difference compared to a general video chat headset. Lastly, it is useful to share and guide surgery without the restriction of space through remote access. It can be expected that doctors in other hospitals and even in other countries will be able to communicate smoothly during surgery with this technique.

Nevertheless, there are some drawbacks and areas for improvement. Firstly, the view from the front camera did not exactly match the surgeon’s line of vision since the camera of the AR headset was located on the forehead, slightly above the eyes. Therefore, the surgeon should pay attention to the camera’s position. In addition, the camera sometimes focused on the hands of the surgeon or the patient’s mouth because the surgical field was narrow and deep, although the out-of-focus time was brief, usually only 2 to 3 seconds. Secondly, the weight of the headset can be a burden on the surgeon. Although the headset weighed 566 g, it could become heavy if the surgeon wears it for a long time. Of course, the AR headset could be used for a short time only during difficult procedures, but considering its convenience in real surgery, it is thought that these limitations should be improved. Since the AR headset is marketed for general industrial use, it would be useful to develop a medical AR headset that overcomes these limitations. As our study shows the feasibility and practicality of AR technology with HUD, it could be more widely used in medical fields with advanced AR headset.

In conclusion, AR-based surgical training of is a feasible and alternative method for surgical training, which may be especially useful in surgeries with a deep, narrow field. Although we have experienced only a small number of cases, we think that we showed the usefulness and potential of AR headsets for surgical training. An instructor can provide real-time feedback without space constraints through this technique. We anticipate that this method could be applied to numerous surgical fields, ultimately leading to more effective and efficient surgical training in the near future.