An Augmented Vision of Our Medical and Surgical Future,Today?

Abstract

Incorporating consumer electronics into the operating room, we evaluated the Apple Vision Pro (AVP) during limb preservation surgeries, just as we evaluated Google Glass and FaceTime more than a decade ago. Although AVP’s real-time mixed-reality data overlay and controls offer potential enhancements to surgical precision and team communication, our assessment recognized limitations in adapting consumer technology to clinical environments. The initial use facilitated intraoperative decision-making and educational interactions with trainees. The current mixed-reality pass-through resolution allows for input but not for highly dexterous surgical interactions. These early observations indicate that while AVP may soon improve aspects of surgical performance and education, further iteration, evaluation, and experience are needed to fully understand its impact on patient outcomes and to refine its integration into clinical practice.

Introduction

Every second, someone around the world develops a diabetic foot ulcer. Half of these become infected. Twenty percent of these infections end up in hospital. This yields the reality of an amputation every 20 seconds.^1,2 Over the past generation, a shift away from reactive and ablative surgeries such as amputation toward more proactive and reconstructive surgeries such as revascularization and foot reconstruction has become more common. These procedures require a high degree of coordination, communication, and interdisciplinarity.^3-5 Over these years, we have documented the use of consumer electronics in limb preservation procedures. In 2011, we discussed the use of FaceTime (Apple, Cupertino, California) for rapid assessment of patients between clinicians. ⁶ In 2014, we documented the use of Google Glass (Google, Mountain View, California) to assist in real-time intraoperative communication. ⁷ In per-pandemic 2020, we reviewed the successful use of WhatsApp to assist in a home-based application of larvae to assist in complex limb preservation when hospital-based resources were not available. ⁸

In the evolving landscape of medical technology, the integration of consumer electronics into the surgical environment has seen a progressive shift toward more advanced and intuitive tools. Recently, the Apple Vision Pro (AVP) (Apple) stands out as a potential significant iterative step forward. Products like the AVP have been described as “video see-through head-mounted display[s]” (VST-HMDs) compared with an optical see through (OST) HMDs like the Google Glass (Google, Mountain View, California).^7,9 This article builds on our team’s history of utilizing consumer electronics—from FaceTime to Google Glass—to enhance medical and surgical care.^6,7 The AVP’s potential in improving surgical outcomes, particularly in the challenging field of limb preservation, is a testament to the growing synergy between technology and healthcare.

Description of Apple Vision Pro (Figure 1)

The AVP is equipped with a dual micro-organic light-emitting diode (OLED) display system, each providing a resolution of 23 million pixels, which may ultimately assist in detailed visual tasks such as those in surgical settings or pre/perioperative review. ¹⁰ The pixel pitch is 7.5 microns, and the displays cover 92% of the DCI-P3 color gamut. The device supports variable refresh rates of 90, 96, and 100 Hz, which can be critical for maintaining visual fidelity during various procedures. Internally, the Vision Pro is powered by an Apple M2 chip, which includes an 8-core CPU and a 10-core GPU, complemented by a 16-core Neural Engine. The system has 16 GB of unified memory and offers a memory bandwidth of 256 GB/s. The device has a host of sensors, including light detection and ranging (LiDAR), twelve cameras, six microphones, and additional sensors like flicker and ambient light sensors. The LiDAR sensor, used for precise measurements, allows for placement of digital objects within the user’s space. The camera system features two high-resolution main cameras and six world-facing tracking cameras that capture the environment in 3D. The main camera is a stereoscopic 3D system with an 18 mm focal length and an f/2.00 aperture, capable of capturing 6.5 stereo megapixels. Internally, four infrared eye-tracking cameras monitor the user’s gaze, enabling content selection and engagement. A TrueDepth camera, which has already been employed in Apple’s facial recognition technology, supports iris-based authentication. There are four internal inertial measurement units that aid in maintaining orientation and spatial positioning. In addition, flicker and ambient light sensors refine visual quality by adjusting display brightness and reducing flicker from real-world artificial lighting. The device has a 600 g headset and an external battery pack which was waist-mounted in this instance. The battery provides up to two hours of general use and the device includes Wi-Fi and Bluetooth capabilities. ¹¹ Figure 2 depicts the surgeon’s view with examples of an overlay of images and data which can be manipulated using hands-free gestures. These images/apps remain in place in the virtual overlay even as the surgeon/clinician moves through space. This is further illustrated in Supplemental Video 1.

Figure 1.

Apple Vision Pro.

Figure 2.

Surgeon view, entering operating room with mixed-reality overlay of imaging and pre-operative photography.

Applications

The AVP was recently used in two cases involving a multidisciplinary team of podiatric, orthopedic, and vascular surgeons working in concert during limb salvage surgeries. The first of two cases was a 64-year-old woman with diabetes, neuropathy, and midfoot and rearfoot Charcot arthropathy with medial tissue loss (Figure 3). The procedure involved placement of an Ilizarov external fixator (frame) for Charcot reconstruction and was documented with the aid of the AVP. The technology allowed for a nuanced understanding and communication of the complex, 3D surgical steps involved in real-time, a critical component in the educational continuum of limb reconstruction (Figure 3).

Figure 3.

Photos taken during procedure wearing AVP. High visual resolution evident during the first and second cases with spatial photograph. Similarly detailed video is available. All is visible in 3D when wearing device.

The second case focused on a limb preservation procedure in a 62-year-old man with a long-standing history of diabetes mellitus, peripheral artery disease, and hallux gangrene. The focus of the index procedure was aimed at eradicating osteomyelitis through the use of antibiotic-loaded cement. This surgery was performed 24 hours following a distal angioplasty procedure aimed at limb salvage (Figure 3). The AVP’s advanced imaging capabilities afforded the team a real-time view of blood flow, informing understanding of vascular runoff and aiding in the strategic planning of the surgical approach (Figures 4 and 5).

Figure 4.

The data displays were rotated throughout the second case in order to provide views of the pre-surgical planning images and to interact and guide the medical student during the procedure.

Figure 5.

Post-procedure educational lecture with surgical resident and medical student.

The use of AVP in the operating room yielded notable improvements in surgical precision and intraoperative decision-making. The device’s ability to provide high-definition visualization and data overlay enhanced the surgeons’ perception and understanding of complex anatomical structures and surgical dynamics. This was particularly evident in the 3D deformity management of Charcot reconstruction and the assessment of highly complex vascular integrity post-angioplasty in each respective case.

Discussion

Augmented reality has been a source of interest for surgeons and other clinicians since the 1990s but has been limited by various factors such as cost, applicability, and capability.^12,13 The AVP represents an iterative step toward the widespread adoption of augmented reality in surgery. Current uses of HMDs can be divided into communication and documentation, training and education, and data display and imaging guidance. ¹⁴ In our experience, we were able to use the AVP to document operations, guide trainees based in another country throughout the case, and display pre-surgical planning data in real-time—all simultaneously.

Our group was the first to utilize FaceTime and Google Glass in the setting of limb salvage.^6,7 FaceTime marked an initial step in integrating digital communication tools into surgical operations, allowing for real-time video consultations between surgeons and remote experts. This rudimentary application provided a way to overcome geographical barriers, enhancing decision-making and educational opportunities during procedures without complex equipment. Then, the introduction of Google Glass represented a significant advancement by incorporating wearable technology that offered hands-free communication and information access. Surgeons could view patient data, images, and consult with colleagues through a head-mounted display (HMD), reducing the need to divert attention from the surgical field, facilitating a more interactive approach to accessing and sharing information during surgery.

In our estimates, the realm of communication and documentation is the most likely to be integrated into the OR first. For example, HMDs were utilized to improve quality assurance in organ procurement by allowing off-site members of the transplant team to assess the donor organ quality. ¹⁵ Head-mounted displays can also play a role in providing remote surgical or nonsurgical consultations.^16,17 With an increasing burden of documentation due to regulatory requirements, HMDs can also aid in the field of medicolegal documentation.^18,19 Future software can also translate the video into a written operative report for efficient summarization. Furthermore, the use of HMDs in the context of surgical training has been well documented in which an attending physician would wear the HMD and the video would be streamed to trainees in real- time or in which the trainee wore the HMD and received feedback during the video analysis.^14,20 Furthermore, as noted with other HMDs, AVP provides the optimal vantage point for trainees to see from the operator’s perspective (Figure 3). ²¹ Our team was able to walk through a medical student during the second case and have a post-operative educational debrief with both a medical student and surgical resident by watching the videos of the case (Figures 4 and 5). Another potentially practical and promising feature is the use of data displays and imaging guidance. Our team utilized Smart Perfusion (Phillips, Amsterdam, The Netherlands) to provide surgical guidance during the second case. Although the image was static, future software updates can provide dynamic analysis of pre-surgical imaging during the operation, reducing the burden of having to step away from the operating field to consult with the desktop application. In addition, the relative fluidity of the hand gestures to control the AVP and related applications is a notable advantage to other past products.^8,14 An example of dynamic use of data displays and image guidance is when one group utilized HMDs in performing vascular pedunculated flaps of the lower extremities. They overlayed pre-operative computed tomography angiography (CTA) images onto the patient’s body, enabling surgeons to visualize the underlying anatomy without incisions. ²¹ Three-dimensional models of osseous, vascular, and soft tissue structures were generated from CTA scans and correlated intraoperatively to the patient, aiding in the accurate identification and dissection of the perforating vessels. Preliminary results showed this modality was comparable with traditional Doppler ultrasound methods. Other reported uses include real-time monitoring of vital signs or electromyography responses during operations, limiting the need of the surgeon to frequently change his field of vision and improving response times.^22,23 Future uses of data displays and imaging guidance for AVP in limb salvage surgeries are promising. For example, AVP’s advanced sensors, such as LiDAR and infrared sensor, along with future additional sensors can be used for dynamic imaging analysis. This can allow for the real-time analysis and overlays of blood flow and tissue conditions during surgery, helping to identify potential complications early. In addition, the design of custom prosthetics and implants benefits significantly from AVP. By leveraging pre-operative and post-operative scans, surgeons can design and 3D print prosthetics and implants that perfectly match a patient’s anatomy. Regarding comparisons and limitations among various HMDs, OST HMDs have the advantage of offering a natural view of the world through a semi-transparent display. However, with current available commercial devices, the “spatial coherence” is suboptimal because the virtual content has to be perfectly aligned with the natural view of the environment. ⁹ In addition, the discrepancy between the 4D real world and the 2D virtual content on the display leads to perceptual issues such as difficulties with depth of field perception. Although VST-HMDs alleviate some of the aforementioned problems, the reliance on this technology can pose a safety risk if there is a system malfunction in which the operators view is blocked, especially during a critical moment in the operation which is not a concern in OST HMDs. In addition, the field of view specifically for the AVP is limited at roughly 100 degrees.

Limitations of the Apple Vision Pro

Although controlling the AVP was notably more fluid and straightforward than with other HMDs, refinement of the hand gesture control system is necessary. This refinement will ensure that control inputs are detected on the first attempt. In addition, further refinements can improve the economy of movement, allowing fewer gestures to produce the desired outcome. In the future, the AVP could detect hand gestures in the middle of surgical procedures, eliminating the need to put down a surgical instrument to change the controls on the AVP. Other general limitations of HMDs include the weight, battery life, and reliance on wireless Internet or Bluetooth connections for imaging transmissions. Furthermore, Figure 3a-c depicts the high-quality spatial images captured in the surgical field. However, the actual current pass-through real-time view for the surgeon has less high definition, which is a limitation, particularly when compared with the high quality in the spatial photos/video. Advantages and disadvantages of the current iteration of the AVP are detailed in Table 1.

Table 1.

Advantages and Disadvantages of the Current Iteration of the AVP.

Aspect	Advantages	Disadvantages
Surgical Precision	Enhanced visualization and understanding of complex anatomical structures and surgical dynamics. Photos and videos are high definition	Limited by current real-time mixed-reality pass-through resolution for highly dexterous surgical interactions
Educational Value	Facilitates real-time educational interactions with trainees, improving the educational continuum of limb reconstruction	Requires further iteration and evaluation to fully integrate into clinical practice for educational purposes
Communication	Improves intraoperative decision-making and team communication through real-time mixed-reality data overlay and controls	Adaptation to clinical environments may present challenges in communication without further refinement
Technology Integration	Represents an iterative step forward in integrating consumer electronics into the surgical environment	Limitations in adapting consumer technology to clinical environments need to be addressed
Battery Life	External battery pack provides up to two hours of general use	Limited battery life may restrict longer surgical procedures without an additional power source. It is not “hot swappable”
Form Factor	The device is notably more comfortable than other HMDs	The device has a 600 g headset, which may be considered heavy for prolonged use
Connectivity	Includes Wi-Fi and Bluetooth capabilities for seamless integration and data transmission	Reliance on wireless Internet or Bluetooth connections for imaging transmissions could be a limitation in environments with poor connectivity
Cost	Up to $5000 USD may be less than some hospital-based visualization systems	Up to $5000 USD with accessories may be out of the reach of some clinics and clinicians

Abbreviations: HMD, head-mounted display.

Conclusions

The AVP has further evolved surgical technology integration by leveraging augmented reality to superimpose digital information, such as pre-operative scans and anatomical structures, directly onto the surgeon’s field of view. This advancement provided an unprecedented level of precision and context, allowing surgeons to “see through” the patient’s skin and directly visualize critical information without interrupting the workflow. The AVP’s immersive AR capabilities, especially in the field of limb salvage, mark a significant leap toward enhancing surgical accuracy, efficiency, and safety.

Supplemental Material

sj-docx-1-dst-10.1177_19322968241236458 – Supplemental material for An Augmented Vision of Our Medical and Surgical Future, Today?

Supplemental material, sj-docx-1-dst-10.1177_19322968241236458 for An Augmented Vision of Our Medical and Surgical Future, Today? by David G. Armstrong, Sebouh Bazikian, Alexandria A. Armstrong, Giacomo Clerici, Andrea Casini and Anand Pillai in Journal of Diabetes Science and Technology