VolumetricOR: A New Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education

Abstract

Background

Surgical training is primarily carried out through observation during assistance or on-site classes, by watching videos as well as by different formats of simulation. The simulation of physical presence in the operating theatre in virtual reality might complement these necessary experiences. A prerequisite is a new education concept for virtual classes that communicates the unique workflows and decision-making paths of surgical health professions (i.e. surgeons, anesthesiologists and surgical assistants) in an authentic and immersive way. For this project, media scientists, designers and surgeons worked together to develop the foundations for new ways of conveying knowledge using virtual reality in surgery.

Materials and method

A technical workflow to record and present volumetric videos of surgical interventions in a photorealistic virtual operating room was developed. Situated in the virtual reality demonstrator called VolumetricOR, users can experience and navigate through surgical workflows as if they are physically present. The concept is compared with traditional video-based formats of digital simulation in surgical training.

Results

VolumetricOR let trainees experience surgical action and workflows (a) three-dimensionally, (b) from any perspective and (c) in real scale. This improves the linking of theoretical expertise and practical application of knowledge and shifts the learning experience from observation to participation.

Discussion

Volumetric training environments allow trainees to acquire procedural knowledge before going to the operating room and could improve the efficiency and quality of the learning and training process for professional staff by communicating techniques and workflows when the possibilities of training on-site are limited.

Background: The Simulation Paradigm in Surgical Education

Today, surgical training with the exchange of surgical skills and experience is primarily carried out through observation during assistance or on-site classes, by presenting videos, for instance, at conferences or online as well as by different formats of simulation. These methods are essential to connect theory and practice in order to transfer knowledge into skills. Observing is an essential part of life-long surgical education and remains valuable for the provision of care. However, digital media fosters newly formatted ways of training and educating that challenge and expand the traditional ways of knowledge transfer. Today, it is widely accepted that digital training environments can improve operating room performance.^1,2 Digital training particularly expands the scope of video-based education as well as of existing forms of physical simulations.^3,4 Besides the well-established methods of simulation that rely on mannequins, role-playing games, standardized patients or cadavers, digital media has yielded a variety of new formats that enhance the range of skills and competencies of simulation-based education. Especially the field of virtual reality simulation allows trainees and residents to develop clinical competence by experiencing working situations in an authentic way. Virtual reality promises a more effective bridging between theoretical and participatory forms of learning and training, as it allows physicians to acquire knowledge and practical skills outside of the operating room.

Historically the rise of digital simulation is often associated with the decline of the apprenticeship model of surgical residency training^5,6 devised by William Stewart Halsted at Johns Hopkins Hospital in Baltimore in the late 19th century. ⁷ Halsted established an opportunity-based training approach of ‘see one, do one, teach one’ that was built upon the subjective as well as the time and case dependent knowledge transfer between the resident and the supervising surgeon. This type of on-site class within the operating room is still relevant and will remain valuable to help physicians and trainees learn and improve surgical skills and workflows. But given the limited availability and scalability of the possibilities of participatory observation and assisted intervention, the operating room no longer seems to be the most efficient environment for knowledge transfer to junior surgeons and colleagues. ⁸ This has a number of reasons: The demand for digital training environments is fostered by constraints of the economic, social and regulatory aspects of surgical training and practice. ⁸ On the economic level, the pressure for increased productivity has shortened operating room time. ⁹ The implementation of work-hour regulations has further diminished the resources available for the time-consuming process of transferring knowledge in the operating room. ¹⁰ As an effect, operating room time has become ‘too valuable to permit acquisition of basic technical skills’ ¹¹ and ‘residents are often relegated to roles as mere observers’. ¹² This limitation is further emphasized with the availability of image-guided and computer-assisted procedures that physically limit the ability to supervise surgeons to guide interventions. Especially in robot-assisted procedures the surgeon’s attention and focus shift from the operating table to computer terminals that limit knowledge exchange, impede a shared view of a situation and leave less space for cooperation. Altogether these developments have a profound impact on the quality of education and the efficiency of surgical care, which become visible and are further intensified by the restrictions posed upon clinical infrastructures by the COVID-19 pandemic. In order to cope with these challenges of surgical training and knowledge transfer, hospitals need to enhance their existing training facilities with digital tools and environments.

With the goal to provide a new method for knowledge exchange, VolumetricOR proposes such a digital solution for the simulation of surgical workflows based on the combination of virtual reality and 4D video. The concept of VolumetricOR is based on a workflow to transfer real surgical interventions in the operating room into a virtual reality environment without the need for cost-intensive hardware and software solutions or time-consuming post-production. The developed prototype should demonstrate how virtual reality can serve as a digital classroom to simulate the complex interaction and collaboration of surgical health professions, that is, surgeons, anesthesiologists and surgical assistants in a virtual operating room. It offers an immersive and interactive education and training tool for students and staff with the objective to improve and enhance the mutual understanding of the unique workflows and decision-making paths of each individual team member (see video abstract in supplemental material or via https://youtu.be/jFQOU1nyThI).

Materials and Methods

The concept of VolumetricOR is based on a virtual reality application, that has been developed during an interdisciplinary research collaboration between surgery, design and media studies at the Department of Surgery at Charité – Universitätsmedizin Berlin and the Cluster of Excellence Image Knowledge Gestaltung at Humboldt-Universität zu Berlin. The virtual reality application lets users explore an operation room at Charité – Universitätsmedizin Berlin and navigate through sequences of a living kidney donation surgery. By wearing a virtual reality headset VolumetricOR places users within the surgical scene in a way that lets them experience surgical action and workflows three-dimensionally, from any perspective and in real scale. The core of VolumetricOR is a technical workflow that consists in 3 elements: a) a photogrammetric reconstruction of an operating room at Charité – Universitätsmedizin Berlin, Campus Mitte, b) volumetric video sequences which are recorded directly in the operating room and c) location-based metadata that adds a digital layer of interactive information inside the surgical scene.

Photogrammetry, a photography-based image measurement method in which the spatial position of objects is reconstructed from the coordinates of overlapping pixels within photos was used: The photos of the operating room are transformed into a 3D model using the software Metashape (Agisoft Inc., St. Petersburg, Russia.). The 3D model is then imported into the game engine Unreal Engine (Epic Games Inc., Cary, USA), a software development environment to build video games (Figure 1). This digital representation of an operating room serves as the scene for displaying volumetric video sequences and additional metadata. Using any SteamVR compatible virtual reality headset, such as Vive (HTC Inc., New Taipei, Taiwan) users can experience the model in real scale, from any perspective, and in a photorealistic manner.

Figure 1.

Photogrammetry model of the operating room at Campus Mitte, Charité - Universitätsmedizin Berlin (a); detail (b) with wireframe model overlay (c).

In order to represent the actions and routines of the surgical staff, we developed a workflow to capture dynamic three-dimensional scenes in the operating room (Figure 2). For this purpose, we combined the spatial information of depth sensors (Kinect, Microsoft Corporation, Redmond, U.S. and Realsense, Intel Corporation, Santa Clara, U.S.) with the surface textures of a video camera (EOS 80D, Canon Inc., Tokyo, Japan). The result is a spatial video that presents a surgical scene in 4D, viewable from any perspective. This so-called volumetric video is an emerging image format that provides the ability to capture and display spatial information in a video-based format. This method of recording surgical interventions is a new approach to convey space, that goes beyond traditional formats of video-based recording of surgical interventions. The core of the workflow is the software Depthkit (Scatter Studios LLC, Brooklyn, U.S.) that consists in a volumetric capture module and a volumetric post-processing module. The capture module records the surgical scene using multiple depth sensors, so-called RGBD cameras and video cameras. The depth sensors record spatial information as a point cloud and the video camera provides colour information in the form of a texture as well as an audio signal. Each individual sensor-camera combination is connected to a microcomputer and is controlled by a server computer outside of the operating room using the software TeamViewer (TeamViewer GmbH, Göppingen, Germany).

Figure 2.

Workflow overview of volumetric capture: recording (a), image fusion (b), quality improvement (c), VR export in game engine (d).

After the recording, the data are imported in the post-processing software module in order to fuse depth information and colour values. For this purpose, the video data are mapped as surface textures onto the depth information. For this step (the so-called image fusion) the point cloud data of the RGBD camera are converted into a polygon net (a so-called mesh) in order to form a closed surface on which the colour information can be projected as RGB images. An optional step is the quality improvement of the video by removing artifacts using the software 3ds Max (Autodesk Inc., San Rafael, U.S.) and FFmpeg (Fabrice Bellard). In the last step the data of each individual frame are exported to the game engine Unreal Engine in the alembic format, where the volumetric video of each sensor module is combined and is then integrated into the photogrammetric scene (Figure 3).

Figure 3.

Volumetric recording of a living donor kidney transplantation procedure at Charité - Universitätsmedizin Berlin (a), embodied learning - users can view the surgical scene in real space and scale (b,c).

In order to be able to interact with the scene, users can access a layer of metadata and context information. With the controllers of the VR headset Vive (HTC Inc., New Taipei, Taiwan) they can navigate through the scene based on an interactive interface that lets them select, stop, loop and rewind the sequences of the intervention, identify objects and areas, such as the sterile zone and select points of interest to access additional metadata via interactive text boxes, that adapt to each user’s position (Figure 4), such as information on tools or machines. In this way, users can get familiar with instrument arrangements and positions as well as with other site-specific objects and infrastructure. Location-based graphical metadata overlays allow for a context-sensitive access to learning content that can be adapted to the hospital’s infrastructure and working situations.

Figure 4.

User interface overlay for navigating through the surgical workflow (a), and points of interest, that offer additional information by selecting spatial markers via a “+” button (b,c).

Results: Embodied Training in Virtual Reality

The concept of VolumetricOR allows hospitals to transfer surgical training from the operating room to a virtual reality environment for comparatively low costs of approximately 5000 US Dollars covering expenses for software licenses, depth sensors, cameras and computers. Based on the concept and technical setup of VolumetricOR hospitals can create virtual classrooms based on their original operating rooms, technical infrastructure and individual workflows in order to improve the acquisition of surgical skills. Studies have already shown that virtual reality simulations provide a more immersive, interactive and authentic experience of surgical action and workflows than other forms of simulation, which places users in front of a monitor.^{4,8,9,13-20,23,25} They extend video-based formats of digital education and training as trainees can experience surgical action and workflows (a) three-dimensionally, (b) from any perspective and (c) in real scale. VolumetricOR further extends this potential by offering the integration of video into virtual reality simulations. This has 3 major benefits for the use of videos in surgical education and training:

First-Person Perspective

Most video-based education concepts, such as videos presented at conferences or surgical learning platforms primarily show the operating field in order to present specific techniques and procedural steps. Volumetric videos expand this limited experience of the surgical situation, as they show the surgical workspace from the individual perspective of the user. This distinguishes them from conventional and even stereoscopic video formats, such as 360-degree video, which present a scene only from a fixed point of view – the camera’s perspective. Volumetric videos allow for the combination of translational movement (moving forward and backward, up and down, left and right) and rotational movement (tilting side to side, forward and backward, left and right) inside the scene (Figure 5). While 360-degree video only allows users to look left or right and up and down from a fixed point, native virtual reality environments such as VolumetricOR, allow spectators to physically move through the scene due to the ability to measure the user’s exact location within the scene. This allows users to move naturally within the virtual operating room.

Figure 5.

Difference between 360-degree video (a) and volumetric video (b). Volumetric video is a native virtual reality video format that can address movement, size and the position of the user, whereas 360-degree video only accounts for rotational movement.

VolumetricOR can furthermore strengthen the cooperation between individual team members, as volumetric recordings make the whole surgical scene visible and physically accessible. As a result, users are no longer bound to the perspective and narration of the video. Instead, they can experience the surgical scene and the operating room from a native first-person view. This provides more context information and spatial awareness which is particularly helpful to understand specific workflows. ¹⁴ Users can situate themselves anywhere in the surgical scene and watch the intervention from the point of view of any role or staff member, for instance, to study the exchange of instruments between the surgeon and the surgical assistant or particular settings and gestures. Trainees and residents can observe routines and protocols and the interaction of different professions working together in the operating room, for example, the surgical ‘time out’ from the perspective and position of each individual staff, or specific manoeuvers like an emergency resuscitation. The first-person view complements and extends existing forms of video-based learning, ¹⁵ such as monitor-based or tablet-based tutorials. This could make it an effective strategy to understand the complex cooperation within the surgical team and to accelerate a practice-based knowledge transfer. ¹⁶

Interaction in Real Space and Scale

Standard formats of video-based training and education are cost and time effective methods to study surgical practices and workflows. But they cannot depict the whole surgical scene. Videos impede an active accumulation of skills and techniques because they assign viewers a passive role as observers who cannot interact with the scene. Although stereoscopic video has added depth to the viewing experience and 360-degree video technology has extended the viewing angle, viewers are bound to the camera’s position. In VolumetricOR, the virtual scene becomes responsive to the user’s scale, position and movement. Persons and objects appear in a human-scale dimension. The software even accounts for the user’s individual height, which is a factor that significantly limits the experience of 360-degree video in virtual reality environments. Being able to experience the surgical scene not only in 3D but also in real scale transforms the way users navigate through the surgical workflow. This requires new strategies of narration and interaction in order to engage trainees with the surgical scene. Techniques such as cut, zoom or frame do longer comply with the concept of volumetric video. Instead, learning objectives need to be conceptualized separately from the perspective of the camera and can be integrated as 3D coordinates that respond to the users position and point of view. The interaction in real scale lets users move naturally as if they would be situated in a real operating room. Compared to observing a video, this connection of sensor and motor activity gives users a feeling of presence and lets them apply acquired theoretical knowledge in a practice-oriented manner. ¹⁸ Instead of consuming information passively in front of a monitor virtual reality settings such as VolumetricOR could improve visuospatial skills ¹⁷ as they create a better appreciation of space. Being able to physically imitate actions, memorize gestures and movements as well as understanding the dimensions of tools and objects before going to the operating room facilitates ‘the transition from the learning environment to the clinical environment’, for instance, regarding instrumental arrangements or the testing of new equipment or specific workflows. ¹⁵ The learning and training experience is shifted ‘from observation to participation’ and ‘from screen to space’.

Realism

A major critique of current virtual reality simulations is their lack of authenticity. ¹⁹ Especially due to their computer-based graphic design, most virtual reality learning scenarios lack a realism of tasks ²⁰ which gives participants the awareness of being in a training environment. ²¹ Accordingly, the majority of virtual reality simulators on the market are not designed to look photorealistic. Their aim is not to communicate the Halstedian case-related and process-related knowledge but to replicate single tasks and action sequences of a procedure with the goal to improve psychomotor skills, such as hand-eye coordination. Current virtual reality simulations particularly facilitate tacit knowledge, such as movement and mechanical force used on tissue ²² but lack context and authenticity. This makes them mainly a tool that targets skill acquisition rather than a tool for the simulation of workflows and settings. ²³ While this is not a significant drawback when training standardized interventions or single tasks that are easily repeatable, it considerably falls short when communicating factors such as stress, teamwork or gestures.

Especially in the operating room, where the quality of care significantly depends on the complex interaction and collaboration of the team, computer-animated simulations cannot yet cope with video-based forms of learning. ²¹ The operating room is a complex and information-rich working environment in which team members with divergent disciplinary backgrounds need to work closely together in order to deliver the best results. The knowledge and the experience distributed amongst the surgical team are difficult to communicate outside of real working situations. ²⁴ Although existing analogue simulation methods, such as role-playing games can effectively convey certain techniques and learning tasks, the experience usually remains separated from the real situation in the operating room and is often not capable of representing working contexts and routines in a realistic manner. This is why current digital simulation solutions on the market are usually not designed for teamwork or for situational decision-making. ¹⁶ For acquiring processual knowledge, knowledge exchange on-site in the (real) operating room is still the preferred choice because it provides an authentic experience and a better contextual awareness.

In summary, the concept of VolumetricOR fills a gap within the existing formats of simulation. By connecting video-based data with the advantages of virtual reality it enables users to cognitively and physically enter a virtual operating room with a sense of presence for the real clinical environment. This makes it particularly useful to convey the process and the context of conducting surgical interventions. Hospitals equipped with the proposed sensor setup could create site-specific content based on their individual clinical operating room infrastructure and workflows. In this way clinical staff does not only need to rely on standardized and sometimes poorly reproduced virtual training scenes. Compared to the aesthetics of virtual training scenes, video-based simulation is supposed to increase identification with the learning experience and fosters a higher involvement with the situation. However, those conceptual considerations require further validation by empirical data. Backing these advantages with data needs to be the next step to establish volumetric video as a training tool.

Furthermore, the technical devices used in the developed prototype do not yet offer a sufficient video quality. While the photogrammetric reconstruction of VolumetricOR achieves a highly photorealistic representation of the surgical working environment that users could hardly distinguish from reality, the volumetric recording of the surgical scene still requires quality improvements. A more stable video image with less artifacts can be achieved by deploying the latest generation of depth sensors, such as the Azure Kinect system (Microsoft Corporation, Redmond, U.S.), a time-of-flight depth sensor that features multi-sensor calibration and onboard graphics processing. Furthermore, commercial volumetric service solutions have recently entered the market and offer out-of-the-box software solutions, such as Depthkit (Similie Inc., Brooklyn, U.S.), Metastage (Metastage Inc., Los Angeles, U.S.), Jaunt (Jaunt Inc. San Mateo, U.S.) or EF Eve (EF Eve Inc., London, U.K.). However, their setup is not yet capable of operating in sensitive environments such as the operating room. Overall, the concept of VolumetricOR does not seek to provide a market-ready solution, instead, the goal is to demonstrate the potential of volumetric simulation for surgical education to healthcare providers and clinical practitioners.

Conclusion: Procedure Simulation for Surgical Education

Technical, economic, social and regulatory constraints of surgical learning and training in the operating room require alternative approaches for the exchange of surgical skills and experience. Hospitals need to respond to these challenges by providing customized and scalable digital education solutions that allow surgical staff to effectively convey the knowledge they need. With the emerging technology of virtual reality, hospitals can conduct practice-based digital learning and training simulations outside of the operating room more effectively. ²⁵ The transfer of elements of on-site classes from the operating room into a photorealistic virtual learning environment will not replace on-site observation and participation but intertwines digital and on-site strategies of surgical education. Compared to on-site classes, virtual reality has the advantage to separate complex operative tasks into comprehensible and measurable steps without risking a patient’s safety. ¹⁶

The surplus of VolumetricOR is to provide a concept and a technical setup that enhance virtual reality applications with volumetric videos of surgical workflows. This gives user the advantage to be able to move and see in the same scale and dimension as in a real surgical environment. This embodied form of knowledge acquisition distinguishes VolumetricOR from other forms of digital simulation. While current virtual reality simulation capabilities mostly focus on skill-based training, such as improving dexterity with instruments, ¹¹ VolumetricOR lets trainees experience surgical interventions similar to being physically present. With the ability to swap to any position within the operating room, trainees can learn both the cognitive reasoning and the understanding of the workflow from the perspective of each individual team member – something which is usually not possible during a real intervention. In this way, they could better understand and appreciate the different decision-making paths that affect the complex cooperation of the surgical team. This could facilitate the acquisition of procedural knowledge.

Furthermore, volumetric video does not stand in opposition to established formats of video-based learning and training but complement each other. In many situations standard video will remain a more practical and cost-effective medium to study surgical interventions. Volumetric video may exceed the limitations of standard video when it comes to understanding spatially related information, such as cooperation between team members or the setup of technical infrastructure. A possible use case is the spatial configuration of a robotic system: studying the technical setup three-dimensionally, from any perspective and in real scale could decrease the time of the docking process and shorten operating room time. Additionally, a ‘picture-in-picture’ integration of the laparoscopic video stream on top of the surgical site could help to train the linking between the video and the robotic arms, that is, when adjusting the camera manually.

Altogether, VolumetricOR provides a new type of virtual class based on real workflows and real infrastructure, that is particularly suitable to communicate spatial and procedural knowledge, which is usually acquired over time through observation, assistance and individual experience in the operating room. ⁸ We propose 4 areas of application to further validate the potential and impact of VolumetricOR for surgical learning and training:

a) From our perspective the key area of application will be training for junior physicians during the transition between general education and initial work experience. They will be able to acquire procedural knowledge in an authentic way before going to the operating room. For trainees we expect to increase the learning curve as they could more effectively close the gap between theoretical expertise and practical application of knowledge, that is, to transfer knowledge into skills more effectively and reduce the risk of making mistakes. In later, more experienced career stages, the added value will probably drop, as physicians will have sufficient on-site experience.

b) For residents, VolumetricOR may still extend the levels of professionalization as it improves the communication of new techniques and workflows when the possibilities of on-site training are limited. Based on the ability to record and transmit on-site classes in 4D and to present them in a virtual reality environment in real time could facilitate knowledge exchange and reduce travel costs for residents.

c) With the ability to quantitatively track, measure and review the performance of the surgical team, VolumetricOR can also function as a tool for quality assessment and control as it records interventions as machine-readable three-dimensional geometric data. Connected to automated image and data analysis tools, this could provide a powerful tool to improve workflows and logistics in the operating room.

d) In order to provide potential trainees such as surgical assistants or nurses with an immersive insight into the operating room, volumetric simulations can also facilitate recruitment processes. Using a virtual reality headset, candidates could experience the tasks and working situations of respective job profiles in an authentic way, for example, at job fairs or online. Taking a first-person perspective in real scale enables participants to get a more precise idea of their potential tasks and working situations. The demand for this type of simulation will be further intensified by the demographic change that will significantly strengthen the labour market in many countries.

Supplemental Material