Mixed Reality for Localisation of Rib Fractures Before Surgical Stabilization: A Pilot Study of an Innovative Tool

Abstract

Background

Accurate localisation of rib fractures is crucial for successful surgical stabilisation of rib fractures (SSRF). Mixed reality (MR) technology can overlay anatomical imaging onto a patient in real-time, which may facilitate the visualisation of rib fractures, allowing accurate skin surface markings for incision. This pilot study assessed the feasibility of using mixed reality (MR) for rib fracture localisation in SSRF.

Methods

This study involved a pre-clinical phase with two healthy patients and a clinical phase with six patients undergoing SSRF in a single tertiary trauma centre. CT scans were transformed into patient-specific 3D holographic models, which were projected through Microsoft HoloLens2TM (HL2) onto the patient after anatomical calibration. The study assessed hologram projection, number of fractures identified, time taken, and distance from skin marking to the fracture site. Iterative improvements to the MR system were implemented throughout the study.

Results

Stable and accurate hologram projection was achieved in both phases. In the clinical portion, MR identified 54 rib fractures, including subscapular fractures, compared to 30 identified by US. The mean time to mark all fractures was 9.07 minutes for MR and 10.02 minutes for US. The mean displacement from skin marking to the fracture site was 2.89 cm for MR and 2.04 cm for US. Technical challenges included distorted surface anatomy and positional variations.

Conclusion

MR technology in the setting of SSRF is feasible and facilitates de-novo visualisation of rib fractures. Technical limitations must be addressed before widespread clinical use.

Background

Rib fractures are a prevalent issue in trauma, sustained by almost half of all patients with blunt chest wall trauma. ¹ Surgical stabilisation of rib fractures (SSRF) is now supported by real-world evidence. ² For both flail and non-flail fracture patterns, SSRF yields less time on ventilation, shorter hospital length of stay, reduced pain, and improved function.^1,3,4

Operative exposure is a key component of SSRF. Larger incisions result in increased postoperative pain and wound complications at the incision site. Multiple guidelines recommend targeted incisions with muscle preserving/splitting dissection.^4,5 Targeted incisions require precise knowledge of rib fracture location and thoracic wall musculature anatomy. However, accurate identification is complicated by intricate injury patterns, variations in patient anatomy, and soft tissue deformation due to positional changes in the operating room.^6,7 Currently, various methods are employed for pre-operative planning, including CT scans and 3D reconstructions. However, these methods are limited by their static nature and rely on surgeons’ subjective judgement of position changes. While ultrasound (US) has been used perioperatively to identify surface markings of rib fractures, its use is not universally adopted. It is associated with inter-user variability, ⁶ cannot be used to identify sub-scapular fractures, and these effects are compounded in patients who are obese, which is an increasingly prevalent issue. ⁸ There is a clear need for improved methods for accurate and reliable rib fracture localisation to optimise SSRF and minimise surgical invasiveness.

Mixed reality (MR) technology allows users to experience virtual objects anchored into the real world within aspects of the environment while being responsive to them. It has emerged as a real-time tool to superimpose anatomy onto a patient, first demonstrated in 2017. ⁹ Early applications in controlled environments have succeeded with mannequin and cadaver models. ⁹ Recent clinical applications have also been shown in orthopaedics, neurosurgery, spinal surgery, and ear, nose, and throat surgery, whereby 3D holograms of patient anatomy have been superimposed onto the patient on the operating table.^10-14 In addition, these applications have primarily focussed on body surfaces with clearly defined landmarks with low density of subcutaneous tissue. However, this technology has yet to be applied to the chest wall, specifically SSRF, which presents unique challenges, including a large surface area, variation in soft tissue density and deformation, greater variation in scan acquisition, and variable patient positioning.

Our pilot study aimed to assess the feasibility of using MR technology to localise rib fractures for SSRF in two stages: a pre-clinical proof-of-concept phase and then a real-world clinical application in an operating room. We hypothesised that MR would be applicable for SSRF in a real clinical setting and could address some of the challenges encountered with existing localisation techniques.

Methods

This study was conducted in accordance with the World Medical Association Declaration of Helsinki ethical principles. All patients gave informed consent, and their privacy rights were observed. Before the study began, ethics approval was obtained from the Local Health District (2021/ETH00209). The STARD checklist was used for writing the report. ¹⁵

Scan Acquisition and Overlay

Axial CT scans in the bone window were extracted from enrolled patient records using Sectra^TM (Sectra AB, Teknikringen 20, SE-583 30 Linköping, Sweden). All scans were in the supine position as per routine care for patients after major trauma; no additional or specific imaging was obtained for this study. A software engineer (GJ, HJ) performed image processing, which took approximately 30 minutes per patient. A U-Net pre-trained chest segmentation model was used to segment the rib cage, followed by manual refinement using a 3D slicer (version 5.0.3). ¹⁶ The final segmentation was converted into a 3D surface model reconstruction (Hologram). A custom software, “RibMR”, was developed using C# and Unity Engine (version 2019.4.28f1). It was used to project and overlay the hologram onto the patient using Microsoft HoloLens2^TM (1 Microsoft Way, Redmond, Washington, USA) after calibration with anatomical landmarks on the patient. This is an unlabelled use for the HoloLens2 device and is used here purely for investigative purposes.

Part 1: Pre-clinical – Proof of Concept

Trauma patients without rib fractures who had a chest CT scan were recruited. Images were projected onto the patients in a non-clinical, controlled environment in supine (P1) and right lateral positions (P2), using 5 anatomical landmarks as calibration points for registration. These landmarks were selected due to the likely consistency of identification on a patient coupled with accurate identification on the segmented images. The distance between the projected holographic landmark and the corresponding palpable bony landmarks was measured as a marker of accuracy.

Part 2: Clinical Use and Testing

All patients between March 2022 and June 2023 undergoing surgical stabilisation of rib fractures (SSRF) were eligible and recruited prospectively for this study performed at a level 1 trauma centre. After anaesthesia, patients were positioned in lateral decubitus (with varying degrees of lateral positioning depending on fracture location and concurrent injuries) as required for the operation. Five fiducial markers were placed onto the patient at palpable bony landmarks and their corresponding locations were marked on the 3D render. These landmarks were selected based on their rigidity during patient position changes and their distinctiveness within the patient model and on the patient’s body. These are reported in the results.

Rib fractures, as seen on the holographic projection, were marked using a surgical marker by the surgical trainee. The surgeon made separate markings for all fractures visible by US, not just those planned for fixation. A surgical trainee marked MR fracture positions to limit bias when the surgeon was marking with US. The operations were conducted routinely with no regard to markings made by MR, given the investigative nature of the study. Time to mark fractures, number of fractures identified, and the shortest distance between rib fracture intra-operatively to corresponding skin marking were measured for US and MR.

Results

Part 1: Pre-clinical – Proof of Concept

Two male trauma patients aged 23 and 26 without rib fractures were recruited, with BMIs of 23.4 and 26.0, respectively. Stable and accurate hologram projection onto both patients was achieved in both supine and right lateral positions. Ribs were able to be palpated directly under their projection. Palpation of the bony landmarks was a mean of 2.12 cm (SD 0.79 cm) away from the projection in the supine position compared to 3.03 cm (SD 1.58 cm) in the right lateral position.

Five bony landmarks were identified as ideal fiducial markers for successfully registering MR to the body and used for part 2 of the study. These were the spinous process of the 7th cervical vertebra (C7), the acromioclavicular joint (AC), the jugular notch, the xiphisternum, and the spinous process of the 10th vertebra (T10).

Part 2: Clinical Use and Testing

Six patients who underwent SSRF between March 2022 and June 2023 were recruited for the study. Table 1 reports patient demographics, including the number of fractures and fractures fixed.

Table 1.

Patient Demographics

Pt	Sex	Height	Weight	BMI	Side	Ribs #	Ribs fixed	Mechanism of injury
		(cm)	(kg)		of #
1	Male	180	120	37.04	R	1-10	4- 8 lateral	MVC
								Car vs truck
2	Male	159	81	32.04	L	3-10	4-9 lateral	Mountain bike fall injury
							5-6 posterior
3	Male	179	90	28.09	R	2-10	3-7 posterolateral	Motorbike collision
							3,5,7 lateral & posterior
4	Male	180	80	24.69	L	2-10	4-8 lateral & posterior	Pedestrian vs car collision
							9-10 posterior
5	Male	181	100	30.52	L	1-10	4-10 lateral	3m free fall
6	Male	178	80	25.25	R	1-9	4-9 lateral	2.5 m free fall
							4-6 multiple

Abbreviations: BMI, Body Mass Index; ORIF, Open reduction and internal fixation; ISS, Injury Severity Score; Pt, Patient; MVC, Motor vehicle crash.

#: fracture.

High-fidelity holographic reconstructions of patient CT scans were successfully projected onto all 6 patients in a real clinical setting, demonstrating the feasibility of using MR for intraoperative rib fracture localisation. An example is shown in Figure 1 and video 1. Rib numbers could also be identified and marked on the chest wall. There were no observable differences in outcomes between anterior, lateral, or posterior rib fractures.

Figure 1.

Demonstration of Mixed Reality (MR) Application for Surgical Stabilisation of Rib Fractures (SSRF)

Iterative Improvements

We report iterative improvement in our process with each patient. The stepwise approach is outlined in Table 2. Chest, abdomen, and pelvis scans were combined to segment the rib cage, spine, and pelvis to access bony landmarks of the lower spine and pelvis, which served as additional fiducial markers compared to the chest only prior to this. A bony rather than skin view of the segmentation was then used to improve the visualisation of fractures. Due to persistent difficulty visualising fractures, annotation of CT at fracture sites was introduced as a new step during segmentation. This allowed fractures to be highlighted within the hologram. The annotation was then modified to be red for easier identification. Annotations were then extended to cross the skin, helping find overlying skin more precisely, as shown in Figure 1A. A systematic method was then introduced to reduce visuospatial subjectivity in skin marking over fractures. This involved placing a digit where the annotated holographic line crosses the skin and viewing this junction from at least three angles.

Table 2.

Representation of the Iterative Improvements Made During the Project

Patients	Challenge	Solution
1	First patient, images incomplete	Combining chest, abdomen, and pelvis scans for segmentation
2	Identifying fractures difficult if poor quality of segmentation	Use of bone model view rather than skin with bone and annotation at fracture site with grey cylinder
3	Cylinder hard to see	Annotation changed to a red bar
4	Identifying fracture behind scapula and spatial visualisation error	Red bar annotations extend to skin
5	Occasional difficulty identifying fiducial markers due to obstruction of view, eg, by neck collar	Manually localise the fiducial marker
6	Spatial visualisation error	Use of finger tracking with view from 3 perspectives prior to marking

Column ‘challenge’ indicates difficulty faced and column ‘solution’ indicates the improvement made to address the issue as further described in the manuscript.

Clinical Outcomes

Given this is a pilot study, no direct statistical comparisons are made between MR and US, which is also not universally used. Instead, findings are presented here with US as a reference to demonstrate feasibility and context for future research. Once the projection was aligned, 54 rib fractures were identified in the 6 patients, including subscapular fractures, which are challenging to identify using ultrasound. In comparison, 30 fractures were identified and marked by US, and no subscapular fractures could be observed. MR was able to identify 96.7% of the fractures identified by US. Depending on exposed anatomy, often limited by concurrent injuries, each patient required various anatomical landmarks to be used as fiducial markers; these are reported in Table 3. As presented in Table 4, the mean time to mark all fractures was slightly faster with MR (9.07 minutes, Range: 7.3-12.07 mins, SD 1.62 mins) compared to US (10.02 minutes, Range: 5.47-14.5 mins, SD 3.51 mins). Similarly, on a per-fracture basis, MR took a mean of 1.23 minutes (Range: 0.61-2.10 mins, SD 0.57 mins) compared to 2.32 minutes (Range: 1.37-4.59 mins, SD 1.30 mins) for US. For rib fractures identified by both MR and US, the mean displacement from intra-operative localisation by MR was 2.89 cm (Range: 0-6 cm, SD 2 cm) vs US 2.04 cm (0-13 cm, SD 4.25 cm).

Table 3.

Fiducial Markers Used for Registration of the Hologram Onto the Patient

Pt	Sternal notch	Xiphi	AC joint	ASIS	PSIS	C7 spinous process	Other	Other: Details	Concurrent surgery
1		X		X			X	Pubic arch, sacral tuberosity, T6 spinous process	Nil
2	X			X (L)	X (L,R)		X	T1 spinous process	Nil
3	X	X	X (R)		X (R)	X			RIGHT clavicle ORIF
4	X		X (L)	X (L)	X (L)	X			LEFT clavicle ORIF
5	X				X (L,R)		X	Rib 10, rib 12	LEFT glenoid ORIF
6	X	X	X (R)	X (R)		X			Nil

X: denotes the anatomical landmark was used as a fiducial marker.

Abbreviations: Pt, Patients; Xiphi, Xiphisternum; AC, Acromioclavicular; ASIS, Anterior Superior Iliac Spine; PSIS, Posterior Superior Iliac Spine.

Table 4.

Comparison of Mixed Reality System (RibMR) to Ultrasound (US)

Patients	RibMR			Ultrasound
	Time taken (mins)	Number of fractures localized	Subscapular	Time taken (mins)	Number of fractures localized	Subscapular
1	7.30	5	N/A	9.33	5	N/A
2	8.42	4	N/A	9.50	3	N/A
3	9.12	15	Yes	7.53	5	No
4	9.22	14	Yes	14.50	10	No
5	12.07	8	Yes	13.77	3	No
6	8.30	8	Yes	5.47	4	No

Time taken is from start of registration process to skin marking of the last fracture.

Subscapular column refers to rib fractures under the scapula. N/A implies there were no such fractures. Yes indicates such fractures are present and able to be marked. No indicates such fractures are present but not able to be marked.

Discussion

Rib fractures are common in blunt chest wall trauma. While SSRF improves outcomes for selected patients, it involves large skin incisions and tissue dissection for proper exposure, which can lead to complications such as increased postoperative pain, infection, and delayed wound healing. To explore localisation methods that could minimise incision size, we developed and used Mixed Reality (MR) to locate rib fractures on the skin. This is a significant finding as it represents, to our knowledge, the first application of MR for rib fracture localisation in actual SSRF cases. We successfully achieved stable and accurate hologram projection of patient-specific CT data onto patients in both pre-clinical and real-world clinical environments.

In the clinical part of our study, we recruited six patients undergoing SSRF. In all six cases, the holographic projections were successfully used for rib fracture localisation. MR identified 54 fractures compared to 30 by US. Additionally, MR allowed for marking subscapular fractures and rib numbers, which could significantly aid operative planning. Per patient, MR took a mean of 9 minutes and 4 seconds, including time taken for registration, compared to 10 minutes and 1 s using US. The mean displacement from intraoperative localisation to skin marking by MR was 2.89 cm (Range: 0-6 cm, SD 2 cm) vs US 2.04 cm (0-13 cm, SD 4.25 cm).

Throughout the clinical phase of the study, we made incremental improvements to our MR system based on the challenges encountered, as highlighted in Table 2. These challenges included variability in the acquisition of CT scans, difficulty exposing bony landmarks for fiducial marker placement, and 3D distortion caused by subcutaneous tissue and patient positioning. For instance, routine CT scans obtained as part of trauma care sometimes fragmented the chest from the abdomen and pelvis, requiring additional processing steps to create a unified anatomical model for a wider range of potential landmarks, including those in the lower spine and pelvis. Furthermore, the visibility of the fiducial markers was sometimes affected by factors such as lighting in the operating room, treatment requirements, such as other fractures or essential tubing obscuring the view, and the patient’s body habitus. Concurrent injuries affected fiducial marker selection in three patients with upper limb or shoulder girdle fractures (Table 3). When traditional landmarks like the clavicle or sternum are fractured, they may be mobile or inaccessible, requiring alternative marker selection or soft tissue approximations. This challenge, unique to polytrauma, may have contributed to observed variability in accuracy. In cases where bony landmarks were obscured or inaccessible, fiducial markers were placed over an approximate corresponding soft tissue position. However, this is prone to error due to the deformation of soft tissue with position exacerbated in obese patients with increased distance from anatomy to skin. ¹⁴ This is likely a significant source of random error in our study. While other studies have reported similar challenges with registration, often requiring manual adjustments,^12,17 our system was designed to maintain a fixed registration after the initial calibration, which increases reliability and reduces the need for intraoperative adjustments. We observed improved accuracy of hologram registration when calibration points had a wider distribution over anterior, posterior, lateral, and medial aspects. To improve the visualisation of fractures, we initially annotated the fractures on the hologram with a grey bar, which was later changed to red for better visibility. However, 3D distortion caused by variations in soft tissue thickness made it difficult to identify the corresponding point on the skin surface accurately. We addressed this by extending the annotation to the skin surface and adopting a systematic method of viewing the intersection of the annotation and the skin from multiple angles.^12,14,18 These iterative improvements, driven by real-world challenges, led to a more accurate and efficient MR system for rib fracture localisation.

To contextualise our findings further, it is helpful to compare our MR system with other applications of MR in surgery. Similar applications have been demonstrated in spinal surgery, identifying the site for percutaneous needle puncture.^12-14 These studies show higher first-pass success, as confirmed by fluoroscopy, which is their routine method to assess needle puncture without MR. Similar applications to identify percutaneous puncture sites have been seen for nephrolithotomy, ¹⁸ anatomical liver segment resection, ¹⁷ and rhizotomy. ¹¹ In contrast, rather than focusing on a single skin point, we demonstrate the identification of multiple fractures on the skin over a large area. These form a pattern that is then used to plan the operative incision for SSRF in the anaesthetised patient. Notably, previous studies used scans obtained in the same position as the patient’s operation, unlike in our study, where the CT scan is obtained from urgent trauma imaging in the supine compared to the operative position, which is varying degrees of lateral decubitus. As is often impossible in trauma, previous studies used fit-for-purpose CT or CT combined with MRI to segment their hologram reconstructions. This highlights the robustness of our system in adapting to real-world variations encountered in trauma settings.

Accurate localisation of rib fractures is crucial for successful SSRF. Current techniques, such as relying solely on preoperative CT scans or ultrasound, have limitations. ¹⁹ CT scans are static and require surgeons to mentally translate the images to the patient’s position in the operating room. ²⁰ While ultrasound provides real-time imaging, it is operator-dependent, may not visualise all fractures, and can be challenging to use in obese patients. We compared MR to ultrasound as it represents current best practice for real-time localization; while not universally adopted, US improves upon CT review and palpation alone, making it an appropriate benchmark for evaluating MR technology. In this study, a key finding was the ability of MR to identify a greater number of rib fractures compared to US, including subscapular fractures. Identifying additional fractures, particularly subscapular fractures, the ability to mark rib numbers, and being faster could potentially improve surgical planning by allowing surgeons to anticipate the full extent of the injury and plan their incisions accordingly. This could lead to more targeted and less invasive surgical approaches, potentially reducing operative time, tissue trauma, and postoperative pain.^6,7,21

Limitations and Future Direction

There are several limitations to this study. Firstly, we did not use MR to guide any incisions, nor was it used intra-operatively, so we could not test for clinical benefits directly. As incremental improvements were made, thereby introducing additional independent variables, no valid statistical testing could be performed for any recorded outcomes. Secondly, while the initial use of MR required training and the presence of engineering staff for technical support, ongoing software development aims to make the system more user-friendly and enable surgeons to operate independently. Thirdly, bias may be introduced due to the non-blinded marking of MR and US. However, the order between MR and US was randomised to reduce this.

Future work will address this study’s limitations, including a larger test cohort and an assessment of clinical benefits. We plan to explore markerless registration techniques to eliminate the need for fiducial markers. It may also be capable of dealing with soft tissue deformation from repositioning, a major source of random error. Beyond SSRF, the technology developed in this study can be applied to a broader range of surgical procedures that require accurate localisation of anatomical structures. For example, MR could guide the resection of tumours, such as breast masses or liver lesions, by overlaying a 3D tumour model onto the patient’s body. This could allow for more precise and minimally invasive resections, potentially improving patient outcomes. A study by Prasad et al ²² demonstrated the feasibility of using mixed reality for tumour re-resection of positive margins in head and neck cancer in cadaveric models, suggesting the potential of this technology in oncological surgery. ²² Future studies should explore the application of this technology to other surgical domains to develop a versatile MR platform for image-guided surgery.

Conclusion

This pilot study demonstrated the feasibility of using mixed reality (MR) for accurate localisation of rib fractures in patients undergoing surgical stabilisation of rib fractures (SSRF). We successfully achieved stable and accurate hologram projection of a patient’s CT onto patients in a real-world clinical environment. MR use identified more rib fractures than ultrasound in a shorter time, including subscapular fractures, and enabled the numbering of ribs, potentially aiding in surgical planning. Future work will address the technical limitations identified in this study, conduct larger clinical trials, and explore the use of markerless registration and intra-operative MR guidance. The technology developed in this study holds promise not only for SSRF but also for a wider range of image-guided surgical procedures, highlighting the potential of MR to transform surgical practice.

Supplemental Material