External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Abstract

Purpose:

This study investigates the accuracy of fully-automated maximum aortic diameter measurements in abdominal aortic aneurysm (AAA) patients using artificial intelligence software (PRAEVAorta 2, Nurea, Bordeaux, France).

Materials and Methods:

This is a multicenter, retrospective validation study using prospectively collected data from the Zenith alpha for aneurysm Repair Registry (ZEPHYR). Automated measurements of PRAEVAorta 2 are compared with measurements of an internationally recognized core laboratory (Syntactx, New York, New York State). The reviewers at the core laboratory were measurement technologists trained to and utilizing established measurement standards, overseen by vascular surgeons and radiologists. The data set comprised 871 computed tomography angiography scans from the ZEPHYR registry with 347 patients who underwent endovascular aneurysm repair (EVAR) with the Zenith Alpha Endovascular Abdominal Graft (Cook Medical, Bloomington, Indiana) in Germany, Belgium, and The Netherlands between 2016 and 2019.

Results:

The analysis demonstrated excellent correlation of the measurements (r=0.97) with an intraclass correlation (ICC) of 0.972 (95% confidence interval [CI]=0.968-0.976) across all scans. For preoperative computed tomography (CT) scans, ICC was 0.953 (95% CI=0.941-0.963), and for postoperative scans, ICC was 0.979 (95% CI=0.975-0,983), respectively. In total, 95.4% of measurements were within the clinically acceptable range of 5 mm in absolute difference. In total, 10% of scans demonstrated obvious segmentation errors, mainly due to failure in detecting vessel segments (renal arteries, aortic bifurcation) or due to mis-detecting the outer border of the AAA (duodenum, inferior vena cava, aortic branches) and were excluded from the analysis.

Conclusion:

In this study, the maximum AAA diameter could be accurately measured fully-automatically by PRAEVAorta 2 (Nurea) in most cases demonstrating that artificial intelligence (AI) software could serve as an important adjunct for research and clinical practice. However, critical review of the generated reports by an experienced observer and cautious use is warranted to identify flawed segmentations.

Clinical Impact

This multicenter, retrospective validation study assessed the accuracy of fully-automated maximum infrarenal aortic aneurysm diameter measurements. It was demonstrated, that 95.4% of measurements were within the clinically acceptable range of 5 mm in absolute difference, positioning the software as a potential adjunct for clinical practice and research. It is also highlighted however, that critical review of the measurements is obligatory, due to a 10% segmentation error rate.

Introduction

Analysis of large data sets from real-world registries is an increasingly important source of evidence.^1,2 The insights that can be gathered from such registries are largely a function of the underlying quality, structure, granularity, and volume of data. In the context of abdominal aortic aneurysm research, information derived from computed tomography angiography (CTA) is essential. Frequently, the measurements of aortic diameters are performed by local researchers, introducing both intraobserver and interobserver biases of up to 5 mm in absolute difference due to heterogeneous methodology and definitions.^3–7 In addition, the availability of resources in research facilities naturally restricts the volume of processable data for research purposes. Availability of data could turn out to be a decisive factor for the ability to adopt big data analysis to guide personalized aortic therapy in the future. ⁸

At present, imaging core laboratories can deliver high-quality analysis at scale. However, these services are labor-intensive, infrastructurally complex and therefore associated with relevant costs. In recent years, tools that allow fully-automated aortic diameter measurements have been introduced.^9–13 One of which is PRAEVAorta 2 (Nurea, Bordeaux, France), which received its CE mark in 2023. Previous studies have demonstrated promising results with regard to the software’s ability to accurately segment volumes and determine the maximum aortic diameter in patients with abdominal aortic aneurysm.^10,11,14 However, studies on the accuracy of PRAEVAorta 2 and similar tools are often limited by a lack of external validity on larger sets of data from multiple sites. External validation of any automated measurement tool is essential.

The Zephyr registry included 347 patients with an abdominal aorto-iliac aneurysm across 14 sites in Germany, Belgium, and The Netherlands who were treated with the Zenith Alpha Abdominal Endovascular Graft (Cook Medical, Bloomington, Indiana). Preoperative and postoperative CT scans were uploaded to a core laboratory (Syntactx, New York, New York State) for the assessment of aortic morphology, including the maximum aneurysm diameter. ¹⁵

The objective of the present study was to evaluate the accuracy of a fully-automated deep-learning–based method developed to measuring the maximum aortic diameter (PRAEVAorta 2) by comparing it to the measurements provided by the core laboratory (Syntactx).

Methods

Study Design

This is a multicenter, retrospective validation study using prospectively collected data from the Zenith alpha for aneurysm Repair Registry (ZEPHYR). Core laboratory assessed maximum AAA diameter measurements were compared with fully-automated measurements generated by a deep-learning–based software (PRAEVAorta 2).

Data Set

All available images from the ZEPHYR registry at the time of the initiation of the present study were included. A total of 871 CTA images were obtained from patients with a juxtarenal or infrarenal AAA who had multidetector CT scans of the entire abdominal aorta and arterial phase intravenous injection of contrast medium. Patients were recruited for the ZEPHYR registry between 2016 and 2019. Images were provided in Digital Imaging and Communications in Medicine (DICOM) format with a matrix size of 512 × 512 for each slice. Data were de-identified prior to the upload on the core lab’s medidata platform. Scans were performed according to local protocols in all participating sites. The study protocol was approved by the local Institutional Review Board (S-652/2023). Informed consent for the purposes of this study was waived. Informed consent for the ZEPHYR registry was obtained from all patients.

Core Laboratory Diameter Measurements (Syntactx)

Preoperative and postoperative CTA scans were uploaded via a web browser to the medidata platform. Among others, the core laboratory performed measurements of the outer-to-outer maximum diameter of the abdominal aortic aneurysm on a slice orthogonal to the outer wall of the aorta in multiplanar reconstruction images (MPRs). A representative image of the methodology is displayed in Supplemental Figure 1. Measurements were performed in TeraRecon iNtuition (Fremont, California). The reviewers are measurement technologists trained to and utilizing established measurement standards, overseen by vascular surgeons and radiologists. Each image is reviewed by an initial reviewer with a quality check by a second reviewer. The images were collected over a 21 CFR part 11 compliant image transfer system.

PRAEVAorta 2 (Nurea)

All available CTA scans were uploaded to an offline research-only version of the PRAEVAorta 2 software installed at one of the participating study sites (Software version 20230321). The method of aneurysm segmentation consists of 4 sequential steps: (1) image preprocessing, (2) segmentation of the aortic lumen and thrombus, (3) centerline generation, and (4) parameter computation. Models were created using the PyTorch framework. The models designed by PRAEVAorta 2 are based on convolutional neural networks (U-Net). Image preprocessing was fully-automated by the software. Standard image filtering and denoising algorithms for medical imaging were used. The maximum transverse diameter was defined as the maximum diameter computed in the planes orthogonal to the centerline, including lumen and thrombus. The software divides the aorta in distinct segments by planes orthogonally cutting the centerline at the level of predefined anatomical landmarks. The infrarenal segment is defined proximally by the lowest point of the lowest renal artery, orthogonally cutting the centerline. The distal end of the infrarenal segment is defined by the plane orthogonally cutting the centerline at the level of the right and left common iliac artery ostium.

The application offers a dashboard view displaying all previously uploaded scans. Custom software that was provided for the purposes of this study by Nurea enabled the queuing of larger badges of CTA scans, thereby enabling to analyze multiple scans without further intervention of the user. The analysis of each scan requires about 1 to 5 minutes, depending on the number of images per scan. After selection of the DICOM CTA images, no further intervention of the user was required. The software then produces a report in pdf format for every scan that can be accessed through the dashboard view within the application. Larger sets of results can be exported in tabular format as well. An excerpt of the report is presented in Supplemental Figure 2. A full report is presented in Supplementary Material 1. The software recommends for the user to visually review the report to verify plausibility of the segmentation and provides 2-dimensional and 3-dimensional visualization of the segmented volumes, including an overview and imaging slices, from which the measurements are derived.

Statistical Analysis

Statistical analysis was performed using “R” (version 4.3.1, 64 bit, Vienna, Austria). ¹⁶ For the validation of the PRAEVAorta software’s accuracy in measuring the maximum AAA diameter (outer-to-outer), descriptive statistical analysis is used with mean values and standard deviations (SDs). Furthermore, a correlation of the manual and fully-automated diameter measurements is calculated as well as a Bland-Altman plot and intraclass correlations (ICCs). The ICC is presented with 95% confidence intervals (95% CIs). An absolute, maximum difference of <5 mm was defined as clinically acceptable, considering interobservership differences in the literature.^5,7,17–19

Results

Imaging Data

There were 871 contrast CT scans available for the purposes of this study displaying the full abdominal aorto-iliac anatomy in arterial phase. These 871 scans were analyzed fully-automatically by the PRAEVAorta software. After analysis, generated pdf reports were independently assessed by 2 observers (J.H., A.B.). Subsequently, 87 scans (10.0%) were excluded for obvious segmentation errors due to misdetection of the aortic segments or misdetection of the outer circumference of the aortic wall. Representative images of excluded scans are presented in Figure 1. The remaining 784 scans were matched with the corelab data set, which resulted in 762 available matched pairs for analysis. For 22 scans (2.8%), there were no corelab measurements available at the time of data curation for this study.

Figure 1.

Scans with obvious errors in segmentations (N=87/871, 10.0%). All reports were scanned for these errors and independently assessed for plausibility by 2 observers (J.H., A.B.). Obviously flawed segmentations were excluded from further analysis.

There were 263 preoperative scans and 499 postoperative scans. Most scans had a slice thickness of 1 to 3 mm (N=477, 62.6%), followed by a slice thickness of <1 mm (N=213, 30.0%) and a slice thickness of >3 mm (N=72, 9.4%). The mean slice thickness across the included scans was 1.2 mm (SD=0.7). The excluded scans had a slice thickness of 1.4 mm (SD=0.9) with 33 scans <1 mm (37.9%), 41 with 1 to 3 mm (47.1%), and 13 with >3 mm (15.0%). A flowchart presenting included scans is presented in Figure 2.

Figure 2.

Flow chart displaying the data set of the study.

PRAEVAorta 2 vs Core Laboratory

The mean aortic diameter across all scans as measured by core laboratory and PRAEVAorta 2 was 58.8 mm (SD=10.0) and 58.5 mm (SD=10.2), respectively. The mean absolute difference on a per scan basis between core laboratory and PRAEVAorta 2 was 0.28 mm (SD=2.4). 727 of 762 (95.4%) measurements showed a clinically acceptable accuracy with an absolute difference of <5 mm. A total of 628 (82.4%) scans showed a measurement difference of <3 mm. There was a strong correlation between the measurements provided by PRAEVAorta and the core laboratory with a correlation coefficient of 0.97. The correlation is displayed in Supplemental Figure 3. A total of 727 analyzed matched pairs (95.4%) were within a range of 5 mm in difference. The Bland-Altman plot is presented in Figure 3. The limits of agreement (95% confidence interval) were −4.36 mm for the lower and 4.93 mm for the upper limit of agreement with a bias of 0.28 mm.

Figure 3.

Bland-Altman plot presenting the agreement between both methods. The lower limit of agreement was −4.36 mm, the upper limit was 4.93 mm with a bias of 0.28 mm. In total, 95.4% of measurements were within a ±5 mm absolute difference.

Intraclass Correlation

The ICC for all included scans was 0.972 (95% CI=0.968-0.976). For preoperative CT scans, ICC was 0.953 (95% CI=0.941-0.963), and for postoperative scans, ICC was 0.979 (95% CI=0.975-0.983), respectively. Violin plots are displayed in Figure 4.

Figure 4.

Violin plots. The intraclass correlation for preoperative scans was 0.953, and for postoperative scans, it was 0.979. For preoperative and postoperative scans combined, it was 0.972.

Discussion

This study evaluated the accuracy of a fully-automated deep-learning–based method developed to automatically measure the maximum aortic diameter (PRAEVAorta 2; Nurea) in CTA scans by comparing it to the measurements provided by the core laboratory (Syntactx).

It comprised the data of 871 CT scans of AAA patients across Europe. ¹⁵ To the knowledge of the authors, this analysis represents one of the largest validation cohorts investigating fully-automated outer-to-outer AAA diameter measurements to date and the only study that uses exclusively externally acquired measurements as the reference standard. The analysis revealed that the software was able to accurately determine the maximum AAA diameter in about 95% of scans with excellent ICC. This was consistent across preoperative and postoperative images following endovascular aneurysm repair (EVAR). Thereby, the technology offers a time-effective and cost-effective methodology for the assessment of these important outcome criteria in AAA research and clinical practice, while simultaneously eliminating interobservership and intraobservership biases. These results are in line with results of a recent validation study investigating the Augmented Radiology for Vascular Aneurysm (ARVA) software (Incepto Medical, France) for aortic diameter measurements—among other segments—in the infrarenal segment. ¹²

But with new technological advancements, new issues arise as well. The study demonstrated that in 10% of scans the visual review of the provided report revealed obviously flawed segmentations. A similar observation was made in a recent publication investigating ARVA (Incepto Medical).^10,12 In the case of PRAEVAorta 2, these errors resulted from failure to correctly divide the aorto-iliac anatomy in its segments, mainly due to the failure to correctly identify the renal arteries or the aortic bifurcation, and thereby mis-segmenting the infrarenal aneurysm and subsequently reporting a flawed diameter measurement. Or from failure to accurately differentiate the outer circumference of the aneurysm from surrounding tissue such as the duodenum, the inferior vena cava, or adjacent vasculature such as the inferior mesenteric artery (Figure 4). These types of errors are multifactorial, including the contiguity of tissues with similar pixel intensity, or in rare cases, the presence of artifacts, such as coils and other implants, illuminating the image. Even though the provided report allows the user to detect these errors, these findings highlight the need for continued improvements in the future. Until then, complementary critical review by a trained human observer with experience about sources of potential segmentation errors is obligatory to ensure its safety.

In addition, to allow full utilization of artificial intelligence (AI) algorithms in the medical future, it is of importance to recognize the necessity of continued standardization of CTA protocols such as the injection timing of contrast, nomenclature of series descriptions across sites, including the definition of regions of interest in preoperative and postoperative EVAR imaging. With overall improvement of imaging quality, AI-based image analysis is likely to improve as well. For AI software solutions to be truly useful, these tools need to be flawlessly applicable to as many scans as possible with minimal to no human preprocessing.^4,6 Another prerequisite is to adapt the current health care systems to cloud-based data storage to catalyze integration of such systems into existing clinical imaging repositories.

Limitations

This study presents with several limitations. Although it was a multicenter study with many included scans from an international registry, patients were treated for infrarenal abdominal aortic aneurysm with standard bifurcated endovascular repair (Zenith Alpha Abdominal Endovascular Graft, Cook Medical, Inc). Therefore, results might not be generalizable to situations with aneurysms following non-standard EVAR, such as fenestrated or branched procedures. In addition, this study only investigated infrarenal maximum AAA diameter, and results are not applicable to other aortic segments, in contrary to other current AI tools. ¹² Slight differences in definition of core laboratory and PRAEVAorta 2 measurements prevented the external validation of additional measurements, such as neck or iliac morphology. Furthermore, while an external core laboratory, blinded for the software’s measurements with internal quality checks, was used as the gold standard in this study, human errors in establishing gold standards for the validation of any AI tools cannot be excluded with absolute certainty.

Conclusion

In this study, PRAEVAorta 2 accurately and fully-automatically assessed the maximum AAA diameter, positioning the software as an important adjunct for research and clinical practice. However, critical review of the provided report by an experienced observer and cautious use is warranted. Standardization of CT protocols and further improvements of the software might facilitate its extended use in very large data sets fully capitalizing on its time-effectiveness and cost-effectiveness when compared with conventional methods.

Supplemental Material

sj-jpg-1-jet-10.1177_15266028241295563 – Supplemental material for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Supplemental material, sj-jpg-1-jet-10.1177_15266028241295563 for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2) by Johannes Hatzl, Christian Uhl, Alexandru Barb, Daniel Henning, Jonathan Fiering, Edris El-Sanosy, Philippe W.M. Cuypers and Dittmar Böckler in Journal of Endovascular Therapy

Supplemental Material

sj-jpg-2-jet-10.1177_15266028241295563 – Supplemental material for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Supplemental material, sj-jpg-2-jet-10.1177_15266028241295563 for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2) by Johannes Hatzl, Christian Uhl, Alexandru Barb, Daniel Henning, Jonathan Fiering, Edris El-Sanosy, Philippe W.M. Cuypers and Dittmar Böckler in Journal of Endovascular Therapy

Supplemental Material

sj-jpg-3-jet-10.1177_15266028241295563 – Supplemental material for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Supplemental material, sj-jpg-3-jet-10.1177_15266028241295563 for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2) by Johannes Hatzl, Christian Uhl, Alexandru Barb, Daniel Henning, Jonathan Fiering, Edris El-Sanosy, Philippe W.M. Cuypers and Dittmar Böckler in Journal of Endovascular Therapy

Supplemental Material

sj-pdf-4-jet-10.1177_15266028241295563 – Supplemental material for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Supplemental material, sj-pdf-4-jet-10.1177_15266028241295563 for External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2) by Johannes Hatzl, Christian Uhl, Alexandru Barb, Daniel Henning, Jonathan Fiering, Edris El-Sanosy, Philippe W.M. Cuypers and Dittmar Böckler in Journal of Endovascular Therapy