An artificial intelligence (AI)-based approach to clinical trial recruitment: The impact of Viz RECRUIT on enrollment in the EMBOLISE trial

Abstract

Background

EMBOLISE (NCT 04402632) is an ongoing randomized controlled trial investigating the safety and efficacy of middle meningeal artery embolization for the treatment of subacute or chronic subdural hematoma (SDH). Viz RECRUIT SDH is an artificial intelligence (AI)-based software platform that can automatically detect SDH in noncontrast computed tomography (NCHCT) images and report the volume, maximum thickness, and midline shift. We hypothesized that the mobile recruitment platform would aid enrollment and coordinate communication and image sharing among the entire research team.

Materials and methods

Patient enrollment in EMBOLISE prior to and after implementation of Viz RECRUIT SDH at a large comprehensive stroke center was compared along with the performance of the software platform. The EMBOLISE trial was activated on May 5, 2021, and Viz RECRUIT SDH was activated on October 6, 2021. The pre-AI cohort consisted of all patients from EMBOLISE to AI activation (153 days), and the post-AI cohort consisted of all patients from AI activation until August 18, 2022 (316 days). All alerts for suspected SDH candidates were manually reviewed to determine the positive predictive value (PPV) of the algorithm.

Results

Prior to AI-software implementation, there were 5 patients enrolled (0.99 patients/month) and one screen failure. After the implementation of the software, enrollment increased by 36% to 1.35 patients/month (14 total enrolled), and there were no screen failures. Over the entire post-AI period, a total of 6244 NCHCTs were processed by the system with 207 total SDH detections (3% prevalence). 35% of all alerts for suspected SDH were viewed within 10 min, and 50% were viewed within an hour. The PPV of the algorithm was 81.4 (CI [75.3, 86.7]).

Conclusion

The implementation of an AI-based software for the automatic screening of SDH patients increased the enrollment rate in the EMBOLISE trial, and the software performed well in a real-world, clinical trial setting.

Introduction

In recent years, middle meningeal artery (MMA) embolization has gained prominence as a treatment modality for subacute or chronic subdural hematoma (SDH). Only a few studies have indicated the superiority of MMA embolization for both initial and recurrent management, and there is a need for more comprehensive data that address this approach and its utility further.^1–4 Our previous work has examined MMA embolization for chronic SDH in a large multicenter retrospective study in the adult population and a unique case report in the pediatric population.^5,6 These studies have demonstrated that MMA embolization for chronic SDH can be performed safely and effectively.^5,6 Among other investigative efforts, EMBOLISE (NCT 04402632) is an ongoing randomized controlled trial that aims to evaluate the safety and efficacy of MMA embolization using the Onyx™ Liquid Embolic System (LES) for the treatment of symptomatic subacute or chronic SDH. ⁷

The process of determining hematoma size, effacement of the cisterns, or midline shift to determine management approach and consequently eligibility for clinical trial recruitment in cases such as EMBOLISE can be complicated by reading times, subjectivity in reader interpretation, and delayed communication across the healthcare team workflow system. These challenges have prompted the development of artificial intelligence (AI) and machine learning systems to address the inherent issues of efficient patient identification and the coordination of treatment or clinical trial recruitment. The proposed advantages of AI-based recruitment include an automatic, 24/7 screening of all relevant scans (e.g., noncontrast computed tomography [NCHCT]) conducted at an institution, mobile alerting of suspected patients meeting customizable eligibility requirements, standardized measurements of key radiographic features, and immediate availability of all information to the entire research team in a mobile app with Health Insurance Portability and Accountability Act-compliant messaging for next steps. This system is set up to automatically receive NCHCT studies of patients for SDH detection and calculation of the volume, maximum thickness, and midline shift, which is provided to the stroke care team on the mobile app with an accompanying notification of meeting prespecified eligibility criteria. The stroke care team then reviews the results in the mobile image viewer feature of the application, minimizing the need for multiple forms of communication outside of the mobile app.

Recent studies have examined the workflow improvements^8–10 and detection capabilities^11–13 of an AI-based care coordination platform for LVO, ICH, and SDH, but there remains a paucity of data that examines the role of these applications in the setting of clinical trial recruitment. The primary focus of our study was to determine if the implementation of a mobile recruitment platform that detects SDH and reports volume, thickness, and midline shift, would assist enrollment by automatically pre-screening suspected candidates for SDH and coordinating communication and image sharing among the entire research team.

Methods

A retrospective study was conducted comparing enrollment in the EMBOLISE trial prior to and after the implementation of the AI recruitment platform at a large comprehensive stroke center. The EMBOLISE trial was activated on May 5, 2021. The AI recruitment system was activated on October 6, 2021. All head NCHCTs performed from two facilities, Valley Baptist Harlingen and Valley Baptist Brownsville, were automatically sent to the system. Two cohorts were formed for analysis: the pre-AI group, which consisted of all patients from the start of EMBOLISE to AI activation (153 days total), and the post-AI group, which included all patients from AI activation until August 18, 2022 (316 days total).

The AI software utilized in this study was Viz RECRUIT SDH (Viz.ai, Inc. San Francisco, CA). The application can notify the end user that an SDH has been identified and compiles the images for review in the mobile app. However, the images within the mobile app are compressed for informational purposes and do not serve a diagnostic purpose apart from initial notification and review. A confirmed SDH notification can be customized with a built-in ringtone specific for these cases to draw more focused attention to these notifications when they arise.

For both cohorts, the inclusion and exclusion criteria were in accordance with the EMBOLISE trial criteria. Inclusion criteria included patients who presented with a confirmed diagnosis of subacute or chronic SDH, premorbid Modified Rankin Score (mRS) ≤ 3, and completed informed consent. The exclusion criteria included patients diagnosed with acute SDH, life expectancy <1 year, Markwalder Grading Scale score ≥ 3, potentially dangerous anatomic variations leading to increased procedural risk or unsafe access for MMA embolization, contraindication for the use of Onyx™ LES, unmanaged, or uncontrolled bleeding disorders/blood diathesis. The risk and benefits of endovascular therapy treatment were discussed with patient families. The study received local institutional review board approval of waiver of consent and is in compliance with HIPPA.

Data collected

Patients who met the inclusion and exclusion criteria were selected from an institutional database that was compiled for patients who were enrolled in the EMBOLISE trial. Data was collected from the software including the alert notification viewing time longs, total NCHCTs processed and identified as SDH by the system, and SDH measurements such as volume, thickness, and midline shift.

Assessment of algorithm performance

To determine the positive predictive value (PPV), which is an assessment of the algorithm performance to alert correctly on cases that are true positives, the alerted NCHCTs from October 6, 2021 until August 18, 2022 were evaluated by one board-certified neurosurgeon and one board-certified neurologist. Any scans that were post-operative or contained surgical drains, shunts, or metal artifacts were excluded. After 13 exclusions, a total of 194 AI-alerted SDH NCHCTs were compared to manual diagnosis.

Statistical analysis

The enrollments were summarized with descriptive statistics, frequencies, and percentages for categorical variables. The PPV confidence intervals were calculated as Clopper–Pearson 95% confidence intervals.

Results

A total of 19 patients were enrolled in EMBOLISE trial during the study period. Prior to AI implementation, there were 5 patients enrolled at a rate of 0.99 patients/month and one screen failure as shown in Figure 1. After the implementation of the software for trial screening, there were an additional 14 patients enrolled at a rate of 1.35 patients/month with no screen failures (Figure 1). After implementing the software, the total enrollment increased by 36%. Furthermore, as demonstrated by Figure 2, there is a steady increase in enrollment after the implementation of the AI recruitment system. The enrollment increase tracked with the increase in the number of alerts, indicating the correct utility of the software.

Figure 1.

Pre-AI and post-AI patients enrolled per month.

Figure 2.

Temporal trend of AI-based alerts and trial enrollment.

Over the entire post-AI period, there were a total of 6244 NCHCTs processed by the software. Among these images, 207 were identified as SDH detections, indicating a 3% prevalence. A further breakdown of alert viewing time was also performed for additional clarification. Of all alerts for suspected SDH, 35% of these were viewed within 10 min and 50% were viewed within an hour. An example of the alerted information in these notifications, shared image viewing, and care team communication in the mobile application is presented in Figure 3.

Figure 3.

AI-based patient identification for an enrolled case.

In addition, a review of SDH alerts was performed to determine the PPV. Out of the 194 manually reviewed NCHCTs that were alerted as positive for SDH by the software and met the inclusion criteria, 158 were found to be true positive cases with a resulting PPV of 81.4 (CI [75.3, 86.7]). The true positive SDH cases had a median hematoma volume of 29.5 mL (IQR [12.0–53.5]), median thickness of 10.0 mm (IQR [7.0–13.8]), and median midline shift of 2.0 mm (IQR [0.0–Q3 4.0]). The most common reason (28/36 cases) for a false positive was brain atrophy.

Discussion

This study represents a small, single-center, retrospective assessment of patient enrollment and user experience before and after the implementation of an AI-based recruitment platform in the ongoing EMBOLISE clinical trial. After implementing the AI software for trial screening, enrollment rate increased by 36% compared to the pre-AI period. This increase in enrollment is indicative of the potential that AI software has in increasing efficiency and reducing chances of missed opportunities for clinical trial enrollment. By streamlining the process of identifying suspected SDH patients on a single, mobile application that alerts all relevant team members immediately for coordinated care, an “all eyes on the patient” approach can be optimized with in-app communication. Due to all NCHCTs processed by the system, more patients are automatically screened for potential inclusion based on radiographic parameters than ultimately enrolled. Yet, with a PPV of 81.5, the algorithm reliably alerted on primarily true positive cases. This finding is important as a low PPV could lead to “alert fatigue” ¹⁴ and likely cessation of using the tool.

The enrollment changes were likely not related to changes in the number of research staff or studies. In 2021 (mostly pre-AI), there were 5 clinical staff members and 10 active studies while in 2022 (all post-AI), there were 4 clinical staff members and 9 active studies. Thus, the staffing and number of studies stayed relatively constant across the pre- and post-AI period.

Currently, there are a few AI-based tools for the detection of SDH including Viz SDH, RAPID Hyperdensity, and TeraRecon Neuro. These tools have not been evaluated on their ability to impact clinical trial enrollment, and the literature on the real-world performance of these systems remains limited with only one large retrospective study of algorithm performance for Viz SDH. Our study is the first to assess the utility of such an algorithm to screen patients in a clinical trial. Prior work has investigated the impact of a more general intracranial hemorrhage detection tool, Viz RECRUIT ICH, in the ENRICH clinical trial with an observed increase in enrollment after software implementation.^15,16

Historically, numerous other studies have examined the role of AI and machine learning in the detection of ICH and SDH as a means to decrease delays in detection and diagnosis.^17–24 The primary limitation in these studies is the lack of one coherent system that allows for an interface that minimizes delays in detection along with communication and further management.^17,24 The system we used in this study was a kind of “one-stop-shop” platform that combined in-application imaging viewing, alert notification, and communication with AI-based detection and measurements. We hypothesize that this combination of features improved enrollment in EMBOLISE by allowing our entire research team to view suspected candidates and communicate the next steps in a single mobile application.

Limitations

The primary limitations of this study are the relatively small sample size and retrospective nature. With small sample sizes, the power of detecting differences in enrollment frequency is reduced. However, the general findings of our study are quite promising and can be bolstered and expanded upon with a large, multicenter, prospective study to provide more comprehensive data on the AI-based enhancement of enrollment in randomized clinical trials such as EMBOLISE.

In addition, our study found that 21% (43/207) of all notification alerts delivered by the mobile application were not viewed. Missed alerts potentially represent a missed enrollment opportunity. However, these alerts may not all have been “missed.” One possibility is that the images were viewed on an external computer application without opening the mobile application (from which the alert analysis was conducted). Due to push notifications being enabled on a mobile device, the alert can directly appear on the mobile device’s home screen and images can then be viewed on an external computer application without opening the mobile application itself. Another possible explanation for missed alerts could be that for many patients, the images were outpatients whose history was already known to the clinical team, and thus their evaluation in the mobile system was not necessary. Finally, for post-operative patients who underwent neurosurgical intervention such as a craniectomy or burr hole drainage, the images may have been previously viewed by the clinical team.

Other limitations of the study may be institution specific. Our enrollment numbers are based on a single trial for chronic SDH treated with MMA embolization via EMBOLISE, but a center with multiple trials involving SDH recruitment for MMA embolization may have enrollment dispersed among the trials. Furthermore, our center had a high motivation to recruit patients for the EMBOLISE trial (going from 8th highest enrolling site in July 2021 to 3rd highest in December 2022) with an understanding among all attending neurosurgeons that all eligible chronic SDH cases would be considered for trial enrollment. Together, these two institutional factors may limit the generalizability of our findings, and multicenter studies are warranted to minimize these potential institutional limitations.

Conclusions

The implementation of AI-based software for the automatic screening of SDH patients greatly increased our enrollment rate in the EMBOLISE trial. The early and quick detection of suspected SDH in all head NCHCTs performed well in a real-world, clinical trial setting with a PPV of 81%. The rest of the screening process was more efficient because the research team could evaluate a candidate in parallel with all imaging and communication shared among the entire team on a single mobile application. Additional studies should be conducted at other institutions to determine the generalizability of the observed enrollment increases and the effectiveness of AI-based recruitment.