Flag on the Field: Identifying Epilepsy Surgery Candidates Using Machine Learning

Abstract

Early Identification of Candidates for Epilepsy Surgery: A Multicenter, Machine Learning, Prospective Validation Study

Wissel BD, Greiner HM, Glauser TA, Pestian JP, Ficker DM, Cavitt JL, Estofan L, Holland-Bouley KD, Mangano FT, Szczesniak RD, Dexheimer JW. Neurology. 2024 Feb;102(4):e208048. doi:10.1212/WNL.0000000000208048. Epub 2024 Feb 5. PMID: 38315952

Background and objectives: Epilepsy surgery is often delayed. We previously developed machine learning (ML) models to identify candidates for resective epilepsy surgery earlier in the disease course. In this study, we report the prospective validation. Methods: In this multicenter, prospective, longitudinal cohort study, random forest models were validated at a pediatric epilepsy center consisting of 2 hospitals and 14 outpatient neurology clinic sites and an adult epilepsy center with 2 hospitals and 27 outpatient neurology clinic sites. The models used neurology visit notes, EEG and MRI reports, visit patterns, hospitalizations, and medication, laboratory, and procedure orders to identify candidates for surgery. The models were trained on historical data up to May 10, 2019. Patients with an ICD-10 diagnosis of epilepsy who visited from May 11, 2019, to May 10, 2020, were screened by the algorithm and assigned surgical candidacy scores. The primary outcome was area under the curve (AUC), which was calculated by comparing scores from patients who underwent epilepsy surgery before November 10, 2020, against scores from nonsurgical patients. Nonsurgical patients’ charts were reviewed to determine whether patients with high scores were more likely to be missed surgical candidates. Delay to surgery was defined as the time between the first visit that a surgical candidate was identified by the algorithm and the date of the surgery. Results: A total of 5285 pediatric and 5782 adult patients were included to train the ML algorithms. During the study period, 41 children and 23 adults underwent resective epilepsy surgery. In the pediatric cohort, AUC was 0.91 (95% CI 0.87-0.94), positive predictive value (PPV) was 0.08 (0.05-0.10), and negative predictive value (NPV) was 1.00 (0.99-1.00). In the adult cohort, AUC was 0.91 (0.86-0.97), PPV was 0.07 (0.04-0.11), and NPV was 1.00 (0.99-1.00). The models first identified patients at a median of 2.1 years (interquartile range [IQR]: 1.2-4.9 years, maximum: 11.1 years) before their surgery and 1.3 years (IQR: 0.3-4.0 years, maximum: 10.1 years) before their presurgical evaluations. Discussion: ML algorithms can identify surgical candidates earlier in the disease course. Even at specialized epilepsy centers, there is room to shorten the time to surgery. Classification of evidence: This study provides Class II evidence that a machine learning algorithm can accurately distinguish patients with epilepsy who require resective surgery from those who do not.

Commentary

Approximately one-third of patients with epilepsy have seizures that are resistant to anti-seizure medications (ASMs), and these individuals may be candidates for epilepsy surgery. Many patients achieve complete seizure freedom after epilepsy surgery, but reduced seizure frequency and severity can also lead to meaningful improvements in quality of life. However, epilepsy surgery is dramatically under-utilized, as less than 5% of individuals who are candidates receive surgical intervention, and the mean duration of epilepsy in patients who do have surgery is often upwards of 20 years. ¹ These delays contribute to cognitive decline, difficulty with work and school, progressive brain network disturbances, and increased risk of death. Numerous factors contribute to underutilization of epilepsy surgery, including patient and family-related issues, as well as barriers related to physicians and health systems. ² In recent decades, a perceived lack of progress in improving access to epilepsy surgery has led to growing frustration in the field. How can we better identify individuals in our healthcare system that may be favorable candidates for further intervention?

In the presently highlighted study, Wissel and colleagues utilize machine learning approaches to “flag” patients who may be favorable candidates for epilepsy surgery using data from the electronic medical records (EMR). ³ A large cohort of over 11,000 children and adults from multiple hospitals and neurology clinics were prospectively included over a one-year period. The authors’ primary outcome measure in testing their model was the area under the receiver operating characteristic curve in identifying individuals who received surgery, which was favorable in both adults (0.91) and children (0.91). While many machine learning studies contribute limited understanding about which variables drive their models (“black box” approach), the authors’ here used permutation measures to help interpret predictions for individual patients. These analyses suggested that the number and duration of office/hospital visits and number and type of ASMs were predictors of surgical candidacy in both children and adults, and age was also a predictor in the pediatric cohort. Overall, these interpretable results suggest that early identification of surgical patients from EMR data may indeed be feasible using a neural network approach.

The use of machine learning approaches to facilitate epilepsy care is rapidly increasing, and includes models to help guide ASM changes, ⁴ improve localization of a seizure focus, ⁵ and direct surgical decisions. ⁶ However, this work by Wissel et al using machine learning to help identify surgical candidates is relatively novel in the field of epilepsy surgery, although it has been explored in several other surgical subspecialties (eg, ⁷ ). The currently highlighted manuscript follows a preliminary investigation by Wissel and colleagues using machine learning to flag surgical candidates from EMR data, but that prior study used retrospective data and a narrower breath of clinical features. ⁸ One strength of the presently highlighted manuscript is that the authors chose a random forest model that is relatively easy to implement and train compared to large language models, which may help lower the barrier to future clinical implementation. Despite heterogeneity across individual practices throughout the cohort, the algorithm utilized was able to identify consistent signals in patient records that may aid in early identification of surgical candidacy. Interestingly, despite historical imbalances in access to epilepsy surgery among racial minorities, race was not a predictor of surgical candidacy. As noted by the authors, this implies that once alerted, provider may refer patients at similar rates regardless of race, and that this work may help narrow the gap in access to care.

One limitation of the study is that while it is presented as a validation cohort, one could argue it functions more as a test set, as it is not utilizing an exact model from the authors’ prior study, and novel clinical variables are now incorporated. Thus, further validation in additional cohorts will be worthwhile in the future. Also notable, the positive predictive value in identifying patients who underwent epilepsy surgery was low at 0.07 to 0.08 for adults and children, respectively. However, these values need to be taken in context, as they refer to the prediction of individuals that were candidates for surgery and then actually underwent surgery. The proportion of favorable candidates for epilepsy surgery that ultimately find their way to the operating room is unfortunately low, influencing these results. In the future, the authors may consider comparing the model's scores against clinical experts, to validate that the model's ability to identify appropriate surgical candidates, even if an individual patient does not ultimately receive surgery.

What are next steps and potential barriers to moving similar research models to the clinical realm in epilepsy surgery? Throughout various areas of medicine, a large body of work now exists evaluating the best data sources, modelling methods, and reporting strategies for EMR phenotyping, ⁹ and this direction should be further explored in epilepsy surgery. One could envision a computational model that could flag a patient as a potential surgical candidate during a healthcare visit, and the patient's provider could then direct care accordingly. In a recent clinical study also by Wissel and colleagues, the authors used such a model to identify surgical candidates at the beginning of pediatric epilepsy clinic visits. ¹⁰ Patients whose provider received the alert were then more likely to be referred for a presurgical evaluation, suggesting feasibility and value in this approach. Implementation of such models based on EMR data does require adequate informatics infrastructure, which may be a challenge in small clinical practices, but is becoming more common in large health care systems.

The rapidly growing use of machine learning approaches to guide medical care will continue, although careful validation and appropriate caution are needed. Given the tremendous societal burden associated with drug resistant epilepsy, new help to identify appropriate patients and guide surgical decision will be needed and welcomed.