Abstract
Objective:
In the pivotal clinical trial that led to Food and Drug Administration De Novo “approval” of the first fully autonomous artificial intelligence (AI) diabetic retinal disease diagnostic system, a reflexive dilation protocol was used. Using real-world deployment data before implementation of reflexive dilation, we identified factors associated with nondiagnostic results. These factors allow a novel predictive dilation workflow, where patients most likely to benefit from pharmacologic dilation are dilated a priori to maximize efficiency and patient satisfaction.
Methods:
Retrospective review of patients who were assessed with autonomous AI at Johns Hopkins Medicine (8/2020 to 5/2021). We constructed a multivariable logistic regression model for nondiagnostic results to compare characteristics of patients with and without diagnostic results, using adjusted odds ratio (aOR). P < .05 was considered statistically significant.
Results:
Of 241 patients (59% female; median age = 59), 123 (51%) had nondiagnostic results. In multivariable analysis, type 1 diabetes (T1D, aOR = 5.82, 95% confidence interval [CI]: 1.45-23.40, P = .01), smoking (aOR = 2.86, 95% CI: 1.36-5.99, P = .005), and age (every 10-year increase, aOR = 2.12, 95% CI: 1.62-2.77, P < .001) were associated with nondiagnostic results. Following feature elimination, a predictive model was created using T1D, smoking, age, race, sex, and hypertension as inputs. The model showed an area under the receiver-operator characteristics curve of 0.76 in five-fold cross-validation.
Conclusions:
We used factors associated with nondiagnostic results to design a novel, predictive dilation workflow, where patients most likely to benefit from pharmacologic dilation are dilated a priori. This new workflow has the potential to be more efficient than reflexive dilation, thus maximizing the number of at-risk patients receiving their diabetic retinal examinations.
Keywords
Introduction
Diabetic retinal disease (DRD) is a highly prevalent complication in patients with diabetes mellitus (DM) and is a leading cause of blindness and visual loss in working-age adults. 1 More than 100 million adults worldwide are currently diagnosed with DRD, and this patient population is projected to double over the next 20 years. 2 While early diagnosis and intervention can significantly improve patient outcomes and avoid progression to irreversible vision loss, 3 the current management of DRD is limited by the inadequate adherence to regular eye screening in people with diabetes on a population level. 4 DRD and its complications are particularly pronounced in underserved/under-represented populations. 5 Efforts to improve access include development of an autonomous AI system for the diagnosis of DRD (IDx-DR, Digital Diagnostics, Coralville, IA). 6 This breakthrough device detects Early Treatment Diabetic Retinopathy Study (ETDRS) level 35 and/or center involved and/or clinically significant macular edema (referred to as “more than mild diabetic retinopathy” [mtmDR]), and was the first fully autonomous diagnostic AI system approved by the Food and Drug Administration (FDA) in any medical fieldIt is meant to be deployed in a primary care setting to evaluate patients with diabetes with no known DRD.
To operate the autonomous AI system, minimally skilled operators take images of patients’ retinae using a nonmydriatic retinal camera (NW400, Topcon Medical Systems, Oakland, NJ), resulting in four images per patient (one centered on the optic disk and one centered on the fovea, per eye). While these images are being taken, the operator is guided by the AI system, and once the quality is sufficient, then a patient-level output is generated (mtmDR present, mtmDR absent, or nondiagnostic result). The AI system will attempt to determine whether mtmDR is present or not, only if the images are of sufficient quality to be deemed diagnostic in the first place. Clinic efficiency is maximized when the number of nondiagnostic results is low.
In the pivotal clinical trial that led to this autonomous AI system's FDA De Novo approval, the protocol included reflexive dilation: if the AI system reported no diagnostic result after three attempts, the patient was (reflexively) pharmacologically dilated. While 76% of subjects received a diagnostic result without pharmacologic dilation, another 20% did require dilation, leading to a reported 96% overall diagnostic result, which is part of the FDA labeling. Similarly, in a real-world deployment study of IDx-DR in Poland, Grzybowski et al found that the diagnostic rate was 78% without pharmacologic dilation. 7
Although the pivotal clinical trial and other studies showedthat diagnostic rates can be increased by a reflexive dilation protocol,6,8,9 such a protocol results in workflow inefficiencies and risks the potential of patients dropping out entirely, as multiple iterations of imaging are required before pharmacologic dilation can be administered. Therefore, during the initial deployment (August 2020 and May 2021) at Johns Hopkins Medicine, reflexive dilation was not implemented. This allowed us to collect data and thereby identify pre-existing factors associated with nondiagnostic results, to characterize those patients most likely to have nondiagnostic results without dilation before the first images are taken. The novel predictive dilation protocol, as compared with a reflexive dilation protocol, has the potential to increase workflow efficiency, reduce delays for operators and patients, and increase patient and clinical staff satisfaction, thereby maximizing the potential of AI-based DRD screening in the primary care setting.
Materials and Methods
Study Design
This study was approved by the Johns Hopkins University Institutional Review Board (IRB00269812) with a waiver of informed consent. This was a retrospective case–control study.
Population
During the initial deployment phase between August 2020 and May 2021, autonomous AI systems were implemented at four Johns Hopkins Medicine primary care clinics in Maryland, with the standard protocol conforming to FDA labeling, except that no reflexive (pharmacologic) dilation was used. Consecutive adult patients with diabetes but no known DRD and thus eligible for screening, who presented for a primary care visit, were assessed using the autonomous AI system. Patients with nondiagnostic images had their images re-taken for a maximum of three total attempts. If the autonomous system still determined the scans to be nondiagnostic after these three attempts, then this patient was determined to be “nondiagnostic.” No pharmacological dilation was performed on any of these patients. A patient’s images can be deemed “nondiagnostic” for two reasons: poor image quality or inadequate field of view. We analyzed the nondiagnostic subjects and excluded those subjects that were nondiagnostic due to field-of-view deficiencies (10 subjects)—these deficiencies were likely the result of operators’ errors and would not benefit from pharmacologic dilation.
Only adult patients evaluated during this initial pilot deployment phase between August 2020 and May 2021 were included in our study; that is, patients evaluated by the autonomous AI system after May 2021, after which reflexive dilation was implemented, were excluded. We extracted demographic and medical variables from the electronic health record, including age, sex, race (African American, Asian, Native Hawaiian or Pacific Islander, White, and other), ethnicity (Hispanic or non-Hispanic), type of diabetes, body mass index (BMI), HbA1c, history of hypertension, smoking status, and time between initial system deployment and examination date (as a proxy for a learning effect).
Outcome
The primary outcome was the adjusted odds ratio (aOR) for each factor associated with a nondiagnostic examination result.
Statistical Analysis
Continuous variables are presented as median (interquartile range/IQR), and categorical variables are presented as n (%). Mann-Whitney U and Pearson’s chi-square tests were used to compare differences between diagnostic and nondiagnostic groups, as appropriate.
Multivariable logistic regression analysis was used to identify a priori risk factors for nondiagnostic results. Ten such factors were considered: sex, ethnicity, race, smoking status, hypertension, type 1 DM, type 2 DM, BMI, age, and HbA1c. We included these factors for analysis for two reasons. First, these factors are readily retrievable at each primary care visit where autonomous AI screening is performed, from either the electronic health record (EHR) or, alternatively, the patient. Second, these factors have been associated with either poor retinal image quality or poor pupillary dilation in the published literature, and hence are likely to be associated with nondiagnostic results as well. Variables such as age, BMI, and HbA1c were treated as continuous, since they produced better-fitting models compared with categorical terms. A final logistic regression model was selected using backward stepwise selection based on the lowest Akaike’s information criterion (AIC). This methodology starts with building a model using all variables under consideration (the 10 listed previously). Variables are subsequently removed from the model, one at a time starting from the least significant variable, until a reduced model with an optimal AIC is found. Risk factors are reported as aORs with 95% CIs. Statistical significance was set at P< .05.
Predictive Dilation Model
Using the results of this analysis, a predictive dilation model was built using stepwise backward selection. Five-fold cross-validation was used to evaluate model performance in identifying patients who would have nondiagnostic results without dilation. The population was evaluated five times, each containing a random and mutually exclusive data partition (80% as the training fold and 20% as the validation fold). Area under the receiver-operator characteristics curve (AUC) was calculated as the primary outcome. The final AUC measurement is the result of averaging the AUC of each evaluation to yield a more realistic real-world estimate of model performance. We also report sensitivity/specificity pairs and corresponding 95% CIs at different test thresholds.
Results
Descriptive Statistics
Out of 241 patients (41% male, median age = 59 years) who underwent autonomous AI screening, 123 (51%) were nondiagnostic. Those with nondiagnostic imaging had a significantly higher median age (64 vs 53 years, P < .001) and lower median BMI (32.0 vs 35.5 kg/m2, P = .005), compared with those with a diagnostic result (Table 1). In addition, patients with nondiagnostic results were more likely to be current smokers (30% vs 14%, P = .002) and less likely to have type 2 diabetes (T2D) (104/123; 85% vs 110/118; 93%, P = .03). Sex, race, ethnicity, history of hypertension, HbA1c, and time between examination date and initial system deployment were not different between the two groups.
Demographic and Clinical Variables of Patients With and Without Diagnostic Results by the Autonomous AI System.
Continuous variables: median (IQR). Categorical variables: n (%). Bolded P values are statistically significant.
Abbreviation: AI, artificial intelligence; BMI, body mass index; T1D, type 1 diabetes; T2D, type 2 diabetes.
In the multivariable logistic regression model, we identified three factors associated with nondiagnostic results: type 1 DM (aOR = 5.82, 95% CI: 1.45-23.40, P = .01), current smoking (2.86, 95% CI: 1.36-5.99, P = .005), and older age (every 10-year increase, aOR = 2.12, 95% CI: 1.62-2.77, P < .001) (Table 2). Nonwhite race (aOR = 1.85, 95% CI: 1.00-3.43, P = .05) and male sex (aOR = 1.80, 95% CI: 1.00-3.24, P = .05) trended toward statistical significance (Figure 1). As there were four different deployment sites with possible site-dependent variations, we conducted an exploratory clustering and mixed effects analysis with site as a random effect. The mixed effects analysis identified the same statistically significant factors associated with nondiagnostic results.
Multivariable Logistic Regression Model for Prediction of Nondiagnostic Results.
Abbreviations: aOR, adjusted odds ratio; CI, confidence interval.
Factors associated with nondiagnostic examinations. Receiver-operator characteristics (ROC) curves for five-fold internal cross-validation using 50% cutoff value; average area under the receiver-operator characteristics curve (AUC) = 0.76.
Predictive Dilation Model
Based on the results of the multivariate analysis, we used six priors (Table 2) to build the predictive dilation model. The model’s AUC was 0.76 (Figure 2), and the sensitivities and specificities of this model at different probability cutoffs were reported in Table 3. At a 50% cutoff value, the sensitivity and specificity were 77% and 68%, respectively.
Five-fold cross-validated ROC curves for the multivariable predictive dilation model. Each dotted line represents a single fold, while the solid red line represents the mean across all folds.
Sensitivity, Specificity, and AUC, With Corresponding 95% Confidence Intervals, as Well as Projected # Unnecessary Dilations (i.e., False Positives) and Projected # Missed Dilations (i.e., False Negatives) in a Theoretical Sample Size of 100 Patients.
Our multivariable model for predicting nondiagnostic results at different classification cutoff values. For example, a 50% cutoff value means that patients with a >50% probability of nondiagnosis will be classified as “nondiagnostic.”
Abbreviation: AUC, area under the curve.
Discussion
In our study, we determined that there are significant priors associated with lower diagnosability: type 1 DM, current smoker, and older age. As these priors are straightforward to determine, they are central in our novel predictive dilation workflow, in which patients at a high risk of nondiagnostic results are identified immediately and can then be dilated proactively, prior to the first retinal images being taken. The autonomous AI system (IDx-DR) utilized in our study is the first FDA-approved fully autonomous system in any medical field, 6 and several studies have evaluated the gradeability of consecutively obtained retinal images.7,10-13 However, a detailed analysis of patient factors related to nondiagnostic images was not conducted in any of these studies.
Real-world experience in deploying AI systems for automatic DRD detection has proven to be challenging. Clinical workflows are highly variable across different sites, and AI system implementation is limited by pre-existing logistics. For example, in a white article on AI-based DRD screening deployment in Thailand, Google researchers found issues with gradeability due to poor image quality (21% of patients), as in our study, to be a major limitation in AI implementation. 14 Nondiagnostic retinal images hinder widespread deployment and adoption of this technology, since it is resource intensive and frustrating for patients to retake several sets of images. Also, the resources and time deployed to take nondiagnostic images will be “wasted,” as these patients will ultimately need to be re-examined by humans. While the risk of complications from pharmacologic dilation is extremely low, 15 it is inconvenient for patients, as it increases light sensitivity and may prevent them from being able to drive after the examination.
In our multivariable logistic regression model, type 1 DM and age are among the strongest predictors for nondiagnostic imaging. Interestingly, compared with the pivotal clinical trial of IDx-DR, 6 our median age (59 years in both studies) and percentage of patients with type 1 DM (roughly 7% in both studies) were similar. However, our study population had nearly double the proportion of nonwhite patients (64% vs 38%), and nonwhite race showed a trend toward nondiagnostic images in our multivariable logistic regression model. Perhaps, the high percentage of nonwhite patients in our population was one of the reasons for our higher nondiagnostic rate (without reflexive dilation), as compared with the pivotal clinical trial.
While it is beyond the scope of this study to determine the causal relationship between our identified patient-level risk factors and nondiagnostic results, we investigated whether the associations were plausible from a pathophysiologic point of view. Type 1 DM patients typically have longer duration of DM, as compared with type 2 DM patients, which likely translates into more advanced systemic microvascular and nerve damage. It has been speculated that autonomic neuropathy affecting the pupillary sphincter and dilator muscles lead to small pupils in patient with diabetes under low luminance conditions. 16 Similarly, smoking affects autonomic function. Specifically, smoking reduces sympathetic nerve activity, 17 which drives pupillary dilation in humans. Regarding age, studies have shown that an older age corresponds to a significantly increased rate of ungradable retinal images during DRD screening, 18 and this correlation is likely due to the fact that older patients have smaller pupils that could lead to poor image quality. 19 To this point, our institution’s experience with implementation of IDx-DR in the pediatric population (the current study only included adult patients) demonstrated diagnosability of 97.5% without dilation, and in this population of 310 youth, age, sex, and race/ethnicity were not associated with increased imaging attempts. 20
We have created a “plug-and-play” multivariable logistic regression model to enable predictive dilation. The model requires six inputs: age, DM type, smoking status, race, sex, and hypertension status. The resulting novel predictive dilation workflow had an AUC of 0.76. This is exciting as it is a clinical decision support tool created for real-life clinical practice, to predict which patients will benefit from pharmacological dilation before the first image is taken. Its six priors are readily retrievable at a wide range of primary care settings. In addition, we reported different sensitivities and specificities of the model at different user-dependent cutoff thresholds (Table 3), which will allow individual clinics to set their own minimum thresholds for initiating predictive dilation protocols.
This workflow has the potential to improve clinical workflow efficiency. Specifically, a patient predicted by the model to have a high chance of a nondiagnostic result can be dilated and will be dilated by the time he/she undergoes autonomous AI screening. In contrast, although the traditional reflexive dilation model (per current FDA labeling) is capable of increasing diagnostic rate (from 76% to 96% in the pivotal clinical trial and from 49% to roughly 80% at our institution), efficiency is lost and will inevitably prolong the visit duration and create inconvenience for the patient and clinical team.
The strength of our study lies in the real-world implementation and the diverse patient population, in both community-based and hospital-based primary care clinics within our integrated healthcare system. Our patient population had a higher portion of minority patients, as compared with other studies that investigated the usability and diagnostic rate of the IDx-DR system, including its pivotal clinical trial. The weaknesses of our study include its retrospective nature and relatively small sample size. Another limitation of our study is that the model did not control for operator or operator experience. However, a learning effect was not present, as there was no difference in time between examination date and initial system deployment between the nondiagnostic and diagnostic groups. We purposefully did not include field-of-view deficiencies in the analysis to account for this. As the next step, we plan to use our data to train other common machine learning models, such as random forests and boosting, test our models prospectively in a clinical setting, and to compare the efficiency of clinics that utilize a reflexive dilation protocol versus clinics that utilize a proactive dilation protocol.
Conclusions
In summary, we show that nondiagnostic results in real-world point-of-care DRD detection using autonomous AI can be predicted, and the resulting predictive dilation workflow has the potential to increase screening rates and improve access to early treatment for DRD, and thereby preventing visual loss and blindness in these vulnerable populations.
Footnotes
Abbreviations
AI, artificial intelligence; AIC, Akaike’s information criterion; aOR, adjusted odds ratio; AUC, area under receiver-operator characteristics curve; BMI, body mass index; CI, confidence interval; DM, diabetes mellitus; DRD, diabetic retinal disease; ETDRS, Early Treatment Diabetic Retinopathy Study; FDA, Food and Drug Administration; IQR, interquartile range; mtmDR, more than mild diabetic retinopathy.
Authors’ Contributions
Conceptualization—TYAL
Methodology—BLS, CC, RG, JHL, JW, MDA, TYAL
Formal analysis—BLS, CC, RG, JHL
Investigation—all authors
Data curation—BLS, KV, TYAL
Writing—Original draft—BLS
Writing—Review & Editing—all authors
Visualization—BLS
Supervision—TYAL, MDA, RMW, RC
Project administration—TYAL
Data Availability
The data set generated during this study that was used to generate the predictive model parameters is available upon reasonable request from the corresponding author TYAL.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: RMW receives research funding from Dexcom, Inc. and Boehringer Ingelheim, unrelated to this research. MDA reports the following conflicts of interest: Investor, Director, Consultant, and Digital Diagnostics; patents and patent applications assigned to the University of Iowa and Digital Diagnostics that are relevant to the subject matter of this article; Chair Healthcare AI Coalition, Washington DC; member, American Academy of Ophthalmology (AAO) AI Committee; member, AI Workgroup Digital Medicine Payment Advisory Group (DMPAG). The remaining authors declare no competing financial or nonfinancial interests.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Conference Presentations
This project was presented as a podium presentation at the Association for Research in Vision & Ophthalmology 2022 Annual Meeting, May 1-5, 2022, Denver, CO, USA.
