The Real-World Impact of Artificial Intelligence on Diabetic Retinopathy Screening in Primary Care

Abstract

The study by Shah et al published in this issue of the Journal of Diabetes Science and Technology validates the IDx autonomous diabetic retinopathy (DR) screening program in a real-world setting. The study found high sensitivity (100%) but low specificity (82%) for referable DR. The resulting positive predictive value of 19% means that four out of five patients without referable DR would be referred to ophthalmology causing a significant burden to ophthalmologists, primary care clinics, and patients. Artificial intelligence programs that provide better specificity, multiple levels of DR, and annotations of where lesions are located in the retina may function better than a simple referral/no referral output. This will allow for better engagement of patients through the difficult process of adhering to treatment recommendations and control their diabetes.

Key Message

High specificity and low failure rates are important attributes of AI-based diabetic retinopathy screening.

Analysis

Deep learning algorithms for screening diabetic retinal disease are celebrated as a successful application of artificial intelligence (AI) in health care. In 2018 IDx became the first autonomous computer program to be approved for screening diabetic retinopathy and diabetic macular edema (collectively called DR in this article) by the US Food and Drug Administration. Hundreds of similar AI programs exist in different stages of development throughout the world and promise to improve accuracy and access to diabetic retinal disease detection. The study by Shah et al published in this issue of the Journal of Diabetes Science and Technology validates the IDx retinopathy grading program in a real-world setting. The study compares IDx to manual grading by ophthalmologists of retinal images captured in primary care clinics in Valencia, Spain. This article comments on the implications of the results for clinical care.

The rationale for screening DR is to partition those patients with diabetes who need ophthalmological intervention from those who can continue with general routine eye care services. We expect that early detection of vision-threatening DR leads to successful treatment with retinal laser or intraocular injections. This expectation should be tempered by the multiple barriers affecting adherence to treatment recommendations among patients with diabetes. Many, if not most, patients do not adhere to referral recommendations from DR screening.^1-3 Barriers such as insurance eligibility, appointment backlogs, transportation, comorbidities, distrust of the health systems, social isolation, fear, depression, and other social determinants of health will affect patient adherence to ophthalmic treatment just as these factors contribute to the uncontrolled diabetes that gave rise to the complications. ⁴ These common barriers need to be considered when developing AI programs for DR screening so that the intervention has the best chance of preventing blindness. The Shah study found nearly 100% sensitivity, but only 82% specificity when IDx was compared to the adjudicated grades. This led to a positive predictive value of only 19%. Four out of five patients who are referred to ophthalmology with suspected referable retinopathy will then be told that they don’t have more than mild retinopathy. False-positive referrals caused by low specificity significantly affect the resources required by the primary care clinic to make and follow the specialist referrals and the effort required by the patient to attend the ophthalmology visits. Patients and clinics lose trust in the referrals and likely decline future ophthalmology appointments, when more advanced retinal disease is likely. Diabetes UK developed guidelines for DR screening programs. They recommend a minimum of 95% specificity ⁵ for any screening program, perhaps for this same reason.

When images are not adequate to be graded by the algorithm, then the patient can be imaged again or referred for a live exam. In the Shah article this occurred in over 400 patients. In a primary care setting, this amounts to the same as a false-positive referral. This problem is worsened when using devices such as current handheld retinal cameras ⁶ whose image quality is usually insufficient for full interpretation.

Rethinking the role of AI in DR screening is important to have a viable intervention in real-world clinical settings. Instead of focusing on lowering the cost of ophthalmology services, it is more cost effective to focus on developing interventions that ensure blindness prevention by engaging patients in their eye care and their chronic disease. Effective DR screening should guide the patient in their journey through ophthalmology treatment, a journey that is often initiated before the patient feels any symptoms. AI in DR screening provides immediate identification of patients who might be at risk of vision impairment creating an opportunity for point-of-care engagement, by showing patients the microvascular lesions that are marked by the AI. In this scenario the AI should not only segment referable retinopathy with high specificity, but it should also stratify all levels of retinopathy and provide region of interest information. This is an opportunity for patients to understand and accept the steps they need to take to prevent vision loss.

Conclusion

Low specificity for detecting referable retinopathy is unacceptable for DR screening programs. Currently available AI for DR screening may not have enough specificity to serve as a standalone intervention to refer patients for ophthalmology treatment. AI provides an excellent point-of-care opportunity to engage patients about the risk of vision impairment and may lead to improved glycemic control when lesions can be identified that demonstrate microvascular complications of diabetes. AI + human intervention at the point of care can refine and ensure the referral to ophthalmology in order to preserve this scarce specialty resource.