Automated Diabetic Retinopathy Screening and Monitoring Using Retinal Fundus Image Analysis

Abstract

Background:

Diabetic retinopathy (DR)—a common complication of diabetes—is the leading cause of vision loss among the working-age population in the western world. DR is largely asymptomatic, but if detected at early stages the progression to vision loss can be significantly slowed. With the increasing diabetic population there is an urgent need for automated DR screening and monitoring. To address this growing need, in this article we discuss an automated DR screening tool and extend it for automated estimation of microaneurysm (MA) turnover, a potential biomarker for DR risk.

Methods:

The DR screening tool automatically analyzes color retinal fundus images from a patient encounter for the various DR pathologies and collates the information from all the images belonging to a patient encounter to generate a patient-level screening recommendation. The MA turnover estimation tool aligns retinal images from multiple encounters of a patient, localizes MAs, and performs MA dynamics analysis to evaluate new, persistent, and disappeared lesion maps and estimate MA turnover rates.

Results:

The DR screening tool achieves 90% sensitivity at 63.2% specificity on a data set of 40 542 images from 5084 patient encounters obtained from the EyePACS telescreening system. On a subset of 7 longitudinal pairs the MA turnover estimation tool identifies new and disappeared MAs with 100% sensitivity and average false positives of 0.43 and 1.6 respectively.

Conclusions:

The presented automated tools have the potential to address the growing need for DR screening and monitoring, thereby saving vision of millions of diabetic patients worldwide.

Keywords

diabetic retinopathy screening image processing turnover analysis DR monitoring automated analysis

Diabetic retinopathy (DR) is a common microvascular complication of diabetes that affects nearly all the type 1 diabetes patients and >60% of type 2 diabetes patients¹ and is the leading cause of new-onset blindness and vision loss among the working-age population in the developed world,¹ with approximately 24 000 people losing vision from DR each year in the United States alone. Studies have demonstrated that early detection and treatment of DR helps reverse DR signs, and slow DR progression.² Therefore, identification of patients in early stages of DR who are at high risk for progression to the sight-threatening stages of DR is crucial to reduce vision loss due to DR.² Annual DR screening is the current clinical recommendation¹ for diabetic patients. However, many diabetic patients do not get annual DR screening since DR can progress over several years without symptoms or discomfort before reaching a sight-threatening stage. The CDC reports that in 2010 US eye care practitioners examined less than 63% of the estimated 23 million people with diabetes, leaving millions of people at risk for potentially preventable visual loss and blindness.³

Most DR screening programs use digital fundus cameras to acquire color images of the retina. Multiple photographs are obtained for each patient encounter (or a visit), and then examined for the presence of pathologies indicative of DR including microaneurysms (MAs), hemorrhages, hard exudates, cotton wool spots, intraretinal microvascular abnormalities (IRMAs), and neovascularization. Manual DR screening is time-consuming and prone to inter- and intragrader variability. Manual DR screening alone cannot meet the screening needs of the large and growing diabetic population (The worldwide diabetic population is estimated to be 387 million in 2014 and reach 592 million by 2035.)⁴ Therefore, automated DR screening methods are essential for screening a large diabetic patient population for DR. Moreover, automated DR screening methods can make the DR screening process more efficient, cost-effective, reproducible, and accessible.⁵

The primary goal of DR screening systems is to identify patients who need referral to an eye care specialist for possible treatment due to elevated risks progressing to vision loss and blindness due to DR. The risks for progression are different for the various stages of the disease.⁶ The International Clinical Diabetic Retinopathy and Diabetic Macular Edema Disease Severity Scales (ICDR)⁶ were formulated by a consensus of international experts to capture the DR stages relevant to screening and standardize DR classification. The ICDR scale defines 5 stages of DR—no apparent DR, mild nonproliferative DR (NPDR), moderate NPDR, severe NPDR, and proliferative DR (PDR)—and apparent macular edema (ME) present or absent. The severe NPDR and PDR stages are termed sight-threatening DR due to the high risk of vision loss in these stages. There is clinical evidence that the risk of progression to sight threatening DR over several years in patients with no DR or mild NPDR without ME is very low, and therefore such patients usually need not be immediately referred to an eye care specialist and can be screened again in one or two years.⁵ Therefore, a DR screening system that identifies patients with moderate NPDR or ME and recommends referral to an eye care specialist for DR monitoring and/or clinical intervention will help reduce the burden on the scarce resources of eye care specialists.

Although the risk of patients with mild NPDR (characterized by only MAs) progressing to sight threatening DR is usually very low, there is clinical evidence that the rate of appearance and disappearance of MAs, termed as MA turnover, is a better predictor of DR progression risk than the DR level.^7-9 MAs, which are small swellings that occur in capillaries, are the first clinically evident signs of DR and are dynamic¹⁰ with new MAs being continuously formed while existing ones may disappear. Therefore, MA identification and monitoring is the first step in the clinical process of risk stratification of DR for appropriate patient education and possible treatment of high risk disease.

Despite the clinical utility, MA turnover is not widely used in current clinical practice since manual evaluation of MA turnover is time-consuming requiring two labor-intensive tasks: careful alignment of longitudinal fundus images (ie, fundus images from two different encounters) and identification of the MAs. In addition, the process of identifying and matching MAs suffers from poor reproducibility.¹¹ Automated analysis can provide fast, robust, and reproducible MA turnover metrics that can be used as a potential biomarker for DR risk monitoring.

In this article, we present and evaluate an automated tool for screening DR patients and its extension for estimating the MA turnover via longitudinal analysis. First we discuss EyeArt v1.2 (Eyenuk, Inc, Los Angeles, CA), a novel and robust automated DR screening system that can analyze variable number of images per patient encounter, identify gradable images (Figures 1A, 1B), exclude nonretinal images (Figures 1C, 1D) and nongradable images (Figures 1E, 1F) from analysis, and provide a recommendation for referral to an eye care specialist for patients whose encounter images show signs of moderate NPDR or higher or surrogate markers for ME on the ICDR scale. The EyeArt tool is implemented as a scalable, high-throughput, cloud-based system to enable analysis of large DR screening data sets. Next we extend the DR screening tool to perform longitudinal analysis across multiple encounters of a patient to estimate MA turnover, a potential biomarker for DR risk that can be used for DR monitoring. This novel MA turnover estimation tool, EyeMark v0.7 (Eyenuk, Inc, Los Angeles, CA), can automatically analyze images from multiple encounters of a patient with each encounter containing multiple images, detect MAs, identify corresponding retinal fields across encounters by registering the images, and then estimate MA turnover. Similar to the screening tool, the MA turnover estimation tool is cloud-based and designed for high scalability and throughput.

Figure 1.

Examples of images from the data set. A, B: Gradable retinal images. C, D: Nonretinal external images. E, F: Nongradable poor quality retinal images.

On a data set of 5084 diabetic patient encounters from EyePACS,¹² a DR telescreening system, we show that the DR screening tool achieves high screening sensitivity at reasonably high specificity values. With minimal workload of human graders having to review 26% of the encounters, system performance of 90% sensitivity and 90% specificity can be achieved. On a subset of 7 longitudinal pairs, we show that the MA turnover estimation tool identifies new and disappeared MAs with 100% sensitivity and average false positives of 0.43 and 1.6, respectively.

Methods

Data Set

We present the analysis on a diabetic patient encounter screening data set from EyePACS,¹² a DR telescreening system with users in over 360 primary care clinics in the United States and elsewhere. The EyePACS data set included a total of 40 542 color fundus images of 5084 diabetic patient encounters without any patient identifying data. The images were captured using multiple color fundus cameras at different clinics and had resolutions ranging from 1396 × 1396 (2 million pixels) to 3168 × 4752 (15 million pixels). The distribution of the DR and diabetic macular edema (DME) levels in this data set is presented in Table 1. Each of the 5084 encounters had 1 to 17 images including external eye images. Of these encounters, 60 were from 25 patients with at least 2 encounters, and these were identified by an anonymized patient id. These deidentified images were captured between April 2006 and December 2013 at various DR screening centers that use the EyePACS Retinal Reading Program.

Table 1.

Distribution of DR and DME Levels in the EyePACS Data Set as per Reference Standard Grading.

ICDR scale	Number of encounters (percentage of encounters)
ICDR scale	Without DME	With DME	Total
No apparent DR (0)	3619 (71.2)	0 (0.0)	3619 (71.2)
Mild NPDR (1)	608 (12.0)	1 (0.0)	609 (12.0)
Moderate NPDR (2)	543 (10.7)	117 (2.3)	660 (13.0)
Severe NPDR (3)	64 (01.3)	43 (0.8)	107 (02.1)
PDR (4)	41 (00.8)	42 (0.8)	83 (01.6)
Unknown (–1)	0 (00.0)	6 (0.1)	6 (00.1)
Total	4875 (95.9)	209 (4.1)	5084 (100.0)

Based on the presence and severity of the various DR lesions each patient encounter is assigned a DR severity level as per the international clinical diabetic retinopathy (ICDR) scale⁶ by a human expert, with 0 for no apparent DR, 1 for mild NPDR, 2 for moderate NPDR, 3 for severe NPDR, and 4 for PDR, and an ME level with 0 for no ME and 1 for ME. A patient was deemed to have nonreferable DR if there was no NPDR or mild NPDR and no ME in both eyes. The “refer” and “no refer” labels from the human experts were used as the reference standard.

Automated DR Screening

The computerized screening tool EyeArt assesses the severity of DR based on multiple color fundus images captured during a patient encounter, and provides a recommendation for referral to an eye care specialist if the patient is deemed to have ME or moderate NPDR or higher on the ICDR scale. The primary output of the tool is a “refer” or “no refer” screening recommendation. The screening tool is designed to provide a “refer” recommendation when it detects signs of moderate NPDR or higher on the ICDR scale, detects signs of ME, or deems the encounter as nongradable. EyeArt outputs a decision statistic that can be used to assess the confidence in the recommendation. In addition, the screening tool provides a “confident refer” recommendation when the decision statistic is above the 98% specificity threshold and a “borderline refer” recommendation when the decision statistic is between the 90% sensitivity and 98% specificity thresholds. The overall block diagram of the EyeArt system is shown in Figure 2. In the following sections we describe the different blocks.

Figure 2.

Overall block diagram of EyeArt DR screening system.

Image Gradability

The patient encounter comprising multiple color fundus images is first analyzed for gradability. Nonretinal images such as those of the external image of the eye and retinal images that cannot be graded for signs of DR due to various factors such as poor illumination, over-exposure and poor focus are excluded from further analysis. Generic descriptors including histogram of gradients (HOG),¹³ color histograms, sum modified Laplacian focus descriptors,¹⁴ Michelson-contrast descriptors¹⁵ are computed for each image and input to a supervised-learning classifier to produce an image-level gradability output. A patient encounter is deemed as gradable if it has at least one gradable retinal image per eye showing the macula. Nongradable patient encounters are not analyzed and flagged as having insufficient data.

Image Enhancement

Retinal fundus images have different colors levels, different dynamic range, and different sensor noise. Image enhancement is necessary to normalize the appearance and enhance the appearance of lesions (see Figure 3). For enhancement, the gradable images are first subjected to an edge-preserving bilateral filter¹⁶ to remove noise without affecting the important landmarks such as lesions and vessels and then subject to median normalization.

Figure 3.

Image enhancement examples on images from two different cameras. A, B: Original captured images. C, D: Corresponding enhanced images. Images from these different cameras are normalized to images with similar dynamic ranges, making them suitable for further analysis using the same algorithms with identical algorithm parameters.

The median normalized intensity $I_{N o r m, S}$ at pixel location $(x, y)$ is

$I_{N o r m, S} (x, y) = {\begin{cases} C_{m i d} + (C_{m i d} - 1) \cdot \frac{I (x, y) - I_{B a c k, S} (x, y)}{C_{m a x} - I_{B a c k, S} (x, y)}, if I (x, y) \geq I_{B a c k, S} (x, y) \\ C_{m i d} \cdot \frac{I (x, y)}{I_{B a c k, S} (x, y)}, o t h e r w i s e \end{cases}$

where $I$ is the input image with pixel intensities in the range $[C_{m i n}, C_{m a x}] = [0, 2^{B} - 1]$ , $B$ is the image depth, $C_{m i d} = 2^{B - 1}$ is the “middle” gray pixel intensity value, and $I_{B a c k, S}$ is the background image obtained using a median filter over the area $S$ .

Interest Region Detection

The enhanced images are then analyzed to identify regions of the image potentially containing interesting anatomical or pathological structures. The interest region detector is based on multiscale morphological filterbank analysis that identifies putative lesion-specific interest regions. On average, less than 1% of the pixels are identified as interest regions. For example (see Figure 4), on a retinal image with over 5 million pixels, further processing is focused on about 25,000 interesting pixels which is about 0.5% of the pixels in the image.

Figure 4.

Examples of interest regions identification. A: Original image showing lesions (bright yellow and dark red spots). B: Binary map showing interesting pixels forming less than 1% of the total pixels in the image.

Descriptor Computation

The pixels identified by the interest region detector are described using a set of feature descriptors (multiscale median filterbank descriptors, oriented median filterbank descriptors, and other established descriptors) that allow local description at multiple scales. The interesting pixels are analyzed using supervised learning classifiers for identifying MAs, hemorrhages, hard exudates, cotton wool spots, IRMAs, and neovascularization.

DR Screening Classification

The pixel-level classifier decision statistics are averaged within each lesion to obtain the lesion-level decision statistics. The histograms of lesion-level decision statistics for different lesion types are collated to generate the image-level descriptor. The image-level descriptors for all the gradable images in a patient encounter are combined to generate an encounter-level descriptor that is classified using a supervised learning ensemble classifier to detect the presence of moderate NPDR or higher on the ICDR scale.

Automated MA Turnover Estimation for DR Monitoring

The overall block diagram of the EyeMark system for automated computation of MA turnover is shown in Figure 5. Typically there are multiple images per patient encounter. Fovea centered images for each eye are automatically identified among the multiple images captured during both the baseline and longitudinal encounter and paired. These images are then aligned/registered using vasculature feature based multiscale registration. The aligned images are analyzed for MA using ensemble classification of many local image properties. The obtained MA confidence maps are used to evaluate turnover.

Figure 5.

Overview of automated MA turnover estimation.

Image Registration

Accurate registration is essential for tracking changes in small lesions such as MAs. Registration involves determining landmark structures that can be consistently identified in both the baseline and longitudinal images. We use retinal vasculature based landmarks¹⁷ since they can be assumed to be stationary/nonchanging across multiple encounters. Robust registration is achieved using a multiscale approach. Coarse vascular features at the coarsest scale are matched first to get a robust, coarse registration model that is then progressively refined using finer vasculature features at higher scales that are geometrically within an error window (“close enough”) as per the registration model computed in the coarser scale.

Ensemble Classification

The registered images are processed for localizing MAs as described in the section on Automated DR screening. The computed pixel descriptors are classified using an ensemble supervised learning classifier to obtain decision statistics describing the confidence of a pixel belonging to a MA. The pixel-level decision statistics are averaged within each “blob” and run through a sigmoid nonlinearity with negative values being clipped to zero, to generate the MA confidence map with confidence values in range $[0, 1]$ .

Turnover Estimation

The difference confidence map is the computed by subtracting the longitudinal MA confidence map from the baseline MA confidence map. Blob locations in the difference confidence map that have absolute values less than persistent lesion threshold ( $T_{p}$ ) and values higher than baseline lesion threshold ( $T_{b}$ ) in the baseline MA confidence map are marked as persistent MA locations with confidence as per the value in the baseline confidence map. Blob locations in the difference confidence map that are over a new lesion threshold ( $T_{n}$ ) are marked as new MA locations with the difference value as the confidence level. Blob locations in the difference image that are below disappeared lesion threshold ( $T_{d}$ ) are marked as disappeared MAs with the absolute of the difference value as the confidence level. Values of $T_{p} = 0.1$ , $T_{b}$ = 0.3, $T_{n} = 0.3$ and $T_{d} = - 0.3$ were empirically found to give the best results. Positive turnover rate is computed as the sum of confidence values of the new MAs divided by the time difference between encounters in years and negative turnover rate is the sum of confidence values of disappeared MAs divided by the time difference between encounters in years. The maps of new, persistent, and disappeared MAs are also output (see Figure 6).

Figure 6.

MA turnover analysis on a pair of longitudinal images. (A) Image from baseline encounter; (B) image from month 6 encounter; (C) MA changes visualization showing new MAs (N1, N2, N3, and N4) (in red), disappeared MA (D1) (in green), and persistent MA (P1, P2, P3, and P4) (in blue). Rectangles show example zoomed-in regions. The classification confidence is indicated by the color saturation for each lesion blob. Positive turnover rate was computed to be 2.3 and negative turnover rate was computed to be 1.1.

Results

All of the 5084 encounters from the EyePACS data set were processed by EyeArt screening tool. EyeArt marks 70 (1.4% of the data set) as nongradable and these encounters are considered to have a “refer” recommendation for this analysis. EyeArt screening sensitivity is 90.0% (95% CI: 88.0%-92.0%) at specificity of 63.2% (95% CI: 61.7%-64.6%) at an operating point of −0.84 on the decision statistic. The receiver operating characteristic (ROC) curve is shown in Figure 7. The area under the ROC curve (AUROC) is 0.879 (95% CI: 0.865-0.893). There are 87 false negatives (only 1.7% of the encounters) out of which 77 do not meet the treatment criteria, that is, have moderate NPDR and no ME.⁶ The sensitivity for detecting encounters with sight-threatening DR is 96.8%, that is, 304 encounters are recommended for referral of the 314 that are marked in the reference standard as having sight-threatening DR.

Figure 7.

ROC curve for identifying encounters with referable DR across 5084 patient encounters. The area under the ROC curve is 0.879 (95% CI: 0.865-0.893).

If the 1304 encounters with “borderline refer” recommendations are reviewed by a human grader then the system achieves 90% sensitivity at 90% specificity with 26% workload. This assumes that “borderline refer” encounters are assigned the same grade as the reference standard grade.

In all, 25 patients were imaged at multiple encounters and the longitudinal images from the patient encounters were analyzed using EyeMark. All the longitudinal pairs were successfully registered and MA turnover analysis was performed. On a subset of 7 longitudinal encounter pairs from the above set, an expert from Doheny Eye Institute marked the new, persistent, and disappeared MAs. As listed in Table 2, EyeMark has high sensitivity for new and disappeared MAs. Persistent MAs do not have any bearing for turnover estimation.

Table 2.

Sensitivity and Average False Positives in MA Dynamics Output by EyeMark.

	Sensitivity (%)	Average false positives
New MAs	100	0.43
Disappeared MAs	100	1.6
Persistent MAs	50	0.72

Discussion

We have presented a DR screening approach and have extended it to develop an image-based biomarker for DR. Together these tools can potentially be useful in automation of critical and time consuming aspects of DR screening and monitoring.

The EyePACS data set that we have analyzed in this article was chosen from an actual DR screening setup to demonstrate utility in actual clinical practice. On this data set, the automated screening tool has high sensitivity at good specificity values for identifying encounters with moderate NPDR or higher or with DME. With minimal workload of human graders having to review 26% of the encounters, system performance of 90% sensitivity and 90% specificity can be achieved in a telemedicine DR screening setup.

Generally, mild NPDR encounters are not referred to the eye care specialist. With the biomarker from the MA turnover estimation tool, mild NPDR encounters with high turnover rates can be flagged and scheduled for DR screening more frequently than the annual screening which would be commonly recommended. The noninvasive biomarker can be a valuable tool for clinicians and retina specialists to monitor DR progress and educate patients. Thus with DR monitoring, the progress of DR can be significantly slowed and risk of vision loss can be substantially reduced.

Conclusions

The DR screening tool achieves high sensitivity of 90% at reasonable specificity of 63.2% on a data set of 40 542 color retinal images from 5084 patient encounters obtained from the EyePACS telemedicine screening systems. With humans grading only the “borderline refer” encounters the screening tool provides a workload reduction of 74% (with 90% sensitivity and 90% specificity) on this data set. The MA turnover estimation tool achieves high sensitivity for new and disappeared MAs and can be reliably used to compute MA turnover rates.

The use of automated DR screening and monitoring tools as presented in this article can greatly reduce the burden on health care systems while providing improved care to the diabetic population.

Footnotes

Abbreviations

AUROC, area under receiver operating characteristic curve; DR, diabetic retinopathy; HOG, histogram of gradients; ICDR, International Clinical Diabetic Retinopathy; MA, microaneurysm; ME, macular edema; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy; ROC, receiver operating characteristic.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Malavika Bhaskaranand, Chaithanya Ramachandra, Sandeep Bhat, and Kaushal Solanki are employees of Eyenuk, Inc. Jorge Cuadros is an employee of EyePACS LLC. Muneeswar G. Nittala and SriniVas Sadda are employees of Doheny Eye Institute. All authors are part of the grants that partially funded this work.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was partially funded by NIH grants TR000377 and EB013585.

References

Fong

Aiello

Gardner

. Retinopathy in diabetes. Diabetes Care. 2004;27(suppl 1):s84-s87.

Klonoff

Schwartz

DM.

An economic analysis of interventions for diabetes. Diabetes Care. 2000;23(3):390-404.

Centers of Disease Control and Prevention. Diabetes report card 2012: national and state profile of diabetes and its complications. 2012. Available at: http://www.cdc.gov/diabetes/library/factsheets.html. Accessed April 16, 2015.

Whiting

Guariguata

Weil

Shaw

IDF diabetes atlas: global estimates of the prevalence of diabetes for 2011 and 2030. Diabetes Res Clin Pract. 2011;94(3):311-321.

Abràmoff

Niemeijer

. Mass screening of diabetic retinopathy using automated methods. In: Michelson

, ed. Teleophthalmology in Preventive Medicine. Berlin, Germany: Springer; 2015:41-50.

Wilkinson

Ferris

III Klein

. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110(9):1677-1682.

Nunes

Pires

Rosa

Duarte

Bernardes

Cunha-Vaz

Microaneurysm turnover is a biomarker for diabetic retinopathy progression to clinically significant macular edema: findings for type 2 diabetics with nonproliferative retinopathy. Ophthalmologica. 2009;223(5):292-297.

Ribeiro

Nunes

Cunha-Vaz

JG.

Microaneurysm turnover at the macula predicts risk of development of clinically significant macular edema in persons with mild nonproliferative diabetic retinopathy. Diabetes Care. 2013;36(5):1254-1259.

Haritoglou

Kernt

Neubauer

. Microaneurysm formation rate as a predictive marker for progression to clinically significant macular edema in nonproliferative diabetic retinopathy. Retina. 2014;34:157-164.

10.

Hellstedt

Immonen

Disappearance and formation rates of microaneurysms in early diabetic retinopathy. Br J Ophthalmol. 1996;80(2):135-139.

11.

Venkatesh

Sharma

Vashist

Vohra

Garg

Detection of retinal lesions in diabetic retinopathy: comparative evaluation of 7-field digital color photography versus red-free photography. Int Ophthalmol. 2015;35:635-640.

12.

Cuadros

Bresnick

EyePACS: an adaptable telemedicine system for diabetic retinopathy screening. J Diabetes Sci Technol. 2009;3(3):509-516.

13.

Dalal

Triggs

Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005;1:886-893.

14.

Nayar

Nakagawa

Shape from focus. IEEE Trans Pattern Anal Mach Intell. 1994;16(8):824-831.

15.

Michelson

AA.

Studies in Optics. Dover Publications, Inc., New York; 1995.

16.

Tomasi

Manduchi

Bilateral filtering for gray and color images. Sixth International Conference on Computer Vision. 1998:839-846.

17.

Frangi

Niessen

Vincken

Viergever

MA.

Multiscale vessel enhancement filtering. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI’98. New York, NY: Springer; 1998:130-137.