Abstract
Background:
Medical image segmentation is a well-studied subject within the field of image processing. The goal of this research is to create an AI retinal screening grading system that is both accurate and fast. We introduce a new segmentation network which achieves state-of-the-art results on semantic segmentation of color fundus photographs. By applying the net-work to identify anatomical markers of diabetic retinopathy (DR) and diabetic macular edema (DME), we collect sufficient information to classify patients by grades R0 and R1 or above, M0 and M1.
Methods:
The AI grading system was trained on screening data to evaluate the presence of DR and DME. The core algorithm of the system is a deep learning network that segments relevant anatomical features in a retinal image. Patients were graded according to the standard NHS Diabetic Eye Screening Program feature-based grading protocol.
Results:
The algorithm performance was evaluated with a series of 6,981 patient retinal images from routine diabetic eye screenings. It correctly predicted 98.9% of retinopathy events and 95.5% of maculopathy events. Non-disease events prediction rate was 68.6% for retinopathy and 81.2% for maculopathy.
Conclusion:
This novel deep learning model was trained and tested on patient data from annual diabetic retinopathy screenings can classify with high accuracy the DR and DME status of a person with diabetes. The system can be easily reconfigured according to any grading protocol, without running a long AI training procedure. The incorporation of the AI grading system can increase the graders’ productivity and improve the final outcome accuracy of the screening process.
Introduction
The devastating consequences of diabetes mellitus is set to continue as a result of the predicted increase in prevalence from 463 million in 2019 to 700 million in 2045 due to population expansion, increased ageing, urbanization, reduced physical activity, and adverse dietary changes. 1 The anticipated increase in prevalence of diabetes related retinopathy and sight threatening diabetic retinopathy world-wide from 2019 to 2045 is 160 to 245 million and 51 to 77 million, respectively.1,2 Vision loss is regarded as the most feared complication of diabetes. In the UK since 2000 the earlier detection of diabetic retinopathy utilizing digital photography of the retina has contributed to relegating diabetic retinopathy from being the commonest cause of blindness in the working age population.3,4 Preserving vision in this vulnerable population remains a priority commensurate with the objective of the World Health Organization (WHO) Universal Eye Health Global Initiative Action Plan 2014-2019 and emphasized in the World Report on Vision, 2020. 5 The NHS Diabetic Eye Screening Program (NDESP) covers all persons with diabetes aged 12 years or over with visual acuity of light perception or more in at least 1 eye, with screening performed every 1 or 2 years according to the perceived risk of developing, slight-threatening retinopathy. The protocol consists of testing visual acuity followed by two 45° field color eye retinal photographs: macula centered and optic disk centered. The disease can be categorized into 3 main stages - background diabetic retinopathy, pre-proliferative diabetic retinopathy and proliferative diabetic retinopathy, each stage is characterized by different anatomical features (Figure 1).
Examples of DR features.
Anatomical features in the background stage:
Microaneurysm—Saccular outpouchings of the capillary walls that can lead to fluid leakage and result in intraretinal edema and hemorrhages. Microaneurysms appear as small red dots which are often in clusters.
Dot/blot hemorrhages—Intraretinal hemorrhages may be round “dot” or oval “blot” shaped if they appear deep in the inner layer. The clinical differentiation between dot hemorrhages and microaneurysms is difficult and of little consequence in the absence of other lesions. Since both represent background retinopathy.
Hard exudates—made up of extracellular lipid and proteinaceous material which has leaked from abnormal retinal capillaries, hence there is often associated retinal oedema. The hard exudates can vary from small specks to larger patches and which may evolve into rings known as circinate exudates. Ultimately large confluent plaques can form. They are found principally in the macular region and as the lipids coalesce and extend into the center of the macula and impinge on the fovea, vision can be severely compromised.
Cotton-wool spots—also called soft exudates. They have a white fluffy appearance and are often located in the posterior pole of the fundus.
Anatomical features in the pre-proliferative stage:
Multiple blot hemorrhages (MBH)—Blot hemorrhages are larger than the width of the smallest of the 4 branches of the central retinal vein as it crosses the edge of the disc. Although the inherent difficulties of counting blot hemorrhages are recognized, most expert graders graded the MBH image set as having between 8 and 10 blot hemorrhages per eye.
Venous beading—a localized increase and decrease in caliber of the vein.
Venous looping—a localized looping of a vein deviating from its linear course.
Venous reduplication—a localized venous segment with 2 or more reuniting parallel branches.
Intraretinal microvascular abnormalities (IRMA)—a collection of dilated capillaries and abnormal intra-retinal new vessel formation. IRMA is considered present if it can still be seen on the color image, that has not been enlarged.
Anatomical features in the proliferative stage:
Neovascularization—as the retina becomes more ischemic new blood vessels may arise bud into fronds of multiple fine vessels on or within one-disc diameter of the optic disc (NVD) or beyond one-disc diameter away in the periphery of the retina (NVE).
Pre-retinal hemorrhage—Neovascularization (NV) can form abnormal adhesions with the vitreous body and those abnormal neovascularizations lack the normal layers of blood vessels can lead to hemorrhage. Where there is a localized detachment of the vitreous body from the retina, bleeding could occur. This blood can accumulate between the retina and the vitreous adopting the characteristic appearance of a subhyaloid hemorrhage. This is often said to be boat-shaped.
Vitreous hemorrhage—bleeding into the jelly-like filling of the back part of your eye named the vitreous body. Vitreous hemorrhage varies in degree from mild to complete loss of vision.
Fibrovascular proliferation—neovascularisation is accompanying by the process of fibrovascular proliferation. Fibrous tissue formation (gliosis) can progress and cause traction on the retina leading to retinal detachment. If the detachment extends across the fovea then the central vision will be deteriorated.
Segmentation of blood vessels, micro-aneurysms, exudates and hemorrhages in retinal images plays an important role in the diagnosis and monitoring of DR.6,7 Due to the large number of people with diabetes needing screening on a regular basis, and the scarcity of board-certified ophthalmologists, particularly in rural areas, 8 an automated DR segmentation tool could easily minimize the manual segmentation burden of ophthalmologists. Artificial intelligence systems using deep learning algorithms have been utilized for the automated detection of DR features from color fundus photographs.5,9-21 These systems are starting to be implemented into screening programs and have the potential to increase efficiency, reproducibility and coverage whilst also reducing screening costs and improving patient experience (earlier detection and treatment). This publication presents a novel convolutional neural network (CNN) architecture for medical image segmentation entitled W-Net, which achieves state of the art results on segmentation of fundus images for classification into 3 categories: adequate image quality or not, disease present or not, and finally sight-threatening DR present or not. The latter category will result in the patient being referred to the ophthalmologist for further assessment or for treatment if necessary.
Methods
The AI grading system evaluates the presence of diabetic retinopathy (DR) and diabetic macular edema (DME) from retinal images. The core algorithm of the system is a deep learning segmentation network (W-net) that locates and segments relevant anatomical features in a retinal image. Both eyes of the patients are graded individually, based on the detected features and according to the standard feature-based grading forms that are used in the NHS Diabetic Eye Screening Program.
The system is made of various subsystems algorithms and rulesets: (A) Image quality filter, (B) Macula localization, (C) Optic disc segmentation, (D) Lesion detection, (E) Laterality classifier, (F) Visual artefacts detection, and (G) Automatic feature-based grading (Figure 2).
AI grading system workflow.
Initially, each retinal image is run through an image classification algorithm, Low quality filter (A) (Figure 3), and is classified into the 3 mutually-exclusive states of “Adequate quality,” “Inadequate quality” or “Anterior eye image.” Next, we localize the macula and the optic disc with the Macula localization algorithm (B) and the Optic disc segmentation algorithm (C). The macula location data is later used in the final maculopathy grading stage. The optic disc segmentation data is used in the Laterality classifier (E).
Image quality samples by class: an image with an adequate image quality, an inadequate quality image and an anterior eye image.
Retinal images which were classified as “Inadequate quality” or which the macula is not localized are not subjected to any further processing and will be re-examined by a human grader. Because the final output of the system is a DR and DME grading of each eye separately, the patient images are then classified by laterality with the Laterality classifier algorithm (E). Images that are graded as “Adequate quality” are sent to the lesion detecting algorithm (D).
With the deep learning architecture, W-Net (D), the algorithm has the ability to detect very small findings in high resolution images, using a set of state-of-the-art deep learning models that scan the image at different size scales. The change from a single 1:1 image scale to multiple scales has positive effects on the final results since the findings can be small such as microaneurysms and exudates or large such as hemorrhages or fibrous proliferation. The lesion segmentation algorithm (D) was trained on 3 publicly available retinal images datasets - IDRiD, 22 e-ophtha 23 and OIA-DDR 24 and a private dataset provided by Northgate Public Services, UK - from routine diabetic eye screening Programs. All images used for the research project were anonymized. All rights in and to the private data provided for the research project vested in and remains with Northgate Public Services. Each retinal anatomical feature associated with diabetic retinopathy or diabetic macular edema are detected and classified. The predicted anatomical features are: microaneurysms (MA), hard exudates (HE), retinal hemorrhages (H), intraretinal microvascular abnormalities (IRMA), venous beading (VB), new vessels on the disc (NVD), new vessels elsewhere (NVE), retinal detachment (RD), fibrous proliferation (FP), vitreous hemorrhage (VH), and exudates (EX).
The imaging process itself sometimes introduces artefacts resulting from small imperfections or masking in the optical path. These artefacts bear some resemblance to small hemorrhages or microaneurysm and are commonly filtered out using a list compiled to include known locations of these image imperfections. We have developed the Visual artifact detection algorithm (F) that checks if the same patient has the same finding in more than one image at the exact same location, if so, we treat the finding as an artifact. This algorithm takes the hemorrhages and micro-aneurysm detections of the lesion detection algorithm (D) and removes the suspected artefacts from the final lesion detections set of findings.
The final diabetic retinopathy and macular edema grades are calculated by the feature-based grading module (G). The feature segmentation prediction data that comes of the subsystems is converted to tabular data and then used as an input of the feature-based grading module. The attributes that are extracted from the segmentations are the lesion type, the eye that has the lesion—left or right, its location and the size of the lesion. The feature-based grading is made of simple rule-based decision-making algorithms that are in line with the NHS Diabetic Eye Screening Program’s standard feature-based grading form. Every grading instruction in the NHS screening protocol form was translated to rule-based instruction. For example, if a set of patient images only has either a microaneurysm, a retinal hemorrhage according to the AI-predicted lesion table, then the AI will give the final verdict of “R1.” Additionally, the feature-based grading module uses the optic disc, fovea and macula location and segmentation data to determine whether the patient has diabetic macular edema or not. A patient will receive a DME grading of M1 if one of the following conditions is met: (1) one or more visible exudate within one disc diameter of the center of the fovea, (2) retinal thickening within one disc diameter of the center of the fovea (if stereo available), (3) any visible microaneurysm or hemorrhage within one disc diameter of the center of the fovea only if associated with a best visual acuity lower or equal to 6/12 (if no stereo).
In addition to the final grading algorithm outputs, DR positive/negative, DME positive/negative, the algorithm can also abstain from giving a final result. Results abstain means that the algorithm has found a suspicious finding, but the finding cannot be classified with a high-level of certainty. In this case, if no other findings were found—the system reports that the algorithm has failed to classify the region with high confidence and it presents the suspicious areas of such findings. Lesions other than DR or DME maybe present that require to be seen by a human grader.
Results
We evaluated the automated retinal image analysis system (ARIAS) performance on 6,981 screening episodes. The ARIAS classification is tested for both retinopathy and maculopathy. Section 2.1 presents classification results for retinopathy, while classification results for maculopathy appear in Section 2.2.
Retinopathy
Table 1 presents the ARIAS classification of disease or no-disease for each screening episode. The ARIAS demonstrates high sensitivity, namely a proportion of 98.9% of those with observable retinopathy were classified as disease. For screening episodes manually graded as R0 (no observable retinopathy), the specificity (ARIAS classification of nodisease) was 68.6%. The sensitivity and false-positive rates in terms of the 95% confidence limits (CI) around the estimates (obtained by bootstrapping) are presented in Table 2. The sensitivity for non-proliferative retinopathy with no referable maculopathy (R1M0) was 97.98% (95% CI 97.04%−98.87%), 96.83% (95% CI 94.56%−100%) for non-proliferative retinopathy with referable maculopathy (R1M1), 99.59% for pre-proliferative retinopathy (R2), 99.05% for stable proliferative disease (R3S), and 99.48% for active proliferative disease (R3A). The false-positive rate was 31.36% (95% CI 29.87%−32.68%).
Retinopathy: Outcome Classification of the Automated Retinal Image Analysis System Compared with manual classification that was performed by a grader (Manual Grade).
Retinopathy: Sensitivity and False-Positive Rates (%) for the Automated Retinal Image Analysis Systems Compared with Manual Grade.
Maculopathy
Classification results for maculopathy appear in Table 3. Table 4 presents the sensitivity (detection rates) point estimates and 95.5% CIs of the ARIAS. The overall sensitivity of the ARIAS for maculopathy is 95.52% (95% CI 58.1%−79.7%). The sensitivity appears to improve with the severity of detected retinopathy: the sensitivity for maculopathy in the presence of mild non-proliferative retinopathy (R1M1) is 66.7% (95% CI 55.5%−77.8%), increasing to 98.95% (95% CI 97.33%−100%) with pre-proliferative retinopathy (R2M1), and 96.13% with proliferative retinopathy (R3M1). The specificity of the ARIAS was 81.2%, namely 4,152 out of 6,981 screening episodes graded as no maculopathy were classified as no-disease. The false positive rate was 31.4% (95% CI 17.6%−19.8%).
Maculopathy: Outcome Classification of the Automated Retinal Image Analysis System Compared with Manual Grade.
Maculopathy: Sensitivity and False-Positive Rates (%) for the Automated Retinal Image Analysis Systems Compared with Manual Grade.
Discussion
It is widely acknowledged that with the ever-increasing prevalence of diabetes world-wide, 1 there is an urgent and continuing need to prevent sight loss and blindness in this vulnerable population, as diabetes remains one of the leading causes of preventable blindness in the world.25,26 The early detection and timely treatment of sight threatening retinopathy is essential to preserve vision. In addition, being aware of early retinopathy lesions will encourage health care professionals to address the modifiable risk factors in an attempt to delay its progression or even reverse retinopathy in its early stages. The introduction of community digital retinal image-based screening Programs for the detection and monitoring of diabetic retinopathy/maculopathy in the UK and elsewhere has contributed to a reduction in incident blindness registrations, 3 thereby reducing both the personal and socio-economic burden of this devastating and most feared complication of diabetes, in a cost-effective way in this vulnerable population.
Without the aid of automated retinal image analysis (ARIA), accommodating the volume of retinal images emanating from the recommended annual or biennial screening of all people with diabetes, which need to be graded in a timely manner, is unachievable when reliant on the limited human resources available. However, since 2008, when Abramoff et al 9 published the results of his automated retinal image analysis system, artificial intelligence using deep learning algorithms have been developed and evaluated for the automated detection of diabetic retinopathy based on color digital images with variable success.27,28 Auto-grading has also been achieved at lower costs when compared to human grading.29-31 Recently, a small number have received approval for clinical use by the U.S. Food and Drug Administration 32 whilst others have already been incorporated into screening services elsewhere in the world as a filter (disease/no disease) or as a substitute for human graders. Smartphone based image capture using hand held non-mydriatic cameras is currently being deployed in certain screening services to facilitate access to an even wider population of people with diabetes.21,33,34 The introduction of AI offers a solution to the anticipated increase in demand for grading in the foreseeable future.
The results of our research show a very high level of performance when using our novel deep learning segmentation network (W-net) algorithm to identify, locate and segment relevant retinopathy and maculopathy features in a color digital retinal image. Our neural network diabetic retinopathy classifier achieved a very high sensitivity of 98.9% for the presence of mild-to-moderate non-proliferative diabetic retinopathy in the absence of maculopathy with a specificity of 68.6%. The overall sensitivity of the classifier for the detection of maculopathy was 95.5% with a specificity of 81.2%. These findings compare very favorably with other AI based screening algorithms already adopted for clinical use. 28 The new algorithm also detects and falsely classifies abnormalities in the eye that are not related to DR. Typical false positive lesions include: various pigmentation, central retinal vein occlusion (CRVO), and age-related macular degeneration (AMD).
The current study involved a relatively small data set of color retinal images derived from a number of sources, thereby involving different populations and procedures which can influence the performance of the grading Program. Further research is therefore planned involving larger real-world population groups of different ages, sex and ethnicity derived from screening Programs with standardized image capture and grading procedures.
We plan to conduct a series of evaluations of our AI algorithm, with the first evaluation being this study to assess the AI’s ability to filter out those persons with ungradable poor-quality images, who will need to be referred to a specialist, and those with no disease who can be reviewed on a regular basis. Another evaluation will be on those with diabetic retinopathy or maculopathy who will be graded by a human grader to decide on the need for referral to an ophthalmologist, or not. It will be essential for us to fully explore discrepancies between human grader and the auto-grader to arrive at more meaningful assessment of sensitivity and specificity.
We will need to utilize a much larger data set in our future validation program to be able to confidently answer your question. We certainly acknowledge that the current data set is not rich in severe cases which are the main concern for ophthalmologists. This study is our initial evaluation where the main purpose was to identify disease or no disease or ungradable images. The context is that if disease is detected then the images would be referred to a human grader for stratification into referable or non-referable according to the severity of the retinopathy grade and or the presence of maculopathy. The auto-grader would therefore be functioning at this stage merely as a filter in order to reduce the number of images needing to be seen by the human grader. This will then allow the human grader to focus on grading the severity of the retinopathy or maculopathy whilst also still checking approximately 10% of those screened negative for retinopathy or maculopathy. The next stage in our evaluation of the algorithm will be to involve a much large pool of retinal images with a higher proportion of more severe retinopathy in order to evaluate the performance of the auto-grader in determining the need for referral, or not, to an ophthalmologist for further assessment or treatment as required.
The algorithm has been trained for lesion detection, and therefore, having identified the lesions, can allocate a severity grade according to the guidelines being applied. This next step as you suggest is very necessary to avoid the over referral of query maculopathy which is happening at present. Poor quality of images is a major limitation of any screening service. In the circumstances AI can only identify if the images are gradable or not, which means that the individual will need to be referred for specialist assessment. Purely technical failure can usually be resolved by re-photography with dilation.
The current version of the AI is able to differentiate no retinopathy from the presence of any retinopathy as observed from the sensitivity and specificity results where the majority of the lesions are at the lower end of retinopathy severity. The next version of our AI will be able to predict the severity of the retinopathy - R0, R1, R2, R3S, and R3A, with and higher sensitivity and specificity.
Conclusions
This new deep learning neural network diabetic retinopathy grading system evaluated in this study shows a high level of performance compared to compared to recent similar researches such as EyeArt by Eyenuk (sensitivity of 91.3%, specificity of 67.9%). 11 Our system achieves 98.9% detection sensitivity of DR combined with 68.6% specificity. This is despite the notion that the algorithm was trained on a relatively small dataset, due to the utilization of state-of-the-art neural network segmentation models. The system is not restricted to a specific grading protocol as the core algorithm detects anatomical features that are sent to a feature-based grading module that can easily be reconfigured without the need for a lengthy training procedure.
Our future plans include adding the ability to detect diabetic retinopathy and diabetic macular edema anatomical features that are missing from the current iteration of the AI, to classify other retinal lesions and to compare patient cases over different time periods.
It is now acknowledged that automated grading offers major benefits to screening Programs which includes enhanced grading efficiency and consistency along with increased capacity and reduced costs as compared with human grading. AI also can positively influence many other elements of the screening process to fully integrate eye health care within the existing health care systems to ensure accessible, timely and affordable care to prevent the loss of vision and blindness in this vulnerable population with diabetes.
Footnotes
Acknowledgements
All images used for the research project were anonymized. All rights in and to the private data provided for the research project vested in and remains with Northgate Public Services.
Abbreviations
(ARIAS) automated retinal image analysis system, (CI) confidence interval, (DME) diabetic macular edema, (DR) diabetic retinopathy, (NHS) National Health Service, (NDESP) NHS Diabetic Eye Screening Program
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
