A study of whether automated Diabetic Retinopathy Image Assessment could replace manual grading steps in the English National Screening Programme

Abstract

Objectives

Diabetic retinopathy screening in England involves labour intensive manual grading of digital retinal images. We present the plan for an observational retrospective study of whether automated systems could replace one or more steps of human grading.

Methods

Patients aged 12 or older who attended the Diabetes Eye Screening programme, Homerton University Hospital (London) between 1 June 2012 and 4 November 2013 had macular and disc-centred retinal images taken. All screening episodes were manually graded and will additionally be graded by three automated systems. Each system will process all screening episodes, and screening performance (sensitivity, false positive rate, likelihood ratios) and diagnostic accuracy (95% confidence intervals of screening performance measures) will be quantified. A sub-set of gradings will be validated by an approved Reading Centre. Additional analyses will explore the effect of altering thresholds for disease detection within each automated system on screening performance.

Results

2,782/20,258 diabetes patients were referred to ophthalmologists for further examination. Prevalence of maculopathy (M1), pre-proliferative retinopathy (R2), and proliferative retinopathy (R3) were 7.9%, 3.1% and 1.2%, respectively; 4749 (23%) patients were diagnosed with background retinopathy (R1); 1.5% were considered ungradable by human graders.

Conclusions

Retinopathy prevalence was similar to other English diabetic screening programmes, so findings should be generalizable. The study population size will allow the detection of differences in screening performance between the human and automated grading systems as small as 2%. The project will compare performance and economic costs of manual versus automated systems.

Introduction

Diabetic Retinopathy (DR) is a common complication of diabetes. It is no longer the leading cause of vision loss in the working age population in England and Wales¹, probably largely due to the effectiveness of the English National Screening Committee for Diabetic Retinopathy (ENSCDR) programme² in identifying those requiring treatment, particularly early treatment, which is effective in preventing visual loss.¹ Over 2.5 million people aged 12 or over have diagnosed diabetes in England, of whom 2.3 million have been offered screening and 1.9 million have been screened.³ This represents a major challenge for the NHS. Retinal screening is labour intensive, requiring trained human graders, and future numbers will escalate as the prevalence and incidence of diabetes are increasing markedly.⁴

In Scotland one automated system achieved very high detection rates (100% for proliferative retinopathy and over 97% for maculopathy) in a large, unselected population attending two regional screening programmes.⁵ The system differentiated between those with sight-threatening DR or other retinal abnormalities, and those at low risk. Other systems have since become available, with reported diagnostic accuracy comparable with that of experts,^6–12 but these are not currently authorized for use in ENSCDR programmes and their clinical validity and applicability in routine screening remains unclear. Recent work has shown that Optical Coherence Tomography (OCT) imaging is a useful adjunct to colour fundus photography in screening for referable diabetic maculopathy, with the potential to significantly reduce over-referral of cases of macula oedema compared with human assessment of colour fundus images.¹³ Some Hospital Services are already incorporating OCT assessment within their NHS Diabetic Eye Screening Programmes (DESP).¹⁴

This paper presents the research methods and analysis plan for a study to quantify the screening performance and diagnostic accuracy of three commercially available automated grading systems, compared with ENSCDR manual grading in diabetic patients undergoing diabetic retinopathy screening in a large sample of real life screening episodes. A sub-set will be validated by an approved Reading Centre. The health economics of automated versus manual grading will provide a key decision-making tool to determine whether and how automated screening should take place for diabetic retinopathy in the future. We will also evaluate the health economics of including OCT pathways within the DESP, which may further improve screening performance.

Study Design and Methods

Study population

This observational retrospective study is based on data from 20,258 consecutive diabetes patients aged 12 or older who attended the annual DESP of the Homerton University Hospital in London between 1 June 2012 and 4 November 2013. Following ENSCDR guidelines, 142,018 images from 28,079 screening episodes (repeat episodes are available for 7,821 patients) have been graded manually.^15,16 All manual grades were stored and accessed using the Digital Health Care system version 3.6. Patients with no perception of light were excluded from the study.

Clinical practice

The NHS diabetic retinopathy screening protocol is followed at the time of screening has been published elsewhere.¹⁷ Four images are required per patient, for each eye one image centred on the optic disc, and one centred on the macula. Additional images are often taken to ensure sufficient quality and to document other pathology (eg. cararact). Gradings for ‘eye for which action is most urgently required’ in order of increasing severity are no retinopathy (R0), background retinopathy (R1), ungradable images (U), maculopathy (M1), pre-proliferative retinopathy (R2) and proliferative retinopathy (R3).^15,16 Up to three human graders who meet ENSCDR quality assurance standards assess images. Level 2 grading of images is carried out by senior graders. Disagreements between level 1 and level 2 graders for episodes that are potentially M1 or R2 are sent for arbitration to a third grader, whose assessment is final. Patients with grades U, M1, R2, and R3 are referred to hospital ophthalmologists, and those with grades R0 and R1 are invited for re-screening in 12 months. In the screening service for much of the period during which images were retrieved, patients known to be photographically ungradable were technically failed at the photography level and underwent slit lamp biomicroscopy in clinic. These patients have been omitted from the dataset (since they have no gradable images). This has reduced the percentage of ungradable images in the set analysed.

Automated grading systems

Automated DR grading systems with a CE mark obtained or applied for up to 6 months after July 2013 were eligible for evaluation. From a literature search and discussions with experts, three software systems were selected and all three agreed to participate in the study: iGrading (version 1.1 by Medalytix),^18,19 Retmarker (version 0.8.2. 2014/02/10 by Critical-Health),²⁰ IDx-DR (by IDx).^21,22 For commercial reasons, IDx withdrew from the study before the analysis, and in 2013 another company, Eyenuk Inc, (software EyeArt²³) asked to join the study, confirming that it would meet the CE mark eligibility criteria. Three systems (iGrading, Retmarker and EyeArt) will therefore process all 28,079 screening episodes with 142,018 images. Permission to extract pseudoanonymized images and process these using automated systems has been obtained from the Caldicott Guardian. No formal ethics approval was required, as all extracted images were anonymized, and no change in clinical pathway will occur.

All automated systems are designed to identify DR cases of R1 or above. EyeArt is additionally designed to identify cases requiring referral to ophthalmology (U, M1, R2 or R3). Retmarker and EyeArt process all the images associated with a screening episode and provide a classification per episode. The iGrading system provides an outcome classification for each image.

To compare screening performance across the automated systems with the reference standard, episodes must be classified as (i) Disease present or technical failure or (ii) Disease absent. Retmarker automatically classifies screening episodes in this way. For EyeArt, episodes classified as Disease, Definite disease, and Ungradable will form group (i) and those classified as ‘Definite no disease’ and ‘No disease’ will form group (ii). For iGrading, screening episodes group (i) will include episodes classified as ‘Disease present’ or ‘Ungradable’, using the following approach:

If the outcome of at least one image in a screening episode is classified as ‘Disease’, then the outcome classification for the episode will be ‘Disease present’.

If the outcome of at least one image in a screening episode is classified as ‘Ungradable’ and ‘No Disease’ is detected in all the other images of the screening episode, then the outcome classification for the episode will be ‘Ungradable’.

If the outcome of all images in a screening episode is ‘No disease’, then the outcome classification for the episode will be ‘Disease absent’.

All automated systems also report a numerical value for a decision statistic; values above a certain threshold imply that DR is present.

Reference standards

Final human grade

The study is not designed to establish the performance of human graders but to compare the automated systems with outcomes from clinical practice. Screening performance of each automated system will be assessed using the final human grade as the reference standard, as well as that modified by arbitration.

Final human grade modified by arbitration

An internationally recognized fundus photographic reading centre (Doheny Image Reading Centre; Los Angeles, USA),²⁴ masked to the final human grading, will arbitrate on disagreements between the final human grade and the grades assigned by the automated systems. Arbitration is limited by the available resources to no more than 1700 episodes. Emphasis will be placed on arbitration of all discrepancies with final human grades R3, R2, M1. A random sample of screening episodes where two or more systems disagree with the final human grade of R1 or R0 will also be sent for arbitration. Table 1 summarizes the process of arbitration and how the final reference grade will be modified. The percentage of screening episodes falling in cases 3 and 4 (see Table 1) to be sent for arbitration will be determined by the sensitivity and false positive rate of the automated systems. Diabetes UK sets standards for DR screening of at least 80% sensitivity for sight-threatening retinopathy;²⁵ any automated system that does not meet this operating standard will not be considered in the process of identifying episodes for arbitration.

Table 1

Process of selection of discrepancies between human grade and automated systems for external arbitration

Cases		Criteria		Arbitration	Final human grade modified by arbitration
1	IF	Human grade:	M1, R2 or R3	No arbitration	Final human grade
1	IF	All software:	Disease present	No arbitration	Final human grade
2	IF	Human grade:	M1, R2 or R3	Send all images to reading centre	Grade from reading centre
2	IF	One or more software:	Disease not present	Send all images to reading centre	Grade from reading centre
3	IF	Human grade:	R0	For each software send a proportion of all screening episodes falling in this case to reading centre (random selection)^*	Grade from reading centre
3	IF	Two or more software:	Disease present		Grade from reading centre
4	IF	Human grade:	R1	For each software send a proportion of all screening episodes falling in this case to reading centre (random selection)^*	Grade from reading centre
4	IF	Two or more software:	Disease not present		Grade from reading centre
5	IF	Human grade:	R0	No arbitration	Final human grade
5	IF	Only one software:	Disease present	No arbitration	Final human grade
6	IF	Human grade:	R1	No arbitration	Final human grade
6	IF	Only one software:	Disease not present	No arbitration	Final human grade
7	IF	Human grade:	R0	No arbitration	Final human grade
7	IF	All software:	Disease not present	No arbitration	Final human grade
8	IF	Human grade:	R1	No arbitration	Final human grade
8	IF	All software:	Disease present	No arbitration	Final human grade
9	IF	Human grade:	R0, R1, M1, R2 or R3	No arbitration	Final human grade
9	IF	More than one software:	Technical failure	No arbitration	Final human grade
10	IF	Human grade:	R0, R1, M1, R2 or R3	Proceed as in cases 1-8 considering the grades of the remaining software
10	IF	Only one software:	Technical failure
11	IF	Human grade:	Ungradable	No arbitration	Final human grade

Appropriate weighting will be used in the random selection process of screening episodes to give increasing preference to episodes for which more than one or all software agree with each other but disagree with the final human grade.

Consensus grade

For automated systems achieving at least 80% sensitivity in identifying cases of DR of R1 or above and 100% sensitivity for R3, two approaches to define a “consensus grade” will be used:

A majority classification based on using data from the final human grade and the automated system classifications (disease absent/disease present or technical failure)

A majority classification based on the automated system classification only (disease absent/disease present or technical failure)

The change in screening performance for each system will be compared with the consensus grade. Currently there is no recommended level for the false positive rate (i.e. 1-specificity); we will explore the influence this may have on the consensus grade.

Statistical methods

Primary analyses

Screening performance of the automated systems v final human grades in current screening practice:

The sensitivity, false positive rate, and likelihood ratios of each automated grading system will be determined using the final human grade (with and without reading centre arbitration) as the reference standard for (i) any retinopathy or ungradable episodes (final human grades R1, U, M1, R2, or R3 v R0); (ii) episodes with final human grades U, M1, R2, or R3 v final human grades R1 or R0; and (iii) for each grade of retinopathy separately (R0, R1, M1, R2, R3, and U), using data from all 28,079 screening episodes. As 7,821 of the 20,258 patients were screened twice (in total 28,079 screening episodes), screening performance for each automated system will be estimated using mixed-effects logistic regression models including the final human grade (with and without reader centre adjustment) as the explanatory variable and a random effect for each participant to take account of repeated measurements. Hence, for each system the model will estimate the probability of an automated system suggesting that retinopathy is present, conditional on human grading. The sensitivity and false positive rate will be presented for each final human grade (all outcome classifications reported by each automated system). Diagnostic accuracy of sensitivity, false positive rate and likelihood ratios will be defined by 95% confidence intervals (CI) obtained using bootstrapping. Clinical interpretation of these measures will focus on the lower limit of the 95% CI for the detection of R3, R2, and M1 grades. This will give an indication of the number of screening episodes requiring clinical intervention that could be missed by the automated systems. The upper confidence limit for false positive rates for retinopathy grades R1 and R0 will be important in assessing the additional number of screening episodes requiring further investigation. The level of uncertainty around these estimated lower and upper limits will also be investigated using bootstrapping. Sensitivity analyses will examine the influence of excluding episodes classified as technical failures by an automated system.

Assessing the predictive value of the automated grading systems:

The positive predictive value of the automated grading systems will be evaluated using data from the 21,929 independent screening episodes by fitting multinomial logistic regression models with the final human grade as outcome (with and without reader centre arbitration) and the outcome classification of each automated system as an explanatory variable.

Comparing technical failure rate between manual and automated grading systems:

The rate of technical failure for automated and manual grading will be compared. The proportion (with 95% CI) of screening episodes identified by each automated system as disease-negative offset against technical failures will be estimated as this measure could be used to quantify the potential saving (in terms of both money and time) in the number of screening episodes requiring manual grading.

Secondary analyses

Alternative reference standard:

Sensitivity and false positive rates will be estimated using the consensus grade on all 28,079 screening episodes to quantify the bias in the measures of screening performance compared with that from the primary analysis.

Influence of potential confounders in screening performance measures:

Multiple variable mixed-effects logistic regression models, including patients’ ethnicity (South Asians, black African-Caribbean and white Europeans), age, and duration of diabetes as explanatory variables, will be used to examine the effect of potential confounders on the screening performance of each automated system. Multiple variable logistic regression will be used to explore the influence of these patient factors on the frequency of grade of retinopathy and technical failure. The study was not specifically powered to examine these individual factors, but may suggest hypotheses requiring further investigation.

Health-Economic Analyses

Health-economic analyses will investigate the implications (i) if an automated system were to replace level 1 human graders, (ii) if the automated system were to be used as a filter prior to level 1 graders. This investigation will also (i) determine the optimal operating threshold for the detection of DR for each software using Receiver Operating Characteristics analysis, (ii) quantify the rate of disease progression from the literature and the data, and (iii) examine whether OCT should be incorporated into the primary screening programmes for diabetic retinopathy and maculopathy. The health economic model will take into account the screening performance of automated systems, efficacy of manual screening, likelihood and timing of re-screening, referral rates to ophthalmologists, prevalence of different levels of retinopathy in the study population, and rates of progression of disease. Cost data will come from Personal Social Services Research Unit²⁶ unit costs, hospital cost data, the existing literature and expert opinion. More information is available online (See Appendix).

Results

Table 2 reports the prevalence of different levels of DR obtained from the final human grades using data from the 20,258 independent screening episodes. Of 20,258 diabetes patients, 2,782 (13.7%) were referred to ophthalmologists for further examination. The prevalence of maculopathy (M1), pre-proliferative retinopathy (R2), and proliferative retinopathy (R3) was 7.9%, 3.1% and 1.2%, respectively. An additional 4,749 (23%) patients were diagnosed with background retinopathy, and were invited for re-examination after 12 months. The rate of episodes with images of inadequate quality was 1.5%. Given the observed prevalence for each grade of DR, the statistical precision (95% CI) around expected detection rates (or specificity) between 80% and 95% is shown in Table 2. If the detection rate of an automated system were found to be 85% for R3 grades, then the 95% CI around the estimate of 85% sensitivity would be 81–89%. If the specificity of an automated system were found to be 85% for R0 grades, then the 95% CI around the estimate of 85% specificity would be 84-86%.

Table 2

Prevalence of retinopathy grades and associated 95% confidence intervals for detection rates ranging from 80% to 95%.

	Screening episodes	Associated 95% CI of the detection rate^*, in four possible scenarios for the detection rate of an automated system
Final human grade	N (prevalence)	80%	85%	90%	95%
Retinopathy grades
R0	12727 (62.8%)	79% to 81%	84% to 86%	89% to 91%	94.6% to 95.4%
R1	4749 (3.4%)	79% to 81%	84% to 86%	89% to 91%	94% to 96%
M1	1609 (7.9%)	78% to 82%	83% to 87%	89% to 91%	94% to 96%
R2	637 (3.1%)	77% to 83%	82% to 88%	88% to 92%	93% to 97%
R3	236 (1.2%)	75% to 85%	81% to 89%	86% to 94%	92% to 97%
U	300 (1.5%)	75% to 84%	81% to 89%	86% to 93%	92% to 97%
Combination of grades
R1, U, M1, R2 or R3	7531 (37.2%)	79% to 81%	84% to 86%	89% to 91%	94.5% to 95.5%
U, M1, R2 or R3	2782 (13.7%)	79% to 81%	84% to 86%	89% to 91%	94% to 96%
Total	20258 (100%)

CI = Confidence interval.

Interpretation of Table 2 with an example: If the detection rate of an R3 by the automated system is 85%, then the 95% CI around the estimate of 85% sensitivity will be 80% to 90%.

For R0, the reported 95% CI correspond to specificity.

Discussion

While the performance of automated grading systems is comparable to that of trained graders in ideal circumstances (published sensitivities over 97%, and false positive rates of 37% or less^6–12), this study in a ‘real life’ setting should provide robust evidence on whether these systems might efficaciously and cost effectively replace one or more steps of current DESPs. In Scotland, macular centred images are already automatically processed, but this study will include newer systems that process both macula and disc centred images. The three systems (RetMarker, iGrading, and EyeArt) will use the same dataset of images collected from the same screening programme. The programme conforms to ENSCDR guidelines, and the ongoing performance of manual grading is maintained by regular external auditing and continually grading a test dataset to nationally agreed standards. Prevalences of retinopathy grades in the 20,258 screening episodes to date are similar to published ENSCDR data.^18,27 To further validate the quality of human grading, a subset of images (including all images considered higher grades of retinopathy, ie. R2 and R3) will be sent to an US Reading Centre compliant with all federal and FDA regulations, including FDA 21 CFR Part 11, for masked assessment, primarily graded using the ENSCDR system, but also according to the Early Treatment of Diabetic Retinopathy Scale used in FDA studies. While this validation process will ensure that the programme is as close to a ‘gold standard’ as possible, the study is not designed to compare the automated systems, but to see if software performance is the same or potentially better than that of a human grader. We aim to ensure that findings are generalizable and directly relevant to other DESPs with similar methodologies.

The large number of episodes will allow screening performance and diagnostic accuracy to be determined with a high level of precision for each grade of retinopathy. This is particularly important at more severe grades, because of the potential sight threatening consequences of missing retinopathy. Previous Diabetes UK recommendations suggest that the sensitivity of a screening programme should be at least 80%²⁵, but controversy exists over whether this is adequate for potentially sight threatening retinopathy grades. Recommendations for acceptable false positives rates and other screening performance measures do not exist.²⁵ False positives cause patient anxiety, and have cost implications due to unnecessary referrals to ophthalmologists. Given the large number of episodes in the dataset, this study will be well placed to examine the economic impact and other consequences for different levels of sensitivity and false positive rates.

Sub-group analysis will examine the effect of age, gender, and ethnicity on screening performance. Examining ethnic differences in screening performance is novel. Including a DESP in an ethnically diverse population (East London), with large representations of Black African and mixed ethnic groups²⁸ will allow the study to examine screening performance in non-white ethnic groups, where higher levels of retinal pigmentation may affect system performance.

While all three automated systems are compatible with the same image management system (Digital Health Care system V3.6), they report different outcomes. This paper outlines the methodology used to homogenize output across systems, to allow comparison of performance with human grades. The unifying approach requires little or no change for some systems (eg. Retmarker), but loss of detail for some (eg. EyeArt). Because in ‘real life’, some screening systems may be unable to cope with unfiltered images or images of insufficient quality, the software companies will each receive an identical test set of undedited images with which to develop and test algorithms for image selection and processing.

Human grading can use information from past image sets or patients’ history to facilitate more accurate grading. For example, yellow/white well demarcated lesions in the macular area which look like exudates but have been present unchanged on 15 sets of prior images with normal visual acuity are unlikely to be M1 even if they have that appearance. Image reading centre analysis and most automated systems cannot do this, although one system has developed an algorithm to detect retinal changes over time (ie. formation and disappearance of microaneurysms).

The cost of diabetes to the NHS is estimated at £24 billion (10% of current NHS expenditure) and is set to rise.²⁹ Complications of diabetes account for a substantial proportion of direct health costs, and strategies to prevent unnecessary complications, such as diabetic retinopathy, are key.²⁹ Human graders are expensive to train, incur on-going costs and are difficult to retain, so if low risk patients could be identified automatically, limited human resources could be concentrated on high risk groups with more severe retinopathy. This study will also allow the health economic assessment of the potential inclusion of OCT in screening pathways, as is already being done in some Hospital Eye Services.¹⁴ The research findings will be published and disseminated to inform policy, and the image dataset collected will provide an important resource to compare performance of future systems.

Footnotes

Funding

This project was funded by the National Institute for Health Research HTA programme (project no. 11/21/02); a Fight For Sight Grant (Hirsch grant award); and the Department of Health's NIHR Biomedical Research Centre for Ophthalmology at Moorfields Eye Hospital and UCL Institute of Ophthalmology. The views expressed are those of the authors, not necessarily those of the Department of Health.

Conflict of interest

All investigators are independent of commercial software development and have no commercial interest or intellectual property in automated diabetic retinopathy screening.

References

Liew

Michaelides

Bunce

. A comparison of the causes of blindness certifications in England and Wales in working age adults (16-64 years), 1999-2000 with 2009-2010. BMJ Open 2014; 4(2): e004015–e004015.

http://diabeticeye.screening.nhs.uk/ . Accessed 10th July 2014.

http://diabeticeye.screening.nhs.uk/statistics. Accessed 20 August 2014.

Shaw

Sicree

Zimmet

. Global estimates of the prevalence of diabetes for 2010 and 2030. Diabetes Res Clin Pract 2010 January; 87(1): 4–14.

Fleming

Goatman

Philip

Prescott

Sharp

Olson

. Automated grading for diabetic retinopathy: a large-scale audit using arbitration by clinical experts. Br J Ophthalmol 2010 December; 94(12): 1606–10.

Abramoff

Niemeijer

Suttorp-Schulten

Viergever

Russell

van

. Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes. Diabetes Care 2008 February; 31(2): 193–8.

Abramoff

Reinhardt

Russell

. Automated early detection of diabetic retinopathy. Ophthalmology 2010 June; 117(6): 1147–54.

Fleming

Goatman

Philip

. The role of haemorrhage and exudate detection in automated grading of diabetic retinopathy. Br J Ophthalmol 2010 June; 94(6): 706–11.

Philip

Fleming

Goatman

. The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol 2007 November; 91(11): 1512–7.

10.

Scotland

McNamee

Fleming

. Costs and consequences of automated algorithms versus manual grading for the detection of referable diabetic retinopathy. Br J Ophthalmol 2010 June; 94(6): 712–9.

11.

Scotland

McNamee

Philip

. Cost-effectiveness of implementing automated grading within the national screening programme for diabetic retinopathy in Scotland. Br J Ophthalmol 2007 November; 91(11): 1518–23.

12.

Niemeijer

van

Cree

. Retinopathy online challenge: automatic detection of microaneurysms in digital color fundus photographs. IEEE Trans Med Imaging 2010 January; 29(1): 185–95.

13.

Mackenzie

Schmermer

Charnley

. SDOCT imaging to identify macular pathology in patients diagnosed with diabetic maculopathy by a digital photographic retinal screening programme. PLoS One 2011; 6(5): e14811–e14811.

14.

NHS England. Public health functions to be exercised by NHS England service specification no 22: NHS Diabetic Eye Screening Programme. 2013. Report No.: 11 November 2013.

15.

Diabetic Eye Screening. Feature Based Grading Forms. Guidance on standard feature based grading forms to be used in the NHS Diabetic Eye Screening Programme. 2012. Report No.: Version 1.4, 1 November 2012.

16.

Diabetic Eye Screening Revised Grading Definitions: To provide guidance on revised grading definitions for the NHS. Diabetic Eye Screening Programme. 2012. Report No.: Version 1.3, 1 November 2012.

17.

Essential elements in developing a diabetic retinopathy screening programme. UK National Screening Committee. Workbook 4.3, Pathway 2: full disease grading, page 32. 16 June 2009.

18.

Goatman

Charnley

Webster

Nussey

. Assessment of automated disease detection in diabetic retinopathy screening using two-field photography. PLoS One 2011; 6(12): e27524–e27524.

19.

http://www.shil.co.uk/Products/igradingtm-platform-diabetic-retinopathy-screening-software.html. Accessed 20 August 2014.

20.

http://www.retmarker.com/. Accessed 20 August 2014.

21.

Abramoff

Folk

Han

. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol 2013 March; 131(3): 351–7.

22.

http://www.eyediagnosis.net/#!automated-dr-screening/c1f3f. Accessed 20 August 2014.

23.

http://www.eyenuk.com/eyeart.html. Accessed 20 August 2014.

24.

http://dirc.doheny.org/Default.aspx. Accessed 10 July 2014.

25.

British Diabetic Association. Retinal photography screening for diabetic eye disease. London: BDA; 1997.

26.

http://www.pssru.ac.uk/project-pages/unit-costs/2011/. Accessed 20 August 2014.

27.

Stratton

Aldington

Taylor

Adler

Scanlon

. A simple risk stratification for time to development of sight-threatening diabetic retinopathy. Diabetes Care 2013 March; 36(3): 580–5.

28.

http://www.hackney.gov.uk/eqal-race.htm#.VBnbsBD7SSk. Accessed 17 September 2014.

29.

Hex

Bartlett

Wright

Taylor

Varley

. Estimating the current and future costs of Type 1 and Type 2 diabetes in the UK, including direct health costs and indirect societal and productivity costs. Diabet Med 2012 July; 29(7): 855–62.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB