Assessment of Trained Image Grader Performance in Screening for Retinopathy Among Youth With Diabetes

Screening for diabetic retinopathy (DR) allows for timely detection and treatment of vision threatening disease ¹ but adherence with DR screening exams remains low. Point-of-care fundus photography with remote interpretation has been employed to improve access to and adherence with screening. ² However, in resource-poor settings there are very few eye care providers to review images and accurately identify referable DR. Previous literature has found that non-physicians can be trained to detect DR in adult fundus photos. ³ Artificial intelligence (AI) algorithms can also be used to detect DR in digital images, but they have not been cleared by the Food and Drug Administration (FDA) for use in pediatric patients.

In our study, two image graders completed an online “Diabetic Retinopathy Grading” course (link here) and one hour of training by retina specialists over videoconferencing. The two graders were a master’s graduate and an undergraduate student, both with no prior experience in ophthalmology. Our objective was to assess the performance of these graders in evaluating digital fundus photos from pediatric patients with diabetes, and to our knowledge this is the first reported study of its kind. We compared their sensitivity and specificity in detecting retinopathy from fundus photos to assessments from retina specialists and to a reference standard.

Retinal images from 90 patients were randomly chosen from a set of 308 patients, enrolled in a previously published prospective study conducted at the Johns Hopkins Children’s center. ⁴ Both graders independently reviewed the same retinal images from 90 patients and assigned each patient a DR grade according to the International Clinical Diabetic Retinopathy classification. ⁵ If the two eyes had different grades, then the eye with the higher grade was considered the DR grade for the patient. The reference standard determination is described in our prior publication. ⁴ Sensitivity, specificity, and agreement were determined against this reference standard.

The average age of our patients was 13.8 years (range = 7-19 years), 52% were non-Hispanic white, 38% were non-Hispanic black, and 52% were female. Most patients had T1D (76%), and their average hemoglobin A1c was 9.1% (range = 5.6%-15%). Of the participants, 87% had no DR, 5% had mild DR, and 8% had moderate DR, per the reference standard. The performance of graders and retina specialists in comparison with reference standard is shown in Table 1.

Table 1.

Reader Comparison to Reference Standard on Presence of Retinopathy.

	Trained grader 1	Trained grader 2	Retina specialist 1	Retina specialist 2
Cases judged as gradable, n	87	90	88	88
Agreement with reference standard, %	70	91	74	84
Sensitivity, %	91	83	92	91
Specificity, %	67	92	71	83
PPV, %	29	63	33	44
NPV, %	98	97	98	99

Each column in this table is a comparison of an individual reader against the reference standard. Readers first determined whether pictures were of sufficient quality to be graded. The sensitivity, specificity, PPV, and NPV are calculated in regard to the reference standard.

Abbreviations: PPV, positive predictive value; NPV, negative predictive value.

Our analysis demonstrates that graders with no prior experience in ophthalmology had high sensitivity and NPV, comparable to that of retina specialists, for detection of DR from fundus photos of pediatric patients after undergoing a standardized training program. The generalizability of these findings should be further evaluated in a larger group of graders. As prevalence of youth onset type 2 diabetes ⁶ and DR increases, policy planners will need to develop local solutions for DR screening in children. In areas with low resources or limited availability of retina specialists, graders can be used for triaging images for further review. In the future, as AI algorithms become FDA approved for children, AI-based screening may be an additional viable option.