“Human vs Machine” Validation of a Deep Learning Algorithm for Pediatric Middle Ear Infection Diagnosis

Abstract

Objective

We compared the diagnostic performance of human clinicians with that of a neural network algorithm developed using a library of tympanic membrane images derived from children taken to the operating room with the intent of performing myringotomy and possible tube placement for recurrent acute otitis media (AOM) or otitis media with effusion (OME).

Study Design

Retrospective cohort study.

Setting

Tertiary academic medical center from 2018 to 2021.

Methods

A training set of 639 images of tympanic membranes representing normal, OME, and AOM was used to train a neural network as well as a proprietary commercial image classifier from Google. Model diagnostic prediction performance in differentiating normal vs nonpurulent vs purulent effusion was scored based on classification accuracy. A web-based survey was developed to test human clinicians’ diagnostic accuracy on a novel image set, and this was compared head to head against our model.

Results

Our model achieved a mean prediction accuracy of 80.8% (95% CI, 77.0%-84.6%). The Google model achieved a prediction accuracy of 85.4%. In a validation survey of 39 clinicians analyzing a sample of 22 endoscopic ear images, the average diagnostic accuracy was 65.0%. On the same data set, our model achieved an accuracy of 95.5%.

Conclusion

Our model outperformed certain groups of human clinicians in assessing images of tympanic membranes for effusions in children. Reduced diagnostic error rates using machine learning models may have implications in reducing rates of misdiagnosis, potentially leading to fewer missed diagnoses, unnecessary antibiotic prescriptions, and surgical procedures.