Automated comet assay analysis using YOLOv5-based deep learning

Abstract

Background

The comet assay is a sensitive and widely used technique for assessing DNA damage at the single-cell level. Despite its advantages, traditional manual scoring methods remain time-consuming, subjective and limited in scalability, posing challenges for high-throughput and standardized analysis.

Objective

This study aims to develop and evaluate a deep learning-based system for automated comet assay image classification, addressing limitations of manual and semi-automated approaches while enhancing accuracy, reproducibility and processing efficiency.

Method

A YOLOv5-based object detection model was trained on a dataset of 875 annotated comet assay images, curated through a three-step expert-reviewed process. Various hyperparameters and data augmentation techniques were optimized to improve performance. The dataset was split into training, validation and test sets, and model performance was evaluated using mAP, precision, recall and confusion matrix analysis.

Results

The model achieved strong performance, with mAP@0.5 reaching 0.98 and recall exceeding 0.8. Detailed analyses revealed robust learning behavior and generalization capacity. Visual outputs, including precision-recall curves and class-wise confusion matrices, confirmed high classification accuracy, although overlapping comet structures and class imbalance posed challenges. The model demonstrated improved scalability and processing speed compared to traditional tools, supporting its integration into web-based applications.

Conclusion

The proposed YOLOv5-based system offers a scalable and accurate solution for automating comet assay analysis. It significantly enhances throughput and reduces human error, supporting its application in genotoxicity testing, biomonitoring and molecular epidemiology. Future work will focus on handling overlapping structures, benchmarking against existing tools and optimizing deployment in real-world laboratory settings.