Despite the exponential growth in academic publications and industrial investments in artificial intelligence (AI) in medical imaging, clinical translation remains disproportionately low. Notably, the absence of internationally recognized guidelines for evaluating AI model performance and ethical considerations creates a critical gap in current practices. In this regard, we aim to offer a practical concise perspective exploring performance challenges to implementation while focusing on their mitigation. The dialog continues in subsequent work (part 2) which focuses on ethical issues. In this part, we explore the challenges inherent to the performance evaluation of AI in radiology, focusing on data heterogeneity, the choice of performance metrics and their interpretability, and data access. By shedding light on these issues and discussing potential opportunities, this work contributes to the ongoing dialog surrounding the practical integration of AI in clinical settings. It highlights the imperative need for established guidelines to ensure the safe and efficient deployment of AI technologies in medical imaging, ultimately bridging the gap between theoretical potential and practical implementation.
ChenMMTerzicABeckerAS, et al.Artificial intelligence in oncologic imaging. Eur J Radiol Open2022; 9: 100441.
4.
HosnyAParmarCQuackenbushJ, et al.Artificial intelligence in radiology. Nat Rev Cancer2018; 18(8): 500–510.
5.
KimDWJangHYKimKW, et al.Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol2019; 20(3): 405–410.
6.
FarahLMurrisJMBorgetI, et al.Assessment of performance, interpretability, and explainability in artificial intelligence–based health technologies: what healthcare stakeholders need to know. Mayo Clin Proc Digit Health2023; 1(2): 120–138.
7.
WuEWuKDaneshjouR, et al.How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med2021; 27(4): 582–584.
WestEMutasaSZhuZ, et al.Global trend in artificial intelligence-based publications in radiology from 2000 to 2018. AJR Am J Roentgenol2019; 213(6): 1204–1206.
10.
FaghaniSKhosraviBZhangK, et al.Mitigating bias in radiology machine learning: 3. Performance metrics. Radiol Artif Intell2022; 4(5): e220061.
11.
RouzrokhPKhosraviBFaghaniS, et al.Mitigating bias in radiology machine learning: 1. Data handling. Radiol Artif Intell2022; 4(5): e210290.
12.
ChangQYanZZhouM, et al.Mining multi-center heterogeneous medical data with distributed synthetic learning. Nat Commun2023; 14(1): 5510.
13.
ChenPHCMermelCHLiuY. Evaluation of artificial intelligence on a reference standard based on subjective interpretation. Lancet Digit Health2021; 3(11): e693–e695.
14.
Foltyn-DumitruMSchellMSahmF, et al.Advancing noninvasive glioma classification with diffusion radiomics: exploring the impact of signal intensity normalization. Neurooncol Adv2024; 6(1): vdae043.
15.
RizzoSBottaFRaimondiS, et al.Radiomics: the facts and the challenges of image analysis. Eur Radiol Exp2018; 2(1): 36.
16.
HanXJovicichJSalatD, et al.Reliability of MRI-derived measurements of human cerebral cortical thickness: the effects of field strength, scanner upgrade and manufacturer. Neuroimage2006; 32(1): 180–194.
17.
CarréAKlausnerGEdjlaliM, et al.Standardization of brain MR images across machines and protocols: bridging the gap for MRI-based radiomics. Sci Rep2020; 10(1): 12340.
18.
KumarGBasriSImamAA, et al.Data harmonization for heterogeneous datasets: a systematic literature review. Applied Sciences2021; 11(17): 8275.
19.
MarziCGiannelliMBarucciA, et al.Efficacy of MRI data harmonization in the age of machine learning: a multicenter study across 36 datasets. Sci Data2024; 11(1): 115.
20.
ChlapPMinHVandenbergN, et al.A review of medical image data augmentation techniques for deep learning applications. J Med Imag Rad Onc2021; 65(5): 545–563.
21.
HussainZGimenezFYiD, et al.Differential data augmentation techniques for medical imaging classification tasks. AMIA Annu Symp Proc2017; 2017: 979–984.
22.
SanfordTHZhangLHarmonSA, et al.Data augmentation and transfer learning to improve generalizability of an automated prostate segmentation model. AJR Am J Roentgenol2020; 215(6): 1403–1410.
23.
HarveyHGlockerB. A standardised approach for preparing imaging data for machine learning tasks in radiology. In: RanschaertERMorozovSAlgraPR (eds). Artificial Intelligence in Medical Imaging. Springer International Publishing, 2019, pp. 61–72.
24.
LuyckxEBosmansJMLBroeckxBJG, et al.Radiologists as Co-Authors in case reports containing radiological images: does their presence influence quality?J Am Coll Radiol2019; 16(4): 526–527.
25.
KouwWMLoogM. A review of domain adaptation without target labels. IEEE Trans Pattern Anal Mach Intell2021; 43(3): 766–785.
26.
PatelVMGopalanRLiR, et al.Visual domain adaptation: a survey of recent advances. IEEE Signal Process Mag2015; 32(3): 53–69.
27.
GuanHLiuM. Domain adaptation for medical image analysis: a survey. IEEE Trans Biomed Eng2022; 69(3): 1173–1185.
28.
OuyangCKamnitsasKBiffiC, et al.Data efficient unsupervised domain adaptation for cross-modality image segmentation. In: ShenDLiuTPetersTM, et al., editors. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Springer International Publishing; 2019. 669–677. (Lecture Notes in Computer Science; vol. 11765).
29.
DasSNayakGKSabaL, et al.An artificial intelligence framework and its bias for brain tumor segmentation: a narrative review. Comput Biol Med2022; 143: 105273.
30.
CardenasCEYangJAndersonBM, et al.Advances in auto-segmentation. Semin Radiat Oncol2019; 29(3): 185–197.
31.
Puyol-AntonERuijsinkBPiechnikSK, et al.Fairness in cardiac MR image analysis: an investigation of bias due to data imbalance in deep learning based segmentation. 2021. [cited 2023 Nov 10]; Available from: https://arxiv.org/abs/2106.12387
32.
GoldmanJHagiwaraAYaoJ, et al.Paradoxical association between relative cerebral blood volume dynamics following chemoradiation and increased progression-free survival in newly diagnosed IDH wild-type MGMT promoter methylated glioblastoma with measurable disease. Front Oncol2022; 12: 849993.
33.
WilleminkMJKoszekWAHardellC, et al.Preparing medical imaging data for machine learning. Radiology2020; 295(1): 4–15.
34.
MorleyJMurphyLMishraA, et al.Governing data and artificial intelligence for health care: developing an international understanding. JMIR Form Res2022; 6(1): e31623.
35.
HandelmanGSKokHKChandraRV, et al.Peering into the Black box of artificial intelligence: evaluation metrics of machine learning methods. AJR Am J Roentgenol2019; 212(1): 38–43.
36.
Maier-HeinLReinkeAGodauP, et al.Metrics reloaded: recommendations for image analysis validation. 2022. https://arxiv.org/abs/2206.01653
37.
ReinkeATizabiMDSudreCH, et al.Common limitations of image processing metrics: a picture story. 2021. https://arxiv.org/abs/2104.05642
38.
OzenneBSubtilFMaucort-BoulchD. The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol2015; 68(8): 855–859.
39.
Sokolova M, Japkowicz N, Szpakowicz S. Beyond accuracy, F-Score and ROC: A family of discriminant measures for performance evaluation. In: Sattar A, Kang B Ho, editors. AI 2006: Advances in Artificial Intelligence [Internet]. Berlin (DE): Springer; 2006. p. 1015–21. (Lecture Notes in Computer Science; vol. 4304). http://link.springer.com/10.1007/11941439_114
40.
OstromQTPriceMNeffC, et al.CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2016—2020. Neuro Oncol2023; 25(Supplement_4): iv1–99.
41.
TahaAAHanburyA. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imag2015; 15(1): 29.
42.
KellyBSJudgeCBollardSM, et al.Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). Eur Radiol2022; 32(11): 7998–8007.
43.
TahaAAHanburyA. An efficient algorithm for calculating the exact hausdorff distance. IEEE Trans Pattern Anal Mach Intell2015; 37(11): 2153–2163.
44.
DoyenSDadarioNB. 12 plagues of AI in healthcare: a practical guide to current issues with using machine learning in a medical context. Front Digit Health2022; 4: 765406.
45.
DengJDongWSocherR, et al.ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition [Internet]. IEEE, 2009, pp. 248–255. https://ieeexplore.ieee.org/document/5206848/
46.
AnnasGJ. HIPAA regulations — a new era of medical-record privacy?N Engl J Med2003; 348(15): 1486–1490.
47.
VoigtPVon Dem BusscheA. The EU General Data Protection Regulation (GDPR) [Internet]. Springer International Publishing, 2017.
48.
BadgeleyMAZechJROakden-RaynerL, et al.Deep learning predicts hip fracture using confounding patient and healthcare variables. Npj Digit Med2019; 2(1): 31.
49.
ShellerMJEdwardsBReinaGA, et al.Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci Rep2020; 10(1): 12598.
50.
PatiSBaidUEdwardsB, et al.Federated learning enables big data for rare cancer boundary detection. Nat Commun2022; 13(1): 7346.
51.
PatiSBaidUEdwardsB, et al.The federated tumor segmentation (FeTS) tool: an open-source solution to further solid tumor research. Phys Med Biol2022; 67(20): 204002.
52.
TeoZLTingDSW. Non-fungible tokens for the management of health data. Nat Med2023; 29(2): 287–288.
