Content-based medical image retrieval (CBMIR) has become an important part of computer-aided diagnostics (CAD) systems. The complex medical semantic information inherent in medical images is the most difficult part to improve the accuracy of image retrieval. Highly expressive feature vectors play a crucial role in the search process. In this paper, we propose an effective deep convolutional neural network (CNN) model to extract concise feature vectors for multiple semantic X-ray medical image retrieval.
METHODS:
We build a feature pyramid based CNN model with ResNet50V2 backbone to extract multi-level semantic information. And we use the well-known public multiple semantic annotated X-ray medical image data set IRMA to train and test the proposed model.
RESULTS:
Our method achieves an IRMA error of 32.2, which is the best score compared to the existing literature on this dataset.
CONCLUSIONS:
The proposed CNN model can effectively extract multi-level semantic information from X-ray medical images. The concise feature vectors can improve the retrieval accuracy of multi-semantic and unevenly distributed X-ray medical images.
LiZ., ZhangX., MüllerH.andZhangS., Large-scale retrieval for medical image analytics: A comprehensive review, Medical Image Analysis43 (2018), 66–84. https://doi.org/10.1016/j.media.2017.09.007.
2.
MüllerH., MichouxN., BandonD.andGeissbuhlerA., A review of content-based image retrieval systems in medical applications— clinical benefits and future directions, International Journal of Medical Informatics73(1) (2004), 1–23. https://doi.org/10.1016/j.ijmedinf.2003.11.024.
3.
AielloM., CavaliereC., D’AlboreA.andSalvatoreM., The Challenges of Diagnostic Imaging in the Era of Big Data, Journal of Clinical Medicine8(3) (2019), doi: 10.3390/jcm8030316.
4.
KahnC.E., CarrinoJ.A., FlynnM.J., PeckD.J.andHoriiS.C., DICOM and Radiology: Past, Present and Future, Journal of the American College of Radiology4(9) (2007), 652–657. doi: https://doi.org/10.1016/j.jacr.2007.06.004.
5.
LehmannT.M., et al., Content-based image retrieval in medical applications, (in eng), Methods Inf Med43(4) (2004), 354–361. doi: 10.1055/s-0038-1633877.
6.
AkgülC.B., RubinD.L., NapelS., BeaulieuC.F., GreenspanH.andAcarB., Content-Based Image Retrieval in Radiology: Current Status and Future Directions, Journal of Digital Imaging24(2) (2011), 208–222. doi:10.1007/s10278-010-9290-9.
7.
RothH.R.et al., Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation, IEEE Transactions on Medical Imaging35(5) (2016), 1170–1181. doi: 10.1109/tmi.2015.2482920.
8.
KhatamiA., BabaieM., KhosraviA., TizhooshH.R.andNahavandiS., Parallel deep solutions for image retrieval from imbalanced medical imaging archives, Applied Soft Computing63 (2018), 197–205. doi: https://doi.org/10.1016/j.asoc.2017.11.024.
9.
QayyumA., AnwarS.M., AwaisM.andMajidM., Medical image retrieval using deep convolutional neural network, Neurocomputing266 (2017), 8–20. doi: https://doi.org/10.1016/j.neucom.2017.05.025.
10.
SrinivasM., NaiduR.R., SastryC.S.andMohanC.K., Content based medical image retrieval using dictionary learning, Neurocomputing168 (2015), 880–895. doi: https://doi.org/10.1016/j.neucom.2015.05.036.
11.
PanL., QiangY., YuanJ.andWuL., Rapid Retrieval of Lung Nodule CT Images Based on Hashing and Pruning Methods, BioMed Research International2016(2016), 3162649. doi: 10.1155/2016/3162649.
12.
ShinH., RobertsK., LuL., Demner-FushmanD., YaoJ.andSummersR.M., Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27–30 June 2016 2016, pp. 2497–2506. doi: 10.1109/cvpr.2016.274.
13.
GreenspanH., GinnekenB.V.andSummersR.M., Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique, IEEE Transactions on Medical Imaging35(5) (2016), 1153–1159. doi: 10.1109/tmi.2016.2553401.
14.
SetioA.A.A.et al., Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks, IEEE Transactions on Medical Imaging35(5) (2016), 1160–1169. doi: 10.1109/tmi.2016.2536809.
15.
ShinH., OrtonM.R., CollinsD.J., DoranS.J.andLeachM.O., Stacked Autoencoders for Unsupervised Feature Learning and Multiple Organ Detection in a Pilot Study Using 4D Patient Data, IEEE Transactions on Pattern Analysis and Machine Intelligence35(8), (2013), 1930–1943. doi: 10.1109/tpami.2012.277.
16.
KhatamiA., BabaieM., TizhooshH.R., KhosraviA., NguyenT.andNahavandiS., A sequential search-space shrinking using CNN transfer learning and a Radon projection pool for medical image retrieval, Expert Systems with Applications100 (2018), 224–233. doi: https://doi.org/10.1016/j.eswa.2018.01.056.
17.
PelkaO., NensaF.andFriedrichC.M., Annotation of enhanced radiographs for medical image retrieval with deep convolutional neural networks, PLOS ONE13(11, p. e0206229, 2018. 10.1371/journal.pone.0206229.
18.
AhnE., KumarA., FulhamM., FengD.andKimJ., Convolutional sparse kernel network for unsupervised medical image analysis, Medical Image Analysis56 (2019), 140–151. doi: https://doi.org/10.1016/j.media.2019.06.005.
19.
KhatamiA., BabaieM., KhosraviA., TizhooshH.R., SalakenS.M.andNahavandiS., A deep-structural medical image classification for a Radon-based image retrieval, in 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE), 30 April-3 May 2017 2017, pp. 1–4. doi: 10.1109/ccece.2017.7946756.
20.
LiuX., TizhooshH.R.andKofmanJ., Generating binary tags for fast medical image retrieval based on convolutional nets and Radon Transform, in 2016 International Joint Conference on Neural Networks (IJCNN), 24–29 July 2016 2016, pp. 2872–2878. doi: 10.1109/ijcnn.2016.7727562.
21.
RussakovskyO.et al., ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision115(3) (2015), 211–252. doi: 10.1007/s11263-015-0816-y.
22.
LinT.-Y.et al., Microsoft COCO: Common Objects in Context, in Computer Vision – ECCV 2014, ChamD., FleetT., Pajdla and SchieleB., TuytelaarsT., Eds., 2014// 2014: Springer International Publishing, pp. 740–755.
23.
EveringhamM., Van GoolL., WilliamsC.K.I., WinnJ.andZissermanA., The Pascal Visual Object Classes (VOC) Challenge, International Journal of Computer Vision88(2) (2010), 303–338. doi:10.1007/s11263-009-0275-4.
24.
ShenH., TaoD.andMaD.Multiview Locally Linear Embedding for Effective Medical Image Retrieval, PLoS ONE8(12) (2013), e82409. doi: 10.1371/journal.pone.0082409.
25.
ShenH., TaoD.andMaD.Dual-Force ISOMAP: A New Relevance Feedback Method for Medical Image Retrieval, PLoS ONE8(12) (2013), e84096. doi: 10.1371/journal.pone.0084096.
26.
HeK., ZhangX., RenS.andSunJ., Deep Residual Learning for Image Recognition, in2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27–30 June 2016 2016, pp. 770–778. doi: 10.1109/cvpr.2016.90.
27.
LinT., DollárP., GirshickR., HeK., HariharanB., BelongieS., Feature Pyramid Networks for Object Detection, in2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21–26 July 2017 2017, pp. 936–944. doi: 10.1109/cvpr.2017.106.
28.
HwangK.H., LeeH.andChoiD., Medical Image Retrieval: Past and Present, Hir18(1) (2012), 3–9. doi: 10.4258/hir.2012.18.1.3.
29.
KumarA., KimJ., CaiW., FulhamM.andFengD., Content-Based Medical Image Retrieval: A Survey of Applications to Multidimensional and Multimodality Data, Journal of Digital Imaging26(6) (2013), 1025–1039. doi: 10.1007/s10278-013-9619-2.
30.
ÇamlicaZ., TizhooshH.R.andKhalvatiF., Medical Image Classification via SVM Using LBP Features from Saliency-Based Folded Data, in 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), , 9–11 Dec. 2015 2015, pp. 128–132. doi: 10.1109/ICMLA.2015.131.
31.
MüllerH., de HerreraA.G.S., Kalpathy-CramerJ., Demner-FushmanD., AntaniS.K.andEggelI., Overview of the ImageCLEF 2012 medical image retrieval and classiFIcation tasks, 2012, pp. 1–16.
32.
JiangM., ZhangS., LiH.andMetaxasD.N., Computer-Aided Diagnosis of Mammographic Masses Using Scalable Image Retrieval, IEEE Transactions on Biomedical Engineering62(2) (2015), 783–792. doi: 10.1109/tbme.2014.2365494.
33.
XuX., LeeD., AntaniS.andLongL.R., A Spine X-Ray Image Retrieval System Using Partial Shape Matching, IEEE Transactions on Information Technology in Biomedicine12(1) (2008), 100–108. doi: 10.1109/titb.2007.904149.
34.
CaicedoJ.C., CruzA.andGonzalezF.A., Histopathology Image Classification Using Bag of Features and Kernel Functions, in Artificial Intelligence in Medicine, Berlin, HeidelbergC.Combi, Y.Shahar and A.Abu-Hanna, Eds., 2009// 2009: Springer Berlin Heidelberg, pp. 126–135.
35.
VenkatachalamK., SiulyS., BacaninN., HubálovskýS.andTrojovskýP., An Efficient Gabor Walsh-Hadamard Transform Based Approach for Retrieving Brain Tumor Images From MRI, IEEE Access9 (2021), 119078–119089. doi: 10.1109/access.2021.3107371.
36.
HofmanningerJ.andLangsG., Mapping visual features to semantic profiles for retrieval in medical imaging, in2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 June 2015 2015, pp. 457–465. doi: 10.1109/cvpr.2015.7298643.
37.
ZhangS., ZhiL.andZhouT., Medical Image Retrieval Using Empirical Mode Decomposition with Deep Convolutional Neural Network, BioMed Research International2020 (2020), 6687733. doi: 10.1155/2020/6687733.
38.
ÖztürkŞ, ÇelikE.andÇukurT., Content-based medical image retrieval with opponent class adaptive margin loss, Information Sciences637 (2023), 118938. doi: https://doi.org/10.1016/j.ins.2023.118938.
39.
Mohammad AlizadehS., Sadegh HelfroushM.andMüllerH., A novel Siamese deep hashing model for histopathology image retrieval, Expert Systems with Applications225 (2023), 120169. doi: https://doi.org/10.1016/j.eswa.2023.120169.
40.
ChenY., TangY., HuangJ.andXiongS.Multi-scale Triplet Hashing for Medical Image Retrieval, Computers in Biology and Medicine155 (2023), 106633. doi: https://doi.org/10.1016/j.compbiomed.2023.106633.
41.
LehmannT.M.et al., IRMA–content-based image retrieval in medical applications, in Stud Health Technol Inform vol. 107, no. Pt 2. Netherlands, 2004, pp. 842–6.
42.
HeK., ZhangX., RenS.andSunJ., Identity Mappings in Deep Residual Networks, in Computer Vision – ECCV 2016, Cham,B. Leibe, J.Matas, N.Sebe and M.Welling, Eds., 2016// 2016: Springer International Publishing, pp. 630–645.
43.
LinT., GoyalP., GirshickR., HeK.andDollárP., Focal Loss for Dense Object Detection, in2017 IEEE International Conference on Computer Vision (ICCV) , 22–29 Oct. 2017 2017, pp. 2999–3007. doi: 10.1109/iccv.2017.324.
44.
MüllerH., DeselaersT., DesernoT.M., Kalpathy—CramerJ., KimE.andHershW., Overview of the ImageCLEFmed 2007 Medical Retrieval and Medical Annotation Tasks, in: Advances in Multilingual and Multimodal Information Retrieval: 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers: Springer-Verlag, 2008, pp. 472–491. .
45.
DeselaersT., KeysersD.andNeyH., Features for image retrieval: an experimental comparison, Information Retrieval11(2) (2008), 77–107. doi: 10.1007/s10791-007-9039-3.
TanM.andLeQ.V., EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, CoRR, vol. abs/1905.11946, / 2019 [Online]. Available: http://arxiv.org/abs/1905.11946.
48.
TommasiT., CaputoB., WelterP., GüldM.O.andDesernoT.M., Overview of the CLEF 2009 Medical Image Annotation Track, in Multilingual Information Access Evaluation II. Multimedia Experiments, Berlin, Heidelberg, C.Peterset al., Eds., 2010// 2010: Springer Berlin Heidelberg, pp. 85–93.
