MSM-ViT: A multi-scale MobileViT for pulmonary nodule classification using CT images

Abstract

BACKGROUND:

Accurate classification of benign and malignant pulmonary nodules using chest computed tomography (CT) images is important for early diagnosis and treatment of lung cancer. In terms of natural image classification, the ViT-based model has greater advantages in extracting global features than the traditional CNN model. However, due to the small image dataset and low image resolution, it is difficult to directly apply the ViT-based model to pulmonary nodule classification.

OBJECTIVE:

To propose and test a new ViT-based MSM-ViT model aiming to achieve good performance in classifying pulmonary nodules.

METHODS:

In this study, CNN structure was used in the task of classifying pulmonary nodules to compensate for the poor generalization of ViT structure and the difficulty in extracting multi-scale features. First, sub-pixel fusion was designed to improve the ability of the model to extract tiny features. Second, multi-scale local features were extracted by combining dilated convolution with ordinary convolution. Finally, MobileViT module was used to extract global features and predict them at the spatial level.

RESULTS:

CT images involving 442 benign nodules and 406 malignant nodules were extracted from LIDC-IDRI data set to verify model performance, which yielded the best accuracy of 94.04% and AUC value of 0.9636 after 10 cross-validations.

CONCLUSION:

The proposed new model can effectively extract multi-scale local and global features. The new model performance is also comparable to the most advanced models that use 3D volume data training, but its occupation of video memory (training resources) is less than 1/10 of the conventional 3D models.

Keywords

Pulmonary nodules classification MobileViT multi-scale nodules computer-aided diagnosis.

Get full access to this article

View all access options for this article.

References

Mohammad

B. Al

, Brennan

P.C.

and Mello-Thoms

, A review of lung cancer screening and the role of computer-aided detection, Clinical Radiology 72(6) (2017), 433–442.

Nasrullah

, Sang

, Alam

M.S.

et al., Automated lung nodule detection and classification using deep learning combined with multiple strategies, Sensors 19(17) (2019), 3722.

Liu

, Zhou

, Liu

et al., CT and CEA-based machine learning model for predicting malignant pulmonary nodules, Cancer Science 113(12) (2022), 4363–4373.

, Lin

, Dehmeshki

et al., Shape-based computer-aided detection of lung nodules in thoracic CT images, IEEE Transactions on Biomedical Engineering 56(7) (2009), 1810–1820.

Hua

K.L.

, Hsu

C.H.

, Hidayati

S.C.

et al., Computer-aided classification of lung nodules on computed tomography images via deep learning technique, OncoTargets and Therapy (2015), 8.

Manickavasagam

, Selvan

CAD system for lung nodule detection using deep learning with CNN, Medical & Biological Engineering & Computing 60(1) (2022), 221–228.

Zhang

, Lin

, Wang

Lung nodule classification in CT images using 3D densenet, In: Journal of Physics: Conference Series. vol. 1827. IOP Publishing (2021), 012155.

Mehta

, Jain

, Mangalagiri

et al., Lung nodule classification using biomarkers, volumetric radiomics, and 3D CNNs, Journal of Digital Imaging 34(3) (2021), 647–666.

Agnes

S.A.

, Anitha

, Solomon

A.A.

Two-stage lung nodule detection framework using enhanced UNet and convolutional LSTM networks in CT images, Computers in Biology and Medicine 149 (2022), 106059.

10.

Jiang

, Shen

, Gao

, Han

Learning efficient, explainable and discriminative representations for pulmonary nodules classification, Pattern Recognition 113 (2021), 107825.

11.

Jiang

, Gao

, Xu

et al., Attentive and ensemble 3D dual path networks for pulmonary nodules classification, Neurocomputing 398 (2020), 422–430.

12.

Liu

, Liu

, Tie

et al., Res-trans networks for lung nodule classification, International Journal of Computer Assisted Radiology and Surgery 17(6) (2022), 1059–1068.

13.

, Wang

, Gao

et al., Uniformer: Unified transformer for efficient spatiotemporal representation learning, arXiv preprint arXiv:220104676. 2022.

14.

Dosovitskiy

, Beyer

, Kolesnikov

et al., An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929, 2020.

15.

Lee

, Kim

, Willette

, Hwang

S.J.

MPViT: Multi-path vision transformer for dense prediction, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), 7287–7296.

16.

Wang

, Chen

, Ding

, et al., Transbts: Multimodal brain tumor segmentation using transformer, In: International Conference on Medical Image Comuting and Computer-Assisted Intervention, Springer (2021), 109–119.

17.

Chen

, Li

, Wang

et al., GasHis-Transformer: A multi-scale visual transformer approach for gastric histopathological image detection, Pattern Recognition 130 (2022), 108827.

18.

, Qi

, Sun

et al., A vision transformer for emphysema classification using CT images, Physics in Medicine & Biology 66(24) (2021), 245016.

19.

Ranftl

, Bochkovskiy

, Koltun

Vision transformers for dense prediction, In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), 12179–12188.

20.

Chen

L.C.

, Zhu

, Papandreou

et al., Encoder-decoder with atrous separable convolution for semantic image segmentation, In: Proceedings of the European conference on computer vision (ECCV) (2018), 801–118.

21.

Matsoukas

, Haslum

J.F.

, Soderberg

, Smith

Is it time to replace cnns with transformers for medical images?

arXiv preprint arXiv::210809038. (2021).

22.

Kim

B.C.

, Yoon

J.S.

, Choi

J.S.

, Suk

H.I.

Multi-scale gradual integration CNN for false positive reduction in pulmonary nodule detection, Neural Networks 115 (2019), 1–10.

23.

Al-Shabi

, Lee

H.K.

, Tan

Gated-dilated networks for lung nodule classification in CT scans, IEEE Access 7 (2019), 178827–178838.

24.

Wang

, Xie

, Li

et al., Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), 568–578.

25.

Mehta

, Rastegari

Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer, arXiv preprint arXiv:211002178. 2021.

26.

Shi

, Caballero

, Huszar

et al., Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016), 1874–1883.

27.

Dey

, Lu

, Hong

Diagnostic classification of lung nodules using 3D neural networks, In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). IEEE (2018), 774–778.

28.

Sandler

, Howard

, Zhu

et al., Mobilenetv2: Inverted residuals and linear bottlenecks, In: Proceedings of the IEEE conference on computer vision and pattern recognition (2018), 4510–4520.

29.

Sourlos

, Wang

, Nagaraj

et al., Possible bias in supervised deep learning algorithms for CT lung nodule detection and classification, Cancers 14(16) (2022), 3867.

30.

Al-Shabi

, Shak

, Tan

3D axial-attention for lung nodule classification, International Journal of Computer Assisted Radiology and Surgery 16(8) (2021), 1319–1324.

31.

Xie

, Xia

, Zhang

et al., Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT, IEEE Transactions on Medical Imaging 38(4) (2018), 991–1004.

32.

Tang

, Ma

, Li

et al., Classification of benign and malignant pulmonary nodules based on the multiresolution 3D DPSECN model and semisupervised clustering, IEEE Access 9 (2021), 43397–43410.