Classification of Benign and Malignant Pulmonary Nodules by Multimodal Feature Fusion Network

doi:10.3778/j.issn.1002-8331.2208-0207

Abstract

Abstract: Accurate classification of benign and malignant pulmonary nodules in computed tomography（CT） is crucial for the early diagnosis of lung cancer. However, due to the complexity of the background of pulmonary nodules in CT images and the incompleteness of image feature extraction, it is difficult to realize the accurate classification of benign and malignant pulmonary nodules. To this end, a method of classification of benign and malignant pulmonary nodules by multimodal feature fusion network is proposed. Specifically, with MobileNet V3 as the backbone network and the original CT image of the lung nodule and the extracted nodule image as the input, a dual-path feature extraction network is designed, which can not only effectively extract the global information of the original CT image, but also the discriminative features of the lung nodule area are effectively mined to make up for the problem that when the nodule is small, the network pays too much attention to the surrounding tissue, resulting in misjudgment. For further enhancing feature expression ability of the network, convolutional block attention module（CBAM） and channel shuffle mechanism are employed. Meanwhile, the last four bottlenecks（bnecks） based on inverted residuals in the original MobileNet V3 network are deleted, so that malignant nodules can be accurately diagnosed with less time and space complexity. The experiments on LIDC-IDRI dataset show that this method can significantly reduce the network parameters and FLOPs, and realize the accurate classification of benign and malignant pulmonary nodules. The classification accuracy, sensitivity, specificity, precision, F1 score and AUC are 93.71%, 94.03%, 93.48%, 95.56%, 92.65% and 98.66% respectively.

Key words: classification of benign and malignant pulmonary nodules, feature fusion, convolutional block attention module（CBAM）, channel shuffle, MobileNet V3

摘要： 胸部计算机断层扫描（computed tomography，CT）中肺结节良恶性的精确分类对于肺癌的早期诊断具有重要意义。然而，CT影像中肺结节背景的复杂，以及图像特征提取不全面等问题，为实现肺结节良恶性的精确分类带来困扰。为此，提出了多模式特征融合网络肺结节良恶性分类方法。具体地，以MobileNet V3为骨干网络，以原始肺结节CT图像及提取出的结节图像为输入，设计了一种双路径特征提取网络，不仅能够有效提取原CT图像的全局信息，还能有效挖掘肺结节区域的判别性特征，以弥补结节较小时网络过多关注其周围组织从而产生误判的问题。此外，在特征提取阶段引入convolutional block attention module（CBAM）和通道混洗机制，进一步增强了网络的特征表达能力。同时，对原MobileNet V3网络结构做出修改，删除最后四组基于倒残差结构的bottlenecks（bnecks）模块，使模型能够以较小的时间和空间复杂度精确诊断恶性结节。在LIDC-IDRI数据集上的实验表明，提出的方法能够在显著降低网络参数量和FLOPs的同时实现对肺结节良恶性的精确分类，分类准确率、敏感性、特异性、精确率、F1值和AUC值分别达到了93.71%、94.03%、93.48%、95.56%、92.65%和98.66%。

关键词: 肺结节良恶性分类, 特征融合, 卷积块注意力模块（CBAM）, 通道混洗, MobileNet V3

YIN Zhixian, XIA Kewen, WU Panpan. Classification of Benign and Malignant Pulmonary Nodules by Multimodal Feature Fusion Network[J]. Computer Engineering and Applications, 2023, 59(23): 228-236.

尹智贤, 夏克文, 武盼盼. 多模式特征融合网络肺结节良恶性分类方法[J]. 计算机工程与应用, 2023, 59(23): 228-236.

References

[1] ZHENG R，ZHANG S，ZENG H，et al.Cancer incidence and mortality in China，2016[J].Journal of the National Cancer Center，2022，2（1）：1-9.
[2] SIEGEL R L，MILLER K D，FUCHS H E，et al.Cancer statistics，2022[J].CA：A Cancer Journal for Clinicians，2022，72（1）：7-33.
[3] JACOBS C，RIKXOORT E V，TWELLMANN T，et al.Automatic detection of subsolid pulmonary nodules in thoracic computed tomography images[J].Medical Image Analysis，2014，18（2）：374-384.
[4] FARAG A A，ALI A，ELSHAZLY S，et al.Feature fusion for lung nodule classification[J].International Journal of Computer Assisted Radiology and Surgery，2017，12（10）：1809-1818.
[5] LI X，LI B，TIAN L，et al.Automatic benign and malignant classification of pulmonary nodules in thoracic computed tomography based on random forests algorithm[J].IET Image Processing，2018，12（7）：1253-1264.
[6] Vanbang L E，朱煜，郑兵兵，等.图像灰度密度分布计算模型及肺结节良恶性分类[J].计算机应用研究，2020，37（1）：296-299.
VANBANG L E，ZHU Y，ZHENG B B，et al.Pulmonary nodule image grey density distribution feature extraction algorithm and adenocarcinoma benign/malignant classification[J].Application Research of Computers，2020，37（1）：296-299.
[7] 朱英亮，仇旭阳，徐磊.基于改进ReliefF与k-means算法的良恶性肺结节分类模型[J].小型微型计算机系统，2021，42（3）：566-571.
ZHU Y L，QIU X Y，XU L.Classification model of benign and malignant pulmonary nodules based on unbiased ReliefF and weighted k-means algorithm[J].Journal of Chinese Computer Systems，2021，42（3）：566-571.
[8] ZHANG G，ZHU D，LIU X，et al.Multi-scale pulmonary nodule classification with deep feature fusion via residual network[J].Journal of Ambient Intelligence and Humanized Computing，2018：1-12.
[9] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），USA，2016.Hawaii：IEEE，2016：770-778.
[10] XIE Y，XIA Y，ZHANG J，et al.Transferable multi-model ensemble for benign-malignant lung nodule classification on chest CT[C]//Medical Image Computing and Computer-Assisted Intervention，Canada，2017.Switzerland：Springer，2017：656-664.
[11] XIE Y，XIA Y，ZHANG J，et al.Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT[J].IEEE Transactions on Medical Image，2019，38（4）：991-1004.
[12] SAHU P，YU D，DASARI M，et al.A lightweight multi-section CNN for lung nodule classification and malignancy estimation[J].IEEE Journal of Biomedical and Health Informatics，2019，23（3）：960-968.
[13] AL-SHABI M，LAN B L，CHAN W，et al.Lung nodule classification using deep local-global networks[J].International Journal of Computer Assisted Radiology and Surgery，2019，14：1815-1819.
[14] HUANG G，LIU Z，LAURENS V，et al.Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），USA，2017.Hawaii：IEEE，2017：2261-2269.
[15] ZHANG H，GOODFELLOW I，METAXAS D，et al.Self-attention generative adversarial networks[J].arXiv：1805. 08318，2018.
[16] WANG Y，ZHANG H，CHAE K J，et al.Novel convolutional neural network architecture for improved pulmonary nodule classification on computed tomography[J].Multidimensional Systems and Signal Processing，2020，31：1163-1183.
[17] 汤宁，卫泽良，张瑞，等.基于多尺度多模式图像的肺结节分类对比研究[J].计算机工程与应用，2020，56（3）：165-175.
TANG N，WEI Z L，ZHANG R，et al.Comparative study on classification of pulmonary nodules based on multi-scale and multi-mode image[J].Computer Engineering and Applications，2020，56（3）：165-175.
[18] 孙浩天，袁刚，杨杨，等.基于三维各向异性卷积的肺结节良恶性分类[J].计算机工程与应用，2021，57（10）：133-138.
SUN H T，YUAN G，YANG Y，et al.3D anisotropic convolution based pulmonary nodule classification[J].Computer Engineering and Applications，2021，57（10）：133-138.
[19] SHI F，CHEN B，CAO Q，et al.Semi-supervised deep transfer learning for benign-malignant diagnosis of pulmonary nodules in chest CT images[J].IEEE Transactions on Medical Imaging，2022，41（4）：771-781.
[20] HOWARD A，SANDLER M，CHEN B，et al.Searching for MobileNetV3[C]//Proceedings of the IEEE International Conference on Computer Vision（ICCV），Korea （South），2019.Hawaii：IEEE，2019：1314-1324.
[21] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//European Conference on Computer Vision，Germany，2018.Switzerland：Springer，2018：3-19.
[22] ZHANG X，ZHOU X，LIN M，et al.ShuffleNet：an extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，USA，2018.Hawaii：IEEE，2018：6848-6856.
[23] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer-Assisted Intervention，Germany，2015.Switzerland：Springer，2015：234-241.
[24] ARMATO S G，MCLENNAN G，BIDAUT L，et al.The lung image database consortium（LIDC） and image database resource initiative（IDRI）：a completed reference database of pulmonary nodules on CT scans[J].Medical Physics，2011，38（2）：915-931.
[25] 吴保荣，强彦，王三虎，等.融合多维度卷积神经网络的肺结节分类方法[J].计算机工程与应用，2019，55（24）：171-177.
WU B R，QIANG Y，WANG S H，et al.Fusing multi-dimensional convolution neural network for lung nodules classification[J].Computer Engineering and Applications，2019，55（24）：171-177.
[26] SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：visual explanations from deep networks via gradient-based localization[J].International Journal of Computer Vision，2020，128（2）：336-359.