Overview of Research on Attention Mechanism in Medical Image Processing

doi:10.3778/j.issn.1002-8331.2108-0266

Abstract

Abstract: The attention mechanism is expected to be a safe support for applying deep learning to clinical practice by visualizing the judgments of deep learning models. By incorporating the attention mechanism, not only can the judgment basis of deep learning models be verified, but also deep learning models can be made to focus more on important features to improve deep learning model performance. In the future, this will help to improve AI interpretability, assist physicians in diagnosis, and discover new diagnostic methods using attention mechanisms. It introduces and analyzes common datasets and evaluation metrics for medical image processing, states the types of attention mechanisms in medical image processing, presents examples from different types of attention mechanisms that can be effectively used in medical image analysis and diagnosis, and then discusses future prospects and development directions based on the latest trends in their application to medical image processing.

Key words: attention mechanism, medical image processing, computer-aided diagnosis

摘要： 注意力机制通过对深度学习模型判断的可视化，有望成为将深度学习应用于临床实践的安全支撑。通过结合注意力机制，不仅可以验证深度学习模型的判断依据，而且可以让深度学习模型更多地关注重要特征，以提升深度学习模型性能。在未来，这将有助于提高人工智能可解释性、辅助医生诊断以及运用注意力机制发现新诊断方法。介绍并分析医学图像处理常用数据集及评价指标，陈述了医学图像处理中的注意力机制种类，从不同种类介绍了注意力机制可以有效地用于医学图像分析和诊断方面的例子，根据其应用于医学图像处理的最新趋势讨论未来前景和发展方向。

关键词: 注意力机制, 医学图像处理, 计算机辅助诊断

CHEN Chaoyi, XU Bo, WU Ying, WU Kaiwen. Overview of Research on Attention Mechanism in Medical Image Processing[J]. Computer Engineering and Applications, 2022, 58(5): 23-33.

陈朝一, 许波, 吴英, 吴凯文. 医学图像处理中的注意力机制研究综述[J]. 计算机工程与应用, 2022, 58(5): 23-33.

References

[1] GARGEYA R，LENG T.Automated identification of diabetic retinopathy using deep learning[J].Ophthalmology，2017，124（7）：962-969.
[2] CHUKWUEKE U N，WEN P Y.Use of the response assessment in neuro-oncology（RANO） criteria in clinical trials and clinical practice[J].CNS Oncology，2019，8（1）：CNS28.
[3] WULCZYN E，STEINER D F，XU Z，et al.Deep learning-based survival prediction for multiple cancer types using histopathology images[J].PLoS One，2020，15（6）：e0233678.
[4] HAN Y，YOO J，KIM H H，et al.Deep learning with domain adaptation for accelerated projection-reconstruction MR[J].Magnetic Resonance in Medicine，2018，80（3）：1189-1205.
[5] NIE D，TRULLO R，LIAN J，et al.Medical image synthesis with deep convolutional adversarial networks[J].IEEE Transactions on Biomedical Engineering，2018，65（12）：2720-2730.
[6] HAJI-VALIZADEH H，SHEN D，AVERY R J，et al.Rapid reconstruction of four-dimensional MR angiography of the thoracic aorta using a convolutional neural network[J].Radiology：Cardiothoracic Imaging，2020，2（3）：e190205.
[7] KANG E，CHANG W，YOO J，et al.Deep convolutional framelet denosing for low-dose CT via wavelet residual network[J].IEEE Transactions on Medical Imaging，2018，37（6）：1358-1369.
[8] GINSBURG G S，PHILLIPS K A.Precision medicine：from science to value[J].Health Affairs，2018，37（5）：694-701.
[9] 张可可，许永生，高雅，等.基于深度学习的影像组学在结直肠癌中的研究进展[J].磁共振成像，2021，12（3）：112-115.
ZHANG K K，XU Y S，GAO Y，et al.Advances in application of radiomics in colorectal cancer[J].Chinese Journal of Magnetic Resonance Imaging，2021，12（3）：112-115.
[10] 徐可文，许波，吴英，等.机器学习在超声图像中的应用综述[J].计算机工程与应用，2021，57（4）：11-17.
XU K W，XU B，WU Y，et al.Overview of application of machine learning in ultrasound images[J].Computer Engineering and Applications，2021，57（4）：11-17.
[11] LI Z，WANG Y，YU J，et al.Deep learning based radiomics（DLR） and its usage in noninvasive IDH1 prediction for low grade glioma[J].Scientific Reports，2017，7（1）：1-11.
[12] 孟凡奎，银温社，贺建峰.基于深度学习的眼底图像出血点检测方法[J].山东大学学报（理学版），2020，55（9）：62-71.
MENG F K，YIN W S，HE J F.Detection method of hemorrhages of fundus image based on deep learning[J].Journal of Shandong University（Natural Science），2020，55（9）：62-71.
[13] ZHOU Z，SIDDIQUEE M M R，TAJBAKHSH N，et al.Unet++：a nested U-net architecture for medical image segmentation[M]//Deep learning in medical image analysis and multimodal learning for clinical decision support.Cham：Springer，2018：3-11.
[14] 贡荣麟，施俊，王骏.面向乳腺超声图像分割的混合监督双通道反馈U-Net[J].中国图象图形学报，2020，25（10）：2206-2217.
GONG R L，SHI J，WANG J.Hybrid supervised dual-channel feedback U-Net for segmentation of breast ultrasound images[J].Journal of Image and Graphics，2020，25（10）：2206-2217.
[15] DE FAUW J，LEDSAM J R，ROMERA-PAREDES B，et al.Clinically applicable deep learning for diagnosis and referral in retinal disease[J].Nature Medicine，2018，24（9）：1342-1350.
[16] POPLIN R，VARADARAJAN A V，BLUMER K，et al.Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning[J].Nature Biomedical Engineering，2018，2（3）：158-164.
[17] RUDIE J D，RAUSCHECKER A M，XIE L，et al.Subspecialty-level deep gray matter differential diagnoses with deep learning and Bayesian networks on clinical brain MRI：a pilot study[J].Radiology：Artificial Intelligence，2020，2（5）：e190146.
[18] DORAN D，SCHULZ S，BESOLD T R.What does explainable AI really mean? A new conceptualization of perspectives[J].arXiv：1710.00794，2017.
[19] 化盈盈，张岱墀，葛仕明.深度学习模型可解释性的研究进展[J].信息安全学报，2020，5（3）：1-12.
HUA Y Y，ZHANG D C，GE S M.Research progress in the interpretability of deep learning models[J].Journal of Cyber Security，2020，5（3）：1-12.
[20] LI W，JAFARI O H，ROTHER C.Deep object co-segmentation[C]//Asian Conference on Computer Vision.Cham：Springer，2018：638-653.
[21] SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：visual explanations from deep networks via gradient-based localization[J].International Journal of Computer Vision，2020，128（2）：336-359.
[22] ADEBAYO J，GILMER J，MUELLY M，et al.Sanity checks for saliency maps[J].arXiv：1810.03292，2018.
[23] KOSIOREK A.神经网络中的注意力机制[J].机器人产业，2017（6）：12-17.
KOSIOREK A.Attention mechanisms in neural networks[J].Robot Industry，2017（6）：12-17.
[24] CHIOCCA E A，JOHN S Y，LUKAS R V，et al.Regulatable interleukin-12 gene therapy in patients with recurrent high-grade glioma：results of a phase 1 trial[J].Science Translational Medicine，2019，11（505）：5680.
[25] RAJPURKAR P，IRVIN J，BAGUL A，et al.Mura：large dataset for abnormality detection in musculoskeletal radiographs[J].arXiv：1712.06957，2017.
[26] YAN K，WANG X，LU L，et al.DeepLesion：automated mining of large-scale lesion annotations and universal lesion detection with deep learning[J].Journal of Medical Imaging，2018，5（3）：036501.
[27] BAKR S，GEVAERT O，ECHEGARAY S，et al.A radiogenomic dataset of non-small cell lung cancer[J].Scientific Data，2018，5（1）：1-9.
[28] WANG X，PENG Y，LU L，et al.Chestx-ray8：hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2097-2106.
[29] LAMONTAGNE P J，BENZINGER T L S，MORRIS J C，et al.OASIS-3：longitudinal neuroimaging，clinical，and cognitive dataset for normal aging and Alzheimer disease[J].MedRxiv，2019.
[30] 刘星，吴影，李洋，等.人工智能医疗器械应用中的伦理问题分析[J].中国临床药理学与治疗学，2021，26（6）：695-699.
LIU X，WU Y，LI Y，et al.Analysis of ethical issue in the application of artificial intelligence medical devices[J].Chinese Journal of Clinical Pharmacology and Therapeutics，2021，26（6）：695-699.
[31] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[32] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[33] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[34] GIRSHICK R，DONAHUE J，DARRELL T，et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：580-587.
[35] BIEN N，RAJPURKAR P，BALL R L，et al.Deep-learning-assisted diagnosis for knee magnetic resonance imaging：development and retrospective validation of MRNet[J].PLoS Medicine，2018，15（11）：e1002699.
[36] PAUL H Y，WEI J，KIM T K，et al.Radiology “forensics”：determination of age and sex from chest radiographs using deep learning[J].Emergency Radiology，2021：1-6.
[37] HWANG E J，NAM J G，LIM W H，et al.Deep learning for chest radiograph diagnosis in the emergency department[J].Radiology，2019，293（3）：573-580.
[38] CHENG C T，HO T Y，LEE T Y，et al.Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs[J].European Radiology，2019，29（10）：5469-5477.
[39] OH Y，PARK S，YE J C.Deep learning COVID-19 features on CXR using limited training data sets[J].IEEE Transactions on Medical Imaging，2020，39（8）：2688-2700.
[40] LI L，QIN L，XU Z，et al.Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT[J].Radiology，2020：200905.
[41] PASA F，GOLKOV V，PFEIFFER F，et al.Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization[J].Sci Rep，2019，9：6268.
[42] SMILKOV D，THORAT N，KIM B，et al.Smoothgrad：removing noise by adding noise[J].arXiv：1706.03825，2017.
[43] SITAULA C，HOSSAIN M B.Attention-based VGG-16 model for COVID-19 chest X-ray image classification[J].Applied Intelligence，2021，51（5）：2850-2863.
[44] BA J，MNIH V，KAVUKCUOGLU K.Multiple object recognition with visual attention[J].arXiv：1412.7755，2014.
[45] PESCE E，WITHEY S J，YPSILANTIS P P，et al.Learning to detect chest radiographs containing pulmonary lesions using visual attention networks[J].Medical Image Analysis，2019，53：26-38.
[46] GUAN Q，HUANG Y，ZHONG Z，et al.Diagnose like a radiologist：attention guided convolutional neural network for thorax disease classification[J].arXiv：1801. 09927，2018.
[47] HUANG J，ZHUO E，LI H，et al.Achieving accurate segmentation of nasopharyngeal carcinoma in MR images through recurrent attention[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2019：494-502.
[48] BENTAIEB A，HAMARNEH G.Predicting cancer with a recurrent visual attention model for histopathology images[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2018：129-137.
[49] MOMENI A，THIBAULT M，GEVAERT O.Deep recurrent attention models for histopathological image analysis[J].bioRxiv，2018：438341.
[50] OKTAY O，SCHLEMPER J，FOLGOC L L，et al.Attention u-net：learning where to look for the pancreas[J].arXiv：1804.03999，2018.
[51] SCHLEMPER J，OKTAY O，SCHAAP M，et al.Attention gated networks：learning to leverage salient regions in medical images[J].Medical Image Analysis，2019，53：197-207.
[52] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[53] ABRAMOVA V，CLèRIGUES A，QUILES A，et al.Hemorrhagic stroke lesion segmentation using a 3D U-Net with squeeze-and-excitation blocks[J].Computerized Medical Imaging and Graphics，2021，90：101908.
[54] ROY A G，NAVAB N，WACHINGER C.Concurrent spatial and channel “squeeze & excitation” in fully convolutional networks[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2018：421-429.
[55] ZHANG J，JIANG Z，DONG J，et al.Attention gate ResU-Net for automatic MRI brain tumor segmentation[J].IEEE Access，2020，8：58533-58545.
[56] 于金扣，余南南，于贺，等.基于多尺度层级化注意力模型的脑血肿分割算法[J].航天医学与医学工程，2021，34（1）：44-51.
YU J K，YU N N，YU H，et al.Segmentation algorithm of intracranial hematoma based on multi-scale hierarchical attention model[J].Space Medicine & Medical Engineering，2021，34（1）：44-51.
[57] SUN H，LI C，LIU B，et al.AUNet：attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms[J].Physics in Medicine & Biology，2020，65（5）：055005.
[58] WANG H，WANG S，QIN Z，et al.Triple attention learning for classification of 14 thoracic diseases using chest radiography[J].Medical Image Analysis，2021，67：101846.
[59] 胡屹杉，秦品乐，曾建潮，等.基于特征融合和动态多尺度空洞卷积的超声甲状腺分割网络[J].计算机应用，2021，41（3）：891-897.
HU Y S，QIN P L，ZENG J C，et al.Ultrasound thyroid segmentation network based on feature fusion and dynamic multi-scale dilated convolution[J].Journal of Computer Applications，2021，41（3）：891-897.
[60] WEI Z，SONG H，CHEN L，et al.Attention-based DenseUnet network with adversarial training for skin lesion segmentation[J].IEEE Access，2019，7：136616-136629.
[61] NI Z L，BIAN G B，ZHOU X H，et al.Raunet：residual attention u-net for semantic segmentation of cataract surgical instruments[C]//International Conference on Neural Information Processing.Cham：Springer，2019：139-149.
[62] 罗圣钦，陈金怡，李洪均.基于注意力机制的多尺度残差UNet实现乳腺癌灶分割[J/OL].计算机应用：1-6[2021-07-09].http：//kns.cnki.net/kcms/detail/51.1307.TP.20210708.
1645.016.html.
LUO S Q，CHEN J Y，LI H J.Multiscale residual UNet based on attention mechanism to realize breast cancer lesion segmentation[J/OL].Journal of Computer Applications：1-6[2021-07-09].http：//kns.cnki.net/kcms/detail/51.
1307.TP.20210708.1645.016.html.
[63] 郭宁，柏正尧.注意力机制下密集空洞卷积的肺部图像分割[J].中国图象图形学报，2021，26（9）：2146-2155.
GUO N，BO Z Y.The integration of attention mechanism and dense atrous convolution for lung image segmentation[J].Journal of Image and Graphics，2021，26（9）：2146-2155.
[64] 井杰，王蓓蕾，刘善荣.可解释人工智能在疾病诊疗中的应用[J].检验医学，2021，36（9）：976-980.
JING J，WANG B L，LIU S R.Explainable artificial intelligence in disease diagnosis and treatment[J].Laboratory Medicine，2021，36（9）：976-980.
[65] JIANG H，XU J，SHI R，et al.A multi-label deep learning model with interpretable grad-CAM for diabetic retinopathy classification[C]//2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society（EMBC），2020：1560-1563.
[66] CAI C J，WINTER S，STEINER D，et al.“Hello AI”：uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making[J].Proceedings of the ACM on Human-Computer Interaction，2019，3（CSCW）：1-24.
[67] DRAELOS R L，CARIN L.HiResCAM：faithful location representation in visual attention for explainable 3D medical image classification[J].arXiv：2011.08891，2020.
[68] GOTKOWSKI K，GONZALEZ C，BUCHER A，et al.M3d-CAM：a PyTorch library to generate 3D data attention maps for medical deep learning[J].arXiv：2007.00453，2020.
[69] TOSUN A B，PULLARA F，BECICH M J，et al.Explainable AI（xAI） for anatomic pathology[J].Advances in Anatomic Pathology，2020，27（4）：241-250.
[70] 胡耿，蔡延光.新冠肺炎CT影像的DNN对抗攻击研究[J].计算机工程与应用，2022，58（1）：152-157.
HU G，CAI Y G.Research of DNN adversarial attack on COVID-19 CT image dataset[J].Computer Engineering and Applications，2022，58（1）：152-157.
[71] CHEN S，BORTSOVA G，JUáREZ A G U，et al.Multi-task attention-based semi-supervised learning for medical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2019：457-465.
[72] WANG L，LIN Z Q，WONG A.COVID-Net：a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images[J].Scientific Reports，2020，10（1）：1-12.