Review of Attention Mechanism in Convolutional Neural Networks

doi:10.3778/j.issn.1002-8331.2105-0135

Abstract

Abstract:

Attention mechanism is widely used in deep learning tasks because of its excellent effect and plug and play convenience. This paper mainly focuses on convolution neural network, introduces various mainstream methods in the development process of convolution network attention mechanism, extracts and summarizes its core idea and implementation process, realizes each attention mechanism method, and makes comparative experiments and results analysis on the measured data of the same type of emitter equipment. According to the main ideas and experimental results, the research status and future development direction of attention mechanism in convolutional networks are summarized.

Key words: attention mechanism, convolutional neural network, development direction

摘要：

注意力机制因其优秀的效果与即插即用的便利性，在深度学习任务中得到了越来越广泛的应用。主要着眼于卷积神经网络，对卷积网络注意力机制发展过程中的各种主流方法进行介绍，并对其核心思想与实现过程进行提取与总结，同时对每种注意力机制方法进行实现，针对同型号辐射源设备实测数据进行对比实验与结果分析，并依据主流方法的思想与实验的结果总结并阐述了卷积网络中的注意力机制的研究现状与未来其发展方向。

关键词: 注意力机制, 卷积神经网络, 发展方向

ZHANG Chenjia, ZHU Lei, YU Lu. Review of Attention Mechanism in Convolutional Neural Networks[J]. Computer Engineering and Applications, 2021, 57(20): 64-72.

张宸嘉，朱磊，俞璐. 卷积神经网络中的注意力机制综述[J]. 计算机工程与应用, 2021, 57(20): 64-72.

References

[1] 王文冠，沈建冰，贾云得.视觉注意力检测综述[J].软件学报，2019，30（2）：416-439.
WANG Wenguan，SHEN Jianbing，JIA Yunde.Review of visual attention detection[J].Acta Sinica Sinica，2019，30（2）：416-439.
[2] 任欢，王旭光.注意力机制综述[J].计算机应用，2021，41（21）：1-6.
REN Huan，WANG Xuguang.Review of attention mechanism[J].Computer Application，2021，41（21）：1-6.
[3] 冯兴杰，张志伟，史金钏.基于卷积神经网络和注意力模型的文本情感分析[J].计算机应用研究，2018，35（5）：1434-1436.
FENG Xingjie，ZHANG Zhiwei，SHI Jinchuan.Text sentiment analysis based on convolutional neural network and attention model[J].Computer Application Research，2018，35（5）：1434-1436.
[4] ZHONG Z，LIN Z Q，BIDART R. et al.Squeeze-and-attention networks for semantic segmentation[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Seattle，WA，USA，2020：13062-13071.
[5] ULUTAN O，IFTEKHAR A S M，MANJUNATH B S.VSGNet：spatial attention network for detecting human object interactions using graph convolutions[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Seattle，WA，USA，2020：13614-13623.
[6] 张卫锋.跨媒体数据语义分析技术研究[D].杭州：杭州电子科技大学，2019.
ZHANG Weifeng.Research on semantic analysis of cross media data[D].Hangzhou：Hangzhou University of Electronic Science and Technology，2019.
[7] 朱丰，胡晓峰，吴琳，等.从态势认知走向态势智能认知[J].系统仿真学报，2018，30（3）：761-771.
ZHU Feng，HU Xiaofeng，WU Lin，et al.From situation cognition to situation intelligent cognition[J].Journal of System Simulation，2018，30（3）：761-771.
[8] 梁斌，刘全，徐进，等.基于多注意力卷积神经网络的特定目标情感分析[J].计算机研究与发展，2017，54（8）：1724-1735.
LIANG Bin，LIU Quan，XU Jin，et al.Target specific emotion analysis based on multi attention convolutional neural network[J].Computer Research and Development，2017，54（8）：1724-1735.
[9] XU Tingting，ZHENG Guping.Semantic segmentation of street scene based on multi-scale and attention mechanism[J].World Scientific Research Journal，2021，7（5）.
[10] 朱张莉，饶元，吴渊，等.注意力机制在深度学习中的研究进展[J].中文信息学报，2019，33（6）：1-11.
ZHU Zhangli，RAO Yuan，WU Yuan，et al.Research progress of attention mechanism in deep learning[J].Chinese Journal of Information，2019，33（6）：1-11.
[11] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[J].
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[12] 耿磊，邱玲，吴骏，等.结合深度可分离卷积与通道加权的全卷积神经网络视网膜图像血管分割[J].生物医学工程学杂志，2019，36（1）：107-115.
GENG Lei，QIU Ling，WU Jun，et al.Retinal vascular segmentation based on deep separable convolution and channel weighted convolution neural network[J].Journal of Biomedical Engineering，2019，36（1）：107-115.
[13] 盛纾纬.基于弱监督学习的细粒度图像识别技术研究[D].成都：电子科技大学，2019.
SHENG Shuwei.Research on fine-grained image recognition technology based on weak supervised learning[D].Chengdu：University of Electronic Science and Technology，2019.
[14] WANG Q，WU B，ZHU P，et al.ECA-Net：efficient channel attention for deep convolutional neural networks[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Seattle，WA，USA，2020：11531-11539.
[15] LI X，WANG W，HU X，et al.Selective kernel networks[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，CA，USA，2019：510-519.
[16] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Las Vegas，NV，USA，2016：770-778.
[17] ZHANG H，WU C R，ZHANG Z Y，et al.ResNeSt：split-attention networks[J].arXiv：2004.08955，2020.
[18] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[J].arXiv：1807.06521v2，2018.
[19] WANG Yibin，WANG Haifeng，PENG Zhaohua.Rice diseases detection and classification using attention based neural network and Bayesian optimization[J].Expert Systems with Applications，2021，178.
[20] YU Y，LIU M，FENG H，et al.Split-attention multiframe alignment network for image restoration[J].IEEE Access，2020：39254-39272.
[21] 李启行，廖薇，孟静雯.基于注意力机制的双通道DAC-RNN文本分类模型[J/OL].计算机工程与应用：1-9（2021-04-20）[2021-05-10].http：//kns.cnki.net/kcms/detail/11.2127.tp.20210420.1354.070.html.
LI Qixing，LIAO Wei，MENG Jingwen.Two channel DAC-RNN text classification model based on attention mechanism[J/OL].Computer Engineering and Application：1-9（2021-04-20）[2021-05-10].http：//kns.cnki.net/kcms/detail/11.2127.tp.20210420.1354.070.html.
[22] LI Zhongzhi，QU Na，LI Xiaoxue，et al.Partial discharge detection of insulated conductors based on CNN-LSTM of attention mechanisms[J].Journal of Power Electronics，2021.
[23] LIU Yifan，ZHU Qigang，CAO Feng，et al.High-resolution remote sensing image segmentation framework based on attention mechanism and adaptive weighting[J].ISPRS International Journal of Geo-Information，2021，10（4）：241.
[24] ZHU Xiaoqing，HAN Junjun.Research on visual question answering based on deep stacked attention network[J].Journal of Physics：Conference Series，2021，1873（1）.
[25] YIN Fulian，LI Sitong，JI Meiqi，et al.Neural TV program recommendation with label and user dual attention[J].Applied Intelligence，2021.
[26] XIE L，HUANG C.A residual network of water scene recognition based on optimized inception module and convolutional block attention module[C]//6th International Conference on Systems and Informatics（ICSAI），Shanghai，China，2019：1174-1178.
[27] FU J.Dual attention network for scene segmentation[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，CA，USA，2019：3141-3149.
[28] ZHAO T，WU X.Pyramid feature attention network for saliency detection[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，CA，USA，2019：3080-3089.
[29] 石磊，王毅，成颖，等.自然语言处理中的注意力机制研究综述[J].数据分析与知识发现，2020，4（5）：1-14.
SHI Lei，WANG Yi，CHENG Ying，et al.A review of attention mechanism in natural language processing[J].Data Analysis and Knowledge Discovery，2020，4（5）：1-14.
[30] 乔若羽.基于注意力机制的神经网络预测模型[D].合肥：中国科学技术大学，2020.
QIAO Ruoyu.Neural network prediction model based on attention mechanism[D].Hefei：University of Science and Technology of China，2020.
[31] 陈龙杰，张钰，张玉梅，等.基于多注意力多尺度特征融合的图像描述生成算法[J].计算机应用，2019，39（2）：354-359.
CHEN Longjie，ZHANG Yu，ZHANG Yumei，et al.Image description generation algorithm based on multi attention and multi-scale feature fusion[J].Computer Applications，2019，39（2）：354-359.
[32] 边继龙，王厚博，李金凤.基于多尺度注意力网络的立体匹配方法[J/OL].北京邮电大学学报：1-7[2021-05-10].https：//doi.org/10.13190/j.jbupt.2020-204.
BIAN Jilong，WANG Houbo，LI Jinfeng.Stereo matching method based on multi-scale attention network[J/OL].Journal of Beijing University of Posts and Telecommunications：1-7[2021-05-10].https：//doi.org/10.13190/j.jbupt.
2020-204.
[33] CHENG Q，LI H，WU Q，et al.Learn to pay attention via switchable attention for image recognition[C]//IEEE Conference on Multimedia Information Processing and Retrieval（MIPR），2020：291-296.
[34] ZENG Y，GUO X，WANG H，et al.Efficient dual attention module for real-time visual tracking[C]//IEEE Visual Communications and Image Processing（VCIP），2019：1-4.
[35] FANG P，ZHOU J，ROY S K，et al.Attention in attention networks for person retrieval[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021.
[36] YANG Z，ZHANG T，YANG J.Research on classification algorithms for attention mechanism[C]//19th International Symposium on Distributed Computing and Applications for Business Engineering and Science（DCABES），2020：194-197.
[37] LI C.Attention Unet++：a nested attention-aware U-Net for liver CT image segmentation[C]//IEEE International Conference on Image Processing（ICIP），2020：345-349.
[38] HE A，LI T，LI N，et al.CABNet：category attention block for imbalanced diabetic retinopathy grading[J].IEEE Transactions on Medical Imaging，2021：143-153.
[39] ZHU M，JIAO L，LIU F，et al.Residual spectral-spatial attention network for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2021：449-462.
[40] XING X，YUAN Y，MENG M Q H.Zoom in lesions for better diagnosis：attention guided deformation network for WCE image classification[J].IEEE Transactions on Medical Imaging，2020：4047-4059.
[41] PENG B，YU Z，LEI J，et al.Attention-guided fusion network of point cloud and multiple views for 3D shape recognition[C]//IEEE International Conference on Visual Communications and Image Processing（VCIP），2020：185-188.
[42] MEI R，WANG H，MEN A.Attention-enhanced and more balanced R-CNN for object detection[C]//IEEE International Conference on Image Processing（ICIP），2020：2136-2140.
[43] SUN L，SHAO W，ZHANG D，et al.Anatomical attention guided deep networks for ROI segmentation of brain MR images[J].IEEE Transactions on Medical Imaging，2020：2000-2012.
[44] YI Z，SUN Z，FENG J，et al.3D residual networks with channel-spatial attention module for action recognition[C]//Chinese Automation Congress（CAC），2020：5171-5174.
[45] GAO Y，GONG H，DING X，et al.Image recognition based on mixed attention mechanism in smart home appliances[C]//IEEE 5th Advanced Information Technology，Electronic and Automation Control Conference（IAEAC），2021：1501-1505.