Parallel Selective Kernel Attention Based on HardSoftmax

doi:10.3778/j.issn.1002-8331.2010-0085

Abstract

Abstract:

Attention has been widely used in Convolutional Neural Networks（CNNs）, and the performance of CNNs is effectively improved. At the same time, attention is very lightweight, and almost does not need to change the original architecture of CNNs. This paper proposes Parallel Selective Kernel（PSK） attention based on HardSoftmax. Firstly, to solve the problem that Softmax contains exponential operation, which is easy to occur computational overflow for large positive inputs, this paper proposes computationally safer HardSoftmax to replace Softmax. Then, different from Selective Kernel（SK） attention which puts the extraction and transformation of global features after feature fusion, PSK attention puts it in one branch alone, thus being in parallel connection with multiple branches with different kernel sizes. Meanwhile, the transformation of global features uses group convolution to further reduce the number of parameters and Multiply Adds（MAdds）. Finally, multiple branches with different kernel sizes are fused using HardSoftmax attention that is guided by the information in these branches. A wide range of image classification experiments show that just simply replacing Softmax with HardSoftmax can maintain or improve the performance of original attention. HardSoftmax also runs faster than Softmax in the experiments of this paper. PSK attention can match or outperform SK attention with less parameters and MAdds.

Key words: Convolutional Neural Networks（CNNs）, HardSoftmax, Parallel Selective Kernel（PSK） attention

摘要：

注意力被广泛地运用在卷积神经网络中，并有效地提升了卷积神经网络的性能。同时，注意力是非常轻量的，且几乎不需要改变卷积神经网络原来的架构。提出了基于HardSoftmax的并行选择核注意力。针对Softmax包含指数运算，对于较大的正输入很容易发生计算溢出的问题，提出了计算更安全的HardSoftmax来替换Softmax。不同于选择核注意力将全局特征的提取和转换放在特征融合之后，并行选择核注意力将全局特征的提取和转换单独放在一个分支，与具有不同核大小的多个分支构成并行结构。同时，并行选择核注意力的全局特征转换使用分组卷积，进一步减少参数量和计算量。并行选择核注意力通过HardSoftmax注意来关注不同核大小的多个分支。一系列的图像分类实验表明，只是简单地用HardSoftmax替换Softmax，也能保持或提升原注意力的性能。HardSoftmax的运行速度在实验中也比Softmax更快速。并行选择核注意力能够以更少的参数量和计算量追平或超越选择核注意力。

关键词: 卷积神经网络, HardSoftmax, 并行选择核注意力

ZHU Meng, MIN Weidong, ZHANG Yu, DUAN Jingwen. Parallel Selective Kernel Attention Based on HardSoftmax[J]. Computer Engineering and Applications, 2021, 57(21): 95-101.

朱梦，闵卫东，张煜，段静雯. 基于HardSoftmax的并行选择核注意力[J]. 计算机工程与应用, 2021, 57(21): 95-101.

References

[1] MIN W，FAN M，GUO X，et al.A new approach to track multiple vehicles with the combination of robust detection and two classifiers[J].IEEE Transactions on Intelligent Transportation Systems，2018，19（1）：174-186.
[2] MIN W D，GUO X G，HAN Q.An improved vibe algorithm and its application in traffic video processing[J].Guangxue Jingmi Gongcheng/Optics Precision Engineering，2017，25（3）：806-811.
[3] ZHOU L，MIN W，LIN D，et al.Detecting motion blurred vehicle logo in iov using filterdeblurgan and vl-yolo[J].IEEE Transactions on Vehicular Technology，2020，69（4）：3604-3614.
[4] YAO L，MIN W，LU K.A new approach to fall detection based on the human torso motion model[J].Applied Sciences，2017，7（10）：993.
[5] MIN W，CUI H，RAO H，et al.Detection of human falls on furniture using scene analysis based on deep learning and activity characteristics[J].IEEE Access，2018，6：9324-9335.
[6] XIONG X，MIN W，ZHENG W，et al.S3D-CNN：Skeletonbased 3d consecutive-low-pooling neural network for fall detection[J].Applied Intelligence，2020，50（1）：1-14.
[7] 赵中堂，陈继光，马倩.摔倒检测中的样本失衡问题研究[J].计算机工程与应用，2017，53（23）：142-146.
ZHAO Z T，CHEN J G，MA Q.Research on imbalance learning problem in fall detection[J].Computer Engineering and Applications，2017，53（23）：142-146.
[8] MIN W，SHI J，HAN Q.A distributed face recognition method and performance optimization[J].Optics and Precision Engineering，2017，25（3）：779-785.
[9] ZOU F，LI J，MIN W.Distributed face recognition based on load balancing and dynamic prediction[J].Applied Sciences，2019，9（4）.
[10] 董艳花，张树美，赵俊莉.有遮挡人脸识别方法综述[J].计算机工程与应用，2020，56（9）：1-12.
DONG Y H，ZHANG S M，ZHAO J L.Review of occlusion face recognition method[J].Computer Engineering and Applications，2020，56（9）：1-12.
[11] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012：1097-1105.
[12] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[13] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for largescale image recognition[C]//International Conference on Learning Representations，2015.
[14] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[15] HE K，ZHANG X，REN S，et al.Identity mappings in deep residual networks[C]//European Conference on Computer Vision，2016：630-645.
[16] ZAGORUYKO S，KOMODAKIS N.Wide residual networks[C]//Proceedings of the British Machine Vision Conference，2016：8701-8712.
[17] XIE S，GIRSHICK R，DOLLAR P，et al.Aggregated residual transformations for deep neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition，2017：1492-1500.
[18] IOFFE S，SZEGEDY C.Batch normalization：accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning，2015：448-456.
[19] SZEGEDY C，VANHOUCKE V，IOFFE S，et al.Rethinking the inception architecture for computer vision[C]//IEEE Conference on Computer Vision and Pattern Recognition，2016：2018-2826.
[20] SZEGEDY C，IOFFE S，VANHOUCKE V，et al.Inception-v4，inception-resnet and the impact of residual connections on learning[C]//AAAI Conference on Artificial Intelligence，2017：4278-4284.
[21] HUANG G，LIU Z，MAATEN L，et al.Densely connected convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[22] CHEN Y，LI J，XIAO H，et al.Dual path networks[C]//Advances in Neural Information Processing Systems，2017：4467-4475.
[23] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[24] HOWARD A，SANDLER M，CHU G，et al.Searching for mobilenetv3[C]//IEEE International Conference on Computer Vision，2019：1314-1324.
[25] LI X，WANG W，HU X，et al.Selective kernel networks[C]//IEEE Conference on Computer Vision and Pattern Recognition，2019：510-519.
[26] ZHANG H，WU C，ZHANG Z，et al.Resnest：split-attention networks[EB/OL].[2020].https：//arxiv.org/pdf/2004.
08955.pdf.
[27] TAN M，CHEN B，PANG R，et al.Mnasnet：platform-aware neural architecture search for mobile[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：2820-2828.
[28] SANDLER M，HOWARD A，ZHU M，et al.Mobilenetv2：Inverted residuals and linear bottlenecks[C]//IEEE Conference on Computer Vision and Pattern Recognition，2018：4510-4520.
[29] TAN M，LE Q.Efficientnet：rethinking model scaling for convolutional neural networks[C]//Proceedings of the 36th International Conference on Machine Learning，2019：6105-6114.
[30] BAKER B，GUPTA O，NAIK N，et al.Designing neural network architectures using reinforcement learning[C]//International Conference on Learning Representations，2017.
[31] CHEN X，XIE L，WU J，et al.Progressive differentiable architecture search：bridging the depth gap between search and evaluation[C]//IEEE/CVF International Conference on Computer Vision，2019：294-303.
[32] ZOPH B，VASUDEVAN V，SHLENS J，et al.Learning transferable architectures for scalable image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition，2018：8697-8710.
[33] REAL E，AGGARWAL A，HUANG Y，et al.Regularized evolution for image classifier architecture search[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：4780-4789.
[34] LIU H，SIMONYAN K，YANG Y.Darts：differentiable architecture search[C]//International Conference on Learning Representations，2019.
[35] KANDASAMY K，NEISWANGER W，SCHNEIDER J，et al.Neural architecture search with Bayesian optimisation and optimal transport[C]//Advances in Neural Information Processing Systems，2018：2016-2025.
[36] NAIR V，HINTON G E.Rectified linear units improve restricted boltzmann machines[C]//International Conference on Machine Learning，2010：807-814.
[37] HAWARD A G，ZHU M，CHEN B，et al.Mobilenets：efficient convolutional neural networks for mobile vision applications[J].arXiv：1704.04861，2017.
[38] PASZKE A，GROSS S，MASSA F，et al.Pytorch：an imperative style，highperformance deep learning library[C]//Advances in Neural Information Processing Systems，2019：8026-8037.
[39] XIAO H，RASUL K，VOLLGRAF R.Fashion-mnist：a novel image dataset for benchmarking machine learning algorithms[J].arXiv：1708.07747，2017.
[40] KRIZHEVSKY A，HINTON G.Learning multiple layers of features from tiny images[EB/OL].[2009].http：//citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&
rep=rep1&type=pdf.
[41] HE K，ZHANG X，REN S，et al.Delving deep into rectifiers：surpassing human-level performance on imagenet classification[C]//IEEE International Conference on Computer Vision，2015：1026-1034.
[42] LOSHCHILOW H，HUTTER F.Decoupled weight decay regularization[C]//International Conference on Learning Representations，2019.
[43] SUTSKEVER I，MARTENS J，DAHL G，et al.On the importance of initialization and momentum in deep learning[C]//International Conference on Machine Learning，2013：1139-1147.