Facial Expression Recognition in Wild Based on Attention and Vision Transformer

doi:10.3778/j.issn.1002-8331.2111-0044

Abstract

Abstract: Facial expression recognition（FER） nowadays pays more attention to images in the wild that contain factors such as facial occlusion and image blurring than laboratory images, and the epidemic of COVID-19 makes people have to wear masks in public places, which brings new challenges to the task of FER. Due to the recent success of the Transformer on numerous computer vision tasks, an Attention-Transformer network is proposed, which first using CSWin Transformer as the backbone. Meanwhile, channel-spatial attention module is designed to increase the attention of the network to global features. Moreover, the Sub-center ArcFace loss function is used to further optimize the classification ability of the model. The proposed method is evaluated on two public in-the-wild facial expression datasets, RAF-DB and FERPlus, and their corresponding masked datasets. In addition, their accuracy rates are 88.80% and 89.31% on RAF-DB and FERPlus datasets, and the accuracy rates are 76.12% and 72.28% on their masked datasets. In a nutshell, the results demonstrate that the model performs superior to the state-of-the-art methods.

Key words: facial expression recognition, Transformer, attention mechanism, Sub-center ArcFace

摘要： 目前的人脸表情识别更关注包含面部遮挡、图像模糊等因素的野外图像而非实验室图像，且COVID-19的流行使得人们不得不在公共场合佩戴口罩，这给表情识别任务带来了新的挑战。受启发于最近Transformer在众多计算机视觉任务上的成功，提出了基于注意力及视觉Transformer的野外人脸表情识别模型，并率先使用CSWin Transformer作为主干网络。加入通道-空间注意力模块来提高模型对于全局特征的注意力。Sub-center ArcFace损失函数被用来进一步优化模型的分类能力。在两个公开的野外表情数据集RAF-DB和FERPlus上以及它们对应的口罩遮挡数据集上对所提出的方法进行了评估，识别准确率分别为88.80%、89.31%和76.12%、72.28%，提高了表情识别精度。

关键词: 人脸表情识别, Transformer, 注意力机制, Sub-center ArcFace

LUO Yan, FENG Tianbo, SHAO Jie. Facial Expression Recognition in Wild Based on Attention and Vision Transformer[J]. Computer Engineering and Applications, 2022, 58(10): 200-207.

罗岩, 冯天波, 邵洁. 基于注意力及视觉Transformer的野外人脸表情识别[J]. 计算机工程与应用, 2022, 58(10): 200-207.

References

[1] BARENTINE C，MCNAY A，PFAFFENBICHLER R，et al.A VR teleoperation suite with manipulation assist[C]//2021 ACM/IEEE International Conference on Human-Robot Interaction，2021：442-446.
[2] FEI Z，YANG E，LI D D U，et al.Deep convolution network based emotion analysis towards mental health care[J].Neurocomputing，2020，388：212-227.
[3] JOSHI A，KYAL S，BANERJEE S，et al.In-the-wild drowsiness detection from facial expressions[C]//2020 IEEE Intelligent Vehicles Symposium，2020：207-212.
[4] ZHAO X，LIANG X，LIU L，et al.Peak-piloted deep network for facial expression recognition[C]//14th European Conference on Computer Vision.Cham：Springer，2016：425-442.
[5] YU Z，LIU G，LIU Q，et al.Spatio-temporal convolutional features with nested LSTM for facial expression recognition[J].Neurocomputing，2018，317：50-57.
[6] LI Y，ZENG J，SHAN S，et al.Patch-gated CNN for occlusion-aware facial expression recognition[C]//2018 24th International Conference on Pattern Recognition，2018：2209-2214.
[7] LI Y，ZENG J，SHAN S，et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing，2018，28（5）：2439-2450.
[8] WANG K，PENG X，YANG J，et al.Region attention networks for pose and occlusion robust facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4057-4069.
[9] DING H，ZHOU P，CHELLAPPA R.Occlusion-adaptive deep network for robust facial expression recognition[C]//2020 IEEE International Joint Conference on Biometrics，2020：1-9.
[10] WANG K，PENG X，YANG J，et al.Suppressing uncertainties for large-scale facial expression recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6897-6906.
[11] SHAO J，LUO Y.TAMNet：two attention modules-based network on facial expression recognition under uncertainty[J].Journal of Electronic Imaging，2021，30（3）：033021.
[12] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16x16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[13] TOUVRON H，CORD M，DOUZE M，et al.Training data-efficient image transformers & distillation through attention[C]//2021 International Conference on Machine Learning，2021：10347-10357.
[14] MA F，SUN B，LI S.Robust facial expression recognition with convolutional visual transformers[J].arXiv：2103.16854，2021.
[15] LI H，SUI M，ZHAO F，et al.MViT：mask vision transformer for facial expression recognition in the wild[J].arXiv：2106.04520，2021.
[16] DONG X，BAO J，CHEN D，et al.CSWin transformer：a general vision transformer backbone with cross-shaped windows[J].arXiv：2107.00652，2021.
[17] PARK J，WOO S，LEE J Y，et al.BAM：bottleneck attention module[C]//British Machine Vision Conference，2018.
[18] DENG J，GUO J，XUE N，et al.Arcface：additive angular margin loss for deep face recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4690-4699.
[19] DENG J，GUO J，LIU T，et al.Sub-center ArcFace：boosting face recognition by large-scale noisy web faces[C]//16th European Conference on Computer Vision.Cham：Springer，2020：741-757.
[20] LOSHCHILOV I，HUTTER F.Decoupled weight decay regularization[C]//2018 International Conference on Learning Representations，2018.
[21] LI S，DENG W，DU J P.Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：2852-2861.
[22] BARSOUM E，ZHANG C，FERRER C C，et al.Training deep networks for facial expression recognition with crowd-sourced label distribution[C]//18th ACM International Conference on Multimodal Interaction，2016：279-283.
[23] WANG Z，WANG G，HUANG B，et al.Masked face recognition dataset and application[J].arXiv：2003.09093，2020.
[24] 吕诲，童倩倩，袁志勇.基于人脸分割的复杂环境下表情识别实时框架[J].计算机工程与应用，2020，56（12）：134-140.
LV H，TONG Q Q，YUAN Z Y.Realtime architecture for facial expression recognition in complex scenes based on face region segmentation[J].Computer Engineering and Applications，2020，56（12）：134-140.
[25] 毛君宇，何廷年，郭艺，等.基于全局注意力及金字塔卷积网络的表情识别[J/OL].计算机工程与应用（2021-09-13）[2021-11-30].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210913.
1327.014.html.
MAO J Y，HE Y N，GUO Y，et al.Expression recognition based on global attention and pyramidal convolution network[J/OL].Computer Engineering and Applications（2021-09-13）[2021-11-30].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210913.1327.014.html.
[26] FAN X，DENG Z，WANG K，et al.Learning discriminative representation for facial expression recognition from un-certainties[C]//2020 IEEE International Conference on Image Processing，2020：903-907.
[27] SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：visual explanations from deep networks via gradient-based localization[C]//2017 IEEE International Conference on Computer Vision，2017：618-626.
[28] GRAHAM B，EL-NOUBY A，TOUVRON H，et al.LeViT：a vision Transformer in ConvNet’s clothing for faster inference[J].arXiv：2104.01136，2021.
[29] ZHANG Q，YANG Y.ResT：an efficient transformer for visual recognition[J].arXiv：2105.13677，2021.
[30] CHU X，TIAN Z，WANG Y，et al.Twins：revisiting spatial attention design in vision transformers[J].arXiv：2104.13840，
2021.
[31] LIU Z，LIN Y，CAO Y，et al.Swin Transformer：hierarchical vision transformer using shifted windows[J].arXiv：2103.14030，2021.
[32] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.