Facial Expression Recognition Method Embedded with Attention Mechanism Residual Network

doi:10.3778/j.issn.1002-8331.2207-0315

Abstract

Abstract: Aiming at the problems that face images in uncontrollable environments are susceptible to complex factors such as illumination and pose changes, which in turn cause low face detection rate and poor expression recognition accuracy in face expression recognition, an expression recognition method with an embedded attention mechanism residual network is proposed. In the stage of face detection, the improved RetinaFace algorithm is used to complete multi-view face detection and obtain the face region. In the stage of feature extraction, ResNet-50 is used as the backbone network for feature extraction. Firstly, the pre-processed face images are sequentially passed through the channel attention network and spatial attention network of this model to explicitly model the global image interdependence. Secondly, in the shortcut connection of the dashed residual cells, an average ensemble layer is added for the downsampling operation. By fine-tuning the operation of the residual module, the mapping between the input features is enhanced, so that the extracted expression features can be passed between the networks more completely, so as to reduce the loss of feature information. Finally, the convolutional block attention module（CBAM） attention mechanism module is passed into the network again to enhance the channel dimension information and spatial dimension information of local expression features, strengthen the focus information of feature regions with high relevance to expressions in the feature map, and suppress the interference of irrelevant regions in the feature map, thus speeding up the convergence speed of the network and improving the expression recognition rate. Compared with the baseline algorithm, this method achieves 87.65% and 73.57% accuracy on the RAF-DB and FER2013 expression datasets, respectively.

Key words: attention mechanism, residual network, expression recognition, convolutional block attention module, RetinaFace

摘要： 针对非可控环境下人脸图像易受光照、姿态变化等复杂因素的影响，进而造成人脸表情识别中人脸检测率低、表情识别精度差的问题，提出了一种嵌入注意力机制残差网络的表情识别方法。在人脸检测阶段，采用改进的RetinaFace算法完成多视角人脸检测，获取人脸区域。在特征提取阶段，使用ResNet-50作为特征提取的主干网络。将预处理后的人脸图片，依次通过该网络的通道注意力网络和空间注意力网络，显式地建模全局图像的相互依赖性。在虚线残差单元的快捷连接中，加入平均池化层进行下采样操作，通过微调残差模块的操作，加强输入特征之间的映射，使提取的表情特征能够较完整地在网络之间传递，以减小特征信息的损失；在网络中再次传入卷积注意力机制模块，增强局部表情特征的通道维度信息和空间维度信息，加强特征图中与表情相关性高的特征区域的重点信息，同时抑制特征图中无关区域的干扰，进而加快网络的收敛速度，提高表情识别率。与基线算法相比，该方法在RAF-DB和FER2013表情数据集上分别取得了87.65%和73.57%的准确率。

关键词: 注意力机制, 残差网络, 表情识别, 卷积注意力机制模块, RetinaFace

ZHONG Rui, JIANG Bin, LI Nanxing, CUI Xiaomei. Facial Expression Recognition Method Embedded with Attention Mechanism Residual Network[J]. Computer Engineering and Applications, 2023, 59(11): 88-97.

钟瑞, 蒋斌, 李南星, 崔晓梅. 嵌入注意力机制残差网络的人脸表情识别方法[J]. 计算机工程与应用, 2023, 59(11): 88-97.

References

[1] HANAFI W N W，DAUD S.Managing sustainable develop-ment of government link companies（GLCs） in Malaysia through emotional intelligence and organisational politics[J].International Journal of Innovation and Sustainable Develop-ment，2021，15（1）：126-141.
[2] LI B，LIMA D.Facial expression recognition via ResNet-50[J].International Journal of Cognitive Computing in Engineering，2021，2：57-64.
[3] TRAN L，YIN X，LIU X.Representation learning by rotating your faces[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，41（12）：3007-3021.
[4] WANG K，PENG X，YANG J，et al.Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6897-6906.
[5] WANG K，PENG X，YANG J，et al.Region attention networks for pose and occlusion robust facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4057-4069.
[6] YADAV R，KACKER P.Efficient technique for real-time face detection[J].Materials Today：Proceedings，2022，56：3306-3311.
[7] VIOLA P，JONES M J.Robust real-time face detection[J].International Journal of Computer Vision，2004，57（2）：137-154.
[8] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[9] ZHANG K，ZHANG Z，LI Z，et al.Joint face detection and alignment using multitask cascaded convolutional networks[J].IEEE Signal Processing Letters，2016，23（10）：1499-1503.
[10] 张玉云.基于MTCNN和S-LBPH算法的多人脸识别在课堂考勤中的应用[D].成都：西南交通大学，2019.
ZHANG Y Y.Application of multiple face recognition based on MTCNN and S-LBPH algorithm in classroom attendance[D].Chengdu：Southwest Jiaotong University，2019.
[11] 杨泽睿.监控场景下的人脸检测及识别算法研究[D].广州：华南理工大学，2020.
YANG Z R.Pedestrian face detection and recognition algorithm research based on surveillance entrance video[D].Guangzhou：South China University of Technology，2020.
[12] 刘成攀.基于深度学习的动态人脸识别[D].绵阳：西南科技大学，2020.
LIU C P.Dynamic face recognition based on deep learning[D].Mianyang：Southwest University of Science and Technology，2020.
[13] 刘晨.基于机器人视觉系统的人脸检测技术研究[J].电子设计工程，2019，27（5）：109-112.
LIU C.Research on face detection technology based on robot vision system[J].Electronic Design Engineering，2019，27（5）：109-112.
[14] FENG Y，YU S，PENG H，et al.Detect faces efficiently：a survey and evaluations[J].IEEE Transactions on Biometrics，Behavior，and Identity Science，2021，4（1）：1-18.
[15] DENG J，GUO J，VERVERAS E，et al.RetinaFace：single-shot multi-level face localisation in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5203-5212.
[16] MAJUMDER A，BEHERA L，SUBRAMANIAN V K.Emotion recognition from geometric facial features using self-organizing map[J].Pattern Recognition，2014，47（3）：1282-1293.
[17] TONG Y，CHEN R，YANG J，et al.Robust facial expression recognition based on local tri-directional coding pattern[C]//Conference on Complex，Intelligent，and Software Intensive Systems.Cham：Springer，2018：606-614.
[18] WOLD S，ESBENSEN K，GELADI P.Principal component analysis[J].Chemometrics and Intelligent Laboratory Systems，1987，2（1/2/3）：37-52.
[19] OJALA T，PIETIKAINEN M，MAENPAA T.Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2002，24（7）：971-987.
[20] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition（CVPR’05），2005：886-893.
[21] HESSE N，GEHRIG T，GAO H，et al.Multi-view facial expression recognition using local appearance features[C]//Proceedings of the 21st International Conference on Pattern Recognition（ICPR2012），2012：3533-3536.
[22] 陈香敏，柯丽，杜强.基于KPCANet和线性判别分析的表情识别[J].华中科技大学学报（自然科学版），2020，48（9）：95-99.
CHEN X M，KE L，DU Q.Kernel principal component analysis network and linear discriminate analysis for expression recoanition[J].Journal of Huazhong University of Science and Technology（Nature Science Edition），2020，48（9）：95-99.
[23] 乔桂芳，侯守明，刘彦彦.基于改进卷积神经网络与支持向量机结合的面部表情识别算法[J].计算机应用，2022，42（4）：1253-1259.
QIAO G F，HOU S M，LIU Y Y.Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine[J].Journal of Computer Applications，2022，42（4）：1253-1259.
[24] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[25] HINTON G E，SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science，2006，313（5786）：504-507.
[26] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012.
[27] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[28] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[29] 周本君.基于卷积神经网络的人脸表情识别研究[D].南京：南京邮电大学，2019.
ZHOU B J.Research on facial expression recognition based on convolutional neural network[D].Nanjing：Nanjing University of Posts and Telecommunications，2019.
[30] 王珂.基于深度学习的人脸表情识别算法研究与实现[D].南京：南京信息工程大学，2019.
WANG K.Research and implementation of face expression recognition algorithm based on deep learning[D].Nanjing：Nanjing University of Information Engineering，2019.
[31] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[32] ZHONG R，JIANG B，LI N X，et al.A multi-view face detection and expression recognition method with improved RetinaFace[C]//2021 6th International Conference on Mechanical，Control and Computer Engineering（ICMCCE），Sanya，China，December 24-26，2021：1409-1412.
[33] ALMAHAIRI A，BALLAS N，COOIJMANS T，et al.Dynamic capacity networks[C]//International Conference on Machine Learning，2016：2549-2558.
[34] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：3-19.
[35] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[36] ZHANG H，SU W，YU J，et al.Identity-expression dual branch network for facial expression recognition[J].IEEE Transactions on Cognitive and Developmental Systems，2020，13（4）：898-911.
[37] YU N，BAI D.A visual self-attention network for facial expression recognition[C]//2021 International Joint Conference on Neural Networks（IJCNN），2021：1-8.
[38] WANG Z，ZENG F，LIU S，et al.OAENet：oriented attention ensemble for accurate facial expression recognition[J].Pattern Recognition，2021，112：107694.
[39] ZHOU Y，JIN L，MA G，et al.Quaternion capsule neural network with region attention for facial expression recognition in color images[J].IEEE Transactions on Emerging Topics in Computational Intelligence，2022，6（4）：893-912.
[40] Xia Y，Yu H，Wang X，et al.Relation-aware facial expression recognition[J].IEEE Transactions on Cognitive and Developmental Systems，2022，14（3）：1143-1154.
[41] JIA C，LI C L，YING Z.Facial expression recognition based on the ensemble learning of CNNs[C]//2020 IEEE International Conference on Signal Processing，Communications and Computing（ICSPCC），2020：1-5.
[42] XIE W，SHEN L，DUAN J.Adaptive weighting of handcrafted feature losses for facial expression recognition[J].IEEE Transactions on Cybernetics，2019，51（5）：2787-2800.
[43] HAYALE W，NEGI P S，MAHOOR M.Deep siamese neural networks for facial expression recognition in the wild[J].IEEE Transactions on Affective Computing，2021.
[44] 陈加敏，徐杨.注意力金字塔卷积残差网络的表情识别[J].计算机工程与应用，2022，58（22）：123-131.
CHEN J M，XU Y.Expression recognition based on convolution residual network of attention pyramid[J].Computer Engineering and Applications，2022，58（22）：123-131.