结合关键点与权重分配残差网络的表情识别

doi:10.3778/j.issn.1002-8331.2101-0200

摘要/Abstract

摘要： 现有人脸表情识别算法易受图像背景、非表情内容等无关因素的影响。此外，部分人脸表情（例如害怕、生气、伤心等表情）的类间差异较小也制约着算法的性能。针对上述两个问题，提出了一种融合面部关键点和权重分配残差网络的表情识别算法。通过面部关键点获取最大的表情范围以消除图像背景和非表情内容的干扰，将预处理后的表情图像作为深度残差网络的输入，引入权重分配机制从通道和空间维度上进行注意权重推断，实现不同区域的权重分配，进而引导深度残差网络学习对表情具有鉴别力的局部特征。该算法分别在FER2013和CK+表情数据集上达到了74.14%和98.99%的识别准确率，有效改善了生气、伤心、害怕等类间差异较小的表情识别准确率。

关键词: 面部关键点, 权重分配, 残差网络, 表情识别

Abstract: Existing facial expression recognition algorithms are susceptible to irrelevant factors such as image background and non-expressive content. In addition, the small differences in some facial expressions（such as fear, angry, sad, etc.） between classes also restrict the performance of the algorithm. Aiming at the above two problems, this paper proposes an expression recognition algorithm that combines facial key points and a residual network of weight allocation. First, it obtains the largest expression range through facial key points to eliminate the interference of image background and non-expressive content. Then it uses the preprocessed expression image as the input of the deep residual network. Next, a weight distribution mechanism is introduced to infer attention weights from the channel and space dimensions to realize the weight distribution of different regions, and then it guides the deep residual network to learn local features that are discriminatory to expressions. The algorithm achieves recognition accuracy of 74.14% and 98.99% on the FER2013 and CK+ expression data sets, respectively, which effectively improves the recognition accuracy of expressions with little difference between angry, sad and fear categories.

Key words: facial key points, weight allocation, residual network, facial expression recognition

姜月武, 张玉金, 施建新. 结合关键点与权重分配残差网络的表情识别[J]. 计算机工程与应用, 2022, 58(17): 181-188.

JIANG Yuewu, ZHANG Yujin, SHI Jianxin. Expression Recognition Method Combining Key Points and Residual Network of Weight Allocation[J]. Computer Engineering and Applications, 2022, 58(17): 181-188.

参考文献

[1] EKMAN P.Strong evidence for universals in facial expressions：a reply to Russell’s mistaken critique[J].Psychological Bulletin，1994，115（2）：268-287.
[2] LI H，LI G M.Research on facial expression recognition based on LBP and deep learning[C]//Proceedings of 2019 International Conference on Robots & Intelligent System（ICRIS），2019：94-97.
[3] BASHYAL S，VENAYAGAMOORTHY G K.Recognition of facial expressions using Gabor wavelets and learning vector quantization[J].Engineering Applications of Artificial Intelligence，2008，21（7）：1056-1064.
[4] XIANG Z，TAN H，YE W.The excellent properties of a dense grid-based HOG feature on face recognition compared to Gabor and LBP[J].IEEE Access，2018，6：29306-29319.
[5] MATSUGU M，MORI K，MITARI Y，et al.Subject independent facial expression recognition with robust face detection using a convolutional neural network[J].Neural Networks，2003，16（5/6）：555-559.
[6] BARGAL S A，BARSOUM E，FERRER C C，et al.Emotion recognition in the wild from videos using images[C]//The 18th ACM International Conference on Multimodal Interaction，2016：433-436.
[7] LV Y，FENG Z，XU C.Facial expression recognition via deep learning[C]//2014 International Conference on Smart Computing（SMARTCOMP），2014：303-308.
[8] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[9] LI K，JIN Y，AKRAM M W，et al.Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy[J].The Visual Computer，2020，36（2）：391-404.
[10] KIM B K，ROH J，DONG S Y，et al.Hierarchical committee of deep convolutional neural networks for robust facial expression recognition[J].Journal on Multimodal User Interfaces，2016，10（2）：1-17.
[11] 褚晶辉，汤文豪，张姗，等.一种基于注意力模型的面部表情识别算法[J].激光与光电子学进展，2020，57（12）：197-204.
CHU J H，TANG W H，ZHANG S，et al.An attention model-based facial expression recognition algorithm[J].Laser and Optoelectronics Progress，2020，57（12）：197-204.
[12] JADERBERG M，SIMONYAN K，ZISSERMAN A.Spatial transformer networks[C]//Advances in Neural Information Processing Systems，2015：2017-2025.
[13] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[14] WANG Q，WU B，ZHU P，et al.ECA-net：efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：11534-11542.
[15] 张爱梅，徐杨.注意力分层双线性池化残差网络的表情识别[J].计算机工程与应用，2020，56（23）：161-166.
ZHANG A M，XU Y.Attention hierarchical bilinear pooling residual network for expression recognition[J].Computer Engineering and Applications，2020，56（23）：161-166.
[16] LUCEY P，COHN J F，KANADE T，et al.The extended Cohn-Kanade dataset（CK+）：a complete dataset for action unit and emotion-specified expression[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops，2010：94-101.
[17] DHALL A，GOECKE R，LUCEY S，et al.Static facial expression analysis in tough conditions：data，evaluation protocol and benchmark[C]//2011 IEEE International Conference on Computer Vision Workshops（ICCV Workshops），2011：2106-2112.
[18] KAZEMI V，SULLIVAN J.One millisecond face alignment with an ensemble of regression trees[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：1867-1874.
[19] HE K M，ZHANG X Y，REN S Q，et al.Deep residual learning for image recognition[C]//Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，2016：770-778.
[20] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//European Conference on Computer Vision.Cham：Springer，2018.
[21] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012：1097-1105.
[22] SIMONYAN K，ZISSERMAN A.Very deep convolutional net works for large-scale image recognition[J].arXiv：1409.1556，2014.
[23] SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：visual explanations from deep networks via gradient-based localization[J].International Journal of Computer Vision，2020，128（2）：336-359.
[24] MINAEE S，ABDOLRASHIDI A.Deep-emotion：facial expression recognition using attentional convolutional network[J].arXiv：1902.01019，2019.
[25] LIU X，ZHOU F.Improved curriculum learning using SSM for facial expression recognition[J].The Visual Computer，2019，35（10）：1-15.
[26] GAN Y，CHEN J，YANG Z，et al.multiple attention network for facial expression recognition[J].IEEE Access，2020，8：7383-7393.
[27] GAN Y，CHEN J，XU L.Facial expression recognition boosted by soft label with a diverse ensemble[J].Pattern Recognition Letters，2019，125：105-112.
[28] HUA W，DAI F，HUANG L，et al.HERO：human emotions recognition for realizing intelligent Internet of Things[J].IEEE Access，2019，7：24321-24332.