采用支路辅助学习的人脸表情识别

doi:10.3778/j.issn.1002-8331.2111-0113

摘要/Abstract

摘要： 人脸表情识别属于一种细粒度识别，模型需要同时聚焦于浅层与深层特征。针对独立结构的卷积神经网络对细粒度特征的提取、融合能力不足的问题，提出一种基于支路辅助学习的网络结构。在基础网络的输入层引入一条支路辅助网络，该网络将逐层使用金字塔卷积块提取全局特征；通过特征映射模块不断将支路提取到的决策信息传导至基础网络，辅助基础网络提取细节特征；在模型输出层采用特征拼接的方式将支路网络与主路网络融合。将所提出的网络在公开人脸表情数据集CK+、JAFFE、FER2013和MMEW上进行识别实验，结果表明：支路辅助学习模块能够有效提升基础网络的特征提取能力和泛化能力，提出的方法识别率达到了98.89%、94.80%、71.88%和86.67%，比仅采用基础网络（例如：ResNet50）进行识别提高了3.49、2.2、5.51和1.48个百分点。

关键词: 人脸表情识别, 卷积神经网络, 支路辅助学习, 金字塔卷积块

Abstract: Facial expression recognition is a kind of fine-grained recognition, this model needs to focus on both shallow and deep features. Aiming at the problem that the convolutional neural network with independent structure has insufficient ability to extract and fuse fine-grained features, a network structure based on branch-assisted learning is proposed in this paper. Firstly, a branch network is introduced into the input layer of the basic network, which will extract global features by using pyramid convolution blocks. Then, through the feature-mapping module, the decision information extracted by the branch is continuously transmitted to the basic network to assist the basic network to extract the detailed features. Finally, in the output layer of the model, the branch network and base network are fused by feature stitching. The recognition experiments of the proposed network on the public facial expression data sets CK+, JAFFE, FER2013 and MMEW show that the branch-assisted learning module can effectively improve the feature extraction ability and generalization ability of the basic network, and the recognition rate of the proposed method reaches 98.89%, 94.80%, 71.88% and 86.67%, which is 3.49, 2.2, 5.51 and 1.48 percentage points higher than that of only using the basic network（e.g.ResNet50）.

Key words: facial expression recognition, convolutional neural network, branch-assisted learning, pyramid convolution block

赵家琦, 周颖玥, 王欣宇, 李驰. 采用支路辅助学习的人脸表情识别[J]. 计算机工程与应用, 2022, 58(23): 151-160.

ZHAO Jiaqi, ZHOU Yingyue, WANG Xinyu, LI Chi. Facial Expression Recognition Method Based on Branch-Assisted Learning Network[J]. Computer Engineering and Applications, 2022, 58(23): 151-160.

参考文献

[1] EKMAN P.Strong evidence for universals in facial expressions：a reply to Russel’s mistaken critique[J].Psychological Bulletin，1994，115（2）：268-287.
[2] LI H，LI G.Research on facial expression recognition based on LBP and deep learning[C]//Proceedings of 2019 International Conference on Robots & Intelligent System（ICRIS），2019：94-97.
[3] BASHYAL S，VENAYAGAMOORTHY G K.Recognition of facial expressions using Gabor wavelets and learning vector quantization[J].Engineering Applications of Artificial Intelligence，2007，21（7）：1056-1064.
[4] BERRETTI S，BIMBO A D，PAIA P，et al.A set of selected SIFT features for 3D facial expression recognition[C]//2010 IEEE 20th International Conference on Pattern Recognition（ICPR），2010：4125-4128.
[5] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[6] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[7] SZEGEDY C，VANHOUCKE V，IOFFE S，et al.Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2818-2826.
[8] SANDLER M，HOWARD A，ZHU M，et al.MobileNetv2：inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4510-4520.
[9] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[10] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：3-19.
[11] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16×16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[12] CONNIE T，AL-SHABI M，CHEAH W P，et al.Facial expression recognition using a hybrid CNN-SIFT aggregator[C]//International Workshop on Multi-disciplinary Trends in Artificial Intelligence.Cham：Springer，2017：139-149.
[13] GU S T，XU C，FENG B.Facial expression recognition based on global and local feature fusion with CNNs[C]//2019 IEEE International Conference on Signal Processing，Communications and Computing（ICSPCC），2019：1-5.
[14] 柳璇，唐颖军，黄淑英.结合多特征和跨连通道加权的面部表情识别[J].小型微型计算机系统，2021，42（2）：399-404.
LIU X，TANG Y J，HUANG S Y.Facial expression recognition combined with multiple features and cross-channel weighting[J].Journal of Chinese Computer Systems，2021，42（2）：399-404.
[15] 高红霞，郜伟.融合关键点属性与注意力表征的人脸表情识别[J/OL].计算机工程与应用：1-10[2021-10-14].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210927.2023.002.html.
GAO H X，GAO W.Facial expression recognition integrating key point attributes and attention representation[J/OL].Computer Engineering and Applications：1-10[2021-10-14].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210927.
2023.002.html.
[16] DING X，ZHANG X，MA N，et al.RepVGG：making VGG-style convnets great again[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：13733-13742.
[17] MA J B，WANG W，WANG L.Irregular convolutional neural networks[C]//2017 4th IAPR Asian Conference on Pattern Recognition（ACPR），2017：268-273.
[18] LUCEY P，COHN J F，KANADE T，et al.The extended Cohn-Kanade dataset（CK+）：a complete dataset for action unit and emotion-specified expression[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops，2010：94-101.
[19] GOODFELLOW I J，ERHAN D，CARRIER P L，et al.Challenges in representation learning：a report on three machine learning contests[J].Neural Networks，2013，64：59-63．
[20] BEN X，REN Y，ZHANG J，et al.Video-based facial micro-expression analysis：a survey of datasets，features and algorithms[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，44（9）：5826-5846.
[21] CHEN P，LIU S，ZHAO H，et al.Gridmask data augmentation[J].arXiv：2001.04086，2020.
[22] SELVARAJU R R，COGSWELL M，DAS A，et al.Grad-CAM：visual explanations from deep networks via gradient-based localization[J].International Journal of Computer Vision，2020，128（2）：336-359.
[23] GAN Y L，CHEN J Y，YANG Z K，et al.Multiple attention network for facial expression recognition[J].IEEE Access，2020，8：7383-7393.
[24] 张鹏，孔韦韦，滕金保.基于多尺度特征注意力机制的人脸表情识别[J].计算机工程与应用，2022，58（1）：182-189.
ZHANG P，KONG W W，TENG J B.Facial expression recognition based on multi-scale feature attention mechanism[J].Computer Engineering and Applications，2022，58（1）：182-189.
[25] 崔子越，皮家甜，陈勇，等.结合改进VGGNet和Focal Loss的人脸表情识别[J].计算机工程与应用，2021，57（19）：171-178.
CUI Z Y，PI J T，CHEN Y，et al.Facial expression recognition combined with improved VGGNet and focal loss[J].Computer Engineering and Applications，2021，57（19）：171-178.
[26] MINAEE S，MINAEI M，ABDOLRASHIDI A.Deep-emotion：facial expression recognition using attentional convolutional network[J].Sensors，2021，21（9）：3046.
[27] 陈加敏，徐杨.注意力金字塔卷积残差网络的表情识别[J].计算机工程与应用，2022，58（22）：123-131.
CHEN J M，XU Y.Expression recognition based on the convolution residual network of attention pyramid[J].Computer Engineering and Applications，2022，58（22）：123-131.