Multi-Branch Network Facial Expression Recognition Based on Gender Constraint

doi:10.3778/j.issn.1002-8331.2202-0120

Abstract

Abstract: Aiming at the large intra-class variation and small inter-class differences of facial expressions under different genders, the thesis proposes a multi-branch network facial expression recognition based on gender constraints. Firstly, through the method of clustering algorithm K-means and convolutional neural network, the relationship between facial expression classes under gender constraints is obtained. Then, according to the obtained inter-class relationships, a backbone network and a branch network with a channel attention mechanism are constructed to further distinguish between strongly similar inter-class relationships and highlight intra-class changes in facial expressions of different genders. Finally, experiments and analysis are performed on CK+, FER2013 and RAF-DB datasets. Experiments show that the average recognition rate of the proposed network structure on the CK+, FER2013 and RAF-DB datasets is superior to other advanced methods, reaching 97.60%, 73.58% and 87.98%.

Key words: facial expression recognition, gender constraint, multi-branch network, attention mechanism

摘要： 针对不同性别下人脸表情类内变化大、类间差异小的问题，提出一种基于性别约束的多分支网络人脸表情识别方法。通过聚类算法K-means与卷积神经网络相结合的方法，得到性别约束下人脸表情类间关系。根据类间关系，构建主干网络和具有通道注意力机制的分支网络，进一步区分强相似的类间关系和突出不同性别人脸表情的类内变化。最后在CK+、FER2013和RAF-DB数据集上进行实验并分析。实验表明，提出的网络结构在CK+、FER2013和RAF-DB数据集上的平均识别率均优于其他先进方法，分别达到了97.60%、73.58%和87.98%。

关键词: 人脸表情识别, 性别约束, 多分支网络, 注意力机制

ZENG Xi, XIN Yuelan, XIE Qiqi. Multi-Branch Network Facial Expression Recognition Based on Gender Constraint[J]. Computer Engineering and Applications, 2023, 59(9): 245-254.

曾曦, 辛月兰, 谢琪琦. 基于性别约束的多分支网络人脸表情识别[J]. 计算机工程与应用, 2023, 59(9): 245-254.

References

[1] SCHROER M，BEVACQUA E，COWIE R，et al.Building autonomous sensitive artificial listeners[J].IEEE Transactions on Affective Computing，2012，3（2）：165-183.
[2] LUCEY P，COHN J F，MATTHEWS I，et al.Automatically detecting pain in video through facial action units[J].IEEE Transactions on Systems，Man，and Cybernetics（Cybernetics），2010，41（3）：664-674.
[3] KRIZHEVSKY A，SUTSKEVER I，HINTON G.ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012：45-55.
[4] SIMONYAN K，ZISSEMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.1556，2014.
[5] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，Jun 26-Jul 1，2016.Piscataway：IEEE，2016：770-778.
[6] SZEGEDY C，IOFFE S，VANHOUCKEV V，et al.Inception-v4：Inception-resNet and the impact of residual connections on learning[C]//National Conference on Artificial Intelligence，Phoenix，Feb 12-17，2016：4278-4284.
[7] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，Jun 18-23，2018：7132-7141.
[8] BARGAL S A，BARSOUM E，FERRER C C，et al.Emotion recognition in the wild from videos using images[C]//Proceedings of the ACM International Conference on Multimodal Interaction.Tokyo：ACM，Nov 12-16，2016：433-436.
[9] CAI J，MENG Z B，KHAN A S，et al.Island loss for learning discriminative features in facial expression recognition[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition（FG 2018），May 15-19，2018，Xi’an，China.New York：IEEE，2018：302-309.
[10] FARZANEH A H，QI X.Facial expression recognition in the wild via deep attentive center loss[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，Jan 5-9，2021：2402-2411.
[11] 赵家琦，周颖玥，王欣宇.采用支路辅助学习的人脸表情识别[J].计算机工程与应用，2022，58（23）：151-160.
ZHAO J Q，ZHOU Y Y，WANG X Y.Facial expression recognition method based on branch-assisted learning network[J].Computer Engineering and Applications，2022，58（23）：151-160.
[12] 蒋斌，钟瑞，张秋闻，等.采用深度学习方法的非正面表情识别综述[J].计算机工程与应用，2021，57（8）：48-61.
JIANG B，ZHONG R，ZHANG Q W，et al.Survey of non-frontal facial expression recognition by using deep learning methods[J].Computer Engineering and Applications，2021，57（8）：48-61.
[13] 廖海斌，徐斌.基于性别和年龄因子分析的鲁棒性人脸表情识别[J].计算机研究与发展，2021，58（3）：528-538.
LIAO H B，XU B.Robust face expression recognition based on gender and age factor analysis[J].Journal of Computer Research and Development，2021，58（3）：528-538.
[14] LI S，DENG W H.Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition[J].IEEE Transactions on Image Processing，2019，28（1）：356-370.
[15] 吴慧华，苏寒松，刘高华.基于余弦距离损失函数的人脸表情识别算法[J].激光与光电子学进展，2019，56（24）：196-202.
WU H H，SU H S，LIU G H.Facial expression recognition algorithm based on cosine distance loss function[J].Laser & Optoelectronics Progress，2019，56（24）：196-202.
[16] WANG K，PENG X，YANG J，et al.Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，Jun 16-20，2020：6897-6906.
[17] LUCEY P，COHN J F，KANADE T，et al.The extended cohn-kanade dataset（ck+）：a complete dataset for action unit and emotion-specifified expression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，San Francisco，Jun 13-18，2010：94-101.
[18] GOODFELLOW I J，ERHAN D，CARRIER P L，et al.Challenges in representation learning：a report on three machine learning contests[C]//International Conference on Neural Information Processing.Berlin，Heidelberg：Springer，2013：117-124.
[19] LI S，DENG W，DU J P.Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017.Washington：IEEE Computer Society，2017：2852-2861.
[20] LIU Z，LUO P，WANG X，et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision（ICCV），Santiago，Chile，Dec 13-16，2015：3730-3738.
[21] SZEGEDY C.Going deeper with convolutions[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Boston，MA，Jun 8-10，2015：1-9.
[22] CHOLLET F.Xception：deep learning with depthwise separable convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017：1800-1807.
[23] MA F，SUN B，LI S.Robust facial expression recognition with convolutional visual transformers[J].arXiv：2103.16854，2021.
[24] 吕诲，童倩倩，袁志勇.基于人脸分割的复杂环境下表情识别实时框架[J].计算机工程与应用，2020，56（12）：134-140.
LV H，TONG Q Q，YUAN Z Y.Realtime architecture for facial expression recognitionin complex scenes based on face region segmentation[J].Computer Engineering and Applications，2020，56（12）：134-140.
[25] 黄浩，葛洪伟.强化类间区分的深度残差表情识别网络[J].计算机科学与探索，2022，16（8）：1842-1849.
HUANG H，GE H W.Deep residual expression recognition network to enhance inter-class discrimination[J].Journal of Frontiers of Computer Science and Technology，2022，16（8）：1842-1849.
[26] 徐琳琳，张树美，赵俊莉.构建并行卷积神经网络的表情识别算法[J].中国图象图形学报，2019，24（2）：227-236.
XU L L，ZHANG S M，ZHAO J L.Expression recognition algorithm for parallel convolutional neural networks[J].Journal of Image and Graphics，2019，24（2）：227-236.
[27] MOLLAHOSSEINI A，DAVID C，MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision（WACV），NY，Mar 7-10，2016：1-10.
[28] 陈加敏，徐杨.注意力金字塔卷积残差网络的表情识别[J].计算机工程与应用，2022，58（22）：123-131.
CHEN J M，XU Y.Expression recognition based on convolution residual network of attention pyramid[J].Computer Engineering and Applications，2022，58（22）：123-131.
[29] 张鹏，孔韦韦，滕金保.基于多尺度特征注意力机制的人脸表情识别[J].计算机工程与应用，2022，58（1）：182-189.
ZHANG P，KONG W W，TENG J B.Facial expression recognition based on multi-scale feature attention mechanism[J].Computer Engineering and Applications，2022，58（1）：182-189.