Facial Expression Recognition Method with Attention-Free Capsule Network

doi:10.3778/j.issn.1002-8331.2206-0329

Abstract

Abstract: Expression recognition technology can analyze the emotional activities of recognized objects from human expressions. Aiming at the problem that an effective feature extraction and mapping model cannot be established due to the complex spatial relationship and feature information in facial expression images, the spare multilayer perceptron（sMLP） uses a small amount of parameters to allow each spatial location to perform, and the capsule network can also express the spatial pose information of the feature, so this paper proposes a new facial expression recognition model sMLP-CapsNet to improve the ability of expression recognition spatial relationship mapping. The CK+ dataset and the RAF-DB dataset are used to extract facial expression picture features from contour to detail by using an improved capsule neural network to achieve facial expression classification. Compared with other facial expression recognition algorithms, the accuracy of the model is significantly improved, and the recognition rates on the CK+ dataset and RAF-DB dataset can reach 99.48% and 85.69%, respectively, showing the advanced nature of the algorithm.

Key words: deep learning, capsule neural network, facial expression recognition, spare multilayer perceptron（sMLP）, sMLP-CapsNet

摘要： 表情识别技术可以从人类的表情中分析出识别对象的情感活动，针对面部表情图像复杂的空间关系和特征信息时，不能建立有效特征提取和映射模型的问题，稀疏多层感知机（spare multilayer perceptron，sMLP）使用很少的参数量让每个空间位置进行交流，而胶囊网络也可以表现特征的空间姿态信息，因此提出了一种新的面部表情识别模型sMLP-CapsNet，以提升表情识别空间关系映射的能力。采用CK+数据集和RAF-DB数据集，通过改进的胶囊神经网络从轮廓到细节提取面部表情图片特征，进而实现面部表情分类。相比于其他面部表情识别算法，模型精度提升效果明显，在CK+数据集和RAF-DB数据集上分别可达到99.48%以及85.69%的识别率，展现了该算法的先进性。

关键词: 深度学习, 胶囊神经网络, 面部表情识别, 稀疏多层感知机（sMLP）, sMLP-CapsNet

XU Xuebin, LIU Chenguang, LU Longbin, CAO Shuxin, XU Zongyu. Facial Expression Recognition Method with Attention-Free Capsule Network[J]. Computer Engineering and Applications, 2023, 59(22): 251-258.

许学斌, 刘晨光, 路龙宾, 曹淑欣, 徐宗瑜. 无注意力胶囊网络的面部表情识别方法[J]. 计算机工程与应用, 2023, 59(22): 251-258.

References

[1] HE J，BAXTER S L，XU J，et al.The practical implementation of artificial intelligence technologies in medicine[J].Nature Medicine，2019，25（1）：30-36.
[2] TIAN Y，KANADE T，COHN J F.Facial expression recognition，and handbook of face recognition[M].London：Springer，2011：487-519.
[3] 李瑞新，蔡兆信，王冰冰，等.基于人脸图像和脑电的连续情绪识别方法[J].计算机系统应用，2021，30（2）：1-11.
LI R X，CAI Z X，WANG B B，et al.Continuous emotion recognition method based on face image and EEG[J].Computer Systems & Applications，2021，30（2）：1-11.
[4] EKMAN P，FRIESEN W V.Constants across cultures in the face and emotion[J].Journal of Personality and Social Psychology，1971，17（2）：124-129.
[5] MATSUMOTO D.More evidence for the universality of a contempt expression[J].Motivation and Emotion，1992，16（4）：363-368.
[6] NG P C，HENIKOFF S.SIFT：predicting amino acid changes that affect protein function[J].Nucleic Acids Research，2003，31（13）：3812-3814.
[7] BAY H，TUYTELAARS T，GOOL L V.SURF：speeded up robust features[C]//Proceedings of the 9th European Conference on Computer Vision.Berlin，Heidelberg：Springer，2006：404-417.
[8] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2005：886-893.
[9] OJALA T，PIETIKAINEN M，MAENPAA T.Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2002，24（7）：971-987.
[10] LIU K，ZHANG M，PAN Z.Facial expression recognition with CNN ensemble[C]//Proceedings of the 2016 International Conference on Cyberworlds，2016：163-166.
[11] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-
Net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems 25，Lake Tahoe，Dec 3-6，2012：1106-1114.
[12] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[13] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[14] MOLLAHOSSEINI A，CHAN D，MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision，2016：1-10.
[15] CHEN J，LV Y，XU R，et al.Automatic social signal analysis：facial expression recognition using difference convolution neural network[J].Journal of Parallel and Distributed Computing，2019，131：97-102.
[16] ZHANG H，HUANG B，TIAN G.Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture[J].Pattern Recognition Letters，2020，131：128-134.
[17] CHANG T，LI H，WEN G，et al.Facial expression recognition sensing the complexity of testing samples[J].Applied Intelligence，2019，49（12）：4319-4334.
[18] KRITHIKA L B，PRIYA G G.Graph based feature extraction and hybrid classification approach for facial expression recognition[J].Journal of Ambient Intelligence and Humanized Computing，2021，12（2）：2131-2147.
[19] PHAM L，VU T H，TRAN T A.Facial expression recognition using residual masking network[C]//Proceedings of the 2020 25th International Conference on Pattern Recognition，2021：4513-4519.
[20] SABOUR S，FROSST N，HINTON G E.Dynamic routing between capsules[C]//Advances in Neural Information Processing Systems 30，2017.
[21] TANG C，ZHAO Y，WANG G，et al.Sparse MLP for image recognition：is self-attention really necessary?[J].arXiv：2109.05422，2021.
[22] ZHANG Y，WANG C，DENG W.Relative uncertainty learning for facial expression recognition[C]//Advances in Neural Information Processing Systems 34，2021：17616-17627.
[23] ZHANG Y H，WANG C R.Learn from all：erasing attention consistency for noisy label facial expression recognition[C]//Proceedings of the 2022 IEEE Conference on Computer Vision and Pattern Recognition，2022.
[24] XIE S，HU H.Facial expression recognition with FRR-CNN[J].Electronics Letters，2017，53（4）：235-237.
[25] LIU M，LI S，SHAN S，et al.Deeply learning deformable facial action parts model for dynamic expression analysis[C]//Proceedings of the 2014 Asian Conference on Computer Vision.Cham：Springer，2014：143-157.
[26] HASANI B，MAHOOR M H.Spatio-temporal facial expression recognition using convolutional neural networks and conditional random fields[C]//Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition，2017：790-795.
[27] HASANI B，MAHOOR M H.Facial expression recognition using enhanced deep 3D convolutional neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：30-40.
[28] HU M，WANG H，WANG X，et al.Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks[J].Journal of Visual Communication and Image Representation，2019，59：176-185.
[29] LIU C，HIROTA K，MA J，et al.Facial expression recognition using hybrid features of pixel and geometry[J].IEEE Access，2021，9：18876-18889.
[30] ZHANG T，ZHENG W，CUI Z，et al.Spatial-temporal recurrent neural network for emotion recognition[J].IEEE Transactions on Cybernetics，2018，49（3）：839-847.
[31] RAVI R，YADHUKRISHNA S V.A face expression recognition using CNN & LBP[C]//Proceedings of the 2020 4th International Conference on Computing Methodologies and Communication，2020：684-689.
[32] MINAEE S，MINAEI M，ABDOLRASHIDI A.Deep-Emotion：facial expression recognition using attentional convolutional network[J].Sensors，2021，21（9）：3046.
[33] LI S，LI W，WEN S，et al.Auto-FERNet：a facial expression recognition network with architecture search[J].IEEE Transactions on Network Science and Engineering，2021，8（3）：2213-2222.
[34] JEONG D，KIM B G，DONG S Y.Deep joint spatiotemporal network （DJSTN） for efficient facial expression recognition[J].Sensors，2020，20（7）：1936.
[35] LIU Z，LI S，DENG W.Real-world facial expression recognition using metric learning method[C]//Proceedings of the 2016 Chinese Conference on Biometric Recognition.Cham：Springer，2016：519-527.
[36] LIU Z，LI S，DENG W.Boosting-POOF：boosting part based one vs one feature for facial expression recognition in the wild[C]//Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition，2017：967-972.
[37] FAN Y，LAM J C K，LI V O K.Multi-region ensemble convolutional neural network for facial expression recognition[C]//Proceedings of the 27th International Conference on Artificial Neural Networks.Cham：Springer，2018：84-94.
[38] VIELZEUF V，KERVADEC C，PATEUX S，et al.An Occam’s Razor view on learning audiovisual emotion recognition with small training sets[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction，2018：589-593.
[39] ZHAO S，CAI H，LIU H，et al.Feature selection mechanism in CNNs for facial expression recognition[C]//Proceedings of the British Machine Vision Conference 2018，2018：317.
[40] YU J，LIN Z，YANG J，et al.Generative image inpainting with contextual attention[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：5505-5514.
[41] LI S，DENG W，DU J P.Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：2852-2861.
[42] LI Y，ZENG J，SHAN S，et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing，2018，28（5）：2439-2450.
[43] ZHOU Y，JIN L，MA G，et al.Quaternion capsule neural network with region attention for facial expression recognition in color images[J].IEEE Transactions on Emerging Topics in Computational Intelligence，2022，6（4）：893-912.
[44] CAO S，YAO Y，AN G.E2‐capsule neural networks for facial expression recognition using AU‐aware attention[J].IET Image Processing，2020，14（11）：2417-2424.