New Research Advances in Facial Expression Recognition Under Partial Occlusion

doi:10.3778/j.issn.1002-8331.2112-0268

Abstract

Abstract: With the development of artificial intelligence technology, facial expression recognition can extract expression states from images or videos to recognize the mental emotions of human beings, achieving a better human-computer interaction effect. However, there are many methods only focusing on facial expression recognition without occlusion, which is not applicable to objective and complex scenarios and greatly limits the practicality of the algorithm. In recent years, aiming at different types of occlusion such as illumination occlusion, low resolution occlusion, pose variations, and physical occlusion, researchers have proposed various new methods to solve these problems. The survey introduces the core ideas of relevant methods, and conducts a comparison analysis about these methods, meanwhile looks forward to future research and development.

Key words: facial expression recognition, partial occlusion, new research advances

摘要： 随着人工智能技术的发展，人脸表情识别可以从图像或视频中抽取表情状态，识别对象心理情绪，从而达到更好的人机交互效果。然而多数方法只关注正面无遮挡的人脸表情识别，并不能适用于客观复杂的场景，极大地限制了算法的实用性。近几年，针对光照遮挡、噪声遮挡、姿态遮挡、实物遮挡等不同类型的遮挡，研究者们提出了各种新方法来挑战人脸部分遮挡下的表情识别，综述了这些方法的主要原理并进行了对比分析，同时对未来的研究和发展方向进行了展望。

关键词: 人脸表情识别, 部分遮挡, 新进展

JIANG Bin, LI Nanxing, ZHONG Rui, WU Qinggang, CHANG Huawen. New Research Advances in Facial Expression Recognition Under Partial Occlusion[J]. Computer Engineering and Applications, 2022, 58(12): 12-24.

蒋斌, 李南星, 钟瑞, 吴庆岗, 常化文. 人脸部分遮挡条件下表情识别研究的新进展[J]. 计算机工程与应用, 2022, 58(12): 12-24.

References

[1] SOLEYMANI M，PANTIC M，PUN T.Multimodal emotion recognition in response to videos[J].IEEE Transactions on Affective Computing，2011，3（2）：211-223.
[2] NOROOZI F，MARJANOVIC M，NJEGUS A，et al.Audio-visual emotion recognition in video clips[J].IEEE Transactions on Affective Computing，2017，10（1）：60-75.
[3] JEONG M，KO B C.Driver’s facial expression recognition in real-time for safe driving[J].Sensors，2018，18（12）：4270.
[4] LI Q，LIU X，GONG X，et al.INDReview on facial expression analysis and its application in education[C]//2019 Chinese Automation Congress（CAC），2019：4526-4530.
[5] WANG K，PENG X，YANG J，et al.Region attention networks for pose and occlusion robust facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4057-4069.
[6] 蒋斌，钟瑞，张秋闻，等.采用深度学习方法的非正面表情识别综述[J].计算机工程与应用，2021，57（8）：48-61.
JIANG B，ZHONG R，ZHANG Q W，et al.Survey of non-frontal facial expression recognition by using deep learning methods[J].Computer Engineering and Applications，2021，57（8）：48-61.
[7] KOTSIA I，BUCIU I，PITAS I.An analysis of facial expression recognition under partial facial image occlusion[J].Image and Vision Computing，2008，26（7）：1052-1067.
[8] 王锴.挑战条件下的人脸表情识别[D].深圳：中国科学院大学（中国科学院深圳先进技术研究院），2020.
WANG K.Challenge under the condition of facial expression recognition[D].Shenzhen：University of Chinese Academy of Sciences（Shenzhen Institutes of Advanced Technology，Chinese Academy of Sciences），2020.
[9] ABDULRAHMAN M，GWADABE T R，ABDU F J，et al.Gabor wavelet transform based facial expression recognition using PCA and LBP[C]//2014 22nd Signal Processing and Communications Applications Conference（SIU），2014：2265-2268.
[10] LUCEY P，COHN J F，KANADE T，et al.The extended cohn-kanade dataset（ck+）：a complete dataset for action unit and emotion-specified expression[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops，2010：94-101.
[11] CALVO M G，LUNDQVIST D.Facial expressions of emotion（KDEF）：identification under different display-duration conditions[J].Behavior Research Methods，2008，40（1）：109-115.
[12] ZHANG K，HUANG Y，DU Y，et al.Facial expression recognition based on deep evolutional spatial-temporal networks[J].IEEE Transactions on Image Processing，2017，26（9）：4193-4203.
[13] PANTIC M，VALSTAR M，RADEMAKER R，et al.Web-based database for facial expression analysis[C]//2005 IEEE International Conference on Multimedia and Expo，2005：5.
[14] LANGNER O，DOTSCH R，BIJLSTRA G，et al.Presentation and validation of the radboud faces database[J].Cognition and Emotion，2010，24（8）：1377-1388.
[15] GOODFELLOW I J，ERHAN D，CARRIER P L，et al.Challenges in representation learning：a report on three machine learning contests[C]//International Conference on Neural Information Processing.Berlin，Heidelberg：Springer，2013：117-124.
[16] YIN L，WEI X，SUN Y，et al.A 3D facial expression database for facial behavior research[C]//7th International Conference on Automatic Face and Gesture Recognition（FGR06），2006：211-216.
[17] LI S，DENG W，DU J P.Reliable crowd sourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2852-2861.
[18] DHALL A，GOECKE R，LUCEY S，et al.Static facial expression analysis in tough conditions：data，evaluation protocol and benchmark[C]//2011 IEEE International Conference on Computer Vision Workshops（ICCV Workshops），2011：2106-2112.
[19] MOLLAHOSSEINI A，HASANI B，MAHOOR M H.Affectnet：a database for facial expression，valence，and arousal computing in the wild[J].IEEE Transactions on Affective Computing，2017，10（1）：18-31.
[20] DHALL A，GOECKE R，LUCEY S，et al.Collecting large，richly annotated facial-expression databases from movies[J].IEEE Multimedia，2012，19（3）：34-41.
[21] DHALL A，RAMANA MURTHY O V，GOECKE R，et al.Video and image based emotion recognition challenges in the wild：EmotiW 2015[C]//Proceedings of the 2015 ACM on International Conference on Multimodal Interaction，2015：423-426.
[22] LI Y，ZENG J，SHAN S，et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing，2018，28（5）：2439-2450.
[23] FABIAN BENITEZ-QUIROZ C，SRINIVASAN R，MARTINEZ A M.Emotionet：an accurate，real-time algorithm for the automatic annotation of a million facial expressions in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：5562-5570.
[24] DHALL A，GOECKE R，LUCEY S，et al.Static facial expression analysis in tough conditions：data，evaluation protocol and benchmark[C]//2011 IEEE International Conference on Computer Vision Workshops（ICCV Workshops），2011：2106-2112.
[25] ZHANG F，ZHANG T，MAO Q，et al.Geometry guided pose-invariant facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4445-4460.
[26] 张继威，牛少彰，曹志义，等.基于深度学习和约束稀疏表达的人脸识别算法[J].北京理工大学学报，2019，39（3）：255-261.
ZHANG J W，NIU S Z，CAO Z Y，et al.Face recognition algorithm based on deep learning and constrained sparse representation[J].Journal of Beijing Institute of Technology，2019，39（3）：255-261.
[27] TOWNER H，SLATER M.Reconstruction and recognition of occluded facial expressions using PCA[C]//International Conference on Affective Computing and Intelligent Interaction.Berlin，Heidelberg：Springer，2007：36-47.
[28] 王慧星，黄勃，高永彬，等.部分遮挡人脸识别的方法综述[J].武汉大学学报（理学版），2020，66（5）：451-461.
WANG H X，HUANG B，GAO Y B，et al.A review of partially occluded face recognition methods[J].Journal of Wuhan University（Natural Science），2020，66（5）：451-461.
[29] YAN Y，ZHANG Z，CHEN S，et al.Low-resolution facial expression recognition：a filter learning perspective[J].Signal Processing，2020，169：107370.
[30] ZHANG Z，LAI C，LIU H，et al.Infrared facial expression recognition via Gaussian-based label distribution learning in the dark illumination environment for human emotion detection[J].Neurocomputing，2020，409：341-350.
[31] HU Y，ZENG X，HUANG Z，et al.A preprocessing method of facial expression image under different illumination[C]//2021 13th International Conference on Communication Software and Networks（ICCSN），2021：318-322.
[32] REDDY C V R，REDDY U S，KISHORE K V K.Facial emotion recognition using NLPCA and SVM[J].Traitement du Signal，2019，36（1）：13-22.
[33] SHI Y，LV Z，BI N，et al.An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations[J].Neural Computing and Applications，2020，32（13）：9267-9281.
[34] GUPTA O，RAVIV D，RASKAR R.Illumination invariants in deep video expression recognition[J].Pattern Recognition，2018，76：25-35.
[35] CUI Y，MA Y，LI W，et al.Multi-emonet：a novel multi-task neural network for driver emotion recognition[J].IFAC-Papers OnLine，2020，53（5）：650-655.
[36] KOLA D G R，SAMAYAMANTULA S K.A novel approach for facial expression recognition using local binary pattern with adaptive window[J].Multimedia Tools and Applications，2021，80（2）：2243-2262.
[37] LIU T，WANG J，YANG B，et al.Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom[J].Infrared Physics & Technology，2021，112：103594.
[38] SHAO J，CHENG Q.E-FCNN for tiny facial expression recognition[J].Applied Intelligence，2021，51（1）：549-559.
[39] ZHOU W，HONG C，WANG X，et al.FSRGAN-DB：super-resolution reconstruction based on facial prior knowledge[C]//2020 IEEE International Conference on Big Data，2020：3380-3386.
[40] 廖海斌，徐斌.基于性别和年龄因子分析的鲁棒性人脸表情识别[J].计算机研究与发展，2021，58（3）：528-538.
LIAO H B，XU B.Robust facial expression recognition based on gender and age factor analysis[J].Computer Research and Development，2021，58（3）：528-538.
[41] BELLAMKONDA S，GOPALAN N P.Facial expression recognition using kirsch edge detection，LBP and gabor wavelets[C]//2018 Second International Conference on Intelligent Computing and Control Systems（ICICCS），2018：1457-1461.
[42] HE Z，KAN M，ZHANG J，et al.A fully end-to-end cascaded cnn for facial landmark detection[C]//2017 12th IEEE International Conference on Automatic Face & Gesture Recognition（FG 2017），2017：200-207.
[43] WANG K，PENG X，YANG J，et al.Region attention networks for pose and occlusion robust facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4057-4069.
[44] DING H，ZHOU P，CHELLAPPA R.Occlusion-adaptive deep network for robust facial expression recognition[C]//2020 IEEE International Joint Conference on Biometrics（IJCB），2020：1-9.
[45] NAIK N，MEHTA M A.Hand-over-face gesture based facial emotion recognition using deep learning[C]//2018 International Conference on Circuits and Systems in Digital Enterprise Technology（ICCSDET），2018：1-7.
[46] LAI Y H，LAI S H.Emotion-preserving representation learning via generative adversarial network for multi-view facial expression recognition[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition（FG 2018），2018：263-270.
[47] WANG W，SUN Q，FU Y，et al.Comp-GAN：compositional generative adversarial network in synthesizing and recognizing facial expression[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：211-219.
[48] YANG L，TIAN Y，SONG Y，et al.A novel feature separation model exchange-GAN for facial expression recognition[J].Knowledge-Based Systems，2020，204：106217.
[49] ZHANG F，ZHANG T，MAO Q，et al.Geometry guided pose-invariant facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：4445-4460.
[50] 吕诲，童倩倩，袁志勇.基于人脸分割的复杂环境下表情识别实时框架[J].计算机工程与应用，2020，56（12）：134-140.
LV H，TONG Q Q，YUAN Z Y.Realtime architecture for facial expression recognitionin complex scenes based on face region segmentation[J].Computer Engineering and Applications，2020，56（12）：134-140.
[51] LI Y，ZENG J，SHAN S，et al.Patch-gated CNN for occlusion-aware facial expression recognition[C]//2018 24th International Conference on Pattern Recognition（ICPR），2018：2209-2214.
[52] LI Y，ZENG J，SHAN S，et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing，2018，28（5）：2439-2450.
[53] WANG C，XUE J，LU K，et al.Light attention embedding for facial expression recognition[J].IEEE Transactions on Circuits and Systems for Video Technology，2021.
[54] ZHAO Z，LIU Q，WANG S.Learning deep global multi-scale and local attention features for facial expression recognition in the wild[J].IEEE Transactions on Image Processing，2021，30：6544-6556.
[55] XIA Y，YU H，WANG X，et al.Relation-aware facial expression recognition[J].IEEE Transactions on Cognitive and Developmental Systems，2021.
[56] GEORGESCU M I，DU?ǎ G E，IONESCU R T.Teacher-student training and triplet loss to reduce the effect of drastic face occlusion[J].Machine Vision and Applications，2022，33（1）：1-19.
[57] 韦赛远，林丽媛，张怡然.基于M-Xception网络的戴口罩人脸表情识别[J].天津科技大学学报，2021，36（3）：72-76.
WEI S Y，LIN L Y，ZHANG Y R.Facial expression recognition for mask wearing based on M-Xception network[J].Journal of Tianjin University of Science and Technology，2021，36（3）：72-76.
[58] HOUSHMAND B，KHAN N M.Facial expression recognition under partial occlusion from virtual reality headsets based on transfer learning[C]//2020 IEEE Sixth International Conference on Multimedia Big Data（BigMM），2020：70-75.
[59] ZOU M，YOU M，AKASHI T.Reconstruction of partially occluded facial image for classification[J].IEEJ Transactions on Electrical and Electronic Engineering，2021，16（4）：600-608.
[60] HU K，HUANG G，YANG Y，et al.Rapid facial expression recognition under part occlusion based on symmetric SURF and heterogeneous soft partition network[J].Multimedia Tools and Applications，2020，79（41）：30861-30881.
[61] DONG J，ZHANG L，ZHANG H，et al.Occlusion-aware GAN for face de-occlusion in the wild[C]//2020 IEEE International Conference on Multimedia and Expo（ICME），2020：1-6.
[62] 杨鲁月，张树美，赵俊莉.基于并行Gan的有遮挡动态表情识别[J].计算机工程与应用，2021，57（24）：168-178.
YANG L Y，ZHANG S M，ZHAO J L.Dynamic expression recognition with partial occlusion based on parallel Gan[J].Computer Engineering and Applications，2021，57（24）：168-178.
[63] LU Y，WANG S，ZHAO W，et al.WGAN-based robust occluded facial expression recognition[J].IEEE Access，2019，7：93594-93610.
[64] 刘全明，辛阳阳.端到端的低质人脸图像表情识别[J].小型微型计算机系统，2020，41（3）：668-672.
LIU Q M，XIN Y Y.Face expression recognition based on end-to-end low-quality face images[J].Journal of Chinese Computer Systems，2020，41（3）：668-672.
[65] POUX D，ALLAERT B，IHADDADENE N，et al.Dynamic facial expression recognition under partial occlusion with optical flow reconstruction[J].IEEE Transactions on Image Processing，2021，31：446-457.
[66] ALCHALABI B，FAUBERT J，LABBé D R.A multi-modal modified feedback self-paced BCI to control the gait of an avatar[J].Journal of Neural Engineering，2021，18（5）：056005.
[67] 卢洋.人脸表情图像识别关键技术的分析与研究[D].长春：吉林大学，2019.
LU Y.Analysis and research on key technologies of facial expression image recognition[D].Changchun：Jilin University，2019.
[68] 王海涌，梁红珠.基于改进的GAN的局部遮挡人脸表情识别[J].计算机工程与应用，2020，56（5）：141-146.
WANG H Y，LIANG H Z.Recognition of local occluded facial expressions based on improved generative adversarial network[J].Computer Engineering and Applications，2020，56（5）：141-146.
[69] LEE J，KIM S，KIM S，et al.Multi-modal recurrent attention networks for facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：6977-6991.
[70] 林克正，白婧轩，李昊天，等.深度学习下融合不同模型的小样本表情识别[J].计算机科学与探索，2020，14（3）：482-492.
LIN K Z，BAI J X，LI H T，et al.Facial expression recognition with small samples fused with different models under deep learning[J].Journal of Frontiers of Computer Science and Technology，2020，14（3）：482-492.
[71] DU S，TAO Y，MARTINEZ A M.Compound facial expressions of emotion[J].Proceedings of the National Academy of Sciences，2014，111（15）：1454-1462.
[72] CHENG Y，WANG D，ZHOU P，et al.Model compression and acceleration for deep neural networks：the principles，progress，and challenges[J].IEEE Signal Processing Magazine，2018，35（1）：126-136.
[73] 李勇，曾加贝，刘昕，等.面部动作单元检测方法进展与挑战[J].中国图象图形学报，2020，25（11）：2293-2305.
LI Y，ZENG J B，LIU X，et al.Progress and challenges in facial action unit detection[J].Journal of Image and Graphics，2020，25（11）：2293-2305.
[74] PUMAROLA A，AGUDO A，MARTINEZ A M，et al.Ganimation：one-shot anatomically consistent facial animation[J].International Journal of Computer Vision，2020，128（3）：698-713.
[75] NOROOZI F，KAMINSKA D，CORNEANU C，et al.Survey on emotional body gesture recognition[J].arXiv：1801.
07481，2018.
[76] LIU P，YIN L.Spontaneous facial expression analysis based on temperature changes and head motions[C]//2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition（FG），2015：1-6.