Dynamic Expression Recognition with Partial Occlusion Based on Parallel Gan

doi:10.3778/j.issn.1002-8331.2007-0130

Abstract

Abstract:

In order to reduce the influence of partial occlusion in dynamic Facial Expression Recognition（FER）, a method of dynamic FER with occlusion based on parallel Gan network is proposed. The P-IncepNet（Para Inception Network） constructed for context feature extraction is connected to the Gan, this connected net is trained for image repairing. The P-IncepNet is cascaded with LSTM to train a more robust dynamic FER network, which makes full use of the feature extraction of P-IncepNet and the Spatio-temporal information acquisition of the LSTM. The experimental results show that on CelebA and MMI datasets, the occlusion completion network is better than other networks for medium-sized occlusion. In the cascade expression recognition network, the average recognition rate of the repaired image is 4.45 percentage points higher than that of the unrepaired image. Especially with regard to anger, surprise and happy, 6.36 percentage points of the recognition rates are improved due to the repair of the occlusion. The experiments on AFEW and MMI databases show that the dynamic FER network is also superior to other networks in uncovered recognition, the average recognition is 51.12% and 80.31%. Therefore, the P-IncepNet is stable, and the performance of occlusion repair and expression recognition is significantly improved.

Key words: partial occlusion, dynamic facial expression recognition, deep learning, parallel processing, cascaded network, generative adversarial net

摘要：

为了解决实际中动态表情识别存在的局部遮挡问题，提出一种基于并行Gan网络的有遮挡动态表情识别方法。构建一个并联网络P-IncepNet进行上下文特征提取，利用条件对抗网络训练了一个处理不同程度遮挡的图像修复网络。将构建的并联网络与LSTM进行级联，充分利用并联网络的特征提取和LSTM的时空信息获取能力，训练得到一个更具鲁棒性的动态表情识别网络。实验结果表明，在CelebA和MMI数据集上训练的局部遮挡补全网络对中小程度遮挡的补全优于其他网络；构建的级联表情识别网络对于不同程度遮挡的识别结果显示，修复表情图的平均识别率比未修复表情图高4.45个百分点，尤其愤怒、惊讶、高兴有6.36个百分点的较大识别率提升得益于遮挡图像的修复；在AFEW和MMI数据集的无遮挡实验表明，该网络对无遮挡的识别同样具有优越性能，平均识别准确率达51.12%和80.31%。因此构建的P-IncepNet是稳定的，对图像的遮挡修复和表情识别性能均有明显改善。

关键词: 局部遮挡, 动态表情识别, 深度学习, 并行处理, 级联网络, 生成对抗网络

YANG Luyue, ZHANG Shumei, ZHAO Junli. Dynamic Expression Recognition with Partial Occlusion Based on Parallel Gan[J]. Computer Engineering and Applications, 2021, 57(24): 168-178.

杨鲁月，张树美，赵俊莉. 基于并行Gan的有遮挡动态表情识别[J]. 计算机工程与应用, 2021, 57(24): 168-178.

References

[1] MEHRABIAN A，RUSSELL J A.An approach to environmental psychology[M].[S.l.]：MIT Press，1974.
[2] 梁华刚，张志伟，王亚茹.自适应Gabor卷积核编码网络的表情识别方法[J].计算机工程与应用，2020，56（10）：149-156.
LIANG H G，ZHANG Z W，WANG Y R.Expression recognition method for adaptive Gabor convolution kernel coding network[J].Computer Engineering Applications，2020，56（10）：149-156.
[3] WANG K，PENG X，YANG J，et al.Region attention networks for pose and occlusion robust facial expression recognition[J].arXiv：1905.04075，2019.
[4] KUMAR J D，ZHANG Z，KAIQI H，et al.Multi angle optimal pattern based deep learning for automatic facial expression recognition[J].Pattern Recognition Letters，2019，10（2）：223-236.
[5] AVOTS E，SAPINSKI T，BACHMANN M，et al.Audiovisual emotion recognition in wild[J].Machine Vision and Applications，2019，30（5）：975-985.
[6] MOLLAHOSSEINI A，CHAN D，MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision，2016：1-10.
[7] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，NV United States，2016：770-778.
[8] 张建明，张晓翠.一种处理部分遮挡表情图像的方法[J].计算机工程与应用，2011，47（3）：170-173.
ZHANG J M，ZHANG X C.Processing method of facial expression images under partial occlusion[J].Computer Engineering and Applications，2011，47（3）：170-173.
[9] ZHANG L，TJONDRONEGORO D，CHANDRAN V.Random Gabor based templates for facial expression recognition in images with facial occlusion[J].Neurocomputing，2014，145（5）：451-464.
[10] SHEN J，CHAN T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics，2002，62：1019-1043.
[11] BARNES C，SHECHTMAN E，FINKELSTEIN A，et al.Patch match：a randomized correspondence algorithm for structural image editing[J].ACM Transactions on Graphics，2009，28（3）：1-11.
[12] ZHANG J，KAN M N，SHAN S G，et al.Occlusion-free face alignment：deep regression networks coupled with de-corrupt autoencoders[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Las Vegas，USA：IEEE，2016：3428-3437.
[13] 王海涌，梁红珠.基于改进的GAN的局部遮挡人脸表情识别[J].计算机工程与应用，2020，56（5）：141-146.
WANG H Y，LIANG H Z.Recognition of local occluded facial expressions based on improved generative adversarial network[J].Computer Engineering and Applications，2020，56（5）：141-146.
[14] LEE J，KIM S，KIM S，et al.Context-aware emotion recognition networks[C]//IEEE/CVF International Conference on Computer Vision（ICCV），2020.
[15] PATHAK D，KRAHENBUHL P，DONAHUE J，et al.Context encoders：feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2016.
[16] SUN M C，HSU S H，YANG M C，et al.Context-aware cascade attention-based RNN for video emotion recognition[C]//2018 First Asian Conference on Affective Computing and Intelligent Interaction（ACII Asia），2018：1-6.
[17] DENTON E，GROSS S，FERGUS R.Semi-supervised learning with context-conditional generative adversarial networks[J].arXiv：1805.12098，2018.
[18] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of IEEE Computer Vision and Pattern Recognition，2015：1-9.
[19] 李郅琴，杜建强，聂斌，等.特征选择方法综述[J].计算机工程与应用，2019，55（24）：10-19.
LI Z Q，DU J Q，NIE B，et al.Summary of feature selection methods[J].Computer Engineering and Applications，2019，55（24）：10-19.
[20] WU L，WANG Y，LI X，et al.Deep attention based spatially recursive networks for fine grained visual recognition[J].IEEE Transactions on Cybernetics，2018：1791-1802.
[21] ZHU X，LEI Z，YAN J，et al.High-fidelity pose and expression normalization for face recognition in the wild[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2015：787-796.
[22] 胡敏，张柯柯，王晓华，等.结合滑动窗口动态时间规整和CNN的视频人脸表情识别[J].中国图象图形学报，2018，23（8）：1144-1153.
HU M，ZHANG K K，WANG X H，et al.Video facial expression recognition combined with sliding window dynamic time warping and CNN[J].Journal of Image and Graphics，2018，23（8）：1144-1153.