深度度量注意力混合模型表情识别方法

doi:10.3778/j.issn.1002-8331.2311-0323

摘要/Abstract

摘要： 深度学习网络在人脸表情识别中已广泛采用，但因表情图像复杂多变，受光照、个体差异等各个因素的影响，现有方法的识别效果有待提高。为了提高深度学习网络的表达能力，在深度学习网络中，结合面部关键区域的位置特征，提出融合位置信息的深层注意力反馈机制卷积神经网络模型。同时，由于表情特征的类间差异小，为了提高分类器的分类学习能力，引入度量学习方法增强特征的判别性，使同类之间的距离减小，异类之间的距离加大。通过度量学习将面部表情图像的特征映射到具有表情判别性的新的特征空间中，由此判断各表情样本的表情类别。对原图进行人脸检测，确定人脸裁剪出人脸关键区域，去除头发、背景等因素的干扰；通过深层注意力反馈机制的CNN模型对人脸关键区域进行特征学习，学习获得面部表情深度特征，之后引入判别性度量学习方法，通过度量矩阵将特征向量映射为新的学习后的特征向量；将提取的样本表情特征送入全连接层并通过Softmax分类器识别划分到预先定义好的7种基本表情。在CK+和RAF-DB数据库的实验表明，该方法取得了98.69%和87.68%的平均识别率，提高了分类器的分类学习能力。

关键词: 深度注意力, 表情识别, 卷积神经网络, 度量学习

Abstract: Deep learning network has been widely used in facial expression recognition, but the recognition effect of existing methods needs to be improved due to the complexity and variety of expression images and the influence of various factors such as illumination and individual differences. In order to improve the expressive ability of deep learning network, a deep attention feedback mechanism convolutional neural network (CNN) model is proposed in the deep learning network, which combines the location features of key facial areas. At the same time, due to the small differences between different classes of expression features, in order to improve the classification learning ability of the classifier, metric learning method is introduced to enhance the discrimination of features, which reduces the distance between the same kind and increases the distance between different kinds. Through metric learning, the features of facial expression images are mapped into a new feature space with discriminative expression, so as to judge the expression categories of each expression sample. Firstly, face detection is carried out on the original image, and the key areas of human face are determined, and the interference of hair, background and other factors is removed. Secondly, the CNN model with deep attention feedback mechanism is used to learn the features of key facial areas, and the facial expression depth features are obtained by learning. Then, the discriminant metric learning method is introduced, and the feature vectors are mapped into new learned feature vectors by metric matrix. Finally, the extracted sample expression features are sent to the full connection layer, and identified by Softmax classifier and divided into seven predefined basic expressions. Experiments on CK+ and RAF-DB databases show that the average recognition rate of this method is 98.69% and 87.68%. The classification learning ability of the classifier is improved.

Key words: depth attention, expression recognition, convolutional neural network, metric learning

姚丽莎. 深度度量注意力混合模型表情识别方法[J]. 计算机工程与应用, 2025, 61(7): 245-254.

YAO Lisha. Facial Expression Recognition Method Based on Mixed Model of Measure Depth Attention[J]. Computer Engineering and Applications, 2025, 61(7): 245-254.

参考文献

[1] 王秀友, 范建中, 刘华明, 等. 基于概率协作表示的多表情序列融合识别[J]. 激光与光电子学进展, 2019, 56(13): 79-85.
WANG X Y, FAN J Z, LIU H M, et al. Multi-expression sequence fusion recognition based on probabilistic cooperative representation[J]. Laser & Optoelectronics Progress, 2019, 56(13): 79-85.
[2] ACEVEDO D, NEGRI P, BUEMI M E, et al. A simple geometric-based descriptor for facial expression recognition[C]//Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition. Piscataway: IEEE, 2017: 802-808.
[3] LIU Y P, LI Y B, MA X, et al. Facial expression recognition with fusion features extracted from salient facial areas[J]. Sensors, 2017, 17(4): 712.
[4] 叶珍, 白璘, 粘永健. 基于Gabor特征与局部保护降维的高光谱图像分类算法[J]. 光学学报, 2016, 36(10): 502-511.
YE Z, BAI L, NIAN Y J. Hyperspectral image classification algorithm based on Gabor feature and locality-preserving dimensionality reduction[J]. Acta Optica Sinica, 2016, 36(10): 502-511.
[5] YU J, WANG Z F. A video-based facial motion tracking and expression recognition system[J]. Multimedia Tools and Applications, 2017, 76(13): 14653-14672.
[6] 翟懿奎, 刘健. 面向人脸表情识别的迁移卷积神经网络研究[J]. 信号处理, 2018, 34(6): 729-738.
ZHAI Y K, LIU J. Facial expression recognition based on transferring convolutional neural network[J]. Journal of Signal Processing, 2018, 34(6): 729-738.
[7] 李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1): 176-182.
LI Y, LIN X Z, JIANG M Y. Facial expression recognition with cross-connect LeNet-5 network[J]. Acta Automatica Sinica, 2018, 44(1): 176-182.
[8] 何志超, 赵龙章, 陈闯. 用于人脸表情识别的多分辨率特征融合卷积神经网络[J]. 激光与光电子学进展, 2018, 55(7): 370-375.
HE Z C, ZHAO L Z, CHEN C. Convolution neural network with multi-resolution feature fusion for facial expression recognition[J]. Laser & Optoelectronics Progress, 2018, 55(7): 370-375.
[9] ZHAO X Y, LIANG X D, LIU L Q, et al. Peak-piloted deep network for facial expression recognition[C]//Proceedings of the European Conference on Computer Vision. Cham: Springer, 2016: 425-442.
[10] LIANG X C, XU L S, ZHANG W X, et al. A convolution-transformer dual branch network for head-pose and occlusion facial expression recognition[J]. The Visual Computer, 2023, 39(6): 2277-2290.
[11] POURMIRZAEI M, MONTAZER G A, ESMAILI F. Using self-supervised auxiliary tasks to improve fine-grained facial representation[J]. arXiv:2105.06421, 2021.
[12] VO T H, LEE G S, YANG H J, et al. Pyramid with super resolution for in-the-wild facial expression recognition[J]. IEEE Access, 2020, 8: 131988-132001.
[13] LIANG D D, LIANG H G, YU Z B, et al. Deep convolutional BiLSTM fusion network for facial expression recognition[J]. The Visual Computer, 2020, 36(3): 499-508.
[14] AOUAYEB M, HAMIDOUCHE W, SOLADIE C, et al. Learning vision transformer with squeeze and excitation for facial expression recognition[J]. arXiv:2107.03107, 2021.
[15] FARD A P, MAHOOR M H. Ad-corre: adaptive correlation-based loss for facial expression recognition in the wild[J]. IEEE Access, 2022, 10: 26756-26768.
[16] YAO L S, ZHAO H F. Deep learning method of facial expression recognition based on Gabor filter bank combined with PCNN[J]. Wireless Personal Communications, 2023, 131(2): 955-971.
[17] 邓志平, 王冬霞, 马晓冬, 等. 基于注意力机制的NewVGG16-BiGRU鼾声分类[J]. 计算机应用, 2023, 43(S1): 276-280.
DENG Z P, WANG D X, MA X D, et al. NewVGG16-BiGRU snoring classification based on attention mechanism[J]. Journal of Computer Applications, 2023, 43(S1): 276-280.
[18] ALKURDY N H, ALJOBOURI H K, WADI Z K. Ultrasound renal stone diagnosis based on convolutional neural network and VGG16 features[J]. International Journal of Electrical and Computer Engineering, 2023, 13(3): 3440-3448.
[19] 朱敏, 毛莺池, 程永, 等. 基于双重注意力机制的事件抽取方法[J]. 软件学报, 2023, 34(7): 3226-3240.
ZHU M, MAO Y C, CHENG Y, et al. Event extraction method based on dual attention mechanism[J]. Journal of Software, 2023, 34(7): 3226-3240.
[20] 庞俊, 徐浩, 秦宏超, 等. 联合注意力与卷积网络的知识超图链接预测[J]. 计算机科学与探索, 2023, 17(11): 2734-2742.
PANG J, XU H, QIN H C, et al. Link prediction in knowledge hypergraph combining attention and convolution network[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(11): 2734-2742.
[21] KANADE T, COHN J F, TIAN Y L. Comprehensive database for facial expression analysis[C]//Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway: IEEE, 2000: 46-53.
[22] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset(CK+): a complete expression dataset for action unit and emotion-specified expression[C]//Proceedigs of the 3rd International Workshop on CVPR for Human Communicative Behavior Analysis, San Francisco, USA, 2010: 94-101.