Expression Recognition and Interaction of Pharyngeal Swab Collection Robot

doi:10.3778/j.issn.1002-8331.2203-0479

Abstract

Abstract: The recognition of the facial expression of the collected person of pharyngeal swabs, and the interactive control of the human emotional expression have important significance for the successful collection of pharyngeal swabs of the pharyngeal swab collection robot. When training the data sampled by the throat swab robot, there are some problems, such as the expansion of occlusion removal and micro-expression recognition, the definition of self-collected data sets, key frame recognition and processing, expression classification and delimitation, and interactive integration. To solve these problems, it proposes an effective framework named efficient self-cured network（ESCN）. First, a weighted feature extraction model with the ability to recognize faces is constructed. Second, the subjects are captured by a multi-scale attention mechanism. Finally, the facial expressions of throat swab collectors are identified by a relabel-corrected linear aggregation model. Experiments are validated on real datasets RAF-DB, FER2013 and self-collected datasets. The experimental results show that the accuracy of ESCN is 4.643~11.058 percentage points higher than the classic model. The parameter amount is relatively small, which is convenient for lightweight and integration.

Key words: pharyngeal swab collection, facial expression recognition, feature extraction, perceptual interaction

摘要： 咽拭子采集机器人对采集人的表情识别、情绪表达等感知交互，对于咽拭子采集机器人成功执行任务具有重要的意义。咽拭子机器人采集到的数据样本在训练过程中要解决遮挡去除及微表情识别的拓展、自采集数据集的定义、关键帧识别及处理、表情分类及划定、交互集成等关键问题。针对以上问题，设计了一种高效自修复网络（efficient self-cured network，ESCN），构建了具有类识别人脸能力的特征提取加权模型，通过多尺度注意力机制捕捉受试者的面部表情区，通过重标签校正的线性聚合模型识别咽拭子采集者的面部表情。实验在真实数据集RAF-DB、FER2013和自采集数据集上进行验证。实验结果表明，ESCN比经典模型的准确率提升4.643~11.058个百分点，同时参数量相对较小，便于轻量化和集成。

关键词: 咽拭子采集, 表情识别, 特征提取, 感知交互

GUO Xinwei, MA Nan, LIU Weifeng, SUN Fuchun, ZHANG Jinli, CHEN Yang, ZHANG Guoping. Expression Recognition and Interaction of Pharyngeal Swab Collection Robot[J]. Computer Engineering and Applications, 2022, 58(8): 125-135.

郭馨蔚, 马楠, 刘伟锋, 孙富春, 张津丽, 陈洋, 张国平. 咽拭子采集机器人表情识别与交互[J]. 计算机工程与应用, 2022, 58(8): 125-135.

References

[1] ZEKHNINE C，BERRACHED N E.Human-robots interaction by facial expression recognition[J].International Journal of Engineering Research in Africa，2020，46：76-87.
[2] XIA H Y，LI C，TAN Y，et al.Destruction and reconstruction learning for facial expression recognition[J].IEEE Multimedia，2021，28（2）：20-28.
[3] EKMAN P，FRIESEN W V.Facial action coding system（FACS）：a technique for the measurement of facial actions[J].Rivista Di Psichiatria，1978，47（2）：126-138.
[4] AHMED H，BAJWA U I，ZHANG F，et al.Deception detection in videos using the facial action coding system[J].arXiv：2105.13659，2021.
[5] WANG Y，LI M，ZHANG C，et al.Weighted-fusion feature of MB-LBPUH and HOG for facial expression recognition[J].Soft Computing，2020，24（3）：5859-5875.
[6] LI J，ZHANG D，ZHANG J，et al.Facial expression recognition with Faster R-CNN[J].Procedia Computer Science，2017，107：135-140.
[7] WANG Y，LI M，WAN X，et al.Multiparameter space decision voting and fusion features for facial expression recognition[J].Computational Intelligence and Neuroscience，2020，2：1-17.
[8] FAN X，TJAHJADI T.A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences[J].Pattern Recognition，2015，48（11）：3407-3416.
[9] MA N，LI D Y，HE W，et al.Future vehicles：interactive wheeled robots[J].SCIENCE CHINA Information Sciences，2021，64（5）：56101.
[10] DIYASA G S M，FAUZI A，IDHOM M，et al.Multi-face recognition for the detection of prisoners in jail using a modified cascade classifier and CNN[J].Journal of Physics：Conference Series，2021，1844（1）：012005.
[11] QUANG N V，CHUN J，TAKESHI T.CapsuleNet for micro-expression recognition[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition，2019.
[12] FU Y，WU X，LI X，et al.Semantic neighborhood-aware deep facial expression recognition[J].IEEE Transactions on Image Processing，2020，29：6535-6548.
[13] FAN Y，LI V，LAM J.Facial expression recognition with deeply-supervised attention network[J].IEEE Transactions on Affective Computing，2020，99.
[14] MOLLAHOSSEINI A，CHAN D，MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//2016 IEEE Winter Conference on Applications of Computer Vision（WACV），2016.
[15] BARSOUM E，ZHANG C，FERRER C C，et al.Training deep networks for facial expression recognition with crowd-sourced label distribution[C]//The 18th ACM International Conference on Multimodal Interaction，2016.
[16] LIU S，LONG Y，XU G，et al.An optimized Capsule-LSTM model for facial expression recognition with video sequences[J].arXiv：2106.07564，2021.
[17] VALSTAR M F，SáNCHEZ-LOZANO，COHN J F，et al.FERA 2017-addressing head pose in the third facial expression recognition and analysis challenge[C]//IEEE International Conference on Automatic Face & Gesture Recognition，2017.
[18] MADHAVAN S，KUMAR N.Incremental methods in face recognition：a survey[J].Artificial Intelligence Review，2021，54：253-303.
[19] JAIN D K，ZHANG Z，HUANG K.Multi angle optimal pattern-based deep learning for automatic facial expression recognition-ScienceDirect[J].Pattern Recognition Letters，2020，139：157-165.
[20] DU H，SHI H，LIU Y，et al.Towards NIRVIS masked face recognition[J].IEEE Signal Processing Letters，2021，28：768-772.
[21] LIU Y P，LI Y B，MA X，et al.Facial expression recognition with fusion features extracted from salient facial areas[J].Sensors，2017，4：712-730.
[22] LEE S H，PLATANIOTIS K N K，YONG M R.Intra-class variation reduction using training expression images for sparse representation based facial expression recognition[J].IEEE Transactions on Affective Computing，2017，5（3）：340-351.
[23] ZHAO G，PIETIKAINEN M.Dynamic texture recognition using local binary patterns with an application to facial expressions[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2007，29：915-928.
[24] PERVEEN N，ROY D，MOHAN C K.Facial expression recognition in videos using dynamic kernels[J].IEEE Transactions on Image Processing，2020，29：8316-8325.
[25] AAMIR M，ALI T，SHAF A，et al.ML-DCNNet：multi-level deep convolutional neural network for facial expression recognition and intensity estimation[J].Arabian Journal for Science and Engineering，2020，45（12）：10605-10620.
[26] CORNEANU C，A，SIMóN M O，COHN J F，et al.Survey on RGB，3D，thermal，and multimodal approaches for facial expression recognition：history，trends，and affect-related applications.[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2016，38（8）：1548-1568.
[27] LIU Y J，ZHANG J K，YAN W J，et al.A main directional mean optical flow feature for spontaneous micro-expression recognition[J].IEEE Transactions on Affective Computing，2016，7（4）：299-310.
[28] ZHANG C，SONG J，ZHANG Q，et al.Deep adaptive attention for joint facial action unit detection and face alignment[J].arxiv：1803.05588，2018.
[29] SHE J，HU Y，SHI H，et al.Dive into ambiguity：latent distribution mining and pairwise uncertainty estimation for facial expression recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2021：6248-6257.
[30] CHEN Y，JOO J.Understanding and mitigating annotation bias in facial expression recognition[C]//2021 IEEE International Conference on Computer Vision（ICCV），2021.
[31] LI Y，ZENG J，SHAN S，et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing，2019，28（5）：2439-2450.
[32] SAJJANHAR A，WU Z Q，WEN Q.Deep learning models for facial expression recognition[C]//2018 Digital Image Computing：Techniques and Applications（DICTA），2018：1-6.
[33] ZHANG Y H，WANG C R，DENG W H.Relative uncertainty learning for facial expression recognition[C]//35th Conference on Neural Information Processing Systems（NeurIPS），2021.
[34] WANG K，PENG X，YANG J，et al.Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2020：6897-6906.