Review of Deep Learning Methods for Lip Recognition

doi:10.3778/j.issn.1002-8331.2109-0196

Abstract

Abstract:

With the continuous development of deep learning, significant progress has been made in the field of lip recognition, and many deep learning algorithms for lip recognition have emerged. According to the continuity of the recognition object, this paper divides lip recognition into isolated lip recognition and continuous lip recognition, analyzes and summarizes the deep learning methods of each recognition task in detail and in-depth. Firstly, it introduces the mainstream lip recognition methods from the two aspects of the deep learning method of isolated lip recognition and the deep method of continuous lip recognition, and compares the advantages, disadvantages and performance of each method. Secondly, the characteristics and performance of representative methods for different data sets are compared, and the advantages, disadvantages and application scope of the two methods are expounded. Finally, the problems and challenges of lip recognition methods are discussed, and the research trend of lip recognition methods is prospected.

Key words: lip recognition, deep learning, convolutional neural networks, attentional mechanisms

摘要：

随着深度学习的不断发展，唇语识别领域的研究取得了重大进展，涌现了许多唇语识别的深度学习算法。依据识别对象的连续性，将唇语识别分为孤立唇语识别和连续唇语识别，并对各识别任务的深度学习方法进行了详细和深入的分析总结。从孤立唇语识别的深度学习方法和连续唇语识别的深度方法两个方面介绍了主流唇语识别方法，并对各方法的优缺点和性能进行比较；对不同数据集下代表性方法的特点和性能进行比较，对两类方法的优缺点和适用范围进行阐述；讨论了唇语识别方法存在的问题和挑战，并对唇语识别方法的研究趋势进行了展望。

关键词: 唇语识别, 深度学习, 卷积神经网络, 注意力机制

MA Jinlin, ZHU Yanbin, MA Ziping, GONG Yuanwen, CHEN Deguang, LIU Yuhao. Review of Deep Learning Methods for Lip Recognition[J]. Computer Engineering and Applications, 2021, 57(24): 61-73.

马金林，朱艳彬，马自萍，巩元文，陈德光，刘宇灏. 唇语识别的深度学习方法综述[J]. 计算机工程与应用, 2021, 57(24): 61-73.

References

[1] SUMBY W H，POLL ACK I.Visual contribution to speech intelligibility in noise[J].Journal of the Acoustical Society of America，1954，26（2）：212-215.
[2] PETAJAN E D.Automatic lipreading to enhance speech recognition（speech reading）[D].University of Illinois at Urbana-Champaign，1984.
[3] TAMURA S，NINOMIYA H，KITAOKA N，et al.Audio-visual speech recognition using deep bottleneck features and high-performance lipreading[C]//Proceedings of the 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference，HK，China，Dec 16-19，2015.New York：IEEE，2015：575-582.
[4] WATANABE T，KATSURADA K，KANAZAWA Y，et al.Lip reading from multi view facial images using 3D-AAM[C]//Proceedings of the 13th Asian Conference on Computer Vision，Taipei，Taiwan，Nov 20-24，2016.Berlin，Cham：Springer，2016：303-316.
[5] BAART M，SAMUEL A G.Turning a blind eye to the lexicon：ERPs show no cross-talk between lip-read and lexical context during speech sound processing[J].Journal of Memory and Language，2015，85：42-59.
[6] LESANI F S，GHAZVINI F F，DIANAT R，et al.Mobile phone security using automatic lip reading[C]//Proceedings of the 9th International Conference on e-Commerce in Developing Countries：With Focus on e-Business，Isfahan，Iran，April 16，2015.New York：IEEE，2015：1-5.
[7] MATHULAPRANGSAN S，WANG C Y，USUM A Z，et al.A survey of visual lip reading and lip-password verification[C]//Proceedings of the 2015 International Conference on Orange Technologies，Hong Kong，China，Dec 19-22，2015.New York：IEEE，2015：22-25.
[8] BAHDANAU D，CHOROWSKI J，SEERDYUK D，et al.End-to-end attention-based large vocabulary speech recognition[C]//Proceedings of the 2016 IEEE International Conference on Acoustics，Speech and Signal Processing，Shanghai，China，March 20-25，2016.New York：IEEE，2016：4945-4949.
[9] HUANG J T，LI J，GONG Y，et al.An analysis of convolutional neural networks for speech recognition[C]//Proceedings of the 2015 IEEE International Conference on Acoustics，Speech and Signal Processing，South Brisbane，QLD，Australia，April 19-24，2015.New York：IEEE，2015：4989-4993.
[10] CHAE H，KANG C M，KIM B，et al.Autonomous braking system via deep reinforcement learning[C]//Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems，Yokohama，Japan，Oct 16-19，2017.New York：IEEE，2017：1-6.
[11] PUJARI S，SNEHA S，VINUSHA R，et al.A survey on deep learning based lip-reading techniques[C]//Proceedings of the Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks，Tirunelveli，India，Feb 4-6，2021.New York：IEEE，2021：1286-1293.
[12] ZHANG Z，QU W，LIU F et al.Review of the lip-reading recognition[C]//Proceedings of the 2014 IEEE 5th International Conference on Software Engineering and Service Science，Beijing，China，June 27-29，2014.New York：IEEE，2014：593-596.
[13] 马金林，陈德光，郭贝贝，等.唇语语料库综述[J].计算机工程与应用，2019，55（22）：1-13.
MA J L，CHEN D G，GUO B B，et al.Lip corpus review[J].Computer Engineering and Applications，2019，55（22）：1-13.
[14] 陈小鼎，盛常冲，匡纲要，等.唇读研究进展与展望[J].自动化学报，2020，46（11）：2275-2301.
CHEN X D，SHENG C C，KUANG G Y，et al.The state of the art and prospects of lip reading[J].Acta Automatica Sinica，2020，46（11）：2275-2301.
[15] PALECEK K.Utilizing lipreading in large vocabulary continuous speech recognition[C]//Proceedings of the International Conference on Speech and Computer，Hatfield，UK，Sep 12-16，2017.Berlin，Cham：Springer，2017：767-776.
[16] MCGURK H，MACDONALD J.Hearing lips and seeing voices[J].Nature，1976，264（5588）：746-748.
[17] WENG X，KITANI K.Learning spatio-temporal features with two-stream deep 3d CNNs for lipreading[J].arXiv：1905.02540，2019.
[18] SUN D，YANG X，LIU M Y，et al.Pwc-net：Cnns for optical flow using pyramid，warping，and cost volume[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，UT，USA，June 18-23，2018.New York：IEEE，2018：8934-8943.
[19] CHUNG J S，ZISSERMAN A.Out of time：automated lip sync in the wild[C]//Proceedings of the Asian Conference on Computer Vision，Taipei，Taiwan，Nov 20-24，2016.Berlin，Cham：Springer，2016：251-263.
[20] ZHAO G，BARNARD M，PIETIKAINEN M.Lipreading with local spatiotemporal descriptors[J].IEEE Transactions on Multimedia，2009，11（7）：1254-1265.
[21] WAND M，KOUTIK J，SCHMIDHUBER J.Lipreading with long short-term memory[C]//Proceedings of the 2016 IEEE International Conference on Acoustics，Speech and Signal Processing，Shanghai，China，March 20-25，2016.New York：IEEE，2016：6115-6119.
[22] GARG A，NOYOLA J，BAGADIA S，et al.Lip reading using CNN and LSTM，CS231 n project report[R].Stanford University，2016.
[23] NICOLAS A，FERRERO E E，MARTENS K，et al.Deformation and flow of amorphous solids：Insights from elastoplastic models[J].Reviews of Modern Physics，2018，90（4）：045006.
[24] TSCHANNEN M，DJOLONGA J，RUBENSTEIN P K，et al.On mutual information maximization for representation learning[J].arXiv：1907.13625，2019.
[25] CHUNG J S，ZISSERMAN A.Learning to lip read words by watching videos[J].Computer Vision and Image Understanding，2018，173：76-85.
[26] GUNES H，PICCARDI M.Affect recognition from face and body：early fusion vs.late fusion[C]//Proceedings of the 2005 IEEE International Conference on Systems，Man and Cybernetics，Waikoloa，HI，USA，Oct 12-12，2005.New York：IEEE，2005：3437-3443.
[27] ARBES F，LANDMAN W，WEINREBE G，et al.Multi tower systems and simulation tools[C]//Proceedings of the AIP Conference Proceedings，Casablanca，Morocco，Oct 2-5，2018.New York：AIP，2019.
[28] CHATFIELD K，SIMONYAN K，VEDALDI A，et al.Return of the devil in the details：delving deep into convolutional nets[J].arXiv：1405.3531，2014.
[29] CHUNG J S，ZISSERMAN A.Lip reading in the wild[C]//Proceedings of the Asian Conference on Computer Vision，Taipei，Taiwan，Nov 20-24，2016.Berlin，Cham：Springer，2016：87-103.
[30] SAITOH T，ZHOU Z，ZHAO G，et al.Concatenated frame image based CNN for visual speech recognition[C]//Proceedings of the Asian Conference on Computer Vision，Taipei，Taiwan，Nov 20-24，2016.Berlin，Cham：Springer，2016：277-289.
[31] LIN M，CHEN Q，YAN S.Network in network[J].arXiv：1312.4400，2013.
[32] MESBAH A，BERRAHOU A，HAMMOUCHI H，et al.Lip reading with Hahn convolutional neural networks[J].Image and Vision Computing，2019，88：76-83.
[33] STAFYLAKIS T，TZIMIROPOULOS G.Combining residual networks with LSTMs for lipreading[J].arXiv：1703.
04105，2017.
[34] TORFI A，IRANMANESH S M，NASRABADI N，et al.
3D convolutional neural networks for cross audio-visual matching recognition[J].IEEE Access，2017，5：22081-22091.
[35] NISAR S，KHAN M A，USMAN M.Iris recognition using mel-fequency cepstral coefficient[J].International Journal of Engineering Research，2014，3（2）：126-129.
[36] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Las V，USA，June 27-30，2016.Las Vegas：IEEE，2016：770-778.
[37] GRAVES A，SCHMIDHUBER J.Framewise phoneme classification with bidirectional LSTM and other neural network architectures[J].Neural Networks，2005，18（5/6）：602-610.
[38] PETRIDIS S，STAFYLAKIS T，MA P，et al.End-to-end audiovisual speech recognition[C]//Proceedings of the 2018 IEEE International Conference on Acoustics，Speech and Signal Processing，Calgary，Canada，April 15-20，2018.New York：IEEE，2018：6548-6552.
[39] YANG S，ZHANG Y，FENG D，et al.LRW-1000：a naturally-distributed large-scale benchmark for lip reading in the wild[C]//Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition，Lille，France，May 14-18，2019.New York：IEEE，2019：222-229.
[40] WIRIYATHAMMABHUM P.SpotFast networks with memory augmented lateral transformers for lipreading[C]//International Conference on Neural Information Processing，Bangkok Thailand，Nov 18-22，2020.Berlin，Cham：Springer，2020：554-561.
[41] GUTIERREZ A，ROBERT Z.Lip reading word classification[C]//Asian Conference on Computer Vision，2017.
[42] CHUNG J S，SENIOR A，VINYALS O，et al.Lip reading sentences in the wild[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，HI，Jul 21-26，2017.New York：IEEE，2017：3444-3453.
[43] CHUNG J S，ZISSERMAN A.Lip reading in profile[C]//British Machine Vision Conference，2017.
[44] NADEEMHASHMI S，GUPTA H，MITTAL D，et al.A lip reading model using CNN with batch normalization[C]//Proceedings of the 2018 Eleventh International Conference on Contemporary Computing，Noida，India，Aug 2-4，2018.New York：IEEE，2018：1-6.
[45] STAFYLAKIS T，KHAN M H，TZIMIROPOULOS G，et al.Pushing the boundaries of audiovisual word recognition using residual networks and LSTMs[J].Computer Vision and Image Understanding，2018，176：22-32.
[46] COURYNEY L，SREENIVAS R.Learning from videos with deep convolutional LSTM networks[J].arXiv：1904.04817，2019.
[47] WANG C.Multi-grained spatio-temporal modeling for lip-reading[J].arXiv：1908.11618，2019.
[48] ZHANG Y，YANG S，XIAO J，et al.Can we read speech beyond the lips? rethinking ROI selection for deep visual speech recognition[J].arXiv：2003.03206，2020.
[49] XIAO J，YANG S，ZHANG Y，et al.Deformation flow based two-stream network for lip reading[J].arXiv：2003.
05709，2020.
[50] ZHAO X，YANG S，SHAN S，et al.Mutual information maximization for effective lip reading[J].arXiv：2003.06439，2020.
[51] PAREKH D，GUPTA A，CHHATPAR S，et al.Lip reading using convolutional auto encoders as feature extractor[C]//Proceedings of the 2019 IEEE 5th International Conference for Convergence in Technology，Bombay，India，March 29-31，2019.New York：IEEE，2019：1-6.
[52] FENG D，YANG S，SHAN S，et al.Learn an effective lip reading model without pains[J].arXiv：2011.07557，2020.
[53] GRAVES A，FERNANDEZ S，GOMEZ F，et al.Connectionist temporal classification：labelling unsegmented sequence data with recurrent neural networks[C]//Proceedings of the 23rd International Conference on Machine Learning，Pennsylvania，USA，June 25-29，2006.New York：ACM，2006：369-376.
[54] ASSAEL Y M，SHILLINGFORD B，WHITESON S，et al.Lipnet：sentence-level lipreading[J].arXiv：1611.01599，2016.
[55] PRABHAVALKAR R，SAINATH T N，WU Y，et al.Minimum word error rate training for attention-based sequence-to-sequence models[C]//Proceedings of the 2018 IEEE International Conference on Acoustics，Speech and Signal Processing，Calgary，AB，Canada，April 15-20，2018.New York：IEEE，2018：4839-4843.
[56] ZHANG X，GONG H，DAI X，et al.Understanding pictograph with facial features：end-to-end sentence-level lip reading of Chinese[C]//Proceedings of the AAAI Conference on Artificial Intelligence，Hawaii，USA，January 27-30，2019.California：AAAI，2019：9211-9218.
[57] SHILLINGFORD B，ASSAEL Y，HOFFMAN M W，et al.Large-scale visual speech recognition[J].arXiv：1807.05162，2018.
[58] XU K，LI D，CASSIMATIS N，et al.LCANet：end-to-end lip reading with cascaded attention-CTC[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition，Xi’an，China，May 15-19，2018.New York：IEEE，2018：548-555.
[59] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[60] AFOURAS T，CHUNG J S，SENIOR A，et al.Deep audio-visual speech recognition[J].arXiv：1809.02108，2018.
[61] TOOMARIAN N B，BARHEN J.Learning a trajectory using adjoint functions and teacher forcing[J].Neural Networks，1992，5（3）：473-484.
[62] LUO M，YANG S，SHAN S，et al.Pseudo-convolutional policy gradient for sequence-to-sequence lip-reading[J].arXiv：2003.03983，2020.
[63] SENIOR A，VANHOUCKE V，NGUYEN P，et al.Deep neural networks for acoustic modeling in speech recognition[J].IEEE Signal Processing Magazine，2012.
[64] ZHOU P，YANG W，CHEN W，et al.Modality attention for end-to-end audio-visual speech recognition[C]//Proceedings of the 2016 IEEE International Conference on Acoustics，Speech and Signal Processing，Brighton，UK，May 12-17，2019.New York：IEEE，2019：6565-6569.
[65] SARHAN A M，ELSHENNAWY N M，IBRAHIM D M.HLR-Net：a hybrid lip-reading model based on deep convolutional neural networks[J].CMC-Computers Materials & Continua，2021，68（2）：1531-1549.
[66] ELASHMAWY S，RAMSIS M，ERAQI H M，et al.Spatio-temporal attention mechanism and knowledge distillation for lip reading[J].arXiv：2108.03543，2021.
[67] LU Y，LI H.Automatic lip-reading system based on deep convolutional neural network and attention-based long short-term memory[J].Applied Sciences，2019，9（8）：1599.
[68] LIN Z，ZHAO Z，LI H，et al.SimulLR：simultaneous lip reading transducer with attention-guided adaptive memory[J].arXiv：2108.13630，2021.
[69] LIU J，REN Y，ZHAO Z，et al.FastLR：non-autoregressive lipreading model with integrate-and-fire[C]//Proceedings of the 28th ACM International Conference on Multimedia，Seattle WA，USA，Oct 12-16，2020.New York：ACM，2020：4328-4336.
[70] CHEN H，DU J，HU Y，et al.Lip-reading with hierarchical pyramidal convolution and self-attention[J].arXiv：2012.
14360，2020.
[71] ZHAO H，ZHANG B，YIN Z.Lip-corrector：application of BERT-based model in sentence-level lipreading[C]//Proceedings of the Journal of Physics：Conference Series，Nanjing，China，March 12-14，2021.Nanjing：IOP Publishing，2021.
[72] QU L，WEBWER C，WERMTER S.LipSound：neural mel-spectrogram reconstruction for lip reading[C]//Proceedings of the Interspeech 2019，Graz，Austria，Sep 15-19，2019.Denmark：ISCA，2019：2768-2772.
[73] LUO M，YANG S，CHEN X，et al.Synchronous bidirectional learning for multilingual lip reading[J].arXiv：2005.
03846，2020.
[74] ZHANG X，CHENG F，WANG S，et al.Spatio-temporal fusion based convolutional sequence learning for lip reading[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision，Seoul，Korea，Oct 27-31，2019.New York：IEEE，2019：713-722.
[75] PETRIDIS S，STAFYLAKIS T，MA P，et al.Audio-visual speech recognition with a hybrid ctc/attention architecture[C]//Proceedings of the 2018 IEEE Spoken Language Technology Workshop（SLT），Athens，Greece，Dec 18-21，2018.New York：IEEE，2018：513-520.
[76] WAND M，SCHMIDHUBER J.Improving speaker-independent lipreading with domain-adversarial training[J].arXiv：1708.01565，2017.
[77] WAND M，SCHMIDHUBER J，VU N T，et al.Investigations on end-to-end audiovisual fusion[C]//Proceedings of the 2018 IEEE International Conference on Acoustics，Speech and Signal Processing，Calgary，AB，Canada，April 15-20，2018.New York：IEEE，2018：3041-3045.
[78] AFOURAS T，CHUNG J S，ZISSERMAN A.Deep lip reading：a comparison of models and an online application[J].arXiv：1806.06053，2018.
[79] FENGHOUR S，CHEN D，XIAO P.Decoder-encoder LSTM for lip reading[C]//Proceedings of the 2019 8th International Conference on Software and Information Engineering，Cairo，Egypt，April 9-12，2019.New York：ACM，2019：162-166.
[80] MARGAM D K，ARALIKATTI R，SHARMA T，et al.Lip reading with 3D-2D-CNN BLSTM-HMM and word-CTC models[J].arXiv：1906.12170，2019.
[81] GRAVES A，JAITLY N.Towards end-to-end speech recognition with recurrent neural networks[C]//Proceedings of the 31st International Conference on Machine Learning，Beijing，China，June 22-24，2014：1764-1772.
[82] HANNUN A，CASE C，CASPER J，et al.Deep speech：scaling up end-to-end speech recognition[J].arXiv：1412.
5567，2014.
[83] ZHOU Z，ZHAO G，HONG X，et al.A review of recent advances in visual speech decoding[J].Image and Vision Computing，2014，32（9）：590-605.
[84] FERNANDEZ-LOPEZ A，SUKNO F M.Survey on automatic lip-reading in the era of deep learning[J].Image and Vision Computing，2018，78：53-72.
[85] LIU C，GONG S，LOY C C，et al.Person re-identification：What features are important?[C]//Proceedings of European Conference on Computer Vision，Florence，Italy，Oct 7-13，2012.Berlin，Cham：Springer，2012：391-401.
[86] AFOURAS T，CHUNG J S，ZISSERMAN A.LRS3-TED：a large-scale dataset for visual speech recognition[J].arXiv：1809.00496，2018.
[87] XU B，LU C，GUO Y，et al.Discriminative multi-modality speech recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，USA，June 15-18，2020.New York：IEEE，2020：14433-14442.
[88] MARTINEZ B，MA P，PETRIDIS S，et al.Lipreading using temporal convolutional networks[C]//Proceedings of 2020 IEEE International Conference on Acoustics，Speech and Signal Processing，Barcelona，Spain，May 4-8，2020.New York：IEEE，2020：6319-6323.
[89] LI X，WANG W，HU X，et al.Selective kernel networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，CA，USA，June 15-20，2019.New York：IEEE，2019：510-519.
[90] JETHANANDANI M，TANG D.Adversarial attacks against LipNet：end-to-end sentence level lipreading[C]//Proceedings of the 2020 IEEE Security and Privacy Workshops，San Francisco，CA，USA，May 21-21，2020.New York：IEEE，2020：15-19.
[91] ZHAO Y，XU R，WANG X，et al.Hearing lips：improving lip reading by distilling speech recognizers[C]//Proceedings of the AAAI Conference on Artificial Intelligence，New York Hilton Midtown，New York，Feb 7-12，2012.Palo Alto，CA：AAAI Press，2020：6917-6924.
[92] AFOURAS T，CHUNG J S，ZISSERMAN A.Asr is all you need：cross-modal distillation for lip reading[C]//Proceedings of the 2020 IEEE/CVF International Conference on Acoustics，Speech and Signal Processing，Barcelona，Spain，May 4-8，2020.New York：IEEE，2020：2143-2147.
[93] YU J，ZHANG S X，WU J，et al.Audio-visual recognition of overlapped speech for the lrs2 dataset[C]//Proceedings of 2020 IEEE International Conference on Acoustics，Speech and Signal Processing，Barcelona，Spain，May 4-8，2020.New York：IEEE，2020：6984-6988.
[94] ALMAJAI I，COX S，HAARVEY R，et al.Improved speaker independent lip reading using speaker adaptive training and deep neural networks[C]//Proceedings of 2016 IEEE International Conference on Acoustics，Speech and Signal Processing，Shanghai，China，March 20-25 2016.New York：IEEE，2016：2722-2726.
[95] FERNANDEZ-LOPEZ A，SUKNO F M.Automatic viseme vocabulary construction to enhance continuous lip-reading[J].arXiv：1704.08035，2017.