Research Progress of Natural Language Processing Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2106-0166

Abstract

Abstract:

This paper comprehensively analyzes the research of deep learning in the field of natural language processing through a combination of quantitative and qualitative methods. It uses CiteSpace and VOSviewer to draw a knowledge graph of countries, institutions, journal distribution, keywords co-occurrence, co-citation network clustering, and timeline view of deep learning in the field of natural language processing to clarify the research. Through mining important researches in the field, this paper summarizes the research trend, the main problems, development bottlenecks, and gives corresponding solutions and ideas. Finally, suggestions are given on how to track the research of deep learning in the field of natural language processing, and provides references for subsequent research and development in the field.

Key words: Deep Learning（DL）, Natural Language Processing（NLP）, knowledge graph, visualization

摘要：

通过定量与定性相结合的方式全面分析了深度学习在自然语言处理领域的研究情况。采用CiteSpace和VOSviewer对深度学习在自然语言处理领域的研究国家、机构、期刊分布、关键词共现、共被引网络聚类及时间轴视图等进行知识图谱绘制，理清研究脉络。通过深入挖掘领域内的重要文献，总结深度学习在自然语言处理领域的研究趋势、存在的主要问题或发展瓶颈，并给出相应的解决办法与思路。对于如何跟踪深度学习在自然语言处理领域的研究成果给出建议，为该领域的后续研究与发展提供参考。

关键词: 深度学习（DL）, 自然语言处理（NLP）, 知识图谱, 可视化

JIANG Yangyang, JIN Bo, ZHANG Baochang. Research Progress of Natural Language Processing Based on Deep Learning[J]. Computer Engineering and Applications, 2021, 57(22): 1-14.

江洋洋，金伯，张宝昌. 深度学习在自然语言处理领域的研究进展[J]. 计算机工程与应用, 2021, 57(22): 1-14.

References

[1] YOUNG T，HAZARIKA D，PORIA S，et al.Recent trends in deep learning based natural language processing[J].IEEE Computational Intelligence Magazine，2018，13（3）：55-75.
[2] 赵京胜，宋梦雪，高祥.自然语言处理发展及应用综述[J].信息技术与信息化，2019（7）：142-145.
ZHAO J S，SONG M X，GAO X.Development and application of natural language processing[J].Information Technology and Informatization，2019（7）：142-145.
[3] 徐翼龙，李文法，周纯洁.基于深度学习的自然语言处理综述[C]//中国计算机用户协会网络应用分会2018年第二十二届网络新技术与应用年会论文集，2018：4.
XU Y L，LI W F，ZHOU C J.Review of natural language processing based on deep learning[C]//Proceedings of the 22nd Annual Meeting of Network New Technology and Application Held by Network Application Branch of China Computer Users Association in 2018，2018：4.
[4] 何玉洁，杜方，史英杰，等.基于深度学习的命名实体识别研究综述[J].计算机工程与应用，2021，57（11）：21-36.
HE Y J，DU F，SHI Y J，et al.Survey of named entity recognition based on deep learning[J].Computer Engineering and Applications，2021，57（11）：21-36.
[5] 郝超，裘杭萍，孙毅，等.多标签文本分类研究进展[J].计算机工程与应用，2021，57（10）：48-56.
HAO C，QIU H P，SUN Y，et al.Research progress of multi-label text classification[J].Computer Engineering and Applications，2021，57（10）：48-56.
[6] 余同瑞，金冉，韩晓臻，等.自然语言处理预训练模型的研究综述[J].计算机工程与应用，2020，56（23）：12-22.
YU T R，JIN R，HAN X Z，et al.Review of pre-training models for natural language processing[J].Computer Engineering and Applications，2020，56（23）：12-22.
[7] OTTER D W，MEDINA J R，KALITA J K.A survey of the usages of deep learning for natural language processing[J].IEEE Transactions on Neural Networks and Learning Systems，2021，32（2）：604-624.
[8] 王国胤，张军平，何清，等.中国人工智能发展报告2019—2020[M].北京：机械工业出版社，2020.
WANG G Y，ZHANG J P，HE Q，et al.China AI development report 2019—2020[M].Beijing：China Machine Press，2020.
[9] HINTON G，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，8（7）：1527-1554.
[10] LI H.Deep learning for natural language processing：advantages and challenges[J].National Science Review，2018，5（1）：24-26.
[11] 罗枭.基于深度学习的自然语言处理研究综述[J].智能计算机与应用，2020，10（4）：133-137.
LUO X.A survey of natural language processing based on deep learning[J].Intelligent Computer and Applications，2020，10（4）：133-137.
[12] 李杰，陈超美.CiteSpace：科技文本挖掘及可视化[M].2版.北京：首都经济贸易大学出版社，2017.
LI J，CHEN C M.CiteSpace：text mining and visualization in scientific literature[M].2nd ed.Beijing：Capital University of Economics and Business Press，2017.
[13] 李杰.科学知识图谱原理及应用[M].北京：高等教育出版社，2018.
LI J.Principles and applications of mapping knowledge domains[M].Beijing：Higher Education Press，2018.
[14] LIU W S，LIAO H C.A bibliometric analysis of fuzzy decision research during 1970—2015[J].International Journal of Fuzzy Systems，2017，19：1-14.
[15] YU D J，LIAP H C.Visualization and quantitative research on intuitionistic fuzzy studies[J].Journal of Intelligent and Fuzzy Systems，2016，30：3653-3663.
[16] TASKIN Z，AL U.Natural language processing applications in library and information science[J].Online Information Review，2019，43（4）：676-690.
[17] HOU L，CHEN H S，ZHANG G M，et al.Deep learning-based applications for safety management in the AEC industry：a review[J].Applied Sciences Basel，2021，11（2）：821.
[18] POWELL T H，KOUROPALATIS Y，MORGAN R E，et al.Mapping knowledge and innovation research themes：using bibliometrics for classification，evolution，proliferation and determinism[J].International Entrepreneurship and Management Journal，2016，20：174-199.
[19] UpHonest Capital：美国人工智能投资分析报告[EB/OL].（2017-12-12）[2021-05-08].https：//www.sohu.com/a/210056479_
483389.
UpHonest Capital：investment analysis report of American artificial intelligence[EB/OL].（2017-12-12）[2021-05-08].https：//www.sohu.com/a/210056479_483389.
[20] 王惠铃.2017美国人工智能产业分析[EB/OL].（2017-08-21）[2021-05-08].https：//www.jieju.cn/News/20170821/Detail799453.shtml.
WANG H L.Analysis of American AI industry in 2017[EB/OL].（2017-08-21）[2021-05-08].https：//www.jieju.cn/News/20170821/Detail799453.shtml.
[21] ARGOUBI M，JAMMELI H，MASRI H.The intellectual structure of the waste management field[J].Annals of Operations Research，2020，294（1/2）：655-676.
[22] CHEN C M.CiteSpace II：detecting and visualizing emerging trends and transient patterns in scientific literature[J].Journal of the American Sociaty for Information Science and Technology，2006，57（3）：359-377.
[23] BENGIO Y，DUCHARME R，VINCENT P，et al.A neural probabilistic language model[J].Journal of Machine Learning Research，2003，3：1137-1155.
[24] COLLOBERT R，WESTON J.A unified architecture for natural language processing：deep neural networks with multitask learning[C]//25th International Conference on Machine Learning，2008：160-167.
[25] MIKOLOV T，SUTSKEVER I，CHEN K，et al.Distributed representations of words and phrases and their compositionality[C]//Advances in Neural Information Processing Systems，2013：3111-3119.
[26] PENNINGTON J，SOCHER R，MANNING C D.Glove：global vectors for word representation[C]//2014 Conference on Empirical Methods in Natural Language Processing，2014：1532-1543.
[27] GITTENS A，ACHLIOPTAS D，MAHONEY M W.Skip-gram-zipf+uniform=vector additivity[C]//55th Annual Meeting of the Association for Computational Linguistics，2017，1：69-76.
[28] ETHAYARAJH K，DUVENAUD D，HIRST G.Towards understanding linear word analogies[J].arXiv：1810.04882，2018.
[29] COLLOBERT R，WESTON J，BOTTOU L，et al.Natural language processing（almost） from scratch[J].Journal of Machine Learning Research，2011，12：2493-2537.
[30] KALCHBRENNER N，GREFENSTETTE E，BLUNSOM P.A convolutional neural network for modelling sentences[J].arXiv：1404.2188v1，2014.
[31] KIM Y.Convolutional neural networks for sentence classification[C]//2014 Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[32] MIKOLOV T，KOMBRINK S，BURGET L，et al.Extensions of recurrent neural network language model[C]//2011 IEEE International Conference on Acoustics，Speech and Signal Processing，2011：5528-5531.
[33] SUTSKEVER I，MARTENS J，HINTON G E.Generating text with recurrent neural networks[C]//28th International Conference on Machine Learning，2011：1017-1024.
[34] LIU S，YANG N，LI M，et al.A recursive recurrent neural network for statistical machine translation[C]//52nd Annual Meeting of the Association for Computational Linguistics，2014，1：1491-1500.
[35] AULI M，GALLEY M，QUIRK C，et al.Joint language and translation modeling with recurrent neural networks[C]//2013 Conference on Empirical Methods in Natural Language Processing，2013：1044-1054.
[36] SUTSKEVER I，VINYALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems，2014：3104-3112.
[37] GRAVES A，MOHAMED A R，HINTON G.Speech recognition with deep recurrent neural networks[C]//2013 IEEE International Conference on Acoustics，Speech，and Signal Processing，2013：6645-6649.
[38] GRAVES A，JAITLY N.Towards end-to-end speech recognition with recurrent neural networks[C]//31st International Conference on Machine Learning，2014：3771-3779.
[39] SAK H，SENIOR A，BEAUFAYS F.Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition[J].arXiv：1402.1128，2014.
[40] VINYALS O，TOSHEV A，BENGIO S，et al.Show and tell：a neural image caption generator[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition，2015：3156-3164.
[41] CHO K，VAN MERRIENBOER B，GULCEHRE C，et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//2014 Conference on Empirical Methods in Natural Language Processing，2014：1724-1734.
[42] KARPATHY A，LI F.Deep visual-semantic alignments for generating image descriptions[C]//2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2015：3128-3137.
[43] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[44] RUSH A M，CHOPRA S，WESTON J.A neural attention model for abstractive sentence summarization[J].arXiv：1509.00685，2015.
[45] XU K，BA J，KIROS R，et al.Show，attend and tell：neural image caption generation with visual attention[C]//32nd International Conference on Machine Learning，2015，3：2048-2057.
[46] VINYALS O，KAISER L，KOO T，et al.Grammar as a foreign language[C]//Advances in Neural Information Processing Systems，2015：2773-2781.
[47] VINYALS O，FORTUNATO M，JAITLY N.Pointer networks[C]//Advances in Neural Information Processing Systems，2015：2692-2700.
[48] PAULUS R，XIONG C，SOCHER R.A deep reinforced model for abstractive summarization[J].arXiv：1705.04304，2017.
[49] WANG Y，HUANG M，ZHAO L，et al.Attention-based LSTM for aspect-level sentiment classification[C]//2016 Conference on Empirical Methods in Natural Language Processing，2016：606-615.
[50] MA Y，PENG H，CAMBRIA E.Targeted aspect-based sentiment analysis via embedding common sense knowledge into an attentive LSTM[C]//Association Advancement Artificial Intelligence Conference，2018：5876-5883.
[51] SOCHER R，HUVAL B，MANNING C D，et al.Semantic compositionality through recursive matrix-vector spaces[C]//2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning，2012：1201-1211.
[52] BOWMAN S R，POTTS C，MANNING C D.Recursive neural networks can learn logical semantics[J].arXiv：1406.
1827，2014.
[53] DAI A M，LE Q V.Semi-supervised sequence learning[C]//Advances in Neural Information Processing Systems，2015：3079-3087.
[54] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5999-6009.
[55] PETERS M E，NEUMANN M，IYYER M，et al.Deep contextualized word representations[C]//2018 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2018：2227-2237.
[56] RADFORD A，NARASIMHAN K，SALIMANS T，et al.Improving language understanding with unsupervised learning[EB/OL].（2018-06-11）[2021-05-08].https：//openai.com/blog/language-unsupervised/.
[57] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[58] LIU Y H，OTT M，GOYAL N，et al.RoBERTa：a robustly optimized BERT pretraining approach[J].arXiv：1907.
11692，2019.
[59] SONG K T，TAN X，QIN T，et al.MASS：masked sequence to sequence pre-training for language generation[C]//36th International Conference on Machine Learning，2019：10384-10394.
[60] DONG L，YANG N，WANG W H，et al.Unified language model pre-training for natural language understanding and generation[C]//Advances in Neural Information Processing Systems，2019：13042-13054.
[61] ZHANG Z Y，HAN X，LIU Z Y，et al.ERNIE：enhanced language representation with informative entities[J].arXiv：1905.07129v3，2019.
[62] JOSHI M，CHEN D，LIU Y，et al.SpanBERT：improving pre-training by representing and predicting spans[J].arXiv：1907.10529，2019.
[63] LAN Z Z，CHEN M D，GOODMAN S，et al.ALBERT：a lite BERT for self-supervised learning of language representations[J].arXiv：1909.11942，2019.
[64] YANG Z L，DAI Z H，YANG Y M，et al.XLNet：generalized autoregressive pretraining for language understanding[C]//Advances in Neural Information Processing Systems，2019：5754-5764.
[65] LUO J，WU J.An entropy-based pruning method for CNN compression[J].arXiv：1706.05791，2017.
[66] YANG T，CHEN Y，SZE V.Designing energy-efficient convolutional neural networks using energy-aware pruning[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：6071-6079.
[67] HU Y，SUN S，LI J，et al.A novel channel pruning method for deep neural network compression[J].arXiv：1805.1139419，
2018.
[68] HE Y H，ZHANG X Y，SUN J.Channel pruning for accele-
rating very deep neural networks[C]//2017 IEEE International Conference on Computer Vision，2017：1389-1397.
[69] ANWAR S，SUNG W Y.Coarse pruning of convolutional neural networks with random masks[C]//2017 International Conference on Learning Representations，2017：134-145.
[70] LI H，KADAV A，DURDANOVIC I，et al.Pruning filters for efficient ConvNets[J].arXiv：1608.08710，2016.
[71] PAVLO M，STEPHEN T，TERO K，et al.Pruning convolutional neural networks for resource efficient inference[J].arXiv：1611.06440，2016.
[72] HU H，PENG R，TAI Y W，et al.Network trimming：a datadriven neuron pruning approach towards efficient deep architectures[J].arXiv：1611.05128，2016.
[73] DEEPAK M，SHWETA B，MITESH M，et al.Recovering from random pruning：on the plasticity of deep convolutional neural networks[J].arXiv：1801.10447，2018.
[74] LIU X Y，HAN S，MAO H Z，et al.Efficient sparse-winograd convolutional neural networks[C]//5th International Conference on Learning Representations，2017.
[75] GORDON M A，DUH K，ANDREWS N.Compressing BERT：studying the effects of weight pruning on transfer learning[J].arXiv：2002.08307，2020.
[76] MCCARLEY J S，CHAKRAVARTI R，SIL A.Pruning a BERT-based question answering model[J].arXiv：1910.
06360，2019.
[77] GUO F M，LIU S J，MUNGALL F S，et al.Reweighted proximal pruning for large-scale language representation[J].arXiv：1909.12486，2019.
[78] HOWARD A G，ZHU M，CHEN B，et al.Mobilenets：efficient convolutional neural networks for mobile vision applications[J].arXiv：1704.04861，2017.
[79] SANDLER M，HOWARD A，ZHU M，et al.Mobilenetv2：inverted residuals and linear bottlenecks[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：4510-4520.
[80] ZHANG X，ZHOU X，LIN M，et al.Shufflenet：an extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：6848-6856.
[81] MA N，ZHANG X，ZHENG H T，et al.Shufflenet v2：practical guidelines for efficient CNN architecture design[C]//15th European Conference on Computer Vision.Cham：Springer，2018：116-131.
[82] SZEGEDY C，VANHOUCKE V，IOFFE S.Rethinking the inception architecture for computer vision[J].arXiv：1512.
00567v3，2015.
[83] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-
Net classification with deep convolutional neural networks[J].Communications of the ACM，2017，60（6）：84-90.
[84] DEHGHANI M，GOUWS S，VINYALS O，et al.Universal transformers[J].arXiv：1807.03819，2018.
[85] HAO J，WANG X，YANG B S，et al.Modeling recurrence for transformer[C]//2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：1198-1207.
[86] PHAM H，GUAN M Y，ZOPH B，et al.Efficient neural architecture search via parameter sharing[J].arXiv：1802.
03268，2018.
[87] WANG Z H，WOHLWEND J，TAO L.Structured pruning of large language models[J].arXiv：1910.04732v2，2021.
[88] BUCILUA C，CARUANA R，NICULESCU MIZIL A.Model compression[C]//12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2006：535.
[89] HINTON G，VINYALS O，DEAN J.Distilling the knowledge in a neural network[J].Computer Science，2015，14（7）：38-39.
[90] 耿丽丽，牛保宁.深度神经网络模型压缩综述[J].计算机科学与探索，2020，14（9）：1441-1455.
GENG L L，NIU B N.Survey of deep neural networks model compression[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1441-1455.
[91] ZHAO S Q，GUPTA R，SONG Y，et al.Extreme language model compression with optimal subwords and shared projections[J].arXiv：1909.11687，2019.
[92] SANH V，DEBUT L，CHAUMOND J，et al.DistilBERT，a distilled version of BERT：smaller，faster，cheaper and lighter[J].arXiv：1910.01108，2019.
[93] JIAO X Q，YIN Y C，SHANG L F，et al.TinyBERT：distilling BERT for natural language understanding[J].arXiv：1909.10351，2019.
[94] MUKHERJEE S，AWADALLAH A H.Distilling BERT into simple neural networks with unlabeled transfer data[J].arXiv：1910.01769，2019.
[95] 李江昀，赵义凯，薛卓尔，等.深度神经网络模型压缩综述[J].工程科学学报，2019，41（10）：1229-1239.
LI J Y，ZHAO Y K，XUE Z E，et al.A survey of model compression for deep neural networks[J].Chinese Journal of Engineering，2019，41（10）：1229-1239.
[96] HAN S，MAO H Z，DALLY W J.Deep compression：compressing deep neural networks with pruning，trained quantization and Huffman coding[J].arXiv：1510.00149，2015.
[97] JACOB B，KLIGYS S，CHEN B，et al.Quantization and training of neural networks for efficient integer-arithmetic-only inference[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：2704-2713.
[98] ZAFRIR O，BOUDOUKH G，IZSAK P，et al.Q8BERT：quantized 8bit BERT[J].arXiv：1910.06188，2019.
[99] SHEN S，DONG Z，YE J Y，et al.Q-BERT：Hessian based ultra low precision quantization of BERT[J].arXiv：1909.
05840，2019.
[100] ZHANG Q，WU Y N，ZHU S C.Interpretable convolutional neural networks[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：8827-8836.
[101] CASTANON G，BYRNE J.Visualizing and quantifying discriminative features for face recognition[C]//13th IEEE International Conference on Automatic Face and Gesture Recognition，2018：16-23.
[102] ZHOU B L，SUN Y Y，BAU D，et al.Interpretable basis decomposition for visual explanation[C]//15th European Conference on Computer Vision.Cham：Springer，2018：122-138.
[103] HOOKER S，ERHAN D，KINDERMANS P J，et al.Evaluating feature importance estimates[J].arXiv：1806.10758，
2018.
[104] VENTURA F，CERQUITELLI T，GIACALONE F.Black-box model explained through an assessment of its interpretable features[C]//New Trends in Databases and Information Systems-ADBIS 2018 Short Papers and Workshops，2018：138-149.
[105] FONG R C，VEDALDI A.Interpretable explanations of black boxes by meaningful perturbation[C]//2017 IEEE International Conference on Computer Vision，2017：3449-3457.
[106] PETSIUK V，DAS A，SAENKO K.RISE：randomized input sampling for explanation of black-box models[J].arXiv：1806.07421，2018.
[107] ZHANG Q，WANG W，ZHU S C.Examining CNN representations with respect to dataset bias[C]//32nd AAAI Conference on Artificial Intelligence，2018：4464-4473.
[108] BAUMGARTNER C F，KOCH L M.Supplementary material for visual feature attribution using Wasserstein GANs[J].arXiv：1711.08998，2018.
[109] HOU B J，ZHOU Z H.Learning with interpretable structure from RNN[J].arXiv：1810.10708，2018.
[110] WARNECKE A，ARP D，WRESSNEGGER C，et al.Don’t paint it black：white-box explanations for deep learning in computer security[J].arXiv：1906.02108，2019.
[111] 化盈盈，张岱墀，葛仕明.深度学习模型可解释性的研究进展[J].信息安全学报，2020，5（3）：1-12.
HUA Y Y，ZHANG D C，GE S M.Research progress in the interpretability of deep learning models[J].Journal of Cyber Security，2020，5（3）：1-12.
[112] 吴飞，廖彬兵，韩亚洪.深度学习的可解释性[J].航空兵器，2019（1）：39-46.
WU F，LIAO B B，HAN Y H.Interpretability for deep learning[J].Aero Weaponry，2019（1）：39-46.
[113] SPRING R，SHRIVASTAVA A.Scalable and sustainable deep learning via randomized hashing[C]//23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2017：445-454.
[114] WIGGERS K.MIT researchers warn that deep learning is approaching computational limits[EB/OL].（2020-07-15）[2021-06-30].https：//venturebeat.com/2020/07/15/mit-researchers-warn-that-deep-learning-is-approaching-computational-limits.
[115] YANG Y Z，XU Z.Rethinking the value of labels for impro-
ving class-imbalanced learning[J].arXiv：2006.07529v2，2020.
[116] TIAN J R，WANG C B，LI B Y，et al.Equalization loss for long-tailed object recognition[J].arXiv：2003.05176v2，
2020.
[117] MINAEE S，KALCHBRENNER N，CAMBRIA E，et al.Deep learning based text classification：a comprehensive review[J].arXiv：2004.03705，2020.
[118] LI Q，PENG H，LI J，et al.A text classification survey：from shallow to deep learning[J].arXiv：2008.00364，2020.