Survey of Open Source Natural Language Processing Tools

doi:10.3778/j.issn.1002-8331.2211-0358

Abstract

Abstract: Natural language processing tools are functional integration components that realize various subtasks in the field of natural language processing, and provide effective support for text processing and text analysis. At present, there are many types of natural language processing tools, various tools have different levels of support for subtasks, and some tools are only suitable for some special text fields, which will cause confusion in the selection of tools. Firstly, according to the processing order, the subtasks supported by the tools are divided into auxiliary tasks, basic tasks and application tasks, and are introduced. 23 domestic and foreign natural language processing open source tools such as LTP, NLPIR and OpenNLP are selected, and the call methods and supported programming languages of these tools are compared to summarize the characteristics of various tools. Then, the implementation principles of various tool subtasks are divided into rule methods, statistical methods, neural network methods and combination methods for sorting and analysis, the shortcomings of current tools are discussed. Finally, the future development of natural language processing tools are prospected from the aspects of multimodal fusion, cognitive intelligence, model compression and efficient computing.

Key words: natural language processing tools, text processing, text analysis

摘要： 自然语言处理工具是实现自然语言处理领域各项子任务的功能集成构件，为文本处理和文本分析提供有效的支撑。当前自然语言处理工具种类较多，各种工具对子任务支持程度不同，同时某些工具只适用于一些特殊的文本领域，这些差异会对工具选用造成困扰。依据处理顺序将工具支持的子任务划分为辅助任务、基础任务以及应用任务并进行介绍，选取LTP、NLPIR、OpenNLP等23种国内外自然语言处理开源工具，对这些工具的调用方式、支持的程序语言等方面进行比较，总结各种工具特点。再将各种工具子任务的实现原理分为规则方法、统计方法、神经网络方法以及组合方法进行整理和分析，探讨当前工具存在的不足之处。最后从多模态融合、认知智能、模型压缩与高效计算等方面对自然语言处理工具未来的发展进行展望。

关键词: 自然语言处理工具, 文本处理, 文本分析

LIAO Chunlin, ZHANG Hongjun, LIAO Xianglin, CHENG Kai, LI Dashuo, WANG Hang. Survey of Open Source Natural Language Processing Tools[J]. Computer Engineering and Applications, 2023, 59(22): 36-56.

廖春林, 张宏军, 廖湘琳, 程恺, 李大硕, 王航. 开源自然语言处理工具综述[J]. 计算机工程与应用, 2023, 59(22): 36-56.

References

[1] BIRD?S，KLEIN?E，LOPER E.Natural language processing with Python[M].Sebastopol：O’Reilly Media，2009：42-44.
[2] MANNING C D，SURDEANU M，BAUER J，et al.The Stanford CoreNLP natural language processing toolkit[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics：System Demonstrations，2014：55-60.
[3] CHE W，FENG Y，QIN L，et al.N-LTP：an open-source neural language technology platform for Chinese[J].arXiv：2009.11616，2020.
[4] 刘挺，车万翔，李正华.语言技术平台[J].中文信息学报，2011，25（6）：53-63.
LIU T，CHE W P，LI Z H.Language technology platform[J].Journal of Chinese Information Processing，2011，25（6）：53-63.
[5] 刘梦迪，梁循.基于偏旁部首知识表示学习的汉字字形相似度计算方法[J].中文信息学报，2021，35（12）：47-59.
LIU M D，LIANG X.A method of Chinese character glyph similarity calculation based on radical knowledge representation learning[J].Journal of Chinese Information Processing，2021，35（12）：47-59.
[6] LOU J，LU Y，DAI D，et al.Universal information extraction as unified semantic matching[J].arXiv：2301.03282，2023.
[7] CARRIóN S，CASACUBERTA F.On the effectiveness of quasi character-level models for machine translation[C]//Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas（Volume 1：Research Track），2022：131-143.
[8] GARDNER M，GRUS J，NEUMANN M，et al.A deep semantic natural language processing platform[J].arXiv：1803.07640，2017.
[9] SUN M，CHEN X，ZHANG K，et al.THULAC：an efficient lexical analyzer for Chinese[R].Tsinghua University，2016.
[10] SUN M，LI J，GUO Z，et al.THUCTC：an efficient Chinese text classifier[R].Tsinghua University，2016.
[11] ZENG G，QI F，ZHOU Q，et al.OpenAttack：an open-source textual adversarial attack toolkit[J].arXiv：2009.09191，2020.
[12] ZHANG J，DING Y，SHEN S，et al.THUMT：an open source toolkit for neural machine translation[J].arXiv：1706.06415，2017.
[13] HAN X，GAO T，YAO Y，et al.OpenNRE：an open and extensible toolkit for neural relation extraction[J].arXiv：1909.13078，2019.
[14] GENG Z，YAN H，QIU X，et al.fastHan：a BERT-based multi-task toolkit for Chinese NLP[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics，2021：99-106.
[15] JIAO Z，SUN S，SUN K.Chinese lexical analysis with deep Bi-GRU-CRF network[J].arXiv：1807.01882，2018.
[16] 张华平，商建云.NLPIR-Parser：大数据语义智能分析平台[J].语料库语言学，2019，6（1）：87-104.
ZHANG H P，SHANG J Y.NLPIR-Parser：an intelligent semantic analysis toolkit for big data[J].Corpus Linguistics，2019，6（1）：87-104.
[17] HE H，CHOI J D.The stem cell hypothesis：Dilemma behind multi-task learning with transformer encoders[J].arXiv：2109.06939，2021.
[18] AKBIK A，BERGMANN T，BLYTHE D，et al.FLAIR：an easy-to-use framework for state-of-the-art NLP[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics（Demonstrations），2019：54-59.
[19] STRAKA M，STRAKOVá J.Tokenizing，pos tagging，lemmatizing and parsing UD 2.0 with UDPipe[C]//Proceedings of the CoNLL 2017 Shared Task：Multilingual Parsing from Raw Text to Universal Dependencies，2017：88-99.
[20] STRAKA M，STRAKOVá J，HAJI? J.Evaluating contextualized embeddings on 54 languages in POS tagging，lemmatization and dependency parsing[J].arXiv：1908.
07448，2019.
[21] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[22] QI P，ZHANG Y，ZHANG Y，et al.Stanza：a Python natural language processing toolkit for many human languages[J].arXiv：2003.07082，2020.
[23] WANG C，XU B.Convolutional neural network with word embeddings for Chinese word segmentation[J].arXiv：1711.
04411，2017.
[24] PIETRA S，PIETRA S，PIETRA S.A maximum entropy approach to natural language processing[M].Cambridge：MIT Press，1996.
[25] RABINER L，JUANG B.An introduction to hidden Markov models[J].IEEE ASSP Magazine，1986，3（1）：4-16.
[26] LAFFERTY J，MCCALLUM A，PEREIRA F C N.Conditional random fields：probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning，2001：282-289.
[27] FINE S，SINGER Y，TISHBY N.The hierarchical hidden Markov model：analysis and applications[J].Machine Learning，1998，32（1）：41-62.
[28] 刘群，张华平，俞鸿魁，等.基于层叠隐马模型的汉语词法分析[J].计算机研究与发展，2004，41（8）：1421-1429.
LIU Q，ZHANG H P，YU H K，et al.Chinese lexical analysis using cascaded hidden Markov model[J].Journal of Computer Research and Development，2004，41（8）：1421-1429.
[29] MCCALLUM A，FREITAG D，PEREIRA F C N.Maximum entropy Markov models for information extraction and segmentation[C]//Proceedings of the 17th International Conference on Machine Learning，2000：591-598.
[30] JELINEK F，LAFFERTY J D，MERCER R L.Basic methods of probabilistic context free grammars[M].Berlin，Heidelberg：Springer，1992.
[31] MIHALCEA R，TARAU P.TextRank：bringing order into text[C]//Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing，2004：404-411.
[32] BENGIO Y，DUCHARME R，VINCENT P.A neural probabilistic language model[C]//Advances in Neural Information Processing Systems 13，2000.
[33] CHEN D，MANNING C D.A fast and accurate dependency parser using neural networks[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：740-750.
[34] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[35] KIM Y.Convolutional neural networks for sentence classification[J].arXiv：1408.5882，2014.
[36] YIN W，KANN K，YU M，et al.Comparative study of CNN and RNN for natural language processing[J].arXiv：1702.01923，2017.
[37] SATHASIVAM S，ABDULLAH W A T W.Logic learning in hopfield networks[J].arXiv：0804.4075，2008.
[38] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[39] CHO K，MERRIENBOER B V，GULCEHRE C，et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1724-1734.
[40] LIPTON Z C，BERKOWITZ J，ELKAN C.A critical review of recurrent neural networks for sequence learning[J].arXiv：1506.00019，2015.
[41] SUTSKEVER I，VINYALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems 27，2014.
[42] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[43] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems 30，2017：5998-6008.
[44] BACCIU D，ERRICA F，MICHELI A，et al.A gentle introduction to deep learning for graphs[J].Neural Networks，2020，129：203-221.
[45] DEFFERRARD M，BRESSON X，VANDERGHEYNST P.Convolutional neural networks on graphs with fast localized spectral filtering[C]//Advances in Neural Information Processing Systems 29，2016.
[46] VELICKOVIC P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.
[47] ZHANG J，SHI X，XIE J，et al.GAAN：gated attention networks for learning on large and spatiotemporal graphs[J].arXiv：1803.07294，2018.
[48] 王思力，张华平，王斌.双数组Trie树算法优化及其应用研究[J].中文信息学报，2006，20（5）：24-30.
WANG S L，ZHANG H P，WANG B.Research of optimization on double-array trie and its application[J].Journal of Chinese Information Processing，2006，20（5）：24-30.
[49] HUANG Z，XU W，YU K.Bidirectional LSTM-CRF models for sequence tagging[J].arXiv：1508.01991，2015.
[50] DOZAT T，MANNING C D.Deep biaffine attention for neural dependency parsing[J].arXiv：1611.01734，2016.
[51] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[J].arXiv：1301.3781，2013.
[52] BOJANOWSKI P，GRAVE E，JOULIN A，et al.Enriching word vectors with subword information[J].Transactions of the Association for Computational Linguistics，2017，5：135-146.
[53] PENNINGTON J，SOCHER R，MANNING C D.GloVe：global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1532-1543.
[54] PETERS M E，NEUMANN M，IYYER M，et al.Deep contextualized word representations[J].arXiv：1802.05365，2018.
[55] RADFORD A，NARASIMHAN K，SALIMANS T，et al.Improving language understanding by generative pre-training[EB/OL].（2018）[2022-10-18].https：//cdn.openai.com/research-covers/language-unsupervised/language_
understanding_paper.pdf.
[56] SUN Y，WANG S，LI Y，et al.ERNIE：enhanced representation through knowledge integration[J].arXiv：1904.09223，2019.
[57] RADFORD A，WU J，CHILD R，et al.Language models are unsupervised multitask learners[J].OpenAI Blog，2019，1（8）：9.
[58] BROWN T，MANN B，RYDER N，et al.Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33，2020：1877-1901.
[59] LIU Y，OTT M，GOYAL N，et al.RoBERTa：a robustly optimized BERT pretraining approach[J].arXiv：1907.11692，2019.
[60] LAN Z，CHEN M，GOODMAN S，et al.ALBERT：a lite BERT for self-supervised learning of language representations[J].arXiv：1909.11942，2019.
[61] CLARK K，LUONG M T，LE Q V，et al.ELECTRA：pre-training text encoders as discriminators rather than generators[J].arXiv：2003.10555，2020.
[62] BRANTS T.TnT—a statistical part-of-speech tagger[J].arXiv：cs/0003055，2000.
[63] HAJIC J，RAAB J，SPOUSTA M.Semi-supervised training for the averaged perceptron POS tagger[C]//Proceedings of the 12th Conference of the European Chapter of the ACL，2009：763-771.
[64] NIVRE J.Dependency parsing[J].Language and Linguistics Compass，2010，4（3）：138-152.
[65] CHARNIAK E.Statistical parsing with a context-free grammar and word statistics[C]//Proceedings of the 14th National Conference on Artificial Intelligence and 9th Conference on Innovative Applications of Artificial Intelligence，1997：598-603.
[66] JOSHI V，PETERS M，HOPKINS M.Extending a parser to distant domains using a few dozen partially annotated examples[J].arXiv：1805.06556，2018.
[67] XU K，WU L，WANG Z，et al.Exploiting rich syntactic information for semantic parsing with graph-to-sequence model[J].arXiv：1808.07624，2018.
[68] CHEN Y，WU L，ZAKI M J.Reinforcement learning based graph-to-sequence model for natural question generation[J].arXiv：1908.04942，2019.
[69] XU K，WU L，WANG Z，et al.Graph2seq：graph to sequence learning with attention-based neural networks[J].arXiv：1804.00823，2018.
[70] YAN H，DENG B，LI X，et al.TENER：adapting transformer encoder for named entity recognition[J].arXiv：1911.04474，2019.
[71] XU C，ZHOU W，GE T，et al.BERT-of-Theseus：compressing BERT by progressive module replacing[J].arXiv：2002.02925，2020.
[72] WANG K，ZONG C，SU K Y.Which is more suitable for Chinese word segmentation，the generative model or the discriminative one?[C]//Proceedings of the 23rd Pacific Asia Conference on Language，Information and Computation，2009，2：827-834.
[73] RASOOLI M S，FAILI H.Fast unsupervised dependency parsing with arc-standard transitions[C]//Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP，2012：1-9.
[74] 张李义，李亚子.基于反序词典的中文逆向最大匹配分词系统设计[J].现代图书情报技术，2006（8）：42-45.
ZHANG L Y，LI Y Z.A Chinese reverse-order directional maximum mathching segmentation system design based converse dictionary[J].New Technology of Library and Information Service，2006（8）：42-45.
[75] XU W.LSTM Shift-reduce CCG parsing[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：1754-1764.
[76] LUO R，XU J，ZHANG Y，et al.PKUSEG：a toolkit for multi-domain Chinese word segmentation[J].arXiv：1906.
11455，2019.
[77] ZHANG Y，ZHOU H，LI Z.Fast and accurate neural CRF constituency parsing[J].arXiv：2008.03736，2020.
[78] KUMAR A，ALBUQUERQUE V H C.Sentiment analysis using XLM-R transformer and zero-shot transfer learning on resource-poor Indian language[J].Transactions on Asian and Low-Resource Language Information Processing，2021，20（5）：1-13.
[79] HONNIBAL M，GOLDBERG Y，JOHNSON M.A non-monotonic arc-eager transition system for dependency parsing[C]//Proceedings of the 17th Conference on Computational Natural Language Learning，2013：163-172.
[80] TOUTANOVA K，KLEIN D，MANNING C D，et al.Feature-rich part-of-speech tagging with a cyclic dependency network[C]//Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics，2003：252-259.
[81] ZHU M，ZHANG Y，CHEN W，et al.Fast and accurate shift-reduce constituent parsing[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2013：434-443.
[82] PELLEG D，MOORE A.Accelerating exact k-means algorithms with geometric reasoning[C]//Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，1999：277-281.
[83] HINTON G E，NEAL R M.A view of the EM algorithm that justifies incremental，sparse，and other variants[M]//Learning in graphical models.Dordrecht：Springer，1998：355-368.
[84] MCCANN B，BRADBURY J，XIONG C，et al.Learned in translation：contextualized word vectors[C]//Advances in Neural Information Processing Systems 30，2017.
[85] RIBEIRO M T，SINGH S，GUESTRIN C.Semantically equivalent adversarial rules for debugging NLP models[C]//Proceedings of the 56th Annual meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2018：856-865.
[86] IYYER M，WIETING J，GIMPEL K，et al.Adversarial example generation with syntactically controlled paraphrase networks[J].arXiv：1804.06059，2018.
[87] ZHAO Z，DUA D，SINGH S.Generating natural adversarial examples[J].arXiv：1710.11342，2017.
[88] JIN D，JIN Z，ZHOU J T，et al.Is BERT really robust? A strong baseline for natural language attack on text classification and entailment[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：8018-8025.
[89] REN S，DENG Y，HE K，et al.Generating natural language adversarial examples through probability weighted word saliency[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：1085-1097.
[90] ALZANTOT M，SHARMA Y，ELGOHARY A，et al.Generating natural language adversarial examples[J].arXiv：1804.
07998，2018.
[91] PAPERNOT N，MCDANIEL P，SWAMI A，et al.Crafting adversarial input sequences for recurrent neural networks[C]//Proceedings of the 2016 IEEE Military Communications Conference，2016：49-54.
[92] LI J，JI S，DU T，et al.TextBugger：generating adversarial text against real-world applications[J].arXiv：1812.05271，2018.
[93] WALLACE E，FENG S，KANDPAL N，et al.Universal adversarial triggers for attacking and analyzing NLP[J].arXiv：1908.07125，2019.
[94] EBRAHIMI J，RAO A，LOWD D，et al.HotFlip：white-box adversarial examples for text classification[J].arXiv：1712.
06751，2017.
[95] EGER S，?AHIN G G，RüCKLé A，et al.Text processing like humans do：visually attacking and shielding NLP systems[J].arXiv：1903.11508，2019.
[96] GAO J，LANCHANTIN J，SOFFA M L，et al.Black-box generation of adversarial text sequences to evade deep learning classifiers[C]//Proceedings of the 2018 IEEE Security and Privacy Workshops，2018：50-56.
[97] SNELL J，SWERSKY K，ZEMEL R.Prototypical networks for few-shot learning[C]//Advances in Neural Information Processing Systems 30，2017.
[98] WESTON J，CHOPRA S，BORDES A.Memory networks[J].arXiv：1410.3916，2014.
[99] ESTER M，KRIEGEL H P，SANDER J，et al.A density-based algorithm for discovering clusters in large spatial databases with noise[C]//Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining，1996：226-231.
[100] 刘斌.基于Mask-RCNN的图像中文描述生成器[D].北京：北京理工大学，2018.
LIU B.Mask-RCNN based image chinese caption generator[D].Beijing：Beijing Institute of Technology，2018.
[101] RECASENS M，MARNEFFE M D，POTTS C.The life and death of discourse entities：identifying singleton mentions[C]//Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2013：627-633.
[102] LEE H，PEIRSMAN Y，CHANG A，et al.Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task[C]//Proceedings of the 15th Conference on Computational Natural Language Learning：Shared Task，2011：28-34.
[103] RAGHUNATHAN K，LEE H，RANGARAJAN S，et al.A multi-pass sieve for coreference resolution[C]//Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing，2010：492-501.
[104] LEE H，CHANG A，PEIRSMAN Y，et al.Deterministic coreference resolution based on entity-centric，precision-ranked rules[J].Computational Linguistics，2013，39（4）：885-916.
[105] CLARK K，MANNING C D.Entity-centric coreference resolution with model stacking[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2015：1405-1415.
[106] CLARK K，MANNING C D.Deep reinforcement learning for mention-ranking coreference models[J].arXiv：1609.08667，2016.
[107] KURNIASARI L，SETYANTO A.Sentiment analysis using recurrent neural network[J].Journal of Physics Conference Series，2020，1471：012018.
[108] SURDEANU M，MCCLOSKY D，SMITH M，et al.Customizing an information extraction system to a new domain[C]//Proceedings of the ACL 2011 Workshop on Relational Models of Semantics，2011：2-10.
[109] AMEED H，KETBI S，KAABI A，et al.Arabic light stemmer：a new enhanced approach[C]//Proceedings of the 2nd International Conference on Innovations in Information Technology，2005.
[110] ABAINIA K，REBBANI H.Comparing the effectiveness of the improved ARLSTem algorithm with existing Arabic light stemmers[C]//Proceedings of the 2019 International Conference on Theoretical and Applicative Aspects of Computer Science，Skikda，2019：1-8.
[111] WEISSWEILER L，FRASER A.Developing a stemmer for German based on a comparative analysis of publicly available stemmers[C]//Proceedings of the 2017 International Conference of the German Society for Computational Linguistics and Language Technology.Cham：Springer，2017：81-94.
[112] TAGHVA K，ELKHOURY R，COOMBS J.Arabic stemming without a root dictionary[C]//Proceedings of the 2005 International Conference on Information Technology：Coding and Computing，2005：152-157.
[113] PAICE C D.Another stemmer[J].ACM SIGIR Forum，1990，24（3）：56-61.
[114] PORTER M F.An algorithm for suffix stripping[J].Program，1980，14（3）：130-137.
[115] YANG Z，DAI Z，YANG Y，et al.XLNet：generalized autoregressive pretraining for language understanding[C]//Advances in Neural Information Processing Systems 32，2019.
[116] REIMERS N，GUREVYCH I.Sentence-BERT：sentence embeddings using Siamese BERT-networks[J].arXiv：1908.
10084，2019.
[117] BARROW J，PESKOV D.UMDeep at SemEval-2017 task 1：end-to-end shared weight LSTM model for semantic textual similarity[C]//Proceedings of the 11th International Workshop on Semantic Evaluation，2017：180-184.
[118] HOU M，SONG Y H，XU D，et al.A multi-pattern matching algorithm based on double array Trie[M]//Advances in Computer Science and Its Applications.Cham：Springer，2014：863-868.
[119] RANJAN M N M，GHORPADE Y R，KANTHALE G R，et al.Document classification using LSTM neural network[J].Journal of Data Mining and Management，2017，2（2）：1-9.