Agglutinative Languages Named Entity Recognition Based on Pruner and Multilingual Fine-Tuning

doi:10.3778/j.issn.1002-8331.2208-0109

Abstract

Abstract: Minority languages, represented by Uyghur, are characterized by agglutination and lack resources, which pose great challenges for their named entity recognition tasks. Meanwhile, the multilingual model suffers from problems such as large parameter scale, large word vocabularies, and slow inference speed. In order to explore the best fine-tuning strategy to alleviate the low-resource problem, monolingual and multilingual fine-tuning are performed for five agglutinative languages, namely Uyghur, Kazakh, Kirghiz, Uzbek, and Tatar, respectively. The experimental results show that CINO-Agglu reduces the model size, number of parameters, word list size, and inference time by 45%, 44%, 92%, and 38%, respectively, compared with the period before pruning, and the average F1 score on the five languages is 85.9%, which exceeds all baseline models. The inclusion of appropriately sized data from the same language branch is beneficial to enhance the fine-tuning effect.

Key words: agglutinative language, low-resource language, named entity recognition, cross-lingual transfer, model pruner

摘要： 以维吾尔语为代表的少数民族语言具有黏着性和资源匮乏等特点，为其命名实体识别任务带来了巨大挑战。与此同时，多语言模型存在参数规模和词表大、推理速度慢等问题。为此，通过对CINO进行重新剪枝，提出针对低资源黏着语命名实体识别的CINO新版本：CINO-Agglu。为了探讨最佳微调策略，缓解低资源问题，对维吾尔语、哈萨克语、柯尔克孜语、乌兹别克语、塔塔尔语等五种黏着语分别进行单语言和多语言微调。实验结果表明，CINO-Agglu相较于剪枝前，模型大小、参数量、词表大小、推理时间分别减少45%、44%、92%、38%，并且在五种语言上的平均F1值为85.9%，超过了所有基线模型。加入适当规模的同语族数据有利于提升微调效果。

关键词: 黏着语, 低资源语言, 命名实体识别, 多语言迁移, 模型剪枝

LUO Kai’ang, Abudukelimu Halidanmu, LIU Chang, Abudukelimu Abulizi, GUO Wenqiang. Agglutinative Languages Named Entity Recognition Based on Pruner and Multilingual Fine-Tuning[J]. Computer Engineering and Applications, 2023, 59(24): 121-130.

罗凯昂, 哈里旦木·阿布都克里木, 刘畅, 阿布都克力木·阿布力孜, 郭文强. 融合剪枝和多语微调的黏着语命名实体识别[J]. 计算机工程与应用, 2023, 59(24): 121-130.

References

[1] NI J，DINU G，FLORIAN R.Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers）.Vancouver，Canada：Association for Computational Linguistics，2017：1470-1480.
[2] CHEN S，PEI Y，KE Z，et al.Low-resource named entity recognition via the pre-training model[J].Symmetry，2021，13（5）：786.
[3] MUELLER D，ANDREWS N，DREDZE M.Sources of transfer in multilingual named entity recognition[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：8093-8104.
[4] CHEN W，JIANG H，WU Q，et al.AdvPicker：effectively leveraging unlabeled data via adversarial discriminator for cross-lingual NER[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2021：743-753.
[5] STOLLENWERK F.Adaptive fine-tuning of transformer-based language models for named entity recognition[J].arXiv：2202.02617，2022.
[6] 吐尔根·依布拉音，卡哈尔江·阿比的热西提，艾山·吾买尔，等.中亚语言自然语言处理综述[J].中文信息学报，2018，32（5）：1-13.
TUERGEN I，KAHAERJIANG A，AISHAN W，et al.A survey of central Asian language processing[J].Journal of Chinese Information Processing，2018，32（5）：1-13.
[7] 艾斯卡尔·肉孜，宗成庆，姑丽加玛丽·麦麦提艾力，等.基于条件随机场的维吾尔人名识别方法[J].清华大学学报（自然科学版），2013，53（6）：873-877.
ROZI A，ZONG C Q，MAMATELI G，et al.Approach to recognizing Uyhgur names based on conditional random fields[J].Journal of Tsinghua University（Science and Technology），2013，53（6）：873-877.
[8] 麦合甫热提，米日姑·肉孜，麦热哈巴·艾力，等.基于语法语义知识的维吾尔文机构名识别[J].计算机工程与设计，2014，35（8）：2944-2948.
MAIHEFURETI，ROUZI M，AILI M，et al.Uyghur organization name recognition based on syntactic and semantic knowledge[J].Computer Engineering and Design，2014，35（8）：2944-2948.
[9] 买合木提·买买提，卡哈尔江·阿比的热西提，艾山·吾买尔，等.CRF与规则相结合的维吾尔文地名识别研究[J].中文信息学报，2017，31（6）：110-118.
MAIHEMULI T M，KAHAERJIANG A，AISHAN W，et al.Uyghur location names recognition based on conditional random fields and rules[J].Journal of Chinese Information Processing，2017，31（6）：110-118.
[10] BHARADWAJ A，MORTENSEN D R，DYER C，et al.Phonologically aware neural model for named entity recognition in low resource transfer settings[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.Austin，Texas：Association for Computational Linguistics，2016：1462-1472.
[11] FENG X，HUANG L，QIN B，et al.Multi-level cross lingual attentive neural architecture for low resource name tagging[J].Tsinghua Science and Technology，2017，22（6）：633-645.
[12] XIE J，YANG Z，NEUBIG G，et al.Neural cross-lingual named entity recognition with minimal resources[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing，Brussels，Belgium，2018：369-379.
[13] 王路路，艾山·吾买尔，买合木提·买买提，等.基于CRF和半监督学习的维吾尔文命名实体识别[J].中文信息学报，2018，32（11）：16-26.
WANG L L，WUMAIER A，MAIMAITI M，et al.A semi-supervised approach to Uyghur named entity recognition based on CRF[J].Journal of Chinese Information Processing，2018，32（11）：16-26.
[14] HAISA G，ALTENBEK G.Deep learning with word embedding improves Kazakh named-entity recognition[J].Information，2022，13（4）：180.
[15] AYIFU M，WUSHOUER S，PAlIDAN M.Multilingual named entity recognition based on the BiGRU-CNN-CRF hybrid model[J].International Journal of Information and Communication Technology，2019，15（3）：223-242.
[16] AKHMED-ZAKI D，MANSUROVA M，BARAKHNIN V，et al.Development of Kazakh named entity recognition models[C]//International Conference on Computational Collective Intelligence.Cham：Springer，2020：697-708.
[17] 孔祥鹏，吾守尔·斯拉木，杨启萌，等.基于迁移学习的维吾尔语命名实体识别[J].东北师大学报（自然科学版），2020，52（2）：58-65.
KONG X P，SILAMU W，YANG Q M，et al.Uyghur named entity recognition based on transfer learning[J].Journal of Northeast Normal University（Nature Science Edition），2020，52（2）：58-64.
[18] MULLER B，ANASTASOPOULOS A，SAGOT B，et al.When being unseen from mBERT is just the beginning：handling new languages with multilingual language models[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2021：448-462.
[19] LEI S，ZHANG X，HE J，et al.Uncertainty-aware cross-lingual transfer with pseudo partial labels[C]//Findings of the Association for Computational Linguistics，2022：1987-1997.
[20] DEVLIN J，CHANG M W，LEE K，et al.Bert：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，Volume 1（Long and Short Papers），2019：4171-4186.
[21] BAUMANN A.Multilingual language models for named entity recognition in German and english[C]//Proceedings of the Student Research Workshop Associated with RANLP 2019，2019：21-27.
[22] WU Q，LIN Z，KARLSSON B F，et al.Unitrans：unifying model transfer and data transfer for cross-lingual named entity recognition with unlabeled data[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence（IJCAI2020），2020：3926-3932.
[23] LI Z，HU C，GUO X，et al.An unsupervised multiple-task and multiple-teacher model for cross-lingual named entity recognition[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2022：170-179.
[24] GALPERIN R，SCHNAPP S，ELHADAD M.Cross-lingual UMLS named entity linking using UMLS dictionary fine-tuning[C]//Findings of the Association for Computational Linguistics（ACL 2022），2022：3380-3390.
[25] LAMPLE G，CONNEAU A.Cross-lingual language model pretraining[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems，2019：7059-7069.
[26] CONNEAU A，KHANDELWAL K，GOYAL N，et al.Unsupervised cross-lingual representation learning at scale[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2019：8440-8451.
[27] LIU Y，OTT M，GOYAL N，et al.RoBERTa：a robustly optimized BERT pretraining approach[J].arXiv：1907.11692，2019.
[28] CHEN G，MA S，CHEN Y，et al.Towards making the most of cross-lingual transfer for zero-shot neural machine translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2022：142-157.
[29] CHI Z，DONG L，WEI F，et al.InfoXLM：an information-theoretic framework for cross-lingual language model pre-training[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2020：3576-3588.
[30] CHI Z，DONG L，ZHENG B，et al.Improving pretrained cross-lingual language models via self-labeled word alignment[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing，2021：3418-3430.
[31] YANG Z，XU Z，CUI Y，et al.CINO：a Chinese minority pre-trained language model[J].arXiv：2202.13558，2022.
[32] ABDAOUI A，PRADEL C，SIGEL G.Load what you need：smaller versions of multilingual BERT[C]//Proceedings of SustaiNLP：Workshop on Simple and Efficient Natural Language Processing，2020：119-123.
[33] LAGUNAS F，CHARLAIX E，SANH V，et al.Block pruning for faster transformers[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing，2021：10619-10629.
[34] YANG Z，CUI Y，CHEN Z.TextPruner：a model pruning toolkit for pre-trained language models[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics System Demonstrations，2022：35-43.
[35] PAN X，ZHANG B，MAY J，et al.Cross-lingual name tagging and linking for 282 languages[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2017：1946-1958.
[36] WOLF T，DEBUT L，SANH V，et al.Transformers：state-of-the-art natural language processing[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing：System Demonstrations，2020：38-45.
[37] WANG Z，LIPTON Z C，TSVETKOV Y.On negative interference in multilingual models：findings and a meta-learning treatment[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing（EMNLP），2020：4438-4450.
[38] DAI X，ADEL H.An analysis of simple data augmentation for named entity recognition[C]//Proceedings of the 28th International Conference on Computational Linguistics，Barcelona，Spain，2020：3861-3867.
[39] BUAPHET W，UDOMCHAROENCHAIKIT C，LIMKONCHOTIWAT P，et al.Thai nested named entity recognition corpus[C]//Findings of the Association for Computational Linguistics，2022：1473-1486.
[40] SHEN Y，WANG X，TAN Z，et al.Parallel instance query network for named entity recognition[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2022：947-961.
[41] ?AHIN G G，STEEDMAN M.Data augmentation via dependency tree morphing for low-resource languages[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels，Belgium：Association for Computational Linguistics，2018：5004-5009.
[42] LI B H，HOU Y T，CHE W X.Data augmentation approaches in natural language processing：a survey[J].arXiv：2110.
01852v3，2021.
[43] LIU L，DING B，BING L，et al.MulDA：a multilingual data augmentation framework for low-resource cross-lingual NER[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing（Volume 1：Long Papers），2021：5834-5846.
[44] ZHOU R，LI X，HE R，et al.MELM：data augmentation with masked entity language modeling for low-resource NER[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2022：2251-2262.