Meta-Learning Method of Uyghur Morphological Segmentation

doi:10.3778/j.issn.1002-8331.2201-0087

Abstract

Abstract: With the development of deep learning, the accuracy of Uyghur morphological segmentation has been dramatically improved, but the demand for data volume is high, while meta-learning method can effectively alleviate the model’s reliance on data volume by learning from previous tasks, and is widely used in low-resource domains. Therefore, the meta-learning method of Uyghur morphological segmentation is proposed, which focuses on fast generalisation on new tasks by training on previous tasks and obtaining a set of parameters with the ability to quickly adapt to new tasks. The experiments are first constructed with N pseudo-meta-learning tasks based on the similarity of the data for the partitioning of meta-learning support sets and query sets. Afterwards, the Uyghur data is encoded using Transformer’s encoder. Finally, the meta-learning method is used to achieve morphological segmentation for Uyghur language in few shot environments. Experimental results show the meta-learning method outperforms the pre-trained model in the few shot task, effectively avoiding overfitting of the model and mitigating the impact of data sparsity on the model.

Key words: meta-learning, morphological segmentation, Uyghur

摘要： 随着深度学习的发展，维吾尔语形态切分的准确率得到了大幅提升，但对数据量的需求较高，而元学习方法通过对以往任务的学习，有效缓解了模型对数据量的依赖，在低资源领域应用广泛。因此提出维吾尔语形态切分的元学习方法，该方法主要通过对以往任务的训练，获得一组具有快速适应新任务能力的参数，从而在新任务上实现快速泛化。实验根据数据的相似度构建[N]个伪元学习任务，完成元学习支撑集和查询集的划分，使用Transformer的编码器对维吾尔语数据进行编码，采用元学习方法实现对少样本环境下的维吾尔语形态切分。实验结果表明，在维吾尔语形态切分的少样本任务中元学习方法优于预训练模型，有效避免了模型的过拟合，缓解了数据稀疏性对模型的影响。

关键词: 元学习, 形态切分, 维吾尔语

ZHANG Yuning, LI Wenzhuo, Abudukelimu Halidanmu, Abulizi Abudukelimu. Meta-Learning Method of Uyghur Morphological Segmentation[J]. Computer Engineering and Applications, 2023, 59(11): 98-104.

张雨宁, 李文卓, 哈里旦木·阿布都克里木, 阿布都克力木·阿布力孜. 维吾尔语形态切分的元学习方法[J]. 计算机工程与应用, 2023, 59(11): 98-104.

References

[1] 李凡长，刘洋，吴鹏翔，等.元学习研究综述[J].计算机学报，2021，44（2）：422-446.
LI F C，LIU Y，WU P X，et al.A survey on recent advances in meta-learning[J].Chinese Journal of Computers，2021，44（2）：422-446.
[2] 哈里旦木·阿布都克里木，程勇，刘洋，等.基于双向门限递归单元神经网络的维吾尔语形态切分[J].清华大学学报（自然科学版），2017，57（1）：1-6.
ABUDUKELIMU H，CHENG Y，LIU Y，et al.Uyghur morphological segmentation with bidirectional GRU neural networks[J].Journal of Tsinghua University（Science and Technology），2017，57（1）：1-6.
[3] CHELSEA F，PIETER A，SERGEY L.Model-agnostic meta-learning for fast adaptation of deep networks[C]//International Conference on Machine Learning，2017：1126-1135.
[4] NICHOL A，SCHULMAN J.Reptile：a scalable meta learning algorithm[J].arXiv：1803.02999，2018.
[5] SUN Q，LIU Y，CHUA T S，et al.Meta-transfer learning for few-shot learning[C]//Proceeding of the IEEE Conference on Computer Vision，Seoul，Korea（South），2019：403-412.
[6] RUSU A A，RAO D，SYGNOWSKI J，et al.Meta-learning with latent embedding optimization[C]//ICLR，2019.
[7] ZHANG R，CHE T，GHAHRAMANI Z，et al.MetaGAN：an adversarial approach to few-shot learning[C]//Advances in Neural Information Processing Systems，2018：975-985.
[8] GU J T，WANG Y，CHEN Y，et al.Meta-learning for low-resource neural machine translation[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing，2018：3622-3631.
[9] DOU Z Y，YU K，ANASTASOPOULOS A.Investigating meta-learning algorithms for low-resource natural language understanding tasks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP），2019：1192-1197.
[10] SUN Y，TANG D，DUAN N，et al.Neural semantic parsing in low-resource settings with back-translation and meta-learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：8960-8967.
[11] DONG M Q，PAN C，LUO Z.MapRE：an effective semantic mapping approach for low-resource relation extraction[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing，2021：2694-2704.
[12] ZENG J L，YIN Y，LIU Y，et al.Domain adaptive meta-learning for dialogue state tracking[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021，29：2493-2501.
[13] ZHAO Y，TIAN Z，YAO H，et al.Improving meta-learning for low-resource text classification and generation via memory imitation[J].arXiv：2203.11670，2022.
[14] 赵凯琳，靳小龙，王元卓.小样本学习研究综述[J].软件学报，2021，32（2）：349-369.
ZHAO K L，JIN X L，WANG Y Z.Survey on few-shot learning[J].Journal of Software，2021，32（2）：349-369.
[15] LU J，GONG P，YE J，et al.Learning from very few samples：a survey[J].arXiv：2009.02653，2020.
[16] 刘畅，阿布都克力木·阿布力孜，姚登峰，等.维吾尔语形态分析研究综述[J].计算机工程与应用，2021，57（15）：42-61.
LIU C，ABUDUKELIMU A，YAO D F，et al.Survey for Uyghur morphological analysis[J].Computer Engineering and Applications，2021，57（15）：42-61.
[17] ORHUN M，TANTUG A，ADALO E.Rule based analysis of the Uyghur nouns[J].International Journal on Asian Language Processing，2008，19（1）：33-43.
[18] 古丽拉·阿东别克，米吉提·阿布力米提.维吾尔语词切分析法初探[J].中文信息学报，2004（6）：61-65.
GULILA A，MIJIT M.A Preliminary study on the segmentation methods of Uygur words[J].Journal of Chinese Information Processing，2004（6）：61-65.
[19] AISHA B，SUN M.A statistical method for Uyghur tokenization[C]//Natural Language Processing and Knowledge Engineering，2009.
[20] LAFFERTY J，MCCALLUM A，PEREIRA F.Conditional random fields：probabilistic models for segmenting and labeling sequence data[J/OL].Departmental Papers（CIS），2001[2021-12-20].https：//repository.upenn.edu/cis_papers/159.
[21] MAHMOUD A，PATTARL A，HAMDULLA A.Uyghur stemming using conditional random fields[J].International Journal of Signal Processing，Image Processing and Pattern Recognition，2015，8（8）：43-50.
[22] MI C，YANG Y，DONG R，et al.Log-linear model for Uyghur segmentation in spoken language translation[C]//Proceedings of the International Conference Recent Advances in Natural Language Processing，2017：492-500.
[23] ABULIMITI A，SCHULTZ T.Building language models for morphological rich low-resource languages using data from related donor languages：the case of Uyghur[C]//Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-Resourced Languages（SLTU） and Collaboration and Computing for Under-Resourced Languages（CCURL），2020：271-276.
[24] 薛化建，董兴华，王磊，等.基于词缀库的非监督维吾尔语词切分析法[J].计算机工程与设计，2011，32（9）：3191-3194.
XUE H J，DONG X H，WANG L，et al.Unsupervised Uyghur word segmentation method based on affix database[J].Computer Engineering and Design，2011，32（9）：3191-3194.
[25] ABLIMIT M，HAMDULLA A，KAWAHARA T.Morpheme concatenation approach in language modeling for large-vocabulary Uyghur speech recognition[C]//2011 International Conference on Speech Database and Assessments，2011.
[26] LIU C，ZHANG Z，ZHANG P，et al.Character-aware sub-word level language modeling for Uyghur and Turkish ASR[C/OL]//InterSpeech，2019：3495-3499[2021-03-27].http：//www.isca-speech.org/archive/Interspeech_2019/abstracts/1484.html.
[27] YANG Y，LI S，ZHANG Y，et al.Point the point：Uyghur morphological segmentation using pointer network with GRU[C]//China National Conference on Computational Linguistics，2019.
[28] WU Q，LIN Z，WANG G，et al.Enhanced meta-learning for cross-lingual named entity recognition with minimal resources[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：9274-9281.
[29] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//NIPS，2017.
[30] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics，2019：4171-4186.
[31] ZHU Y，KIROS R，ZEMEL R，et al.Aligning books and movies：towards story-like visual explanations by watching movies and reading books[C]//2015 IEEE International Conference on Computer Vision（ICCV）.Santiago，Chile：IEEE，2015：19-27.
[32] SANG E F，DE MEULDER F.Introduction to the CoNLL-2003 shared task：language-independent named entity recognition[J].arXiv preprint cs/0306050，2003.
[33] CHI Z，DONG L，WEI F，et al.InfoXLM：an information-theoretic framework for cross-lingual language model pre-training[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2021：3576-3588.
[34] WOLF T，DEBUT L，SANH V，et al.Transformers：state-of-the-art natural language processing[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing：System Demonstrations，2020：38-45.