Application of Non-Autoregressive Translation Model in Mongolian and Chinese Translation

doi:10.3778/j.issn.1002-8331.2011-0275

Abstract

Abstract: Most machine translation models are autoregressive models. To solve the problems existing in the autoregressive models, this paper conducts knowledge distillation and cross-language word embedding on corpus, on the translation models of Transformer and non-autoregressive Transformer. Experimental results show that non-autoregressive Transformer models with knowledge distillation achieve significant improvements in terms of BLEU and improve the generation rate. Experimental results show that knowledge distillation can significantly reduce the dependence between source and target language and improve the BLEU of Mongolian-Chinese machine translation, compared to Transformer models, BLEU values are improved by 2.8 and time consumption is reduced by 19.34 hours.

Key words: Transformer models, non-autoregressive Transformer（NAT）, knowledge distillation, cross-language word embedding

摘要： 当前大多数机器翻译模型都属于自回归模型，不支持解码器并行生成翻译结果，且生成速率过低。针对当前自回归模型中存在的问题，基于Transformer和非自回归Transformer（non-autoregressive Transformer，NAT）的翻译模型进行实验，对蒙汉语料进行知识蒸馏和语跨语言词语嵌入的处理。实验结果表明，引入知识蒸馏的NAT模型在BLEU值方面有显著提升，同时也提高了模型生成速率。NAT模型进行知识蒸馏与跨语言词嵌入处理后能显著减少源语言和目标语言之间的依赖关系，提高蒙汉机器翻译的BLEU值，相比Transformer模型，BLEU值提高了2.8，时间消耗减少了19.34?h。

关键词: Transformer模型, NAT模型, 知识蒸馏, 跨语言词嵌入

ZHAO Xu, SU Yila, RENQING Dao’erji, SHI Bao. Application of Non-Autoregressive Translation Model in Mongolian and Chinese Translation[J]. Computer Engineering and Applications, 2022, 58(12): 310-316.

赵旭, 苏依拉, 仁庆道尔吉, 石宝. 非自回归翻译模型在蒙汉翻译上的应用[J]. 计算机工程与应用, 2022, 58(12): 310-316.

References

[1] HUTCHINS W J.Machine translation：past，present，future[M].Chichester，UK：Ellis Horwood，Ltd，1986：55.
[2] CHIANG D.Hierarchical phrase-based translation[J].Computational Linguistics，2007，33（2）：201-228.
[3] 百顺.基于派生文法的日-蒙动词短语机器翻译研究[J].中文信息学报，2008，22（2）：47-54.
BAI S.Research on Japanese-Mongolian machine translation of verb phrase based on the derivational grammar[J].Journal of Chinese Information Processing，2008，22（2）：47-54.
[4] 任众，侯宏旭，吉亚图，等.子字粒度切分在蒙汉神经机器翻译中的应用[J].中文信息学报，2019，33（1）：85-92.
REN Z，HOU H X，JI Y T，et al.Application of sub-word segmentation in Mongolian-Chinese neural machine translation[J].Journal of Chinese Information Processing，2019，33（1）：85-92.
[5] 苏依拉，乌尼尔，刘婉婉.基于统计分析的蒙汉自然语言机器翻译[J].北京工业大学学报，2017，43（1）：36-42.
SU Y L，WU N E，LIU W W.Machine translation of Mongolian and Chinese natural language based on statistical analysis[J].Journal of Beijing University of Technology，2017，43（1）：36-42.
[6] 苏依拉，刘婉婉，乌尼尔.基于实例相似度检测的蒙汉机器翻译方法[J].北京工业大学学报，2017，43（9）：1366-1372.
SU Y L，LIU W W，WU N E.Method of Mongolian and Chinese machine translation based on instance similarity detection[J].Journal of Beijing University of Technology，2017，43（9）：1366-1372.
[7] SUTSKEVER I，VINYALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Proceeding of Advances in Neural Information Processing Systems，Montreal，2014：3104-3112.
[8] 杜健.融合统计机器翻译特征的蒙汉神经网络机器翻译技术[D].呼和浩特：内蒙古大学，2017.
DU J.Mongolian-Chinese neural network machine translation based on the features of statistical machine translation[D].Hohhot：Inner Mongolia University，2017.
[9] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Proceeding of Advances in Neural Information Processing Systems，2017：5998-6008.
[10] LUONG T，PHAN H，MANING C D.Bilingual word representations with monolingual quality in mind[C]//Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing，2015：151-159.
[11] YOON K，ALEXANDER R.Sequence-level knowledge distillation[C]//Proceeding of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：1317-1327.
[12] GU J，BRADBURY J，XIONG C，et al.Non-autoregressive neural machine translation[C]//Proceedings of the 6th International Conference on Learning Representations（ICLR），Vancouver，Canada，2018.
[13] CHRIS D，VICTOR C，NOAH S.A simple，fast，and effective reparameterization of IBM model 2[C]//The North American Chapter of the Association for Computational Linguistics，Berlin，Germany，2013：644-648.
[14] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[15] MIKEL A，GORKA L，ENEKO A.Learning bilingual word embeddings with （almost） no bilingual data[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，Vancouver，Canada，2017：451-462.
[16] CONNEAU A，LAMPLE G，RANZATO，M A，et al.Word translation without parallel data[J].arXiv：1710.04087，2017.
[17] LIU H B，LEE I.Bridging the gap between training and inference for spatio-temporal forecasting[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，Florence，Italy，2019：4334-4343.
[18] ZHOU C，NEUBIG G，GU J.Understanding knowledge distillation in non-autoregressive machine translation[C]//Proceedings of International Conference on Learning Representations，2020：186-195.
[19] SENNRICH R，HADDOW B，BIRCH A.Neural machine translation of rare words with subword units[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics，2016：1715-1725.
[20] BARUN P，?JOEL R A M，?SARTHAK G，?et al.Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，Florence，Italy，2019：184-193.
[21] SHAULI R，YANAI E，JACOB G，et al.Unsupervised distillation of syntactic information from contextualized word representations[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：91-106.