Survey of Pivot Methods for Neural Machine Translation

doi:10.3778/j.issn.1002-8331.2201-0227

Abstract

Abstract: Neural machine translation（NMT）for low-resource languages has always been a difficult and hot research topic in the field of machine translation, and pivot-based methods provide ideas for improving its performance. Aiming at the application of the pivot idea in low-resource NMT, this paper divides the research into three aspects：pivot translation, pivot-based pseudo-parallel data generation and pivot-based model construction, then analyzes and compares the research status at home and abroad, main problems and trends of different methods, which can provide a reference for the research in this field.

Key words: pivot methods, neural machine translation（NMT）, low-resource machine translation

摘要： 低资源语言的神经机器翻译（neural machine translation，NMT）一直是机器翻译领域研究的难点和热点，基于枢轴的方法为其性能的提升和改进提供了思路。针对枢轴思想在低资源语言神经机器翻译中的应用，从枢轴翻译、基于枢轴的伪平行数据生成和基于枢轴的模型构建三方面，对不同方法的国内外研究现状、主要问题和趋势进行了分析和比较，为该领域的研究提供参考和借鉴。

关键词: 枢轴方法, 神经机器翻译（NMT）, 低资源机器翻译

LI Jiaquan, WANG Liqing, LI Peng, JIANG Xiaomin, XU Yongyue. Survey of Pivot Methods for Neural Machine Translation[J]. Computer Engineering and Applications, 2022, 58(16): 49-55.

黎家全, 王丽清, 李鹏, 蒋晓敏, 徐永跃. 面向神经机器翻译的枢轴方法研究综述[J]. 计算机工程与应用, 2022, 58(16): 49-55.

References

[1] SUTSKEVER I，VINYALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems，2014：3104-3112.
[2] BELINKOV Y，DURRANI N，DALVI F，et al.On the linguistic representational power of neural machine translation models[J].Computational Linguistics，2020，46（1）：1-57.
[3] GEHRING J，AULI M，GRANGIER D，et al.Convolutional sequence to sequence learning[C]//Proceedings of the 34th International Conference on Machine Learning，2017：1243-1252.
[4] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[5] SENNRICH R，ZHANG B.Revisiting low-resource neural machine translation：a case study[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：211-221.
[6] KOEHN P，KNOWLES R.Six challenges for neural machine translation[C]//Proceedings of the 1st Workshop on Neural Machine Translation，2017：28-39.
[7] 李洪政，冯冲，黄河燕.稀缺资源语言神经网络机器翻译研究综述[J].自动化学报，2021，47（6）：1217-1231.
LI H Z，FENG C，HUANG H Y.A survey on low-resource neural machine translation[J].Acta Automatica Sinica，2021，47（6）：1217-1231.
[8] UTIYAMA M，ISAHARA H.A comparison of pivot methods for phrase-based statistical machine translation[C]//Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics，2007：484-491.
[9] WU H，WANG H.Pivot language approach for phrase-based statistical machine translation[J].Machine Translation，2007，21（3）：165-181.
[10] WU H，WANG H.Revisiting pivot language approach for machine translation[C]//Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP，2009：154-162.
[11] AHMADNIA B，SERRANO J，HAFFARI G.Persian-Spanish low-resource statistical machine translation through English as pivot language[C]//Proceedings of the 2017 International Conference Recent Advances in Natural Language Processing，2017：24-30.
[12] DE GISPERT A，MARINO J B.Catalan-English statistical machine translation without parallel corpus：bridging through Spanish[C]//Proceedings of the 5th International Conference on Language Resources and Evaluation，2006：65-68.
[13] TOKARCHUK E，ROSENDAHL J，WANG W，et al.Integrated training for sequence-to-sequence models using non-autoregressive transformer[C]//Proceedings of the 18th International Conference on Spoken Language Translation，2021：276-286.
[14] RUIZ COSTA-JUSSà M，CASAS N，MELERO M.English-catalan neural machine translation in the biomedical domain through the cascade approach[C]//Proceedings of the 2018 Workshop Multilingual BIO：Multilingual Biomedical Text Processing，2018：21-24.
[15] LIU S，WANG L，LIU C H.Chinese-Portuguese machine translation：a study on building parallel corpora from comparable texts[C]//Proceedings of the 11th International Conference on Language Resources and Evaluation，2018.
[16] SANTOS R，SILVA J，BRANCO A，et al.The direct path may not be the best：Portuguese-Chinese neural machine translation[C]//Proceedings of the 2019 EPIA Conference on Artificial Intelligence.Cham：Springer，2019：757-768.
[17] GUO H，LIU W，HE Y，et al.ISTIC’s triangular machine translation system for WMT2021[C]//Proceedings of the 6th Conference on Machine Translation，2021：320-324.
[18] FIRAT O，SANKARAN B，AL-ONAIZAN Y，et al.Zero-resource translation with multi-lingual neural machine translation[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：268-277.
[19] DABRE R，IMANKULOVA A，KANEKO M，et al.Simultaneous multi-pivot neural machine translation[J].arXiv：2104.07410，2021.
[20] NISHIMURA Y，SUDOH K，NEUBIG G，et al.Multi-source neural machine translation with missing data[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2019，28：569-580.
[21] AHMADNIA B，SERRANO J.Employing pivot language technique through statistical and neural machine translation frameworks：the case of under-resourced Persian-Spanish language pair[J].International Journal on Natural Language Computing，2017，6（5）：37-47.
[22] CHENG Y.Joint training for pivot-based neural machine translation[M]//Joint training for neural machine translation.Singapore：Springer，2019：41-54.
[23] REN S，CHEN W，LIU S，et al.Triangular architecture for rare language translation[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：56-65.
[24] PAUL M，FINCH A，SUMITA E.How to choose the best pivot language for automatic translation of low-resource languages[J].ACM Transactions on Asian Language Information Processing，2013，12（4）：1-17.
[25] LIU C H，SILVA C C，WANG L，et al.Pivot machine translation using chinese as pivot language[C]//Proceedings of the China Workshop on Machine Translation.Singapore：Springer，2018：74-85.
[26] LENG Y，TAN X，QIN T，et al.Unsupervised pivot translation for distant languages[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：175-183.
[27] SENNRICH R，HADDOW B，BIRCH A.Improving neural machine translation models with monolingual data[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics，2016：86-96.
[28] BERTOLDI N，BARBAIANI M，FEDERICO M，et al.Phrase-based statistical machine translation with pivot languages[C]//Proceedings of the 5th International Workshop on Spoken Language Translation：Papers，2008.
[29] 李强，王强，肖桐，等.稀缺资源机器翻译中改进的语料级和短语级中间语言方法研究[J].计算机学报，2017，40（4）：925-938.
LI Q，WANG Q，XIAO T.Research on improved corpus-level and phrase-level pivot language based methods in low-resource machine translation[J].Chinese Journal of Computers，2017，40（4）：925-938.
[30] CASAS N，FONOLLOSA J A R，ESCOLANO C，et al.The TALP-UPC machine translation systems for WMT19 news translation task：pivoting techniques for low resource MT[C]//Proceedings of the 4th Conference on Machine Translation，2019：155-162.
[31] LI B，LI Y，XU C，et al.The niutrans machine translation systems for wmt19[C]//Proceedings of the 4th Conference on Machine Translation，2019：257-266.
[32] PARK J，SONG J，YOON S.Building a neural machine translation system using only synthetic parallel data[J].arXiv：1704.00253，2017.
[33] CHOI G H，SHIN J H，KIM Y G.Improving a multi-source neural machine translation model with corpus extension for low-resource languages[C]//Proceedings of the 11th International Conference on Language Resources and Evaluation，2018.
[34] YANG H，HU B，XIE W，et al.TenTrans multilingual low-resource translation system for WMT21 Indo-European languages task[C]//Proceedings of the 6th Conference on Machine Translation，2021：376-382.
[35] COSTA-JUSSA M R，CASAS N，ESCOLANO C，et al.Chinese-Catalan：a neural machine translation approach based on pivoting and attention mechanisms[J].ACM Transactions on Asian and Low-Resource Language Information Processing，2019，18（4）：1-8.
[36] ZHENG H，CHENG Y，LIU Y.Maximum expected likelihood estimation for zero-resource neural machine translation[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，2017：4251-4257.
[37] CHEN Y，LIU Y，CHENG Y，et al.A teacher-student framework for zero-resource neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics，2017：1925-1935.
[38] CAI Y，ZHU M.A teacher-student model with the bilingual word embedding for zero-shot machine translation[J].Journal of Physics：Conference Series，2020，1693（1）：012218.
[39] CURREY A，HEAFIELD K.Zero-resource neural machine translation with monolingual pivot data[C]//Proceedings of the 3rd Workshop on Neural Generation and Translation，2019：99-107.
[40] KIDANE L，KUMAR S，TSVETKOV Y.An exploration of data augmentation techniques for improving English to Tigrinya Translation[J].arXiv：2103.16789，2021.
[41] XIA M，KONG X，ANASTASOPOULOS A，et al.Generalized data augmentation for low-resource translation[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：5786-5796.
[42] 贾承勋，赖华，余正涛，等.基于枢轴语言的汉越神经机器翻译伪平行语料生成[J].计算机工程与科学，2021，43（3）：542-550.
JIA C X，LAI H，YU Z T，et al.Pseudo-parallel corpus generation for Chinese-Vietnamese neural machine translation based on pivot language[J].Computer Engineering & Science，2021，43（3）：542-550.
[43] 赵小兵，金波，孙媛.基于枢轴语言系统融合的词汇混淆神经网络神经机器翻译[C]//第二十届中国计算语言学大会论文集，2021：46-56.
ZHAO X B，JIN B，SUN Y.Neural machine translation for vocabulary confusion network based on pivotal language system fusion）[C]//Proceedings of the 20th Chinese National Conference on Computational Linguistics，2021：46-56.
[44] ZOPH B，YURET D，MAY J，et al.Transfer learning for low-resource neural machine translation[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing，2016：1568-1575.
[45] KOCMI T，BOJAR O.Trivial transfer learning for low-resource neural machine translation[C]//Proceedings of the 3rd Conference on Machine Translation，2018：244-252.
[46] AJI A F，BOGOYCHEV N，HEAFIELD K，et al.In neural machine translation，what does transfer learning transfer?[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：7701-7710.
[47] KIM Y，PETROV P，PETRUSHKOV P，et al.Pivot-based transfer learning for neural machine translation between non-English languages[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing，2019：866-876.
[48] MHASKAR S，BHATTACHARYYA P.Pivot based transfer learning for neural machine translation：CFILT IITB@WMT 2021 triangular MT[C]//Proceedings of the 6th Conference on Machine Translation，2021：336-340.
[49] BANERJEE A，JAIN A，MHASKAR S，et al.Neural machine translation in low-resource setting：a case study in English-Marathi pair[C]//Proceedings of the 18th Biennial Machine Translation Summit，2021：35-47.
[50] YU Z，YU Z，HUANG Y，et al.Transfer learning for Chinese-Lao neural machine translation with linguistic similarity[C]//Proceedings of the China Conference on Machine Translation.Singapore：Springer，2020：1-10.
[51] JI B，ZHANG Z，DUAN X，et al.Cross-lingual pre-training based transfer for zero-shot neural machine translation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：115-122.
[52] DONG D，WU H，HE W，et al.Multi-task learning for multiple language translation[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing，2015：1723-1732.
[53] FIRAT O，CHO K，SANKARAN B，et al.Multi-way，multilingual neural machine translation[J].Computer Speech & Language，2017，45：236-252.
[54] LU Y，KEUNG P，LADHAK F，et al.A neural interlingua for multilingual machine translation[C]//Proceedings of the 3rd Conference on Machine Translation，2018：84-92.
[55] JOHNSON M，SCHUSTER M，LE Q V，et al.Google’s multilingual neural machine translation system：enabling zero-shot translation[J].Transactions of the Association for Computational Linguistics，2017，5：339-351.
[56] AHARONI R，JOHNSON M，FIRAT O.Massively multilingual neural machine translation[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：3874-3884.
[57] ARIVAZHAGAN N，BAPNA A，FIRAT O，et al.Massively multilingual neural machine translation in the wild：findings and challenges[J].arXiv：1907.05019，2019.
[58] DABRE R，CHU C，KUNCHUKUTTAN A.A survey of multilingual neural machine translation[J].ACM Computing Surveys，2020，53（5）：1-38.
[59] TAN X，CHEN J，HE D，et al.Multilingual neural machine translation with language clustering[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing，2019：963-973.
[60] ZHANG B，WILLIAMS P，TITOV I，et al.Improving massively multilingual neural machine translation and zero-shot translation[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：1628-1639.
[61] GU J，WANG Y，CHO K，et al.Improved zero-shot neural machine translation via ignoring spurious correlations[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：1258-1268.
[62] PHAM N Q，NIEHUES J，HA T L，et al.Improving zero-shot translation with language-independent constraints[C]//Proceedings of the 4th Conference on Machine Translation，2019：13-23.
[63] KUNCHUKUTTAN A.An empirical investigation of multi-bridge multilingual NMT models[J].arXiv：2110.07304，2021.
[64] RIOS A，MüLLER M，SENNRICH R.Subword segmentation and a single bridge language affect zero-shot neural machine translation[C]//Proceedings of the 5th Conference on Machine Translation，2020：528-537.
[65] FAN A，BHOSALE S，SCHWENK H，et al.Beyond english-centric multilingual machine translation[J].Journal of Machine Learning Research，2021，22（107）：1-48.
[66] CHEN Y，LIU Y，LI V O K.Zero-resource neural machine translation with multi-agent communication game[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018：5086-5093.
[67] NAKAYAMA H，NISHIDA N.Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot[J].Machine Translation，2017，31（1）：49-64.
[68] HUANG P Y，HU J，CHANG X，et al.Unsupervised multimodal neural machine translation with pseudo visual pivoting[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：8226-8237.
[69] WU C K，SHIH C C，WANG Y C，et al.Improving low-resource machine transliteration by using 3-way transfer learning[J].Computer Speech & Language，2022，72：101283.