Research on Sentence Length Sensitivity in Neural Network Machine Translation

doi:10.3778/j.issn.1002-8331.2102-0274

Abstract

Abstract: With the development of deep learning, neural network machine translation has made considerable progress.It is well known that neuro-machine translation is sensitive to sentence length. In order to make full use of the large number of parallel corpus, this paper divides the original parallel corpus into several modules, trains a sub-model for each module, and proposes a neuro-machine translation method based on sentence length fusion strategy. At the end of the training, the translations are obtained by model fusion and three-feature（confusion, sentence length ratio and classifier） fusion sorting methods after the division of sentence length boundaries. The experimental results show that the BLEU points are increased by about 1.2 in English and Chinese tasks on three different test sets and 0.4 to 0.6 in Uyghur tasks. This method has some reference value.

Key words: machine translation, extreme sentence length data, perplexity（PPL）, ensemble, deep learning

摘要： 随着深度学习的发展神经网络机器翻译有了长足的进步。众所周知，神经机器翻译方法对句子长度比较敏感。为了充分利用海量平行语料，考虑平行语料句子长度信息，把原平行语料划分若干个模块，为每一个模块训练一个子模型，提出一种按句子长度融合策略的神经机器翻译方法。当训练结束时，通过句长边界划分后的模型融合与三特征（困惑度、句长比与分类器）融合排序方法得到译文。实验结果表明，提出的方法在三个不同测试集上英中任务中平均提高了1.2左右的BLEU点，维汉任务中提升了0.4至0.6的BLEU点。说明该方法具有一定的参考意义。

关键词: 机器翻译, 极端句长数据, 困惑度, 融合, 深度学习

Alim Samat, Sirajahmat Ruzmamat, 　Maihefureti, 　Aishan Wumaier, 　Wushuer　Silamu, Turgun Ebrayim. Research on Sentence Length Sensitivity in Neural Network Machine Translation[J]. Computer Engineering and Applications, 2022, 58(9): 195-200.

阿里木·赛买提, 斯拉吉艾合麦提·如则麦麦提, 麦合甫热提, 艾山·吾买尔, 吾守尔·斯拉木, 吐尔根·依不拉音. 神经机器翻译面对句长敏感问题的研究[J]. 计算机工程与应用, 2022, 58(9): 195-200.

References

[1] 刘庆峰，刘晨璇，王亚楠，等.会议场景下融合外部词典知识的领域个性化机器翻译方法[J].中文信息学报，2019，33（10）：31-37.
LIU Q F，LIU C X，WANG Y N，et al.Domain-specific machine translation with external dictionary knowledge for conference scenarios[J].Journal of Chinese Information Processing，2019，33（10）：31-37
[2] ARTHUR P，NEUBIG G，NAKAMURA S.Incorporating discrete translation lexicons into neural machine translation[J].arXiv：1606.02006，2016.
[3] LUONG M T，SUTSKEVER I，LE Q V，et al.Addressing the rare word problem in neural machine translation[J].arXiv：1410.8206，2014.
[4] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[5] WU Y，SCHUSTER M，CHEN Z，et al.Google’s neural machine translation system：bridging the gap between human and machine translation[J].arXiv：1609.08144，2016.
[6] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[7] 李志峰，张家硕，洪宇，等.融合覆盖机制的多模态神经机器翻译[J].中文信息学报，2020，34（3）：44-55.
LI Z F，ZHANG J S，HONG Y，et al.Merging coverage mechanism for multimodal neural machine translation[J].Journal of Chinese Information Processing，2020，34（3）：44-55.
[8] SUTSKEVER I，VINYALS O，LE Q V.Sequenceto sequence learning with neural networks[C]//Proceedings of the Neural Information Processing Systems（NIPS 2014），Montreal，Canada，2014：3104-3112.
[9] DEVLIN J，CHANG M W，LEE K，et al.Bert：pretraining of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[10] SENNRICH R，HADDOW B，BIRCH A.Neural machine translation of rare words with subword units[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2016：1715-1725.
[11] VASWANI A，BENGIO S，BREVDO E，et al.Tensor2tensor for neural machine translation[J].arXiv：1803.07416，2018.
[12] LI J，TU Z，YANG B，et al.Multi-head attention with disagreement regularization[J].arXiv：1810.10183，2018.
[13] LI Z，WANG R，CHEN K，et al.Explicit sentence compression for neural machine translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：8311-8318.
[14] KUMAR K，REZAGHOLIZADEH M，LAU Y S，et al.Improving neural machine translation with compact word embedding tables[J].arXiv：2104.08677，2021.

[1]	LUO Xianglong, GUO Huang, LIAO Cong, HAN Jing, WANG Lixin. Spatiotemporal Short-Term Traffic Flow Prediction Based on Broad Learning System [J]. Computer Engineering and Applications, 2022, 58(9): 181-186.
[2]	CHEN Yixiao, Alifu·Kuerban, LIN Wenlong, YUAN Xu. CA-YOLOv5 for Crowded Pedestrian Detection [J]. Computer Engineering and Applications, 2022, 58(9): 238-245.
[3]	FANG Yiqiu, LU Zhuang, GE Junwei. Forecasting Stock Prices with Combined RMSE Loss LSTM-CNN Model [J]. Computer Engineering and Applications, 2022, 58(9): 294-302.
[4]	GAO Guangshang. Survey on Attention Mechanisms in Deep Learning Recommendation Models [J]. Computer Engineering and Applications, 2022, 58(9): 9-18.
[5]	JI Meng, HE Qinglong. AdaSVRG: Accelerating SVRG by Adaptive Learning Rate [J]. Computer Engineering and Applications, 2022, 58(9): 83-90.
[6]	SHI Jie, YUAN Chenxiang, DING Fei, KONG Weixiang. Survey of Building Target Detection in SAR Images [J]. Computer Engineering and Applications, 2022, 58(8): 58-66.
[7]	CHEN Qin, CHEN Lanlan, JIANG Runqiang. Emotion Recognition of EEG Based on Ensemble CapsNet [J]. Computer Engineering and Applications, 2022, 58(8): 175-184.
[8]	XIONG Fengguang, ZHANG Xin, HAN Xie, KUANG Liqun, LIU Huanle, JIA Jionghao. Research on Improved Semantic Segmentation of Remote Sensing [J]. Computer Engineering and Applications, 2022, 58(8): 185-190.
[9]	YANG Jinfan, WANG Xiaoqiang, LIN Hao, LI Leixiao, YANG Yanyan, LI Kecen, GAO Jing. Review of One-Stage Vehicle Detection Algorithms Based on Deep Learning [J]. Computer Engineering and Applications, 2022, 58(7): 55-67.
[10]	HAN Haiyun, YANG Youlong, SUN Liqin. Multi-Instance Ensemble Algorithm Combined with Fuzzy Clustering [J]. Computer Engineering and Applications, 2022, 58(7): 87-96.
[11]	WANG Bin, LI Xin. Research on Multi-Source Domain Adaptive Algorithm Integrating Dynamic Residuals [J]. Computer Engineering and Applications, 2022, 58(7): 162-166.
[12]	TAN Shuqiu, TANG Guofang, TU Yuanya, ZHANG Jianxun, GE Panjie. Classroom Monitoring Students Abnormal Behavior Detection System [J]. Computer Engineering and Applications, 2022, 58(7): 176-184.
[13]	ZHANG Meiyu, LIU Yuehui, HOU Xianghui, QIN Xujia. Automatic Coloring Method for Gray Image Based on Convolutional Network [J]. Computer Engineering and Applications, 2022, 58(7): 229-236.
[14]	ZHANG Zhuangzhuang, QU Licheng, LI Xiang, ZHANG Minghao, LI Zhaolu. Traffic Flow Prediction with Missing Data Based on Spatial-Temporal Convolutional Neural Networks [J]. Computer Engineering and Applications, 2022, 58(7): 259-265.
[15]	XU Jie, ZHU Yukun, XING Chunxiao. Research on Financial Trading Algorithm Based on Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2022, 58(7): 276-285.