{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T13:44:15Z","timestamp":1784382255544,"version":"3.55.0"},"reference-count":30,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T00:00:00Z","timestamp":1616025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The pre-training fine-tuning mode has been shown to be effective for low resource neural machine translation. In this mode, pre-training models trained on monolingual data are used to initiate translation models to transfer knowledge from monolingual data into translation models. In recent years, pre-training models usually take sentences with randomly masked words as input, and are trained by predicting these masked words based on unmasked words. In this paper, we propose a new pre-training method that still predicts masked words, but randomly replaces some of the unmasked words in the input with their translation words in another language. The translation words are from bilingual data, so that the data for pre-training contains both monolingual data and bilingual data. We conduct experiments on Uyghur-Chinese corpus to evaluate our method. The experimental results show that our method can make the pre-training model have a better generalization ability and help the translation model to achieve better performance. Through a word translation task, we also demonstrate that our method enables the embedding of the translation model to acquire more alignment knowledge.<\/jats:p>","DOI":"10.3390\/info12030133","type":"journal-article","created":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T11:41:06Z","timestamp":1616067666000},"page":"133","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Pre-Training on Mixed Data for Low-Resource Neural Machine Translation"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8932-923X","authenticated-orcid":false,"given":"Wenbo","family":"Zhang","sequence":"first","affiliation":[{"name":"The Xinjiang Technical Institute of Physics &amp; Chemistry, Chinese Academy of Sciences, Urumqi 830011, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"},{"name":"Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi 830011, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiao","family":"Li","sequence":"additional","affiliation":[{"name":"The Xinjiang Technical Institute of Physics &amp; Chemistry, Chinese Academy of Sciences, Urumqi 830011, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"},{"name":"Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi 830011, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yating","family":"Yang","sequence":"additional","affiliation":[{"name":"The Xinjiang Technical Institute of Physics &amp; Chemistry, Chinese Academy of Sciences, Urumqi 830011, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"},{"name":"Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi 830011, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Dong","sequence":"additional","affiliation":[{"name":"The Xinjiang Technical Institute of Physics &amp; Chemistry, Chinese Academy of Sciences, Urumqi 830011, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"},{"name":"Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi 830011, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,18]]},"reference":[{"key":"ref_1","unstructured":"Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Macherey, M.N.W., Krikun, J.M., Cao, Y., Cao, Q., Macherey, K., and Klinger, J. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv."},{"key":"ref_2","unstructured":"Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. (2017, January 6\u201311). Convolutional sequence to sequence learning. Proceedings of the 34th International Conference on Machine Learning-Volume 70, Sydney, Australia."},{"key":"ref_3","unstructured":"Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., and Li, M. (2018). Achieving human parity on automatic chinese to english news translation. arXiv."},{"key":"ref_4","unstructured":"Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8\u201313). Sequence to sequence learning with neural networks. Proceedings of the Advances in Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_5","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Luong, M.T., Pham, H., and Manning, C.D. (2015, January 17\u201321). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.","DOI":"10.18653\/v1\/D15-1166"},{"key":"ref_7","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., and Kaiser, L. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zoph, B., Yuret, D., May, J., and Knight, K. (2016). Transfer learning for low-resource neural machine translation. arXiv.","DOI":"10.18653\/v1\/D16-1163"},{"key":"ref_9","unstructured":"Nguyen, T.Q., and Chiang, D. (2017). Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation. arXiv."},{"key":"ref_10","unstructured":"Gulcehre, C., Firat, O., Xu, K., Cho, K., Barrault, L., Lin, H., Bougares, F., Schwenk, H., and Bengio, Y. (2015). On using monolingual corpora in neural machine translation. arXiv."},{"key":"ref_11","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2021, March 18). Improving Neural Machine Translation Models with Monolingual Data. Available online: https:\/\/arxiv.org\/abs\/1511.06709."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, J., and Zong, C. (2016, January 1\u20135). Exploiting source-side monolingual data in neural machine translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1160"},{"key":"ref_13","unstructured":"Xia, Y., He, D., Qin, T., Wang, L., Yu, N., Liu, T.Y., and Ma, W.Y. (2018, January 6\u201312). Dual learning for machine translation. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_14","unstructured":"Cheng, Y., Xu, W., He, Z., He, W., Wu, H., Sun, M., and Liu, Y. (2021, March 18). Semi-Supervised Learning for Neural Machine Translation. Available online: https:\/\/link.springer.com\/chapter\/10.1007\/978-981-32-9748-7_3."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gu, J., Wang, Y., Chen, Y., Cho, K., and Li, V.O.K. (November, January 31). Meta-Learning for Low-Resource Neural Machine Translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1398"},{"key":"ref_16","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_17","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_18","unstructured":"Lample, G., and Conneau, A. (2019). Crosslingual language model pretraining. arXiv."},{"key":"ref_19","unstructured":"Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.Y. (2019). Mass: Masked sequence to sequence pre-training for language generation. arXiv."},{"key":"ref_20","first-page":"79","article-title":"A statistical approach to machine translation","volume":"16","author":"Brown","year":"1990","journal-title":"Comput. Linguist."},{"key":"ref_21","unstructured":"Skorokhodov, I., Rykachevskiy, A., Emelyanenko, D., Slotin, S., and Ponkratov, A. (2021, March 18). Semi-Supervised Neural Machine Translation with Language Models. Available online: https:\/\/www.aclweb.org\/anthology\/W18-2205.pdf."},{"key":"ref_22","unstructured":"Zhu, J., Xia, Y., and Wu, L. (2020). Incorporating bert into neural machine translation. arXiv."},{"key":"ref_23","unstructured":"Guo, J., Zhang, Z., and Xu, L. (2020). Incorporating BERT into Parallel Sequence Decoding with Adapters. arXiv."},{"key":"ref_24","first-page":"263","article-title":"The mathematics of statistical machine translation: Parameter estimation","volume":"19","author":"Brown","year":"1993","journal-title":"Comput. Linguist."},{"key":"ref_25","unstructured":"Dyer, C., Chahuneau, V., and Smith, N.A. (2013, January 9\u201314). A simple, fast, and effective reparameterization of ibm model 2. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, GA, USA."},{"key":"ref_26","unstructured":"Koehn, P., Hoang, H., Birch, A., Burch, C.C., Federico, M., Bertoldi, N., Corwan, B., Shen, W., Moran, C., and Zens, R. (2021, March 18). Moses: Open Source Toolkit for Statistical Machine Translation. Available online: https:\/\/www.aclweb.org\/anthology\/P07-2.pdf."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv.","DOI":"10.18653\/v1\/P16-1162"},{"key":"ref_28","unstructured":"Diederik, K., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_29","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Nitish","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_30","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2021, March 18). BLEU: A Method for Automatic Evaluation of Machine Translation. Available online: https:\/\/www.aclweb.org\/anthology\/P02-1040.pdf."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/3\/133\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:37:39Z","timestamp":1760161059000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/3\/133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,18]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["info12030133"],"URL":"https:\/\/doi.org\/10.3390\/info12030133","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,18]]}}}