{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T12:30:49Z","timestamp":1770813049305,"version":"3.50.1"},"reference-count":47,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T00:00:00Z","timestamp":1713916800000},"content-version":"vor","delay-in-days":114,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The utilization of monolingual data has been shown to be a promising strategy for addressing low-resource machine translation problems. Previous studies have demonstrated the effectiveness of techniques such as back-translation and self-supervised objectives, including masked language modeling, causal language modeling, and denoise autoencoding, in improving the performance of machine translation models. However, the manner in which these methods contribute to the success of machine translation tasks and how they can be effectively combined remains an under-researched area. In this study, we carry out a systematic investigation of the effects of these techniques on linguistic properties through the use of probing tasks, including source language comprehension, bilingual word alignment, and translation fluency. We further evaluate the impact of pre-training, back-translation, and multi-task learning on bitexts of varying sizes. Our findings inform the design of more effective pipelines for leveraging monolingual data in extremely low-resource and low-resource machine translation tasks. Experiment results show consistent performance gains in seven translation directions, which provide further support for our conclusions and understanding of the role of monolingual data in machine translation.<\/jats:p>","DOI":"10.1162\/coli_a_00496","type":"journal-article","created":{"date-parts":[[2023,11,15]],"date-time":"2023-11-15T19:26:09Z","timestamp":1700076369000},"page":"25-47","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Rethinking the Exploitation of Monolingual Data for Low-Resource\n                    Neural Machine Translation"],"prefix":"10.1162","volume":"50","author":[{"given":"Jianhui","family":"Pang","sequence":"first","affiliation":[{"name":"NLP2CT Lab, University of Macau. nlp2ct.pangjh3@gmail.com"}]},{"given":"Baosong","family":"Yang*","sequence":"additional","affiliation":[{"name":"Alibaba Group. yangbaosong.ybs@alibaba-inc.com"}]},{"given":"Derek Fai","family":"Wong*","sequence":"additional","affiliation":[{"name":"NLP2CT Lab, University of Macau. derekfw@um.edu.mo"}]},{"given":"Yu","family":"Wan","sequence":"additional","affiliation":[{"name":"Alibaba Group. wanyu.wy@alibaba-inc.com"}]},{"given":"Dayiheng","family":"Liu","sequence":"additional","affiliation":[{"name":"Alibaba Group. liudayiheng.ldyh@alibaba-inc.com"}]},{"given":"Lidia Sam","family":"Chao","sequence":"additional","affiliation":[{"name":"NLP2CT Lab, University of Macau. lidiasc@um.edu.mo"}]},{"given":"Jun","family":"Xie","sequence":"additional","affiliation":[{"name":"Alibaba Group. qingjing.xj@alibaba-inc.com"}]}],"member":"281","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"key":"2024042419445279600_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.615","article-title":"Language model prior for low-resource neural\n                        machine translation","author":"Baziotis","year":"2020","journal-title":"arXiv preprint\n                        arXiv:2004.14928"},{"key":"2024042419445279600_bib2","doi-asserted-by":"publisher","first-page":"182","DOI":"10.3115\/1626431.1626468","article-title":"Domain adaptation for statistical machine\n                        translation with monolingual resources","volume-title":"Proceedings of the Fourth Workshop on Statistical Machine\n                        Translation","author":"Bertoldi","year":"2009"},{"key":"2024042419445279600_bib3","first-page":"330","article-title":"Improving translation model by monolingual\n                        data","volume-title":"Proceedings of the Sixth Workshop on\n                        Statistical Machine Translation","author":"Bojar","year":"2011"},{"key":"2024042419445279600_bib4","doi-asserted-by":"publisher","first-page":"53","DOI":"10.18653\/v1\/W19-5206","article-title":"Tagged back-translation","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume\n                        1: Research Papers)","author":"Caswell","year":"2019"},{"key":"2024042419445279600_bib5","doi-asserted-by":"publisher","first-page":"566","DOI":"10.18653\/v1\/2020.emnlp-main.42","article-title":"Accurate word alignment induction from neural machine\n                        translation","volume-title":"Proceedings of the 2020 Conference\n                        on Empirical Methods in Natural Language Processing (EMNLP)","author":"Chen","year":"2020"},{"key":"2024042419445279600_bib6","article-title":"SentEval: An evaluation toolkit for universal\n                        sentence representations","author":"Conneau","year":"2018","journal-title":"CoRR"},{"key":"2024042419445279600_bib7","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language Technologies,\n                        Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2024042419445279600_bib8","doi-asserted-by":"publisher","first-page":"489","DOI":"10.18653\/v1\/D18-1045","article-title":"Understanding back-translation at\n                        scale","volume-title":"Proceedings of the 2018 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Edunov","year":"2018"},{"key":"2024042419445279600_bib9","first-page":"986","article-title":"Better alignments = better\n                        translations?","volume-title":"Proceedings of ACL-08:\n                    HLT","author":"Ganchev","year":"2008"},{"key":"2024042419445279600_bib10","doi-asserted-by":"publisher","first-page":"4453","DOI":"10.18653\/v1\/D19-1453","article-title":"Jointly learning to align and translate\n                        with transformer models","volume-title":"Proceedings of the 2019\n                        Conference on Empirical Methods in Natural Language Processing and the 9th\n                        International Joint Conference on Natural Language Processing\n                        (EMNLP-IJCNLP)","author":"Garg","year":"2019"},{"key":"2024042419445279600_bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.132","article-title":"Cross-attention is all you need: Adapting pretrained\n                        transformers for machine translation","author":"Gheini","year":"2021","journal-title":"arXiv preprint\n                        arXiv:2104.08771"},{"key":"2024042419445279600_bib12","doi-asserted-by":"publisher","first-page":"344","DOI":"10.18653\/v1\/N18-1032","article-title":"Universal neural machine translation for extremely low\n                        resource languages","volume-title":"Proceedings of the 2018\n                        Conference of the North American Chapter of the Association for\n                        Computational Linguistics: Human Language Technologies, Volume 1 (Long\n                        Papers)","author":"Gu","year":"2018"},{"key":"2024042419445279600_bib13","doi-asserted-by":"publisher","first-page":"3622","DOI":"10.18653\/v1\/D18-1398","article-title":"Meta-learning for low-resource neural machine\n                        translation","volume-title":"Proceedings of the 2018 Conference\n                        on Empirical Methods in Natural Language Processing","author":"Gu","year":"2018"},{"key":"2024042419445279600_bib14","article-title":"On using monolingual corpora in neural\n                        machine translation","author":"Gulcehre","year":"2015","journal-title":"arXiv preprint\n                        arXiv:1503.03535"},{"key":"2024042419445279600_bib15","doi-asserted-by":"publisher","first-page":"1198","DOI":"10.18653\/v1\/N19-1122","article-title":"Modeling recurrence for\n                        transformer","volume-title":"Proceedings of the 2019 Conference\n                        of the North American Chapter of the Association for Computational\n                        Linguistics: Human Language Technologies, Volume 1 (Long and Short\n                        Papers)","author":"Hao","year":"2019"},{"key":"2024042419445279600_bib16","doi-asserted-by":"publisher","first-page":"3651","DOI":"10.18653\/v1\/P19-1356","article-title":"What does BERT learn about the structure\n                        of language?","volume-title":"Proceedings of the 57th Annual\n                        Meeting of the Association for Computational Linguistics","author":"Jawahar","year":"2019"},{"key":"2024042419445279600_bib17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1101","article-title":"Improving unsupervised word-by-word translation with language\n                        model and denoising autoencoder","author":"Kim","year":"2019","journal-title":"CoRR"},{"key":"2024042419445279600_bib18","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2015","journal-title":"CoRR"},{"key":"2024042419445279600_bib19","doi-asserted-by":"publisher","first-page":"1613","DOI":"10.18653\/v1\/2021.eacl-main.138","article-title":"Multilingual neural machine translation with deep encoder and\n                        multiple shallow decoders","volume-title":"Proceedings of the\n                        16th Conference of the European Chapter of the Association for Computational\n                        Linguistics: Main Volume","author":"Kong","year":"2021"},{"key":"2024042419445279600_bib20","doi-asserted-by":"publisher","first-page":"66","DOI":"10.18653\/v1\/D18-2012","article-title":"SentencePiece: A simple and language\n                        independent subword tokenizer and detokenizer for neural text\n                        processing","volume-title":"EMNLP","author":"Kudo","year":"2018"},{"key":"2024042419445279600_bib21","article-title":"Unsupervised machine translation using\n                        monolingual corpora only","author":"Lample","year":"2017","journal-title":"arXiv preprint\n                        arXiv:1711.00043"},{"key":"2024042419445279600_bib22","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence\n                        pre-training for natural language generation, translation, and\n                        comprehension","author":"Lewis","year":"2020","journal-title":"Proceedings of the 58th Annual\n                        Meeting of the Association for Computational Linguistics"},{"key":"2024042419445279600_bib23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.247","article-title":"On the complementarity between\n                        pre-training and back-translation for neural machine\n                        translation","author":"Liu","year":"2021","journal-title":"arXiv:2110.01811"},{"key":"2024042419445279600_bib24","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1162\/tacl_a_00343","article-title":"Multilingual denoising pre-training for\n                        neural machine translation","volume":"8","author":"Liu","year":"2020","journal-title":"Transactions of the\n                        Association for Computational Linguistics"},{"key":"2024042419445279600_bib25","article-title":"Multi-task sequence to sequence\n                        learning","author":"Luong","year":"2016","journal-title":"arXiv preprint\n                    arXiv:1511.06114"},{"key":"2024042419445279600_bib26","doi-asserted-by":"publisher","first-page":"15991","DOI":"10.18653\/v1\/2023.acl-long.891","article-title":"Crosslingual generalization through\n                        multitask finetuning","volume-title":"Proceedings of the 61st\n                        Annual Meeting of the Association for Computational Linguistics (Volume 1:\n                        Long Papers)","author":"Muennighoff","year":"2022"},{"key":"2024042419445279600_bib27","article-title":"Improving language understanding by\n                        generative pre-training","author":"Radford","year":"2018","journal-title":"OpenAI\n                    blog"},{"issue":"8","key":"2024042419445279600_bib28","first-page":"9","article-title":"Language models are unsupervised multitask\n                        learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2024042419445279600_bib29","article-title":"Neural machine translation for low-resource languages: A\n                        survey","author":"Ranathunga","year":"2021","journal-title":"arXiv preprint\n                    arXiv:2106.15115"},{"key":"2024042419445279600_bib30","article-title":"Leveraging pre-trained checkpoints for\n                        sequence generation tasks","author":"Rothe","year":"2019","journal-title":"CoRR"},{"key":"2024042419445279600_bib31","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00313","article-title":"Leveraging pre-trained checkpoints for\n                        sequence generation tasks","author":"Rothe","year":"2020","journal-title":"arXiv preprint\n                        arXiv:1907.12461"},{"key":"2024042419445279600_bib32","unstructured":"Schweter,\n                                Stefan\n          .\n                        2020. German GPT-2 model.\n                        10.5281\/zenodo.4275046"},{"key":"2024042419445279600_bib33","doi-asserted-by":"publisher","first-page":"371","DOI":"10.18653\/v1\/W16-2323","article-title":"Edinburgh neural machine translation\n                        systems for WMT 16","volume-title":"Proceedings of the First\n                        Conference on Machine Translation: Volume 2, Shared Task Papers","author":"Sennrich","year":"2016"},{"key":"2024042419445279600_bib34","doi-asserted-by":"publisher","first-page":"86","DOI":"10.18653\/v1\/P16-1009","article-title":"Improving neural machine translation\n                        models with monolingual data","volume-title":"Proceedings of the\n                        54th Annual Meeting of the Association for Computational Linguistics (Volume\n                        1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2024042419445279600_bib35","doi-asserted-by":"publisher","first-page":"259","DOI":"10.3115\/1626431.1626480","article-title":"Fluency, adequacy, or HTER? Exploring\n                        different human judgments with a tunable MT metric","volume-title":"Proceedings of the Fourth Workshop on Statistical Machine\n                        Translation","author":"Snover","year":"2009"},{"key":"2024042419445279600_bib36","first-page":"41","article-title":"Extremely low-resource machine translation\n                        for closely related languages","volume-title":"Proceedings of the\n                        23rd Nordic Conference on Computational Linguistics (NoDaLiDa)","author":"Tars","year":"2021"},{"key":"2024042419445279600_bib37","doi-asserted-by":"publisher","first-page":"2062","DOI":"10.18653\/v1\/N19-1209","article-title":"Overcoming catastrophic forgetting during\n                        domain adaptation of neural machine translation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language Technologies,\n                        Volume 1 (Long and Short Papers)","author":"Thompson","year":"2019"},{"key":"2024042419445279600_bib38","article-title":"LLaMa: Open and efficient foundation\n                        language models","author":"Touvron","year":"2023","journal-title":"arXiv preprint\n                        arXiv:2302.13971"},{"key":"2024042419445279600_bib39","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2024042419445279600_bib40","article-title":"AER: Do we need to \u201cimprove\u201d our\n                        alignments?","volume-title":"Proceedings of the Third\n                        International Workshop on Spoken Language Translation:\n                    Papers","author":"Vilar","year":"2006"},{"key":"2024042419445279600_bib41","doi-asserted-by":"publisher","first-page":"1022","DOI":"10.18653\/v1\/2020.emnlp-main.75","article-title":"Multi-task learning for multilingual\n                        neural machine translation","volume-title":"Proceedings of the\n                        2020 Conference on Empirical Methods in Natural Language Processing\n                        (EMNLP)","author":"Wang","year":"2020"},{"key":"2024042419445279600_bib42","doi-asserted-by":"publisher","first-page":"5786","DOI":"10.18653\/v1\/P19-1579","article-title":"Generalized data augmentation for\n                        low-resource translation","volume-title":"Proceedings of the 57th\n                        Annual Meeting of the Association for Computational Linguistics","author":"Xia","year":"2019"},{"key":"2024042419445279600_bib43","doi-asserted-by":"publisher","first-page":"3069","DOI":"10.18653\/v1\/P19-1295","article-title":"Leveraging local and global patterns for self-attention\n                        networks","volume-title":"Proceedings of the 57th Annual Meeting\n                        of the Association for Computational Linguistics","author":"Xu","year":"2019"},{"key":"2024042419445279600_bib44","doi-asserted-by":"publisher","first-page":"483","DOI":"10.18653\/v1\/2021.naacl-main.41","article-title":"mT5: A massively multilingual pre-trained\n                        text-to-text transformer","volume-title":"Proceedings of the 2021\n                        Conference of the North American Chapter of the Association for\n                        Computational Linguistics: Human Language Technologies","author":"Xue","year":"2021"},{"key":"2024042419445279600_bib45","doi-asserted-by":"publisher","first-page":"1628","DOI":"10.18653\/v1\/2020.acl-main.148","article-title":"Improving massively multilingual neural\n                        machine translation and zero-shot translation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for\n                        Computational Linguistics","author":"Zhang","year":"2020"},{"key":"2024042419445279600_bib46","doi-asserted-by":"publisher","first-page":"1535","DOI":"10.18653\/v1\/D16-1160","article-title":"Exploiting source-side monolingual data in neural machine\n                        translation","volume-title":"Proceedings of the 2016 Conference\n                        on Empirical Methods in Natural Language Processing","author":"Zhang","year":"2016"},{"key":"2024042419445279600_bib47","article-title":"Incorporating BERT into neural machine\n                        translation","author":"Zhu","year":"2020","journal-title":"arXiv preprint\n                        arXiv:2002.06823"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/1\/25\/2367119\/coli_a_00496.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/1\/25\/2367119\/coli_a_00496.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T19:45:18Z","timestamp":1713987918000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/50\/1\/25\/118132\/Rethinking-the-Exploitation-of-Monolingual-Data"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,3,1]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00496","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}