{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T16:55:48Z","timestamp":1782838548097,"version":"3.54.5"},"reference-count":43,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> We describe an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text. We use multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. We validate our technique by extracting parallel sentence pairs on the BUCC 2017 bitext mining task and observe up to a 24.5 point increase (absolute) in F<jats:sub>1<\/jats:sub> scores over previous unsupervised methods. We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation performance by up to 3.5 BLEU on the WMT\u201914 French-English and WMT\u201916 German-English tasks and outperforming the previous state-of-the-art. Finally, we enrich the IWSLT\u201915 English-Vietnamese corpus with pseudo-parallel Wikipedia sentence pairs, yielding a 1.2 BLEU improvement on the low-resource MT task. We demonstrate that unsupervised bitext mining is an effective way of augmenting MT datasets and complements existing techniques like initializing with pre-trained contextual embeddings. <\/jats:p>","DOI":"10.1162\/tacl_a_00348","type":"journal-article","created":{"date-parts":[[2021,1,5]],"date-time":"2021-01-05T20:52:59Z","timestamp":1609879979000},"page":"828-841","source":"Crossref","is-referenced-by-count":10,"title":["Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings"],"prefix":"10.1162","volume":"8","author":[{"given":"Phillip","family":"Keung","sequence":"first","affiliation":[{"name":"Amazon."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julian","family":"Salazar","sequence":"additional","affiliation":[{"name":"Amazon."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yichao","family":"Lu","sequence":"additional","affiliation":[{"name":"Amazon."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noah A.","family":"Smith","sequence":"additional","affiliation":[{"name":"University of Washington"},{"name":"Allen Institute for AI."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings","author":"Artetxe Mikel","year":"2018"},{"key":"bib2","doi-asserted-by":"crossref","first-page":"3197","DOI":"10.18653\/v1\/P19-1309","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Artetxe Mikel","year":"2019"},{"key":"bib3","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00288"},{"key":"bib4","volume-title":"3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings","author":"Bahdanau Dzmitry","year":"2015"},{"key":"bib5","first-page":"597","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL\u201905)","author":"Bannard Colin","year":"2005"},{"key":"bib6","first-page":"2","volume-title":"Proceedings of the 12th International Workshop on Spoken Language Translation","author":"Cettolo Mauro","year":"2015"},{"key":"bib7","doi-asserted-by":"crossref","first-page":"1914","DOI":"10.18653\/v1\/D18-1217","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Clark Kevin","year":"2018"},{"key":"bib8","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019"},{"key":"bib9","doi-asserted-by":"crossref","first-page":"2185","DOI":"10.18653\/v1\/D19-1224","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Dodge Jesse","year":"2019"},{"key":"bib10","first-page":"1442","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Gr\u00e9goire Francis","year":"2018"},{"key":"bib11","first-page":"23:1","volume":"21","author":"Guo Jian","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"bib12","first-page":"165","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Guo Mandy","year":"2018"},{"key":"bib13","first-page":"7","volume-title":"Proceedings of the 15th International Workshop on Spoken Language Translation","author":"Hangya Viktor","year":"2018"},{"key":"bib14","doi-asserted-by":"crossref","first-page":"1224","DOI":"10.18653\/v1\/P19-1118","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Hangya Viktor","year":"2019"},{"key":"bib15","first-page":"200","volume-title":"Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers)","author":"Hieber Felix","year":"2018"},{"key":"bib16","first-page":"18","volume-title":"Proceedings of the 2nd Workshop on Neural Machine Translation and Generation","author":"Cong Duy Hoang Vu","year":"2018"},{"key":"bib17","volume":"1702","author":"Johnson Jeff","year":"2017","journal-title":"CoRR"},{"key":"bib18","doi-asserted-by":"crossref","first-page":"1355","DOI":"10.18653\/v1\/D19-1138","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Keung Phillip","year":"2019"},{"key":"bib19","doi-asserted-by":"crossref","first-page":"pages 244\u2013pages","DOI":"10.18653\/v1\/W18-6325","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Kocmi Tom","year":"2018"},{"key":"bib20","doi-asserted-by":"crossref","first-page":"726","DOI":"10.18653\/v1\/W18-6453","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Koehn Philipp","year":"2018"},{"key":"bib21","first-page":"7057","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8\u201314 December 2019, Vancouver, BC, Canada","author":"Lample Guillaume","year":"2019"},{"key":"bib22","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings","author":"Lample Guillaume","year":"2018"},{"key":"bib23","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 \u2013 May 3, 2018, Conference Track Proceedings","author":"Lample Guillaume","year":"2018"},{"key":"bib24","doi-asserted-by":"crossref","first-page":"5039","DOI":"10.18653\/v1\/D18-1549","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Lample Guillaume","year":"2018"},{"key":"bib25","author":"Libovick\u00fd Jindrich","year":"2019","journal-title":"CoRR"},{"key":"bib26","first-page":"pages 84\u2013pages","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Yichao Lu","year":"2018"},{"key":"bib27","first-page":"76","volume-title":"Proceedings of the 12th International Workshop on Spoken Language Translation","author":"Luong Minh-Thang","year":"2015"},{"key":"bib28","doi-asserted-by":"crossref","first-page":"68","DOI":"10.18653\/v1\/D19-5206","volume-title":"Proceedings of the 6th Workshop on Asian Translation","author":"Marie Benjamin","year":"2019"},{"key":"bib29","first-page":"3912","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Mulcaire Phoebe","year":"2019"},{"key":"bib30","volume-title":"Proceedings of the 16th International Workshop on Spoken Language Translation","author":"Nguyen Toan Q.","year":"2019"},{"key":"bib31","doi-asserted-by":"crossref","first-page":"186","DOI":"10.18653\/v1\/W18-6319","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Post Matt","year":"2018"},{"key":"bib32","first-page":"72","volume-title":"Machine Translation and the Information Soup, Third Conference of the Association for Machine Translation in the Americas, AMTA \u201998, Langhorne, PA, USA, October 28-31, 1998, Proceedings","volume":"1529","author":"Resnik Philip","year":"1998"},{"key":"bib33","doi-asserted-by":"publisher","DOI":"10.1162\/089120103322711578"},{"key":"bib34","doi-asserted-by":"crossref","first-page":"228","DOI":"10.18653\/v1\/P18-2037","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Schwenk Holger","year":"2018"},{"key":"bib35","author":"Schwenk Holger","year":"2019","journal-title":"CoRR"},{"key":"bib36","author":"Schwenk Holger","year":"2019","journal-title":"CoRR"},{"key":"bib37","doi-asserted-by":"crossref","first-page":"pages 86\u2013pages","DOI":"10.18653\/v1\/P16-1009","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich Rico","year":"2016"},{"key":"bib38","first-page":"5926","volume-title":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research","author":"Song Kaitao","year":"2019"},{"key":"bib39","first-page":"4383","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada","author":"Jingjing Xu","year":"2019"},{"key":"bib40","doi-asserted-by":"crossref","first-page":"46","DOI":"10.18653\/v1\/P18-1005","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Yang Zhen","year":"2018"},{"key":"bib41","doi-asserted-by":"crossref","first-page":"189","DOI":"10.3115\/981658.981684","volume-title":"33rd Annual Meeting of the Association for Computational Linguistics","author":"Yarowsky David","year":"1995"},{"key":"bib42","volume-title":"ACL","author":"Zhou Jiawei","year":"2020"},{"key":"bib43","doi-asserted-by":"crossref","first-page":"60","DOI":"10.18653\/v1\/W17-2512","volume-title":"Proceedings of the 10th Workshop on Building and Using Comparable Corpora","author":"Zweigenbaum Pierre","year":"2017"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00348","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:48Z","timestamp":1615585188000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96465"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":43,"alternative-id":["10.1162\/tacl_a_00348"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00348","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}