{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T06:41:09Z","timestamp":1784616069400,"version":"3.55.0"},"reference-count":39,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,9,20]],"date-time":"2023-09-20T00:00:00Z","timestamp":1695168000000},"content-version":"vor","delay-in-days":262,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,9,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Cross-lingual text classification leverages text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning (zero\/ few-shots cross-lingual transfer). Nowadays, cross-lingual text classifiers are typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. However, the performance of these models varies significantly across languages and classification tasks, suggesting that the superposition of the language modelling and classification tasks is not always effective. For this reason, in this paper we propose revisiting the classic \u201ctranslate-and-test\u201d pipeline to neatly separate the translation and classification stages. The proposed approach couples 1) a neural machine translator translating from the targeted language to a high-resource language, with 2) a text classifier trained in the high-resource language, but the neural machine translator generates \u201csoft\u201d translations to permit end-to-end backpropagation during fine-tuning of the pipeline. Extensive experiments have been carried out over three cross-lingual text classification datasets (XNLI, MLDoc, and MultiEURLEX), with the results showing that the proposed approach has significantly improved performance over a competitive baseline.<\/jats:p>","DOI":"10.1162\/tacl_a_00593","type":"journal-article","created":{"date-parts":[[2023,9,20]],"date-time":"2023-09-20T14:58:08Z","timestamp":1695221888000},"page":"1147-1161","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification"],"prefix":"10.1162","volume":"11","author":[{"given":"Inigo Jauregi","family":"Unanue","sequence":"first","affiliation":[{"name":"RoZetta Technology, Australia. inigo.jauregi@rozettatechnology.com"},{"name":"University of Technology Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gholamreza","family":"Haffari","sequence":"additional","affiliation":[{"name":"Monash University, Australia. gholamreza.haffari@monash.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Massimo","family":"Piccardi","sequence":"additional","affiliation":[{"name":"University of Technology Sydney, Australia. massimo.piccardi@uts.edu.au"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,9,19]]},"reference":[{"key":"2023092014493554400_bib1","doi-asserted-by":"publisher","first-page":"4623","DOI":"10.18653\/v1\/2020.acl-main.421","article-title":"On the cross-lingual transferability of monolingual representations","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Artetxe","year":"2020"},{"key":"2023092014493554400_bib2","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1007\/978-3-540-45175-4_13","article-title":"Cross-lingual text categorization","volume-title":"Proceedings of the 7th European Conference on Digital Libraries","author":"Bel","year":"2003"},{"key":"2023092014493554400_bib3","doi-asserted-by":"publisher","first-page":"6974","DOI":"10.18653\/v1\/2021.emnlp-main.559","article-title":"MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Chalkidis","year":"2021"},{"key":"2023092014493554400_bib4","doi-asserted-by":"publisher","first-page":"6170","DOI":"10.18653\/v1\/2022.acl-long.427","article-title":"XLM-E: Cross-lingual language model pre-training via ELECTRA","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","author":"Chi","year":"2022"},{"key":"2023092014493554400_bib5","doi-asserted-by":"publisher","first-page":"8440","DOI":"10.18653\/v1\/2020.acl-main.747","article-title":"Unsupervised cross-lingual representation learning at scale","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Conneau","year":"2020"},{"key":"2023092014493554400_bib6","article-title":"Cross-lingual language model pretraining","volume":"32","author":"Conneau","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023092014493554400_bib7","doi-asserted-by":"publisher","first-page":"2475","DOI":"10.18653\/v1\/D18-1269","article-title":"XNLI: Evaluating cross-lingual sentence representations","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Conneau","year":"2018"},{"key":"2023092014493554400_bib8","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin","year":"2019"},{"key":"2023092014493554400_bib9","article-title":"Non-autoregressive neural machine translation","volume-title":"International Conference on Learning Representations","author":"Jiatao","year":"2018"},{"key":"2023092014493554400_bib10","first-page":"2790","article-title":"Parameter-efficient transfer learning for NLP","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Houlsby","year":"2019"},{"key":"2023092014493554400_bib11","first-page":"4411","article-title":"XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Junjie","year":"2020"},{"key":"2023092014493554400_bib12","doi-asserted-by":"publisher","first-page":"2485","DOI":"10.18653\/v1\/D19-1252","article-title":"Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing","author":"Huang","year":"2019"},{"key":"2023092014493554400_bib13","article-title":"Categorical reparameterization with gumbel-softmax","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Jang","year":"2017"},{"key":"2023092014493554400_bib14","doi-asserted-by":"publisher","first-page":"915","DOI":"10.18653\/v1\/2021.acl-short.115","article-title":"BERTTune: Fine-tuning neural machine translation with BERTScore","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","author":"Unanue","year":"2021"},{"key":"2023092014493554400_bib15","doi-asserted-by":"publisher","first-page":"6282","DOI":"10.18653\/v1\/2020.acl-main.560","article-title":"The state and fate of linguistic diversity and inclusion in the NLP world","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Joshi","year":"2020"},{"key":"2023092014493554400_bib16","first-page":"1459","article-title":"Inducing crosslingual distributed representations of words","volume-title":"Proceedings of the 24th International Conference on Computational Linguistics","author":"Klementiev","year":"2012"},{"key":"2023092014493554400_bib17","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1162\/tacl_a_00447","article-title":"Quality at a glance: An audit of web-crawled multilingual datasets","volume":"10","author":"Kreutzer","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023092014493554400_bib18","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lewis","year":"2020"},{"key":"2023092014493554400_bib19","doi-asserted-by":"publisher","first-page":"6008","DOI":"10.18653\/v1\/2020.emnlp-main.484","article-title":"XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Liang","year":"2020"},{"key":"2023092014493554400_bib20","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1162\/tacl_a_00343","article-title":"Multilingual denoising pre-training for neural machine translation","volume":"8","author":"Liu","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023092014493554400_bib21","doi-asserted-by":"publisher","first-page":"3980","DOI":"10.18653\/v1\/2021.acl-long.308","article-title":"VECO: Variable and flexible cross-lingual pre-training for language understanding and generation","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","author":"Luo","year":"2021"},{"key":"2023092014493554400_bib22","first-page":"1614","article-title":"From softmax to sparsemax: A sparse model of attention and multi-label classification","volume-title":"International Conference on Machine Learning","author":"Martins","year":"2016"},{"key":"2023092014493554400_bib23","doi-asserted-by":"publisher","first-page":"116","DOI":"10.18653\/v1\/2021.mrl-1.11","article-title":"Small data? No problem! Exploring the viability of pretrained multilingual language models for low-resourced languages","volume-title":"Proceedings of the 1st Workshop on Multilingual Representation Learning","author":"Ogueji","year":"2021"},{"key":"2023092014493554400_bib24","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Peters","year":"2018"},{"key":"2023092014493554400_bib25","doi-asserted-by":"publisher","first-page":"3479","DOI":"10.18653\/v1\/2022.naacl-main.255","article-title":"Lifting the curse of multilinguality by pre-training modular transformers","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics","author":"Pfeiffer","year":"2022"},{"key":"2023092014493554400_bib26","doi-asserted-by":"publisher","first-page":"7654","DOI":"10.18653\/v1\/2020.emnlp-main.617","article-title":"MAD-X: An adapter-based framework for multi-task cross-lingual transfer","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Pfeiffer","year":"2020"},{"key":"2023092014493554400_bib27","article-title":"Modelling latent translations for cross-lingual transfer","author":"Ponti","year":"2021","journal-title":"arXiv preprint arXiv:2107.11353"},{"key":"2023092014493554400_bib28","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.18653\/v1\/2021.acl-long.155","article-title":"Glancing transformer for non- autoregressive neural machine translation","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","author":"Qian","year":"2021"},{"key":"2023092014493554400_bib29","article-title":"Improving language understanding by generative pre-training","author":"Radford","year":"2018"},{"issue":"140","key":"2023092014493554400_bib30","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2023092014493554400_bib31","doi-asserted-by":"publisher","first-page":"4512","DOI":"10.18653\/v1\/2020.emnlp-main.365","article-title":"Making monolingual sentence embeddings multilingual using knowledge distillation","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Reimers","year":"2020"},{"key":"2023092014493554400_bib32","article-title":"A corpus for multilingual document classification in eight languages","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation","author":"Schwenk","year":"2018"},{"key":"2023092014493554400_bib33","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.18653\/v1\/D19-1140","article-title":"Machine translation for machines: The sentiment classification use case","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing","author":"Tebbifakhr","year":"2019"},{"key":"2023092014493554400_bib34","first-page":"36","article-title":"Machine-oriented NMT adaptation for zero-shot NLP tasks: Comparing the usefulness of close and distant languages","volume-title":"Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects","author":"Tebbifakhr","year":"2020"},{"key":"2023092014493554400_bib35","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2023092014493554400_bib36","first-page":"165","article-title":"Cross language text categorization using a bilingual lexicon","volume-title":"Third International Joint Conference on Natural Language Processing","author":"Ke","year":"2008"},{"key":"2023092014493554400_bib37","doi-asserted-by":"publisher","first-page":"7361","DOI":"10.18653\/v1\/2021.acl-long.571","article-title":"Vocabulary learning via optimal transport for neural machine translation","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","author":"Jingjing","year":"2021"},{"key":"2023092014493554400_bib38","first-page":"483","article-title":"mT5: A massively multilingual pre-trained text-to-text transformer","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Xue","year":"2021"},{"key":"2023092014493554400_bib39","doi-asserted-by":"publisher","first-page":"362","DOI":"10.18653\/v1\/2022.acl-short.40","article-title":"Translate-train embracing translationese artifacts","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","author":"Sicheng","year":"2022"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00593\/2159097\/tacl_a_00593.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00593\/2159097\/tacl_a_00593.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,20]],"date-time":"2023-09-20T15:02:35Z","timestamp":1695222155000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00593\/117584\/T3L-Translate-and-Test-Transfer-Learning-for-Cross"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":39,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00593","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}