{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T03:19:40Z","timestamp":1769915980631,"version":"3.49.0"},"reference-count":43,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T00:00:00Z","timestamp":1688515200000},"content-version":"vor","delay-in-days":185,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,29]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https:\/\/bit.ly\/frmt-task.<\/jats:p>","DOI":"10.1162\/tacl_a_00568","type":"journal-article","created":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T17:33:48Z","timestamp":1688578428000},"page":"671-685","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation"],"prefix":"10.1162","volume":"11","author":[{"given":"Parker","family":"Riley","sequence":"first","affiliation":[{"name":"Google Research, USA. prkriley@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timothy","family":"Dozat","sequence":"additional","affiliation":[{"name":"Google Research, USA. tdozat@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jan A.","family":"Botha","sequence":"additional","affiliation":[{"name":"Google Research, USA. jabot@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xavier","family":"Garcia","sequence":"additional","affiliation":[{"name":"Google Research, USA. xgarcia@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dan","family":"Garrette","sequence":"additional","affiliation":[{"name":"Google Research, USA. dhgarrette@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Riesa","sequence":"additional","affiliation":[{"name":"Google Research, USA. riesa@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Orhan","family":"Firat","sequence":"additional","affiliation":[{"name":"Google Research, USA. orhanf@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noah","family":"Constant","sequence":"additional","affiliation":[{"name":"Google Research, USA. nconstant@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,6,29]]},"reference":[{"key":"2023070517334359000_bib1","first-page":"1","article-title":"Findings of the 2021 conference on machine translation (WMT21)","volume-title":"Proceedings of the Sixth Conference on Machine Translation","author":"Akhbardeh","year":"2021"},{"issue":"22","key":"2023070517334359000_bib2","doi-asserted-by":"publisher","first-page":"87","DOI":"10.17771\/PUCRio.TradRev.30591","article-title":"e-pact: Esperto paraphrase aligned corpus of en-ep\/bp translations","volume":"1","author":"Barreiro","year":"2017","journal-title":"Tradu\u00e7ao em Revista"},{"key":"2023070517334359000_bib3","doi-asserted-by":"publisher","first-page":"58","DOI":"10.18653\/v1\/2021.gem-1.6","article-title":"A review of human evaluation for style transfer","volume-title":"Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)","author":"Briakou","year":"2021"},{"key":"2023070517334359000_bib4","first-page":"1877","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2023070517334359000_bib5","first-page":"261","article-title":"WIT3: Web inventory of transcribed and translated talks","volume-title":"Proceedings of the 16th Annual conference of the European Association for Machine Translation","author":"Cettolo","year":"2012"},{"key":"2023070517334359000_bib6","article-title":"Palm: Scaling language modeling with pathways","author":"Chowdhery","year":"2022","journal-title":"arXiv preprint arXiv:2204.02311"},{"key":"2023070517334359000_bib7","first-page":"275","article-title":"A neural approach to language variety translation","volume-title":"Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)","author":"Costa-juss\u00e0","year":"2018"},{"key":"2023070517334359000_bib8","doi-asserted-by":"publisher","first-page":"1460","DOI":"10.1162\/tacl_a_00437","article-title":"Experts, errors, and context: A large-scale study of human evaluation for machine translation","volume":"9","author":"Freitag","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023070517334359000_bib9","first-page":"733","article-title":"Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain","volume-title":"Proceedings of the Sixth Conference on Machine Translation","author":"Freitag","year":"2021"},{"key":"2023070517334359000_bib10","article-title":"irr: Various coefficients of interrater reliability and agreement","volume-title":"CRAN","author":"Gamer","year":"2019"},{"key":"2023070517334359000_bib11","article-title":"Towards universality in multilingual text rewriting","author":"Garcia","year":"2021","journal-title":"arXiv preprint arXiv:2107.14749"},{"key":"2023070517334359000_bib12","article-title":"Using natural language prompts for machine translation","author":"Garcia","year":"2022","journal-title":"arXiv preprint arXiv:2202.11822"},{"key":"2023070517334359000_bib13","article-title":"Wiki-40b: Multilingual language model dataset","volume-title":"LREC 2020","author":"Guo","year":"2020"},{"key":"2023070517334359000_bib14","article-title":"Machine translation of low-resource spoken dialects: Strategies for normalizing Swiss German","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Honnet","year":"2018"},{"issue":"1","key":"2023070517334359000_bib15","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1145\/3544903.3544906","article-title":"Text style transfer: A review and experimental evaluation","volume":"24","author":"Zhiqiang","year":"2022","journal-title":"SIGKDD Explorations Newsletter"},{"key":"2023070517334359000_bib16","doi-asserted-by":"publisher","first-page":"10","DOI":"10.18653\/v1\/W17-4902","article-title":"Shakespearizing modern language using copy-enriched sequence to sequence models","volume-title":"Proceedings of the Workshop on Stylistic Variation","author":"Jhamtani","year":"2017"},{"key":"2023070517334359000_bib17","doi-asserted-by":"publisher","first-page":"110","DOI":"10.18653\/v1\/2021.acl-short.16","article-title":"Machine translation into low-resource language varieties","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Kumar","year":"2021"},{"key":"2023070517334359000_bib18","doi-asserted-by":"crossref","first-page":"156","DOI":"10.18653\/v1\/W18-6316","article-title":"Neural machine translation into language varieties","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Lakew","year":"2018"},{"key":"2023070517334359000_bib19","first-page":"1865","article-title":"Delete, retrieve, generate: A simple approach to sentiment and style transfer","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Li","year":"2018"},{"key":"2023070517334359000_bib20","article-title":"OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Lison","year":"2018"},{"key":"2023070517334359000_bib21","doi-asserted-by":"publisher","first-page":"312","DOI":"10.18653\/v1\/P18-2050","article-title":"Extreme adaptation for personalized neural machine translation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Michel","year":"2018"},{"key":"2023070517334359000_bib22","doi-asserted-by":"publisher","first-page":"2814","DOI":"10.18653\/v1\/D17-1299","article-title":"A study of style in machine translation: Controlling the formality of machine translation output","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Niu","year":"2017"},{"key":"2023070517334359000_bib23","first-page":"1008","article-title":"Multi-task neural models for translating between styles within and across languages","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Niu","year":"2018"},{"key":"2023070517334359000_bib24","first-page":"1008","article-title":"Multi-task neural models for translating between styles within and across languages","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Niu","year":"2018"},{"key":"2023070517334359000_bib25","doi-asserted-by":"publisher","first-page":"138","DOI":"10.18653\/v1\/D19-5614","article-title":"Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer","volume-title":"Proceedings of the 3rd Workshop on Neural Generation and Translation","author":"Pang","year":"2019"},{"key":"2023070517334359000_bib26","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2023070517334359000_bib27","doi-asserted-by":"publisher","first-page":"392","DOI":"10.18653\/v1\/W15-3049","article-title":"chrF: Character n-gram F-score for automatic MT evaluation","volume-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation","author":"Popovi\u0107","year":"2015"},{"key":"2023070517334359000_bib28","doi-asserted-by":"publisher","first-page":"186","DOI":"10.18653\/v1\/W18-6319","article-title":"A call for clarity in reporting BLEU scores","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Post","year":"2018"},{"key":"2023070517334359000_bib29","doi-asserted-by":"publisher","first-page":"3786","DOI":"10.18653\/v1\/2021.acl-long.293","article-title":"TextSETTR: Few-shot text style extraction and tunable targeted restyling","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Riley","year":"2021"},{"key":"2023070517334359000_bib30","doi-asserted-by":"publisher","first-page":"5094","DOI":"10.18653\/v1\/2020.coling-main.447","article-title":"AraBench: Benchmarking dialectal Arabic-English machine translation","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Sajjad","year":"2020"},{"key":"2023070517334359000_bib31","article-title":"Multitask prompted training enables zero-shot task generalization","volume-title":"International Conference on Learning Representations","author":"Sanh","year":"2022"},{"key":"2023070517334359000_bib32","doi-asserted-by":"publisher","first-page":"7881","DOI":"10.18653\/v1\/2020.acl-main.704","article-title":"BLEURT: Learning robust metrics for text generation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Sellam","year":"2020"},{"key":"2023070517334359000_bib33","doi-asserted-by":"publisher","first-page":"35","DOI":"10.18653\/v1\/N16-1005","article-title":"Controlling politeness in neural machine translation via side constraints","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Sennrich","year":"2016"},{"key":"2023070517334359000_bib34","first-page":"6830","article-title":"Style transfer from non-parallel text by cross-alignment","volume-title":"Advances in Neural Information Processing Systems 30","author":"Shen","year":"2017"},{"key":"2023070517334359000_bib35","article-title":"Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning","volume":"abs\/2201 .03110","author":"Siddhant","year":"2022","journal-title":"CoRR"},{"key":"2023070517334359000_bib36","doi-asserted-by":"publisher","first-page":"137","DOI":"10.18653\/v1\/2021.eacl-srw.19","article-title":"Towards personalised and document-level machine translation of dialogue","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop","author":"Vincent","year":"2021"},{"issue":"05","key":"2023070517334359000_bib37","doi-asserted-by":"publisher","first-page":"9130","DOI":"10.1609\/aaai.v34i05.6448","article-title":"Unsupervised neural dialect translation with commonality and diversity modeling","volume":"34","author":"Wan","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2023070517334359000_bib38","doi-asserted-by":"publisher","first-page":"3573","DOI":"10.18653\/v1\/D19-1365","article-title":"Harnessing pre-trained neural networks with rules for formality style transfer","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Wang","year":"2019"},{"key":"2023070517334359000_bib39","article-title":"Finetuned language models are zero-shot learners","volume-title":"International Conference on Learning Representations","author":"Wei","year":"2022"},{"key":"2023070517334359000_bib40","first-page":"10534","article-title":"On variational learning of controllable representations for text without supervision","volume-title":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\u201318 July 2020, Virtual Event","author":"Peng","year":"2020"},{"key":"2023070517334359000_bib41","first-page":"483","article-title":"mT5: A massively multilingual pre-trained text-to-text transformer","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Xue","year":"2021"},{"key":"2023070517334359000_bib42","article-title":"Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects","author":"Zampieri","year":"2021"},{"key":"2023070517334359000_bib43","first-page":"49","article-title":"Machine translation of Arabic dialects","volume-title":"Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Zbib","year":"2012"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00568\/2141015\/tacl_a_00568.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00568\/2141015\/tacl_a_00568.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T17:34:09Z","timestamp":1688578449000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00568\/116617\/FRMT-A-Benchmark-for-Few-Shot-Region-Aware-Machine"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":43,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00568","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}