{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T13:03:27Z","timestamp":1772715807979,"version":"3.50.1"},"reference-count":57,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T00:00:00Z","timestamp":1637280000000},"content-version":"vor","delay-in-days":322,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Identifying factors that make certain languages harder to model than others is essential to reach language equality in future Natural Language Processing technologies. Free-order case-marking languages, such as Russian, Latin, or Tamil, have proved more challenging than fixed-order languages for the tasks of syntactic parsing and subject-verb agreement prediction. In this work, we investigate whether this class of languages is also more difficult to translate by state-of-the-art Neural Machine Translation (NMT) models. Using a variety of synthetic languages and a newly introduced translation challenge set, we find that word order flexibility in the source language only leads to a very small loss of NMT quality, even though the core verb arguments become impossible to disambiguate in sentences without semantic cues. The latter issue is indeed solved by the addition of case marking. However, in medium- and low-resource settings, the overall NMT quality of fixed-order languages remains unmatched.<\/jats:p>","DOI":"10.1162\/tacl_a_00424","type":"journal-article","created":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T14:32:23Z","timestamp":1637332343000},"page":"1233-1248","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["On the Difficulty of Translating Free-Order Case-Marking Languages"],"prefix":"10.1162","volume":"9","author":[{"given":"Arianna","family":"Bisazza","sequence":"first","affiliation":[{"name":"Center for Language and Cognition, University of Groningen, The Netherlands. a.bisazza@rug.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmet","family":"\u00dcst\u00fcn","sequence":"additional","affiliation":[{"name":"Center for Language and Cognition, University of Groningen, The Netherlands. a.ustun@rug.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephan","family":"Sportel","sequence":"additional","affiliation":[{"name":"Center for Language and Cognition, University of Groningen, The Netherlands. research@spor.tel"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"key":"2021111914321657300_bib1","first-page":"97","article-title":"An evaluation of two vocabulary reduction methods for neural machine translation","volume-title":"Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers)","author":"Ataman","year":"2018"},{"key":"2021111914321657300_bib2","article-title":"Case syncretism","volume-title":"The World Atlas of Language Structures Online","author":"Baerman","year":"2013"},{"key":"2021111914321657300_bib3","article-title":"Neural machine translation by jointly learning to align and translate","volume-title":"3rd International Conference on Learning Representations, ICLR 2015","author":"Bahdanau","year":"2015"},{"key":"2021111914321657300_bib4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/W19-5301","article-title":"Findings of the 2019 conference on machine translation (WMT19)","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Barrault","year":"2019"},{"key":"2021111914321657300_bib5","doi-asserted-by":"publisher","first-page":"861","DOI":"10.18653\/v1\/P17-1080","article-title":"What do neural machine translation models learn about morphology?","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Belinkov","year":"2017"},{"key":"2021111914321657300_bib6","doi-asserted-by":"publisher","first-page":"745","DOI":"10.3115\/1613715.1613809","article-title":"Predicting success in machine translation","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Birch","year":"2008"},{"key":"2021111914321657300_bib7","doi-asserted-by":"publisher","first-page":"25","DOI":"10.18653\/v1\/W18-2704","article-title":"Inducing grammars with and for neural machine translation","volume-title":"Proceedings of the 2nd Workshop on Neural Machine Translation and Generation","author":"Bisk","year":"2018"},{"key":"2021111914321657300_bib8","doi-asserted-by":"publisher","first-page":"272","DOI":"10.18653\/v1\/W18-6401","article-title":"Findings of the 2018 conference on machine translation (WMT18)","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Bojar","year":"2018"},{"key":"2021111914321657300_bib9","doi-asserted-by":"publisher","first-page":"1640","DOI":"10.18653\/v1\/2020.acl-main.149","article-title":"It\u2019s easier to translate out of English than into it: Measuring neural translation difficulty by cross-mutual information","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Bugliarello","year":"2020"},{"key":"2021111914321657300_bib10","doi-asserted-by":"crossref","first-page":"5166","DOI":"10.18653\/v1\/P19-1509","article-title":"Word-order biases in deep-agent emergent communication","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Chaabouni","year":"2019"},{"key":"2021111914321657300_bib11","article-title":"An introduction to synchronous grammars","author":"Chiang","year":"2006","journal-title":"Tutorial available at"},{"key":"2021111914321657300_bib12","doi-asserted-by":"publisher","first-page":"291","DOI":"10.18653\/v1\/P19-1509","article-title":"Automatically extracting challenge sets for non-local phenomena in neural machine translation","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Choshen","year":"2019"},{"key":"2021111914321657300_bib13","volume-title":"Language Universals and Linguistic Typology","author":"Comrie","year":"1981"},{"key":"2021111914321657300_bib14","first-page":"536","article-title":"Are all languages equally hard to language-model?","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Cotterell","year":"2018"},{"key":"2021111914321657300_bib15","doi-asserted-by":"publisher","first-page":"24","DOI":"10.18653\/v1\/W19-5203","article-title":"Incorporating source syntax into transformer-based neural machine translation","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)","author":"Currey","year":"2019"},{"key":"2021111914321657300_bib16","article-title":"Order of subject, object and verb","volume-title":"The World Atlas of Language Structures Online","author":"Dryer","year":"2013"},{"key":"2021111914321657300_bib17","volume-title":"WALS Online","author":"Dryer","year":"2013"},{"issue":"1","key":"2021111914321657300_bib18","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1515\/pralin-2017-0018","article-title":"Pre-reordering for neural machine translation: Helpful or harmful?","volume":"108","author":"Jinhua","year":"2017","journal-title":"The Prague Bulletin of Mathematical Linguistics"},{"issue":"2","key":"2021111914321657300_bib19","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1207\/s15516709cog1402_1","article-title":"Finding structure in time","volume":"14","author":"Elman","year":"1990","journal-title":"Cognitive Science"},{"key":"2021111914321657300_bib20","doi-asserted-by":"publisher","first-page":"823","DOI":"10.18653\/v1\/P16-1078","article-title":"Tree-to-sequence attentional neural machine translation","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Eriguchi","year":"2016"},{"issue":"33","key":"2021111914321657300_bib21","doi-asserted-by":"publisher","first-page":"10336","DOI":"10.1073\/pnas.1502134112","article-title":"Large-scale evidence of dependency length minimization in 37 languages","volume":"112","author":"Futrell","year":"2015","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2021111914321657300_bib22","first-page":"91","article-title":"Quantifying word order freedom in dependency corpora","volume-title":"Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015)","author":"Futrell","year":"2015"},{"key":"2021111914321657300_bib23","doi-asserted-by":"publisher","first-page":"316","DOI":"10.18653\/v1\/D18-1029","article-title":"On the relation between linguistic typology and (limitations of) multilingual language modeling","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Gerz","year":"2018"},{"issue":"2","key":"2021111914321657300_bib24","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1111\/j.1551-6709.2009.01073.x","article-title":"Do grammars minimize dependency length?","volume":"34","author":"Gildea","year":"2010","journal-title":"Cognitive Science"},{"key":"2021111914321657300_bib25","first-page":"73","article-title":"Some universals of grammar with particular reference to the order of meaningful elements","volume-title":"Universals of Human Language","author":"Greenberg","year":"1963"},{"key":"2021111914321657300_bib26","first-page":"1195","article-title":"Colorless green recurrent networks dream hierarchically","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Gulordava","year":"2018"},{"key":"2021111914321657300_bib27","doi-asserted-by":"publisher","first-page":"121","DOI":"10.18653\/v1\/N18-1108","article-title":"Diachronic trends in word order freedom and dependency length in dependency-annotated corpora of Latin and ancient Greek","volume-title":"Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015)","author":"Gulordava","year":"2015"},{"key":"2021111914321657300_bib28","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1162\/tacl_a_00103","article-title":"Multi-lingual dependency parsing evaluation: A large-scale analysis of word order properties using artificial data","volume":"4","author":"Gulordava","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111914321657300_bib29","doi-asserted-by":"publisher","first-page":"6100","DOI":"10.18653\/v1\/D19-1632","article-title":"The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali\u2013English and Sinhala\u2013English","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Guzm\u00e1n","year":"2019"},{"key":"2021111914321657300_bib30","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1162\/tacl_a_00306","article-title":"Theoretical limitations of self-attention in neural sequence models","volume":"8","author":"Hahn","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111914321657300_bib31","article-title":"Achieving human parity on automatic Chinese to English news translation","author":"Hassan","year":"2018","journal-title":"arXiv preprint arXiv:1803.05567"},{"issue":"8","key":"2021111914321657300_bib32","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2021111914321657300_bib33","first-page":"944","article-title":"Automatic evaluation of translation quality for distant language pairs","volume-title":"Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing","author":"Isozaki","year":"2010"},{"key":"2021111914321657300_bib34","doi-asserted-by":"publisher","first-page":"6282","DOI":"10.18653\/v1\/2020.acl-main.560","article-title":"The state and fate of linguistic diversity and inclusion in the NLP world","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Joshi","year":"2020"},{"key":"2021111914321657300_bib35","first-page":"79","article-title":"Europarl: A parallel corpus for statistical machine translation","volume-title":"The Tenth Machine Translation Summit Proceedings of Conference","author":"Koehn","year":"2005"},{"key":"2021111914321657300_bib36","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1162\/tacl_a_00343","article-title":"Multilingual denoising pre-training for neural machine translation","volume":"8","author":"Liu","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111914321657300_bib37","doi-asserted-by":"publisher","first-page":"1412","DOI":"10.18653\/v1\/D15-1166","article-title":"Effective approaches to attention-based neural machine translation","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Luong","year":"2015"},{"key":"2021111914321657300_bib38","article-title":"Case, word order, and language learnability: Insights from connectionist modeling","volume-title":"Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society","author":"Lupyan","year":"2002"},{"key":"2021111914321657300_bib39","doi-asserted-by":"publisher","first-page":"55","DOI":"10.3115\/v1\/P14-5010","article-title":"The Stanford CoreNLP natural language processing toolkit","volume-title":"Association for Computational Linguistics (ACL) System Demonstrations","author":"Manning","year":"2014"},{"key":"2021111914321657300_bib40","doi-asserted-by":"publisher","DOI":"10.21236\/ADA273556","article-title":"Building a large annotated corpus of English: The Penn Treebank","author":"Marcus","year":"1993"},{"key":"2021111914321657300_bib41","doi-asserted-by":"crossref","first-page":"4975","DOI":"10.18653\/v1\/P19-1491","article-title":"What kind of language is hard to language-model?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Mielke","year":"2019"},{"key":"2021111914321657300_bib42","doi-asserted-by":"publisher","first-page":"311","DOI":"10.18653\/v1\/P19-1491","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2021111914321657300_bib43","doi-asserted-by":"publisher","first-page":"425","DOI":"10.18653\/v1\/D18-1039","article-title":"Contextual parameter generation for universal neural machine translation","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Antonios Platanios","year":"2018"},{"issue":"1","key":"2021111914321657300_bib44","doi-asserted-by":"publisher","first-page":"4381","DOI":"10.1038\/s41467-020-18073-9","article-title":"Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals","volume":"11","author":"Popel","year":"2020","journal-title":"Nature Communications"},{"key":"2021111914321657300_bib45","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-demos.14","article-title":"Stanza: A Python natural language processing toolkit for many human languages","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Qi","year":"2020"},{"key":"2021111914321657300_bib46","doi-asserted-by":"publisher","first-page":"3532","DOI":"10.18653\/v1\/N19-1356","article-title":"Studying the inductive biases of RNNs with synthetic variations of natural languages","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Ravfogel","year":"2019"},{"key":"2021111914321657300_bib47","article-title":"Comparing complexity measures","volume-title":"Computational Approaches to Morphological Complexity","author":"Sagot","year":"2013"},{"key":"2021111914321657300_bib48","first-page":"9834","article-title":"Analyzing the effect of neural network architecture on training performance","volume-title":"Proceedings of Machine Learning and Systems 2020","author":"Sankararaman","year":"2020"},{"key":"2021111914321657300_bib49","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with subword units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2021111914321657300_bib50","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1075\/slcs.94.06sin","article-title":"Complexity trade-offs in core argument marking","volume-title":"Language Complexity","author":"Sinnem\u00e4ki","year":"2008"},{"key":"2021111914321657300_bib51","first-page":"129","article-title":"The acquisition of Russian as a native language","author":"Slobin","year":"1966","journal-title":"The Genesis of Language: A Psycholinguistic Approach"},{"issue":"3","key":"2021111914321657300_bib52","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1016\/0010-0277(82)90033-6","article-title":"Children use canonical sentence schemas: A crosslinguistic study of word order and inflections","volume":"12","author":"Slobin","year":"1982","journal-title":"Cognition"},{"key":"2021111914321657300_bib53","doi-asserted-by":"publisher","first-page":"4731","DOI":"10.18653\/v1\/D18-1503","article-title":"The importance of being recurrent for modeling hierarchical structure","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Ke","year":"2018"},{"key":"2021111914321657300_bib54","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4\u20139 December 2017, Long Beach, CA, USA","author":"Vaswani","year":"2017"},{"key":"2021111914321657300_bib55","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1162\/tacl_a_00113","article-title":"The galactic dependencies treebanks: Getting more data by synthesizing new languages","volume":"4","author":"Wang","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111914321657300_bib56","doi-asserted-by":"crossref","first-page":"6682","DOI":"10.18653\/v1\/2020.acl-main.597","article-title":"Predicting declension class from form and meaning","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Williams","year":"2020"},{"key":"2021111914321657300_bib57","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.597","article-title":"Exploiting pre-ordering for neural machine translation","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)","author":"Zhao","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00424\/1972440\/tacl_a_00424.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00424\/1972440\/tacl_a_00424.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T14:32:51Z","timestamp":1637332371000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00424\/108198\/On-the-Difficulty-of-Translating-Free-Order-Case"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":57,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00424","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}