{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T01:28:50Z","timestamp":1780450130137,"version":"3.54.1"},"reference-count":35,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,8,6]],"date-time":"2021-08-06T00:00:00Z","timestamp":1628208000000},"content-version":"vor","delay-in-days":217,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,2]]},"abstract":"<jats:p>In this paper, we evaluate the translation of negation both automatically and manually, in English\u2013German (EN\u2013DE) and English\u2013 Chinese (EN\u2013ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN\u2192DE, DE\u2192EN, EN\u2192ZH, and ZH\u2192EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under-translation of negation, we study the model\u2019s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement.<\/jats:p>","DOI":"10.1162\/tacl_a_00395","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T19:29:23Z","timestamp":1632166163000},"page":"740-755","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":4,"title":["Revisiting Negation in Neural Machine Translation"],"prefix":"10.1162","volume":"9","author":[{"given":"Gongbo","family":"Tang","sequence":"first","affiliation":[{"name":"Department of Linguistics and Philology, Uppsala University, Sweden. gongbo.tang@lingfil.uu.se"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philipp","family":"R\u00f6nchen","sequence":"additional","affiliation":[{"name":"Department of Linguistics and Philology, Uppsala University, Sweden. philipp.r\u00f6nchen@lingfil.uu.se"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rico","family":"Sennrich","sequence":"additional","affiliation":[{"name":"Department of Computational Linguistics, University of Zurich, Switzerland"},{"name":"School of Informatics, University of Edinburgh. rico.sennrich@ed.ac.uk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joakim","family":"Nivre","sequence":"additional","affiliation":[{"name":"Department of Linguistics and Philology, Uppsala University, Sweden. joakim.nivre@lingfil.uu.se"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,8,2]]},"reference":[{"key":"2021080620253250800_bib1","doi-asserted-by":"publisher","first-page":"4190","DOI":"10.18653\/v1\/2020.acl-main.385","article-title":"Quantifying attention flow in transformers","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Abnar","year":"2020"},{"key":"2021080620253250800_bib2","doi-asserted-by":"publisher","first-page":"187","DOI":"10.18653\/v1\/D19-5619","article-title":"On the importance of word boundaries in character-level neural machine translation","volume-title":"Proceedings of the 3rd Workshop on Neural Generation and Translation","author":"Ataman","year":"2019"},{"issue":"2","key":"2021080620253250800_bib3","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1162\/COLI_a_00099","article-title":"Modality and negation in SIMT use of modality and negation in semantically-informed syntactic MT","volume":"38","author":"Baker","year":"2012","journal-title":"Computational Linguistics"},{"key":"2021080620253250800_bib4","doi-asserted-by":"publisher","first-page":"257","DOI":"10.18653\/v1\/D16-1025","article-title":"Neural versus phrase-based machine translation quality: A case study","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Bentivogli","year":"2016"},{"key":"2021080620253250800_bib5","first-page":"41","article-title":"Can out-of-the-box NMT beat a domain-trained Moses on technical data?","volume-title":"The 20th Annual Conference of the European Association for Machine Translation (EAMT)","author":"Beyer","year":"2017"},{"key":"2021080620253250800_bib6","doi-asserted-by":"crossref","first-page":"169","DOI":"10.18653\/v1\/W17-4717","article-title":"Findings of the 2017 Conference on Machine Translation (WMT17)","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Bojar","year":"2017"},{"key":"2021080620253250800_bib7","unstructured":"Ond\u0159ej\n              Bojar\n            , PhilipWilliams, DavidMare\u010dek, MartinPopel, RudolfRosa, JosefJon, and MichalKa\u0161par. 2018. Final report on employing semantic role labelling and shallow proxies for negation and fidelity checking in MT, The University of Edinburgh."},{"key":"2021080620253250800_bib8","doi-asserted-by":"publisher","first-page":"531","DOI":"10.3115\/1219840.1219906","article-title":"Clause restructuring for statistical machine translation","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics)","author":"Collins","year":"2005"},{"key":"2021080620253250800_bib9","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional Transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2021080620253250800_bib10","doi-asserted-by":"publisher","first-page":"1150","DOI":"10.18653\/v1\/P17-1106","article-title":"Visualizing and understanding neural machine translation","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ding","year":"2017"},{"key":"2021080620253250800_bib11","article-title":"Neural networks for cross-lingual negation scope detection","author":"Fancellu","year":"2018","journal-title":"CoRR"},{"key":"2021080620253250800_bib12","doi-asserted-by":"publisher","first-page":"598","DOI":"10.3115\/v1\/E14-1063","article-title":"Applying the Semantics of Negation to SMT Through N-best List Re-ranking","volume-title":"Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics","author":"Fancellu","year":"2014"},{"key":"2021080620253250800_bib13","doi-asserted-by":"publisher","first-page":"2","DOI":"10.3115\/v1\/W15-1301","article-title":"Translating negation: A manual error analysis","volume-title":"Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExProM 2015)","author":"Fancellu","year":"2015"},{"key":"2021080620253250800_bib14","doi-asserted-by":"publisher","first-page":"140","DOI":"10.18653\/v1\/P16-1014","article-title":"Pointing the unknown words","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Gulcehre","year":"2016"},{"key":"2021080620253250800_bib15","article-title":"Sockeye: A toolkit for neural machine translation","author":"Hieber","year":"2017","journal-title":"CoRR"},{"key":"2021080620253250800_bib16","doi-asserted-by":"publisher","first-page":"3869","DOI":"10.18653\/v1\/2020.findings-emnlp.345","article-title":"It\u2019s not a Non-Issue: Negation as a source of error in machine translation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Md","year":"2020"},{"key":"2021080620253250800_bib17","doi-asserted-by":"publisher","DOI":"10.1017\/9781316423530","volume-title":"The Cambridge grammar of the English language","author":"Huddleston","year":"2002"},{"key":"2021080620253250800_bib18","first-page":"1","article-title":"On using very large target vocabulary for neural machine translation","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Jean","year":"2015"},{"key":"2021080620253250800_bib19","first-page":"5739","article-title":"NegBERT: A transfer learning approach for negation detection and scope resolution","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Khandelwal","year":"2020"},{"key":"2021080620253250800_bib20","article-title":"Adam: A method for stochastic optimization","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Kingma","year":"2015"},{"key":"2021080620253250800_bib21","doi-asserted-by":"crossref","first-page":"190","DOI":"10.3115\/1626431.1626470","article-title":"Chinese syntactic reordering for adequate generation of Korean verbal phrases in Chinese-to-Korean smt","volume-title":"Proceedings of the Fourth Workshop on Statistical Machine Translation","author":"Li","year":"2009"},{"key":"2021080620253250800_bib22","article-title":"NegPar: A parallel corpus annotated for negation","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Liu","year":"2018"},{"key":"2021080620253250800_bib23","first-page":"1563","article-title":"ConanDoyle-neg: Annotation of negation in Conan Doyle stories","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912)","author":"Morante","year":"2012"},{"key":"2021080620253250800_bib24","doi-asserted-by":"publisher","first-page":"pages 311\u2013pages 318","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2021080620253250800_bib25","doi-asserted-by":"publisher","first-page":"pages 2227\u2013pages 2237","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Peters","year":"2018"},{"key":"2021080620253250800_bib26","doi-asserted-by":"publisher","first-page":"186","DOI":"10.18653\/v1\/W18-6319","article-title":"A call for clarity in reporting bleu scores","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Post","year":"2018"},{"key":"2021080620253250800_bib27","doi-asserted-by":"publisher","first-page":"pages 1314\u2013pages 1324","DOI":"10.18653\/v1\/N18-1119","article-title":"Fast lexically constrained decoding with dynamic beam allocation for neural machine translation","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Post","year":"2018"},{"key":"2021080620253250800_bib28","first-page":"299","article-title":"Confidence through attention","volume-title":"Proceedings of the 16th Machine Translation Summit (MT Summit 2017)","author":"Rikters","year":"2017"},{"key":"2021080620253250800_bib29","doi-asserted-by":"publisher","first-page":"376","DOI":"10.18653\/v1\/E17-2060","article-title":"How grammatical is character-level neural machine translation? assessing mt quality with contrastive translation pairs","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers","author":"Sennrich","year":"2017"},{"key":"2021080620253250800_bib30","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural Machine Translation of Rare Words with Subword Units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2021080620253250800_bib31","doi-asserted-by":"publisher","first-page":"178","DOI":"10.18653\/v1\/D19-6221","article-title":"Neural token representations and negation and speculation scope detection in biomedical and general domain text","volume-title":"Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019)","author":"Sergeeva","year":"2019"},{"key":"2021080620253250800_bib32","doi-asserted-by":"publisher","first-page":"4263","DOI":"10.18653\/v1\/D18-1458","article-title":"Why self-attention? A targeted evaluation of neural machine translation architectures","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Tang","year":"2018"},{"key":"2021080620253250800_bib33","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1162\/tacl_a_00048","article-title":"Context gates for neural machine translation","volume":"5","author":"Zhaopeng","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021080620253250800_bib34","first-page":"76","article-title":"Modeling coverage for neural machine translation","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers)","author":"Zhaopeng","year":"2016"},{"key":"2021080620253250800_bib35","first-page":"20","article-title":"Enriching parallel corpora for statistical machine translation with semantic negation rephrasing","volume-title":"Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation","author":"Wetzel","year":"2012"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00395\/1955167\/tacl_a_00395.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00395\/1955167\/tacl_a_00395.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T19:29:42Z","timestamp":1632166182000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00395\/106793\/Revisiting-Negation-in-Neural-Machine-Translation"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":35,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00395","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}