{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:05:05Z","timestamp":1783569905791,"version":"3.55.0"},"reference-count":16,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2018,9]]},"abstract":"<jats:p> The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique\u2014in other words, whether BLEU scores correlate with real-world utility and user-satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for), but does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing. <\/jats:p>","DOI":"10.1162\/coli_a_00322","type":"journal-article","created":{"date-parts":[[2018,6,7]],"date-time":"2018-06-07T18:53:10Z","timestamp":1528397590000},"page":"393-401","source":"Crossref","is-referenced-by-count":219,"title":["A Structured Review of the Validity of BLEU"],"prefix":"10.1162","volume":"44","author":[{"given":"Ehud","family":"Reiter","sequence":"first","affiliation":[{"name":"University of Aberdeen, Department of Computing Science."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","first-page":"65","volume-title":"Proceedings of the ACL-2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005"},{"key":"bib2","doi-asserted-by":"publisher","DOI":"10.3115\/1557690.1557746"},{"key":"bib3","first-page":"313","volume-title":"Proceedings of EACL 2006","author":"Belz Anja","year":"2006"},{"key":"bib4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2302"},{"key":"bib5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2301"},{"key":"bib6","first-page":"207","volume-title":"Proceedings of EMNLP-2014","author":"Bouamor Houda","year":"2014"},{"key":"bib7","first-page":"249","volume-title":"Proceedings of EACL-2006","author":"Callison-Burch Chris","year":"2006"},{"key":"bib8","doi-asserted-by":"publisher","DOI":"10.3115\/1289189.1289273"},{"key":"bib9","first-page":"564","volume-title":"Proceedings of EMNLP-2010","author":"Espinosa Dominic","year":"2010"},{"key":"bib10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1013"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-1019"},{"key":"bib12","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2535"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1238"},{"key":"bib14","first-page":"311","volume-title":"Proceedings of ACL 2002","author":"Papineni Kishore","year":"2002"},{"key":"bib15","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2015.2829"},{"key":"bib16","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2009.35.4.35405"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00322","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:28:14Z","timestamp":1615584494000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/44\/3\/393-401\/1598"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9]]},"references-count":16,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9]]}},"alternative-id":["10.1162\/coli_a_00322"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00322","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,9]]}}}