{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T03:26:58Z","timestamp":1773026818333,"version":"3.50.1"},"reference-count":81,"publisher":"MIT Press","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2019,9]]},"abstract":"<jats:p>Automatic Machine Translation (MT) evaluation is an active field of research, with a handful of new metrics devised every year. Evaluation metrics are generally benchmarked against manual assessment of translation quality, with performance measured in terms of overall correlation with human scores. Much work has been dedicated to the improvement of evaluation metrics to achieve a higher correlation with human judgments. However, little insight has been provided regarding the weaknesses and strengths of existing approaches and their behavior in different settings. In this work we conduct a broad meta-evaluation study of the performance of a wide range of evaluation metrics focusing on three major aspects. First, we analyze the performance of the metrics when faced with different levels of translation quality, proposing a local dependency measure as an alternative to the standard, global correlation coefficient. We show that metric performance varies significantly across different levels of MT quality: Metrics perform poorly when faced with low-quality translations and are not able to capture nuanced quality distinctions. Interestingly, we show that evaluating low-quality translations is also more challenging for humans. Second, we show that metrics are more reliable when evaluating neural MT than the traditional statistical MT systems. Finally, we show that the difference in the evaluation accuracy for different metrics is maintained even if the gold standard scores are based on different criteria.<\/jats:p>","DOI":"10.1162\/coli_a_00356","type":"journal-article","created":{"date-parts":[[2019,6,25]],"date-time":"2019-06-25T15:16:15Z","timestamp":1561475775000},"page":"515-558","source":"Crossref","is-referenced-by-count":14,"title":["Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments"],"prefix":"10.1162","volume":"45","author":[{"given":"Marina","family":"Fomicheva","sequence":"first","affiliation":[{"name":"University of Sheffield."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lucia","family":"Specia","sequence":"additional","affiliation":[{"name":"Imperial College London, University of Sheffield."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"bib1","doi-asserted-by":"publisher","DOI":"10.3115\/1687878.1687923"},{"key":"bib2","first-page":"1","volume-title":"Proceedings of ICLR 2015","author":"Bahdanau Dzmitry","year":"2014"},{"issue":"2","key":"bib3","doi-asserted-by":"crossref","first-page":"167","DOI":"10.2307\/3867666","volume":"46","author":"Baig Taimur","year":"1999","journal-title":"IMF Staff Papers"},{"key":"bib4","first-page":"65","volume-title":"Proceedings of the Association for Computational Linguistics (ACL) Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","volume":"29","author":"Banerjee Satanjeev","year":"2005"},{"issue":"1","key":"bib5","first-page":"1","volume":"56","author":"Berentsen Geir Drage","year":"2014","journal-title":"Journal of Statistical Software"},{"key":"bib6","doi-asserted-by":"publisher","DOI":"10.1016\/0014-2921(90)90073-8"},{"key":"bib7","doi-asserted-by":"publisher","DOI":"10.3115\/1220355.1220401"},{"key":"bib8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4717"},{"key":"bib9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2301"},{"key":"bib10","first-page":"27","volume-title":"Workshop on Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem","author":"Bojar Ond\u0159ej","year":"2016"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4755"},{"key":"bib12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2302"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"bib14","doi-asserted-by":"publisher","DOI":"10.3115\/1626355.1626373"},{"key":"bib15","first-page":"17","volume-title":"Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR","author":"Callison-Burch Chris","year":"2010"},{"key":"bib16","first-page":"249","volume-title":"EACL","volume":"6","author":"Callison-Burch Chris","year":"2006"},{"key":"bib17","doi-asserted-by":"publisher","DOI":"10.1515\/pralin-2017-0013"},{"key":"bib18","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219862"},{"key":"bib19","first-page":"63","volume-title":"Proceedings of MT Summit IX","author":"Coughlin Deborah","year":"2003"},{"key":"bib20","first-page":"71","volume-title":"Proceedings of MT Summit IX","author":"Culy Christopher","year":"2003"},{"key":"bib21","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-008-9090-y"},{"key":"bib22","volume-title":"Proceedings of the Ninth Biennial Conference of the Association for Machine Translation in the Americas (AMTA)","author":"Denkowski Michael","year":"2010"},{"key":"bib23","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3348"},{"key":"bib24","doi-asserted-by":"publisher","DOI":"10.3115\/1289189.1289273"},{"key":"bib25","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1994.10476782"},{"key":"bib26","first-page":"2755","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation","author":"Fomicheva Marina","year":"2016"},{"key":"bib27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3046"},{"key":"bib28","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2339"},{"key":"bib29","doi-asserted-by":"publisher","DOI":"10.1111\/0022-1082.00494"},{"key":"bib30","doi-asserted-by":"publisher","DOI":"10.1075\/cilt.260.17gim"},{"issue":"94","key":"bib31","first-page":"77","author":"Gim\u00e9nez Jes\u00fas","year":"2010","journal-title":"Prague Bulletin of Mathematical Linguistics"},{"key":"bib32","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-011-9088-7"},{"key":"bib33","first-page":"33","volume-title":"Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse","author":"Graham Yvette","year":"2013"},{"key":"bib34","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324915000339"},{"key":"bib35","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1124"},{"key":"bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1124"},{"key":"bib37","first-page":"1619","volume":"48","author":"Hjort Nils Lid","year":"1996","journal-title":"Annals of Statistics"},{"key":"bib38","doi-asserted-by":"publisher","DOI":"10.1093\/rfs\/hhl037"},{"key":"bib39","doi-asserted-by":"publisher","DOI":"10.1023\/A:1024270700807"},{"key":"bib40","volume-title":"Proceedings of 13th International Workshop on Spoken Language Translation","volume":"1","author":"Junczys-Dowmunt Marcin","year":"2016"},{"key":"bib41","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4763"},{"key":"bib42","doi-asserted-by":"publisher","DOI":"10.3115\/1654650.1654666"},{"key":"bib43","first-page":"279","volume-title":"Contributions to Probability and Statistics. Essays in Honor of Harold Hotelling","author":"Levene Howard","year":"1961"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1219032"},{"key":"bib45","doi-asserted-by":"publisher","DOI":"10.3115\/1220355.1220427"},{"key":"bib46","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-010-0201-1_18"},{"key":"bib47","first-page":"25","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Liu Ding","year":"2005"},{"key":"bib48","doi-asserted-by":"publisher","DOI":"10.5565\/rev\/tradumatica.77"},{"key":"bib49","doi-asserted-by":"publisher","DOI":"10.1111\/0022-1082.00340"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3336"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.3115\/1073483.1073504"},{"key":"bib52","first-page":"220","volume-title":"Proceedings of the ACL 2010 Conference Short Papers","author":"Moore Robert C.","year":"2010"},{"key":"bib53","first-page":"39","volume-title":"Proceedings of the Second International Conference on Language Resources and Evaluation (LREC\u201900)","author":"Nie\u00dfen Sonja","year":"2000"},{"key":"bib54","doi-asserted-by":"publisher","DOI":"10.3115\/1687878.1687922"},{"key":"bib55","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the ACL","author":"Papineni Kishore","year":"2002"},{"key":"bib56","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3049"},{"key":"bib57","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2341"},{"key":"bib58","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00072"},{"key":"bib59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2323"},{"key":"bib60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1159"},{"key":"bib61","first-page":"223","volume-title":"Proceedings of Association for Machine Translation in the Americas","volume":"200","author":"Snover Matthew","year":"2006"},{"key":"bib62","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-009-9062-9"},{"key":"bib63","first-page":"73","volume-title":"15th Conference of the European Association for Machine Translation","author":"Specia Lucia","year":"2011"},{"key":"bib64","first-page":"513","volume-title":"Machine Translation Summit XIII","author":"Specia Lucia","year":"2011"},{"key":"bib65","first-page":"55","volume-title":"Proceedings of the 16th Machine Translation Summit","author":"Specia Lucia","year":"2017"},{"key":"bib66","first-page":"28","volume-title":"13th Annual Conference of the European Association for Machine Translation","author":"Specia Lucia","year":"2009"},{"key":"bib67","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3354"},{"key":"bib68","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3031"},{"key":"bib69","doi-asserted-by":"publisher","DOI":"10.1016\/j.jempfin.2013.11.006"},{"key":"bib70","doi-asserted-by":"publisher","DOI":"10.3115\/1706543.1706586"},{"key":"bib71","first-page":"3104","volume-title":"Advances in Neural Information Processing Systems","author":"Sutskever Ilya","year":"2014"},{"key":"bib72","doi-asserted-by":"crossref","first-page":"2667","DOI":"10.21437\/Eurospeech.1997-673","volume-title":"Fifth European Conference on Speech Communication and Technology","author":"Tillmann Christoph","year":"1997"},{"key":"bib73","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2012.08.001"},{"key":"bib74","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4613-9655-0"},{"key":"bib75","doi-asserted-by":"publisher","DOI":"10.2478\/v10108-012-0014-9"},{"key":"bib76","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-1100"},{"key":"bib77","volume-title":"Regression Analysis","volume":"14","author":"Williams Evan James","year":"1959"},{"key":"bib78","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2327"},{"key":"bib79","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3053"},{"key":"bib80","doi-asserted-by":"publisher","DOI":"10.3115\/1626281.1626285"},{"key":"bib81","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2343"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00356","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,20]],"date-time":"2024-07-20T06:08:17Z","timestamp":1721455697000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/45\/3\/515-558\/93373"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9]]},"references-count":81,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.1162\/coli_a_00356"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00356","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9]]}}}