{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T15:52:26Z","timestamp":1776181946233,"version":"3.50.1"},"reference-count":71,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation, and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By utilizing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivaling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE. <\/jats:p>","DOI":"10.1162\/tacl_a_00330","type":"journal-article","created":{"date-parts":[[2020,9,9]],"date-time":"2020-09-09T18:58:55Z","timestamp":1599677935000},"page":"539-555","source":"Crossref","is-referenced-by-count":46,"title":["Unsupervised Quality Estimation for Neural Machine Translation"],"prefix":"10.1162","volume":"8","author":[{"given":"Marina","family":"Fomicheva","sequence":"first","affiliation":[{"name":"University of Sheffield."}]},{"given":"Shuo","family":"Sun","sequence":"additional","affiliation":[{"name":"Johns Hopkins University."}]},{"given":"Lisa","family":"Yankovskaya","sequence":"additional","affiliation":[{"name":"University of Tartu."}]},{"given":"Fr\u00e9d\u00e9ric","family":"Blain","sequence":"additional","affiliation":[{"name":"University of Sheffield."}]},{"given":"Francisco","family":"Guzm\u00e1n","sequence":"additional","affiliation":[{"name":"Facebook AI."}]},{"given":"Mark","family":"Fishel","sequence":"additional","affiliation":[{"name":"University of Tartu."}]},{"given":"Nikolaos","family":"Aletras","sequence":"additional","affiliation":[{"name":"University of Sheffield."}]},{"given":"Vishrav","family":"Chaudhary","sequence":"additional","affiliation":[{"name":"Facebook AI."}]},{"given":"Lucia","family":"Specia","sequence":"additional","affiliation":[{"name":"University of Sheffield"},{"name":"Imperial College London."}]}],"member":"281","reference":[{"key":"bib1","volume-title":"3rd International Conference on Learning Representations, ICLR 2015","author":"Bahdanau Dzmitry","year":"2015"},{"key":"bib2","first-page":"1","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Barrault Lo\u00efc","year":"2019"},{"key":"bib3","doi-asserted-by":"crossref","first-page":"257","DOI":"10.18653\/v1\/D16-1025","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Bentivogli Luisa","year":"2016"},{"key":"bib4","volume-title":"Proceedings of the 22nd Annual Conference of the European Association for Machine Translation","author":"Blain Fr\u00e9d\u00e9ric","year":"2020"},{"key":"bib5","doi-asserted-by":"crossref","first-page":"315","DOI":"10.3115\/1220355.1220401","volume-title":"COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics","author":"Blatz John","year":"2004"},{"key":"bib6","doi-asserted-by":"crossref","first-page":"489","DOI":"10.18653\/v1\/W17-4755","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Bojar Ond\u0159ej","year":"2017"},{"key":"bib7","doi-asserted-by":"crossref","first-page":"199","DOI":"10.18653\/v1\/W16-2302","volume-title":"Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers","author":"Bojar Ond\u0159ej","year":"2016"},{"issue":"1","key":"bib8","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1515\/pralin-2017-0013","volume":"108","author":"Castilho Sheila","year":"2017","journal-title":"The Prague Bulletin of Mathematical Linguistics"},{"key":"bib9","first-page":"3112","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Cho Jaemin","year":"2019"},{"key":"bib10","volume-title":"Proceedings of the EACL 2014 Workshop on Statistical Machine Translation","author":"Denkowski Michael","year":"2014"},{"key":"bib11","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019"},{"key":"bib12","doi-asserted-by":"crossref","first-page":"743","DOI":"10.18653\/v1\/P18-1069","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Li Dong","year":"2018"},{"key":"bib13","first-page":"118","volume-title":"Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks","author":"Espl\u00e0 Miquel","year":"2019"},{"key":"bib14","doi-asserted-by":"crossref","first-page":"782","DOI":"10.18653\/v1\/W18-6461","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Etchegoyhen Thierry","year":"2018"},{"key":"bib15","first-page":"1","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)","author":"Fonseca Erick","year":"2019"},{"key":"bib16","first-page":"1050","volume-title":"International Conference on Machine Learning","author":"Gal Yarin","year":"2016"},{"key":"bib17","first-page":"1409","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","author":"Garmash Ekaterina","year":"2016"},{"key":"bib18","first-page":"3124","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","author":"Graham Yvette","year":"2016"},{"key":"bib19","first-page":"1183","volume-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Graham Yvette","year":"2015"},{"key":"bib20","first-page":"33","volume-title":"Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse","author":"Graham Yvette","year":"2013"},{"key":"bib21","first-page":"1","author":"Graham Yvette","year":"2015","journal-title":"Natural Language Engineering"},{"key":"bib22","first-page":"2348","volume-title":"Advances in Neural Information Processing Systems","author":"Graves Alex","year":"2011"},{"key":"bib23","first-page":"1321","volume-title":"Proceedings of the 34th International Conference on Machine Learning- Volume 70","author":"Guo Chuan","year":"2017"},{"key":"bib24","doi-asserted-by":"crossref","first-page":"6097","DOI":"10.18653\/v1\/D19-1632","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Guzm\u00e1n Francisco","year":"2019"},{"key":"bib25","author":"Hassan Hany","year":"2018","journal-title":"arXiv preprint arXiv:1803.05567"},{"key":"bib26","first-page":"583","volume-title":"Proceedings of the 22nd Conference on Computational Natural Language Learning","author":"He Xuanli","year":"2018"},{"key":"bib27","author":"Hendrycks Dan","year":"2016","journal-title":"arXiv preprint arXiv:1610.02136"},{"key":"bib28","volume-title":"Proceedings of COLING 2018, the 27th International Conference on Computational Linguistics: Technical Papers","author":"Ive Julia","year":"2018"},{"key":"bib29","first-page":"5574","volume-title":"Advances in Neural Information Processing Systems","author":"Kendall Alex","year":"2017"},{"key":"bib30","doi-asserted-by":"crossref","first-page":"78","DOI":"10.18653\/v1\/W19-5406","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)","author":"Kepler Fabio","year":"2019"},{"key":"bib31","doi-asserted-by":"crossref","first-page":"117","DOI":"10.18653\/v1\/P19-3020","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics\u2013System Demonstrations","author":"Kepler F\u00e1bio","year":"2019"},{"key":"bib32","first-page":"562","volume-title":"Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers","author":"Kim Hyun","year":"2017"},{"key":"bib33","first-page":"562","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Kim Hyun","year":"2017"},{"key":"bib34","first-page":"79","volume-title":"MT summit","volume":"5","author":"Koehn Philipp","year":"2005"},{"key":"bib35","first-page":"3474","volume-title":"Advances in Neural Information Processing Systems","author":"Kuleshov Volodymyr","year":"2015"},{"key":"bib36","author":"Kumar Aviral","year":"2019","journal-title":"arXiv preprint arXiv:1903.00802"},{"key":"bib37","first-page":"6402","volume-title":"Advances in Neural Information Processing Systems","author":"Lakshminarayanan Balaji","year":"2017"},{"key":"bib38","doi-asserted-by":"crossref","first-page":"1412","DOI":"10.18653\/v1\/D15-1166","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Luong Thang","year":"2015"},{"key":"bib39","first-page":"671","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Ma Qingsong","year":"2018"},{"key":"bib40","first-page":"62","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Ma Qingsong","year":"2019"},{"key":"bib41","unstructured":"David J. C. MacKay. 1992. Bayesian Methods for Adaptive Models. Ph.D. thesis, California Institute of Technology."},{"key":"bib42","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1007\/978-3-030-31372-2_24","volume-title":"Statistical Language and Speech Processing","author":"Miok Kristian","year":"2019"},{"key":"bib43","first-page":"120","volume-title":"7th Workshop on Statistical Machine Translation","author":"Moreau Erwan","year":"2012"},{"key":"bib44","doi-asserted-by":"crossref","first-page":"pages 1587\u2013page","DOI":"10.18653\/v1\/D15-1182","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Nguyen Khanh","year":"2015"},{"key":"bib45","doi-asserted-by":"crossref","first-page":"575","DOI":"10.18653\/v1\/W19-8671","volume-title":"Proceedings of The 12th International Conference on Natural Language Generation","author":"Niehues Jan","year":"2019"},{"key":"bib46","volume-title":"International Conference on Machine Learning","author":"Ott Myle","year":"2018"},{"key":"bib47","volume-title":"Proceedings of NAACL-HLT 2019: Demonstrations","author":"Ott Myle","year":"2019"},{"key":"bib48","first-page":"1","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Ott Myle","year":"2018"},{"key":"bib49","author":"Pearce Tim","year":"2018","journal-title":"arXiv preprint arXiv:1810.05546"},{"key":"bib50","first-page":"133","volume-title":"Proceedings of the Seventh Workshop on Statistical Machine Translation","author":"Popovi\u0107 Maja","year":"2012"},{"key":"bib51","first-page":"299","volume-title":"Proceedings of MT Summit XVI","author":"Rikters Mat\u0131-ss","year":"2017"},{"key":"bib52","author":"Schwenk Holger","year":"2019","journal-title":"arXiv preprint arXiv:1907.05791"},{"key":"bib53","author":"Shen Tianxiao","year":"2019","journal-title":"arXiv preprint arXiv:1902.07816"},{"key":"bib54","first-page":"13969","volume-title":"Advances in Neural Information Processing Systems","author":"Snoek Jasper","year":"2019"},{"key":"bib55","volume-title":"Proceedings of association for machine translation in the Americas","volume":"200","author":"Snover Matthew","year":"2006"},{"key":"bib56","doi-asserted-by":"crossref","first-page":"689","DOI":"10.18653\/v1\/W18-6451","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Specia Lucia","year":"2018"},{"key":"bib57","first-page":"79","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Specia Lucia","year":"2013"},{"key":"bib58","first-page":"28","volume-title":"13th Conference of the European Association for Machine Translation","author":"Specia Lucia","year":"2009"},{"issue":"1","key":"bib59","first-page":"1929","volume":"15","author":"Srivastava Nitish","year":"2014","journal-title":"The Journal of Machine Learning Research"},{"key":"bib60","first-page":"3104","volume-title":"Advances in Neural Information Processing Systems","author":"Sutskever Ilya","year":"2014"},{"key":"bib61","first-page":"14633","volume-title":"Advances in Neural Information Processing Systems","author":"Tran Dustin","year":"2019"},{"key":"bib62","author":"Vashishth Shikhar","year":"2019","journal-title":"arXiv preprint arXiv:1909.11218"},{"key":"bib63","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017"},{"key":"bib64","first-page":"63","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Vig Jesse","year":"2019"},{"key":"bib65","author":"Vijayakumar Ashwin K.","year":"2016","journal-title":"arXiv preprint arXiv:1610.02424"},{"key":"bib66","doi-asserted-by":"crossref","first-page":"5797","DOI":"10.18653\/v1\/P19-1580","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Voita Elena","year":"2019"},{"key":"bib67","doi-asserted-by":"crossref","first-page":"809","DOI":"10.18653\/v1\/W18-6465","volume-title":"Proceedings of the Third Conference on Machine Translation: Shared Task Papers","author":"Wang Jiayi","year":"2018"},{"key":"bib68","first-page":"791","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing","author":"Wang Shuo","year":"2019"},{"key":"bib69","first-page":"681","volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML-11)","author":"Welling Max","year":"2011"},{"key":"bib70","volume-title":"Regression Analysis","volume":"14","author":"Williams Evan James","year":"1959"},{"key":"bib71","volume-title":"Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers","author":"Yankovskaya Elizaveta","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00330","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:43Z","timestamp":1615585183000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96475"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":71,"alternative-id":["10.1162\/tacl_a_00330"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00330","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}