{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T23:47:12Z","timestamp":1792799232143,"version":"4.5.0"},"reference-count":147,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,1,18]],"date-time":"2022-01-18T00:00:00Z","timestamp":1642464000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Department of Computer Science and Engineering"},{"name":"Robert Bosch Center for Data Science and Artificial Intelligence","id":[{"id":"https:\/\/ror.org\/058w0r841","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,2,28]]},"abstract":"<jats:p>In the last few years, a large number of automatic evaluation metrics have been proposed for evaluating Natural Language Generation (NLG) systems. The rapid development and adoption of such automatic evaluation metrics in a relatively short time has created the need for a survey of these metrics. In this survey, we (i) highlight the challenges in automatically evaluating NLG systems, (ii) propose a coherent taxonomy for organising existing evaluation metrics, (iii) briefly describe different existing metrics, and finally (iv) discuss studies criticising the use of automatic evaluation metrics. We then conclude the article highlighting promising future directions of research.<\/jats:p>","DOI":"10.1145\/3485766","type":"journal-article","created":{"date-parts":[[2022,1,18]],"date-time":"2022-01-18T04:50:32Z","timestamp":1642481432000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":167,"title":["A Survey of Evaluation Metrics Used for NLG Systems"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1953-6214","authenticated-orcid":false,"given":"Ananya B.","family":"Sai","sequence":"first","affiliation":[{"name":"Robert-Bosch Centre for Data Science and AI, Indian Institute of Technology, Madras, Chennai, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akash Kumar","family":"Mohankumar","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology, Madras, Chennai, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mitesh M.","family":"Khapra","sequence":"additional","affiliation":[{"name":"Robert-Bosch Centre for Data Science and AI, Indian Institute of Technology, and AI4Bharat, Chennai, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,1,18]]},"reference":[{"key":"e_1_3_2_2_2","volume-title":"The Hitchhiker\u2019s Guide to the Galaxy","author":"Adams Douglas","year":"2007","unstructured":"Douglas Adams. 2007. The Hitchhiker\u2019s Guide to the Galaxy. Random House Publishing Group."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.12.004"},{"key":"e_1_3_2_4_2","first-page":"3318","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Amidei Jacopo","year":"2018","unstructured":"Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, 3318\u20133329."},{"key":"e_1_3_2_5_2","unstructured":"Ananthakrishnan Pushpak Bhattacharyya Murugesan Sasikumar and Ritesh M. Shah. 2006. Some issues in automatic evaluation of english-hindi MT: More blues for BLEU. ICON ."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","unstructured":"Peter Anderson Basura Fernando Mark Johnson and Stephen Gould. 2016. SPICE: Semantic propositional image caption evaluation. In European Conference on Computer Vision . B. Leibe J. Matas N. Sebe and M. Welling (Eds.) Lecture Notes in Computer Science Vol. 9909. Springer 382\u2013398.","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.568"},{"key":"e_1_3_2_8_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. Association for Computational Linguistics, 65\u201372."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.3115\/1118253.1118255"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/2002736.2002785"},{"key":"e_1_3_2_11_2","volume-title":"Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics","author":"Belz Anja","year":"2006","unstructured":"Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. The Association for Computer Linguistics."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3302"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2301"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4755"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2302"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/1699510.1699548"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics","author":"Callison-Burch Chris","year":"2006","unstructured":"Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Trento, Italy."},{"key":"e_1_3_2_18_2","first-page":"169","volume-title":"EMNLP (Demonstration)","author":"Cer Daniel","year":"2018","unstructured":"Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder for English. In EMNLP (Demonstration). Association for Computational Linguistics, 169\u2013174."},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Arun Tejasvi Chaganty Stephen Mussmann and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. 643\u2013653. https:\/\/doi.org\/10.18653\/v1\/P18-1060","DOI":"10.18653\/v1\/P18-1060"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5817"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1152"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5356"},{"key":"e_1_3_2_23_2","unstructured":"Junyoung Chung Caglar Gulcehre Kyunghyun Cho and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv:1412.3555. Retrieved from https:\/\/arxiv.org\/abs\/1412.3555."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1264"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/2002736.2002774"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1070"},{"key":"e_1_3_2_27_2","first-page":"63","volume-title":"Proceedings of MT Summit IX","author":"Coughlin Deborah","year":"2003","unstructured":"Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX. Citeseer, 63\u201370."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00608"},{"key":"e_1_3_2_29_2","first-page":"459","article-title":"Contextual word similarity","author":"Dagan Ido","year":"2000","unstructured":"Ido Dagan. 2000. Contextual word similarity. In Handbook of Natural Language Processing. Rob Dale, Hermann Moisl, and Harold Somers (Eds.), Marcel Dekker Inc, 459\u2013475.","journal-title":"Handbook of Natural Language Processing"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/1868850.1868901"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3348"},{"key":"e_1_3_2_33_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171\u20134186."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1483"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.5555\/1289189.1289273"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1068"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2019.06.009"},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Echizen\u2019ya Hiroshi","year":"2019","unstructured":"Hiroshi Echizen\u2019ya, Kenji Araki, and Eduard Hovy. 2019. Word embedding-based automatic MT evaluation metric using word position information. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1045"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2074"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"Alexander R. Fabbri Wojciech Kry\u015bci\u0144ski Bryan McCann Caiming Xiong Richard Socher and Dragomir Radev. 2020. SummEval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics 9 (2021) 391\u2013409.","DOI":"10.1162\/tacl_a_00373"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00356"},{"key":"e_1_3_2_43_2","unstructured":"Gabriel Forgues and Joelle Pineau. 2014. Bootstrapping dialog systems with word embeddings. In Modern Machine Learning and Natural Language Processing Workshop (NeurIPS\u201914) Vol. 2."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-2073"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.41"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1017"},{"key":"e_1_3_2_47_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1007\/978-3-642-15573-4_14","volume-title":"Empirical Methods in Natural Language Generation","author":"Gatt Albert","year":"2010","unstructured":"Albert Gatt and Anja Belz. 2010. Introducing shared tasks to NLG: The TUNA shared task evaluation challenges. In Empirical Methods in Natural Language Generation. E. Krahmer and M. Theune (Eds.), Lecture Notes in Computer Science, Vol. 5790, Springer, 264\u2013293."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.5555\/3241691.3241693"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3455511"},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","unstructured":"Sarik Ghazarian Johnny Tian-Zheng Wei Aram Galstyan and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. NAACL HLT 2019 (2019) 82.","DOI":"10.18653\/v1\/W19-2310"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.2478\/v10108-010-0022-6"},{"key":"e_1_3_2_52_2","first-page":"33","volume-title":"Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse","author":"Graham Yvette","year":"2013","unstructured":"Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse. The Association for Computer Linguistics, 33\u201341."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324915000339"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5357"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6454"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/GRC.2008.4664680"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.5555\/2998828.2998898"},{"key":"e_1_3_2_58_2","first-page":"1753","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Horbach Andrea","year":"2020","unstructured":"Andrea Horbach, Itziar Aldabe, Marie Bexte, Oier Lopez de Lacalle, and Montse Maritxalar. 2020. Linguistic appropriateness and pedagogic usefulness of reading comprehension questions. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, 1753\u20131762."},{"key":"e_1_3_2_59_2","unstructured":"Hassan Kan\u00e9 Muhammed Yusuf Kocyigit Ali Abdalla Pelkins Ajanoh and Mohamed Coulibali. 2020. NUBIA: NeUral based interchangeability assessor for text generation. EvalNLGEval 2020 (2020) 28."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-1019"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1014"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1051"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045221"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.104.2.211"},{"key":"e_1_3_2_65_2","volume-title":"Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics","author":"Leusch Gregor","year":"2006","unstructured":"Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT evaluation using block movements. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. The Association for Computer Linguistics."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1230"},{"key":"e_1_3_2_67_2","unstructured":"Margaret Li Jason Weston and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv:1909.03087. Retrieved from https:\/\/arxiv.org\/abs\/1909.03087."},{"key":"e_1_3_2_68_2","first-page":"986","volume-title":"Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Li Yanran","year":"2017","unstructured":"Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, 986\u2013995."},{"key":"e_1_3_2_69_2","article-title":"Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation","author":"Liang Weixin","year":"2020","unstructured":"Weixin Liang, James Zou, and Zhou Yu. 2020. Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation. CoRR abs\/2005.10716. (2020).","journal-title":"CoRR abs\/2005.10716"},{"key":"e_1_3_2_70_2","article-title":"A technique for the measurement of attitudes.","author":"Likert Rensis","year":"1932","unstructured":"Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology 22, 140 (1932), 55.","journal-title":"Archives of Psychology"},{"key":"e_1_3_2_71_2","first-page":"74","volume-title":"Proceedings of the Workshop on Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out. Association for Computational Linguistics, 74\u201381."},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1230"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.100"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4767"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5358"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2124"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.5555\/2393015.2393048"},{"key":"e_1_3_2_78_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Logeswaran Lajanugen","year":"2018","unstructured":"Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of the International Conference on Learning Representations. OpenReview.net."},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1103"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-4640"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6450"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4768"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5302"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.5555\/1626431.1626433"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1269"},{"key":"e_1_3_2_86_2","doi-asserted-by":"crossref","unstructured":"Nitika Mathur Tim Baldwin and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. 4984\u20134997.","DOI":"10.18653\/v1\/2020.acl-main.448"},{"key":"e_1_3_2_87_2","first-page":"688","volume-title":"Proceedings of the 5th Conference on Machine Translation","author":"Mathur Nitika","year":"2020","unstructured":"Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the 5th Conference on Machine Translation. Association for Computational Linguistics, 688\u2013725."},{"key":"e_1_3_2_88_2","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space.  arXiv:1301.3781. Retrieved from https:\/\/arxiv.org\/abs\/1301.3781."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1429"},{"key":"e_1_3_2_90_2","volume-title":"Proceedings of the 2nd International Conference on Language Resources and Evaluation","author":"Nie\u00dfen Sonja","year":"2000","unstructured":"Sonja Nie\u00dfen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation. European Language Resources Association."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1238"},{"key":"e_1_3_2_92_2","doi-asserted-by":"crossref","unstructured":"Juri Opitz and Anette Frank. 2020. Towards a decomposable metric for explainable evaluation of text generation from AMR.  arXiv:2008.08896. Retrieved from https:\/\/arxiv.org\/abs\/2008.08896.","DOI":"10.18653\/v1\/2021.eacl-main.129"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00329"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6455"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_2_96_2","unstructured":"Prasanna Parthasarathi Joelle Pineau and Sarath Chandar. 2020. How to evaluate your dialogue system: Probe tasks as an alternative for token-level evaluation metrics.  arXiv:2008.10427. Retrieved from https:\/\/arxiv.org\/abs\/2008.10427."},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.5555\/1597148.1597310"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1202"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.303"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3049"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4770"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6319"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00322"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2009.35.4.35405"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.5555\/1857999.1858019"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.5555\/2390384.2390403"},{"key":"e_1_3_2_109_2","unstructured":"Ananya B. Sai Tanay Dixit Dev Yashpal Sheth Sreyas Mohan and Mitesh M. Khapra. 2021. Perturbation checklists for evaluating NLG evaluation metrics.  arXiv:2109.05771. Retrieved from https:\/\/arxiv.org\/abs\/2109.05771."},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33016220"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00347"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.5555\/1520025.1520028"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1170"},{"key":"e_1_3_2_114_2","doi-asserted-by":"crossref","unstructured":"Thibault Sellam Dipanjan Das and Ankur P. Parikh. 2020. BLEURT: Learning robust metrics for text generation.  arXiv:2004.04696. Retrieved from https:\/\/arxiv.org\/abs\/2004.04696.","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.5555\/3016387.3016435"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-3003"},{"key":"e_1_3_2_117_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1007\/978-3-030-01237-3_3","volume-title":"Computer Vision \u2014 ECCV 2018","author":"Sharif Naeha","year":"2018","unstructured":"Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. 2018. NNEval: Neural network based evaluation metric for image captioning. In Computer Vision \u2014 ECCV 2018 . V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss (Eds.), Lecture Notes in Computer Science, Vol. 11212, Springer, 39\u201355."},{"key":"e_1_3_2_118_2","unstructured":"Shikhar Sharma Layla El Asri Hannes Schulz and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation.  arXiv:1706.09799. Retrieved from https:\/\/arxiv.org\/abs\/1706.09799."},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6456"},{"key":"e_1_3_2_120_2","unstructured":"Hiroki Shimanaka Tomoyuki Kajiwara and Mamoru Komachi. 2019. Machine translation evaluation with BERT regressor.  arXiv:1907.12679. Retrieved from https:\/\/arxiv.org\/abs\/1907.12679."},{"key":"e_1_3_2_121_2","volume-title":"WebNLG Challenge: Human Evaluation Results","author":"Shimorina Anastasia","year":"2019","unstructured":"Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2019. WebNLG Challenge: Human Evaluation Results. Technical Report."},{"key":"e_1_3_2_122_2","doi-asserted-by":"crossref","unstructured":"Koustuv Sinha Prasanna Parthasarathi Jasmine Wang Ryan Lowe William L. Hamilton and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation.  arXiv:2005.00583. Retrieved from https:\/\/arxiv.org\/abs\/2005.00583.","DOI":"10.18653\/v1\/2020.acl-main.220"},{"key":"e_1_3_2_123_2","volume-title":"Proceedings of the Association for Machine Translation in the Americas","author":"Snover Matthew","year":"2006","unstructured":"Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas."},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5359"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3031"},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1025"},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30586-6_38"},{"key":"e_1_3_2_128_2","unstructured":"Nisan Stiennon Long Ouyang Jeff Wu Daniel M. Ziegler Ryan Lowe Chelsea Voss Alec Radford Dario Amodei and Paul F. Christiano. 2020. Learning to summarize from human feedback.  arXiv:2009.01325. Retrieved from https:\/\/arxiv.org\/abs\/2009.01325."},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992137"},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.5555\/3504035.3504124"},{"key":"e_1_3_2_131_2","unstructured":"Ran Tian Shashi Narayan Thibault Sellam and Ankur P. Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation.  arXiv:1910.08684. Retrieved from https:\/\/arxiv.org\/abs\/1910.08684."},{"key":"e_1_3_2_132_2","volume-title":"European Conf. on Speech Communication and Technology","author":"Tillmann Christoph","year":"1997","unstructured":"Christoph Tillmann, Stephan Vogel, Hermann Ney, A. Zubiaga, and Hassan Sawaf. 1997. Accelerated DP based search for statistical translation. In European Conf. on Speech Communication and Technology. ISCA."},{"key":"e_1_3_2_133_2","unstructured":"Joseph Turian Luke Shen and I. Melamed. 2003. Evaluation of machine translation and its evaluation. In Proceedings of the MT Summit IX ."},{"key":"e_1_3_2_134_2","doi-asserted-by":"publisher","DOI":"10.1093\/mind\/LIX.236.433"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295349"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324919000123"},{"key":"e_1_3_2_138_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_3_2_139_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2342"},{"key":"e_1_3_2_140_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1427"},{"key":"e_1_3_2_141_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1042"},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1239"},{"key":"e_1_3_2_143_2","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3454804"},{"key":"e_1_3_2_144_2","doi-asserted-by":"publisher","DOI":"10.5555\/1626281.1626285"},{"key":"e_1_3_2_145_2","unstructured":"Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT.  arXiv:1904.09675. Retrieved from https:\/\/arxiv.org\/abs\/1904.09675."},{"key":"e_1_3_2_146_2","volume-title":"Proceedings of the 4th International Conference on Language Resources and Evaluation","author":"Zhang Ying","year":"2004","unstructured":"Ying Zhang, Stephan Vogel, and Alex Waibel. 2004. Interpreting BLEU\/NIST scores: How much improvement do we need to have a better system? In Proceedings of the 4th International Conference on Language Resources and Evaluation. European Language Resources Association."},{"key":"e_1_3_2_147_2","doi-asserted-by":"crossref","unstructured":"Tianyu Zhao Divesh Lala and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response evaluators.  arXiv:2004.04908. Retrieved from https:\/\/arxiv.org\/abs\/2004.04908.","DOI":"10.18653\/v1\/2020.acl-main.4"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1053"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3485766","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3485766","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:12:11Z","timestamp":1750176731000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3485766"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,18]]},"references-count":147,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,2,28]]}},"alternative-id":["10.1145\/3485766"],"URL":"https:\/\/doi.org\/10.1145\/3485766","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,18]]},"assertion":[{"value":"2020-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}