{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T23:32:20Z","timestamp":1773358340144,"version":"3.50.1"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:p>Embedding-based evaluation measures have shown promising improvements on the correlation with human judgments in natural language generation. In these measures, various intrinsic metrics are used in the computation, including generalized precision, recall, F-score and the earth mover's distance. However, the relations between these metrics are unclear, making it difficult to determine which measure to use in real applications. In this paper, we provide an in-depth study on the relations between these metrics. Inspired by the optimal transportation theory, we prove that these metrics correspond to the optimal transport problem with different hard marginal constraints. However, these hard marginal constraints may cause the problem of incomplete and noisy matching in the evaluation process. Therefore we propose a family of new evaluation metrics, namely Lazy Earth Mover's Distances, based on the more general unbalanced optimal transport problem. Experimental results on WMT18 and WMT19 show that our proposed metrics have the ability to produce more consistent evaluation results with human judgements, as compared with existing intrinsic metrics.<\/jats:p>","DOI":"10.24963\/ijcai.2020\/516","type":"proceedings-article","created":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T12:12:10Z","timestamp":1594210330000},"page":"3730-3736","source":"Crossref","is-referenced-by-count":3,"title":["Evaluating Natural Language Generation via Unbalanced Optimal Transport"],"prefix":"10.24963","author":[{"given":"Yimeng","family":"Chen","sequence":"first","affiliation":[{"name":"University of Chinese Academy of Sciences"},{"name":"Academy of Mathematics and Systems Science, CAS"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanyan","family":"Lan","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, CAS"},{"name":"University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruinbin","family":"Xiong","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, CAS"},{"name":"University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liang","family":"Pang","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, CAS"},{"name":"University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiming","family":"Ma","sequence":"additional","affiliation":[{"name":"University of Chinese Academy of Sciences"},{"name":"Academy of Mathematics and Systems Science, CAS"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xueqi","family":"Cheng","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, CAS"},{"name":"University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10584","event":{"name":"Twenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence {IJCAI-PRICAI-20}","theme":"Artificial Intelligence","location":"Yokohama, Japan","acronym":"IJCAI-PRICAI-2020","number":"28","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2020,7,11]]},"end":{"date-parts":[[2020,7,17]]}},"container-title":["Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T02:15:39Z","timestamp":1594260939000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2020\/516"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2020,7]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2020\/516","relation":{},"subject":[],"published":{"date-parts":[[2020,7]]}}}