{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:54:21Z","timestamp":1750308861481,"version":"3.41.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,11,12]],"date-time":"2018-11-12T00:00:00Z","timestamp":1541980800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61662031, 61462044, and 61462045"],"award-info":[{"award-number":["61662031, 61462044, and 61462045"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2019,3,31]]},"abstract":"<jats:p>Automatic evaluation of machine translation is critical for the evaluation and development of machine translation systems. In this study, we propose a new model for automatic evaluation of machine translation. The proposed model combines standard n-gram precision features and sentence semantic mapping features with neural features, including neural language model probabilities and the embedding distances between translation outputs and their reference translations. We optimize the model with a representative list-wise learning to rank approach, ListMLE, in terms of human ranking assessments. The experimental results on WMT\u20192015 Metrics task indicated that the proposed approach yields significantly better correlations with human assessments than several state-of-the-art baseline approaches. In particular, the results confirmed that the proposed list-wise learning to rank approach is useful and powerful for optimizing automatic evaluation metrics in terms of human ranking assessments. Deep analysis also demonstrated that optimizing automatic metrics with the ListMLE approach is a reasonable method and adding the neural features can gain considerable improvements compared with the traditional features.<\/jats:p>","DOI":"10.1145\/3226045","type":"journal-article","created":{"date-parts":[[2018,11,13]],"date-time":"2018-11-13T13:20:16Z","timestamp":1542115216000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Optimizing Automatic Evaluation of Machine Translation with the ListMLE Approach"],"prefix":"10.1145","volume":"18","author":[{"given":"Maoxi","family":"Li","sequence":"first","affiliation":[{"name":"Jiangxi Normal University, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingwen","family":"Wang","sequence":"additional","affiliation":[{"name":"Jiangxi Normal University, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,11,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1289189.1289273"},{"volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. Association for Computational Linguistics, 65--72","author":"Banerjee S.","key":"e_1_2_1_3_1"},{"volume-title":"Proceedings of Association for Machine Translation in the Americas. Association for Machine Translation in the Americas, 223--231","author":"Snover M.","key":"e_1_2_1_4_1"},{"volume-title":"Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 55--62","author":"Chan Y. S.","key":"e_1_2_1_5_1"},{"volume-title":"Proceedings of the Joint 5th Workshop on Statistical Machine Translation and Metrics (MATR). Association for Computational Linguistics, 354--359","author":"Liu C.","key":"e_1_2_1_6_1"},{"volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 930--939","author":"Chen B.","key":"e_1_2_1_7_1"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1124"},{"volume-title":"Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 505--510","author":"Wang W.","key":"e_1_2_1_9_1"},{"volume-title":"Proceedings of the 2nd Workshop on Statistical Machine Translation. Association for Computational Linguistics, 136--158","author":"Callison-Burch C.","key":"e_1_2_1_10_1"},{"volume-title":"Proceedings of IWSLT\u20192008","year":"2008","author":"Paul M.","key":"e_1_2_1_11_1"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073012.1073032"},{"volume-title":"Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation. Sk\u00f6vde University Studies in Informatics, 75--84","author":"Kulesza A.","key":"e_1_2_1_13_1"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-008-9046-1"},{"volume-title":"Proceedings of the 9th Conference of the Association for Machine Translation in the Americas. Association for Machine Translation in the Americas, 1--10","author":"Specia L.","key":"e_1_2_1_15_1"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/1626394.1626425"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"F. Guzman S. Joty L. Marquez and P. Nakov. 2015. Pairwise neural machine translation evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Association for Computational Linguistics. 805--814.  F. Guzman S. Joty L. Marquez and P. Nakov. 2015. Pairwise neural machine translation evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Association for Computational Linguistics. 805--814.","DOI":"10.3115\/v1\/P15-1078"},{"volume-title":"Proceedings of the 6th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 123--129","author":"Song X.","key":"e_1_2_1_18_1"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390306"},{"volume-title":"Proceedings of the 3rd Workshop on Statistical Machine Translation. Association for Computational Linguistics, 70-106","author":"Callison-Burch C.","key":"e_1_2_1_20_1"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-009-5108-8"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"T. Joachims. 1999. Making large-scale support vector machine learning practical. Advances in Kernel Methods. MIT Press 169--184.   T. Joachims. 1999. Making large-scale support vector machine learning practical. Advances in Kernel Methods. MIT Press 169--184.","DOI":"10.7551\/mitpress\/1130.003.0015"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"H. Li. 2011. A short introduction to learning to rank. IEICE Transactions on Information and Systems E94-D. 1--9.  H. Li. 2011. A short introduction to learning to rank. IEICE Transactions on Information and Systems E94-D. 1--9.","DOI":"10.1587\/transinf.E94.D.1854"},{"volume-title":"Proceedings of the 9th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 293--301","author":"Machacek M.","key":"e_1_2_1_25_1"},{"volume-title":"Proceedings of the 10th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 256--273","author":"Stanojevic M.","key":"e_1_2_1_26_1"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277790"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341544"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346567"},{"volume-title":"Proceedings of the 7th International Conference on Spoken Language Processing. ISCA, 901--904","year":"2002","author":"Stolcke A.","key":"e_1_2_1_31_1"},{"volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 98--107","author":"He X.","key":"e_1_2_1_32_1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815619"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/972470.972474"},{"volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1724--1734","author":"Cho K.","key":"e_1_2_1_35_1"},{"volume-title":"Proceedings of the 11th Annual Conference of the International Speech Communication Association. International Speech Communication Association, 1045--1048","author":"Mikolov T.","key":"e_1_2_1_36_1"},{"key":"e_1_2_1_37_1","unstructured":"T. Mikolov I. Sutskever K. Chen G. Corrado and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26. 3111--3119.   T. Mikolov I. Sutskever K. Chen G. Corrado and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26. 3111--3119."},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"B. Chen and H. Guo. 2015. Representation based translation evaluation metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Association for Computational Linguistics 150--155.  B. Chen and H. Guo. 2015. Representation based translation evaluation metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Association for Computational Linguistics 150--155.","DOI":"10.3115\/v1\/P15-2025"},{"volume-title":"Proceedings of the 10th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 1--46","author":"Bojar O.","key":"e_1_2_1_39_1"},{"volume-title":"Proceedings of the 9th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 376--380","author":"Denkowski M.","key":"e_1_2_1_40_1"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3047"},{"volume-title":"Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 232--238","author":"Bojar O.","key":"e_1_2_1_42_1"},{"volume-title":"Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 388--395","year":"2004","author":"Koehn P.","key":"e_1_2_1_43_1"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-010-9073-6"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3226045","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3226045","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:15:01Z","timestamp":1750281301000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3226045"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,12]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,3,31]]}},"alternative-id":["10.1145\/3226045"],"URL":"https:\/\/doi.org\/10.1145\/3226045","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"type":"print","value":"2375-4699"},{"type":"electronic","value":"2375-4702"}],"subject":[],"published":{"date-parts":[[2018,11,12]]},"assertion":[{"value":"2017-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-11-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}