{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,4,5]],"date-time":"2022-04-05T13:08:32Z","timestamp":1649164112276},"reference-count":24,"publisher":"World Scientific Pub Co Pte Lt","issue":"01","funder":[{"name":"JSPS KAKENHI","award":["17K18481"],"award-info":[{"award-number":["17K18481"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. As. Lang. Proc."],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:p> Current methods of neural machine translation may generate sentences with different levels of quality. Methods for automatically evaluating translation output from machine translation can be broadly classified into two types: a method that uses human post-edited translations for training an evaluation model, and a method that uses a reference translation that is the correct answer during evaluation. On the one hand, it is difficult to prepare post-edited translations because it is necessary to tag each word in comparison with the original translated sentences. On the other hand, users who actually employ the machine translation system do not have a correct reference translation. Therefore, we propose a method that trains the evaluation model without using human post-edited sentences and in the test set, estimates the quality of output sentences without using reference translations. We define some indices and predict the quality of translations with a regression model. For the quality of the translated sentences, we employ the BLEU score calculated from the number of word [Formula: see text]-gram matches between the translated sentence and the reference translation. After that, we compute the correlation between quality scores predicted by our method and BLEU actually computed from references. According to the experimental results, the correlation with BLEU is the highest when XGBoost uses all the indices. Moreover, looking at each index, we find that the sentence log-likelihood and the model uncertainty, which are based on the joint probability of generating the translated sentence, are important in BLEU estimation. <\/jats:p>","DOI":"10.1142\/s2717554520500022","type":"journal-article","created":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T02:45:27Z","timestamp":1590115527000},"page":"2050002","source":"Crossref","is-referenced-by-count":0,"title":["Estimating Machine Translation Quality of Any Input Sentence"],"prefix":"10.1142","volume":"30","author":[{"given":"Taichi","family":"Aida","sequence":"first","affiliation":[{"name":"Department of Electrical, Electronics and Information Engineering, Nagaoka University of Technology, Nagaoka City, Niigata Prefecture 940-2188, Japan"}]},{"given":"Kazuhide","family":"Yamamoto","sequence":"additional","affiliation":[{"name":"Department of Electrical, Electronics and Information Engineering, Nagaoka University of Technology, Nagaoka City, Niigata Prefecture 940-2188, Japan"}]}],"member":"219","published-online":{"date-parts":[[2020,5,20]]},"reference":[{"key":"S2717554520500022BIB001","first-page":"349","volume-title":"Proc. 2019 Int. Conf. Asian Language Processing","author":"Aida T.","year":"2019"},{"key":"S2717554520500022BIB002","first-page":"311","volume-title":"Proc. 40th Annu. Meeting of the Association for Computational Linguistics","author":"Papineni K.","year":"2002"},{"key":"S2717554520500022BIB003","first-page":"785","volume-title":"Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining","author":"Chen T.","year":"2016"},{"key":"S2717554520500022BIB004","doi-asserted-by":"crossref","first-page":"77","DOI":"10.2478\/v10108-010-0022-6","volume":"94","author":"Gim\u00e9nez J.","year":"2010","journal-title":"Prague Bull. Math. Linguist."},{"key":"S2717554520500022BIB005","doi-asserted-by":"crossref","first-page":"417","DOI":"10.18653\/v1\/W15-3053","volume-title":"Proc. Tenth Workshop on Statistical Machine Translation","author":"Yu H.","year":"2015"},{"key":"S2717554520500022BIB006","doi-asserted-by":"crossref","first-page":"598","DOI":"10.18653\/v1\/W17-4768","volume-title":"Proc. Second Conf. Machine Translation","author":"Ma Q.","year":"2017"},{"key":"S2717554520500022BIB007","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.18653\/v1\/D15-1124","volume-title":"Proc. 2015 Conf. Empirical Methods in Natural Language Processing","author":"Gupta R.","year":"2015"},{"key":"S2717554520500022BIB009","first-page":"590","volume-title":"Proc. Twenty-Fifth Annu. Meeting of the Association for Natural Language Processing","author":"Shimanaka H.","year":"2019"},{"key":"S2717554520500022BIB010","doi-asserted-by":"crossref","first-page":"115","DOI":"10.3115\/v1\/P15-4020","volume-title":"Proc. ACL-IJCNLP 2015 System Demonstrations","author":"Specia L.","year":"2015"},{"key":"S2717554520500022BIB011","doi-asserted-by":"crossref","first-page":"208","DOI":"10.18653\/v1\/K16-1021","volume-title":"Proc. 20th SIGNLL Conf. Computational Natural Language Learning","author":"Beck D.","year":"2016"},{"key":"S2717554520500022BIB013","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1162\/tacl_a_00056","volume":"5","author":"Martins A. F. T.","year":"2017","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"S2717554520500022BIB014","doi-asserted-by":"crossref","first-page":"562","DOI":"10.18653\/v1\/W17-4763","volume-title":"Proc. Second Conf. Machine Translation","author":"Kim H.","year":"2017"},{"key":"S2717554520500022BIB015","first-page":"3146","volume-title":"Proc. 27th Int. Conf. Computational Linguistics","author":"Ive J.","year":"2018"},{"key":"S2717554520500022BIB020","doi-asserted-by":"crossref","first-page":"117","DOI":"10.18653\/v1\/P19-3020","volume-title":"Proc. 57th Annu. Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Kepler F.","year":"2019"},{"key":"S2717554520500022BIB021","doi-asserted-by":"crossref","first-page":"316","DOI":"10.18653\/v1\/W15-3037","volume-title":"Proc. Tenth Workshop on Statistical Machine Translation","author":"Kreutzer J.","year":"2015"},{"key":"S2717554520500022BIB024","first-page":"1050","volume-title":"Proc. 33rd Int. Conf. Machine Learning","volume":"48","author":"Gal Y.","year":"2016"},{"key":"S2717554520500022BIB025","first-page":"1929","volume":"15","author":"Srivastava N.","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"S2717554520500022BIB027","first-page":"791","volume-title":"Proc. 2019 Conf. Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP)","author":"Wang S.","year":"2019"},{"key":"S2717554520500022BIB028","doi-asserted-by":"crossref","first-page":"250","DOI":"10.18653\/v1\/W19-4330","volume-title":"Proc. 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)","author":"Chidambaram M.","year":"2019"},{"key":"S2717554520500022BIB029","doi-asserted-by":"crossref","first-page":"169","DOI":"10.18653\/v1\/D18-2029","volume-title":"Proc. 2018 Conf. Empirical Methods in Natural Language Processing: System Demonstrations","author":"Cer D.","year":"2018"},{"key":"S2717554520500022BIB030","first-page":"48","volume-title":"Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics (Demonstrations)","author":"Ott M.","year":"2019"},{"key":"S2717554520500022BIB031","first-page":"2204","volume-title":"Proc. Tenth Int. Conf. Language Resources and Evaluation (LREC\u201916)","author":"Nakazawa T.","year":"2016"},{"key":"S2717554520500022BIB032","doi-asserted-by":"crossref","first-page":"501","DOI":"10.3115\/1220355.1220427","volume-title":"Proc. 20th Int. Conf. Computational Linguistics (COLING 2004)","author":"Lin C.-Y.","year":"2004"},{"key":"S2717554520500022BIB033","first-page":"7322","volume-title":"Proc. Thirty-Third AAAI Conf. Artificial Intelligence","author":"Xiao Y.","year":"2019"}],"container-title":["International Journal of Asian Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S2717554520500022","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,6,4]],"date-time":"2020-06-04T07:18:31Z","timestamp":1591255111000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/abs\/10.1142\/S2717554520500022"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3]]},"references-count":24,"journal-issue":{"issue":"01","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["10.1142\/S2717554520500022"],"URL":"https:\/\/doi.org\/10.1142\/s2717554520500022","relation":{},"ISSN":["2717-5545","2424-791X"],"issn-type":[{"value":"2717-5545","type":"print"},{"value":"2424-791X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3]]}}}