{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T06:32:50Z","timestamp":1777703570126,"version":"3.51.4"},"reference-count":38,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2018,5,18]],"date-time":"2018-05-18T00:00:00Z","timestamp":1526601600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2018,5,24]]},"abstract":"<jats:p>Errors and residuals are closely related measures of the deviation. An error is a deviation of the observed value (PEMT output) from the expected value (MT output), while the residual of the observed value is the difference between the observed and predicted value of quality. We propose an exploratory data technique representing an ideal instrument to evaluate and improve machine translation (MT) systems. The main contribution consists of a rigorous technique (a statistical method), novel to the research of MT evaluation given by residual analysis to identify differences between MT output and post-edited machine translation output regarding human translation (reference). The residual analysis of the automatic metrics can help us to discover significant differences between MT and PEMT and to identify questionable issues regarding the one reference. In this study, we show the usage of residuals in MT evaluation. Using residual analysis, we identified sentences, in which significant differences were found in the scores of automatic metrics between MT output and post-edited (PE) MT output from Slovak into English.<\/jats:p>","DOI":"10.3233\/jifs-169504","type":"journal-article","created":{"date-parts":[[2018,5,22]],"date-time":"2018-05-22T10:54:20Z","timestamp":1526986460000},"page":"3211-3223","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":10,"title":["Detecting errors in machine translation using residuals and metrics of automatic evaluation"],"prefix":"10.1177","volume":"34","author":[{"given":"Michal","family":"Munk","sequence":"first","affiliation":[{"name":"Department of Informatics, Constantine the Philosopher University in Nitra, Tr. A. Hlinku, Nitra, Slovakia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dasa","family":"Munkova","sequence":"additional","affiliation":[{"name":"Department of Translation Studies, Constantine the Philosopher University in Nitra, Stefanikova, Nitra, Slovakia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2018,5,18]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-014-9155-y"},{"key":"e_1_3_2_3_2","first-page":"441","article-title":"A robust evaluation metric for machine translation with augmented factors","author":"Han A.L.-F.","year":"2012","unstructured":"HanA.L.-F., WongD.F. and ChaoL.S., A robust evaluation metric for machine translation with augmented factors, Proceedings of COLING (2012), 441\u2013450.","journal-title":"Proceedings of COLING"},{"key":"e_1_3_2_4_2","first-page":"59","volume-title":"Improving amber, an mt evaluation metric","author":"Chen B.","year":"2012","unstructured":"ChenB., KuhnR. and FosterG., Improving amber, an mt evaluation metric, Proceedings of the 7th Workshop on Statistical Machine Translation2012, pp. 59\u201363."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.21437\/Eurospeech.1997-673"},{"key":"e_1_3_2_6_2","first-page":"220","article-title":"Meant: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via antic frames","author":"Lo C.-K.","year":"2011","unstructured":"LoC.-K. and WuD., Meant: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via antic frames, Proceedings of ACL (2011), 220\u2013229.","journal-title":"Proceedings of ACL"},{"key":"e_1_3_2_7_2","first-page":"10","volume-title":"Structured vs. flat semantic role representations for machine translation evaluation","author":"Lo C.-K.","year":"2011","unstructured":"LoC.-K. and WuD., Structured vs. flat semantic role representations for machine translation evaluation, Proceedings of the 5th Workshop on Syntax and Structure in Statistical Translation (2011), pp. 10\u201320."},{"key":"e_1_3_2_8_2","first-page":"243","volume-title":"Fully automatic semantic mt evaluation","author":"Lo C.-K.","year":"2012","unstructured":"LoC.-K., TurmuluruA.K. and WuD., Fully automatic semantic mt evaluation, Proceedings of the 7th Workshop on Statistical Machine Translation2012, pp. 243\u2013252."},{"key":"e_1_3_2_9_2","first-page":"203","volume-title":"Task-based evaluation of machine translation (mt) engines: Measuring how well people extract who, when, where-type elements in mt output","author":"Voss C.R.","year":"2006","unstructured":"VossC.R. and TateR.R., Task-based evaluation of machine translation (mt) engines: Measuring how well people extract who, when, where-type elements in mt output, Proceedings of 11th Annual Conference of the European Association for Machine Translation2006, pp. 203\u2013212."},{"key":"e_1_3_2_10_2","first-page":"697","volume-title":"Error analysis of statistical machine translation output","author":"Vilar D.","year":"2006","unstructured":"VilarD., XuJ., D\u2019HaroL.F. and NeyH., Error analysis of statistical machine translation output, Proceedings of the 5th International Conference on Language Resources and Evaluation2006, 697\u2013702."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.3115\/1289189.1289273"},{"key":"e_1_3_2_12_2","unstructured":"TouryG. Descriptive Translation Studies and Beyond 1995. Amsterdam and Philadelphia: John Benjamins Publishing Company."},{"key":"e_1_3_2_13_2","unstructured":"TouryG. In Search of a Theory of Translation Tel Aviv: The Poter Institute for Poetics and Semiotics 1980."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.3115\/1626355.1626393"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"HouseJ. Translation quality assessment: Past and present London and NY: Routledge 2015.","DOI":"10.1057\/9781137025487_13"},{"key":"e_1_3_2_16_2","first-page":"2048","article-title":"Task-based mt evaluation: From who\/when\/where extraction to event understanding","author":"Laoudi J.","year":"2006","unstructured":"LaoudiJ., TateR.R. and VossC.R., Task-based mt evaluation: From who\/when\/where extraction to event understanding, Proceedings of LREC06 (2006), 2048\u20132053.","journal-title":"Proceedings of LREC06"},{"issue":"34","key":"e_1_3_2_17_2","first-page":"67","article-title":"An experiment in evaluating the quality of translation","volume":"9","author":"Carroll J.B.","year":"1966","unstructured":"CarrollJ.B., An experiment in evaluating the quality of translation, Mechanical Translation and Computational Linguistics9(34) (1966), 67\u201375.","journal-title":"Mechanical Translation and Computational Linguistics"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/bimj.200610341"},{"key":"e_1_3_2_19_2","first-page":"21","article-title":"A task-oriented evaluation metric for machine translation, pp","author":"White J.S.","year":"1998","unstructured":"WhiteJ.S. and TaylorK.B., A task-oriented evaluation metric for machine translation, pp, First International Conference on Language Resources & Evaluation (1998), 21\u201325.","journal-title":"First International Conference on Language Resources & Evaluation"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00981759"},{"key":"e_1_3_2_21_2","first-page":"311","article-title":"Bleu: A method for automatic evaluation of machine translation, pp","author":"Papineni K.","year":"2002","unstructured":"PapineniK., RoukosS., WardT. and ZhuW.-J., Bleu: A method for automatic evaluation of machine translation, pp, Proceedings of ACL (2002), pp. 311\u2013318.","journal-title":"Proceedings of ACL"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.2307\/270937"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992137"},{"key":"e_1_3_2_24_2","unstructured":"HildrethL.A. Residual analysis for structural equation modelling Graduate Theses and Dissertations 13400 http:\/\/lib.dr.iastate.edu\/etd\/13400."},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","first-page":"118","DOI":"10.26034\/cm.jostrans.2015.342","article-title":"On the correctness of machine translation: A machine translation post-editing task","volume":"23","author":"Koponen M.","year":"2015","unstructured":"KoponenM. and SalmiL., On the correctness of machine translation: A machine translation post-editing task, Journal of Specialised Translation23 (2015), 118\u2013136.","journal-title":"Journal of Specialised Translation"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21934-4_39"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.2478\/v10108-010-0010-x"},{"key":"e_1_3_2_28_2","first-page":"48","article-title":"Word error rates: Decomposition over pos classes and applications for error analysis","author":"Popovic M.","year":"2007","unstructured":"PopovicM. and NeyH., Word error rates: Decomposition over pos classes and applications for error analysis, Proceedings of the Second ACL Workshop on Statistical Machine Translation (2007), pp. 48\u201355.","journal-title":"Proceedings of the Second ACL Workshop on Statistical Machine Translation"},{"key":"e_1_3_2_29_2","first-page":"223","article-title":"A study of translation edit rate with targeted human annotation","author":"Snover M.","year":"2006","unstructured":"SnoverM., DorrB., SchwartzR., MicciullaL. and MakhoulJ., A study of translation edit rate with targeted human annotation, Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (2006), pp. 223\u2013231.","journal-title":"Proceedings of the 7th Conference of the Association for Machine Translation in the Americas"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.17533\/udea.mut.1825"},{"key":"e_1_3_2_31_2","first-page":"20","article-title":"Comparison of post-editing productivity between professional translators and lay users","author":"Aranberri N.","year":"2014","unstructured":"AranberriN., LabakaG., D\u00edaz de IlarrazaA., and SarasolaK., Comparison of post-editing productivity between professional translators and lay users, Proceedings of the Third Workshop on Post-Editing Technology and Practice2014, pp. 20\u201333.","journal-title":"Proceedings of the Third Workshop on Post-Editing Technology and Practice"},{"key":"e_1_3_2_32_2","unstructured":"NIST\/SEMATECH e-Handbook of Statistical Methods http:\/\/www.itl.nist.gov\/div898\/handbook\/."},{"key":"e_1_3_2_33_2","first-page":"1","volume-title":"Findings of the workshop on statistical machine translation","author":"Bojar O.","year":"2013","unstructured":"BojarO., BuckC., Callison-BurchC., FedermannC., HaddowB., KoehnP., MonzC., PostM., SoricutR. and SpeciaL., Findings of the workshop on statistical machine translation, Proceedings of the eighth Workshop on Statistical Machine Translation2013, pp. 1\u201344."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3302"},{"key":"e_1_3_2_35_2","first-page":"1","article-title":"Findings of the workshop on statistical machine translation","author":"Bojar O.","year":"2015","unstructured":"BojarO., ChatterjeeR., FedermannC., HaddowB., HuckM., HokampC., KoehnP., LogachevaV., MonzC., NegriM., PostM., ScartonC., SpeciaL. and TurchiM., Findings of the workshop on statistical machine translation, Proceedings of the Tenth Workshop on Statistical Machine Translation (2015), pp. 1\u201346.","journal-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1002\/sim.1731"},{"key":"e_1_3_2_37_2","unstructured":"HoerlR.W. The Reality of Residual Analysis Quality Progress 2008."},{"key":"e_1_3_2_38_2","first-page":"65","article-title":"Meteor: An automatic metric for mt evaluationwith improvedcorrelationwith human judgments","author":"Banerjee S.","year":"2005","unstructured":"BanerjeeS. and LavieA., Meteor: An automatic metric for mt evaluationwith improvedcorrelationwith human judgments, Proceedings of the ACL (2005), 65\u201372.","journal-title":"Proceedings of the ACL"},{"key":"e_1_3_2_39_2","unstructured":"O\u2019BrienS. BallingL.W. CarlM. SimardM. and SpeciaL. eds. Post-editing of Machine Translation: Processes and Applications Newcastle upon Tyne: Cambridge Scholars Publishing 2014."}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-169504","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-169504","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-169504","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:39:00Z","timestamp":1777455540000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-169504"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,18]]},"references-count":38,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2018,5,24]]}},"alternative-id":["10.3233\/JIFS-169504"],"URL":"https:\/\/doi.org\/10.3233\/jifs-169504","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,5,18]]}}}