{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T06:32:49Z","timestamp":1777703569174,"version":"3.51.4"},"reference-count":26,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2018,5,18]],"date-time":"2018-05-18T00:00:00Z","timestamp":1526601600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2018,5,24]]},"abstract":"<jats:p>The study describes an experiment with different estimations of reliability. Reliability reflects the technical quality of the measurement procedure such as an automatic evaluation of Machine Translation (MT). Reliability is an indicator of accuracy, the reliability of measuring, in our case, measuring the accuracy and error rate of MT output based on automatic metrics (precision, recall, f-measure, Bleu-n, WER, PER, and CDER). The experiment showed metrics (Bleu-4 and WER) that reduce the overall reliability of the automatic evaluation of accuracy and error rate using entropy. Based on the results we can say, that the use of entropy for the estimation of reliability brings more accurate results than conventional estimations of reliability (Cronbach\u2019s alpha and correlation). MT evaluation, based on n-grams or edit distance, using entropy could offer a new view on lexicon-based metrics in comparison to commonly used ones.<\/jats:p>","DOI":"10.3233\/jifs-169505","type":"journal-article","created":{"date-parts":[[2018,5,22]],"date-time":"2018-05-22T10:54:27Z","timestamp":1526986467000},"page":"3225-3233","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":11,"title":["Towards the use of entropy as a measure for\u00a0the reliability of automatic MT evaluation metrics"],"prefix":"10.1177","volume":"34","author":[{"given":"Michal","family":"Munk","sequence":"first","affiliation":[{"name":"Department of Informatics, Constantine the Philosopher University in Nitra, Tr. A. Hlinku, Nitra, Slovakia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dasa","family":"Munkova","sequence":"additional","affiliation":[{"name":"Department of Translation Studies, Constantine the Philosopher University in Nitra, Stefanikova, Nitra, Slovakia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lubomir","family":"Benko","sequence":"additional","affiliation":[{"name":"Institute of System Engineering and Informatics, University of Pardubice, Studentska, Pardubice, Czech Republic"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2018,5,18]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"209","volume-title":"Interact Knowl Discov Data Min Biomed Informatics State-of-the-Art Futur Challenges","author":"Holzinger A.","year":"2014","unstructured":"HolzingerA., H\u00f6rtenhuberM., MayerC., BachlerM., WassertheurerS., PinhoA.J., KoslickiD., On Entropy-Based Data Mining, HolzingerA. and JurisicaI.Interact Knowl Discov Data Min Biomed Informatics State-of-the-Art Futur ChallengesBerlin Heidelberg, Sringer Heidelberg (2014), 209\u2013226, doi:10.1007\/978-3-662-43968-5_12."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2016.06.002"},{"key":"e_1_3_2_4_2","first-page":"136","article-title":"(Meta-) evaluation of machine translation","author":"Callison-Burch C.","year":"2017","unstructured":"Callison-BurchC., FordyceC., KoehnP., MonzC. and SchroederJ., (Meta-) evaluation of machine translation, Proc Second Work Stat Mach Transl (2017) 136\u2013158. http:\/\/dl.acm.org\/citation.cfm?id=1626373,(accessed August 8, 2017.","journal-title":"Proc Second Work Stat Mach Transl"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.21437\/Eurospeech.1997-673"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/584091.584093"},{"key":"e_1_3_2_7_2","doi-asserted-by":"crossref","unstructured":"LimaC.F.L. de AssisF.M. de SouzaC.P. A Comparative Study of Use of Shannon R\u00e9nyi and Tsallis Entropy for Attribute Selecting in Network Intrusion Detection Berlin Sringer Heidelberg 2012 pp. 492\u2013501. doi:10.1007\/978-3-642-32639-4_60","DOI":"10.1007\/978-3-642-32639-4_60"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICAICT.2014.7035992"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-30298-0_69"},{"key":"e_1_3_2_10_2","first-page":"590","article-title":"Parallel corpora for medium density languages","author":"Varga D.","year":"2005","unstructured":"VargaD., N\u00e9methL., Hal\u00e1csyP., KornaiA., Tr\u00f3nV. and NagyV., Parallel corpora for medium density languages, Proc RANLP 2005 (2005), 590\u2013596.","journal-title":"Proc RANLP 2005"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRev.106.620"},{"key":"e_1_3_2_12_2","article-title":"CDER: Efficient MT Evaluation Using Block Movements","author":"Leusch G.","year":"2006","unstructured":"LeuschG., UeffingN. and NeyH., CDER: Efficient MT Evaluation Using Block Movements, Proc 11th Conf Eur Chapter Assoc Comput Linguist (EACL 2006)2006.","journal-title":"Proc 11th Conf Eur Chapter Assoc Comput Linguist (EACL 2006)"},{"key":"e_1_3_2_13_2","unstructured":"YuH. WuX. JiangW. LiuQ. and LinS. Improve the Evaluation of Fluency Using Entropy for Machine Translation Evaluation Metrics (accessed August 8 (2017) http:\/\/arxiv.org\/abs\/1508.02225."},{"key":"e_1_3_2_14_2","first-page":"311","volume-title":"Proc 40th Annu Meet Assoc Comput Linguist","author":"Papineni K.","year":"2002","unstructured":"PapineniK., RoukosS., WardT., ZhuW., BLEU: A method for automatic evaluation of machine translation, Proc 40th Annu Meet Assoc Comput Linguist , Philadelphia (2002), 311\u2013318."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2392"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2016.2639049"},{"key":"e_1_3_2_17_2","first-page":"45","volume-title":"Proc Work Autom Man Metrics Oper Transl Eval MTE 2014","author":"Carl M.","year":"2014","unstructured":"CarlM., SchaefferM., Word Transition Entropy as an Indicator for Expected Machine Translation QualityProc Work Autom Man Metrics Oper Transl Eval MTE 2014 , ParisELRA (2014), 45\u201350."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49397-8_12"},{"key":"e_1_3_2_19_2","first-page":"20","volume-title":"Proc Work Autom Man Metrics Oper Transl Eval MTE 2014","author":"Vela M.","year":"2014","unstructured":"VelaM., SchumannA.-K., WurmA., Human Translation Evaluation and its Coverage by Automatic ScoresProc Work Autom Man Metrics Oper Transl Eval MTE 2014 , ParisELRA (2014), 20\u201330."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0019875"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-013-9146-4"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2016.2632158"},{"key":"e_1_3_2_23_2","unstructured":"ClausiusR. On the Motive Power of Heat and on the Laws which Can be Deduced from it for the Theory of Heat Dover (1960) https:\/\/books.google.sk\/books?id=WEzmcQAACAAJ."},{"key":"e_1_3_2_24_2","first-page":"24","article-title":"Scale development","author":"DeVellis R.F.","year":"1991","unstructured":"DeVellisR.F., Scale development, Sage Publ (1991), 24\u201333.","journal-title":"Sage Publ"},{"key":"e_1_3_2_25_2","article-title":"Sage, Los Angeles","author":"DeVellis R.F.","year":"2012","unstructured":"DeVellisR.F., Sage, Los Angeles, Scale Development: Theory and applications2012.","journal-title":"Scale Development: Theory and applications"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-015-9176-1"},{"key":"e_1_3_2_27_2","first-page":"39","article-title":"An evaluation tool for machine translation: Fast evaluation for MT research","author":"Nie\u00dfen S.","year":"2000","unstructured":"Nie\u00dfenS., OchF.J., LeuschG. and NeyH., An evaluation tool for machine translation: Fast evaluation for MT research, Proc 2nd Int Conf Lang Resour Eval (2000), 39\u201345.","journal-title":"Proc 2nd Int Conf Lang Resour Eval"}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-169505","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-169505","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-169505","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:39:00Z","timestamp":1777455540000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-169505"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,18]]},"references-count":26,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2018,5,24]]}},"alternative-id":["10.3233\/JIFS-169505"],"URL":"https:\/\/doi.org\/10.3233\/jifs-169505","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,5,18]]}}}