{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T18:31:54Z","timestamp":1769797914970,"version":"3.49.0"},"reference-count":66,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T00:00:00Z","timestamp":1769558400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Ministry of Science and Higher Education of the Republic of Kazakhstan","award":["AP23487816"],"award-info":[{"award-number":["AP23487816"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>This study presents the application of free, open-source artificial intelligence (AI) techniques to advance machine translation for low-resource Turkic languages such as Kazakh, Azerbaijani, Kyrgyz, Turkish, Turkmen, and Uzbek. This machine translation problem for Turkic languages is part of a project to generate meeting minutes from speech transcripts. Due to limited parallel corpora and underdeveloped linguistic tools for these languages, traditional machine translation approaches often underperform. The goal is to reduce digital inequality for these languages and to support scalability. We investigate the effectiveness of free open-source pre-trained specialized and general-purpose AI models for morphologically rich state Turkic languages. This research includes developing parallel corpora for six Turkic languages, fine-tuning, and performance evaluation using BLEU, WER, TER, and chrF metrics. The parallel corpora for five pair languages, each of 300,000 and 500,000 sentences, were generated and cleaned. The results for corpora 500,000 parallel sentences show significant improvements compared with baseline NLLB-200 1.3B on average: BLEU increased by 23.81 points, chrF increased by 26.05 points, and WER and TER decreased by 0.36 and 33.95, respectively, after cleaning and fine-tuning. Six Turkic-language multilingual parallel corpora of 3 885 542 sentences were developed and the fine-tuning of NLLB-200 1.3B shows the following, compared with the results for 500,000 cleaned corpus: BLEU increased by 4.3 points, chrF increased by 1.7 points, and WER and TER decreased by 0.1 and 4.75, respectively These results demonstrate the high efficiency of corpus cleaning and synthetic data generation to improve the quality of machine translation for low-resource Turkic languages using AI models. These results were confirmed by external evaluation on the FLORES 200 dataset and human evaluation. The scientific contribution of this article is the development of a methodology for generating parallel corpora using a specialized AI model of machine translation and fine-tuning the specialized AI model on the created corpora, creating new multilingual parallel corpora of Azerbaijan\u2013Kazakh, Kyrgyz\u2013Kazakh, Turkish\u2013Kazakh, Turkmen\u2013Kazakh, and Uzbek\u2013Kazakh pairs using the proposed methodology, cleaning them, and conducting fine-tuning experiments.<\/jats:p>","DOI":"10.3390\/computers15020073","type":"journal-article","created":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T17:40:22Z","timestamp":1769622022000},"page":"73","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Integrated Approach to Adapting Open-Source AI Models for Machine Translation of Low-Resource Turkic Languages"],"prefix":"10.3390","volume":"15","author":[{"given":"Ualsher","family":"Tukeyev","sequence":"first","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1637-4643","authenticated-orcid":false,"given":"Assem","family":"Shormakova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"given":"Aidana","family":"Karibayeva","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"given":"Diana","family":"Rakhimova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"given":"Balzhan","family":"Abduali","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"given":"Dina","family":"Amirova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7188-4838","authenticated-orcid":false,"given":"Nazym","family":"Rakhmanberdi","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]},{"given":"Rashid","family":"Aliyev","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Artificial Intelligence, Farabi University, Almaty 050040, Kazakhstan"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,28]]},"reference":[{"key":"ref_1","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7\u20139). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA. Available online: https:\/\/arxiv.org\/abs\/1409.0473."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Mirzakhalov, J., Babu, A., Ataman, D., Kariev, S., Tyers, F., Abduraufov, O., Hajili, M., Ivanova, S., Khaytbaev, A., and Laverghetta, A. (2021, January 7\u201311). A Large-Scale Study of Machine Translation in Turkic Languages. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.475"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4","DOI":"10.47611\/jsrhs.v13i4.8405","article-title":"The Quality of Translation of Turkic Languages by AI Translation Tools","volume":"13","author":"Jumashukurov","year":"2024","journal-title":"J. Stud. Res."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mamasaidov, M., and Shopulatov, A. (2024, January 12\u201313). Open Language Data Initiative: Advancing Low-Resource Machine Translation for Kara-kalpak. Proceedings of the Ninth Conference on Machine Translation (WMT), Miami, FL, USA.","DOI":"10.18653\/v1\/2024.wmt-1.48"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kadyrbek, N., Tuimebayev, Z., Mansurova, M., and Viegas, V. (2025). The Development of Small-Scale Language Models for Low-Resource Languages, with a Focus on Kazakh and Direct Preference Optimization. Big Data Cogn. Comput., 9.","DOI":"10.3390\/bdcc9050137"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"13835","DOI":"10.1038\/s41598-024-64848-1","article-title":"Multilingual End-to-End ASR for Low-Resource Turkic Languages with Common Alphabets","volume":"14","author":"Bekarystankyzy","year":"2024","journal-title":"Sci. Rep."},{"key":"ref_7","unstructured":"Veitsman, Y., and Hartmann, M. (2025, January 7). Recent Advancements and Challenges of Turkic Central Asian Language Processing. Proceedings of the First Workshop on Language Models for Low-Resource Languages, Abu Dhabi, United Arab Emirates. Available online: https:\/\/aclanthology.org\/2025.loreslm-1.25\/."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Guzm\u00e1n, F., Chen, P., Ott, M., Du, J., Lewis, M., and Neubig, G. (2019, January 3\u20137). The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali\u2013English and Sinhala\u2013English. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1632"},{"key":"ref_9","unstructured":"Tiedemann, J., and Thottingal, S. (2020, January 3\u20135). OPUS-MT\u2014Building open translation services for the World. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal."},{"key":"ref_10","unstructured":"Tiedemann, J. (2020, January 19\u201320). The Tatoeba Translation Challenge: Realistic Data Sets for Low-Resource and Multilingual MT. Proceedings of the Fifth Conference on Machine Translation (WMT), Online. Available online: https:\/\/aclanthology.org\/2020.wmt-1.139\/."},{"key":"ref_11","unstructured":"Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google\u2019s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv."},{"key":"ref_12","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA. Available online: https:\/\/arxiv.org\/abs\/1706.03762."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1613\/jair.1.12007","article-title":"Neural Machine Translation: A Review","volume":"69","author":"Stahlberg","year":"2020","journal-title":"J. Artif. Intell. Res."},{"key":"ref_14","unstructured":"Yang, S., Wang, Y., and Chu, X. (2020). A Survey of Deep Learning Techniques for Neural Machine Translation. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2016, January 7\u201312). Improving Neural Machine Translation Models with Monolingual Data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), Berlin, Germany.","DOI":"10.18653\/v1\/P16-1009"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Edunov, S., Ott, M., Auli, M., and Grangier, D. (November, January 31). Understanding Back-Translation at Scale. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1045"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhang, J., and Zong, C. (2020). Neural Machine Translation: Challenges, Progress and Future. arXiv.","DOI":"10.1007\/s11431-020-1632-x"},{"key":"ref_18","unstructured":"J\u0131lta, E. (2024, January 2\u20136). Low-Resource Language Processing: Turkish Language Study. Proceedings of the 6th International Conference on Data Science and Applications (ICONDATA\u201924), Pristina, Kosovo. Available online: https:\/\/www.researchgate.net\/publication\/390349163."},{"key":"ref_19","unstructured":"Mirzakhalov, J., Babu, A., Kunafin, A., Wahab, A., Moydinboyev, B., Ivanova, S., Uzokova, M., Pulatova, S., Ataman, D., and Kreutzer, J. (2021). Evaluating Multiway Multilingual NMT in Turkic Languages. arXiv."},{"key":"ref_20","unstructured":"Yeshpanov, R., Polonskaya, A., and Varol, H.A. (2024, January 20\u201325). KazParC: Kazakh Parallel Corpus for Machine Translation. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy. Available online: https:\/\/aclanthology.org\/2024.lrec-main.842\/."},{"key":"ref_21","unstructured":"Di Gangi, M.A., Cattoni, R., Bentivogli, L., Negri, M., and Turchi, M. (2019, January 2\u20137). MuST-C: A Multilingual Speech Translation Corpus. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Minneapolis, MN, USA."},{"key":"ref_22","unstructured":"NLLB Team, Costa-Juss\u00e0, M.R., Cross, J., \u00c7elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., and Licht, D. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv."},{"key":"ref_23","unstructured":"Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., and Chaudhary, V. (2020). Beyond English-Centric Multilingual Machine Translation. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e1224","DOI":"10.7717\/peerj-cs.1224","article-title":"The Neural Machine Translation Models for the Low-Resource Kazakh\u2013English Language Pair","volume":"9","author":"Karyukin","year":"2023","journal-title":"PeerJ Comput. Sci."},{"key":"ref_25","unstructured":"Chen, C.-C., and Chen, W. (2022). Benchmarking Neural Machine Translation for Azerbaijani. arXiv."},{"key":"ref_26","unstructured":"Tiedemann, J. (2026, January 15). OPUS: The Open Parallel Corpus. University of Helsinki. Available online: https:\/\/opus.nlpl.eu\/."},{"key":"ref_27","unstructured":"Nurmaganbet, D., Tukeyev, U., Shormakova, A., and Zhumanov, Z. (2024). Comparative Analysis of Speech Translation Models for Turkic Languages. Intelligent Information and Database Systems (Lecture Notes in Computer Science), Springer. Available online: https:\/\/link.springer.com\/chapter\/10.1007\/978-981-97-4985-0_28."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Adal\u0131, E., Oflazer, K., and Tantu\u011f, A.C. (2007, January 25\u201330). A MT System from Turkmen to Turkish Employing Finite State and Statistical Methods. Proceedings of the MT Summit XI, Copenhagen, Denmark. Available online: https:\/\/aclanthology.org\/2007.mtsummit-papers.61.pdf.","DOI":"10.3115\/1626355.1626359"},{"key":"ref_29","unstructured":"Qamet, B., Zhakypbayeva, A., Turganbayeva, A., and Tukeyev, U. (2022, January 16\u201318). Development of Kazakh\u2013Turkish MT on the Base of CSE Model. Proceedings of the 2022 Asian Conference on Intelligent Information and Database Systems (ACIIDS 2022), Ho Chi Minh City, Vietnam. Available online: https:\/\/link.springer.com\/chapter\/10.1007\/978-981-19-8234-7_42."},{"key":"ref_30","unstructured":"Kuriyozov, E., Doval, Y., and G\u00f3mez-Rodr\u00edguez, C. (2020). Cross-Lingual Word Embeddings for Turkic Languages. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"32341","DOI":"10.1109\/ACCESS.2025.3542491","article-title":"Improving Low-Resource Kazakh-English and Turkish-English NMT Using Transfer Learning and POS Tags","volume":"13","author":"Yazar","year":"2025","journal-title":"IEEE Access"},{"key":"ref_32","unstructured":"Balabekova, T., Kairatuly, B., and Tukeyev, U. (2024, January 10\u201312). Kazakh\u2013Uzbek Speech Cascade Machine Translation on Complete Set of Endings. Proceedings of the 2024 International Conference on Language and Speech Technology (ICLST 2024), Almaty, Kazakhstan. Available online: https:\/\/link.springer.com\/chapter\/10.1007\/978-3-031-41774-0_34."},{"key":"ref_33","unstructured":"Tiedemann, J. (2012, January 23\u201325). Parallel Data, Tools and Interfaces in OPUS. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), Istanbul, Turkey. Available online: http:\/\/www.lrec-conf.org\/proceedings\/lrec2012\/pdf\/463_Paper.pdf."},{"key":"ref_34","first-page":"110011","article-title":"Parallel Texts Dataset for Uzbek\u2013Kazakh Machine Translation","volume":"54","author":"Allaberdiev","year":"2024","journal-title":"Data Brief"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Li, Z., and Specia, L. (2019, January 1\u20132). Improving Neural Machine Translation Robustness via Data Augmentation. Proceedings of the Fourth Conference on Machine Translation (WMT 2019), Florence, Italy.","DOI":"10.18653\/v1\/D19-5543"},{"key":"ref_36","unstructured":"Fadaee, M., Bisazza, A., and Monz, C. (August, January 30). Data Augmentation for Low-Resource NMT. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Vancouver, BC, Canada."},{"key":"ref_37","unstructured":"Wikipedia (2025, December 04). Word Error Rate. Available online: https:\/\/en.wikipedia.org\/wiki\/Word_error_rate."},{"key":"ref_38","unstructured":"Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006, January 8\u201312). A Study of Translation Edit Rate with Targeted Human Annotation. Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 2006), Cambridge, MA, USA. Available online: https:\/\/aclanthology.org\/2006.amta-papers.25.pdf."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Popovi\u0107, M. (2015, January 17\u201318). chrF: Character n-Gram F-Score for Automatic MT Evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-3049"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1162\/coli.2008.07-017-R1-06-83","article-title":"Dependency Parsing of Turkish","volume":"34","author":"Nivre","year":"2008","journal-title":"Comput. Linguist."},{"key":"ref_42","unstructured":"(2025, December 04). Google Translate. Available online: https:\/\/translate.google.com\/."},{"key":"ref_43","unstructured":"(2025, December 04). GPT-4o. Available online: https:\/\/openai.com\/index\/hello-gpt-4o\/?utm_source=chatgpt.com."},{"key":"ref_44","unstructured":"(2025, December 04). Microsoft Copilot. Available online: https:\/\/copilot.microsoft.com\/."},{"key":"ref_45","unstructured":"Meta AI (2025, December 04). No Language Left Behind: High-Quality Machine Translation for 200 Languages. Available online: https:\/\/ai.meta.com\/blog\/nllb-200-high-quality-machine-translation."},{"key":"ref_46","unstructured":"Google (2026, January 15). Introducing Gemma 2. Available online: https:\/\/blog.google\/technology\/developers\/google-gemma-2\/."},{"key":"ref_47","unstructured":"Microsoft (2025, December 04). Introducing Phi-4: Microsoft\u2019s Newest Small Language Model. Available online: https:\/\/techcommunity.microsoft.com\/blog\/azure-ai-foundry-blog\/introducing-phi-4-microsoft%E2%80%99s-newest-small-language-model-specializing-in-comple\/4357090."},{"key":"ref_48","unstructured":"Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., and Wei, H. (2024). Qwen2.5 Technical Report. arXiv."},{"key":"ref_49","unstructured":"Meta AI (2026, January 15). Llama-3.2: Vision on Edge and Mobile Devices. Available online: https:\/\/ai.meta.com\/blog\/llama-3-2-connect-2024-vision-edge-mobile-devices\/."},{"key":"ref_50","unstructured":"Meta AI (2025, December 04). Introducing Meta Llama 3. Available online: https:\/\/ai.meta.com\/blog\/meta-llama-3\/."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021, January 7\u201311). mT5: A Massively Multilingual Pre-Trained Text-to-Text Transformer. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.naacl-main.41"},{"key":"ref_52","unstructured":"Wikipedia (2025, December 03). T5 (Text-to-Text Transfer Transformer). Available online: https:\/\/en.wikipedia.org\/wiki\/T5_%28language_model%29."},{"key":"ref_53","unstructured":"(2025, December 03). T5-Base Model Card. Available online: https:\/\/huggingface.co\/google-t5\/t5-base."},{"key":"ref_54","unstructured":"(2025, December 03). T5-Large Model Card. Available online: https:\/\/huggingface.co\/google-t5\/t5-large."},{"key":"ref_55","unstructured":"(2025, December 03). T5-3B Model Card. Available online: https:\/\/huggingface.co\/google-t5\/t5-3b."},{"key":"ref_56","unstructured":"(2025, December 03). T5-11B Model Card. Available online: https:\/\/huggingface.co\/google-t5\/t5-11b."},{"key":"ref_57","unstructured":"Poncelas, A., Shterionov, D., Way, A., Maillette de Buy Wenniger, G., and Passban, P. (2018, January 28\u201330). Investigating Backtranslation in Neural Machine Translation. Proceedings of the 21st Annual Conference of the European Association for Machine Translation (EAMT 2018), Alacant, Spain. Available online: https:\/\/aclanthology.org\/2018.eamt-main.25\/."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2016, January 7\u201312). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), Berlin, Germany.","DOI":"10.18653\/v1\/P16-1162"},{"key":"ref_59","first-page":"449","article-title":"Resources for Turkish Natural Language Processing: A Critical Survey","volume":"57","year":"2022","journal-title":"Lang. Resour. Eval."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1049\/cje.2016.03.020","article-title":"Agglutinative Language Speech Recognition Using Automatic Allophone Deriving","volume":"25","author":"Xu","year":"2016","journal-title":"Chin. J. Electron."},{"key":"ref_61","unstructured":"Mathur, N., Wei, J., Freitag, M., Ma, Q., and Bojar, O. (2020, January 19\u201320). Results of the WMT20 Metrics Shared Task. Proceedings of the Fifth Conference on Machine Translation (WMT 2020), Online. Available online: https:\/\/aclanthology.org\/2020.wmt-1.77\/."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Sellam, T., Das, D., and Parikh, A. (2020, January 5\u201310). BLEURT: Learning Robust Metrics for Text Generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), Online.","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Rei, R., Stewart, C., Farinha, A.C., and Lavie, A. (2020, January 16\u201320). COMET: A Neural Framework for MT Evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.213"},{"key":"ref_64","unstructured":"Freitag, M., Rei, R., Mathur, N., Lo, C., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., and Martins, A.F.T. (2022, January 7\u20138). Results of WMT22 Metrics Shared Task: Stop Using BLEU\u2014Neural Metrics Are Better and More Robust. Proceedings of the Seventh Conference on Machine Translation (WMT 2022), Abu Dhabi, United Arab Emirates."},{"key":"ref_65","unstructured":"Rei, R., Treviso, M., Guerreiro, N.M., Zerva, C., Farinha, A.C., Maroti, C., de Souza, J.G.C., Glushkova, T., Alves, D., and Coheur, L. (2022, January 7\u20138). CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. Proceedings of the Seventh Conference on Machine Translation (WMT 2022), Abu Dhabi, United Arab Emirates."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"1460","DOI":"10.1162\/tacl_a_00437","article-title":"Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation","volume":"9","author":"Freitag","year":"2021","journal-title":"Trans. Assoc. Comput. Linguist."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/2\/73\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T05:21:24Z","timestamp":1769750484000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/2\/73"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,28]]},"references-count":66,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,2]]}},"alternative-id":["computers15020073"],"URL":"https:\/\/doi.org\/10.3390\/computers15020073","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,28]]}}}