{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T16:19:42Z","timestamp":1790093982323,"version":"4.0.1"},"reference-count":32,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T00:00:00Z","timestamp":1769040000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Higher Education, Science and Technology","award":["001\/LL17\/DT.05.00\/PL\/2025"],"award-info":[{"award-number":["001\/LL17\/DT.05.00\/PL\/2025"]}]},{"name":"Ministry of Higher Education, Science and Technology","award":["29\/DPPM-UIR\/HN-P\/2025"],"award-info":[{"award-number":["29\/DPPM-UIR\/HN-P\/2025"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Digital"],"abstract":"<jats:p>This paper investigates the use of large language models (LLMs) as evaluators in multidimensional machine translation (MT) assessment, focusing on the English\u2013Indonesian language pair. Building on established evaluation frameworks, we adopt an MQM-aligned rubric that assesses translation quality along morphosyntactic, semantic, and pragmatic dimensions. Three LLM-based translation systems (Qwen 3 (0.6B), LLaMA 3.2 (3B), and Gemma 3 (1B)) are evaluated using both expert human judgments and an LLM-based evaluator (GPT\u20135), allowing for a detailed comparison of alignment, bias, and consistency between human and AI-based assessments. In addition, a classroom calibration study is conducted to examine how rubric-guided evaluation supports alignment among novice evaluators. The results indicate that GPT\u20135 exhibits strong agreement with human evaluators in terms of relative quality ranking, while systematic differences in absolute scoring highlight calibration challenges. Overall, this study provides insights into the role of LLMs as reference-free evaluators for MT and illustrates how multidimensional rubrics can support both research-oriented evaluation and pedagogical applications in a mid-resource language setting.<\/jats:p>","DOI":"10.3390\/digital6010008","type":"journal-article","created":{"date-parts":[[2026,1,26]],"date-time":"2026-01-26T16:13:48Z","timestamp":1769444028000},"page":"8","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Beyond BLEU: GPT\u20135, Human Judgment, and Classroom Validation for Multidimensional Machine Translation Evaluation"],"prefix":"10.3390","volume":"6","author":[{"given":"Shalawati","family":"Shalawati","sequence":"first","affiliation":[{"name":"Faculty of Education, Universitas Islam Riau, Pekanbaru 28284, Indonesia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6283-3217","authenticated-orcid":false,"given":"Arbi Haza","family":"Nasution","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, Universitas Islam Riau, Pekanbaru 28284, Indonesia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4637-5735","authenticated-orcid":false,"given":"Winda","family":"Monika","sequence":"additional","affiliation":[{"name":"Department of Library Science, Universitas Lancang Kuning, Pekanbaru 28266, Indonesia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tatum","family":"Derin","sequence":"additional","affiliation":[{"name":"Faculty of Education and Vocation, Universitas Lancang Kuning, Pekanbaru 28266, Indonesia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9434-5880","authenticated-orcid":false,"given":"Aytug","family":"Onan","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Faculty of Engineering, Izmir Institute of Technology, Izmir 35430, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8310-2007","authenticated-orcid":false,"given":"Yohei","family":"Murakami","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ataman, D., Birch, A., Habash, N., Federico, M., Koehn, P., and Cho, K. (2025). Machine Translation in the Era of Large Language Models: A Survey of Historical and Emerging Problems. Information, 16.","DOI":"10.3390\/info16090723"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., Chen, J., and Li, L. (2024, January 16\u201321). Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico.","DOI":"10.18653\/v1\/2024.findings-naacl.176"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kong, M., Fernandez, A., Bains, J., Milisavljevic, A., Brooks, K.C., Shanmugam, A., Avilez, L., Li, J., Honcharov, V., and Yang, A. (2025). Evaluation of the accuracy and safety of machine translation of patient-specific discharge instructions: A comparative analysis. BMJ Qual. Saf.","DOI":"10.1136\/bmjqs-2024-018384"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.21037\/jmai-24-24","article-title":"A Comparative assessment of ChatGPT vs. Google Translate for the translation of patient instructions","volume":"7","author":"Rao","year":"2024","journal-title":"J. Med Artif. Intell."},{"key":"ref_5","first-page":"2472916","article-title":"ChatGPT translation vs. human translation: An examination of a literary text","volume":"11","author":"Rousan","year":"2025","journal-title":"Cogent Soc. Sci."},{"key":"ref_6","unstructured":"Callison-Burch, C., Osborne, M., and Koehn, P. (2006, January 3\u20137). Re-evaluating the Role of Bleu in Machine Translation Research. Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mathur, N., Baldwin, T., and Cohn, T. (2020, January 5\u201310). Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.448"},{"key":"ref_8","unstructured":"Park, D., and Pad\u00f3, S. (2024, January 20\u201325). Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean. Proceedings of the Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1460","DOI":"10.1162\/tacl_a_00437","article-title":"Experts, errors, and context: A large-scale study of human evaluation for machine translation","volume":"9","author":"Freitag","year":"2021","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Freitag, M., Mathur, N., Deutsch, D., Lo, C.K., Avramidis, E., Rei, R., Thompson, B., Blain, F., Kocmi, T., and Wang, J. (2024, January 15\u201316). Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task. Proceedings of the Ninth Conference on Machine Translation, Miami, FL, USA.","DOI":"10.18653\/v1\/2024.wmt-1.2"},{"key":"ref_11","unstructured":"Kocmi, T., and Federmann, C. (2023, January 12\u201315). Large Language Models Are State-of-the-Art Evaluators of Translation Quality. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, Tampere, Finland."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023, January 6\u201310). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"71876","DOI":"10.1109\/ACCESS.2024.3402809","article-title":"ChatGPT Label: Comparing the Quality of Human-Generated and LLM-Generated Annotations in Low-Resource Language NLP Tasks","volume":"12","author":"Nasution","year":"2024","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"94009","DOI":"10.1109\/ACCESS.2025.3574629","article-title":"Benchmarking Open-Source Large Language Models for Sentiment and Emotion Classification in Indonesian Tweets","volume":"13","author":"Nasution","year":"2025","journal-title":"IEEE Access"},{"key":"ref_15","unstructured":"NLLB Team, Costa-juss\u00e0, M.R., Cross, J., \u00c7elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., and Licht, D. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv."},{"key":"ref_16","first-page":"1","article-title":"Beyond English-Centric Multilingual Machine Translation","volume":"22","author":"Fan","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kim, Y., Petrov, P., Petrushkov, P., Khadivi, S., and Ney, H. (2019, January 3\u20137). Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1080"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1162\/tacl_a_00642","article-title":"Exploring Human-Like Translation Strategy with Large Language Models","volume":"12","author":"He","year":"2024","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_19","unstructured":"Freitag, M., Rei, R., Mathur, N., Lo, C.K., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., and Martins, A.F. (2022, January 7\u20138). Results of the WMT22 Metrics Shared Task: Stop Using BLEU. Proceedings of the Seventh Conference on Machine Translation (WMT), Abu Dhabi, United Arab Emirates."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jiang, Y.E., Liu, T., Ma, S., Zhang, D., Yang, J., Huang, H., Sennrich, R., Sachan, M., Cotterell, R., and Zhou, M. (2022, January 10\u201315). BLONDE: An Automatic Evaluation Metric for Document-level Machine Translation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.","DOI":"10.18653\/v1\/2022.naacl-main.111"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., and Grundkiewicz, R. (2023, January 6\u20137). Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here But Not Quite There Yet. Proceedings of the Eighth Conference on Machine Translation, Singapore.","DOI":"10.18653\/v1\/2023.wmt-1.1"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kocmi, T., Zouhar, V., Avramidis, E., Grundkiewicz, R., Karpinska, M., Popovic, M., Sachan, M., and Shmatova, M. (2024, January 15\u201316). Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation. Proceedings of the Ninth Conference on Machine Translation, Miami, FL, USA.","DOI":"10.18653\/v1\/2024.wmt-1.131"},{"key":"ref_23","unstructured":"Kim, A. (August, January 27). RUBRIC-MQM: Span-Level LLM-as-judge in Machine Translation For High-End Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria."},{"key":"ref_24","unstructured":"Li, R., Patel, T., and Du, X. (2024). PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations. Trans. Mach. Learn. Res., 1\u201337. Available online: https:\/\/openreview.net\/forum?id=YVD1QqWRaj."},{"key":"ref_25","unstructured":"Tan, W., and Zhu, K. (2024). NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cahyawijaya, S., Winata, G.I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z.Y., Bahar, S., and Khodra, M. (2021, January 7\u201311). IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.699"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sun, J., Luo, Y., Gong, Y., Lin, C., Shen, Y., Guo, J., and Duan, N. (2024). Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models. Findings of the Association for Computational Linguistics: NAACL 2024, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.findings-naacl.257"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/21582440251369204","article-title":"Exploring the Efficacy of ChatGPT-Based Feedback Compared with Teacher Feedback and Self-Feedback: Evidence From Chinese-English Translation","volume":"15","author":"Cao","year":"2025","journal-title":"SAGE Open"},{"key":"ref_29","unstructured":"Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., and Lv, C. (2025). Qwen3 Technical Report. arXiv."},{"key":"ref_30","unstructured":"Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The Llama 3 Herd of Models. arXiv."},{"key":"ref_31","unstructured":"Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram\u00e9, A., and Rivi\u00e8re, M. (2025). Gemma 3 Technical Report. arXiv."},{"key":"ref_32","first-page":"1220","article-title":"Towards Two-Step Fine-Tuned Abstractive Summarization for Low-Resource Language Using Transformer T5","volume":"16","author":"Nasution","year":"2025","journal-title":"Int. J. Adv. Comput. Sci. Appl."}],"container-title":["Digital"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2673-6470\/6\/1\/8\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T05:16:33Z","timestamp":1769490993000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2673-6470\/6\/1\/8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,22]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["digital6010008"],"URL":"https:\/\/doi.org\/10.3390\/digital6010008","relation":{},"ISSN":["2673-6470"],"issn-type":[{"value":"2673-6470","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,22]]}}}