{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T10:49:57Z","timestamp":1767869397580,"version":"3.49.0"},"reference-count":46,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T00:00:00Z","timestamp":1767657600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This paper presents the results of a study in which we conducted a fine-grained error analysis for intralingual machine translations into Plain Language. As there are no established error schemes for intralingual translations, we adapted the MQM scheme to fit the purposes of intralingual translation and expanded the scheme by error categories that are only relevant to intralingual translation. Our study has revealed that substantial differences exist between general-purpose and domain-specific models, with fine-tuned systems achieving notably higher accuracy and fewer severe errors across most categories. We found that across all four models, most errors occurred in the \u201cAccuracy\u201d category, closely followed by errors in the \u201cLinguistic conventions\u201d category and that all evaluated models produced persistent issues, particularly in terms of accuracy, linguistic conventions, and alignment with the target audience. In addition, we identified subcategories from the MQM scheme that are primarily relevant to interlingual translation, such as \u201cTextual conventions\u201d. Furthermore, we found that manual error annotation is resource-intensive and subjective, highlighting the urgent need for the development of automatic or semi-automatic error annotation tools. We also discuss difficulties that arose in the annotation process and show how methodological limitations might be overcome in future studies. Our findings provide practical directions for improving both machine translation technology and quality assurance frameworks for intralingual translation into Plain Language.<\/jats:p>","DOI":"10.3390\/info17010053","type":"journal-article","created":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T15:09:36Z","timestamp":1767712176000},"page":"53","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating Intralingual Machine Translation Quality: Application of an Adapted MQM Scheme to German Plain Language"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2933-9257","authenticated-orcid":false,"given":"Silvana","family":"Deilen","sequence":"first","affiliation":[{"name":"Institute for Translation Studies and Interpreting, Heidelberg University, 69117 Heidelberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4879-3435","authenticated-orcid":false,"given":"Sergio","family":"Hern\u00e1ndez Garrido","sequence":"additional","affiliation":[{"name":"Institute for Translation Studies and Specialised Communication, University of Hildesheim, 31141 Hildesheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5618-8087","authenticated-orcid":false,"given":"Ekaterina","family":"Lapshinova-Koltunski","sequence":"additional","affiliation":[{"name":"Institute for Translation Studies and Specialised Communication, University of Hildesheim, 31141 Hildesheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3126-5912","authenticated-orcid":false,"given":"Chris","family":"Maa\u00df","sequence":"additional","affiliation":[{"name":"Institute for Translation Studies and Specialised Communication, University of Hildesheim, 31141 Hildesheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0764-1657","authenticated-orcid":false,"given":"Annie","family":"Werner","sequence":"additional","affiliation":[{"name":"Institute for Translation Studies and Specialised Communication, University of Hildesheim, 31141 Hildesheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s12599-023-00795-x","article-title":"Welcome to the era of chatgpt et al. the prospects of large language models","volume":"65","author":"Teubner","year":"2023","journal-title":"Bus. Inf. Syst. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1017\/S1351324923000554","article-title":"A year\u2019s a long time in generative AI","volume":"30","author":"Dale","year":"2024","journal-title":"Nat. Lang. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6\u201312). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_4","unstructured":"Banerjee, S., and Lavie, A. (2005, January 29\u201330). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Ann Arbor, MI, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Rei, R., Stewart, C., Farinha, A.C., and Lavie, A. (2020). COMET: A neural framework for MT evaluation. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.213"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1037\/h0051965","article-title":"\u201cSimplification of Flesch Reading Ease Formula\u201d: Reply","volume":"36","author":"Flesch","year":"1952","journal-title":"J. Appl. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1162\/tacl_a_00107","article-title":"Optimizing statistical machine translation for text simplification","volume":"4","author":"Xu","year":"2016","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Castilho, S., Doherty, S., Gaspari, F., and Moorkens, J. (2018). Approaches to human and machine translation quality assessment. Translation Quality Assessment: From Principles to Practice, Springer.","DOI":"10.1007\/978-3-319-91241-7_2"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"455","DOI":"10.5565\/rev\/tradumatica.77","article-title":"Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics","volume":"12","author":"Lommel","year":"2014","journal-title":"Tradum\u00e0tica"},{"key":"ref_10","first-page":"21","article-title":"Accuracy, Readability, and Acceptability in Translation","volume":"16","author":"McDonald","year":"2022","journal-title":"Appl. Transl."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Baumgart, M., H\u00f6sel, C., Breck, D., Schuster, M., Roschke, C., and Ritter, M. (2021, January 24\u201329). Development of a holistic web-based interface assistance system to support the intralingual translation process. Proceedings of the International Conference on Human\u2013Computer Interaction, Virtual.","DOI":"10.1007\/978-3-030-78635-9_65"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1007\/s10209-023-00975-2","article-title":"Empirical evaluation of Easy Language recommendations: A systematic literature review from journal research in Catalan, English, and Spanish","volume":"23","author":"Matamala","year":"2024","journal-title":"Univers. Access Inf. Soc."},{"key":"ref_13","unstructured":"Luque Lop\u00e9z, L. (2025). Leveraging Large Language Models to Translate into Easy Language: An Exploratory Study on University Websites. [Master\u2019s Thesis, Universit\u00e9 de Gen\u00e8ve]."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1007\/s10676-024-09792-4","article-title":"Easy-read and large language models: On the ethical dimensions of LLM-based text simplification","volume":"26","author":"Freyer","year":"2024","journal-title":"Ethics Inf. Technol."},{"key":"ref_15","unstructured":"Jakobson, R. (1959). On linguistic aspects of translation. The Translation Studies Reader, Routledge."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Maa\u00df, C. (2020). Easy Language\u2013Plain Language\u2013Easy Language Plus: Balancing Comprehensibility and Acceptability, Frank & Timme.","DOI":"10.26530\/20.500.12657\/42089"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Maa\u00df, C., and Hern\u00e1ndez Garrido, S. (2025). Einfache Sprache: Einfach, leicht, verst\u00e4ndlich?. Einfache Sprache mit KI-Tools: Ein Leitfaden f\u00fcr die Redaktionelle Praxis, Springer.","DOI":"10.1007\/978-3-658-47867-4"},{"key":"ref_18","unstructured":"(2024). Einfache Sprache\u2014Teil 1: Grunds\u00e4tze und Leitlinien (Standard No. DIN ISO 24495-1:2024-03)."},{"key":"ref_19","unstructured":"(2024). Einfache Sprache\u2014Anwendung f\u00fcr das Deutsche\u2014Teil 1: Sprachspezifische Festlegungen (Standard No. DIN 8581-1)."},{"key":"ref_20","unstructured":"(2023). Plain Language\u2014Part 1: Governing Principles and Guidelines (Standard No. ISO 24495-1:2023)."},{"key":"ref_21","unstructured":"Deilen, S., Lapshinova-Koltunski, E., Garrido, S., H\u00f6rner, J., Maa\u00df, C., Theel, V., and Ziemer, S. (2024, January 24\u201327). Evaluation of intralingual machine translation for health communication. Proceedings of the 25th Annual Conference of the European Association for Machine Translation, Sheffield, UK."},{"key":"ref_22","unstructured":"Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023). LENS: A Learnable Evaluation Metric for Text Simplification. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada, 9\u201314 July 2023, Association for Computational Linguistics."},{"key":"ref_23","first-page":"37","article-title":"A formula for predicting readability: Instructions","volume":"27","author":"Dale","year":"1948","journal-title":"Educ. Res. Bull."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1086\/458513","article-title":"A new readability formula for primary-grade reading materials","volume":"53","author":"Spache","year":"1953","journal-title":"Elem. Sch. J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"283\u2013284","DOI":"10.1037\/h0076540","article-title":"A computer readability formula designed for machine scoring","volume":"60","author":"Coleman","year":"1975","journal-title":"J. Appl. Psychol."},{"key":"ref_26","first-page":"179","article-title":"Readability of English written materials","volume":"1","author":"Isnaeni","year":"2017","journal-title":"Elite Engl. Lit. J."},{"key":"ref_27","unstructured":"Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (2025). Evaluating the Evaluators: Are readability metrics good measures of readability?. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4\u20139 November 2025, Association for Computational Linguistics."},{"key":"ref_28","first-page":"84","article-title":"Artificial Intelligence and Natural Language Processing for Easy-to-Read Texts","volume":"82","author":"Saggion","year":"2024","journal-title":"J. Lang. Law"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Guo, Y., August, T., Leroy, G., Cohen, T.A., and Wang, L.L. (2024, January 12\u201316). APPLS: Evaluating Evaluation Metrics for Plain Language Summarization. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.","DOI":"10.18653\/v1\/2024.emnlp-main.519"},{"key":"ref_30","unstructured":"Gao, M., Ruan, J., Sun, R., Yin, X., Yang, S., and Wan, X. (2023). Human-like Summarization Evaluation with ChatGPT. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Stodden, R., Momen, O., and Kallmeyer, L. (2023). DEplain: A German parallel corpus with intralingual translations into plain language for sentence and document simplification. arXiv.","DOI":"10.18653\/v1\/2023.acl-long.908"},{"key":"ref_32","unstructured":"Grabar, N., and Saggion, H. (July, January 27). Evaluation of automatic text simplification: Where are we now, where should we go from here. Proceedings of the Traitement Automatique des Langues Naturelles, ATALA, Avignon, France."},{"key":"ref_33","unstructured":"Patil, U., Calvillo, J., Lago, S., and Schumann, A.K. (2025, January 23). Quantifying word complexity for Leichte Sprache: A computational metric and its psycholinguistic validation. Proceedings of the 1st Workshop on Artificial Intelligence and Easy and Plain Language in Institutional Contexts (AI & EL\/PL), Geneva, Switzerland."},{"key":"ref_34","unstructured":"Bredel, U., and Maa\u00df, C. (2016). Leichte Sprache: Theoretische Grundlagen? Orientierung f\u00fcr die Praxis, Duden."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ansch\u00fctz, M., Oehms, J., Wimmer, T., Jezierski, B., and Groh, G. (2023). Language models for German text simplification: Overcoming parallel data scarcity through style-specific pre-training. arXiv.","DOI":"10.18653\/v1\/2023.findings-acl.74"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Elmakias, I., and Vilenchik, D. (2021). An oblivious approach to machine translation quality estimation. Mathematics, 9.","DOI":"10.3390\/math9172090"},{"key":"ref_37","unstructured":"Deilen, S., Garrido, S.H., Lapshinova-Koltunski, E., and Maa\u00df, C. (2023). Using ChatGPT as a CAT tool in Easy Language translation. arXiv."},{"key":"#cr-split#-ref_38.1","unstructured":"Wartena, C., and Heid, U. (2025). Evaluation of Machine Translation Errors in German Plain Language Texts in the Domain of Health Information. Proceedings of the 21st Conference on Natural Language"},{"key":"#cr-split#-ref_38.2","unstructured":"Processing (KONVENS 2025): Workshops, Hildesheim, Germany, 9-12 September 2025, HsH Applied Academics."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"38","DOI":"10.33542\/JTI2025-S-3","article-title":"Evaluation of translations into plain german produced\nby humans and mt systems including chatgpt","volume":"18","author":"Ahrens","year":"2025","journal-title":"SKASE J. Transl. Interpret."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Hansen-Schirra, S., Nitzke, J., and Gutermuth, S. (2021). Language (Geasy Corpus): What Sentence Alignments Can Tell Us About Translation Strategies in Intralingual. New Perspectives on Corpus Translation Studies, Springer.","DOI":"10.1007\/978-981-16-4918-9_11"},{"key":"ref_41","unstructured":"Deilen, S., Lapshinova-Koltunski, E., Garrido, S.H., Maa\u00df, C., H\u00f6rner, J., Theel, V., and Ziemer, S. (2024, January 20). Towards ai-supported health communication in plain language: Evaluating intralingual machine translation of medical texts. Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health) at LREC-COLING 2024, Torino, Italy."},{"key":"ref_42","unstructured":"Kuckartz, U. (2018). Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterst\u00fctzung, Beltz Juventa."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lu, Q., Qiu, B., Ding, L., Zhang, K., Kocmi, T., and Tao, D. (2023). Error analysis prompting enables human-like translation evaluation in large language models. arXiv.","DOI":"10.20944\/preprints202303.0255.v1"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A.F., Neubig, G., Garg, A., Clark, J.H., Freitag, M., and Firat, O. (2023). The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. arXiv.","DOI":"10.18653\/v1\/2023.wmt-1.100"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kocmi, T., and Federmann, C. (2023). GEMBA-MQM: Detecting translation quality error spans with GPT-4. arXiv.","DOI":"10.18653\/v1\/2023.wmt-1.64"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/1\/53\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T05:22:56Z","timestamp":1767849776000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/1\/53"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,6]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["info17010053"],"URL":"https:\/\/doi.org\/10.3390\/info17010053","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,6]]}}}