{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T15:38:18Z","timestamp":1776872298435,"version":"3.51.2"},"reference-count":58,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2019,9,17]],"date-time":"2019-09-17T00:00:00Z","timestamp":1568678400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>This study aims to analyse how translation experts from the German department of the European Commission\u2019s Directorate-General for Translation (DGT) identify and correct different error categories in neural machine translated texts (NMT) and their post-edited versions (NMTPE). The term translation expert encompasses translator, post-editor as well as revisor. Even though we focus on neural machine-translated segments, translator and post-editor are used synonymously because of the combined workflow using CAT-Tools as well as machine translation. Only the distinction between post-editor, which refers to a DGT translation expert correcting the neural machine translation output, and revisor, which refers to a DGT translation expert correcting the post-edited version of the neural machine translation output, is important and made clear whenever relevant. Using an automatic error annotation tool and the more fine-grained manual error annotation framework to identify characteristic error categories in the DGT texts, a corpus analysis revealed that quality assurance measures by post-editors and revisors of the DGT are most often necessary for lexical errors. More specifically, the corpus analysis showed that, if post-editors correct mistranslations, terminology or stylistic errors in an NMT sentence, revisors are likely to correct the same error type in the same post-edited sentence, suggesting that the DGT experts were being primed by the NMT output. Subsequently, we designed a controlled eye-tracking and key-logging experiment to compare participants\u2019 eye movements for test sentences containing the three identified error categories (mistranslations, terminology or stylistic errors) and for control sentences without errors. We examined the three error types\u2019 effect on early (first fixation durations, first pass durations) and late eye movement measures (e.g., total reading time and regression path durations). Linear mixed-effects regression models predict what kind of behaviour of the DGT experts is associated with the correction of different error types during the post-editing process.<\/jats:p>","DOI":"10.3390\/informatics6030041","type":"journal-article","created":{"date-parts":[[2019,9,17]],"date-time":"2019-09-17T04:23:36Z","timestamp":1568694216000},"page":"41","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Translation Quality and Error Recognition in Professional Neural Machine Translation Post-Editing"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5187-1813","authenticated-orcid":false,"given":"Jennifer","family":"Vardaro","sequence":"first","affiliation":[{"name":"English Linguistics and Translation Studies, Johannes Gutenberg University, 76726 Mainz\/Germersheim, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5844-8092","authenticated-orcid":false,"given":"Moritz","family":"Schaeffer","sequence":"additional","affiliation":[{"name":"English Linguistics and Translation Studies, Johannes Gutenberg University, 76726 Mainz\/Germersheim, Germany"}]},{"given":"Silvia","family":"Hansen-Schirra","sequence":"additional","affiliation":[{"name":"English Linguistics and Translation Studies, Johannes Gutenberg University, 76726 Mainz\/Germersheim, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2019,9,17]]},"reference":[{"key":"ref_1","unstructured":"Chartier-Brun, P., and Mahler, K. (2019, April 30). Machine Translation and Neural Networks for a Multilingual EU. Available online: http:\/\/zerl.uni-koeln.de\/chartierbrun-mahler-2018-machine-translation-eu.html."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Moorkens, J. (2018). Translation Quality, Quality Management and Agency: Principles and Practice in the European Union Institutions. Translation Quality Assessment, Springer.","DOI":"10.1007\/978-3-319-91241-7"},{"key":"ref_3","unstructured":"ISO 17100 (2015). Translation Services\u2014Requirements for Translation Services, Technical Committee ISO\/TC37."},{"key":"ref_4","unstructured":"DIN ISO 18587 (2017). Translation Services\u2014Post-Editing of Machine Translation Output\u2014Requirements, Beuth. Technical Committee ISO\/TC 37\/SC 5."},{"key":"ref_5","first-page":"31","article-title":"Quality in Institutional Eu Translation: Parameters, Policies and Practices","volume":"8","author":"Biel","year":"2017","journal-title":"Qual. Asp. Inst. Transl."},{"key":"ref_6","first-page":"75","article-title":"Translation Manuals and Style Guides as Quality Assurance Indicators: The Case of The European Commission\u2019S Directorate-General for Translation","volume":"8","author":"Svoboda","year":"2017","journal-title":"Qual. Asp. Inst. Transl."},{"key":"ref_7","first-page":"16","article-title":"Quality of Written Communication in a Multilingual Organisation","volume":"1","author":"Wagner","year":"2000","journal-title":"Termin. Et Trad."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"651","DOI":"10.54648\/350927","article-title":"The Problem of Quality in EU Legislation: What on Earth is Really Wrong?","volume":"38","author":"Xanthaki","year":"2001","journal-title":"Common. Mark. Law Rev."},{"key":"ref_9","unstructured":"Daems, J., Vandepitte, S., Hartsuiker, R., and Macken, L. (November, January 30). The Impact of Machine Translation Error Types on Post-Editing Effort Indicators. Proceedings of the Fourth Workshop on Post-Editing Technology and Practice, Miami, FL, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.3389\/fpsyg.2017.01282","article-title":"Identifying the Machine Translation Error Types with the Greatest Impact on Post-editing Effort","volume":"8","author":"Daems","year":"2017","journal-title":"Front. Psychol."},{"key":"ref_11","unstructured":"Daems, J., Macken, L., and Vandepitte, S. (2013, January 2\u20136). Quality as the sum of its parts: A two-step approach for the identification of translation problems and translation quality assessment for HT and MT+PE. Proceedings of the MT Summit XIV Workshop on Post-editing Technology and Practice, Nice, France."},{"key":"ref_12","unstructured":"Lommel, A. (2019, March 15). Multidimensional Quality Metrics Definition. Available online: http:\/\/www.qt21.eu\/mqm-definition\/definition-2015-06-16.html."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.csl.2017.11.004","article-title":"Neural versus phrase-based MT quality: An in-depth analysis on English\u2013German and English\u2013French","volume":"49","author":"Bentivogli","year":"2018","journal-title":"Comput. Speech Lang."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bentivogli, L., Bisazza, A., Cettolo, M., and Federico, M. (2016). Neural versus Phrase-Based Machine Translation Quality: A Case Study. arXiv.","DOI":"10.18653\/v1\/D16-1025"},{"key":"ref_15","unstructured":"Daems, J. (2016). A Translation Robot for Each Translator? A Comparative Study of Manual Translation and Post-Editing of Machine Translations: Process, Quality and Translator Attitude, Ghent University."},{"key":"ref_16","unstructured":"O\u2019Brien, S., Winther Balling, L., Carl, M., Simard, M., and Specia, L. (2014). Pauses and Cognitive Effort in Post-Editing. Post-editing of Machine Translation Processes and Applications, Cambridge Scholars Publishing."},{"key":"ref_17","unstructured":"Castilho, S., Moorkens, J., and Gaspari, F. (2017). A Comparative Quality Evaluation of PBSMT and NMT Using Professional Translators, Cambridge Scholars Publishing."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/s10590-018-9214-x","article-title":"Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: A Case Study on English to Croatian","volume":"32","author":"Toral","year":"2018","journal-title":"Mach. Transl."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Toral, A., and S\u00e1nchez-Cartagena, V.M. (2017). A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions. arXiv.","DOI":"10.18653\/v1\/E17-1100"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Moorkens, J., Walker, C., and Federici, F.M. (2018). Eye Tracking as a measure of cognitive effort for post-editing of machine translation. Eye Tracking and Multidisciplinary Studies on Translation, John Benjamins.","DOI":"10.1075\/btl.143.04moo"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/s10590-019-09228-7","article-title":"A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output","volume":"33","author":"Koponen","year":"2019","journal-title":"Mach. Transl."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Schaeffer, M., Nitzke, J., Tardel, A., Oster, K., Gutermuth, S., and Hansen-Schirra, S. (2019). Eye-tracking revision processes of translation students and professional translators. Perspectives, 1\u201315.","DOI":"10.1080\/0907676X.2019.1597138"},{"key":"ref_23","unstructured":"Mertin, E. (2006). Prozessorientiertes Qualit\u00e4tsmanagement im Dienstleistungsbereich \u00dcbersetzen, Lang."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Carl, M., Schaeffer, M., and Bangalore, S. (2016). The CRITT translation process research database. New Directions in Empirical Translation Process Research\u2014Exploring the CRITT TPR-DB, Springer.","DOI":"10.1007\/978-3-319-20358-4"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/978-3-319-91241-7_10","article-title":"Machine Translation Quality Estimation: Applications and Future Perspectives","volume":"Volume 1","author":"Moorkens","year":"2018","journal-title":"Translation Quality Assessment"},{"key":"ref_26","unstructured":"Wang, J., Fan, K., Li, B., Zhou, F., Chen, B., Shi, Y., and Si, L. (November, January 31). Alibaba Submission for WMT18 Quality Estimation Task. Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Brussels, Belgium."},{"key":"ref_27","unstructured":"Shimanaka, H., Kajiwara, T., and Komachi, M. (November, January 31). RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation. Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Brussels, Belgium."},{"key":"ref_28","unstructured":"Specia, L. (2011, January 30\u201331). Exploiting Objective Annotations for Measuring Translation Post-editing Effort. Proceedings of the 15th conference of the European Association for Machine Translation, Leuven, Belgium."},{"key":"ref_29","unstructured":"Specia, L., Cancedda, N., Turchi, M., Cristianini, N., and Dymetman, M. (2009, January 14\u201315). Estimating the sentence-level quality of machine translation systems. Proceedings of the 13th Conference of the European Association for Machine Translation, Barcelona, Spain."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Luong, N.Q., Besacier, L., and Lecouteux, B. (2014, January 26\u201327). LIG System for Word Level QE task at WMT14. Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, MA, USA.","DOI":"10.3115\/v1\/W14-3342"},{"key":"ref_31","unstructured":"Scarton, C., and Specia, L. (2014, January 16\u201318). Document-level translation quality estimation: Exploring discourse and pseudo-references. Proceedings of the 17th Annual Conference of the European Association for Machine Translation, Dubrovnik, Croatia."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Graham, Y., Ma, Q., Baldwin, T., Liu, Q., Parra, C., and Scarton, C. (2017, January 3\u20137). Improving Evaluation of Document-level Machine Translation Quality Estimation. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Short Papers, Valencia, Spain.","DOI":"10.18653\/v1\/E17-2057"},{"key":"ref_33","first-page":"59","article-title":"Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output","volume":"96","year":"2011","journal-title":"Prague Bull. Math. Linguist."},{"key":"ref_34","unstructured":"Vilar, D., Xu, J., D\u2019Haro, L.F., and Ney, H. (2006, January 22\u201328). Error Analysis of Statistical Machine Translation Output. Proceedings of the presented at the International Conference on Language Resources and Evaluation, Genoa, Italy."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Vardaro, J., Schaeffer, M., and Hansen-Schirra, S. (2019, January 5\u20137). Comparing the Quality of Neural Machine Translation and Professional Post-Editing. Proceedings of the QoMEX, Berlin, Germany.","DOI":"10.1109\/QoMEX.2019.8743218"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Popovi\u0107, M., and Ney, H. (2007, January 23). Word error rates: Decomposition over Pos classes and applications for error analysis. Proceedings of the Second Workshop on Statistical Machine Translation\u2014StatMT \u201907, Prague, Czech Republic.","DOI":"10.3115\/1626355.1626362"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Corpas Pastor, G., and Dur\u00e1n-Mu\u00f1oz, I. (2017). SCATE Taxonomy and Corpus of Machine Translation Errors. Trends in E-Tools and Resources for Translators and Interpreters, Brill.","DOI":"10.1163\/9789004351790"},{"key":"ref_38","unstructured":"Lommel, A. (2014). A New Framework for Translation Quality Assessment, Deutsches Forschungszentrum f\u00fcr K\u00fcnstliche Intelligenz (DFKI)."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1037\/bul0000030","article-title":"From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words","volume":"142","author":"Weingarten","year":"2016","journal-title":"Psychol. Bull."},{"key":"ref_40","unstructured":"Green, S., Heer, J., and Manning, C.D. (May, January 27). The efficacy of human post-editing for language translation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems\u2014CHI \u201913, Paris, France."},{"key":"ref_41","unstructured":"\u010culo, O., and Nitzke, J. (June, January 30). Patterns of Terminological Variation in Post-editing and of Cognate Use in Machine Translation in Contrast to Human Translation. Proceedings of the 19th Annual Conference of the European Association for Machine Translation, Riga, Latvia."},{"key":"ref_42","unstructured":"Farrell, M. (2018, January 15\u201316). Machine Translation Markers in Post-Edited Machine Translation Output. Proceedings of the 40th Conference Translating and the Computer, London, UK."},{"key":"ref_43","unstructured":"Toral, A. (2019). Post-editese: An Exacerbated Translationese. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Tupper, D.E., and Cicerone, K.D. (1990). Introduction to the Neuropsychology of Everyday Life. The Neuropsychology of Everyday Life: Assessment and Basic Competencies, Foundations of Neuropsychology, Springer.","DOI":"10.1007\/978-1-4613-1503-2"},{"key":"ref_45","first-page":"1302","article-title":"Word reading and translation in bilinguals: The impact of formal and informal translation expertise","volume":"5","author":"Huepe","year":"2014","journal-title":"Front. Psychol."},{"key":"ref_46","first-page":"136","article-title":"Translog-II: A Program for Recording User Activity Data for Empirical Translation Process Research","volume":"3","author":"Carl","year":"2012","journal-title":"Int. J. Comput. Linguist. Appl."},{"key":"ref_47","unstructured":"Germann, U. (2008, January 15\u201320). Yawat: Yet another word alignment tool. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies Demo Session\u2014HLT \u201908, Columbus, OH, USA."},{"key":"ref_48","unstructured":"R Core Team (2018). R: A Language for Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"48","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting Linear Mixed-Effects Models Using lme4","volume":"67","author":"Bates","year":"2015","journal-title":"J. Stat. Softw."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"26","DOI":"10.18637\/jss.v082.i13","article-title":"lmerTest Package: Tests in Linear Mixed Effects Models","volume":"82","author":"Kuznetsova","year":"2017","journal-title":"J. Stat. Softw."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v087.i09","article-title":"Visualizing Fit and Lack of Fit in Complex Regression Models with Predictor Effect Plots and Partial Residuals","volume":"87","author":"Fox","year":"2018","journal-title":"J. Stat. Softw."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1598\/RRQ.35.4.7","article-title":"The Interplay Between Automatic and Control Processes in Reading","volume":"35","author":"Walczyk","year":"2000","journal-title":"Read. Res. Q."},{"key":"ref_53","unstructured":"Kroll, J.F., and De Groot, A.M.B. (2009). Bilingual Visual Wod Recognition and Lexical Access. Handbook of Bilingualism: Psycholinguistic Approaches, Oxford University Press."},{"key":"ref_54","unstructured":"Lenth, R. (2019). Emmeans: Estimated Marginal Means, aka Least-Squares Means, The Comprehensive R Archive Network. R Package Version 1.3.4."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1080\/00031305.1980.10483031","article-title":"Population Marginal Means in the Linear Model: An Alternative to Least Squares Means","volume":"34","author":"Searle","year":"1980","journal-title":"Am. Stat."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Snowling, M., and Hulme, C. (2005). Editorial Part I. the Science of Reading: A handbook, Blackwell Publishing.","DOI":"10.1111\/b.9781405114882.2005.x"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1075\/ata.xv.04dra","article-title":"Coordination of reading and writing processes in translation: An eye on uncharted territory","volume":"15","author":"Dragsted","year":"2010","journal-title":"Transl. Cogn."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1075\/tis.8.2.03sch","article-title":"Shared representations and the translation process: A recursive model","volume":"8","author":"Schaeffer","year":"2013","journal-title":"Transl. Interpret. Stud."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/6\/3\/41\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:20:49Z","timestamp":1760188849000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/6\/3\/41"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,17]]},"references-count":58,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2019,9]]}},"alternative-id":["informatics6030041"],"URL":"https:\/\/doi.org\/10.3390\/informatics6030041","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9,17]]}}}