{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T15:38:21Z","timestamp":1776872301222,"version":"3.51.2"},"reference-count":31,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,9,14]],"date-time":"2021-09-14T00:00:00Z","timestamp":1631577600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>As Machine Translation (MT) becomes increasingly ubiquitous, so does its use in professional translation workflows. However, its proliferation in the translation industry has brought about new challenges in the field of Post-Editing (PE). We are now faced with a need to find effective tools to assess the quality of MT systems to avoid underpayments and mistrust by professional translators. In this scenario, one promising field of study is MT Quality Estimation (MTQE), as this aims to determine the quality of an automatic translation and, indirectly, its degree of post-editing difficulty. However, its impact on the translation workflows and the translators\u2019 cognitive load is still to be fully explored. We report on the results of an impact study engaging professional translators in PE tasks using MTQE. To assess the translators\u2019 cognitive load we measure their productivity both in terms of time and effort (keystrokes) in three different scenarios: translating from scratch, post-editing without using MTQE, and post-editing using MTQE. Our results show that good MTQE information can improve post-editing efficiency and decrease the cognitive load on translators. This is especially true for cases with low MT quality.<\/jats:p>","DOI":"10.3390\/informatics8030061","type":"journal-article","created":{"date-parts":[[2021,9,14]],"date-time":"2021-09-14T23:34:11Z","timestamp":1631662451000},"page":"61","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["The Role of Machine Translation Quality Estimation in the Post-Editing Workflow"],"prefix":"10.3390","volume":"8","author":[{"given":"Hannah","family":"B\u00e9chara","sequence":"first","affiliation":[{"name":"Hertie School Data Science Lab., 10117 Berlin, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2067-8890","authenticated-orcid":false,"given":"Constantin","family":"Or\u0103san","sequence":"additional","affiliation":[{"name":"Centre of Translation Studies, University of Surrey, Guildford GU2 7XH, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8412-1525","authenticated-orcid":false,"given":"Carla","family":"Parra Escart\u00edn","sequence":"additional","affiliation":[{"name":"RWS Language Weaver, Dublin, Ireland"}]},{"given":"Marcos","family":"Zampieri","sequence":"additional","affiliation":[{"name":"Rochester Institute of Technology, Department of Computer Science, Rochester, NY 14623, USA"}]},{"given":"William","family":"Lowe","sequence":"additional","affiliation":[{"name":"Hertie School Data Science Lab., 10117 Berlin, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,14]]},"reference":[{"key":"ref_1","unstructured":"Zaretskaya, A., Pastor, G.C., and Seghiri, M. (2015, January 29\u201331). Translators\u2019 Requirements for Translation Technologies: A user survey. Proceedings of the 7th International Conference of the Iberian Association of Translation and Interpreting Studies (AIETI), New Horizons in Translation and Interpreting Studies, Malaga, Spain."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1075\/babel.00062.sch","article-title":"Translation Memories and the Translator: A report on a user survey","volume":"64","author":"Schneider","year":"2018","journal-title":"Babel"},{"key":"ref_3","first-page":"131","article-title":"Is machine translation post-editing worth the effort? A survey of research into post-editing and effort","volume":"25","author":"Koponen","year":"2016","journal-title":"J. Spec. Transl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1515\/pralin-2017-0032","article-title":"Questing for Quality Estimation A User Study","volume":"108","year":"2017","journal-title":"Prague Bull. Math. Linguist."},{"key":"ref_5","unstructured":"Guerberof, A. (2014). Productivity and Quality in the Post-Edition of Outputs from Translation Memories and Machine Translation. [Ph.D. Thesis, Rovira and Virgili University]."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zampieri, M., and Vela, M. (2014, January 26). Quantifying the Influence of MT Output in the Translators\u2019 Performance: A case study in technical translation. Proceedings of the EACL 2014 Workshop on Humans and Computer-Assisted Translation, Gothenburg, Sweden.","DOI":"10.3115\/v1\/W14-0314"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Turchi, M., Negri, M., and Federico, M. (2015, January 26\u201331). MT Quality Estimation for Computer-assisted Translation: Does it Really Help?. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), Beijing, China.","DOI":"10.3115\/v1\/P15-2087"},{"key":"ref_8","unstructured":"Federico, M., Bertoldi, N., Cettolo, M., Negri, M., Turchi, M., Trombetti, M., Cattelan, A., Farina, A., Lupinetti, D., and Martines, A. (2014, January 23\u201329). The matecat tool. Proceedings of the COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations, Dublin, Ireland."},{"key":"ref_9","unstructured":"Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006, January 8\u201312). A Study of Translation Edit Rate with Targeted Human Annotation. Proceedings of the Association for Machine Translation in the Americas (AMTA), Cambridge, MA, USA."},{"key":"ref_10","unstructured":"Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., and Zens, R. (2007, January 23\u201330). Moses: Open Source Toolkit for Statistical Machine Translation. Proceedings of the Association for Computational Linguistics (ACL), Prague, Czech Republic."},{"key":"ref_11","unstructured":"Parra Escart\u00edn, C., and Arcedillo, M. (November, January 30). Machine translation evaluation made fuzzier: A study on post-editing productivity and evaluation metrics in commercial settings. Proceedings of the MT Summit XV, Miami, FL, USA."},{"key":"ref_12","unstructured":"Parra Escart\u00edn, C., and Arcedillo, M. (November, January 30). Living on the edge: Productivity gain thresholds in machine translation evaluation metrics. Proceedings of the Fourth Workshop on Post-Editing Technology and Practice, Miami, FL, USA."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/s10590-015-9175-2","article-title":"Correlations of perceived post-editing effort with measurements of actual effort","volume":"29","author":"Moorkens","year":"2015","journal-title":"Mach. Transl."},{"key":"ref_14","first-page":"141","article-title":"Comparing Translator Acceptability of TM and SMT outputs","volume":"4","author":"Moorkens","year":"2016","journal-title":"Balt. J. Mod. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kenny, D. (2017). Assessing User Interface Needs of Post-Editors of Machine Translation. Human Issues in Translation Technology: The IATIS Yearbook, Routledge.","DOI":"10.4324\/9781315648934"},{"key":"ref_16","unstructured":"Teixeira, C., and O\u2019Brien, S. (2017, January 18\u201322). The Impact of MT Quality Estimation on Post-Editing Effort. Proceedings of the MT Summit XVI, Volume 2: Users and Translators Track, Nagoya, Japan."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Graham, Y., Mathur, N., and Baldwin, T. (June, January 31). Accurate evaluation of segment-level machine translation metrics. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, Denver, CO, USA.","DOI":"10.3115\/v1\/N15-1124"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.csl.2018.10.005","article-title":"Estimating Post-editing Time Using a Gold-standard Set of Machine Translation Errors","volume":"55","author":"Tezcan","year":"2019","journal-title":"Comput. Speech Lang."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, K., Wang, J., Ge, N., Shi, Y., Zhao, Y., and Fan, K. (2020, January 16\u201320). Computer Assisted Translation with Neural Quality Estimation and Auotmatic Post-Editing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, Online.","DOI":"10.18653\/v1\/2020.findings-emnlp.197"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Herbig, N., D\u00fcwel, T., Pal, S., Meladaki, K., Monshizadeh, M., Kr\u00fcger, A., and van Genabith, J. (2020, January 5\u201310). MMPE: A multi-modal interface for post-editing machine translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.155"},{"key":"ref_21","unstructured":"Herbig, N., Pal, S., Kr\u00fcger, A., and van Genabith, J. (2021). Multi-modal estimation of cognitive load in post-editing of machine translation. Translation, Interpreting, Cognition, Language Science Press."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chollampatt, S., Susanto, R.H., Tan, L., and Szymanska, E. (2020, January 16\u201320). Can Automatic Post-Editing Improve NMT?. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.217"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/s10590-020-09252-y","article-title":"A review of the state-of-the-art in automatic post-editing","volume":"35","author":"Shterionov","year":"2021","journal-title":"Mach. Transl."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lee, W., Jung, B., Shin, J., and Lee, J.H. (2021, January 19\u201323). Adaptation of Back-translation to Automatic Post-Editing for Synthetic Data Generation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online.","DOI":"10.18653\/v1\/2021.eacl-main.322"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"7","DOI":"10.2478\/v10108-010-0010-x","article-title":"A productivity test of statistical machine translation postediting in a typical localisation context","volume":"93","author":"Plitt","year":"2010","journal-title":"Prague Bull. Math. Linguist."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Specia, L., Paetzold, G., and Scarton, C. (2015, January 26\u201331). Multi-level Translation Quality Prediction with QuEst++. Proceedings of the ACL-IJCNLP 2015 System Demonstrations; Association for Computational Linguistics and The Asian Federation of Natural Language Processing, Beijing, China.","DOI":"10.3115\/v1\/P15-4020"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Gupta, R., Bechara, H., El Maarouf, I., and Orasan, C. (2014, January 23\u201324). UoW: NLP techniques developed at the University of Wolverhampton for Semantic Similarity and Textual Entailment. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval\u201914), Dublin, Ireland.","DOI":"10.3115\/v1\/S14-2139"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"B\u00e9chara, H., Costa, H., Taslimipoor, S., Gupta, R., Orasan, C., Corpas Pastor, G., and Mitkov, R. (2015, January 4\u20135). MiniExperts: An SVM approach for Measuring Semantic Textual Similarity. Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval\u201915, Denver, CO, USA.","DOI":"10.18653\/v1\/S15-2017"},{"key":"ref_29","unstructured":"B\u00e9chara, H., Parra Escart\u00edn, C., Orasan, C., and Specia, L. (June, January 30). Semantic textual similarity in quality estimation. Proceedings of the 19th Annual Conference of the European Association for Machine Translation, Riga, Latvia."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ranasinghe, T., Or\u0103san, C., and Mitkov, R. (2020, January 8\u201313). TransQuest: Translation quality estimation with cross-lingual transformers. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.445"},{"key":"ref_31","unstructured":"Aziz, W., Castilho, S., and Specia, L. (2012, January 23\u201325). PET: A Tool for Post-editing and Assessing Machine Translation. Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/3\/61\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:02:33Z","timestamp":1760166153000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/3\/61"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,14]]},"references-count":31,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["informatics8030061"],"URL":"https:\/\/doi.org\/10.3390\/informatics8030061","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,14]]}}}