{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,8]],"date-time":"2026-03-08T05:24:49Z","timestamp":1772947489631,"version":"3.50.1"},"reference-count":34,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,8,28]],"date-time":"2020-08-28T00:00:00Z","timestamp":1598572800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003130","name":"Fonds Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["G.0064.17N"],"award-info":[{"award-number":["G.0064.17N"]}],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Due to the growing success of neural machine translation (NMT), many have started to question its applicability within the field of literary translation. In order to grasp the possibilities of NMT, we studied the output of the neural machine system of Google Translate (GNMT) and DeepL when applied to four classic novels translated from English into Dutch. The quality of the NMT systems is discussed by focusing on manual annotations, and we also employed various metrics in order to get an insight into lexical richness, local cohesion, syntactic, and stylistic difference. Firstly, we discovered that a large proportion of the translated sentences contained errors. We also observed a lower level of lexical richness and local cohesion in the NMTs compared to the human translations. In addition, NMTs are more likely to follow the syntactic structure of a source sentence, whereas human translations can differ. Lastly, the human translations deviate from the machine translations in style.<\/jats:p>","DOI":"10.3390\/informatics7030032","type":"journal-article","created":{"date-parts":[[2020,8,30]],"date-time":"2020-08-30T06:06:22Z","timestamp":1598767582000},"page":"32","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Gutenberg Goes Neural: Comparing Features of Dutch Human Translations with Raw Neural Machine Translation Outputs in a Corpus of English Literary Classics"],"prefix":"10.3390","volume":"7","author":[{"given":"Rebecca","family":"Webster","sequence":"first","affiliation":[{"name":"LT<sup>3<\/sup>, Language and Translation Technology Team, Ghent University, 9000 Ghent, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9575-7844","authenticated-orcid":false,"given":"Margot","family":"Fonteyne","sequence":"additional","affiliation":[{"name":"LT<sup>3<\/sup>, Language and Translation Technology Team, Ghent University, 9000 Ghent, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8707-6176","authenticated-orcid":false,"given":"Arda","family":"Tezcan","sequence":"additional","affiliation":[{"name":"LT<sup>3<\/sup>, Language and Translation Technology Team, Ghent University, 9000 Ghent, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7516-7487","authenticated-orcid":false,"given":"Lieve","family":"Macken","sequence":"additional","affiliation":[{"name":"LT<sup>3<\/sup>, Language and Translation Technology Team, Ghent University, 9000 Ghent, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3734-5160","authenticated-orcid":false,"given":"Joke","family":"Daems","sequence":"additional","affiliation":[{"name":"LT<sup>3<\/sup>, Language and Translation Technology Team, Ghent University, 9000 Ghent, Belgium"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Matusov, E. (2019, January 19). The Challenges of Using Neural Machine Translation for Literature. Proceedings of the Qualities of Literary Machine Translation, Dublin, Ireland.","DOI":"10.18653\/v1\/W19-5209"},{"key":"ref_2","unstructured":"Voigt, R., and Jurafsky, D. (2012, January 8). Towards a Literary Machine Translation: The Role of Referential Cohesion. Proceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature, Montr\u00e9al, QB, Canada."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Moorkens, J., Castilho, S., Gaspari, F., and Doherty, S. (2018). What Level of Quality Can Neural Machine Translation Attain on Literary Text?. Translation Quality Assessment, Springer International Publishing AG.","DOI":"10.1007\/978-3-319-91241-7"},{"key":"ref_4","unstructured":"Fonteyne, M., Tezcan, A., and Macken, L. (2020, January 11\u201316). Literary Machine Translation under the Magnifying Glass: Assessing the Quality of an NMT-Translated Detective Novel on Document Level. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_5","unstructured":"Hadley, J., Popovi\u0107, M., Afli, H., and Way, A. When a \u2018sport\u2019 is a person and other issues for NMT of novels. Proceedings of the Qualities of Literary Machine Translation."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1163\/9789004351790_012","article-title":"SCATE taxonomy and corpus of machine translation errors","volume":"Volume 45","author":"Pastor","year":"2017","journal-title":"Trends in E-Tools and Resources for Translators and Interpreters"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PL, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_8","unstructured":"Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., and Li, M. (2018). Achieving Human Parity on Automatic Chinese to English News Translation. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1007\/s10590-018-9220-z","article-title":"Human versus automatic quality evaluation of NMT and PBSMT","volume":"32","author":"Shterionov","year":"2018","journal-title":"Mach. Transl."},{"key":"ref_10","unstructured":"Vanmassenhove, E., Shterionov, D., and Way, A. (2019). Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation. Proceedings of Machine Translation Summit XVII Volume 1: Research Track, European Association for Machine Translation."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"602","DOI":"10.3758\/s13428-016-0734-0","article-title":"Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading","volume":"49","author":"Cop","year":"2017","journal-title":"Behav. Res. Methods"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1093\/llc\/17.3.267","article-title":"\u2019Delta\u2019: A Measure of Stylistic Difference and a Guide to Likely Authorship","volume":"17","author":"Burrows","year":"2002","journal-title":"Lit. Linguist. Comput."},{"key":"ref_13","unstructured":"Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., and Mazo, H. (2018). A fine-grained error analysis of NMT, PBMT and RBMT output for English-to-Dutch. Proceedings of the Eleventh International Conference on Language Resources and Evaluation, European Language Resources Association (ELRA)."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1037\/h0093508","article-title":"Sudies in Language Behavior","volume":"56","author":"Johnson","year":"1944","journal-title":"Psychol. Monogr."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"L\u00e4ubli, S., Sennrich, R., and Volk, M. (November, January 31). Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1512"},{"key":"ref_16","unstructured":"Voita, E., Sennrich, R., and Titov, I. (August, January 28). When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_17","unstructured":"Werlen, L.M., Ram, D., Pappas, N., and Henderson, J. (November, January 31). Document-Level Neural Machine Translation with Hierarchical Attention Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"McNamara, D.S., Graesser, A.C., McCarthy, P.M., and Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix, Cambridge University Press.","DOI":"10.1017\/CBO9780511894664"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1227","DOI":"10.3758\/s13428-015-0651-7","article-title":"The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion","volume":"48","author":"Crossley","year":"2016","journal-title":"Behav. Res. Methods"},{"key":"ref_20","first-page":"101","article-title":"Predicting syntactic equivalence between source and target sentences","volume":"9","author":"Vanroy","year":"2019","journal-title":"Comput. Linguist. Neth. J."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Vanroy, B., De Clercq, O., Tezcan, A., Daems, J., and Macken, L. (2020). Metrics of syntactic equivalence to assess translation difficulty. Explorations in Empirical Translation Process Research, Springer International Publishing AG. Submitted.","DOI":"10.1007\/978-3-030-69777-8_10"},{"key":"ref_22","unstructured":"Matthews, P. (1981). Syntax. Cambridge Textbooks in Linguistics, Cambridge University Press."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/322139.322143","article-title":"The tree-to-tree correction problem","volume":"26","author":"Tai","year":"1979","journal-title":"J. ACM"},{"key":"ref_24","unstructured":"Nivre, J., De Marneffe, M.C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C.D., McDonald, R., Petrov, S., Pyysalo, S., and Silveira, N. (2016, January 23\u201328). Universal dependencies v1: A multilingual treebank collection. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), Portoro\u017e, Slovenia."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C.D. (2020, January 5\u201310). Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. Proceedings of the 58th Annual Meetingof the Association for Computational Linguistics: System Demonstrations, Seattle, WA, USA.","DOI":"10.18653\/v1\/2020.acl-demos.14"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1093\/llc\/fqx023","article-title":"Understanding and explaining Delta measures for authorship attribution","volume":"32","author":"Evert","year":"2017","journal-title":"Digit. Scholarsh. Humanit."},{"key":"ref_27","unstructured":"Proisl, T., Evert, S., Jannidis, F., Sch\u00f6ch, C., Konle, L., and Pielstr\u00f6m, S. (2018, January 7\u201312). Delta vs. N-Gram Tracing: Evaluating the Robustness of Authorship Attribution Methods. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bentivogli, L., Bisazza, A., Cettolo, M., and Federico, M. (2016, January 1\u20134). Neural versus Phrase-Based Machine Translation Quality: A Case Study. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1025"},{"key":"ref_29","unstructured":"Kong, X., Tu, Z., Shi, S., Hovy, E., and Zhang, T. (February, January 27). Neural machine translation with adequacy-oriented learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Koehn, P., and Knowles, R. (2017). Six Challenges for Neural Machine Translation. Proceedings of the First, Workshop on Neural Machine Translation, Association for Computational Linguistics.","DOI":"10.18653\/v1\/W17-3204"},{"key":"ref_31","unstructured":"Baker, M. (2001). In Other Words\/Mona Baker, Routledge."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Besacier, L., and Schwartz, L. (2015, January 4). Automated Translation of a Literary Work: A Pilot Study. Proceedings of the Fourth Workshop on Computational Linguistics for Literature, Denver, CO, USA.","DOI":"10.3115\/v1\/W15-0713"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"708","DOI":"10.1093\/llc\/fqt027","article-title":"The stylistics and stylometry of collaborative translation: Woolf\u2019s Night and Day in Polish","volume":"28","author":"Rybicki","year":"2013","journal-title":"Lit. Linguist. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Qin, Y., Zhang, J., and Lu, X. (2019, January 14\u201317). The Gap between NMT and Professional Translation from the Perspective of Discourse. Proceedings of the 2019 3rd International Conference on Innovation in Artificial Intelligence, New York, NY, USA.","DOI":"10.1145\/3319921.3319936"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/7\/3\/32\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:04:29Z","timestamp":1760177069000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/7\/3\/32"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,28]]},"references-count":34,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["informatics7030032"],"URL":"https:\/\/doi.org\/10.3390\/informatics7030032","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,28]]}}}