{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,13]],"date-time":"2025-05-13T22:00:11Z","timestamp":1747173611128,"version":"3.40.5"},"reference-count":45,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2020,7,2]],"date-time":"2020-07-02T00:00:00Z","timestamp":1593648000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Neural Networks applied to Machine Translation need a finite vocabulary to express textual information as a sequence of discrete tokens. The currently dominant subword vocabularies exploit statistically-discovered common parts of words to achieve the flexibility of character-based vocabularies without delegating the whole learning of word formation to the neural network. However, they trade this for the inability to apply word-level token associations, which limits their use in semantically-rich areas and prevents some transfer learning approaches e.g. cross-lingual pretrained embeddings, and reduces their interpretability. In this work, we propose new hybrid linguistically-grounded vocabulary definition strategies that keep both the advantages of subword vocabularies and the word-level associations, enabling neural networks to profit from the derived benefits. We test the proposed approaches in both morphologically rich and poor languages, showing that, for the former, the quality in the translation of out-of-domain texts is improved with respect to a strong subword baseline.<\/jats:p>","DOI":"10.1017\/s1351324920000364","type":"journal-article","created":{"date-parts":[[2020,7,2]],"date-time":"2020-07-02T04:55:26Z","timestamp":1593665726000},"page":"485-506","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":4,"title":["Linguistic knowledge-based vocabularies for Neural Machine Translation"],"prefix":"10.1017","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0248-899X","authenticated-orcid":false,"given":"Noe","family":"Casas","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marta R.","family":"Costa-juss\u00e0","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jos\u00e9 A. R.","family":"Fonollosa","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juan A.","family":"Alonso","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ram\u00f3n","family":"Fanlo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2020,7,2]]},"reference":[{"key":"S1351324920000364_ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6319"},{"key":"S1351324920000364_ref37","unstructured":"Sennrich, R. , Volk, M. and Schneider, G. (2013). Exploiting synergies between open resources for German dependency parsing, POS-tagging, and morphological analysis. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pp. 601\u2013609."},{"key":"S1351324920000364_ref21","unstructured":"Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain. Association for Computational Linguistics, pp. 388\u2013395."},{"key":"S1351324920000364_ref7","unstructured":"Callison-Burch, C. , Osborne, M. and Koehn, P. (2006). Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics."},{"key":"S1351324920000364_ref34","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2209"},{"key":"S1351324920000364_ref6","doi-asserted-by":"publisher","DOI":"10.1109\/72.279181"},{"key":"S1351324920000364_ref24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1166"},{"key":"S1351324920000364_ref28","doi-asserted-by":"crossref","unstructured":"Papineni, K. , Roukos, S. , Ward, T. and Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics, pp. 311\u2013318. 10.3115\/1073083.1073135.","DOI":"10.3115\/1073083.1073135"},{"key":"S1351324920000364_ref26","unstructured":"Mikolov, T. , Sutskever, I. , Chen, K. , Corrado, G.S. and Dean, J. (2013) Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. Curran Associates, Inc., pp. 3111\u20133119."},{"key":"S1351324920000364_ref33","first-page":"895","volume-title":"Advances in Neural Information Processing Systems","author":"Sch\u00fctze","year":"1993"},{"key":"S1351324920000364_ref1","doi-asserted-by":"publisher","DOI":"10.3115\/1614049.1614050"},{"key":"S1351324920000364_ref32","unstructured":"Riezler, S. and Maxwell, J.T. (2005). On some pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Ann Arbor, Michigan. Association for Computational Linguistics, pp. 57\u201364."},{"key":"S1351324920000364_ref38","doi-asserted-by":"crossref","first-page":"1441","DOI":"10.21437\/Interspeech.2011-252","volume-title":"Interspeech","author":"Shaik","year":"2011"},{"key":"S1351324920000364_ref14","unstructured":"Faruqui, M. , Schuetze, H. , Trancoso, I. and Yaghoobzadeh, Y. (2017). Proceedings of the First Workshop on Subword and Character Level Models in NLP. Association for Computational Linguistics."},{"key":"S1351324920000364_ref44","unstructured":"Virpioja, S. , Smit, P. , Gr\u00f6nroos, S.-A. and Kurimo, M. (2013). Morfessor 2.0: Python implementation and extensions for Morfessor baseline. Technical report."},{"key":"S1351324920000364_ref2","unstructured":"Alonso, J.A. and Thurmair, G. (2003). The comprendium translator system. In Proceedings of the Ninth Machine Translation Summit."},{"key":"S1351324920000364_ref11","doi-asserted-by":"publisher","DOI":"10.3115\/1118647.1118650"},{"key":"S1351324920000364_ref17","unstructured":"Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universit\u00e4t M\u00fcnchen 910 (1)."},{"key":"S1351324920000364_ref12","unstructured":"de Marneffe, M.-C. , Dozat, T. , Silveira, N. , Haverinen, K. , Ginter, F. , Nivre, J. and Manning, C.D. (2014). Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), Reykjavik, Iceland. European Languages Resources Association (ELRA), pp. 4585\u20134592."},{"key":"S1351324920000364_ref43","unstructured":"Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A.N. , Kaiser, L.u. and Polosukhin, I. (2017) Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 5998\u20136008."},{"key":"S1351324920000364_ref13","unstructured":"Etchegoyhen, T. , Azpeitia, A. and P\u00e9rez, N. (2016). Exploiting a large strongly comparable corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), Portoro\u017e, Slovenia. European Language Resources Association (ELRA), pp. 3523\u20133529."},{"key":"S1351324920000364_ref16","unstructured":"Garc\u0131a-Mart\u0131nez, M. , Barrault, L. and Bougares, F. (2016). Factored neural machine translation architectures. In Proceedings of the International Workshop on Spoken Language Translation. Seattle, USA, IWSLT, vol. 16."},{"key":"S1351324920000364_ref3","unstructured":"Avramidis, E. and Koehn, P. (2008). Enriching morphologically poor languages for statistical machine translation. In Proceedings of ACL-08: HLT, Columbus, Ohio. Association for Computational Linguistics, pp. 763\u2013770."},{"key":"S1351324920000364_ref35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"S1351324920000364_ref40","unstructured":"Sutskever, I. , Martens, J. and Hinton, G. (2011). Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML\u201911, Madison, WI, USA. Omnipress, pp. 1017\u20131024. ISBN 9781450306195."},{"key":"S1351324920000364_ref42","doi-asserted-by":"crossref","unstructured":"Vania, C. and Lopez, A. (2017). From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp. 2016\u20132027. 10.18653\/v1\/P17-1184.","DOI":"10.18653\/v1\/P17-1184"},{"key":"S1351324920000364_ref25","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"S1351324920000364_ref29","unstructured":"Passban, P. (2017). Machine Translation of Morphologically Rich Languages Using Deep Neural Networks. PhD Thesis, Dublin City University."},{"key":"S1351324920000364_ref18","unstructured":"Honnibal, M. and Montani, I. (to appear) spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing."},{"key":"S1351324920000364_ref36","unstructured":"Sennrich, R. , Schneider, G. , Volk, M. and Warin, M. (2009). A new hybrid dependency parser for German. Proceedings of the German Society for Computational Linguistics and Language Technology 115, 124."},{"key":"S1351324920000364_ref23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1100"},{"key":"S1351324920000364_ref30","doi-asserted-by":"crossref","unstructured":"Ponti, E.M. , Reichart, R. , Korhonen, A. and Vuli\u0106 I. (2018). Isomorphic transfer of syntactic structures in cross-lingual NLP. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics.","DOI":"10.18653\/v1\/P18-1142"},{"key":"S1351324920000364_ref27","unstructured":"Mikolov, T. , Sutskever, I. , Deoras, A. , Le, H.-S. , Kombrink, S. and Cernocky, J. (2012). Subword language modeling with neural networks. Technical report, Faculty of Information Technology, Brno University of Technology."},{"key":"S1351324920000364_ref45","unstructured":"Wu, Y. , Schuster, M. , Chen, Z. , Le, Q.V. , Norouzi, M. , Macherey, W. , Krikun, M. , Cao, Y. , Gao, Q. , Macherey, K. , et al. (2016 ). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint ."},{"key":"S1351324920000364_ref8","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"S1351324920000364_ref10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4123"},{"key":"S1351324920000364_ref22","doi-asserted-by":"publisher","DOI":"10.1075\/li.13.2.06lam"},{"key":"S1351324920000364_ref5","unstructured":"Banerjee, S. and Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Ann Arbor, Michigan. Association for Computational Linguistics, pp. 65\u201372."},{"key":"S1351324920000364_ref19","unstructured":"Jan, N. , Cattoni, R. , Sebastian, S. , Cettolo, M. , Turchi, M. and Federico, M. (2018). The iwslt 2018 evaluation campaign. In International Workshop on Spoken Language Translation, pp. 2\u20136."},{"key":"S1351324920000364_ref41","first-page":"119","article-title":"Using POS information for statistical machine translation into","volume":"34","author":"Ueffing","year":"2003","journal-title":"Pure and Applications Algebra"},{"key":"S1351324920000364_ref20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-4012"},{"key":"S1351324920000364_ref9","unstructured":"Conforti, C. , Huck, M. and Fraser, A. (2018). Neural morphological tagging of lemma sequences for machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), Boston, MA. Association for Machine Translation in the Americas, pp. 39\u201353."},{"key":"S1351324920000364_ref15","unstructured":"Faruqui, M. , Schuetze, H. , Trancoso, I. , Tsvetkov, Y. and Yaghoobzadeh, Y. (2018). Proceedings of the Second Workshop on Subword and Character Level Models in NLP (SCLeM 2018). Association for Computational Linguistics."},{"key":"S1351324920000364_ref4","unstructured":"Bahdanau, D. , Cho, K. and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings."},{"key":"S1351324920000364_ref39","doi-asserted-by":"crossref","unstructured":"Song, K. , Zhang, Y. , Zhang, M. and Luo, W. (2018). Improved English to Russian translation by neural suffix prediction. In Thirty-Second AAAI Conference on Artificial Intelligence.","DOI":"10.1609\/aaai.v32i1.11250"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324920000364","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T18:15:20Z","timestamp":1667240120000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324920000364\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,2]]},"references-count":45,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["S1351324920000364"],"URL":"https:\/\/doi.org\/10.1017\/s1351324920000364","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"type":"print","value":"1351-3249"},{"type":"electronic","value":"1469-8110"}],"subject":[],"published":{"date-parts":[[2020,7,2]]},"assertion":[{"value":"\u00a9 The Author(s), 2020. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}}]}}