{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T08:21:32Z","timestamp":1777537292076,"version":"3.51.4"},"reference-count":8,"publisher":"MIT Press","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["TACL"],"published-print":{"date-parts":[[2018,12]]},"abstract":"<jats:p> Neural architectures are prominent in the construction of language models (LMs). However, word-level prediction is typically agnostic of subword-level information (characters and character sequences) and operates over a closed vocabulary, consisting of a limited word set. Indeed, while subword-aware models boost performance across a variety of NLP tasks, previous work did not evaluate the ability of these models to assist next-word prediction in language modeling tasks. Such subword-level informed models should be particularly effective for morphologically-rich languages (MRLs) that exhibit high type-to-token ratios. In this work, we present a large-scale LM study on 50 typologically diverse languages covering a wide variety of morphological systems, and offer new LM benchmarks to the community, while considering subword-level information. The main technical contribution of our work is a novel method for injecting subword-level information into semantic word vectors, integrated into the neural language modeling training, to facilitate word-level prediction. We conduct experiments in the LM setting where the number of infrequent words is large, and demonstrate strong perplexity gains across our 50 languages, especially for morphologically-rich languages. Our code and data sets are publicly available. <\/jats:p>","DOI":"10.1162\/tacl_a_00032","type":"journal-article","created":{"date-parts":[[2018,12,10]],"date-time":"2018-12-10T19:32:50Z","timestamp":1544470370000},"page":"451-465","source":"Crossref","is-referenced-by-count":17,"title":["Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction"],"prefix":"10.1162","volume":"6","author":[{"given":"Daniela","family":"Gerz","sequence":"first","affiliation":[{"name":"Language Technology Lab, DTAL, University of Cambridge,"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivan","family":"Vuli\u0107","sequence":"additional","affiliation":[{"name":"Language Technology Lab, DTAL, University of Cambridge,"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edoardo","family":"Ponti","sequence":"additional","affiliation":[{"name":"Language Technology Lab, DTAL, University of Cambridge,"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Naradowsky","sequence":"additional","affiliation":[{"name":"Johns Hopkins University,"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roi","family":"Reichart","sequence":"additional","affiliation":[{"name":"Faculty of Industrial Engineering and Management, Technion, IIT,"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna","family":"Korhonen","sequence":"additional","affiliation":[{"name":"Language Technology Lab, DTAL, University of Cambridge,"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"p_4","first-page":"1137","volume":"3","author":"Bengio Yoshua","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"p_13","first-page":"2121","volume":"12","author":"Duchi John","year":"2011","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"p_18","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1006\/csla.2001.0174","volume":"15","author":"Goodman Joshua T.","year":"2001","journal-title":"Computer Speech & Language"},{"key":"p_25","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"issue":"2","key":"p_37","first-page":"313","volume":"19","author":"Marcus Mitchell P.","year":"1993","journal-title":"Computational Linguistics"},{"key":"p_40","first-page":"309","volume":"5","author":"Mrk\u0161i\u0107 Nikola","year":"2017","journal-title":"Transactions of the ACL"},{"issue":"3","key":"p_46","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1109\/TASLP.2015.2400218","volume":"23","author":"Sundermeyer Martin","year":"2015","journal-title":"IEEE Transactions on Audio, Speech and Language Processing"},{"key":"p_53","first-page":"345","volume":"3","author":"Wieting John","year":"2015","journal-title":"Transactions of the ACL"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00032","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:38:00Z","timestamp":1615585080000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12]]},"references-count":8,"alternative-id":["10.1162\/tacl_a_00032"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00032","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,12]]}}}