{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T19:53:20Z","timestamp":1712951600618},"reference-count":65,"publisher":"Cambridge University Press (CUP)","issue":"6","license":[{"start":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T00:00:00Z","timestamp":1594166400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Distributional semantic word representations are at the basis of most modern NLP systems. Their usefulness has been proven across various tasks, particularly as inputs to deep learning models. Beyond that, much work investigated fine-tuning the generic word embeddings to leverage linguistic knowledge from large lexical resources. Some work investigated context-dependent word token embeddings motivated by word sense disambiguation, using sequential context and large lexical resources. More recently, acknowledging the need for an in-context representation of words, some work leveraged information derived from language modelling and large amounts of data to induce contextualised representations. In this paper, we investigate <jats:bold>S<\/jats:bold>yntax-<jats:bold>A<\/jats:bold>ware word <jats:bold>Tok<\/jats:bold>en <jats:bold>E<\/jats:bold>mbeddings (<jats:bold>SATokE<\/jats:bold>) as a way to explicitly encode specific information derived from the linguistic analysis of a sentence in vectors which are input to a deep learning model. We propose an efficient unsupervised learning algorithm based on tensor factorisation for computing these token embeddings given an arbitrary graph of linguistic structure. Applying this method to syntactic dependency structures, we investigate the usefulness of such token representations as part of deep learning models of text understanding. We encode a sentence either by learning embeddings for its tokens and the relations between them from scratch or by leveraging pre-trained relation embeddings to infer token representations. Given sufficient data, the former is slightly more accurate than the latter, yet both provide more informative token embeddings than standard word representations, even when the word representations have been learned on the same type of context from larger corpora (namely pre-trained dependency-based word embeddings). We use a large set of supervised tasks and two major deep learning families of models for sentence understanding to evaluate our proposal. We empirically demonstrate the superiority of the token representations compared to popular distributional representations of words for various sentence and sentence pair classification tasks.<\/jats:p>","DOI":"10.1017\/s1351324920000297","type":"journal-article","created":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T07:20:25Z","timestamp":1594192825000},"page":"691-720","update-policy":"http:\/\/dx.doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":2,"title":["Towards syntax-aware token embeddings"],"prefix":"10.1017","volume":"27","author":[{"given":"Diana Nicoleta","family":"Popa","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julien","family":"Perez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Henderson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric","family":"Gaussier","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2020,7,8]]},"reference":[{"key":"S1351324920000297_ref3","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00016"},{"key":"S1351324920000297_ref41","doi-asserted-by":"crossref","unstructured":"Mrk\u0161ic, N. , OS\u00e9aghdha, D. , Thomson, B. , Ga\u0161ic, M. , Rojas-Barahona, L. , Su, P.-H. , Vandyke, D. , Wen, T.-H. & Young, S. (2016). Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","DOI":"10.18653\/v1\/N16-1018"},{"key":"S1351324920000297_ref12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1070"},{"key":"S1351324920000297_ref22","doi-asserted-by":"crossref","unstructured":"Honnibal, M. and Johnson, M. (2015). An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.","DOI":"10.18653\/v1\/D15-1162"},{"key":"S1351324920000297_ref53","unstructured":"Socher, R. , Huval, B. , Manning, C.D. and Ng, A.Y. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning."},{"key":"S1351324920000297_ref46","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219855"},{"key":"S1351324920000297_ref51","unstructured":"Socher, R. , Bauer, J. , Manning, C.D , and Ng, A.Y. (2013a). Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics."},{"key":"S1351324920000297_ref11","unstructured":"Conneau, A. and Kiela, D. (2018). Senteval: an evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)."},{"key":"S1351324920000297_ref8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1075"},{"key":"S1351324920000297_ref20","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-2710-7"},{"key":"S1351324920000297_ref52","unstructured":"Socher, R. , Huang, E.H. , Pennington, J. , Ng, A.Y. and Manning, C.D. (2011). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of the 24th International Conference on Neural Information Processing Systems."},{"key":"S1351324920000297_ref64","unstructured":"Zou, W.Y. , Socher, R. , Cer, D. and Manning, C.D. (2013). Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing."},{"key":"S1351324920000297_ref58","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-2632"},{"key":"S1351324920000297_ref42","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1113"},{"key":"S1351324920000297_ref33","unstructured":"Liu, P. , Qiu, X. and Huang, X. (2015). Learning context-sensitive word embeddings with neural tensor skip-gram model. In Proceedings of the 24th International Conference on Artificial Intelligence."},{"key":"S1351324920000297_ref7","unstructured":"Bordes, A. , Usunier, N. , Garcia-Duran, A. , Weston, J. and Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26. Curran Associates Inc., pp. 2787\u20132795."},{"key":"S1351324920000297_ref30","unstructured":"Levy, O. and Goldberg, Y. (2014b). Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems 27. MIT Press, pp. 2177\u20132185."},{"key":"S1351324920000297_ref43","unstructured":"Nickel, M. , Tresp, V. and Kriegel, H.-P. (2011). A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning. Omnipress, pp. 809\u2013816."},{"key":"S1351324920000297_ref14","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies."},{"key":"S1351324920000297_ref54","unstructured":"Socher, R. , Manning, C.D. and Ng, A.Y. (2010). Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop."},{"key":"S1351324920000297_ref57","unstructured":"Trouillon, T. , Welbl, J. , Riedel, S. , Gaussier, \u00c9. and Bouchard, G. (2016). Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on International Conference on Machine Learning."},{"key":"S1351324920000297_ref1","unstructured":"Bahdanau, D. , Cho, K. and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. In Proceedings of the 2014 International Conference on Learning Representations."},{"key":"S1351324920000297_ref37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K16-1006"},{"key":"S1351324920000297_ref5","doi-asserted-by":"crossref","unstructured":"Bentivogli, L. , Bernardi, R. , Marelli, M. , Menini, S. , Baroni, M. and Zamparelli, R. (2016). Sick through the semeval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. Journal of Language Resources and Evaluation 50, 95\u2013124.","DOI":"10.1007\/s10579-015-9332-5"},{"key":"S1351324920000297_ref24","unstructured":"\u0130rsoy, O. and Cardie, C. (2014). Deep recursive neural networks for compositionality in language. In Proceedings of the 27th International Conference on Neural Information Processing Systems."},{"key":"S1351324920000297_ref38","unstructured":"Mikolov, T. , Chen, K. , Corrado, G.S. & Dean, J. (2013a). Efficient estimation of word representations in vector space. CoRR abs\/1301.3781."},{"key":"S1351324920000297_ref4","first-page":"1137","volume":"3","author":"Bengio","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324920000297_ref21","doi-asserted-by":"crossref","unstructured":"Henderson, J. (2003). Inducing history representations for broad coverage statistical parsing. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","DOI":"10.3115\/1073445.1073459"},{"key":"S1351324920000297_ref60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1006"},{"key":"S1351324920000297_ref35","doi-asserted-by":"publisher","DOI":"10.21236\/ADA273556"},{"key":"S1351324920000297_ref39","unstructured":"Mikolov, T. , Sutskever, I. , Chen, K. , Corrado, G.S. & Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. Curran Associates, Inc., pp. 3111\u20133119."},{"key":"S1351324920000297_ref27","unstructured":"Kingma, D. and Ba, J. (2014). Adam: a method for stochastic optimization. In Proceedings of the 2014 International Conference on Learning Representations."},{"key":"S1351324920000297_ref29","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2050"},{"key":"S1351324920000297_ref9","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1110"},{"key":"S1351324920000297_ref59","unstructured":"Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A.N. , Kaiser . and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 5998\u20136008."},{"key":"S1351324920000297_ref22a","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"S1351324920000297_ref2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2131"},{"key":"S1351324920000297_ref19","unstructured":"Grefenstette, E. and Sadrzadeh, M. (2011). Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing."},{"key":"S1351324920000297_ref17","unstructured":"Gehring, J. , Auli, M. , Grangier, D. , Yarats, D. and Dauphin, Y. (2017). Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning."},{"key":"S1351324920000297_ref28","unstructured":"Kiros, R. , Zhu, Y. , Salakhutdinov, R.R. , Zemel, R. , Urtasun, R. , Torralba, A. and Fidler, S. (2015). Skip-thought vectors. In Advances in Neural Information Processing Systems 28."},{"key":"S1351324920000297_ref56","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783307"},{"key":"S1351324920000297_ref47","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"S1351324920000297_ref44","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2007.33.2.161"},{"key":"S1351324920000297_ref32","doi-asserted-by":"crossref","unstructured":"Lin, D. (1998). Automatic retrieval and clustering of similar words. In Proceedings of the 17th International Conference on Computational Linguistics - Volume 2.","DOI":"10.3115\/980432.980696"},{"key":"S1351324920000297_ref18","unstructured":"Ghannay, S. , Favre, B. , Est\u00e8ve, Y. and Camelin, N. (2016). Word embedding evaluation and combination. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), Portoro\u017e, Slovenia, pp. 300\u2013305. European Language Resources Association (ELRA)."},{"key":"S1351324920000297_ref16","doi-asserted-by":"crossref","unstructured":"Faruqui, M. , Dodge, J. , Jauhar, S.K. , Dyer, C. , Hovy, E.H. and Smith, N.A. (2015). Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","DOI":"10.3115\/v1\/N15-1184"},{"key":"S1351324920000297_ref31","doi-asserted-by":"publisher","DOI":"10.3115\/1072228.1072378"},{"key":"S1351324920000297_ref63","unstructured":"Zhao, H. , Lu, Z. & Poupart, P. (2015). Self-adaptive hierarchical sentence model. In Proceedings of the 24th International Conference on Artificial Intelligence."},{"key":"S1351324920000297_ref50","doi-asserted-by":"crossref","unstructured":"Salant, S. and Berant, J. (2018). Contextualized word representations for reading comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","DOI":"10.18653\/v1\/N18-2088"},{"key":"S1351324920000297_ref62","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-0410"},{"key":"S1351324920000297_ref6","unstructured":"Blacoe, W. and Lapata, M. (2012). A comparison of vector-based representations for semantic composition. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning."},{"key":"S1351324920000297_ref25","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1062"},{"key":"S1351324920000297_ref40","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2010.01106.x"},{"key":"S1351324920000297_ref61","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00265"},{"key":"S1351324920000297_ref15","unstructured":"Dolan, B. and Brockett, C. (2005). Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing."},{"key":"S1351324920000297_ref34","unstructured":"Liu, Y. , Sun, C. , Lin, L. and Wang, X. (2016). Learning natural language inference using bidirectional lstm model and inner-attention. CoRR abs\/1605.09090."},{"key":"S1351324920000297_ref45","doi-asserted-by":"crossref","unstructured":"Pang, B. and Lee, L. (2004). A sentimental education: sentiment analysis using subjectivity. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics.","DOI":"10.3115\/1218955.1218990"},{"key":"S1351324920000297_ref26","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1181"},{"key":"S1351324920000297_ref23","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014073"},{"key":"S1351324920000297_ref36","unstructured":"McCann, B. , Bradbury, J. , Xiong, C. & Socher, R. (2017). Learned in translation: contextualized word vectors. In Advances in Neural Information Processing Systems 30, pp. 6294\u20136305."},{"key":"S1351324920000297_ref48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1161"},{"key":"S1351324920000297_ref49","doi-asserted-by":"crossref","unstructured":"Peters, M.E. , Neumann, M. , Iyyer, M. , Gardner, M. , Clark, C. , Lee, K. and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","DOI":"10.18653\/v1\/N18-1202"},{"key":"S1351324920000297_ref10","first-page":"2493","volume":"12","author":"Collobert","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324920000297_ref13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1191"},{"key":"S1351324920000297_ref55","unstructured":"Socher, R. , Perelygin, A. , Wu, J. , Chuang, J. , Manning, C.D. , Ng, A.Y. and Potts, C. (2013b). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324920000297","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T13:29:21Z","timestamp":1635254961000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324920000297\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,8]]},"references-count":65,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["S1351324920000297"],"URL":"https:\/\/doi.org\/10.1017\/s1351324920000297","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,8]]},"assertion":[{"value":"\u00a9 The Author(s), 2020. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}}]}}