{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T08:31:37Z","timestamp":1790411497575,"version":"4.1.0"},"reference-count":48,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T00:00:00Z","timestamp":1564531200000},"content-version":"unspecified","delay-in-days":30,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2019,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We present two studies on neural network architectures that learn to represent sentences by composing their words according to automatically induced binary trees, without ever being shown a correct parse tree. We use Tree-Long Short-Term Memories (LSTMs) as our composition function, applied along a tree structure found by a differentiable natural language chart parser. The models simultaneously optimise both the composition function and the parser, thus eliminating the need for externally provided parse trees, which are normally required for Tree-LSTMs. They can therefore be seen as tree-based recurrent neural networks that are unsupervised with respect to the parse trees. Due to being fully differentiable, the models are easily trained with an off-the-shelf gradient descent method and backpropagation.<\/jats:p><jats:p>In the first part of this paper, we introduce a model based on the CKY chart parser, and evaluate its downstream performance on a natural language inference task and a reverse dictionary task. Further, we show how its performance can be improved with an attention mechanism which fully exploits the parse chart, by attending over all possible subspans of the sentence. We find that our approach is competitive against similar models of comparable size and outperforms Tree-LSTMs that use trees produced by a parser.<\/jats:p><jats:p>Finally, we present an alternative architecture based on a shift-reduce parser. We perform an analysis of the trees induced by both our models, to investigate whether they are consistent with each other and across re-runs, and whether they resemble the trees produced by a standard parser.<\/jats:p>","DOI":"10.1017\/s1351324919000184","type":"journal-article","created":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T11:33:09Z","timestamp":1564572789000},"page":"433-449","source":"Crossref","is-referenced-by-count":15,"title":["Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs"],"prefix":"10.1017","volume":"25","author":[{"given":"Jean","family":"Maillard","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Clark","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dani","family":"Yogatama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2019,7,31]]},"reference":[{"key":"S1351324919000184_ref22","unstructured":"Kingma D.P. and Ba J. (2015). Adam: A method for stochastic optimization. In ICLR."},{"key":"S1351324919000184_ref47","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(67)80007-X"},{"key":"S1351324919000184_ref20","volume-title":"An Efficient Recognition and Syntax Analysis Algorithm for Context-Free Languages","author":"Kasami","year":"1965"},{"key":"S1351324919000184_ref48","unstructured":"Zhu X. , Sobhani P. and Guo H. (2015). Long short-term memory over recursive structures. In ICML."},{"key":"S1351324919000184_ref46","unstructured":"Yogatama D. , Blunsom P. , Dyer C. , Grefenstette E. and Ling W. (2016). Learning to compose words into sentences with reinforcement learning."},{"key":"S1351324919000184_ref44","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00019"},{"key":"S1351324919000184_ref24","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"S1351324919000184_ref42","volume-title":"ACL-IJCNLP","author":"Tai","year":"2015"},{"key":"S1351324919000184_ref41","volume-title":"NIPS","author":"Sutskever","year":"2014"},{"key":"S1351324919000184_ref40","unstructured":"Sundermeyer M. , Schl\u00fcter R. and Ney H. (2012). LSTM neural networks for language modeling. In INTERSPEECH."},{"key":"S1351324919000184_ref19","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1062"},{"key":"S1351324919000184_ref38","volume-title":"The Syntactic Process","author":"Steedman","year":"2000"},{"key":"S1351324919000184_ref36","unstructured":"Socher R. , Bauer J. , Manning C.D. and Ng A.Y. (2013). Parsing with compositional vector grammars. In ACL."},{"key":"S1351324919000184_ref35","unstructured":"Shen T. , Zhou T. , Long G. , Jiang J. , Pan S. and Zhang C. (2018). DiSAN: Directional self-attention network for RNN\/CNN-free language understanding. In AAAI."},{"key":"S1351324919000184_ref6","doi-asserted-by":"crossref","DOI":"10.1515\/9783112316009","volume-title":"Syntactic Structures","author":"Chomsky","year":"1957"},{"key":"S1351324919000184_ref32","volume-title":"EMNLP","author":"Pennington","year":"2014"},{"key":"S1351324919000184_ref31","volume-title":"ACL","author":"Paperno","year":"2014"},{"key":"S1351324919000184_ref15","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Htut","year":"2018"},{"key":"S1351324919000184_ref3","first-page":"632","volume-title":"EMNLP","author":"Bowman","year":"2015"},{"key":"S1351324919000184_ref28","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"S1351324919000184_ref29","volume-title":"NAACL-HLT","author":"Mikolov","year":"2013"},{"key":"S1351324919000184_ref8","first-page":"345","article-title":"Mathematical foundations for a compositional distributed model of meaning","volume":"36","author":"Coecke","year":"2011","journal-title":"Linguistic Analysis"},{"key":"S1351324919000184_ref45","volume-title":"NAACL-HLT (Long Papers)","author":"Williams","year":"2018"},{"key":"S1351324919000184_ref14","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"S1351324919000184_ref10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1024"},{"key":"S1351324919000184_ref39","unstructured":"Subramanian S. , Trischler A. , Bengio Y. and Pal C.J. (2018). Learning general purpose distributed sentence representations via large scale multi-task learning. In ICLR."},{"key":"S1351324919000184_ref5","unstructured":"Choi J. , Yoo K.M. and Lee S. (2018). Learning to compose task-specific tree structures. In AAAI, pp. 5094\u20135101."},{"key":"S1351324919000184_ref4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1139"},{"key":"S1351324919000184_ref21","unstructured":"Kim Y. , Denton C. , Hoang L. and Rush A.M. (2017). Structured attention networks. In ICLR."},{"key":"S1351324919000184_ref26","volume-title":"ACL-IJCNLP","author":"Ma","year":"2015"},{"key":"S1351324919000184_ref34","unstructured":"Sha L. , Chang B. , Sui Z. and Li S. (2016). Reading and thinking: Re-read LSTM unit for textual entailment recognition. In COLING."},{"key":"S1351324919000184_ref18","unstructured":"Jozefowicz R. , Zaremba W. and Sutskever I. (2015). An empirical exploration of recurrent network architectures. J Machine Learning Research."},{"key":"S1351324919000184_ref9","unstructured":"Devlin J. , Chang M.-W. , Lee K. and Toutanova K. (2018). Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805"},{"key":"S1351324919000184_ref23","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1162\/tacl_a_00110","article-title":"Easy-first dependency parsing with hierarchical tree lstms","volume":"4","author":"Kiperwasser","year":"2016","journal-title":"TACL"},{"key":"S1351324919000184_ref2","unstructured":"Bengio Y. , L\u00e9onard N. and Courville A.C. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1305.2982."},{"key":"S1351324919000184_ref33","volume-title":"EMNLP","author":"Rush","year":"2015"},{"key":"S1351324919000184_ref11","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.06.042"},{"key":"S1351324919000184_ref12","volume-title":"NAACL-HLT (Short Papers)","author":"Gururangan","year":"2018"},{"key":"S1351324919000184_ref27","volume-title":"Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP","author":"Maillard","year":"2018"},{"key":"S1351324919000184_ref43","unstructured":"Vaswani A. , Shazeer N. , Parmar N. , Uszkoreit J. , Jones L. , Gomez A. N. , Kaiser L. and Polosukhin I. (2017). Attention is all you need. In NIPS."},{"key":"S1351324919000184_ref17","unstructured":"J\u00f3zefowicz R. , Vinyals O. , Schuster M. , Shazeer N. and Wu Y. (2016). Exploring the limits of language modeling. arXiv:1602.02410."},{"key":"S1351324919000184_ref1","unstructured":"Bahdanau D. , Cho K. and Bengio Y. (2015). Neural machine translation by jointly learning to align and translate. In ICLR, pp. 1\u201315."},{"key":"S1351324919000184_ref25","volume-title":"EMNLP","author":"Le","year":"2015"},{"key":"S1351324919000184_ref13","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00080"},{"key":"S1351324919000184_ref7","volume-title":"Programming Languages and Their Compilers: Preliminary Notes","author":"Cocke","year":"1969"},{"key":"S1351324919000184_ref37","volume-title":"EMNLP-CoNLL","author":"Socher","year":"2012"},{"key":"S1351324919000184_ref16","unstructured":"Jang E. , Gu S. and Poole B. (2017). Categorical reparameterization with gumbel-softmax."},{"key":"S1351324919000184_ref30","unstructured":"Neubig G. , Dyer C. , Goldberg Y. , Matthews A. , Ammar W. , Anastasopoulos A. , Ballesteros M. , Chiang D. , Clothiaux D. , Cohn T. , Duh K. , Faruqui M. , Gan C. , Garrette D. , Ji Y. , Kong L. , Kuncoro A. , Kumar G. , Malaviya C. , Michel P. , Oda Y. , Richardson M. , Saphra N. , Swayamdipta S. and Yin P. (2017). Dynet: The dynamic neural network toolkit. arXiv:1701.03980."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324919000184","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,15]],"date-time":"2021-01-15T15:29:25Z","timestamp":1610724565000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324919000184\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,7]]}},"alternative-id":["S1351324919000184"],"URL":"https:\/\/doi.org\/10.1017\/s1351324919000184","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7]]}}}