{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,8]],"date-time":"2024-09-08T17:10:20Z","timestamp":1725815420634},"reference-count":46,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T00:00:00Z","timestamp":1629331200000},"content-version":"vor","delay-in-days":230,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,18]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We present a new conjunctivist framework, neural event semantics (NES), for compositional grounded language understanding. Our approach treats all words as classifiers that compose to form a sentence meaning by multiplying output scores. These classifiers apply to spatial regions (events) and NES derives its semantic structure from language by routing events to different classifier argument inputs via soft attention. NES is trainable end-to-end by gradient descent with minimal supervision. We evaluate our method on compositional grounded language tasks in controlled synthetic and real-world settings. NES offers stronger generalization capability than standard function-based compositional frameworks, while improving accuracy over state-of-the-art neural methods on real-world language tasks.<\/jats:p>","DOI":"10.1162\/tacl_a_00402","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T19:30:56Z","timestamp":1632166256000},"page":"875-890","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["Neural Event Semantics for Grounded Language Understanding"],"prefix":"10.1162","volume":"9","author":[{"given":"Shyamal","family":"Buch","sequence":"first","affiliation":[{"name":"Stanford University, United States. shyamal@cs.stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Li","family":"Fei-Fei","sequence":"additional","affiliation":[{"name":"Stanford University, United States. feifeili@cs.stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noah D.","family":"Goodman","sequence":"additional","affiliation":[{"name":"Stanford University, United States. ngoodman@stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,8,18]]},"reference":[{"key":"2021081914590321100_bib1","doi-asserted-by":"publisher","first-page":"8938","DOI":"10.1109\/ICCV.2019.00903","article-title":"ShapeGlot: Learning language for shape differentiation","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Achlioptas","year":"2019"},{"key":"2021081914590321100_bib2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1181","article-title":"Learning to compose neural networks for question answering","author":"Andreas","year":"2016","journal-title":"NAACL"},{"key":"2021081914590321100_bib3","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1109\/CVPR.2016.12","article-title":"Neural module networks","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Andreas","year":"2016"},{"key":"2021081914590321100_bib4","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.279","article-title":"VQA: Visual question answering","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Antol","year":"2015"},{"key":"2021081914590321100_bib5","article-title":"Systematic generalization: What is required and can it be learned?","author":"Bahdanau","year":"2019","journal-title":"ICLR"},{"key":"2021081914590321100_bib6","article-title":"CLOSURE: Assessing systematic generalization of CLEVR models","author":"Bahdanau","year":"2019","journal-title":"arXiv preprint arXiv:1912.05783"},{"key":"2021081914590321100_bib7","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00361","article-title":"Latent compositional representations improve systematic generalization in grounded question answering","author":"Bogin","year":"2020","journal-title":"arXiv preprint arXiv:2007.00266"},{"key":"2021081914590321100_bib8","article-title":"ShapeNet: An information-rich 3D model repository","author":"Chang","year":"2015","journal-title":"arXiv preprint arXiv:1512.03012"},{"key":"2021081914590321100_bib9","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11832","article-title":"Gated-attention architectures for task-oriented language grounding","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence","author":"Chaplot","year":"2018"},{"key":"2021081914590321100_bib10","article-title":"Microsoft COCO captions: Data collection and evaluation server","author":"Chen","year":"2015","journal-title":"arXiv preprint arXiv: 1504.00325"},{"key":"2021081914590321100_bib11","first-page":"1240","article-title":"Resources for building applications with dependency minimal recursion semantics","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Copestake","year":"2016"},{"key":"2021081914590321100_bib12","doi-asserted-by":"crossref","first-page":"81","DOI":"10.2307\/jj.13027259.6","article-title":"The logical form of action sentences","volume-title":"The Logic of Decision and Action","author":"Davidson","year":"1967"},{"issue":"5\u20136","key":"2021081914590321100_bib13","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Networks"},{"key":"2021081914590321100_bib14","first-page":"2961","article-title":"Mask R-CNN","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"He","year":"2017"},{"key":"2021081914590321100_bib15","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2021081914590321100_bib16","first-page":"53","article-title":"Explainable neural computation via stack neural module networks","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Hu","year":"2018"},{"key":"2021081914590321100_bib17","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV.2017.93","article-title":"Learning to reason: End-to-end module networks for visual question answering","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Hu","year":"2017"},{"key":"2021081914590321100_bib18","article-title":"Compositional attention networks for machine reasoning","volume-title":"International Conference on Learning Representations (ICLR)","author":"Hudson","year":"2018"},{"key":"2021081914590321100_bib19","doi-asserted-by":"publisher","first-page":"6700","DOI":"10.1109\/CVPR.2019.00686","article-title":"GQA: A new dataset for real-world visual reasoning and compositional question answering","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Hudson","year":"2019"},{"key":"2021081914590321100_bib20","article-title":"Transfer learning in visual and relational reasoning","author":"Jayram","year":"2019","journal-title":"arXiv preprint arXiv:1911.11938"},{"key":"2021081914590321100_bib21","doi-asserted-by":"publisher","first-page":"1988","DOI":"10.1109\/CVPR.2017.215","article-title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Johnson","year":"2017"},{"key":"2021081914590321100_bib22","doi-asserted-by":"publisher","first-page":"3008","DOI":"10.1109\/ICCV.2017.325","article-title":"Inferring and executing programs for visual reasoning","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Johnson","year":"2017"},{"key":"2021081914590321100_bib23","doi-asserted-by":"publisher","first-page":"3128","DOI":"10.1109\/CVPR.2015.7298932","article-title":"Deep visual-semantic alignments for generating image descriptions","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Karpathy","year":"2015"},{"key":"2021081914590321100_bib24","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1162\/tacl_a_00220","article-title":"Jointly learning to parse and perceive: Connecting natural language to the physical world","volume":"1","author":"Krishnamurthy","year":"2013","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021081914590321100_bib25","doi-asserted-by":"publisher","first-page":"160","DOI":"10.18653\/v1\/D16-1016","article-title":"Semantic parsing to probabilistic programs for situated question answering","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Krishnamurthy","year":"2016"},{"key":"2021081914590321100_bib26","article-title":"ShapeWorld - A new test methodology for multimodal language understanding","author":"Kuhnle","year":"2017","journal-title":"CoRR"},{"key":"2021081914590321100_bib27","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X16001837","article-title":"Building machines that learn and think like people","volume":"40","author":"Lake","year":"2017","journal-title":"Behavioral and Brain Sciences"},{"key":"2021081914590321100_bib28","first-page":"1682","article-title":"A multi-world approach to question answering about real-world scenes based on uncertain input","volume":"27","author":"Malinowski","year":"2014","journal-title":"Advances in neural information processing systems"},{"key":"2021081914590321100_bib29","article-title":"The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision","volume-title":"International Conference on Learning Representations","author":"Mao","year":"2019"},{"key":"2021081914590321100_bib30","article-title":"On transfer learning using a MAC model variant","author":"Marois","year":"2018","journal-title":"arXiv preprint arXiv:1811.06529"},{"key":"2021081914590321100_bib31","article-title":"A joint model of language and perception for grounded attribute learning","author":"Matuszek","year":"2012","journal-title":"International Conference on Machine Learning"},{"key":"2021081914590321100_bib32","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1162\/tacl_a_00064","article-title":"Colors in Context: A pragmatic neural model for grounded language understanding","volume":"5","author":"Monroe","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"3","key":"2021081914590321100_bib33","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1111\/j.1755-2567.1970.tb00434.x","article-title":"Universal grammar","volume":"36","author":"Montague","year":"1970","journal-title":"Theoria"},{"key":"2021081914590321100_bib34","first-page":"8024","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume-title":"Advances in Neural Information Processing Systems 32","author":"Paszke","year":"2019"},{"key":"2021081914590321100_bib35","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Pennington","year":"2014"},{"key":"2021081914590321100_bib36","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11671","article-title":"FiLM: Visual reasoning with a general conditioning layer","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence","author":"Perez","year":"2018"},{"volume-title":"Events and Semantic Architecture","year":"2005","author":"Pietroski","key":"2021081914590321100_bib37"},{"key":"2021081914590321100_bib38","article-title":"A benchmark for systematic generalization in grounded language understanding","author":"Ruis","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2021081914590321100_bib39","article-title":"Very deep convolutional networks for large-scale image recognition","volume-title":"International Conference on Learning Representations","author":"Simonyan","year":"2015"},{"key":"2021081914590321100_bib40","doi-asserted-by":"publisher","first-page":"5594","DOI":"10.18653\/v1\/2020.acl-main.495","article-title":"Obtaining faithful interpretations from compositional neural networks","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Subramanian","year":"2020"},{"key":"2021081914590321100_bib41","doi-asserted-by":"publisher","first-page":"6418","DOI":"10.18653\/v1\/P19-1644","article-title":"A Corpus for reasoning about natural language grounded in photographs","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Suhr","year":"2019"},{"key":"2021081914590321100_bib42","first-page":"806","article-title":"Learning to follow navigational directions","volume-title":"Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics","author":"Vogel","year":"2010"},{"issue":"3\u20134","key":"2021081914590321100_bib43","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Machine learning"},{"key":"2021081914590321100_bib44","first-page":"1039","article-title":"Neural-symbolic VQA: Disentangling reasoning from vision and language understanding","volume-title":"Advances in Neural Information Processing Systems","author":"Yi","year":"2018"},{"key":"2021081914590321100_bib45","article-title":"Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars","author":"Zettlemoyer","year":"2005","journal-title":"UAI \u201905, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence"},{"key":"2021081914590321100_bib46","doi-asserted-by":"publisher","first-page":"6578","DOI":"10.1109\/CVPR.2019.00674","article-title":"Grounded video description","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Zhou","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00402\/1957718\/tacl_a_00402.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00402\/1957718\/tacl_a_00402.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,8]],"date-time":"2024-09-08T16:16:08Z","timestamp":1725812168000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00402\/106993\/Neural-Event-Semantics-for-Grounded-Language"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":46,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00402","relation":{},"ISSN":["2307-387X"],"issn-type":[{"type":"electronic","value":"2307-387X"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}