{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T14:42:28Z","timestamp":1649342548442},"reference-count":65,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T00:00:00Z","timestamp":1649289600000},"content-version":"vor","delay-in-days":96,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We introduce the Probabilistic Worldbuilding Model (PWM), a new fully symbolic Bayesian model of semantic parsing and reasoning, as a first step in a research program toward more domain- and task-general NLU and AI. Humans create internal mental models of their observations that greatly aid in their ability to understand and reason about a large variety of problems. In PWM, the meanings of sentences, acquired facts about the world, and intermediate steps in reasoning are all expressed in a human-readable formal language, with the design goal of interpretability. PWM is Bayesian, designed specifically to be able to generalize to new domains and new tasks. We derive and implement an inference algorithm that reads sentences by parsing and abducing updates to its latent world model that capture the semantics of those sentences, and evaluate it on two out-of-domain question-answering datasets: (1) ProofWriter and (2) a new dataset we call FictionalGeoQA, designed to be more representative of real language but still simple enough to focus on evaluating reasoning ability, while being robust against heuristics. Our method outperforms baselines on both, thereby demonstrating its value as a proof-of-concept.<\/jats:p>","DOI":"10.1162\/tacl_a_00463","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T14:22:45Z","timestamp":1649341365000},"page":"325-342","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Towards General Natural Language Understanding with Probabilistic\n                    Worldbuilding"],"prefix":"10.1162","volume":"10","author":[{"given":"Abulhair","family":"Saparov","sequence":"first","affiliation":[{"name":"Machine Learning Department, Carnegie Mellon University, USA. asaparov@cs.cmu.edu"}]},{"given":"Tom M.","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Machine Learning Department, Carnegie Mellon University, USA. tom.mitchell@cs.cmu.edu"}]}],"member":"281","published-online":{"date-parts":[[2022,4,6]]},"reference":[{"key":"2022040714222641600_bib1","doi-asserted-by":"publisher","first-page":"242","DOI":"10.18653\/v1\/E17-2039","article-title":"The parallel meaning bank: Towards a multilingual corpus of\n                        translations annotated with compositional meaning\n                        representations","volume-title":"Proceedings of the 15th\n                        Conference of the European Chapter of the Association for Computational\n                        Linguistics, EACL","author":"Abzianidze","year":"2017"},{"key":"2022040714222641600_bib2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BFb0099421","article-title":"Exchangeability and related\n                        topics","volume-title":"Lecture Notes in Mathematics","author":"Aldous","year":"1985"},{"key":"2022040714222641600_bib3","article-title":"Complex query answering with neural link\n                        predictors","volume-title":"International Conference on Learning\n                        Representations","author":"Arakelyan","year":"2021"},{"issue":"2","key":"2022040714222641600_bib4","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1017\/S1471068413000689","article-title":"Structure learning of probabilistic logic\n                        programs by searching the clause space","volume":"15","author":"Bellodi","year":"2015","journal-title":"Theory and\n                        Practice of Logic Programming"},{"key":"2022040714222641600_bib5","doi-asserted-by":"publisher","first-page":"5185","DOI":"10.18653\/v1\/2020.acl-main.463","article-title":"Climbing towards NLU: On meaning, form,\n                        and understanding in the age of data","volume-title":"Proceedings\n                        of the 58th Annual Meeting of the Association for Computational Linguistics,\n                        ACL 2020, Online, July 5-10, 2020","author":"Bender","year":"2020"},{"key":"2022040714222641600_bib6","article-title":"Abductive commonsense reasoning","volume-title":"8th International Conference on Learning Representations, ICLR 2020,\n                        Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Bhagavatula","year":"2020"},{"key":"2022040714222641600_bib7","first-page":"301","article-title":"Open-domain semantic parsing with boxer","volume-title":"Proceedings of the 20th Nordic Conference of Computational\n                        Linguistics, NODALIDA","author":"Bos","year":"2015"},{"key":"2022040714222641600_bib8","article-title":"Language models are few-shot\n                        learners","author":"Brown","year":"2020","journal-title":"CoRR"},{"issue":"1","key":"2022040714222641600_bib9","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0004-3702(93)90060-O","article-title":"A Bayesian model of plan\n                        recognition","volume":"64","author":"Charniak","year":"1993","journal-title":"Artificial Intelligence"},{"issue":"2","key":"2022040714222641600_bib10","doi-asserted-by":"crossref","first-page":"56","DOI":"10.2307\/2266170","article-title":"A formulation of the simple theory of\n                        types","volume":"5","author":"Church","year":"1940","journal-title":"Journal of Symbolic Logic"},{"key":"2022040714222641600_bib11","first-page":"3882","article-title":"Transformers as soft reasoners over\n                        language","volume-title":"Proceedings of the Twenty-Ninth\n                        International Joint Conference on Artificial Intelligence, IJCAI","author":"Clark","year":"2020"},{"key":"2022040714222641600_bib12","doi-asserted-by":"publisher","DOI":"10.33011\/lilt.v10i.1357","article-title":"Probabilistic type theory and natural\n                        language semantics","volume-title":"Linguistic Issues in Language\n                        Technology","author":"Cooper","year":"2015"},{"issue":"4","key":"2022040714222641600_bib13","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1007\/s10994-020-05934-z","article-title":"Learning programs by learning from\n                        failures","volume":"110","author":"Cropper","year":"2021","journal-title":"Machine Learning"},{"issue":"3","key":"2022040714222641600_bib14","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1023\/A:1010924021315","article-title":"Parameter estimation in stochastic logic\n                        programs","volume":"44","author":"Cussens","year":"2001","journal-title":"Machine Learning"},{"key":"2022040714222641600_bib15","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language Technologies,\n                        NAACL-HLT","author":"Devlin","year":"2019"},{"key":"2022040714222641600_bib16","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-009-9065-4_1","volume-title":"Introduction to Montague Semantics","author":"Dowty","year":"1981","edition":"1st ed."},{"key":"2022040714222641600_bib17","first-page":"71","article-title":"From micro-worlds to knowledge\n                        representation: AI at an impasse","volume-title":"Readings in Knowledge Representation","author":"Dreyfus","year":"1985"},{"key":"2022040714222641600_bib18","doi-asserted-by":"publisher","first-page":"7839","DOI":"10.18653\/v1\/2020.acl-main.701","article-title":"To test machine comprehension, start by\n                        defining comprehension","volume-title":"Proceedings of the 58th\n                        Annual Meeting of the Association for Computational Linguistics, ACL 2020,\n                        Online, July 5\u201310, 2020","author":"Dunietz","year":"2020"},{"issue":"2","key":"2022040714222641600_bib19","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1214\/aos\/1176342360","article-title":"A Bayesian analysis of some nonparametric\n                        problems","volume":"1","author":"Ferguson","year":"1973","journal-title":"The Annals of Statistics"},{"key":"2022040714222641600_bib20","first-page":"47","article-title":"Tackling benchmark problems of commonsense\n                        reasoning","volume-title":"Proceedings of the Workshop on\n                        Bridging the Gap between Human and Automated Reasoning - A workshop of the\n                        25th International Conference on Automated Deduction (CADE-25), Berlin,\n                        Germany, August 1, 2015","author":"Furbach","year":"2015"},{"key":"2022040714222641600_bib21","doi-asserted-by":"publisher","first-page":"105","DOI":"10.18653\/v1\/D19-5815","article-title":"On making reading comprehension more\n                        comprehensive","volume-title":"Proceedings of the 2nd Workshop on\n                        Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China,\n                        November 4, 2019","author":"Gardner","year":"2019"},{"key":"2022040714222641600_bib22","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1007\/BF01201363","article-title":"Untersuchungen \u00fcber das logische\n                        schlie\u00dfen i","volume":"39","author":"Gentzen","year":"1935","journal-title":"Mathematische\n                        Zeitschrift"},{"key":"2022040714222641600_bib23","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1016\/S0049-237X(08)70822-X","article-title":"Investigations into Logical\n                        Deduction","volume-title":"The Collected Papers of Gerhard Gentzen","author":"Gentzen","year":"1969"},{"key":"2022040714222641600_bib24","doi-asserted-by":"crossref","DOI":"10.1515\/9780748691647","volume-title":"Language and Logics: An Introduction to the\n                        Logical Foundations of Language","author":"Gregory","year":"2015"},{"issue":"1","key":"2022040714222641600_bib25","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1093\/biomet\/57.1.97","article-title":"Monte Carlo sampling methods using markov\n                        chains and their applications","volume":"57","author":"Hastings","year":"1970","journal-title":"Biometrika"},{"issue":"2","key":"2022040714222641600_bib26","doi-asserted-by":"publisher","first-page":"81","DOI":"10.2307\/2266967","article-title":"Completeness in the theory of\n                        types","volume":"15","author":"Henkin","year":"1950","journal-title":"Journal of Symbolic Logic"},{"key":"2022040714222641600_bib27","doi-asserted-by":"crossref","DOI":"10.1002\/9780470756959.ch32","volume-title":"Abduction in Natural Language Understanding","author":"Hobbs","year":"2006"},{"issue":"1\u20132","key":"2022040714222641600_bib28","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/0004-3702(93)90015-4","article-title":"Interpretation as\n                        abduction","volume":"63","author":"Hobbs","year":"1993","journal-title":"Artificial Intelligence"},{"issue":"4","key":"2022040714222641600_bib29","doi-asserted-by":"publisher","first-page":"71:1\u201371:37","DOI":"10.1145\/3447772","article-title":"Knowledge graphs","volume":"54","author":"Hogan","year":"2021","journal-title":"ACM Computing Surveys"},{"key":"2022040714222641600_bib30","article-title":"Scalable rule learning in probabilistic\n                        knowledge bases","volume-title":"1st Conference on Automated\n                        Knowledge Base Construction, AKBC 2019, Amherst, MA, USA, May 20\u201322,\n                        2019","author":"Jain","year":"2019"},{"key":"2022040714222641600_bib31","first-page":"641","article-title":"Adaptor grammars: A framework for\n                        specifying compositional nonparametric bayesian models","volume-title":"Advances in Neural Information Processing Systems 19, Proceedings of\n                        the Twentieth Annual Conference on Neural Information Processing Systems,\n                        Vancouver, British Columbia, Canada, December 4\u20137, 2006","author":"Johnson","year":"2006"},{"key":"2022040714222641600_bib32","doi-asserted-by":"publisher","first-page":"1896","DOI":"10.18653\/v1\/2020.findings-emnlp.171","article-title":"Unifiedqa: Crossing format boundaries with\n                        a single QA system","volume-title":"Findings of the Association for\n                        Computational Linguistics: EMNLP 2020, Online Event, 16-20 November\n                        2020","author":"Khashabi","year":"2020"},{"issue":"1","key":"2022040714222641600_bib33","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s10462-018-9646-y","article-title":"40 years of cognitive architectures: Core\n                        cognitive abilities and practical applications","volume":"53","author":"Kotseruba","year":"2020","journal-title":"Artificial Intelligence Review"},{"key":"2022040714222641600_bib34","first-page":"1","article-title":"First-order theorem proving and\n                        vampire","volume-title":"Computer Aided Verification - 25th\n                        International Conference, CAV 2013, Saint Petersburg, Russia, July 13-19,\n                        2013. Proceedings","author":"Kov\u00e1cs","year":"2013"},{"issue":"1","key":"2022040714222641600_bib35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0004-3702(87)90050-6","article-title":"SOAR: An architecture for general\n                        intelligence","volume":"33","author":"Laird","year":"1987","journal-title":"Artificial Intelligence"},{"key":"2022040714222641600_bib36","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(87)90050-6","article-title":"Building machines that learn and think\n                        like people","author":"Lake","year":"2016","journal-title":"CoRR"},{"issue":"3","key":"2022040714222641600_bib37","doi-asserted-by":"publisher","first-page":"497","DOI":"10.2307\/1910129","article-title":"An automatic method of solving discrete programming\n                        problems","volume":"28","author":"Land","year":"1960","journal-title":"Econometrica"},{"key":"2022040714222641600_bib38","first-page":"573","article-title":"Type-based MCMC","volume-title":"Human Language Technologies: Conference of the North American\n                        Chapter of the Association of Computational Linguistics, Proceedings, June\n                        2-4, 2010, Los Angeles, California, USA","author":"Liang","year":"2010"},{"key":"2022040714222641600_bib39","doi-asserted-by":"publisher","first-page":"5210","DOI":"10.18653\/v1\/2020.acl-main.465","article-title":"How can we accelerate progress towards\n                        human-like linguistic generalization?","volume-title":"Proceedings\n                        of the 58th Annual Meeting of the Association for Computational Linguistics,\n                        ACL 2020, Online, July 5-10, 2020","author":"Linzen","year":"2020"},{"key":"2022040714222641600_bib40","article-title":"RoBERTa: A robustly optimized BERT\n                        pretraining approach","author":"Liu","year":"2019","journal-title":"CoRR"},{"key":"2022040714222641600_bib41","article-title":"SUMO: Unbiased estimation of log marginal probability for\n                        latent variable models","volume-title":"8th International\n                        Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,\n                        April 26\u201330, 2020","author":"Luo","year":"2020"},{"issue":"5","key":"2022040714222641600_bib42","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1145\/3191513","article-title":"Never-ending learning","volume":"61","author":"Mitchell","year":"2018","journal-title":"Communications of ACM"},{"issue":"4","key":"2022040714222641600_bib43","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/BF03037089","article-title":"Inductive logic\n                        programming","volume":"8","author":"Muggleton","year":"1991","journal-title":"New Generation Computing"},{"key":"2022040714222641600_bib44","first-page":"254","article-title":"Stochastic logic programs","volume-title":"Advances in Inductive Logic Programming","author":"Muggleton","year":"1996"},{"issue":"3","key":"2022040714222641600_bib45","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1145\/360018.360022","article-title":"Computer science as empirical inquiry: Symbols\n                        and search","volume":"19","author":"Newell","year":"1976","journal-title":"Commun. ACM"},{"key":"2022040714222641600_bib46","first-page":"632","article-title":"Learning and inference in tractable\n                        probabilistic knowledge bases","volume-title":"Proceedings of the\n                        Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI 2015,\n                        July 12-16, 2015, Amsterdam, The Netherlands","author":"Niepert","year":"2015"},{"key":"2022040714222641600_bib47","first-page":"1","article-title":"Towards distributed MCMC inference in\n                        probabilistic knowledge bases","volume-title":"Proceedings of the\n                        Joint Workshop on Automatic Knowledge Base Construction and Web-scale\n                        Knowledge Extraction, AKBC- WEKEX@NAACL-HLT 2012, Montr\u00e8al, Canada,\n                        June 7-8, 2012","author":"Niepert","year":"2012"},{"key":"2022040714222641600_bib48","volume-title":"Events in the Semantics of English","author":"Parsons","year":"1990"},{"key":"2022040714222641600_bib49","article-title":"Natural deduction","author":"Pfenning","year":"2004"},{"key":"2022040714222641600_bib50","first-page":"140:1","article-title":"Exploring the limits of transfer learning with a unified\n                        text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine\n                        Learning Research"},{"key":"2022040714222641600_bib51","article-title":"Query2box: Reasoning over knowledge graphs\n                        in vector space using box embeddings","volume-title":"8th\n                        International Conference on Learning Representations, ICLR 2020, Addis\n                        Ababa, Ethiopia, April 26-30, 2020","author":"Ren","year":"2020"},{"key":"2022040714222641600_bib52","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-4145-2","volume-title":"Monte Carlo Statistical Methods","author":"Robert","year":"2004"},{"key":"2022040714222641600_bib53","first-page":"3788","article-title":"End-to-end differentiable\n                        proving","volume-title":"Advances in Neural Information\n                        Processing Systems 30: Annual Conference on Neural Information Processing\n                        Systems 2017, December 4-9, 2017, Long Beach, CA, USA","author":"Rockt\u00e4schel","year":"2017"},{"key":"2022040714222641600_bib54","volume-title":"Artificial Intelligence - A Modern Approach,\n                        Third International Edition","author":"Russell","year":"2010"},{"key":"2022040714222641600_bib55","doi-asserted-by":"crossref","first-page":"122","DOI":"10.18653\/v1\/2020.emnlp-main.9","article-title":"Prover: Proof generation for interpretable\n                        reasoning over rules","volume-title":"Proceedings of the 2020\n                        Conference on Empirical Methods in Natural Language Processing,\n                        EMNLP","author":"Saha","year":"2020"},{"key":"2022040714222641600_bib56","doi-asserted-by":"publisher","first-page":"248","DOI":"10.18653\/v1\/K17-1026","article-title":"A probabilistic generative grammar for\n                        semantic parsing","volume-title":"Proceedings of the 21st\n                        Conference on Computational Natural Language Learning (CoNLL 2017),\n                        Vancouver, Canada, August 3\u20134, 2017","author":"Saparov","year":"2017"},{"key":"2022040714222641600_bib57","first-page":"847","article-title":"Generative modeling with failure in PRISM","volume-title":"IJCAI-05, Proceedings of the Nineteenth International Joint\n                        Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 -\n                        August 5, 2005","author":"Sato","year":"2005"},{"key":"2022040714222641600_bib58","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1007\/978-3-030-29436-6_29","article-title":"Faster, higher, stronger: E\n                        2.3","volume-title":"Automated Deduction - CADE 27 \u2013 27th\n                        International Conference on Automated Deduction, Natal, Brazil, August\n                        27-30, 2019, Proceedings","author":"Schulz","year":"2019"},{"key":"2022040714222641600_bib59","article-title":"Faithful embeddings for knowledge base\n                        queries","volume-title":"Advances in Neural Information\n                        Processing Systems 33: Annual Conference on Neural Information Processing\n                        Systems 2020, NeurIPS 2020, December 6-12, 2020,\n                    virtual","author":"Sun","year":"2020"},{"key":"2022040714222641600_bib60","doi-asserted-by":"publisher","first-page":"3621","DOI":"10.18653\/v1\/2021.findings-acl.317","article-title":"Proofwriter: Generating implications, proofs,\n                        and abductive statements over natural language","volume-title":"Findings of the Association for Computational Linguistics:\n                        ACL\/IJCNLP 2021, Online Event, August 1\u20136, 2021","author":"Tafjord","year":"2021"},{"key":"2022040714222641600_bib61","doi-asserted-by":"publisher","first-page":"6268","DOI":"10.18653\/v1\/2020.acl-main.559","article-title":"Language (re)modelling: Towards embodied\n                        language understanding","volume-title":"Proceedings of the 58th\n                        Annual Meeting of the Association for Computational Linguistics, ACL 2020,\n                        Online, July 5-10, 2020","author":"Tamari","year":"2020"},{"key":"2022040714222641600_bib62","article-title":"Wiktionary data\n                    dumps","author":"Foundation","year":"2020"},{"key":"2022040714222641600_bib63","first-page":"5754","article-title":"Xlnet: Generalized autoregressive pretraining for language\n                        understanding","volume-title":"Advances in Neural Information\n                        Processing Systems 32: Annual Conference on Neural Information Processing\n                        Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC,\n                        Canada","author":"Yang","year":"2019"},{"key":"2022040714222641600_bib64","article-title":"CLEVRER: Collision events for video\n                        representation and reasoning","volume-title":"8th International\n                        Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,\n                        April 26-30, 2020","author":"Yi","year":"2020"},{"key":"2022040714222641600_bib65","first-page":"1050","article-title":"Learning to parse database queries using\n                        inductive logic programming","volume-title":"Proceedings of the\n                        Thirteenth National Conference on Artificial Intelligence and Eighth\n                        Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI\n                        96, Portland, Oregon, USA, August 4\u20138, 1996","author":"Zelle","year":"1996"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00463\/2006984\/tacl_a_00463.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00463\/2006984\/tacl_a_00463.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T14:23:24Z","timestamp":1649341404000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00463\/110435\/Towards-General-Natural-Language-Understanding"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":65,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00463","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}