{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T13:45:44Z","timestamp":1778852744122,"version":"3.51.4"},"reference-count":40,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T00:00:00Z","timestamp":1703116800000},"content-version":"vor","delay-in-days":354,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,12,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Compositional generalization benchmarks for semantic parsing seek to assess whether models can accurately compute meanings for novel sentences, but operationalize this in terms of logical form (LF) prediction. This raises the concern that semantically irrelevant details of the chosen LFs could shape model performance. We argue that this concern is realized for the COGS benchmark (Kim and Linzen, 2020). COGS poses generalization splits that appear impossible for present-day models, which could be taken as an indictment of those models. However, we show that the negative results trace to incidental features of COGS LFs. Converting these LFs to semantically equivalent ones and factoring out capabilities unrelated to semantic interpretation, we find that even baseline models get traction. A recent variable-free translation of COGS LFs suggests similar conclusions, but we observe this format is not semantically equivalent; it is incapable of accurately representing some COGS meanings. These findings inform our proposal for ReCOGS, a modified version of COGS that comes closer to assessing the target semantic capabilities while remaining very challenging. Overall, our results reaffirm the importance of compositional generalization and careful benchmark task design.<\/jats:p>","DOI":"10.1162\/tacl_a_00623","type":"journal-article","created":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T20:20:03Z","timestamp":1703190003000},"page":"1719-1733","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation"],"prefix":"10.1162","volume":"11","author":[{"given":"Zhengxuan","family":"Wu","sequence":"first","affiliation":[{"name":"Stanford University, USA. wuzhengx@stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher D.","family":"Manning","sequence":"additional","affiliation":[{"name":"Stanford University, USA. manning@stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher","family":"Potts","sequence":"additional","affiliation":[{"name":"Stanford University, USA. cgpotts@stanford.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,12,21]]},"reference":[{"key":"2023122120195508100_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.382","article-title":"Lexicon learning for few shot sequence modeling","volume-title":"Association for Computational Linguistics (ACL)","author":"Akyurek","year":"2021"},{"key":"2023122120195508100_bib2","article-title":"Abstract Meaning Representation for sembanking","volume-title":"Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse","author":"Banarescu","year":"2013"},{"key":"2023122120195508100_bib3","article-title":"Systematic generalization with edge transformers","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Bergen","year":"2021"},{"key":"2023122120195508100_bib4","article-title":"Smatch: An evaluation metric for semantic feature structures","volume-title":"Association for Computational Linguistics (ACL)","author":"Cai","year":"2013"},{"key":"2023122120195508100_bib5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.258","article-title":"Meta-learning to compositionally generalize","volume-title":"Association for Computational Linguistics (ACL)","author":"Conklin","year":"2021"},{"key":"2023122120195508100_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.49","article-title":"The devil is in the detail: Simple tricks improve systematic generalization of transformers","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Csord\u00e1s","year":"2021"},{"key":"2023122120195508100_bib7","volume-title":"Combinatory Logic","author":"Curry","year":"1958"},{"key":"2023122120195508100_bib8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2209.15003","article-title":"Compositional semantic parsing with large language models","volume-title":"International Conference on Learning Representations (ICLR)","author":"Drozdov","year":"2023"},{"key":"2023122120195508100_bib9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1456","article-title":"Posing fair generalization tasks for natural language inference","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Geiger","year":"2019"},{"key":"2023122120195508100_bib10","unstructured":"Irene\n              Heim\n            \n          . 1982. The Semantics of Definite and Indefinite Noun Phrases. Ph.D. thesis, University of Massachusetts."},{"key":"2023122120195508100_bib11","doi-asserted-by":"publisher","DOI":"10.1515\/9783110852820.164","article-title":"File change semantics and the familiarity theory of definiteness","volume-title":"Meaning, Use, and Interpretation of Language","author":"Heim","year":"1983"},{"key":"2023122120195508100_bib12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.74","article-title":"Span-based semantic parsing for compositional generalization","volume-title":"Association for Computational Linguistics (ACL)","author":"Herzig","year":"2021"},{"key":"2023122120195508100_bib13","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","author":"Hochreiter","year":"1997","journal-title":"Neural computation"},{"key":"2023122120195508100_bib14","doi-asserted-by":"publisher","DOI":"10.1016\/B978-044481714-3\/50011-4","article-title":"Compositionality","volume-title":"Handbook of Logic and Language","author":"Janssen","year":"1997"},{"key":"2023122120195508100_bib15","article-title":"A theory of truth and semantic representation","volume-title":"Formal Methods in the Study of Language","author":"Kamp","year":"1981"},{"key":"2023122120195508100_bib16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1912.09713","article-title":"Measuring compositional generalization: A comprehensive method on realistic data","volume-title":"International Conference on Learning Representations (ICLR)","author":"Keysers","year":"2020"},{"key":"2023122120195508100_bib17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.731","article-title":"COGS: A compositional generalization challenge based on semantic interpretation","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Kim","year":"2020"},{"key":"2023122120195508100_bib18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.10769","article-title":"Uncontrolled lexical exposure leads to overestimation of compositional generalization in pretrained models","author":"Kim","year":"2022","journal-title":"arXiv preprint arXiv:2212.10769"},{"key":"2023122120195508100_bib19","article-title":"Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks","volume-title":"International Conference on Machine Learning (ICML)","author":"Lake","year":"2018"},{"key":"2023122120195508100_bib20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Association for Computational Linguistics (ACL)","author":"Lewis","year":"2020"},{"key":"2023122120195508100_bib21","article-title":"Lexical insertion in a transformational grammar without deep structure","volume-title":"Papers from the Fourth Meeting of the Chicago Linguistic Society","author":"McCawley","year":"1968"},{"key":"2023122120195508100_bib22","article-title":"Formal philosophy: Selected papers of Richard Montague","author":"Montague","year":"1974","journal-title":"Erkenntnis"},{"key":"2023122120195508100_bib23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.251","article-title":"Making transformers solve compositional tasks","volume-title":"Association for Computational Linguistics (ACL)","author":"Ontanon","year":"2022"},{"key":"2023122120195508100_bib24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.225","article-title":"Improving compositional generalization in semantic parsing","volume-title":"Findings of Empirical Methods in Natural Language Processing (EMNLP)","author":"Oren","year":"2020"},{"key":"2023122120195508100_bib25","volume-title":"Events in the semantics of english: A study in subatomic semantics","author":"Parsons","year":"1990"},{"key":"2023122120195508100_bib26","article-title":"Compositionality","volume-title":"Varieties of Formal Semantics","author":"Partee","year":"1984"},{"key":"2023122120195508100_bib27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-short.46","article-title":"Revisiting the compositional generalization abilities of neural sequence models","volume-title":"Association for Computational Linguistics (ACL)","author":"Patel","year":"2022"},{"key":"2023122120195508100_bib28","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.323","article-title":"Improving compositional generalization with latent structure and data augmentation","volume-title":"North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)","author":"Qiu","year":"2022"},{"key":"2023122120195508100_bib29","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume-title":"Journal of Machine Learning Research (JMLR)","author":"Raffel","year":"2020"},{"key":"2023122120195508100_bib30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1009","article-title":"Universal semantic parsing","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Reddy","year":"2017"},{"key":"2023122120195508100_bib31","article-title":"A benchmark for systematic generalization in grounded language understanding","author":"Ruis","year":"2020","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"2023122120195508100_bib32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.41","article-title":"When can transformers ground and compose: Insights from compositional generalization benchmarks","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Sikarwar","year":"2022"},{"key":"2023122120195508100_bib33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.333","article-title":"Language model acceptability judgements are not always robust to context","volume-title":"Association for Computational Linguistics (ACL)","author":"Sinha","year":"2023"},{"key":"2023122120195508100_bib34","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Vaswani","year":"2017"},{"key":"2023122120195508100_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.563","article-title":"Hierarchical phrase-based sequence-to-sequence learning","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Wang","year":"2022"},{"key":"2023122120195508100_bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.starsem-1.4","article-title":"Compositional generalization with a broad-coverage semantic parser","volume-title":"Proceedings of the 11th Joint Conference on Lexical and Computational Semantics","author":"Wei\u00dfenhorn","year":"2022"},{"key":"2023122120195508100_bib37","article-title":"ReaSCAN: Compositional reasoning in language grounding","volume-title":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Zhengxuan","year":"2021"},{"key":"2023122120195508100_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.337","article-title":"Structural generalization is hard for sequence-to-sequence models","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Yao","year":"2022"},{"key":"2023122120195508100_bib39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.88","article-title":"Compositional generalization via semantic tagging","volume-title":"Findings of Empirical Methods in Natural Language Processing (EMNLP)","author":"Zheng","year":"2021"},{"key":"2023122120195508100_bib40","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.293","article-title":"Disentangled sequence to sequence learning for compositional generalization","volume-title":"Association for Computational Linguistics (ACL)","author":"Zheng","year":"2022"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00623\/2200660\/tacl_a_00623.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00623\/2200660\/tacl_a_00623.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T20:20:14Z","timestamp":1703190014000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00623\/118855\/ReCOGS-How-Incidental-Details-of-a-Logical-Form"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":40,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00623","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}