{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T14:54:28Z","timestamp":1782917668494,"version":"3.54.5"},"reference-count":30,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T00:00:00Z","timestamp":1739232000000},"content-version":"vor","delay-in-days":41,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,2,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Transformers trained on natural language data have been shown to exhibit hierarchical generalization without explicitly encoding any structural bias. In this work, we investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization. We extensively experiment with transformers trained on five synthetic, controlled datasets using several training objectives and show that, while objectives such as sequence-to-sequence modeling, classification, etc., often fail to lead to hierarchical generalization, the language modeling objective consistently leads to transformers generalizing hierarchically. We then study how different generalization behaviors emerge during the training by conducting pruning experiments that reveal the joint existence of subnetworks within the model implementing different generalizations. Finally, we take a Bayesian perspective to understand transformers\u2019 preference for hierarchical generalization: We establish a correlation between whether transformers generalize hierarchically on a dataset and if the simplest explanation of that dataset is provided by a hierarchical grammar compared to regular grammars exhibiting linear generalization. Overall, our work presents new insights on the origins of hierarchical generalization in transformers and provides a theoretical framework for studying generalization in language models.<\/jats:p>","DOI":"10.1162\/tacl_a_00733","type":"journal-article","created":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T19:34:41Z","timestamp":1739302481000},"page":"121-141","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers"],"prefix":"10.1162","volume":"13","author":[{"given":"Kabir","family":"Ahuja","sequence":"first","affiliation":[{"name":"University of Washington, USA. kahuja@cs.washington.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vidhisha","family":"Balachandran","sequence":"additional","affiliation":[{"name":"Microsoft Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madhur","family":"Panwar","sequence":"additional","affiliation":[{"name":"EPFL, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianxing","family":"He","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noah A.","family":"Smith","sequence":"additional","affiliation":[{"name":"University of Washington, USA"},{"name":"Allen Institute for AI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Navin","family":"Goyal","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yulia","family":"Tsvetkov","sequence":"additional","affiliation":[{"name":"University of Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2025,2,12]]},"reference":[{"key":"2025051914252070200_bib1","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/S0022-0000(69)80006-1","article-title":"Syntax directed translations and the pushdown assembler","volume":"3","author":"Aho","year":"1969","journal-title":"Journal of Computer and System Sciences"},{"key":"2025051914252070200_bib2","article-title":"Physics of Language Models: Part 1, Learning Hierarchical Language Structures","volume":"abs\/2305.13673","author":"Allen-Zhu","year":"2023","journal-title":"ArXiv e-prints"},{"key":"2025051914252070200_bib3","doi-asserted-by":"publisher","DOI":"10.1121\/1.2017061","article-title":"Trainable grammars for speech recognition","volume":"65","author":"Baker","year":"1979","journal-title":"Journal of the Acoustical Society of America"},{"key":"2025051914252070200_bib4","first-page":"1877","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2025051914252070200_bib5","article-title":"Random deep neural networks are biased towards simple functions","volume":"32","author":"De Palma","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025051914252070200_bib6","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2025051914252070200_bib7","volume-title":"Unified language model pre-training for natural language understanding and generation","author":"Li","year":"2019"},{"key":"2025051914252070200_bib8","article-title":"Transformational networks","volume":"22","author":"Frank","year":"2007","journal-title":"Models of Human Language Acquisition"},{"key":"2025051914252070200_bib9","article-title":"Adam: A method for stochastic optimization","volume-title":"3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings","author":"Kingma","year":"2015"},{"key":"2025051914252070200_bib10","doi-asserted-by":"crossref","first-page":"241","DOI":"10.18653\/v1\/W19-4825","article-title":"Open sesame: Getting inside BERT\u2019s linguistic knowledge","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Lin","year":"2019"},{"key":"2025051914252070200_bib11","article-title":"Learning sparse neural networks through L__0 regularization","volume-title":"International Conference on Learning Representations","author":"Louizos","year":"2018"},{"key":"2025051914252070200_bib12","first-page":"8","article-title":"The Philosophy and the Approach","volume-title":"Vision","author":"Marr","year":"1982"},{"key":"2025051914252070200_bib13","first-page":"2096","article-title":"Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks","volume-title":"Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018","author":"Thomas McCoy","year":"2018"},{"key":"2025051914252070200_bib14","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1162\/tacl_a_00304","article-title":"Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks","volume":"8","author":"Thomas McCoy","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2025051914252070200_bib15","article-title":"A tale of two circuits: Grokking as competition of sparse and dense subnetworks","volume-title":"ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models","author":"Merrill","year":"2023"},{"key":"2025051914252070200_bib16","doi-asserted-by":"publisher","first-page":"1352","DOI":"10.18653\/v1\/2022.findings-acl.106","article-title":"Coloring the blank slate: Pre-training imparts a hierarchical inductive bias to sequence-to-sequence models","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Mueller","year":"2022"},{"key":"2025051914252070200_bib17","doi-asserted-by":"publisher","first-page":"4761","DOI":"10.18653\/v1\/2024.naacl-long.267","article-title":"In-context learning generalizes, but not always robustly: The case of syntax","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Mueller","year":"2024"},{"key":"2025051914252070200_bib18","article-title":"Transformers can do Bayesian inference","volume-title":"International Conference on Learning Representations","author":"M\u00fcller","year":"2022"},{"key":"2025051914252070200_bib19","doi-asserted-by":"publisher","first-page":"439","DOI":"10.18653\/v1\/2023.acl-short.38","article-title":"Grokking of hierarchical structure in vanilla transformers","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Murty","year":"2023"},{"issue":"3","key":"2025051914252070200_bib20","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1016\/j.cognition.2010.11.001","article-title":"The learnability of abstract syntactic principles","volume":"118","author":"Perfors","year":"2011","journal-title":"Cognition"},{"key":"2025051914252070200_bib21","doi-asserted-by":"publisher","first-page":"1499","DOI":"10.18653\/v1\/D18-1179","article-title":"Dissecting contextual word embeddings: Architecture and representation","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Peters","year":"2018"},{"key":"2025051914252070200_bib22","article-title":"Transformers generalize linearly","author":"Petty","year":"2021"},{"issue":"1","key":"2025051914252070200_bib23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/S0019-9958(64)90223-2","article-title":"A formal theory of inductive inference. Part I","volume":"7","author":"Solomonoff","year":"1964","journal-title":"Information and Control"},{"key":"2025051914252070200_bib24","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1007\/3-540-58473-0_141","article-title":"Inducing probabilistic grammars by Bayesian model merging","volume-title":"International Colloquium on Grammatical Inference","author":"Stolcke","year":"1994"},{"key":"2025051914252070200_bib25","article-title":"Sequence to sequence learning with neural networks","volume":"27","author":"Sutskever","year":"2014","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025051914252070200_bib26","article-title":"What do you learn from context? Probing for sentence structure in contextualized word representations","volume-title":"International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2025051914252070200_bib27","article-title":"Deep learning generalizes because the parameter-function map is biased towards simple functions","volume-title":"International Conference on Learning Representations","author":"Valle-Perez","year":"2019"},{"key":"2025051914252070200_bib28","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2025051914252070200_bib29","doi-asserted-by":"publisher","first-page":"5797","DOI":"10.18653\/v1\/P19-1580","article-title":"Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Voita","year":"2019"},{"key":"2025051914252070200_bib30","doi-asserted-by":"publisher","first-page":"4166","DOI":"10.18653\/v1\/2020.acl-main.383","article-title":"Perturbed masking: Parameter-free probing for analyzing and interpreting BERT","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Zhiyong","year":"2020"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00733\/2501791\/tacl_a_00733.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00733\/2501791\/tacl_a_00733.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T18:25:32Z","timestamp":1747679132000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00733\/127877\/Learning-Syntax-Without-Planting-Trees"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025]]},"references-count":30,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00733","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025]]},"published":{"date-parts":[[2025]]}}}