{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T14:52:43Z","timestamp":1782917563705,"version":"3.54.5"},"reference-count":52,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":336,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpora that target a wide range of linguistic phenomena. Our results show that while transformers are better qua LMs (as measured by perplexity), both models perform equally and surprisingly well on linguistic generalization measures, suggesting that they are capable of generalizing from indirect evidence.<\/jats:p>","DOI":"10.1162\/tacl_a_00720","type":"journal-article","created":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T16:32:07Z","timestamp":1733157127000},"page":"1597-1615","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence"],"prefix":"10.1162","volume":"12","author":[{"given":"Abhinav","family":"Patil","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaap","family":"Jumelet","sequence":"additional","affiliation":[{"name":"ILLC, University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu Ying","family":"Chiu","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andy","family":"Lapastora","sequence":"additional","affiliation":[{"name":"AWS AI Labs, Amazon, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Shen","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lexie","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clevis","family":"Willrich","sequence":"additional","affiliation":[{"name":"Microsoft, Redmond, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shane","family":"Steinert-Threlkeld","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,11,27]]},"reference":[{"issue":"1","key":"2024120216320216100_bib1","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1002\/leap.1210","article-title":"How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship","volume":"32","author":"Allen","year":"2019","journal-title":"Learned Publishing"},{"key":"2024120216320216100_bib2","doi-asserted-by":"publisher","first-page":"301","DOI":"10.18653\/v1\/2022.conll-1.20","article-title":"Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities","volume-title":"Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)","author":"Arehalli","year":"2022"},{"issue":"6","key":"2024120216320216100_bib3","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1007\/s10988-012-9106-0","article-title":"Modularity and intuitions in formal semantics: The case of polarity items","volume":"34","author":"Chemla","year":"2011","journal-title":"Linguistics and Philosophy"},{"key":"2024120216320216100_bib4","doi-asserted-by":"publisher","DOI":"10.1515\/9783110884166","volume-title":"Lectures on Government and Binding","author":"Chomsky","year":"1993"},{"key":"2024120216320216100_bib5","doi-asserted-by":"publisher","first-page":"104791","DOI":"10.1016\/j.cognition.2021.104791","article-title":"The influence of polarity items on inferential judgments","volume":"215","author":"Deni\u0107","year":"2021","journal-title":"Cognition"},{"key":"2024120216320216100_bib6","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1162\/tacl_a_00298","article-title":"What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models","volume":"8","author":"Ettinger","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024120216320216100_bib7","first-page":"61","article-title":"Sequential vs. hierarchical syntactic models of human incremental sentence processing","volume-title":"Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012)","author":"Fossum","year":"2012"},{"key":"2024120216320216100_bib8","doi-asserted-by":"publisher","first-page":"32","DOI":"10.18653\/v1\/N19-1004","article-title":"Neural language models as psycholinguistic subjects: Representations of syntactic state","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Futrell","year":"2019"},{"key":"2024120216320216100_bib9","doi-asserted-by":"publisher","first-page":"70","DOI":"10.18653\/v1\/2020.acl-demos.10","article-title":"SyntaxGym: An online platform for targeted evaluation of language models","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Gauthier","year":"2020"},{"key":"2024120216320216100_bib10","doi-asserted-by":"publisher","first-page":"10","DOI":"10.18653\/v1\/W18-0102","article-title":"Predictive power of word surprisal for reading times is a linear function of language model quality","volume-title":"Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018)","author":"Goodkind","year":"2018"},{"key":"2024120216320216100_bib11","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.18653\/v1\/N18-1108","article-title":"Colorless green recurrent networks dream hierarchically","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Gulordava","year":"2018"},{"key":"2024120216320216100_bib12","doi-asserted-by":"publisher","DOI":"10.3115\/1073336.1073357","article-title":"A probabilistic Earley parser as a psycholinguistic model","volume-title":"Second Meeting of the North American Chapter of the Association for Computational Linguistics","author":"Hale","year":"2001"},{"key":"2024120216320216100_bib13","doi-asserted-by":"publisher","first-page":"75","DOI":"10.18653\/v1\/2020.cmcl-1.10","article-title":"Probabilistic predictions of people perusing: Evaluating metrics of language model performance for psycholinguistic modeling","volume-title":"Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics","author":"Hao","year":"2020"},{"issue":"8","key":"2024120216320216100_bib14","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2024120216320216100_bib15","doi-asserted-by":"publisher","first-page":"1725","DOI":"10.18653\/v1\/2020.acl-main.158","article-title":"A systematic assessment of syntactic generalization in neural language models","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Jennifer","year":"2020"},{"key":"2024120216320216100_bib16","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1038\/s42256-023-00729-y","article-title":"A taxonomy and review of generalization research in NLP","volume":"5","author":"Hupkes","year":"2023","journal-title":"Nature Machine Intelligence"},{"key":"2024120216320216100_bib17","doi-asserted-by":"publisher","first-page":"4958","DOI":"10.18653\/v1\/2021.findings-acl.439","article-title":"Language models use monotonicity to assess NPI licensing","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Jumelet","year":"2021"},{"key":"2024120216320216100_bib18","doi-asserted-by":"publisher","first-page":"222","DOI":"10.18653\/v1\/W18-5424","article-title":"Do language models understand anything? On the ability of LSTMS to understand negative polarity items","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Jumelet","year":"2018"},{"key":"2024120216320216100_bib19","first-page":"287","article-title":"Verb argument structure alternations in word and sentence embeddings","volume-title":"Proceedings of the Society for Computation in Linguistics (SCiL) 2019","author":"Kann","year":"2019"},{"key":"2024120216320216100_bib20","article-title":"Scaling laws for neural language models","author":"Kaplan","year":"2020","journal-title":"CoRR"},{"key":"2024120216320216100_bib21","doi-asserted-by":"publisher","first-page":"9087","DOI":"10.18653\/v1\/2020.emnlp-main.731","article-title":"COGS: A compositional generalization challenge based on semantic interpretation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kim","year":"2020"},{"key":"2024120216320216100_bib22","doi-asserted-by":"publisher","first-page":"5203","DOI":"10.18653\/v1\/2021.acl-long.405","article-title":"Lower perplexity is not always human-like","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Kuribayashi","year":"2021"},{"issue":"5","key":"2024120216320216100_bib23","doi-asserted-by":"publisher","first-page":"1202","DOI":"10.1111\/cogs.12414","article-title":"Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge","volume":"41","author":"Lau","year":"2017","journal-title":"Cognitive Science"},{"issue":"3","key":"2024120216320216100_bib24","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1016\/j.cognition.2007.05.006","article-title":"Expectation-based syntactic comprehension","volume":"106","author":"Levy","year":"2008","journal-title":"Cognition"},{"key":"2024120216320216100_bib25","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1162\/tacl_a_00115","article-title":"Assessing the ability of LSTMs to learn syntax-sensitive dependencies","volume":"4","author":"Linzen","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024120216320216100_bib26","article-title":"Decoupled weight decay regularization","volume-title":"International Conference on Learning Representations","author":"Loshchilov","year":"2017"},{"key":"2024120216320216100_bib27","doi-asserted-by":"publisher","first-page":"1192","DOI":"10.18653\/v1\/D18-1151","article-title":"Targeted syntactic evaluation of language models","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Marvin","year":"2018"},{"key":"2024120216320216100_bib28","doi-asserted-by":"publisher","first-page":"3428","DOI":"10.18653\/v1\/P19-1334","article-title":"Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"McCoy","year":"2019"},{"key":"2024120216320216100_bib29","unstructured":"Gary\n              Milsark\n            \n          . 1974. Existential Sentences in English. Ph.D. thesis, MIT, Cambridge, MA."},{"key":"2024120216320216100_bib30","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2024.emnlp-main.53","article-title":"Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs","author":"Misra","year":"2024"},{"key":"2024120216320216100_bib31","doi-asserted-by":"publisher","first-page":"11237","DOI":"10.18653\/v1\/2023.acl-long.629","article-title":"How to plant trees in language models: Data and architectural effects on the emergence of syntactic inductive biases","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Mueller","year":"2023"},{"key":"2024120216320216100_bib32","doi-asserted-by":"publisher","first-page":"3710","DOI":"10.18653\/v1\/2021.naacl-main.290","article-title":"Refining targeted syntactic evaluation of language models","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Newman","year":"2021"},{"key":"2024120216320216100_bib33","first-page":"4034","article-title":"Universal Dependencies v2: An evergrowing multilingual treebank collection","volume-title":"Proceedings of the Twelfth Language Resources and Evaluation Conference","author":"Nivre","year":"2020"},{"key":"2024120216320216100_bib34","article-title":"Universal Dependencies","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts","author":"Nivre","year":"2017"},{"key":"2024120216320216100_bib35","first-page":"1915","article-title":"Transformer-based language model surprisal predicts human reading times best with about two billion training tokens","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2023","author":"Byung-Doh","year":"2023"},{"key":"2024120216320216100_bib36","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1162\/tacl_a_00548","article-title":"Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?","volume":"11","author":"Byung-Doh","year":"2023","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024120216320216100_bib37","first-page":"2644","article-title":"Frequency explains the inverse correlation of large language models\u2019 size, training data amount, and surprisal\u2019s fit to reading times","volume-title":"Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Byung-Doh","year":"2024"},{"key":"2024120216320216100_bib38","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161","volume-title":"Causality: Models, Reasoning and Inference","author":"Pearl","year":"2009","edition":"2nd edition"},{"key":"2024120216320216100_bib39","doi-asserted-by":"publisher","first-page":"66","DOI":"10.18653\/v1\/K19-1007","article-title":"Using priming to uncover the organization of syntactic representations in neural language models","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Prasad","year":"2019"},{"key":"2024120216320216100_bib40","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-demos.14","article-title":"Stanza: A Python natural language processing toolkit for many human languages","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Qi","year":"2020"},{"key":"2024120216320216100_bib41","doi-asserted-by":"publisher","first-page":"274","DOI":"10.18653\/v1\/2023.conll-1.19","article-title":"The validity of evaluation results: Assessing concurrence across compositionality benchmarks","volume-title":"Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)","author":"Sun","year":"2023"},{"key":"2024120216320216100_bib42","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2024120216320216100_bib43","doi-asserted-by":"crossref","unstructured":"Alex\n              Warstadt\n            \n          . 2022. Artificial Neural Networks as Models of Human Language Acquisition. Ph.D. thesis, New York University. 10.1201\/9781003205388-2","DOI":"10.1201\/9781003205388-2"},{"key":"2024120216320216100_bib44","first-page":"1737","article-title":"Can neural networks acquire a structural bias from raw linguistic data?","volume-title":"42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020","author":"Warstadt","year":"2020"},{"key":"2024120216320216100_bib45","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1201\/9781003205388-2","article-title":"What artificial neural networks can tell us about human language acquisition","volume-title":"Algebraic Structures in Natural Language","author":"Warstadt","year":"2022"},{"key":"2024120216320216100_bib46","doi-asserted-by":"publisher","first-page":"2877","DOI":"10.18653\/v1\/D19-1286","article-title":"Investigating BERT\u2019s knowledge of language: Five analysis methods with NPIs","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Warstadt","year":"2019"},{"key":"2024120216320216100_bib47","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1162\/tacl_a_00321","article-title":"BLiMP: The benchmark of linguistic minimal pairs for English","volume":"8","author":"Warstadt","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024120216320216100_bib48","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1162\/tacl_a_00290","article-title":"Neural network acceptability judgments","volume":"7","author":"Warstadt","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024120216320216100_bib49","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.18653\/v1\/2021.eacl-main.176","article-title":"Language modelling as a multi-task problem","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Weber","year":"2021"},{"key":"2024120216320216100_bib50","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.248","article-title":"Interpretability of language models via task spaces","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Weber","year":"2024"},{"key":"2024120216320216100_bib51","doi-asserted-by":"publisher","first-page":"932","DOI":"10.18653\/v1\/2021.emnlp-main.72","article-title":"Frequency effects on syntactic rule learning in transformers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Wei","year":"2021"},{"key":"2024120216320216100_bib52","article-title":"On the predictive power of neural language models for human real-time comprehension behavior","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","author":"Wilcox","year":"2020"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00720\/2482032\/tacl_a_00720.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00720\/2482032\/tacl_a_00720.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T16:32:17Z","timestamp":1733157137000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00720\/125485\/Filtered-Corpus-Training-FiCT-Shows-that-Language"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":52,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00720","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}