{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T06:10:46Z","timestamp":1778134246920,"version":"3.51.4"},"reference-count":53,"publisher":"MIT Press","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),<jats:sup>1<\/jats:sup>a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs\u2014that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.<\/jats:p>","DOI":"10.1162\/tacl_a_00321","type":"journal-article","created":{"date-parts":[[2020,7,20]],"date-time":"2020-07-20T18:01:16Z","timestamp":1595268076000},"page":"377-392","source":"Crossref","is-referenced-by-count":96,"title":["BLiMP: The Benchmark of Linguistic Minimal Pairs for English"],"prefix":"10.1162","volume":"8","author":[{"given":"Alex","family":"Warstadt","sequence":"first","affiliation":[{"name":"Department of Linguistics, New York University."}]},{"given":"Alicia","family":"Parrish","sequence":"additional","affiliation":[{"name":"Department of Linguistics, New York University."}]},{"given":"Haokun","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York University."}]},{"given":"Anhad","family":"Mohananey","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York University."}]},{"given":"Wei","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Computer Science, New York University."}]},{"given":"Sheng-Fu","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Linguistics, New York University."}]},{"given":"Samuel R.","family":"Bowman","sequence":"additional","affiliation":[{"name":"Department of Linguistics, New York University"},{"name":"Department of Computer Science, New York University"},{"name":"Center for Data Science, New York University."}]}],"member":"281","reference":[{"key":"bib1","first-page":"152","volume-title":"Lingua","volume":"30","author":"Marantz Alec","year":"2013"},{"key":"bib2","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780199243709.001.0001","volume-title":"Core Syntax: A Minimalist Approach","author":"Adger David","year":"2003"},{"key":"bib3","volume-title":"Proceedings of ICLR Conference Track","author":"Adi Yossi","year":"2017"},{"key":"bib4","author":"An Aixiu","year":"2019","journal-title":"arXiv preprint arXiv:1909.04625"},{"issue":"1","key":"bib5","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/0010-0285(91)90003-7","volume":"23","author":"Bock Kathryn","year":"1991","journal-title":"Cognitive Psychology"},{"key":"bib6","volume-title":"Proceedings of the Third Meeting of the Society for Computation in Linguistics (SCiL)","author":"Chaves Rui P.","year":"2020"},{"issue":"4","key":"bib7","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1006\/csla.1999.0128","volume":"13","author":"Chen Stanley F.","year":"1999","journal-title":"Computer Speech & Language"},{"key":"bib8","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780199697977.001.0001","volume-title":"Logic in Grammar","author":"Chierchia Gennaro","year":"2013"},{"key":"bib9","volume-title":"Aspects of the Theory of Syntax","author":"Chomsky Noam","year":"1965"},{"key":"bib10","volume-title":"Lectures on Government and Binding","author":"Chomsky Noam","year":"1981"},{"key":"bib11","first-page":"133","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Chowdhury Shammur Absar","year":"2018"},{"key":"bib12","doi-asserted-by":"crossref","first-page":"204","DOI":"10.18653\/v1\/W19-4821","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Chowdhury Shammur Absar","year":"2019"},{"key":"bib13","first-page":"2126","volume-title":"ACL 2018-56th Annual Meeting of the Association for Computational Linguistics","volume":"1","author":"Conneau Alexis","year":"2018"},{"key":"bib14","volume-title":"Proceedings of the Third Meeting of the Society for Computation in Linguistics (SCiL)","author":"Da Costa Jillian K.","year":"2020"},{"key":"bib15","first-page":"2978","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Dai Zihang","year":"2019"},{"key":"bib16","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019"},{"key":"bib17","first-page":"1790","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Ettinger Allyson","year":"2018"},{"key":"bib18","author":"Futrell Richard","year":"2018","journal-title":"arXiv preprint arXiv:1809.01329"},{"key":"bib19","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1353\/lan.2007.0115","author":"Geurts Bart","year":"2007","journal-title":"Language"},{"issue":"1","key":"bib20","first-page":"34","volume":"4","author":"Graff David","year":"2003","journal-title":"Linguistic Data Consortium, Philadelphia"},{"issue":"1","key":"bib21","first-page":"363","volume":"2","author":"Gulordava Kristina","year":"2019","journal-title":"Proceedings of the Society for Computation in Linguistics"},{"key":"bib22","first-page":"690","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Heafield Kenneth","year":"2013"},{"key":"bib23","doi-asserted-by":"crossref","first-page":"174","DOI":"10.3115\/v1\/P14-2029","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","volume":"2","author":"Heilman Michael","year":"2014"},{"key":"bib24","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"bib25","doi-asserted-by":"crossref","first-page":"328","DOI":"10.18653\/v1\/P18-1031","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Howard Jeremy","year":"2018"},{"key":"bib26","doi-asserted-by":"crossref","first-page":"222","DOI":"10.18653\/v1\/W18-5424","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Jumelet Jaap","year":"2018"},{"issue":"1","key":"bib27","first-page":"287","volume":"2","author":"Kann Katharina","year":"2019","journal-title":"Proceedings of the Society for Computation in Linguistics"},{"issue":"3","key":"bib28","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1007\/s11049-017-9390-z","volume":"36","author":"Kush Dave","year":"2018","journal-title":"Natural Language & Linguistic Theory"},{"issue":"5","key":"bib29","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1111\/cogs.12414","volume":"41","author":"Lau Jey Han","year":"2017","journal-title":"Cognitive Science"},{"key":"bib30","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00115"},{"key":"bib31","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.18653\/v1\/D18-1151","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Marvin Rebecca","year":"2018"},{"key":"bib32","author":"Merity Stephen","year":"2016","journal-title":"CoRR"},{"key":"bib33","volume-title":"Eleventh Annual Conference of the International Speech Communication Association","author":"Mikolov Tom\u00e1\u0161","year":"2010"},{"key":"bib34","author":"Peters Matthew E.","year":"2018","journal-title":"arXiv preprint arXiv:1802.05365"},{"key":"bib35","unstructured":"Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning, Technical report, OpenAI."},{"issue":"8","key":"bib36","volume":"1","author":"Radford Alec","year":"2019","journal-title":"OpenAI Blog"},{"key":"bib37","author":"Raffel Colin","year":"2019","journal-title":"arXiv e-prints"},{"issue":"6","key":"bib38","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1207\/s15516709cog0000_28","volume":"29","author":"Reali Florencia","year":"2005","journal-title":"Cognitive Science"},{"key":"bib39","volume-title":"Syntactic Theory: A Formal Introduction","author":"Sag Ivan A.","year":"2003","edition":"2"},{"key":"bib40","volume-title":"The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology","author":"Sch\u00fctze Carson T.","year":"1996"},{"key":"bib41","first-page":"1526","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Shi Xing","year":"2016"},{"key":"bib42","volume-title":"An Introduction to Syntactic Analysis and Theory","author":"Sportiche Dominique","year":"2013"},{"key":"bib43","volume-title":"Proceedings of ICLR","author":"Tenney Ian","year":"2019"},{"key":"bib44","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani Ashish","year":"2017"},{"key":"bib45","volume-title":"33rd Conference on Neural Information Processing Systems","author":"Wang Alex","year":"2019"},{"key":"bib46","doi-asserted-by":"crossref","first-page":"353","DOI":"10.18653\/v1\/W18-5446","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wang Alex","year":"2018"},{"key":"bib47","unstructured":"Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Haokun Liu, Najoung Kim, Phu Mon Htut, Thibault F\u2019evry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019b.jiant1.2: A software toolkit for research on general-purpose text understanding models.http:\/\/jiant.info\/."},{"key":"bib48","doi-asserted-by":"crossref","first-page":"722","DOI":"10.2307\/415742","author":"Ward Gregory","year":"1995","journal-title":"Language"},{"key":"bib49","author":"Warstadt Alex","year":"2019","journal-title":"arXiv preprint arXiv:1901.03438"},{"key":"bib50","first-page":"2870","volume-title":"Proceedings of EMNLP-IJCNLP","author":"Warstadt Alex","year":"2019"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00290"},{"key":"bib52","doi-asserted-by":"crossref","first-page":"211","DOI":"10.18653\/v1\/W18-5423","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wilcox Ethan","year":"2018"},{"key":"bib53","first-page":"3302","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Wilcox Ethan","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00321","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,10]],"date-time":"2024-08-10T11:25:03Z","timestamp":1723289103000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96452"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":53,"alternative-id":["10.1162\/tacl_a_00321"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00321","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}