{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T14:34:36Z","timestamp":1786113276087,"version":"3.56.0"},"reference-count":60,"publisher":"MIT Press","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p>This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.\u2019s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.<\/jats:p>","DOI":"10.1162\/tacl_a_00290","type":"journal-article","created":{"date-parts":[[2019,9,30]],"date-time":"2019-09-30T19:43:11Z","timestamp":1569872591000},"page":"625-641","source":"Crossref","is-referenced-by-count":329,"title":["Neural Network Acceptability Judgments"],"prefix":"10.1162","volume":"7","author":[{"given":"Alex","family":"Warstadt","sequence":"first","affiliation":[{"name":"New York University."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amanpreet","family":"Singh","sequence":"additional","affiliation":[{"name":"New York University, Facebook AI Research."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel R.","family":"Bowman","sequence":"additional","affiliation":[{"name":"New York University."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780199243709.001.0001","volume-title":"Core Syntax: A Minimalist Approach","author":"Adger David","year":"2003"},{"key":"bib2","volume-title":"Proceedings of ICLR Conference Track","author":"Adi Yossi","year":"2017"},{"issue":"1","key":"bib3","first-page":"1","volume":"13","author":"Baltin Mark R.","year":"1982","journal-title":"Linguistic Inquiry"},{"key":"bib4","doi-asserted-by":"crossref","DOI":"10.1002\/9780470756416","volume-title":"Handbook of Contemporary Syntactic Theory","author":"Baltin Mark R.","year":"2001"},{"issue":"7","key":"bib5","doi-asserted-by":"crossref","first-page":"1207","DOI":"10.1111\/j.1551-6709.2011.01189.x","volume":"35","author":"Berwick Robert C.","year":"2011","journal-title":"Cognitive Science"},{"key":"bib6","doi-asserted-by":"crossref","first-page":"31","DOI":"10.3115\/1219044.1219075","volume-title":"Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions","author":"Bird Steven","year":"2004"},{"issue":"3","key":"bib7","first-page":"275","volume":"4","author":"Bresnan Joan W.","year":"1973","journal-title":"Linguistic Inquiry"},{"key":"bib8","volume-title":"Syntax: A Generative Introduction","author":"Carnie Andrew","year":"2013"},{"key":"bib9","doi-asserted-by":"crossref","DOI":"10.1515\/9783112316009","volume-title":"Syntactic Structures","author":"Chomsky Noam","year":"1957"},{"key":"bib10","volume-title":"Aspects of the Theory of Syntax","author":"Chomsky Noam","year":"1965"},{"key":"bib11","volume-title":"The Minimalist Program","author":"Chomsky Noam","year":"1995"},{"issue":"3","key":"bib12","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/BF01248819","volume":"3","author":"Chung Sandra","year":"1995","journal-title":"Natural Language Semantics"},{"key":"bib13","doi-asserted-by":"crossref","DOI":"10.1002\/9781444390568","volume-title":"Linguistic Nativism and the Poverty of the Stimulus","author":"Clark Alexander","year":"2011"},{"issue":"2","key":"bib14","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1111\/j.1467-9612.2005.00076.x","volume":"8","author":"Collins Chris","year":"2005","journal-title":"Syntax"},{"key":"bib15","first-page":"670","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Conneau Alexis","year":"2017"},{"key":"bib16","first-page":"2126","volume-title":"ACL 2018-56th Annual Meeting of the Association for Computational Linguistics","volume":"1","author":"Conneau Alexis","year":"2018"},{"key":"bib17","doi-asserted-by":"publisher","DOI":"10.1162\/002438999554200"},{"issue":"5","key":"bib18","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1023\/A:1005494000753","volume":"21","author":"Dayal Veneeta","year":"1998","journal-title":"Linguistics and Philosophy"},{"key":"bib19","first-page":"1790","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Ettinger Allyson","year":"2018"},{"key":"bib20","doi-asserted-by":"crossref","first-page":"134","DOI":"10.18653\/v1\/W16-2524","volume-title":"Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP","author":"Ettinger Allyson","year":"2016"},{"key":"bib21","author":"Futrell Richard","year":"2018","journal-title":"arXiv preprint arXiv:1809.01329"},{"key":"bib22","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/978-94-009-3401-6_8","volume-title":"The Formal Complexity of Natural Language","author":"Gazdar Gerald","year":"1981"},{"issue":"6","key":"bib23","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.tics.2010.03.005","volume":"14","author":"Gibson Edward","year":"2010","journal-title":"Trends in Cognitive Sciences"},{"issue":"3","key":"bib24","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1353\/lan.2004.0129","volume":"80","author":"Goldberg Adele E.","year":"2004","journal-title":"Language"},{"issue":"6","key":"bib25","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1037\/0012-1649.28.6.1096","volume":"28","author":"Hart Betty","year":"1992","journal-title":"Developmental Psychology"},{"key":"bib26","doi-asserted-by":"crossref","first-page":"174","DOI":"10.3115\/v1\/P14-2029","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","volume":"2","author":"Heilman Michael","year":"2014"},{"issue":"1","key":"bib27","first-page":"21","volume":"2","author":"Jackendoff Ray S.","year":"1971","journal-title":"Linguistic Inquiry"},{"issue":"4","key":"bib28","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1007\/BF00985272","volume":"16","author":"Kadmon Nirit","year":"1993","journal-title":"Linguistics and Philosophy"},{"key":"bib29","volume-title":"English Syntax: An Introduction","author":"Kim Jong-Bok","year":"2008"},{"key":"bib30","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Kingma Diederik P.","year":"2014"},{"key":"bib31","first-page":"3294","volume-title":"Advances in Neural Information Processing Systems","author":"Kiros Ryan","year":"2015"},{"issue":"5","key":"bib32","first-page":"1202","volume":"41","author":"Lau Jey Han","year":"2016","journal-title":"Cognitive Science"},{"issue":"1","key":"bib33","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1109\/69.842255","volume":"12","author":"Lawrence Steve","year":"2000","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"bib34","volume-title":"English Verb Classes and Alternations: A preliminary investigation","author":"Levin Beth","year":"1993"},{"key":"bib35","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00115"},{"issue":"1","key":"bib36","doi-asserted-by":"crossref","first-page":"100","DOI":"10.5334\/gjgl.528","volume":"3","author":"Linzen Tal","year":"2018","journal-title":"Glossa: A Journal of General Linguistics"},{"issue":"3","key":"bib37","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1353\/lan.2016.0052","volume":"92","author":"Mahowald Kyle","year":"2016","journal-title":"Language"},{"key":"bib38","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.18653\/v1\/D18-1151","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Marvin Rebecca","year":"2018"},{"issue":"2","key":"bib39","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","volume":"405","author":"Matthews Brian W.","year":"1975","journal-title":"Biochimica et Biophysica Acta (BBA)-Protein Structure"},{"key":"bib40","volume-title":"An Introduction to English Syntax","author":"Miller Jim","year":"2002"},{"key":"bib41","first-page":"1532","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","author":"Pennington Jeffrey","year":"2014"},{"key":"bib42","first-page":"2227","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","volume":"1","author":"Peters Matthew","year":"2018"},{"issue":"1","key":"bib43","first-page":"9","volume":"18","author":"Pullum Geoffrey K.","year":"2002","journal-title":"Linguistic Review"},{"issue":"1","key":"bib44","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1017\/S0022226707004975","volume":"44","author":"Hovav Malka Rappaport","year":"2008","journal-title":"Journal of Linguistics"},{"key":"bib45","unstructured":"John Robert Ross. 1967. Constraints on Variables in Syntax. Ph.D. thesis, MIT."},{"issue":"2","key":"bib46","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1017\/S002222679700652X","volume":"33","author":"Sag Ivan A.","year":"1997","journal-title":"Journal of Linguistics"},{"issue":"2","key":"bib47","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/BF00133839","volume":"3","author":"Sag Ivan A.","year":"1985","journal-title":"Natural Language & Linguistic Theory"},{"key":"bib48","volume-title":"Syntactic Theory: A Formal Introduction","author":"Sag Ivan A.","year":"2003","edition":"2"},{"key":"bib49","volume-title":"The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology","author":"Sch\u00fctze Carson T.","year":"1996"},{"key":"bib50","first-page":"1526","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Shi Xing","year":"2016"},{"key":"bib51","volume-title":"An Introduction to Syntactic Analysis and Theory","author":"Sportiche Dominique","year":"2013"},{"issue":"3","key":"bib52","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1017\/S0022226712000011","volume":"48","author":"Sprouse Jon","year":"2012","journal-title":"Journal of Linguistics"},{"key":"bib53","doi-asserted-by":"crossref","first-page":"e311","DOI":"10.1017\/S0140525X17000590","volume":"40","author":"Sprouse Jon","year":"2017","journal-title":"Behavioral and Brain Sciences"},{"key":"bib54","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1016\/j.lingua.2013.07.002","volume":"134","author":"Sprouse Jon","year":"2013","journal-title":"Lingua"},{"issue":"3","key":"bib55","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1558\/cj.v26i3.474-490","volume":"26","author":"Wagner Joachim","year":"2009","journal-title":"CALICO Journal"},{"key":"bib56","doi-asserted-by":"crossref","first-page":"353","DOI":"10.18653\/v1\/W18-5446","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wang Alex","year":"2018"},{"key":"bib57","author":"Warstadt Alex","year":"2019","journal-title":"arXiv preprint arXiv:1901.03438"},{"key":"bib58","doi-asserted-by":"crossref","first-page":"211","DOI":"10.18653\/v1\/W18-5423","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wilcox Ethan","year":"2018"},{"key":"bib59","first-page":"3302","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Wilcox Ethan","year":"2019"},{"issue":"1","key":"bib60","first-page":"203","volume":"11","author":"Williams Edwin","year":"1980","journal-title":"Linguistic Inquiry"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00290","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T07:29:50Z","timestamp":1721806190000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43528"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":60,"alternative-id":["10.1162\/tacl_a_00290"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00290","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}