{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T13:28:08Z","timestamp":1783690088478,"version":"3.55.0"},"reference-count":40,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"vor","delay-in-days":23,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,1,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Many studies have shown that transformers are able to predict subject-verb agreement, demonstrating their ability to uncover an abstract representation of the sentence in an unsupervised way. Recently, Li et al. (2021) found that transformers were also able to predict the object-past participle agreement in French, the modeling of which in formal grammar is fundamentally different from that of subject-verb agreement and relies on a movement and an anaphora resolution.<\/jats:p>\n               <jats:p>To better understand transformers\u2019 internal working, we propose to contrast how they handle these two kinds of agreement. Using probing and counterfactual analysis methods, our experiments on French agreements show that (i) the agreement task suffers from several confounders that partially question the conclusions drawn so far and (ii) transformers handle subject-verb and object-past participle agreements in a way that is consistent with their modeling in theoretical linguistics.<\/jats:p>","DOI":"10.1162\/tacl_a_00531","type":"journal-article","created":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T16:16:04Z","timestamp":1674576964000},"page":"18-33","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement"],"prefix":"10.1162","volume":"11","author":[{"given":"Bingzhi","family":"Li","sequence":"first","affiliation":[{"name":"Universit\u00e9 Paris Cit\u00e9, LLF, CNRS, 75013 Paris, France. bingzhi.li@etu.u-paris.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Wisniewski","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris Cit\u00e9, LLF, CNRS, 75013 Paris, France. guillaume.wisniewski@u-paris.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Beno\u00eet","family":"Crabb\u00e9","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris Cit\u00e9, LLF, CNRS, 75013 Paris, France. benoit.crabbe@u-paris.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"issue":"3","key":"2023012416102798000_bib1","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1017\/S0959269512000312","article-title":"Input frequency and the acquisition of subject-verb agreement in number in spoken and written french","volume":"23","author":"A\u0308gren","year":"2013","journal-title":"Journal of French Language Studies"},{"key":"2023012416102798000_bib2","article-title":"Understanding intermediate layers using linear classifier probes","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings","author":"Alain","year":"2017"},{"issue":"1","key":"2023012416102798000_bib3","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1162\/coli_a_00422","article-title":"Probing classifiers: Promises, shortcomings, and advances","volume":"48","author":"Belinkov","year":"2022","journal-title":"Computational Linguistics"},{"key":"2023012416102798000_bib4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2020.acl-tutorials.1","article-title":"Interpretability and analysis in neural NLP","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts","author":"Belinkov","year":"2020"},{"key":"2023012416102798000_bib5","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1162\/tacl_a_00254","article-title":"Analysis methods in neural language processing: A survey","volume":"7","author":"Belinkov","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102798000_bib6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/9781118358733.wbsyncom081","article-title":"(Past) participle agreement","author":"Belletti","year":"2017","journal-title":"The Wiley Blackwell Companion to Syntax, Second Edition"},{"issue":"1","key":"2023012416102798000_bib7","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1016\/0010-0285(91)90003-7","article-title":"Broken agreement","volume":"23","author":"Bock","year":"1991","journal-title":"Cognitive Psychology"},{"key":"2023012416102798000_bib8","doi-asserted-by":"publisher","first-page":"pages 2126\u2013pages 2136","DOI":"10.18653\/v1\/P18-1198","article-title":"What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Conneau","year":"2018"},{"key":"2023012416102798000_bib9","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1162\/tacl_a_00359","article-title":"Amnesic probing: Behavioral explanation with amnesic counterfactuals","volume":"9","author":"Elazar","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102798000_bib10","unstructured":"Jeffrey L.\n              Elman\n            \n          . 1989. Representation and structure in connectionist models. Techical report, California University San Diego, La Jolla Center for Research in Language."},{"key":"2023012416102798000_bib11","doi-asserted-by":"publisher","first-page":"1828","DOI":"10.18653\/v1\/2021.acl-long.144","article-title":"Causal analysis of syntactic agreement mechanisms in neural language models","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Finlayson","year":"2021"},{"key":"2023012416102798000_bib12","doi-asserted-by":"publisher","first-page":"240","DOI":"10.18653\/v1\/W18-5426","article-title":"Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Giulianelli","year":"2018"},{"key":"2023012416102798000_bib13","article-title":"Assessing BERT\u2019s syntactic abilities","author":"Goldberg","year":"2019","journal-title":"CoRR"},{"key":"2023012416102798000_bib14","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.18653\/v1\/N18-1108","article-title":"Colorless green recurrent networks dream hierarchically","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Gulordava","year":"2018"},{"key":"2023012416102798000_bib15","doi-asserted-by":"publisher","first-page":"2733","DOI":"10.18653\/v1\/D19-1275","article-title":"Designing and interpreting probes with control tasks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Hewitt","year":"2019"},{"key":"2023012416102798000_bib16","doi-asserted-by":"publisher","first-page":"907","DOI":"10.1613\/jair.1.11196","article-title":"Visualisation and\u2019diagnostic classifiers\u2019 reveal how recurrent and recursive neural networks process hierarchical structure","volume":"61","author":"Hupkes","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2023012416102798000_bib17","doi-asserted-by":"publisher","first-page":"2486","DOI":"10.18653\/v1\/D17-1263","article-title":"A challenge set approach to evaluating machine translation","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Isabelle","year":"2017"},{"key":"2023012416102798000_bib18","doi-asserted-by":"publisher","first-page":"3651","DOI":"10.18653\/v1\/P19-1356","article-title":"What does BERT learn about the structure of language?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Jawahar","year":"2019"},{"key":"2023012416102798000_bib19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/K19-1001","article-title":"Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Jumelet","year":"2019"},{"key":"2023012416102798000_bib20","doi-asserted-by":"publisher","first-page":"4801","DOI":"10.18653\/v1\/2020.acl-main.434","article-title":"Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Klafka","year":"2020"},{"key":"2023012416102798000_bib21","doi-asserted-by":"publisher","first-page":"104699","DOI":"10.1016\/j.cognition.2021.104699","article-title":"Mechanisms for handling nested dependencies in neural-network language models and humans","author":"Lakretz","year":"2021","journal-title":"Cognition"},{"key":"2023012416102798000_bib22","doi-asserted-by":"publisher","first-page":"2309","DOI":"10.18653\/v1\/2022.findings-acl.181","article-title":"Does BERT really agree? Fine-grained analysis of lexical dependence on a syntactic task","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Lasri","year":"2022"},{"key":"2023012416102798000_bib23","doi-asserted-by":"crossref","first-page":"4599","DOI":"10.18653\/v1\/2021.emnlp-main.377","article-title":"Are Transformers a modern version of ELIZA? Observations on French object verb agreement","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Li","year":"2021"},{"key":"2023012416102798000_bib24","doi-asserted-by":"crossref","first-page":"501","DOI":"10.18653\/v1\/2022.acl-short.54","article-title":"How distributed are distributed representations? An observation on the locality of syntactic information in verb agreement tasks","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Li","year":"2022"},{"key":"2023012416102798000_bib25","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1146\/annurev-linguistics-032020-051035","article-title":"Syntactic structure from deep learning","volume":"7","author":"Linzen","year":"2021","journal-title":"Annual Review of Linguistics"},{"key":"2023012416102798000_bib26","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1162\/tacl_a_00115","article-title":"Assessing the ability of LSTMs to learn syntax-sensitive dependencies","volume":"4","author":"Linzen","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102798000_bib27","doi-asserted-by":"publisher","first-page":"1192","DOI":"10.18653\/v1\/D18-1151","article-title":"Targeted syntactic evaluation of language models","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Marvin","year":"2018"},{"key":"2023012416102798000_bib28","doi-asserted-by":"publisher","first-page":"3710","DOI":"10.18653\/v1\/2021.naacl-main.290","article-title":"Refining targeted syntactic evaluation of language models","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Newman","year":"2021"},{"key":"2023012416102798000_bib29","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"2023012416102798000_bib30","doi-asserted-by":"publisher","first-page":"3138","DOI":"10.18653\/v1\/2020.emnlp-main.254","article-title":"Pareto probing: Trading off accuracy for complexity","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Pimentel","year":"2020"},{"key":"2023012416102798000_bib31","doi-asserted-by":"publisher","first-page":"4609","DOI":"10.18653\/v1\/2020.acl-main.420","article-title":"Information-theoretic probing for linguistic structure","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Pimentel","year":"2020"},{"key":"2023012416102798000_bib32","doi-asserted-by":"publisher","first-page":"194","DOI":"10.18653\/v1\/2021.conll-1.15","article-title":"Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction","volume-title":"Proceedings of the 25th Conference on Computational Natural Language Learning","author":"Ravfogel","year":"2021"},{"key":"2023012416102798000_bib33","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in BERTology: What we know about how BERT works","volume":"8","author":"Rogers","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102798000_bib34","article-title":"What do you learn from context? Probing for sentence structure in contextualized word representations","volume-title":"International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2023012416102798000_bib35","first-page":"109","article-title":"Investigating \u201caspect\u201d in NMT and SMT: Translating the English simple past and present perfect","volume":"7","author":"Vanmassenhove","year":"2017","journal-title":"Computational Linguistics in the Netherlands Journal"},{"key":"2023012416102798000_bib36","article-title":"Diagnostic classifiers revealing how neural networks process hierarchical structure","volume-title":"CoCo@ NIPS","author":"Veldhoen","year":"2016"},{"key":"2023012416102798000_bib37","first-page":"12388","article-title":"Investigating gender bias in language models using causal mediation analysis","volume-title":"Advances in Neural Information Processing Systems","author":"Vig","year":"2020"},{"key":"2023012416102798000_bib38","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1162\/tacl_a_00321","article-title":"BLiMP: A benchmark of linguistic minimal pairs for English","volume-title":"Proceedings of the Society for Computation in Linguistics 2020","author":"Warstadt","year":"2020"},{"key":"2023012416102798000_bib39","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1162\/tacl_a_00290","article-title":"Neural network acceptability judgments","volume":"7","author":"Warstadt","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102798000_bib40","doi-asserted-by":"publisher","first-page":"932","DOI":"10.18653\/v1\/2021.emnlp-main.72","article-title":"Frequency effects on syntactic rule learning in transformers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Wei","year":"2021"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00531\/2067845\/tacl_a_00531.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00531\/2067845\/tacl_a_00531.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T16:16:15Z","timestamp":1674576975000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00531\/114592\/Assessing-the-Capacity-of-Transformer-to-Abstract"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":40,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00531","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}