{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T16:43:28Z","timestamp":1783183408627,"version":"3.54.6"},"reference-count":43,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2021,3,16]],"date-time":"2021-03-16T00:00:00Z","timestamp":1615852800000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1<\/jats:p>","DOI":"10.1162\/tacl_a_00359","type":"journal-article","created":{"date-parts":[[2021,4,21]],"date-time":"2021-04-21T15:07:57Z","timestamp":1619017677000},"page":"160-175","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":44,"title":["Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals"],"prefix":"10.1162","volume":"9","author":[{"given":"Yanai","family":"Elazar","sequence":"first","affiliation":[{"name":"Computer Science Department, Bar Ilan University"},{"name":"Allen Institute for Artificial Intelligence. yanaiela@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shauli","family":"Ravfogel","sequence":"additional","affiliation":[{"name":"Computer Science Department, Bar Ilan University"},{"name":"Allen Institute for Artificial Intelligence. shauli.ravfogel@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alon","family":"Jacovi","sequence":"additional","affiliation":[{"name":"Computer Science Department, Bar Ilan University. alonjacovi@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoav","family":"Goldberg","sequence":"additional","affiliation":[{"name":"Computer Science Department, Bar Ilan University"},{"name":"Allen Institute for Artificial Intelligence. yoav.goldberg@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,3,11]]},"reference":[{"key":"2021042114435460700_bib1","article-title":"Fine-grained analysis of sentence embeddings using auxiliary prediction tasks","author":"Adi","year":"2016","journal-title":"CoRR"},{"key":"2021042114435460700_bib2","doi-asserted-by":"crossref","first-page":"1534","DOI":"10.18653\/v1\/2020.acl-main.140","article-title":"Probing linguistic features of sentence-level representations in relation extraction","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Alt","year":"2020"},{"key":"2021042114435460700_bib3","article-title":"Experiment tracking with weights and biases","author":"Biewald","year":"2020"},{"key":"2021042114435460700_bib4","doi-asserted-by":"crossref","first-page":"2126","DOI":"10.18653\/v1\/P18-1198","article-title":"What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Conneau","year":"2018"},{"key":"2021042114435460700_bib5","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2021042114435460700_bib6","doi-asserted-by":"crossref","first-page":"11","DOI":"10.18653\/v1\/D18-1002","article-title":"Adversarial removal of demographic attributes from text data","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Elazar","year":"2018"},{"key":"2021042114435460700_bib7","doi-asserted-by":"crossref","unstructured":"Amir Feder , NadavOved, UriShalit, and RoiReichart. 2020. Causalm: Causal model explanation through counterfactual language models.","DOI":"10.1162\/coli_a_00404"},{"key":"2021042114435460700_bib8","article-title":"Do neural language representations learn physical commonsense?","author":"Forbes","year":"2019","journal-title":"Proceedings of the 41st Annual Conference of the Cognitive Science Society"},{"issue":"1","key":"2021042114435460700_bib9","first-page":"2096","article-title":"Domain-adversarial training of neural networks","volume":"17","author":"Ganin","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"2021042114435460700_bib10","doi-asserted-by":"crossref","first-page":"240","DOI":"10.18653\/v1\/W18-5426","article-title":"Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information","volume-title":"Proceedings of the 2018 EMNLP Workshop Blackbox NLP: Analyzing and Interpreting Neural Networks for NLP","author":"Giulianelli","year":"2018"},{"key":"2021042114435460700_bib11","article-title":"Assessing bert\u2019s syntactic abilities","author":"Goldberg","year":"2019","journal-title":"arXiv preprint arXiv:1901.05287"},{"key":"2021042114435460700_bib12","article-title":"Explaining classifiers with causal concept effect (cace)","author":"Goyal","year":"2019","journal-title":"arXiv preprint arXiv:1907.07165"},{"key":"2021042114435460700_bib13","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D19-1275","article-title":"Designing and interpreting probes with control tasks","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Hewitt","year":"2019"},{"key":"2021042114435460700_bib14","first-page":"4129","article-title":"A structural probe for finding syntax in word representations","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT","author":"Hewitt","year":"2019"},{"key":"2021042114435460700_bib15","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1613\/jair.1.11196","article-title":"Visualisation and\u2019diagnostic classifiers\u2019 reveal how recurrent and recursive neural networks process hierarchical structure","volume":"61","author":"Hupkes","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2021042114435460700_bib16","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1162\/tacl_a_00324","article-title":"How can we know what language models know?","volume":"8","author":"Jiang","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021042114435460700_bib17","article-title":"Learning the difference that makes a difference with counterfactually-augmented data","volume-title":"International Conference on Learning Representations","author":"Kaushik","year":"2020"},{"key":"2021042114435460700_bib18","doi-asserted-by":"crossref","first-page":"241","DOI":"10.18653\/v1\/W19-4825","article-title":"Open sesame: Getting inside berts linguistic knowledge","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Lin","year":"2019"},{"key":"2021042114435460700_bib19","first-page":"1073","article-title":"Linguistic knowledge and transferability of contextual representations","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Liu","year":"2019"},{"key":"2021042114435460700_bib20","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"2021042114435460700_bib21","first-page":"92","article-title":"Universal dependency annotation for multilingual parsing","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"McDonald","year":"2013"},{"key":"2021042114435460700_bib22","volume-title":"The book of why: the new science of cause and effect","author":"Pearl","year":"2018"},{"issue":"Oct","key":"2021042114435460700_bib23","first-page":"2825","article-title":"Scikit-learn: Machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"2021042114435460700_bib24","first-page":"2227","article-title":"Deep contextualized word representations","volume-title":"Proceedings of NAACL-HLT","author":"Peters","year":"2018"},{"key":"2021042114435460700_bib25","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.18653\/v1\/D19-1250","article-title":"Language models as knowledge bases?","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Petroni","year":"2019"},{"key":"2021042114435460700_bib26","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.420","article-title":"Information-theoretic probing for linguistic structure","author":"Pimentel","year":"2020"},{"key":"2021042114435460700_bib27","doi-asserted-by":"crossref","first-page":"7237","DOI":"10.18653\/v1\/2020.acl-main.647","article-title":"Null it out: Guarding protected attributes by iterative nullspace projection","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ravfogel","year":"2020"},{"key":"2021042114435460700_bib28","article-title":"Probing the probing paradigm: Does probing accuracy entail task relevance?","author":"Ravichander","year":"2020","journal-title":"arXiv preprint arXiv:2005.00719"},{"key":"2021042114435460700_bib29","first-page":"8592","article-title":"Visualizing and measuring the geometry of bert","volume-title":"Advances in Neural Information Processing Systems","author":"Reif","year":"2019"},{"issue":"5","key":"2021042114435460700_bib30","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/0005-1098(78)90005-5","article-title":"Modeling by shortest data description","volume":"14","author":"Rissanen","year":"1978","journal-title":"Automatica"},{"key":"2021042114435460700_bib31","article-title":"A primer in bertology: What we know about how Bert works","author":"Rogers","year":"2020","journal-title":"arXiv preprint arXiv:2002.12327"},{"key":"2021042114435460700_bib32","article-title":"olmpics \u2013 on what language model pre-training captures","author":"Talmor","year":"2019"},{"key":"2021042114435460700_bib33","article-title":"Investigating transferability in pretrained language models","author":"Tamkin","year":"2020","journal-title":"arXiv preprint arXiv:2004.14975"},{"key":"2021042114435460700_bib34","first-page":"4593","article-title":"BERT rediscovers the classical NLP pipeline","volume-title":"Proceedings of the Conference of the Association for Computational Linguistics, ACL","author":"Tenney","year":"2019"},{"key":"2021042114435460700_bib35","article-title":"What do you learn from context? probing for sentence structure in contextualized word representations","volume-title":"International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2021042114435460700_bib36","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in neural information processing systems","author":"Vaswani","year":"2017"},{"key":"2021042114435460700_bib37","article-title":"Causal mediation analysis for interpreting neural NLP: The case of gender bias","author":"Vig","year":"2020","journal-title":"arXiv preprint arXiv:2004.12265"},{"key":"2021042114435460700_bib38","article-title":"Information-theoretic probing with minimum description length","author":"Voita","year":"2020","journal-title":"arXiv preprint arXiv:2003.12298"},{"key":"2021042114435460700_bib39","doi-asserted-by":"crossref","first-page":"2870","DOI":"10.18653\/v1\/D19-1286","article-title":"Investigating BERTs knowledge of language: Five analysis methods with NPIs","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Warstadt","year":"2019"},{"key":"2021042114435460700_bib40","article-title":"Probing neural language models for human tacit assumptions","volume-title":"42nd Annual Virtual Meeting of theCognitive Science Society (CogSci)","author":"Weir","year":"2020"},{"key":"2021042114435460700_bib41","first-page":"23","article-title":"Ontonotes release 5.0 ldc2013t19","author":"Weischedel","year":"2013","journal-title":"Linguistic Data Consortium, Philadelphia, PA"},{"key":"2021042114435460700_bib42","article-title":"Huggingface\u2019s transformers: State-of-the-art natural language processing","author":"Wolf","year":"2019","journal-title":"ArXiv"},{"key":"2021042114435460700_bib43","article-title":"Evaluating commonsense in pre-trained language models","author":"Zhou","year":"2019","journal-title":"arXiv preprint arXiv:1911.11931"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00359\/1894330\/tacl_a_00359.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00359\/1894330\/tacl_a_00359.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,25]],"date-time":"2022-12-25T01:33:25Z","timestamp":1671932005000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00359\/98091\/Amnesic-Probing-Behavioral-Explanation-with"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,11]]},"references-count":43,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00359","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,11]]}}}