{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T07:18:59Z","timestamp":1771917539940,"version":"3.50.1"},"reference-count":57,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,11,8]],"date-time":"2022-11-08T00:00:00Z","timestamp":1667865600000},"content-version":"vor","delay-in-days":311,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of \u201ccat\u201d is related to our concepts of \u201cears\u201d and \u201cwhiskers\u201d in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system\u2019s behavior is consistent with several key aspects of Fodor\u2019s criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models\u2019 internal states.<\/jats:p>","DOI":"10.1162\/tacl_a_00514","type":"journal-article","created":{"date-parts":[[2022,11,8]],"date-time":"2022-11-08T21:00:04Z","timestamp":1667941204000},"page":"1193-1208","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Unit Testing for Concepts in Neural Networks"],"prefix":"10.1162","volume":"10","author":[{"given":"Charles","family":"Lovering","sequence":"first","affiliation":[{"name":"Department of Computer Science Brown University, USA. charles_lovering@brown.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ellie","family":"Pavlick","sequence":"additional","affiliation":[{"name":"Department of Computer Science Brown University, USA. ellie_pavlick@brown.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,11,7]]},"reference":[{"key":"2022110820595817400_bib1","article-title":"Debugging tests for model explanations","author":"Adebayo","year":"2020","journal-title":"arXiv preprint arXiv:2011.05429"},{"key":"2022110820595817400_bib2","article-title":"Fine- grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks","volume-title":"International Conference on Learning Representations (ICLR)","author":"Adi","year":"2017"},{"key":"2022110820595817400_bib3","article-title":"Measuring compositionality in representation learning","volume-title":"International Conference on Learning Representations","author":"Andreas","year":"2019"},{"key":"2022110820595817400_bib4","first-page":"39","article-title":"Neural module networks","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Andreas","year":"2016"},{"issue":"1","key":"2022110820595817400_bib5","doi-asserted-by":"publisher","first-page":"5","DOI":"10.2307\/2184371","article-title":"Psychologism and behaviorism","volume":"90","author":"Block","year":"1981","journal-title":"The Philosophical Review"},{"key":"2022110820595817400_bib6","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1146\/annurev-linguistics-011619-030303","article-title":"Distributional semantics and linguistic theory","volume":"6","author":"Boleda","year":"2020","journal-title":"Annual Review of Linguistics"},{"key":"2022110820595817400_bib7","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195367638.001.0001","volume-title":"The Origin of Concepts","author":"Carey","year":"2009"},{"key":"2022110820595817400_bib8","article-title":"Explaining image classifiers by counterfactual generation","author":"Chang","year":"2018","journal-title":"arXiv preprint arXiv:1807.08024"},{"key":"2022110820595817400_bib9","doi-asserted-by":"crossref","first-page":"2952","DOI":"10.18653\/v1\/P19-1283","article-title":"Correlating neural and symbolic representations of language","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Chrupa\u0142a","year":"2019"},{"key":"2022110820595817400_bib10","article-title":"An image is worth 16 \u00d7 16 words: Transformers for image recognition at scale","author":"Dosovitskiy","year":"2020","journal-title":"arXiv preprint arXiv:2010.11929"},{"key":"2022110820595817400_bib11","article-title":"When BERT forgets how to POS: Amnesic probing of linguistic properties and MLM predictions","author":"Elazar","year":"2020","journal-title":"arXiv preprint arXiv:2006.00995"},{"issue":"10","key":"2022110820595817400_bib12","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1002\/lnco.362","article-title":"Vector space models of word meaning and phrase meaning: A survey","volume":"6","author":"Erk","year":"2012","journal-title":"Language and Linguistics Compass"},{"key":"2022110820595817400_bib13","doi-asserted-by":"crossref","first-page":"134","DOI":"10.18653\/v1\/W16-2524","article-title":"Probing for semantic evidence of composition by means of simple classification tasks","volume-title":"Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP","author":"Ettinger","year":"2016"},{"key":"2022110820595817400_bib14","doi-asserted-by":"publisher","DOI":"10.1093\/0198236360.001.0001","volume-title":"Concepts: Where Cognitive Science Went Wrong","author":"Fodor","year":"1998"},{"issue":"1\u20132","key":"2022110820595817400_bib15","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/0010-0277(88)90031-5","article-title":"Connectionism and cognitive architecture: A critical analysis","volume":"28","author":"Fodor","year":"1988","journal-title":"Cognition"},{"key":"2022110820595817400_bib16","doi-asserted-by":"publisher","first-page":"3429","DOI":"10.1109\/ICCV.2017.371","article-title":"Interpretable explanations of black boxes by meaningful perturbation","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Fong","year":"2017"},{"key":"2022110820595817400_bib17","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.177","article-title":"Probing linguistic systematicity","volume-title":"Proceedings of ACL","author":"Goodwin","year":"2020"},{"key":"2022110820595817400_bib18","article-title":"Explaining classifiers with causal concept effect","author":"Goyal","year":"2019","journal-title":"arXiv preprint arXiv:1907. 07165"},{"key":"2022110820595817400_bib19","first-page":"2376","article-title":"Counterfactual visual explanations","volume-title":"International Conference on Machine Learning","author":"Goyal","year":"2019"},{"key":"2022110820595817400_bib20","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2022110820595817400_bib21","doi-asserted-by":"publisher","first-page":"65","DOI":"10.18653\/v1\/2020.findings-emnlp.7","article-title":"Reducing sentiment bias in language models via counterfactual evaluation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Huang","year":"2020"},{"key":"2022110820595817400_bib22","doi-asserted-by":"publisher","first-page":"907","DOI":"10.1613\/jair.1.11196","article-title":"Visualisation and \u2018diagnostic classifiers\u2019 reveal how recurrent and recursive neural networks process hierarchical structure","volume":"61","author":"Hupkes","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2022110820595817400_bib23","first-page":"448","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","volume-title":"International Conference on Machine Learning","author":"Ioffe","year":"2015"},{"key":"2022110820595817400_bib24","article-title":"Meaning Holism","volume-title":"The Stanford Encyclopedia of Philosophy, Winter 2020 edition","author":"Jackman","year":"2020"},{"key":"2022110820595817400_bib25","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.215","article-title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Johnson","year":"2017"},{"key":"2022110820595817400_bib26","first-page":"2668","article-title":"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)","volume-title":"International Conference on Machine Learning","author":"Kim","year":"2018"},{"key":"2022110820595817400_bib27","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-main.731","article-title":"COGS: A compositional generalization challenge based on semantic interpretation","volume-title":"Proceedings of EMNLP","author":"Kim","year":"2020"},{"key":"2022110820595817400_bib28","unstructured":"Diederik P.\n              Kingma\n             and JimmyBa. 2014. Adam: A method for stochastic optimization."},{"key":"2022110820595817400_bib29","article-title":"Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks","volume-title":"Proceedings of ICML","author":"Lake","year":"2018"},{"key":"2022110820595817400_bib30","doi-asserted-by":"crossref","first-page":"8595","DOI":"10.1109\/ICASSP.2013.6639343","article-title":"Building high-level features using large scale unsupervised learning","volume-title":"2013 IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Le","year":"2013"},{"key":"2022110820595817400_bib31","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1146\/annurev-linguistics-030514-125254","article-title":"Distributional models of word meaning","volume":"4","author":"Lenci","year":"2018","journal-title":"Annual Review of Linguistics"},{"key":"2022110820595817400_bib32","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1146\/annurev-linguistics-032020-051035","article-title":"Syntactic structure from deep learning","volume":"7","author":"Linzen","year":"2021","journal-title":"Annual Review of Linguistics"},{"key":"2022110820595817400_bib33","article-title":"Predicting inductive biases of pre-trained models","volume-title":"International Conference on Learning Representations","author":"Lovering","year":"2020"},{"key":"2022110820595817400_bib34","volume-title":"Concepts: Core Readings","author":"Margolis","year":"1999"},{"key":"2022110820595817400_bib35","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262514620.001.0001","article-title":"Vision: A Computational Investigation into the Human Representation and Processing of Visual Information","author":"Marr","year":"2010"},{"key":"2022110820595817400_bib36","article-title":"Locating and editing factual knowledge in GPT","author":"Meng","year":"2022","journal-title":"arXiv preprint arXiv:2202.05262"},{"key":"2022110820595817400_bib37","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00412","article-title":"Provable limitations of acquiring meaning from ungrounded form: What will future language models understand?","author":"Merrill","year":"2021","journal-title":"arXiv preprint arXiv:2104.10809"},{"key":"2022110820595817400_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.106","article-title":"Coloring the blank slate: Pre-training imparts a hierarchical inductive bias to sequence-to- sequence models","author":"Mueller","year":"2022","journal-title":"arXiv preprint arXiv:2203 .09397"},{"key":"2022110820595817400_bib39","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1146\/annurev-linguistics-031120-122924","article-title":"Semantic structure in deep learning","volume":"8","author":"Pavlick","year":"2022","journal-title":"Annual Review of Linguistics"},{"key":"2022110820595817400_bib40","doi-asserted-by":"publisher","first-page":"8748","DOI":"10.48550\/arXiv.2103.00020","article-title":"Learning transferable visual models from natural language supervision","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Radford","year":"2021"},{"key":"2022110820595817400_bib41","doi-asserted-by":"publisher","first-page":"7237","DOI":"10.18653\/v1\/2020.acl-main.647","article-title":"Null it out: Guarding protected attributes by iterative nullspace projection","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ravfogel","year":"2020"},{"key":"2022110820595817400_bib42","doi-asserted-by":"publisher","first-page":"1135","DOI":"10.1145\/2939672.2939778","article-title":"\u201cWhy should I trust you?\u201d Explaining the predictions of any classifier","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Ribeiro","year":"2016"},{"key":"2022110820595817400_bib43","doi-asserted-by":"publisher","first-page":"856","DOI":"10.18653\/v1\/P18-1079","article-title":"Semantically equivalent adversarial rules for debugging NLP models","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ribeiro","year":"2018"},{"key":"2022110820595817400_bib44","doi-asserted-by":"publisher","first-page":"4902","DOI":"10.18653\/v1\/2020.acl-main.442","article-title":"Beyond accuracy: Behavioral testing of NLP models with CheckList","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ribeiro","year":"2020"},{"key":"2022110820595817400_bib45","article-title":"Overfeat: Integrated recognition, localization and detection using convolutional networks","author":"Sermanet","year":"2013","journal-title":"arXiv preprint arXiv:1312.6229"},{"key":"2022110820595817400_bib46","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.230","article-title":"Masked language modeling and the distributional hypothesis: Order word matters pre-training for little","author":"Sinha","year":"2021","journal-title":"arXiv preprint arXiv:2104.06644"},{"key":"2022110820595817400_bib47","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780195183115.001.0001","volume-title":"Causal Models: How People Think About the World and its Alternatives","author":"Sloman","year":"2005"},{"issue":"1","key":"2022110820595817400_bib48","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1111\/j.1467-7687.2007.00569.x","article-title":"Core knowledge","volume":"10","author":"Spelke","year":"2007","journal-title":"Developmental Science"},{"key":"2022110820595817400_bib49","first-page":"3319","article-title":"Axiomatic attribution for deep networks","volume-title":"International Conference on Machine Learning","author":"Sundararajan","year":"2017"},{"key":"2022110820595817400_bib50","article-title":"What do you learn from context? Probing for sentence structure in contextualized word representations","volume-title":"International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2022110820595817400_bib51","doi-asserted-by":"publisher","first-page":"862","DOI":"10.18653\/v1\/2021.findings-acl.76","article-title":"What if this modified that? Syntactic interventions with counterfactual embeddings","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Tucker","year":"2021"},{"key":"2022110820595817400_bib52","article-title":"Diagnostic classifiers: Revealing how neural networks process hierarchical structure","volume-title":"CEUR Workshop Proceedings","author":"Veldhoen","year":"2016"},{"key":"2022110820595817400_bib53","first-page":"6","article-title":"Investigating gender bias in language models using causal mediation analysis","volume":"33","author":"Vig","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022110820595817400_bib54","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.16","article-title":"Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually)","author":"Warstadt","year":"2020","journal-title":"arXiv preprint arXiv: 2010.05358"},{"key":"2022110820595817400_bib55","article-title":"Measuring and reducing gendered correlations in pre-trained models","author":"Webster","year":"2020","journal-title":"arXiv preprint arXiv:2010.06032"},{"key":"2022110820595817400_bib56","doi-asserted-by":"crossref","first-page":"11","DOI":"10.18653\/v1\/D19-1002","article-title":"Attention is not not explanation","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Wiegreffe","year":"2019"},{"key":"2022110820595817400_bib57","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-4804","article-title":"Can neural networks understand monotonicity reasoning?","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP","author":"Yanaka","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00514\/2057233\/tacl_a_00514.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00514\/2057233\/tacl_a_00514.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,8]],"date-time":"2022-11-08T21:00:31Z","timestamp":1667941231000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00514\/113663\/Unit-Testing-for-Concepts-in-Neural-Networks"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":57,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00514","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}