{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T19:38:36Z","timestamp":1782934716090,"version":"3.54.5"},"reference-count":71,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,10,2]],"date-time":"2024-10-02T00:00:00Z","timestamp":1727827200000},"content-version":"vor","delay-in-days":275,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Large-scale pretrained language models (LMs) are said to \u201clack the ability to connect utterances to the world\u201d (Bender and Koller, 2020), because they do not have \u201cmental models of the world\u201d (Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT, and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy, and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).1<\/jats:p>","DOI":"10.1162\/tacl_a_00698","type":"journal-article","created":{"date-parts":[[2024,10,2]],"date-time":"2024-10-02T17:17:50Z","timestamp":1727889470000},"page":"1232-1249","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":4,"title":["Do Vision and Language Models Share Concepts? A Vector Space Alignment Study"],"prefix":"10.1162","volume":"12","author":[{"given":"Jiaang","family":"Li","sequence":"first","affiliation":[{"name":"University of Copenhagen, Denmark. jili@di.ku.dk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yova","family":"Kementchedjhieva","sequence":"additional","affiliation":[{"name":"Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates. yova.kementchedjhieva@mbzuai.ac.ae"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Constanza","family":"Fierro","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Denmark. c.fierro@di.ku.dk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anders","family":"S\u00f8gaard","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Denmark. soegaard@di.ku.dk"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,9,30]]},"reference":[{"key":"2024100217173473200_bib1","doi-asserted-by":"publisher","first-page":"109","DOI":"10.18653\/v1\/2021.conll-1.9","article-title":"Can language models encode perceptual structure without grounding? A case study in color","volume-title":"Proceedings of the 25th Conference on Computational Natural Language Learning","author":"Abdou","year":"2021"},{"key":"2024100217173473200_bib2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/nol_a_00087","article-title":"Predictive coding or just feature discovery? an alternative account of why language models fit brain data","author":"Antonello","year":"2022","journal-title":"Neurobiology of Language"},{"key":"2024100217173473200_bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1073","article-title":"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings","volume-title":"ACL","author":"Artetxe","year":"2018"},{"key":"2024100217173473200_bib4","doi-asserted-by":"publisher","first-page":"5185","DOI":"10.18653\/v1\/2020.acl-main.463","article-title":"Climbing towards NLU: On meaning, form, and understanding in the age of data","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Bender","year":"2020"},{"key":"2024100217173473200_bib5","first-page":"1764","article-title":"Learning bilingual lexicons using the visual similarity of labeled web images","volume-title":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three","author":"Bergsma","year":"2011"},{"key":"2024100217173473200_bib6","volume-title":"Natural language processing with Python: Analyzing text with the natural language toolkit","author":"Bird","year":"2009"},{"issue":"1","key":"2024100217173473200_bib7","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1111\/j.1746-8361.2004.tb00293.x","article-title":"Intuition pumps and the proper use of thought experiments","volume":"58","author":"Brendel","year":"2004","journal-title":"Dialectica"},{"key":"2024100217173473200_bib8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10670-021-00491-w","article-title":"Sharing our concepts with machines","author":"Butlin","year":"2021","journal-title":"Erkenntnis"},{"key":"2024100217173473200_bib9","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780192894724.001.0001","volume-title":"Making AI Intelligible: Philosophical Foundations","author":"Cappelen","year":"2021"},{"key":"2024100217173473200_bib10","doi-asserted-by":"publisher","first-page":"9650","DOI":"10.1109\/ICCV48922.2021.00951","article-title":"Emerging properties in self-supervised vision transformers","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Caron","year":"2021"},{"key":"2024100217173473200_bib11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2111.14232","article-title":"Long-range and hierarchical language predictions in brains and algorithms","author":"Caucheteux","year":"2022","journal-title":"Nature Human Behaviour"},{"key":"2024100217173473200_bib12","doi-asserted-by":"publisher","DOI":"10.1038\/s42003-022-03036-1","article-title":"Brains and algorithms partially converge in natural language processing","author":"Caucheteux","year":"2022","journal-title":"Communications Biology"},{"key":"2024100217173473200_bib13","article-title":"Word translation without parallel data","volume-title":"Proceedings of ICLR 2018","author":"Conneau","year":"2018"},{"key":"2024100217173473200_bib14","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2024100217173473200_bib15","article-title":"An image is worth 16x16 words: Transformers for image recognition at scale","volume-title":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\u20137, 2021","author":"Dosovitskiy","year":"2021"},{"key":"2024100217173473200_bib16","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/978-90-481-8847-5_10","article-title":"Wordnet","volume-title":"Theory and Applications of Ontology: Computer Applications","author":"Fellbaum","year":"2010"},{"key":"2024100217173473200_bib17","unstructured":"Wikimedia Foundation. Wikimedia downloads."},{"issue":"14","key":"2024100217173473200_bib18","doi-asserted-by":"publisher","first-page":"12884","DOI":"10.1609\/aaai.v35i14.17524","article-title":"Analogy training multilingual encoders","volume":"35","author":"Garneau","year":"2021","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024100217173473200_bib19","doi-asserted-by":"publisher","first-page":"7548","DOI":"10.18653\/v1\/2020.acl-main.675","article-title":"Non-linear instance-based cross-lingual mapping for non-isomorphic embedding spaces","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Glava\u0161","year":"2020"},{"key":"2024100217173473200_bib20","doi-asserted-by":"publisher","DOI":"10.1101\/2020.12.02.403477","article-title":"Thinking ahead: Spontaneous prediction in context as a keystone of language in humans and machines","author":"Goldstein","year":"2021","journal-title":"bioRxiv"},{"key":"2024100217173473200_bib21","doi-asserted-by":"publisher","DOI":"10.1101\/2022.03.17.484712","article-title":"The combination of hebbian and predictive plasticity learns invariant object representations in deep sensory networks","author":"Halvagal","year":"2022","journal-title":"bioRxiv"},{"key":"2024100217173473200_bib22","doi-asserted-by":"publisher","first-page":"582","DOI":"10.18653\/v1\/D18-1056","article-title":"Why is unsupervised alignment of English embeddings from different algorithms so hard?","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Hartmann","year":"2018"},{"key":"2024100217173473200_bib23","doi-asserted-by":"publisher","first-page":"159","DOI":"10.18653\/v1\/W18-3021","article-title":"Limitations of cross-lingual learning from image search","volume-title":"Proceedings of The Third Workshop on Representation Learning for NLP","author":"Hartmann","year":"2018"},{"key":"2024100217173473200_bib24","doi-asserted-by":"publisher","first-page":"16000","DOI":"10.1109\/CVPR52688.2022.01553","article-title":"Masked autoencoders are scalable vision learners","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"He","year":"2022"},{"key":"2024100217173473200_bib25","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2024100217173473200_bib26","doi-asserted-by":"publisher","first-page":"1801.06126","DOI":"10.18653\/v1\/D18-1043","article-title":"An iterative closest point method for unsupervised word translation","volume-title":"CoRR","author":"Hoshen","year":"2018"},{"key":"2024100217173473200_bib27","article-title":"The platonic representation hypothesis","author":"Huh","year":"2024","journal-title":"arXiv preprint arXiv:2405.07987"},{"key":"2024100217173473200_bib28","doi-asserted-by":"publisher","first-page":"36","DOI":"10.3115\/v1\/D14-1005","article-title":"Learning image embeddings using convolutional neural networks for improved multi-modal semantics","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kiela","year":"2014"},{"key":"2024100217173473200_bib29","doi-asserted-by":"publisher","first-page":"148","DOI":"10.18653\/v1\/D15-1015","article-title":"Visual bilingual lexicon induction with transferred ConvNet features","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Kiela","year":"2015"},{"key":"2024100217173473200_bib30","doi-asserted-by":"publisher","first-page":"1403","DOI":"10.3115\/v1\/P14-1132","article-title":"Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lazaridou","year":"2014"},{"key":"2024100217173473200_bib31","article-title":"Structural similarities between language models and neural response measurements","volume-title":"NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations","author":"Li","year":"2023"},{"issue":"4","key":"2024100217173473200_bib32","doi-asserted-by":"publisher","first-page":"553","DOI":"10.5840\/monist199881427","article-title":"Stepping back inside Leibniz\u2019s mill","volume":"81","author":"Lodge","year":"1998","journal-title":"The Monist"},{"key":"2024100217173473200_bib33","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00522","article-title":"Do language models refer?","author":"Mandelkern","year":"2023"},{"issue":"48","key":"2024100217173473200_bib34","doi-asserted-by":"publisher","first-page":"30046","DOI":"10.1073\/pnas.1907367117","article-title":"Emergent linguistic structure in artificial neural networks trained by self-supervision","volume":"117","author":"Manning","year":"2020","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024100217173473200_bib35","volume-title":"Lexical Competence","author":"Marconi","year":"1997"},{"key":"2024100217173473200_bib36","article-title":"A sentence is worth a thousand pictures: Can large language models understand human language?","author":"Marcus","year":"2023"},{"key":"2024100217173473200_bib37","article-title":"Linearly mapping from image to text space","volume-title":"The Eleventh International Conference on Learning Representations","author":"Merullo","year":"2023"},{"key":"2024100217173473200_bib38","doi-asserted-by":"publisher","first-page":"155","DOI":"10.18653\/v1\/P19-2021","article-title":"From brain space to distributional space: The perilous journeys of fMRI decoding","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop","author":"Minnema","year":"2019"},{"issue":"13","key":"2024100217173473200_bib39","doi-asserted-by":"publisher","first-page":"e2215907120","DOI":"10.1073\/pnas.2215907120","article-title":"The debate over understanding in AI\u2019s large language models","volume":"120","author":"Mitchell","year":"2023","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024100217173473200_bib40","article-title":"The vector grounding problem","author":"Mollo","year":"2023"},{"key":"2024100217173473200_bib41","doi-asserted-by":"publisher","first-page":"512","DOI":"10.18653\/v1\/D18-1047","article-title":"NORMA: Neighborhood sensitive maps for multilingual word embeddings","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Nakashole","year":"2018"},{"key":"2024100217173473200_bib42","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/j.artint.2012.07.001","article-title":"Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network","volume":"193","author":"Navigli","year":"2012","journal-title":"Artificial Intelligence"},{"key":"2024100217173473200_bib43","first-page":"9960","article-title":"Self-supervised learning through the eyes of a child","volume-title":"Advances in Neural Information Processing Systems","author":"Orhan","year":"2020"},{"key":"2024100217173473200_bib44","first-page":"8024","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume-title":"Advances in Neural Information Processing Systems 32","author":"Paszke","year":"2019"},{"key":"2024100217173473200_bib45","article-title":"Meaning without reference in large language models","volume-title":"NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI)","author":"Piantadosi","year":"2022"},{"key":"2024100217173473200_bib46","first-page":"8748","article-title":"Learning transferable visual models from natural language supervision","volume-title":"International Conference on Machine Learning","author":"Radford","year":"2021"},{"key":"2024100217173473200_bib47","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019"},{"issue":"1","key":"2024100217173473200_bib48","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1023\/a:1013765011735","article-title":"Holism, conceptual- role semantics, and syntactic semantics","volume":"12","author":"Rapaport","year":"2002","journal-title":"Minds and Machines"},{"issue":"3","key":"2024100217173473200_bib49","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"International Journal of Computer Vision (IJCV)"},{"issue":"682578","key":"2024100217173473200_bib50","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.682578","article-title":"The singleton fallacy: Why current critiques of language models miss the point","volume":"4","author":"Sahlgren","year":"2021","journal-title":"Frontiers in Artificial Intelligence"},{"issue":"1","key":"2024100217173473200_bib51","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1162\/nol_a_00003","article-title":"Traces of meaning itself: Encoding distributional word vectors in brain activity","volume":"1","author":"Sassenhagen","year":"2020","journal-title":"Neurobiology of Language"},{"issue":"1","key":"2024100217173473200_bib52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF02289451","article-title":"A generalized solution of the orthogonal procrustes problem","volume":"31","author":"Sch\u00f6nemann","year":"1966","journal-title":"Psychometrika"},{"key":"2024100217173473200_bib53","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2105646118","article-title":"The neural architecture of language: Integrative modeling converges on predictive processing","author":"Schrimpf","year":"2021","journal-title":"bioRxiv"},{"key":"2024100217173473200_bib54","doi-asserted-by":"publisher","DOI":"10.1101\/407007","article-title":"Brain-score: Which artificial neural network for object recognition is most brain-like?","author":"Schrimpf","year":"2018","journal-title":"bioRxiv"},{"key":"2024100217173473200_bib55","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1017\/S0140525X00005756","article-title":"Minds, brains, and programs","volume":"3","author":"Searle","year":"1980","journal-title":"Behavioral and Brain Sciences"},{"key":"2024100217173473200_bib56","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780198812883.001.0001","volume-title":"Representation in Cognitive Science","author":"Shea","year":"2018"},{"key":"2024100217173473200_bib57","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1072","volume-title":"On the limitations of unsupervised bilingual dictionary induction","author":"S\u00f8gaard","year":"2018"},{"key":"2024100217173473200_bib58","doi-asserted-by":"publisher","first-page":"146","DOI":"10.18653\/v1\/2022.bigscience-1.11","article-title":"Emergent structures and training dynamics in large language models","volume-title":"Proceedings of BigScience Episode #5 \u2013 Workshop on Challenges & Perspectives in CreatingLarge Language Models","author":"Teehan","year":"2022"},{"key":"2024100217173473200_bib59","article-title":"Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)","volume":"32","author":"Toneva","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024100217173473200_bib60","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2307.09288"},{"key":"2024100217173473200_bib61","article-title":"Well-read students learn better: On the importance of pre-training compact models","author":"Turc","year":"2019","journal-title":"arXiv preprint arXiv:1908.08962v2"},{"key":"2024100217173473200_bib62","doi-asserted-by":"publisher","first-page":"188","DOI":"10.18653\/v1\/P16-2031","article-title":"Multi-modal representations for improved bilingual lexicon learning","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Vuli\u0107","year":"2016"},{"key":"2024100217173473200_bib63","article-title":"Emergent abilities of large language models","author":"Wei","year":"2022","journal-title":"Transactions on Machine Learning Research"},{"issue":"1","key":"2024100217173473200_bib64","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1007\/s11023-017-9441-6","article-title":"Predictive processing and the representation wars","volume":"28","author":"Williams","year":"2018","journal-title":"Minds and Machines"},{"key":"2024100217173473200_bib65","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2024100217173473200_bib66","first-page":"12077","article-title":"Segformer: Simple and efficient design for semantic segmentation with transformers","volume-title":"Advances in Neural Information Processing Systems","author":"Xie","year":"2021"},{"key":"2024100217173473200_bib67","article-title":"Opt: Open pre-trained transformer language models","author":"Zhang","year":"2022"},{"key":"2024100217173473200_bib68","first-page":"3583","article-title":"Non-linearity in mapping based cross-lingual word embeddings","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Zhao","year":"2020"},{"key":"2024100217173473200_bib69","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.544","article-title":"Scene parsing through ade20k dataset","author":"Zhou","year":"2017","journal-title":"Computer Vision and Pattern Recognition"},{"key":"2024100217173473200_bib70","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1109\/ICCV.2015.11","article-title":"Aligning books and movies: Towards story-like visual explanations by watching movies and reading books","author":"Zhu","year":"2015","journal-title":"2015 IEEE International Conference on Computer Vision (ICCV)"},{"key":"2024100217173473200_bib71","article-title":"Representation engineering: A top-down approach to ai transparency","author":"Zou","year":"2023","journal-title":"arXiv preprint arXiv:2310.01405"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00698\/2473674\/tacl_a_00698.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00698\/2473674\/tacl_a_00698.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,2]],"date-time":"2024-10-02T17:17:58Z","timestamp":1727889478000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00698\/124631\/Do-Vision-and-Language-Models-Share-Concepts-A"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":71,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00698","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}