{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T13:57:56Z","timestamp":1781704676175,"version":"3.54.5"},"reference-count":60,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T00:00:00Z","timestamp":1781654400000},"content-version":"vor","delay-in-days":167,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>To what degree and under what conditions do VLMs rely on scene context when generating references to objects? To address this question, we introduce the Common Objects Out-of-Context (COOCo) dataset and conduct experiments on several VLMs under different degrees of scene\u2013object congruency and noise. We find that models leverage scene context adaptively, depending on scene-object semantic relatedness and noise level. Based on these consistent trends across models, we turn to the question of how VLM attention patterns change as a function of target-scene semantic fit, and to what degree these patterns are predictive of categorisation accuracy. We find that successful object categorisation is associated with increased mid-layer attention to the target. We also find a non-monotonic dependency on semantic fit, with attention dropping at moderate fit and increasing for both low and high fit. These results suggest that VLMs dynamically balance local and contextual information for reference generation.<\/jats:p>\n                  <jats:p>Dataset and code are available here: https:\/\/github.com\/cs-nlp-uu\/scenereg.<\/jats:p>","DOI":"10.1162\/tacl.a.701","type":"journal-article","created":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T13:28:04Z","timestamp":1781702884000},"page":"1163-1185","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication"],"prefix":"10.1162","volume":"14","author":[{"given":"Filippo","family":"Merlo","sequence":"first","affiliation":[{"name":"Utrecht University, f.merlo.research@gmail.com"},{"name":"University of Trento"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ece","family":"Takmaz","sequence":"additional","affiliation":[{"name":"Utrecht University, e.k.takmaz@uu.nl"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenkai","family":"Chen","sequence":"additional","affiliation":[{"name":"Utrecht University, w.chen5@students.uu.nl"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Albert","family":"Gatt","sequence":"additional","affiliation":[{"name":"Utrecht University, a.gatt@uu.nl"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,6,1]]},"reference":[{"issue":"1","key":"2026061709275974300_bib1","doi-asserted-by":"crossref","first-page":"42","DOI":"10.3758\/s13428-024-02535-9","article-title":"The Visual Integration of Semantic and Spatial Information of Objects in Naturalistic Scenes (VISIONS) database: attentional, conceptual, and perceptual norms","volume":"57","author":"Allegretti","year":"2025","journal-title":"Behavior Research Methods"},{"key":"2026061709275974300_bib2","volume-title":"Qwen2.5-VL Technical Report","author":"Bai","year":"2025"},{"issue":"8","key":"2026061709275974300_bib3","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1038\/nrn1476","article-title":"Visual objects in context","volume":"5","author":"Bar","year":"2004","journal-title":"Nature Reviews Neuroscience"},{"issue":"2","key":"2026061709275974300_bib4","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/0010-0285(82)90007-X","article-title":"Scene perception: Detecting and judging objects undergoing relational violations","volume":"14","author":"Biederman","year":"1982","journal-title":"Cognitive Psychology"},{"issue":"13","key":"2026061709275974300_bib5","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1167\/18.13.11","article-title":"Anchoring visual search in scenes: Assessing the role of anchor objects on eye movements during visual search","volume":"18","author":"Boettcher","year":"2018","journal-title":"Journal of Vision"},{"key":"2026061709275974300_bib6","volume-title":"When Pigs Fly: Contextual Reasoning in Synthetic and Natural Scenes","author":"Bomatter","year":"2021"},{"key":"2026061709275974300_bib7","volume-title":"WebQA: Multihop and Multimodal QA","author":"Chang","year":"2022"},{"issue":"1","key":"2026061709275974300_bib8","doi-asserted-by":"crossref","first-page":"8776","DOI":"10.1038\/s41598-021-87715-9","article-title":"COCO-Search18 fixation dataset for predicting goal-directed attention control","volume":"11","author":"Chen","year":"2021","journal-title":"Scientific Reports"},{"key":"2026061709275974300_bib9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3389\/fpsyg.2013.00927","article-title":"The impact of attentional, linguistic, and visual features during object naming","volume":"4","author":"Clarke","year":"2013","journal-title":"Frontiers in Psychology"},{"issue":"8","key":"2026061709275974300_bib10","doi-asserted-by":"crossref","first-page":"1995","DOI":"10.1111\/cogs.12313","article-title":"Anticipation in real-world scenes: The role of visual context and visual memory","volume":"40","author":"Coco","year":"2016","journal-title":"Cognitive Science"},{"issue":"6","key":"2026061709275974300_bib11","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1177\/09567976241238217","article-title":"Exploring the Semantic-Inconsistency Effect in Scenes Using a Continuous Measure of Linguistic-Semantic Similarity","volume":"35","author":"Damiano","year":"2024","journal-title":"Psychological Science"},{"key":"2026061709275974300_bib12","volume-title":"Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models","author":"Deitke","year":"2024"},{"key":"2026061709275974300_bib13","volume-title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","author":"Dosovitskiy","year":"2021"},{"issue":"7-8","key":"2026061709275974300_bib14","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1080\/02643294.2019.1656604","article-title":"Semantic memory for objects, actions, and events: A novel test of event-related conceptual semantic knowledge","volume":"36","author":"Dresang","year":"2019","journal-title":"Cognitive Neuropsychology"},{"key":"2026061709275974300_bib15","doi-asserted-by":"crossref","first-page":"418","DOI":"10.3389\/fpsyg.2017.00418","article-title":"Preferential processing of social features and their interplay with physical saliency in complex naturalistic scenes","volume":"8","author":"End","year":"2017","journal-title":"Frontiers in Psychology"},{"issue":"2","key":"2026061709275974300_bib16","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/S0926-6410(02)00244-6","article-title":"An electrophysiological study of scene effects on object identification","volume":"16","author":"Ganis","year":"2003","journal-title":"Cognitive Brain Research"},{"key":"2026061709275974300_bib17","doi-asserted-by":"crossref","first-page":"11462","DOI":"10.18653\/v1\/2025.naacl-long.571","article-title":"What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation","volume-title":"Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Golovanevsky","year":"2025"},{"key":"2026061709275974300_bib18","article-title":"When Meaning Matters: The Temporal Dynamics of Semantic Influences on Visual Attention","volume-title":"Journal of Experimental Psychology: Human Perception and Performance","author":"de Groot","year":"2015"},{"issue":"0","key":"2026061709275974300_bib19","first-page":"1","article-title":"Language-induced visual and semantic biases in visual search are subject to task requirements","volume":"0","author":"de Groot","year":"2017","journal-title":"Visual Cognition"},{"issue":"8","key":"2026061709275974300_bib20","doi-asserted-by":"crossref","first-page":"1262","DOI":"10.1177\/0956797621994768","article-title":"Looking for semantic similarity: What a vector-space model of semantics can tell us about attention in real-world scenes","volume":"32","author":"Hayes","year":"2021","journal-title":"Psychological Science"},{"issue":"10","key":"2026061709275974300_bib21","doi-asserted-by":"crossref","first-page":"e0223792","DOI":"10.1371\/journal.pone.0223792","article-title":"THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images","volume":"14","author":"Hebart","year":"2019","journal-title":"PLOS ONE"},{"issue":"2","key":"2026061709275974300_bib22","doi-asserted-by":"crossref","first-page":"19","DOI":"10.3390\/vision3020019","article-title":"Meaning and Attentional Guidance in Scenes: A Review of the Meaning Map Approach","volume":"3","author":"Henderson","year":"2019","journal-title":"Vision"},{"key":"2026061709275974300_bib23","volume-title":"CLIPScore: A Reference-free Evaluation Metric for Image Captioning","author":"Hessel","year":"2022"},{"issue":"10","key":"2026061709275974300_bib24","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1016\/j.visres.2011.03.010","article-title":"Semantic guidance of eye movements in real-world scenes","volume":"51","author":"Hwang","year":"2011","journal-title":"Vision Research"},{"key":"2026061709275974300_bib25","volume-title":"Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference","author":"Jacob","year":"2017"},{"key":"2026061709275974300_bib26","doi-asserted-by":"crossref","first-page":"344","DOI":"10.18653\/v1\/2024.inlg-main.29","article-title":"Resilience through scene context in visual referring expression generation","volume-title":"Proceedings of the 17th International Natural Language Generation Conference","author":"Junker","year":"2024"},{"key":"2026061709275974300_bib27","article-title":"Scene-Gram: Conceptualizing and Describing Tangrams in Scene Context","volume-title":"arXiv preprint arXiv:2506.11631","author":"Junker","year":"2025"},{"key":"2026061709275974300_bib28","doi-asserted-by":"crossref","first-page":"787","DOI":"10.3115\/v1\/D14-1086","article-title":"ReferItGame: Referring to Objects in Photographs of Natural Scenes","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kazemzadeh","year":"2014"},{"issue":"1","key":"2026061709275974300_bib29","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1162\/COLI_a_00088","article-title":"Computational Generation of Referring Expressions: A Survey","volume":"38","author":"Krahmer","year":"2012","journal-title":"Computational Linguistics"},{"key":"2026061709275974300_bib30","volume-title":"Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations","author":"Krishna","year":"2016"},{"issue":"1","key":"2026061709275974300_bib31","doi-asserted-by":"crossref","first-page":"14666","DOI":"10.1038\/s41598-018-32991-1","article-title":"The role of scene summary statistics in object recognition","volume":"8","author":"Lauer","year":"2018","journal-title":"Scientific Reports"},{"issue":"1","key":"2026061709275974300_bib32","doi-asserted-by":"crossref","first-page":"21988","DOI":"10.1038\/s41598-021-01406-z","article-title":"The role of contextual materials in object recognition","volume":"11","author":"Lauer","year":"2021","journal-title":"Scientific Reports"},{"key":"2026061709275974300_bib33","article-title":"LLaVA-onevision: Easy visual task transfer","volume-title":"Transactions on Machine Learning Research","author":"Li","year":"2025"},{"key":"2026061709275974300_bib34","volume-title":"Microsoft COCO: Common Objects in Context","author":"Lin","year":"2015"},{"key":"2026061709275974300_bib35","volume-title":"Visual Instruction Tuning","author":"Liu","year":"2023"},{"key":"2026061709275974300_bib36","doi-asserted-by":"crossref","first-page":"1780","DOI":"10.3758\/s13428-016-0820-3","article-title":"Scegram: An image database for semantic and syntactic inconsistencies in scenes","volume":"49","author":"\u00d6hlschl\u00e4ger","year":"2016","journal-title":"Behavior Research Methods"},{"issue":"12","key":"2026061709275974300_bib37","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1016\/j.tics.2007.09.009","article-title":"The role of context in object recognition","volume":"11","author":"Oliva","year":"2007","journal-title":"Trends in Cognitive Sciences"},{"issue":"1","key":"2026061709275974300_bib38","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1038\/s44159-023-00254-0","article-title":"Predictive processing of scenes and objects","volume":"3","author":"Peelen","year":"2023","journal-title":"Nature Reviews Psychology"},{"key":"2026061709275974300_bib39","article-title":"Grounding multimodal large language models to the world","volume-title":"The Twelfth International Conference on Learning Representations","author":"Peng","year":"2024"},{"key":"2026061709275974300_bib40","volume-title":"Learning Transferable Visual Models From Natural Language Supervision","author":"Radford","year":"2021"},{"key":"2026061709275974300_bib41","doi-asserted-by":"crossref","first-page":"1067125","DOI":"10.3389\/frai.2023.1067125","article-title":"Rethinking symbolic and visual context in Referring Expression Generation","volume":"6","author":"Sch\u00fcz","year":"2023","journal-title":"Frontiers in Artificial Intelligence"},{"key":"2026061709275974300_bib42","first-page":"20","article-title":"Keeping an Eye on Context: Attention Allocation over Input Partitions in Referring Expression Generation","volume-title":"Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023)","author":"Sch\u00fcz","year":"2023"},{"issue":"5","key":"2026061709275974300_bib43","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.3758\/s13428-021-01540-6","article-title":"You won\u2019t believe what this guy is doing with the potato: The ObjAct stimulusset depicting human actions on congruent and incongruent objects","volume":"53","author":"Shir","year":"2021","journal-title":"Behavior Research Methods"},{"key":"2026061709275974300_bib44","volume-title":"ConceptNet 5.5: An Open Multilingual Graph of General Knowledge","author":"Speer","year":"2018"},{"key":"2026061709275974300_bib45","doi-asserted-by":"crossref","DOI":"10.3758\/s13428-023-02110-8","article-title":"THINGSplus: New norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images","volume-title":"Behavior Research Methods","author":"Stoinski","year":"2023"},{"key":"2026061709275974300_bib46","volume-title":"Resolution-robust Large Mask Inpainting with Fourier Convolutions","author":"Suvorov","year":"2021"},{"issue":"4","key":"2026061709275974300_bib47","doi-asserted-by":"crossref","first-page":"766","DOI":"10.1037\/0033-295X.113.4.766","article-title":"Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search","volume":"113","author":"Torralba","year":"2006","journal-title":"Psychological Review"},{"key":"2026061709275974300_bib48","doi-asserted-by":"crossref","first-page":"63","DOI":"10.18653\/v1\/W19-4808","article-title":"Analyzing the Structure of Attention in a Transformer Language Model","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Vig","year":"2019"},{"key":"2026061709275974300_bib49","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.visres.2020.11.003","article-title":"The meaning and structure of scenes","volume":"181","author":"V\u00f5","year":"2021","journal-title":"Vision Research"},{"key":"2026061709275974300_bib50","doi-asserted-by":"crossref","first-page":"3485","DOI":"10.1109\/CVPR.2010.5539970","article-title":"SUN database: Large-scale scene recognition from abbey to zoo","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition","author":"Xiao","year":"2010"},{"key":"2026061709275974300_bib51","volume-title":"xgen-mm (blip-3): A family of open large multimodal models","author":"Xue","year":"2024"},{"key":"2026061709275974300_bib52","volume-title":"Common Inpainted Objects In-N-Out of Context","author":"Yang","year":"2025"},{"key":"2026061709275974300_bib53","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-46475-6_5","article-title":"Modeling Context in Referring Expressions","volume-title":"ECCV","author":"Yu","year":"2016"},{"key":"2026061709275974300_bib54","doi-asserted-by":"crossref","first-page":"14319","DOI":"10.1109\/ACCESS.2021.3051964","article-title":"Cut-and-Paste Dataset Generation for Balancing Domain Gaps in Object Instance Detection","volume":"9","author":"Yun","year":"2021","journal-title":"IEEE Access"},{"key":"2026061709275974300_bib55","doi-asserted-by":"crossref","DOI":"10.1109\/ICCV51070.2023.01100","volume-title":"Sigmoid Loss for Language Image Pre-Training","author":"Zhai","year":"2023"},{"key":"2026061709275974300_bib56","volume-title":"MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs","author":"Zhang","year":"2025"},{"key":"2026061709275974300_bib57","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52734.2025.01842","volume-title":"Cross-modal Information Flow in Multimodal Large Language Models","author":"Zhang","year":"2025"},{"key":"2026061709275974300_bib58","volume-title":"A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting","author":"Zhuang","year":"2024"},{"key":"2026061709275974300_bib59","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3588015.3588415","article-title":"TF-IDF based Scene-Object Relations Correlate With Visual Attention","volume-title":"2023 Symposium on Eye Tracking Research and Applications","author":"\u00c7elikkol","year":"2023"},{"issue":"5","key":"2026061709275974300_bib60","doi-asserted-by":"crossref","first-page":"1780","DOI":"10.3758\/s13428-016-0820-3","article-title":"SCEGRAM: An image database for semantic and syntactic inconsistencies in scenes","volume":"49","author":"\u00d6hlschl\u00e4ger","year":"2017","journal-title":"Behavior Research Methods"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.701\/2606841\/tacl.a.701.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.701\/2606841\/tacl.a.701.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T13:28:14Z","timestamp":1781702894000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.701\/137201\/Common-Objects-Out-of-Context-COOCo-Investigating"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":60,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.701","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}