{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T13:32:05Z","timestamp":1787146325208,"version":"build-2736575974"},"reference-count":19,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T00:00:00Z","timestamp":1787097600000},"content-version":"vor","delay-in-days":230,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"DARPA Machine Common Sense"},{"name":"MIT-IBM AI LAB"},{"name":"NSF STC","award":["CCF-1231216"],"award-info":[{"award-number":["CCF-1231216"]}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,17]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>People naturally take into account others\u2019 limited perceptual access when attributing mental states such as preferences and knowledge. This ability is central to intuitive psychology, and even infants recognize when others have only partial observability. Understanding how mental state attributions adapt under limited perceptual access is thus crucial for advancing models of intuitive psychology. Here, we introduce POPKI (Physically-grounded Observation, Preference, and Knowledge Inference), a Bayesian inverse-planning framework designed to model graded surprise based on agents\u2019 inferred preferences and knowledge in visually constrained 3D environments. Building on and expanding the AGENT dataset, we develop new scenarios that test how partial observability shapes judgments of surprise. Despite the tremendous progress of state-of-the-art vision-language models (VLMs), our experiments show that they fail to capture the nuances of perceptual access when judging surprise. POPKI, in contrast, offers a principled approach to modeling human-like uncertainty under partial observability and closely replicates human surprise patterns. It provides a foundation for advancing psychological reasoning models and enhancing human-AI interaction.<\/jats:p>","DOI":"10.1162\/opmi.a.375","type":"journal-article","created":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T13:21:24Z","timestamp":1787145684000},"page":"1157-1170","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["POPKI: Modeling Surprise in a Physically-Grounded Joint Inference of Observation, Preference, and Knowledge"],"prefix":"10.1162","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6413-6286","authenticated-orcid":true,"given":"Harry","family":"Chen","sequence":"first","affiliation":[{"name":"Department of Brain & Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9993-7675","authenticated-orcid":true,"given":"Karen","family":"Chung","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering & Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abhishek","family":"Bhandwaldar","sequence":"additional","affiliation":[{"name":"MIT-IBM Watson AI Lab, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1925-2035","authenticated-orcid":true,"given":"Joshua B.","family":"Tenenbaum","sequence":"additional","affiliation":[{"name":"Department of Brain & Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomer D.","family":"Ullman","sequence":"additional","affiliation":[{"name":"Department of Psychology, Harvard University, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianmin","family":"Shu","sequence":"additional","affiliation":[{"name":"Department of Computer Science & Department of Cognitive Science, Johns Hopkins University, Baltimore, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,7,17]]},"reference":[{"issue":"4","key":"2026081909211600000_bib1","doi-asserted-by":"publisher","first-page":"0064","DOI":"10.1038\/s41562-017-0064","article-title":"Rational quantitative attribution of beliefs, desires and percepts in human mentalizing","volume":"1","author":"Baker","year":"2017","journal-title":"Nature Human Behaviour"},{"key":"2026081909211600000_bib2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.15052","article-title":"ToMBench: Benchmarking theory of mind in large language models","author":"Chen","year":"2024","journal-title":"arXiv"},{"key":"2026081909211600000_bib3","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.06261","article-title":"Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities","author":"Comanici","year":"2025","journal-title":"arXiv"},{"key":"2026081909211600000_bib4","volume-title":"Pybullet, a Python module for physics simulation for games, robotics and machine learning","author":"Coumans","year":"2016\u20132019"},{"issue":"1","key":"2026081909211600000_bib5","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1207\/s15516709cog2701_4","article-title":"One-year-old infants use teleological representations of actions productively","volume":"27","author":"Csibra","year":"2003","journal-title":"Cognitive Science"},{"key":"2026081909211600000_bib6","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1145\/3314221.3314642","article-title":"Gen: A general-purpose probabilistic programming system with programmable inference","volume-title":"Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation","author":"Cusumano-Towner","year":"2019"},{"key":"2026081909211600000_bib7","volume-title":"The intentional stance","author":"Dennett","year":"1987"},{"key":"2026081909211600000_bib8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.00324","article-title":"Vision language models see what you want but not what you see","author":"Gao","year":"2025","journal-title":"arXiv"},{"key":"2026081909211600000_bib9","doi-asserted-by":"publisher","first-page":"1478","DOI":"10.1109\/ICRA.2011.5980479","article-title":"Anytime motion planning using the RRT*","volume-title":"2011 IEEE International Conference on Robotics and Automation","author":"Karaman","year":"2011"},{"key":"2026081909211600000_bib10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6980","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2017","journal-title":"arXiv"},{"key":"2026081909211600000_bib11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.17207","article-title":"Perspective-aware reasoning in vision-language models via mental imagery simulation","author":"Lee","year":"2025","journal-title":"arXiv"},{"issue":"1","key":"2026081909211600000_bib12","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1038\/s44159-025-00514-1","article-title":"How physical information is used to make sense of the psychological world","volume":"5","author":"Liu","year":"2026","journal-title":"Nature Reviews Psychology"},{"issue":"3","key":"2026081909211600000_bib13","doi-asserted-by":"publisher","first-page":"489","DOI":"10.1016\/j.cognition.2006.10.007","article-title":"Do 12.5-month-old infants consider what objects others can see when interpreting their actions?","volume":"105","author":"Luo","year":"2007","journal-title":"Cognition"},{"key":"2026081909211600000_bib14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"GPT-4 technical report","author":"OpenAI","year":"2024","journal-title":"arXiv"},{"issue":"4","key":"2026081909211600000_bib15","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1017\/S0140525X00076512","article-title":"Does the chimpanzee have a theory of mind?","volume":"1","author":"Premack","year":"1978","journal-title":"Behavioral and Brain Sciences"},{"key":"2026081909211600000_bib16","first-page":"4218","article-title":"Machine theory of mind","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Rabinowitz","year":"2018"},{"key":"2026081909211600000_bib17","first-page":"9614","article-title":"AGENT: A benchmark for core psychological reasoning","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Shu","year":"2021"},{"key":"2026081909211600000_bib18","doi-asserted-by":"publisher","DOI":"10.4159\/9780674044371","volume-title":"The cultural origins of human cognition","author":"Tomasello","year":"1999"},{"key":"2026081909211600000_bib19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2602.15892","article-title":"Egocentric bias in vision-language models","author":"Wang","year":"2026","journal-title":"arXiv"}],"container-title":["Open Mind"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/opmi\/article-pdf\/doi\/10.1162\/OPMI.a.375\/2621060\/opmi.a.375.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/opmi\/article-pdf\/doi\/10.1162\/OPMI.a.375\/2621060\/opmi.a.375.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T13:21:27Z","timestamp":1787145687000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/opmi\/article\/doi\/10.1162\/OPMI.a.375\/138310\/POPKI-Modeling-Surprise-in-a-Physically-Grounded"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":19,"URL":"https:\/\/doi.org\/10.1162\/opmi.a.375","relation":{},"ISSN":["2470-2986"],"issn-type":[{"value":"2470-2986","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}