{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T15:13:33Z","timestamp":1768490013067,"version":"3.49.0"},"reference-count":60,"publisher":"MIT Press","issue":"4","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Representing a scene and its constituent objects from raw sensory data is a core ability for enabling robots to interact with their environment. In this letter, we propose a novel approach for scene understanding, leveraging an object-centric generative model that enables an agent to infer object category and pose in an allocentric reference frame using active inference, a neuro-inspired framework for action and perception. For evaluating the behavior of an active vision agent, we also propose a new benchmark where, given a target viewpoint of a particular object, the agent needs to find the best matching viewpoint given a workspace with randomly positioned objects in 3D. We demonstrate that our active inference agent is able to balance epistemic foraging and goal-driven behavior, and quantitatively outperforms both supervised and reinforcement learning baselines by more than a factor of two in terms of success rate.<\/jats:p>","DOI":"10.1162\/neco_a_01637","type":"journal-article","created":{"date-parts":[[2024,3,8]],"date-time":"2024-03-08T20:56:52Z","timestamp":1709931412000},"page":"677-704","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["Object-Centric Scene Representations Using Active Inference"],"prefix":"10.1162","volume":"36","author":[{"given":"Toon","family":"Van de Maele","sequence":"first","affiliation":[{"name":"Ghent University, 9000 Ghent, Belgium toon.vandemaele@ugent.be"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Verbelen","sequence":"additional","affiliation":[{"name":"VERSES AI Research Lab, Los Angeles, CA 90016, U.S.A. tim.verbelen@verses.ai"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pietro","family":"Mazzaglia","sequence":"additional","affiliation":[{"name":"Ghent University, 9000 Ghent, Belgium pietro.mazzaglia@ugent.be"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Ferraro","sequence":"additional","affiliation":[{"name":"Ghent University, 9000 Ghent, Belgium stefano.ferraro@ugent.be"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bart","family":"Dhoedt","sequence":"additional","affiliation":[{"name":"Ghent University, 9000 Ghent, Belgium bart.dhoedt@ugent.be"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,3,21]]},"reference":[{"key":"2024032521541624600_bib1","article-title":"Hindsight experience replay","volume-title":"Advances in neural information processing systems","author":"Andrychowicz","year":"2017"},{"key":"2024032521541624600_bib2","first-page":"6027","article-title":"Learning physical graph representations from visual scenes","volume-title":"Advances in neural information processing systems","author":"Bear","year":"2020"},{"key":"2024032521541624600_bib3","author":"Brockman","year":"2016","journal-title":"OpenAi gym"},{"key":"2024032521541624600_bib4","author":"Burgess","year":"2019","journal-title":"MONet: Unsupervised scene decomposition and representation."},{"key":"2024032521541624600_bib5","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1109\/ICAR.2015.7251504","article-title":"The YCB object and model set: Towards common benchmarks for manipulation research","author":"Calli","year":"2015","journal-title":"Proceedings of the 2015 International Conference on Advanced Robotics"},{"key":"2024032521541624600_bib6","author":"Chen","year":"2021","journal-title":"ROOTS: Object-centric representation and rendering of 3D scenes."},{"key":"2024032521541624600_bib7","author":"Crawford","year":"2019","journal-title":"Exploiting spatial invariance for scalable unsupervised object tracking"},{"key":"2024032521541624600_bib8","doi-asserted-by":"crossref","DOI":"10.3389\/fnbot.2018.00076","article-title":"Active fovea-based vision through computationally-effective model-based prediction","volume":"12","author":"Dauc\u00e9","year":"2018","journal-title":"Frontiers in Neurorobotics"},{"issue":"3","key":"2024032521541624600_bib9","doi-asserted-by":"publisher","first-page":"1677","DOI":"10.1007\/s10462-020-09888-5","article-title":"Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: A review","volume":"54","author":"Du","year":"2021","journal-title":"Artificial Intelligence Review"},{"key":"2024032521541624600_bib10","volume-title":"GENESIS: Generative scene inference and sampling of object-centric latent representations.","author":"Engelcke","year":"2020"},{"issue":"6394","key":"2024032521541624600_bib11","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1126\/science.aar6170","article-title":"Neural scene representation and rendering","volume":"360","author":"Eslami","year":"2018","journal-title":"Science"},{"issue":"3","key":"2024032521541624600_bib12","doi-asserted-by":"publisher","DOI":"10.1098\/rsfs.2022.0077","article-title":"Symmetry and complexity in object-centric deep active inference models","volume":"13","author":"Ferraro","year":"2023","journal-title":"Interface Focus"},{"key":"2024032521541624600_bib13","author":"Ferraro","year":"2022","journal-title":"Disentangling shape and pose for object-centric deep active inference models."},{"key":"2024032521541624600_bib14","doi-asserted-by":"publisher","first-page":"862","DOI":"10.1016\/j.neubiorev.2016.06.022","article-title":"Active inference and learning","volume":"68","author":"Friston","year":"2016","journal-title":"Neuroscience and Biobehavioral Reviews"},{"issue":"1","key":"2024032521541624600_bib15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/NECO_a_00912","article-title":"Active inference: A process theory","volume":"29","author":"Friston","year":"2017","journal-title":"Neural Computation"},{"key":"2024032521541624600_bib16","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.00373","article-title":"Kubric: A scalable dataset generator","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Greff","year":"2022"},{"key":"2024032521541624600_bib17","author":"Greff","year":"2020","journal-title":"Multi-object representation learning with iterative variational inference"},{"key":"2024032521541624600_bib18","article-title":"Recurrent world models facilitate policy evolution","volume-title":"Advances in neural information processing systems","author":"Ha","year":"2018"},{"key":"2024032521541624600_bib19","article-title":"Dream to control: Learning behaviors by latent imagination","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hafner","year":"2020"},{"key":"2024032521541624600_bib20","first-page":"2555","article-title":"Learning latent dynamics for planning from pixels","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Hafner","year":"2019"},{"key":"2024032521541624600_bib21","article-title":"Mastering Atari with discrete world models","volume-title":"Proceedings of the 9th International Conference on Learning Representation","author":"Hafner","year":"2021"},{"key":"2024032521541624600_bib22","doi-asserted-by":"publisher","DOI":"10.3389\/fncir.2017.00081","article-title":"A theory of how columns in the neocotex enable learning the structure of the world","volume":"11","author":"Hawkins","year":"2017","journal-title":"Frontiers in Neural Circuits"},{"issue":"3","key":"2024032521541624600_bib23","first-page":"231","article-title":"Some demonstrations of the effects of structural descriptions in mental imagery","volume":"3","author":"Hinton","year":"1979","journal-title":"Cognitive Science"},{"key":"2024032521541624600_bib24","article-title":"Spatial transformer networks","volume-title":"Advances in neural information processing systems","author":"Jaderberg","year":"2015"},{"key":"2024032521541624600_bib25","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1080\/15248372.2012.749481","article-title":"Young children\u2019s self-generated object views and object recognition","volume-title":"Journal of Cognition and Development","author":"James","year":"2014"},{"key":"2024032521541624600_bib26","article-title":"SCALOR: Generative world models with scalable object representations","author":"Jiang","year":"2020","journal-title":"Proceedings of ICLR 2020"},{"key":"2024032521541624600_bib27","article-title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","author":"Johnson","year":"2016","journal-title":"CoRR"},{"key":"2024032521541624600_bib28","author":"Kingma","year":"2014","journal-title":"Auto-encoding variational Bayes"},{"key":"2024032521541624600_bib29","author":"Kingma","year":"2017","journal-title":"Adam: A method for stochastic optimization"},{"key":"2024032521541624600_bib30","author":"Kipf","year":"2019","journal-title":"Contrastive learning of structured world models."},{"key":"2024032521541624600_bib31","author":"Kosiorek","year":"2018","journal-title":"Sequential attend, infer, repeat: Generative modelling of moving objects."},{"key":"2024032521541624600_bib32","article-title":"Active inference in robotics and artificial agents: Survey and challenges","author":"Lanillos","year":"2021","journal-title":"CoRR"},{"key":"2024032521541624600_bib33","article-title":"Improving generative imagination in object-centric world models","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Lin","year":"2020"},{"key":"2024032521541624600_bib34","first-page":"11525","article-title":"Object-centric learning with slot attention","volume-title":"Advances in neural information processing systems","author":"Locatello","year":"2020"},{"key":"2024032521541624600_bib35","author":"Mazzaglia","year":"2022","journal-title":"Choreographer: Learning and adapting skills in imagination."},{"key":"2024032521541624600_bib36","first-page":"24379","article-title":"Discovering and achieving goals via world models","volume-title":"Advances in neural information processing systems","author":"Mendonca","year":"2021"},{"issue":"7","key":"2024032521541624600_bib37","first-page":"3523","article-title":"Image segmentation using deep learning: A survey","volume":"44","author":"Minaee","year":"2021","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2024032521541624600_bib38","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2016.00056","article-title":"Scene construction, visual foraging, and active inference","volume":"10","author":"Mirza","year":"2016","journal-title":"Frontiers in Computational Neuroscience"},{"key":"2024032521541624600_bib39","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1016\/0166-2236(83)90190-X","article-title":"Object vision and spatial vision: Two cortical pathways","volume":"6","author":"Mishkin","year":"1983","journal-title":"Trends in Neurosciences"},{"key":"2024032521541624600_bib40","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/12441.001.0001","volume-title":"Active inference: The free energy principle in mind, brain, and behavior","author":"Parr","year":"2022"},{"key":"2024032521541624600_bib41","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2018.00061","article-title":"Computational neuropsychology and Bayesian inference","volume":"12","author":"Parr","year":"2018","journal-title":"Frontiers in Human Neuroscience"},{"key":"2024032521541624600_bib42","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2021.651432","article-title":"Generative models for active vision","volume":"15","author":"Parr","year":"2021","journal-title":"Frontiers in Neurorobotics"},{"key":"2024032521541624600_bib43","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2022.988977","volume":"16","author":"Pio-Lopez","year":"2022","journal-title":"Frontiers in Computational Neuroscience"},{"key":"2024032521541624600_bib44","author":"Price","year":"2019","journal-title":"Inferring occluded geometry improves performance when retrieving an object from dense clutter."},{"key":"2024032521541624600_bib45","author":"Rezende","year":"2014","journal-title":"Stochastic backpropagation and approximate inference in deep generative models."},{"key":"2024032521541624600_bib46","author":"Rezende","year":"2018","journal-title":"Taming VAEs"},{"key":"2024032521541624600_bib47","article-title":"Object scene representation transformer","volume-title":"Advances in neural information processing systems","author":"Sajjadi","year":"2022"},{"key":"2024032521541624600_bib48","doi-asserted-by":"publisher","first-page":"6219","DOI":"10.1109\/CVPR52688.2022.00613","article-title":"Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations","volume-title":"Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Sajjadi","year":"2022"},{"key":"2024032521541624600_bib49","article-title":"Active 3D shape reconstruction from vision and touch","author":"Smith","year":"2021","journal-title":"CoRR"},{"issue":"1","key":"2024032521541624600_bib50","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1111\/pcn.13138","article-title":"Recent advances in the application of predictive coding and active inference models within clinical neuroscience","volume":"75","author":"Smith","year":"2020","journal-title":"Psychiatry and Clinical Neurosciences"},{"key":"2024032521541624600_bib51","doi-asserted-by":"crossref","first-page":"949","DOI":"10.1109\/3DV50981.2020.00105","article-title":"NodeSLAM: Neural object descriptors for multi-view shape reconstruction","author":"Sucar","year":"2020","journal-title":"Proceedings of the 2020 International Conference on 3D Vision"},{"key":"2024032521541624600_bib52","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1007\/978-3-031-28719-0_4","article-title":"Object-based active inference","volume-title":"Active inference","author":"van Bergen","year":"2023"},{"key":"2024032521541624600_bib53","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2022.840658","article-title":"Embodied object representation learning and recognition","volume":"16","author":"Van de Maele","year":"2022","journal-title":"Frontiers in Neurorobotics"},{"key":"2024032521541624600_bib54","first-page":"1439","article-title":"Entity abstraction in visual model-based reinforcement learning","volume-title":"Proceedings of the Conference on Robot Learning","author":"Veerapaneni","year":"2020"},{"key":"2024032521541624600_bib55","author":"Wang","year":"2019","journal-title":"Normalized object coordinate space for category-level 6D object pose and size estimation."},{"key":"2024032521541624600_bib56","article-title":"COBRA: Data-efficient model-based RL through unsupervised object discovery and curiosity-driven exploration","author":"Watters","year":"2019","journal-title":"CoRR"},{"key":"2024032521541624600_bib57","article-title":"3D ShapeNets: A deep representation for volumetric shape modeling","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Wu","year":"2015"},{"key":"2024032521541624600_bib58","article-title":"PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes","author":"Xiang","year":"2018","journal-title":"Robotics: Science and Systems Conference"},{"key":"2024032521541624600_bib59","author":"Xu","year":"2020","journal-title":"Learning 3D dynamic scene representations for robot manipulation"},{"key":"2024032521541624600_bib60","author":"Yan","year":"2021","journal-title":"CLEVR3D: Compositional language and elementary visual reasoning for question answering in 3D real-world scenes."}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/4\/677\/2351769\/neco_a_01637.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/4\/677\/2351769\/neco_a_01637.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,25]],"date-time":"2024-03-25T21:56:08Z","timestamp":1711403768000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/36\/4\/677\/119792\/Object-Centric-Scene-Representations-Using-Active"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,21]]},"references-count":60,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,3,21]]},"published-print":{"date-parts":[[2024,3,21]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01637","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,4]]},"published":{"date-parts":[[2024,3,21]]}}}