{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T16:53:50Z","timestamp":1775667230608,"version":"3.50.1"},"reference-count":29,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"68","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2022,7,27]]},"abstract":"<jats:p>Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These \u201cself-models\u201d allow robots to consider outcomes of multiple possible future actions without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward kinematic models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot\u2019s state. Such query-driven self-models are continuous in the spatial domain, memory efficient, fully differentiable, and kinematic aware and can be used across a broader range of tasks. In physical experiments, we demonstrate how a visual self-model is accurate to about 1% of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize, and recover from real-world damage, leading to improved machine resiliency.<\/jats:p>","DOI":"10.1126\/scirobotics.abn1944","type":"journal-article","created":{"date-parts":[[2022,7,13]],"date-time":"2022-07-13T17:57:08Z","timestamp":1657735028000},"source":"Crossref","is-referenced-by-count":57,"title":["Fully body visual self-modeling of robot morphologies"],"prefix":"10.1126","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3813-1060","authenticated-orcid":true,"given":"Boyuan","family":"Chen","sequence":"first","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8278-2012","authenticated-orcid":true,"given":"Robert","family":"Kwiatkowski","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1139-9208","authenticated-orcid":true,"given":"Carl","family":"Vondrick","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, New York, NY, USA."},{"name":"Data Science Institute, Columbia University, New York, NY, USA."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0769-4618","authenticated-orcid":true,"given":"Hod","family":"Lipson","sequence":"additional","affiliation":[{"name":"Data Science Institute, Columbia University, New York, NY, USA."},{"name":"Department of Mechanical Engineering, Columbia University, New York, NY, USA."}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1002\/ajp.1350020302"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1053-8100(03)00081-3"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"N. Koenig A. Howard Design and use paradigms for gazebo an open-source multi-robot simulator in 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566 IEEE 2004) vol. 3 pp. 2149\u20132154.","DOI":"10.1109\/IROS.2004.1389727"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","unstructured":"E. Todorov T. Erez Y. Tassa Mujoco: A physics engine for model-based control in 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IEEE 2012) pp. 5026\u20135033.","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","unstructured":"F. Faure C. Duriez H. Delingette J. Allard B. Gilles S. Marchesseau H. Talbot H. Courtecuisse G. Bousquet I. Peterlik in SOFA: A Multi-Model Framework for Interactive Physical Simulation Soft Tissue Biomechanical Modeling for Computer Assisted Surgery (Springer 2012) pp. 283\u2013321.","DOI":"10.1007\/8415_2012_125"},{"key":"e_1_3_2_7_2","unstructured":"E. Coumans Y. Bai PyBullet A Python Module for Physics Simulation for Games Robotics and Machine Learning; http:\/\/pybullet.org."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00500"},{"key":"e_1_3_2_9_2","unstructured":"R. Tedrake Drake Development Team Drake: Model-Based Design and Verification for Robotics https:\/\/drake.mit.edu\/."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1133687"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aau9354"},{"key":"e_1_3_2_12_2","unstructured":"A. Sanchez-Gonzalez N. Heess J. T. Springenberg J. Merel M. Riedmiller R. Hadsell P. Battaglia Graph networks as learnable physics engines for inference and control in International Conference on Machine Learning (PMLR 2018) pp. 4470\u20134479."},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"B. Chen Y. Hu L. Li S. Cummings H. Lipson Smile like you mean it: Driving animatronic robotic face with learned models in 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE 2021) pp. 2739\u20132746.","DOI":"10.1109\/ICRA48506.2021.9560797"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abe1321"},{"key":"e_1_3_2_15_2","unstructured":"H. E. Gardner Frames of Mind: The Theory of Multiple Intelligences (Hachette UK 2011)."},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","unstructured":"J. J. Park P. Florence J. Straub R. Newcombe S. Lovegrove Deepsdf: Learning continuous signed distance functions for shape representation in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2019) pp. 165\u2013174.","DOI":"10.1109\/CVPR.2019.00025"},{"key":"e_1_3_2_17_2","first-page":"7462","article-title":"Implicit neural representa- tions with periodic activation functions","volume":"33","author":"Sitzmann V.","year":"2020","unstructured":"V. Sitzmann, J. Martel, A. Bergman, D. Lindell, G. Wetzstein, Implicit neural representa- tions with periodic activation functions. Adv. Neural Inf. Process. Syst. 33, 7462\u20137473 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/37402.37422"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"S. M. LaValle Planning Algorithms (Cambridge Univ. Press 2006).","DOI":"10.1017\/CBO9780511546877"},{"key":"e_1_3_2_20_2","first-page":"104","article-title":"Incremental sampling-based algorithms for optimal motion plan- ning","author":"Karaman S.","year":"2010","unstructured":"S. Karaman, E. Frazzoli, Incremental sampling-based algorithms for optimal motion plan- ning. Robot. Sci. Syst. VI, 104 (2010).","journal-title":"Robot. Sci. Syst."},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"M. Tancik B. Mildenhall T. Wang D. Schmidt P. P. Srinivasan J. T. Barron R. Ng Learned initializations for optimizing coordinate-based neural representations in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE 2021) pp. 2846\u20132855.","DOI":"10.1109\/CVPR46437.2021.00287"},{"key":"e_1_3_2_22_2","first-page":"199","article-title":"Active touch and robot perception","volume":"7","author":"Goldberg K. Y.","year":"1984","unstructured":"K. Y. Goldberg, R. Bajcsy, Active touch and robot perception. Cogn. Brain Theory 7, 199\u2013214 (1984).","journal-title":"Cogn. Brain Theory"},{"key":"e_1_3_2_23_2","unstructured":"J. Bongard H. Lipson Automatic synthesis of multiple internal models through active exploration in AAAI Fall Symposium: From Reactive to Anticipatory Cognitive Embodied Systems (AAAI Press 2005)."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2017.2721939"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaw6326"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"J. L. Sch\u00f6nberger J.-M. Frahm Structure-from-motion revisited in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE 2016) pp. 4104\u20134113.","DOI":"10.1109\/CVPR.2016.445"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"J. L. Sch\u00f6nberger E. Zheng M. Pollefeys J.-M. Frahm Pixelwise view selection for unstructured multi-view stereo in European Conference on Computer Vision (ECCV) (Springer 2016) pp. 501\u2013518.","DOI":"10.1007\/978-3-319-46487-9_31"},{"key":"e_1_3_2_28_2","unstructured":"D. P. Kingma J. Ba Adam: A method for stochastic optimization arXiv:1412.6980 [cs.LG] (22 December 2014)."},{"key":"e_1_3_2_29_2","unstructured":"A. Paszke S. Gross F. Massa A. Lerer J. Bradbury G. Chanan T. Killeen Z. Lin N. Gimelshein L. Antiga A. Desmaison A. K\u00f6pf E. Yang Z. DeVito M. Raison A. Tejani S. Chilamkurthy B. Steiner L. Fang J. Bai S. Chintala Pytorch: An imperative style high-performance deep learning library in Advances in Neural Information Processing Systems 32 H. Wallach H. Larochelle A. Beygelzimer F. d\u2019Alch\u00e9-Buc E. Fox R. Garnett Eds. (Curran Associates Inc. 2019) pp. 8024\u20138035."},{"key":"e_1_3_2_30_2","unstructured":"W. Falcon PyTorch Lightning Team PyTorch Lightning www.pytorchlightning.ai."}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.abn1944","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,29]],"date-time":"2024-09-29T02:28:51Z","timestamp":1727576931000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.abn1944"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,27]]},"references-count":29,"journal-issue":{"issue":"68","published-print":{"date-parts":[[2022,7,27]]}},"alternative-id":["10.1126\/scirobotics.abn1944"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.abn1944","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,27]]},"article-number":"eabn1944"}}