{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T10:46:07Z","timestamp":1782297967056,"version":"3.54.5"},"reference-count":77,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:p>Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degrees of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, for example, representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensory data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model predictive control (MPC) algorithms to plan the robot\u2019s behavior. We show through experiments that with just 10\u00a0min of real-world robot interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various complex target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot\u2019s manipulation capabilities.<\/jats:p>","DOI":"10.1177\/02783649231219020","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T07:32:06Z","timestamp":1702884726000},"page":"533-549","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":33,"title":["RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks"],"prefix":"10.1177","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3604-465X","authenticated-orcid":false,"given":"Haochen","family":"Shi","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huazhe","family":"Xu","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiao","family":"Huang","sequence":"additional","affiliation":[{"name":"Computer Science &amp; Engineering Department, University of California San Diego, La Jolla, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunzhu","family":"Li","sequence":"additional","affiliation":[{"name":"Stanford University"},{"name":"Computer Science Department, University of Illinois Urbana-Champaign, Urbana, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4176-343X","authenticated-orcid":false,"given":"Jiajun","family":"Wu","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,12,18]]},"reference":[{"key":"e_1_3_5_2_1","unstructured":"Antonova R Shi P Yin H et al. (2021a) Dynamic environments with deformable objects. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track Virtual December 6-14."},{"key":"e_1_3_5_3_1","unstructured":"Antonova R Varava A Shi P et al. (2021b) Sequential topological representations for predictive models of deformable objects. In: Learning for Dynamics and Control Palo Alto United States June 23-24. PMLR pp. 348\u2013360."},{"key":"e_1_3_5_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3157377"},{"key":"e_1_3_5_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-010-0416-3"},{"key":"e_1_3_5_6_1","doi-asserted-by":"crossref","unstructured":"Avigal Y Berscheid L Asfour T et al. (2022) Speedfolding: learning efficient bimanual folding of garments. In: 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE pp. 1\u20138.","DOI":"10.1109\/IROS47612.2022.9981402"},{"key":"e_1_3_5_7_1","first-page":"6","article-title":"Interaction networks for learning about objects, relations and physics","volume":"29","author":"Battaglia P","year":"2016","unstructured":"Battaglia P, Pascanu R, Lai M, et al. (2016) Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems 29: 6.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/2945.817351"},{"key":"e_1_3_5_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/AIM43001.2020.9159044"},{"key":"e_1_3_5_10_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920907684"},{"key":"e_1_3_5_11_1","doi-asserted-by":"publisher","unstructured":"Chi C Burchfiel B Cousineau E et al. (2022) Iterative Residual policy for goal-conditioned dynamic manipulation of deformable objects. In: Proceedings of Robotics: Science and Systems. New York City NY USA. DOI: 10.15607\/RSS.2022.XVIII.016.","DOI":"10.15607\/RSS.2022.XVIII.016"},{"key":"e_1_3_5_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2011.2176115"},{"key":"e_1_3_5_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAES.2016.140952"},{"key":"e_1_3_5_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/174462.156635"},{"key":"e_1_3_5_15_1","doi-asserted-by":"crossref","unstructured":"Fan H Su H Guibas LJ (2017) A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Hawaii United States July 21-26 pp. 605\u2013613.","DOI":"10.1109\/CVPR.2017.264"},{"key":"e_1_3_5_16_1","first-page":"885","volume-title":"Introduction to Finite Element Methods","author":"Felippa CA","year":"2004","unstructured":"Felippa CA (2004) Introduction to Finite Element Methods. Boulder, Colorado, United States: University of Colorado, 885."},{"key":"e_1_3_5_17_1","doi-asserted-by":"crossref","unstructured":"Figueroa N Ureche ALP Billard A (2016) Learning complex sequential tasks from demonstration: a pizza dough rolling case study. In: 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI) Christchurch New Zealand March 7-10. IEEE pp. 611\u2013612.","DOI":"10.1109\/HRI.2016.7451881"},{"key":"e_1_3_5_18_1","volume-title":"Practical Methods of Optimization","author":"Fletcher R","year":"2013","unstructured":"Fletcher R (2013) Practical Methods of Optimization. Hoboken, New Jersey, United States: John Wiley & Sons."},{"key":"e_1_3_5_19_1","doi-asserted-by":"crossref","unstructured":"Ganapathi A Sundaresan P Thananjeyan B et al. (2020) Learning to Smooth and Fold Real Fabric Using Dense Object Descriptors Trained on Synthetic Color Images. Cornell University: New York City NY United States. arXiv preprint arXiv:2003.12698.","DOI":"10.1109\/ICRA40945.2020.9197121"},{"key":"e_1_3_5_20_1","unstructured":"Ha H Song S (2022) Flingbot: the unreasonable effectiveness of dynamic manipulation for cloth unfolding. In: Conference on Robot Learning (CoRL) Auckland New Zealand December 14-18. PMLR pp. 24\u201333."},{"key":"e_1_3_5_21_1","unstructured":"Holl P Thuerey N Koltun V (2019) Learning to control pdes with differentiable physics. In: International Conference on Learning Representations (ICLR) New Orleans United States May 6-9."},{"key":"e_1_3_5_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-021-10001-0"},{"key":"e_1_3_5_23_1","unstructured":"Hu Y Anderson L Li TM et al. (2019) Difftaichi: differentiable programming for physical simulation. In: International Conference on Learning Representations (ICLR) New Orleans United States May 6-9."},{"key":"e_1_3_5_24_1","unstructured":"Huang Z Hu Y Du T et al. (2020) Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In: International Conference on Learning Representations (ICLR) Virtual April 26-May 1."},{"key":"e_1_3_5_25_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2022.XVIII.011"},{"key":"e_1_3_5_26_1","unstructured":"Jatavallabhula KM Macklin M Golemo F et al. (2020) Gradsim: Differentiable Simulation for System Identification and Visuomotor Control. In: International Conference on Learning Representations (ICLR) Virtual April 26-May 1."},{"key":"e_1_3_5_27_1","unstructured":"Kazhdan M Bolitho M Hoppe H (2006) Poisson surface reconstruction. In: Proceedings of the fourth Eurographics Symposium on Geometry Processing Goslar Germany June 26-28. vol. 7."},{"key":"e_1_3_5_28_1","unstructured":"Kingma DP Ba J (2015) Adam: a method for stochastic optimization. In: International Conference on Learning Representation (ICLR) San Diego United States May 7-9."},{"key":"e_1_3_5_29_1","first-page":"4","article-title":"Learning plannable representations with causal infogan","volume":"31","author":"Kurutach T","year":"2018","unstructured":"Kurutach T, Tamar A, Yang G, et al. (2018) Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems 31: 4.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_30_1","unstructured":"Landau LD Lifshitz EM Kosevich AM et al. (1986) Theory of elasticity vol. 7. London UK: Elsevier."},{"key":"e_1_3_5_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3062323"},{"key":"e_1_3_5_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3130377"},{"issue":"1","key":"e_1_3_5_33_1","first-page":"5","article-title":"Constructing neural network-based models for simulating dynamical systems","volume":"1","author":"Leggard CM","year":"2021","unstructured":"Leggard CM, Schranz T, Schweiger G, et al. (2021) Constructing neural network-based models for simulating dynamical systems. ACM Computing Surveys 1(1): 5.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_5_34_1","unstructured":"Li Y Wu J Tedrake R et al. (2018) Learning particle dynamics for manipulating rigid bodies deformable objects and fluids. In: International Conference on Learning Representations (ICLR) Vancouver Canada April 30-May 3."},{"key":"e_1_3_5_35_1","unstructured":"Li Y Lin T Yi K et al. (2020a) Visual grounding of learned physical models. In: International Conference on Machine Learning (ICML) Virtual July 12-18. PMLR pp. 5927\u20135936."},{"key":"e_1_3_5_36_1","unstructured":"Li Y Li S Sitzmann V et al. (2022b) 3d neural scene representations for visuomotor control. In: Conference on Robot Learning (CoRL) Auckland New Zealand December 14-18. PMLR pp. 112\u2013123."},{"key":"e_1_3_5_37_1","first-page":"9180","article-title":"Causal discovery in physical systems from videos","volume":"33","author":"Li Y","year":"2020","unstructured":"Li Y, Torralba A, Anandkumar A, et al. (2020b) Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems 33: 9180\u20139192.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_38_1","unstructured":"Li S Huang Z Du T et al. (2022a) Contact points discovery for soft-body manipulations with differentiable physics. In: International Conference on Learning Representations (ICLR) Virtual April 25 - 29."},{"key":"e_1_3_5_39_1","unstructured":"Lin X Wang Y Huang Z et al. (2022c) Learning visible connectivity dynamics for cloth smoothing. In: Conference on Robot Learning (CoRL) Auckland New Zealand December 14-18. PMLR pp. 256\u2013266."},{"key":"e_1_3_5_40_1","unstructured":"Lin X Wang Y Olkin J et al. (2021) Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In: Conference on Robot Learning (CoRL) London UK November 8-11. PMLR pp. 432\u2013448."},{"key":"e_1_3_5_41_1","unstructured":"Lin X Huang Z Li Y et al. (2022a) Diffskill: skill abstraction from differentiable physics for deformable object manipulations with tools. In: International Conference on Learning Representations (ICLR) Virtual April 25-29."},{"key":"e_1_3_5_42_1","unstructured":"Lin X Qi C Zhang Y et al. (2022b) Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. In: 6th Annual Conference on Robot Learning Auckland New Zealand December 14-18."},{"key":"e_1_3_5_43_1","unstructured":"Luo Y Xu H Li Y et al. (2018) Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In: International Conference on Learning Representations (ICLR) Vancouver Canada April 30-May 3."},{"key":"e_1_3_5_44_1","unstructured":"Manuelli L Li Y Florence P et al. (2021) Keypoints into the future: self-supervised correspondence in model-based reinforcement learning. In: Conference on Robot Learning (CoRL) London UK November 8-11. PMLR pp. 693\u2013710."},{"key":"e_1_3_5_45_1","unstructured":"Matas J James S Davison AJ (2018) Sim-to-real reinforcement learning for deformable object manipulation. In: Conference on Robot Learning (CoRL) Z\u00fcrich Switzerland October 29-31. PMLR pp. 734\u2013743."},{"key":"e_1_3_5_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636808"},{"key":"e_1_3_5_47_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364911430417"},{"key":"e_1_3_5_48_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.aa.30.090192.002551"},{"key":"e_1_3_5_49_1","first-page":"1","article-title":"Flexible neural representation for physics prediction","volume":"31","author":"Mrowca D","year":"2018","unstructured":"Mrowca D, Zhuang C, Wang E, et al. (2018) Flexible neural representation for physics prediction. Advances in Neural Information Processing Systems 31: 1.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_5_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvcir.2007.01.005"},{"key":"e_1_3_5_51_1","doi-asserted-by":"publisher","DOI":"10.3390\/robotics7040074"},{"key":"e_1_3_5_52_1","unstructured":"Nagabandi A Konolige K Levine S et al. (2020) Deep dynamics models for learning dexterous manipulation. In: Conference on Robot Learning (CoRL) Virtual November 16-18. PMLR pp. 1101\u20131112."},{"key":"e_1_3_5_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989247"},{"key":"e_1_3_5_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2016.2533639"},{"key":"e_1_3_5_55_1","volume-title":"Introduction to the Finite Element Method","author":"Reddy JN","year":"2019","unstructured":"Reddy JN (2019) Introduction to the Finite Element Method. New York City, NY, United States: McGraw-Hill Education."},{"key":"e_1_3_5_56_1","doi-asserted-by":"crossref","unstructured":"Rubner Y Tomasi C Guibas LJ (1998) A metric for distributions with applications to image databases. In: Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271) Bombay India January 7. IEEE pp. 59\u201366.","DOI":"10.1109\/ICCV.1998.710701"},{"key":"e_1_3_5_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918779698"},{"key":"e_1_3_5_58_1","unstructured":"Sanchez-Gonzalez A Godwin J Pfaff T et al. (2020) Learning to simulate complex physics with graph networks. In: International Conference on Machine Learning (ICML) Virtual July 12-18. PMLR pp. 8459\u20138468."},{"key":"e_1_3_5_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2974675"},{"key":"e_1_3_5_60_1","first-page":"APS","article-title":"End-to-end differentiable, hardware accelerated, molecular dynamics in pure python","volume":"65","author":"Schoenholz SS","year":"2020","unstructured":"Schoenholz SS, Cubuk ED, Jax M (2020) End-to-end differentiable, hardware accelerated, molecular dynamics in pure python. Bulletin of the American Physical Society 65: APS.","journal-title":"Bulletin of the American Physical Society"},{"key":"e_1_3_5_61_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-03051-4"},{"key":"e_1_3_5_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561391"},{"key":"e_1_3_5_63_1","doi-asserted-by":"publisher","DOI":"10.1177\/02783649211027233"},{"key":"e_1_3_5_64_1","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/abbf9a"},{"key":"e_1_3_5_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/0010-4655(94)00170-7"},{"key":"e_1_3_5_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197121"},{"key":"e_1_3_5_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989275"},{"key":"e_1_3_5_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12530-020-09345-2"},{"key":"e_1_3_5_69_1","doi-asserted-by":"publisher","unstructured":"Viswanath V Shivakumar K Kerr J et al. (2022) Autonomously Untangling long cables. In: Proceedings of Robotics: Science and Systems. New York City NY USA. DOI: 10.15607\/RSS.2022.XVIII.034.","DOI":"10.15607\/RSS.2022.XVIII.034"},{"key":"e_1_3_5_70_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2019.XV.074"},{"key":"e_1_3_5_71_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2020.XVI.065"},{"key":"e_1_3_5_72_1","doi-asserted-by":"publisher","unstructured":"Xu Z Chi C Burchfiel B et al. (2022) DextAIRity: deformable manipulation can be a Breeze. In: Proceedings of Robotics: Science and Systems. New York City NY USA. DOI: 10.15607\/RSS.2022.XVIII.017.","DOI":"10.15607\/RSS.2022.XVIII.017"},{"key":"e_1_3_5_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2969931"},{"key":"e_1_3_5_74_1","unstructured":"Yan W Vangipuram A Abbeel P et al. (2021) Learning predictive representations for deformable objects using contrastive estimation. In: Conference on Robot Learning (CoRL) London UK November 8-11. PMLR pp. 564\u2013574."},{"key":"e_1_3_5_75_1","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abd8803"},{"key":"e_1_3_5_76_1","doi-asserted-by":"crossref","unstructured":"Yoshimoto K Higashimori M Tadakuma K et al. (2011) Active outline shaping of a rheological object based on plastic deformation distribution. In: 2011 IEEE\/RSJ International Conference on Intelligent Robots and Systems San Francisco United States September 25-30. IEEE pp. 1386\u20131391.","DOI":"10.1109\/IROS.2011.6094463"},{"key":"e_1_3_5_77_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12538"},{"key":"e_1_3_5_78_1","first-page":"45","article-title":"Multiphysics modeling with finite element methods","volume":"18","author":"Zimmerman WB","year":"2006","unstructured":"Zimmerman WB (2006) Multiphysics modeling with finite element methods. World Scientific Publishing Company 18: 45.","journal-title":"World Scientific Publishing Company"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231219020","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649231219020","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231219020","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T09:46:58Z","timestamp":1782294418000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649231219020"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,18]]},"references-count":77,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["10.1177\/02783649231219020"],"URL":"https:\/\/doi.org\/10.1177\/02783649231219020","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,18]]}}}