{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T12:10:46Z","timestamp":1770466246515,"version":"3.49.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T00:00:00Z","timestamp":1681257600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T00:00:00Z","timestamp":1681257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Goal-conditioned rearrangement of deformable objects (e.g. straightening a rope and folding a cloth) is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a prescribed <jats:italic>goal<\/jats:italic> configuration with only visual observations. These tasks are typically confronted with two main challenges: the high dimensionality of deformable configuration space and the underlying complexity, nonlinearity and uncertainty inherent in deformable dynamics. To address these challenges, we propose a novel representation strategy that can efficiently model the deformable object states with a set of keypoints and their interactions. We further propose local-graph neural network (GNN), a light local GNN learning to jointly model the deformable rearrangement dynamics and infer the optimal manipulation actions (e.g. pick and place) by constructing and updating two dynamic graphs. Both simulated and real experiments have been conducted to demonstrate that the proposed dynamic graph representation shows superior expressiveness in modeling deformable rearrangement dynamics. Our method reaches much higher success rates on a variety of deformable rearrangement tasks (96.3% on average) than state-of-the-art method in simulation experiments. Besides, our method is much more lighter and has a 60% shorter inference time than state-of-the-art methods. We also demonstrate that our method performs well in the <jats:italic>multi-task<\/jats:italic> learning scenario and can be transferred to real-world applications with an average success rate of 95% by solely fine tuning a keypoint detector. A supplementary video can be found at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/youtu.be\/AhwTQo6fCM0\">https:\/\/youtu.be\/AhwTQo6fCM0<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s40747-023-01048-w","type":"journal-article","created":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T04:03:15Z","timestamp":1681272195000},"page":"5923-5936","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Learning visual-based deformable object rearrangement with local graph neural networks"],"prefix":"10.1007","volume":"9","author":[{"given":"Yuhong","family":"Deng","sequence":"first","affiliation":[]},{"given":"Xueqian","family":"Wang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8508-6699","authenticated-orcid":false,"given":"Lipeng","family":"Chen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,4,12]]},"reference":[{"issue":"54","key":"1048_CR1","doi-asserted-by":"publisher","first-page":"eabd8803","DOI":"10.1126\/scirobotics.abd8803","volume":"6","author":"H Yin","year":"2021","unstructured":"Yin H, Varava A, Kragic D (2021) Modeling, learning, perception, and control methods for deformable object manipulation. Sci Robot 6(54):eabd8803","journal-title":"Sci Robot"},{"key":"1048_CR2","doi-asserted-by":"crossref","unstructured":"Thach B, Cho BY, Kuntz A, Hermans T (2022) Learning visual shape control of novel 3D deformable objects from partial-view point clouds. In: 2022 International conference on robotics and automation (ICRA). IEEE, pp 8274\u20138281","DOI":"10.1109\/ICRA46639.2022.9812215"},{"key":"1048_CR3","doi-asserted-by":"crossref","unstructured":"Seita D, Florence P, Tompson J, Coumans E, Sindhwani V, Goldberg K, Zeng A (2021) Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE, pp 4568\u20134575","DOI":"10.1109\/ICRA48506.2021.9561391"},{"key":"1048_CR4","doi-asserted-by":"crossref","unstructured":"Li Q, Liu C, Yang C, Chen F, Ritter H (2022) Robotic dexterous manipulation: from tele-operation to autonomous learning and adaptive control. Complex Intell Syst 8(4):2809\u20132811","DOI":"10.1007\/s40747-022-00773-y"},{"issue":"4","key":"1048_CR5","doi-asserted-by":"publisher","first-page":"2911","DOI":"10.1007\/s40747-021-00459-x","volume":"8","author":"Z Dong","year":"2022","unstructured":"Dong Z, Tian H, Bao X, Yan Y, Chen F (2022) GraspVDN: scene-oriented grasp estimation by learning vector representations of grasps. Complex Intell Syst 8(4):2911\u20132922","journal-title":"Complex Intell Syst"},{"key":"1048_CR6","doi-asserted-by":"crossref","unstructured":"Deng Y, Guo X, Wei Y, Lu K, Fang B, Guo D, Liu H, Sun F (2019) Deep reinforcement learning for robotic pushing and picking in cluttered environment. In: 2019 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 619\u2013626","DOI":"10.1109\/IROS40897.2019.8967899"},{"issue":"3","key":"1048_CR7","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/MRA.2022.3147415","volume":"29","author":"J Zhu","year":"2022","unstructured":"Zhu J, Cherubini A, Dune C, Navarro-Alarcon D, Alambeigi F, Berenson D, Ficuciello F, Harada K, Kober J, Li X et al (2022) Challenges and outlook in robotic manipulation of deformable objects. IEEE Robot Autom Mag 29(3):67\u201377","journal-title":"IEEE Robot Autom Mag"},{"key":"1048_CR8","doi-asserted-by":"crossref","unstructured":"Zimmermann S, Poranne R, Coros S (2021) Dynamic manipulation of deformable objects with implicit integration. IEEE Robot Autom Lett 6(2):4209\u20134216","DOI":"10.1109\/LRA.2021.3066969"},{"key":"1048_CR9","doi-asserted-by":"crossref","unstructured":"Tang T, Fan Y, Lin HC, Tomizuka M (2017) State estimation for deformable objects by point registration and dynamic simulation. In: 2017 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 2427\u20132433","DOI":"10.1109\/IROS.2017.8206058"},{"key":"1048_CR10","unstructured":"Grannen J, Sundaresan P, Thananjeyan B, Ichnowski J, Balakrishna A, Viswanath V, Laskey M, Gonzalez JE, Goldberg K (2020) Learning robot policies for untangling dense knots in linear deformable structures. In: Conference on robot learning (CoRL), PMLR"},{"key":"1048_CR11","unstructured":"Kulkarni TD, Gupta A, Ionescu C, Borgeaud S, Reynolds M, Zisserman A, Mnih V (2019) Unsupervised learning of object keypoints for perception and control. Adv Neural Inf Process Syst 32:10723\u201310733"},{"key":"1048_CR12","unstructured":"Sanchez-Gonzalez A, Godwin J, Pfaff T, Ying R, Leskovec J, Battaglia P (2020) Learning to simulate complex physics with graph networks. In: International conference on machine learning. PMLR, pp 8459\u20138468"},{"key":"1048_CR13","unstructured":"Li Y, Wu J, Tedrake R, Tenenbaum JB, Torralba A (2019) Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In: ICLR (Poster). OpenReview.net"},{"key":"1048_CR14","doi-asserted-by":"crossref","unstructured":"Deng Y, Xia C, Wang X, Chen L (2022) Graph-transporter: a graph-based learning method for goal-conditioned deformable object rearranging task. In: 2022 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 1910\u20131916","DOI":"10.1109\/SMC53654.2022.9945180"},{"key":"1048_CR15","doi-asserted-by":"crossref","unstructured":"Deng Y, Xia C, Wang X, Chen L (2022) Deep reinforcement learning based on local GNN for goal-conditioned deformable object rearranging. In: 2022 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE","DOI":"10.1109\/IROS47612.2022.9981669"},{"key":"1048_CR16","doi-asserted-by":"crossref","unstructured":"Lim MH, Zeng A, Ichter B, Bandari M, Coumans E, Tomlin C, Schaal S, Faust A (2022) Multi-task learning with sequence-conditioned transporter networks. In: 2022 International conference on robotics and automation (ICRA). IEEE, pp 2489\u20132496","DOI":"10.1109\/ICRA46639.2022.9812096"},{"key":"1048_CR17","unstructured":"Matas J, James S, Davison AJ (2018) Sim-to-real reinforcement learning for deformable object manipulation. In: Conference on robot learning (CoRL). PMLR, pp 734\u2013743"},{"key":"1048_CR18","doi-asserted-by":"crossref","unstructured":"Nair A, Chen D, Agrawal P, Isola P, Abbeel P, Malik J, Levine S (2017) Combining self-supervised learning and imitation for vision-based rope manipulation. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2146\u20132153","DOI":"10.1109\/ICRA.2017.7989247"},{"key":"1048_CR19","unstructured":"Yan W, Vangipuram A, Abbeel P, Pinto L (2021) Learning predictive representations for deformable objects using contrastive estimation. In: Conference on robot learning. PMLR, pp 564\u2013574"},{"key":"1048_CR20","doi-asserted-by":"crossref","unstructured":"Wang C, Zhang Y, Zhang X, Wu Z, Zhu X, Jin S, Tang T, Tomizuka M (2022) Offline-online learning of deformation model for cable manipulation with graph neural networks. IEEE Robot Autom Lett 7(2):5544\u20135551","DOI":"10.1109\/LRA.2022.3158376"},{"key":"1048_CR21","doi-asserted-by":"crossref","unstructured":"Ma X, Hsu D, Lee WS (2022) Learning latent graph dynamics for visual manipulation of deformable objects. In: 2022 International conference on robotics and automation (ICRA). IEEE, pp 8266\u20138273","DOI":"10.1109\/ICRA46639.2022.9811597"},{"issue":"2","key":"1048_CR22","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1177\/0278364911430417","volume":"31","author":"S Miller","year":"2012","unstructured":"Miller S, Van Den Berg J, Fritz M, Darrell T, Goldberg K, Abbeel P (2012) A geometric approach to robotic laundry folding. Int J Robot Res 31(2):249\u2013267","journal-title":"Int J Robot Res"},{"key":"1048_CR23","doi-asserted-by":"crossref","unstructured":"Wu Y, Yan W, Kurutach T, Pinto L, Abbeel P (2020) Learning to manipulate deformable objects without demonstrations. In: Robotics: science and systems","DOI":"10.15607\/RSS.2020.XVI.065"},{"issue":"6","key":"1048_CR24","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1177\/0278364919841431","volume":"41","author":"T Tang","year":"2022","unstructured":"Tang T, Tomizuka M (2022) Track deformable objects from point clouds with structure preserved registration. Int J Robot Res 41(6):599\u2013614","journal-title":"Int J Robot Res"},{"key":"1048_CR25","unstructured":"Lin X, Wang Y, Olkin J, Held D (2021) Softgym: benchmarking deep reinforcement learning for deformable object manipulation. In: Conference on robot learning (CoRL). PMLR, pp 432\u2013448"},{"key":"1048_CR26","unstructured":"Zeng A, Florence P, Tompson J, Welker S, Chien J, Attarian M, Armstrong T, Krasin I, Duong D, Sindhwani V et al (2021) Transporter networks: rearranging the visual world for robotic manipulation. In: Conference on robot learning. PMLR, pp 726\u2013747"},{"key":"1048_CR27","first-page":"10724","volume":"32","author":"T Kulkarni","year":"2019","unstructured":"Kulkarni T, Gupta A, Ionescu C, Borgeaud S, Reynolds M, Zisserman A, Mnih V (2019) Unsupervised learning of object keypoints for perception and control. Adv Neural Inf Process Syst 32:10724\u201310734","journal-title":"Adv Neural Inf Process Syst"},{"key":"1048_CR28","unstructured":"Jakab T, Gupta A, Bilen H, Vedaldi A (2018) Unsupervised learning of object landmarks through conditional image generation. Adv Neural Inf Process Syst 31:4020\u20134031"},{"key":"1048_CR29","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:5998\u20136008"},{"key":"1048_CR30","unstructured":"Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (1). Association for Computational Linguistics, pp 4171\u20134186"},{"key":"1048_CR31","unstructured":"Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u20131901"},{"key":"1048_CR32","unstructured":"Coumans E, Bai Y (2016) Pybullet, a python module for physics simulation for games, robotics and machine learning. http:\/\/pybullet.org"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01048-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-01048-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01048-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T17:28:42Z","timestamp":1695403722000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-01048-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,12]]},"references-count":32,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["1048"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-01048-w","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,12]]},"assertion":[{"value":"1 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}