{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T17:55:04Z","timestamp":1774720504093,"version":"3.50.1"},"reference-count":39,"publisher":"Wiley","issue":"10","license":[{"start":{"date-parts":[[2024,7,3]],"date-time":"2024-07-03T00:00:00Z","timestamp":1719964800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003006","name":"Eidgen\u00f6ssische Technische Hochschule Z\u00fcrich","doi-asserted-by":"publisher","award":["SwissDataScienceCenterC22\u201008"],"award-info":[{"award-number":["SwissDataScienceCenterC22\u201008"]}],"id":[{"id":"10.13039\/501100003006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003006","name":"Eidgen\u00f6ssische Technische Hochschule Z\u00fcrich","doi-asserted-by":"publisher","award":["AICenterDoctoralFellowshipforHehuiZheng"],"award-info":[{"award-number":["AICenterDoctoralFellowshipforHehuiZheng"]}],"id":[{"id":"10.13039\/501100003006","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["advanced.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Advanced Intelligent Systems"],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:p>The precise control of soft and continuum robots requires knowledge of their shape, which has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built\u2010in sensors, resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on expensive tracking systems with reflective markers placed on all components, which are infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, a regression approach is presented for three\u2010dimensional\u00a0key point estimation using a convolutional neural network. The proposed approach uses data\u2010driven supervised learning and is capable of online markerless estimation during inference. Two images of a robotic system are captured\u00a0simultaneously at 25\u2009Hz from different perspectives and fed to the network, which returns for each pair the parameterized key point or piecewise constant curvature shape representations. The proposed approach outperforms markerless state\u2010of\u2010the\u2010art methods by a maximum of 4.5% in estimation accuracy while being more robust and requiring no prior knowledge of the shape. Online evaluations on two types of soft robotic arms and a soft robotic fish demonstrate the method's accuracy and versatility on highly deformable systems.<\/jats:p>","DOI":"10.1002\/aisy.202400105","type":"journal-article","created":{"date-parts":[[2024,7,3]],"date-time":"2024-07-03T07:20:19Z","timestamp":1719991219000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Vision\u2010Based Online Key Point Estimation of Deformable Robots"],"prefix":"10.1002","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4977-0220","authenticated-orcid":false,"given":"Hehui","family":"Zheng","sequence":"first","affiliation":[{"name":"Soft Robotics Lab ETH Zurich  Tannenstrasse 3 Z\u00fcrich 8092 Switzerland"},{"name":"ETH AI Center ETH Zurich  Andreasstrasse 5 Z\u00fcrich 8092 Switzerland"}]},{"given":"Sebastian","family":"Pinzello","sequence":"additional","affiliation":[{"name":"Soft Robotics Lab ETH Zurich  Tannenstrasse 3 Z\u00fcrich 8092 Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7810-6620","authenticated-orcid":false,"given":"Barnabas Gavin","family":"Cangan","sequence":"additional","affiliation":[{"name":"Soft Robotics Lab ETH Zurich  Tannenstrasse 3 Z\u00fcrich 8092 Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0254-811X","authenticated-orcid":false,"given":"Thomas J. K.","family":"Buchner","sequence":"additional","affiliation":[{"name":"Soft Robotics Lab ETH Zurich  Tannenstrasse 3 Z\u00fcrich 8092 Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7143-7259","authenticated-orcid":false,"given":"Robert K.","family":"Katzschmann","sequence":"additional","affiliation":[{"name":"Soft Robotics Lab ETH Zurich  Tannenstrasse 3 Z\u00fcrich 8092 Switzerland"},{"name":"ETH AI Center ETH Zurich  Andreasstrasse 5 Z\u00fcrich 8092 Switzerland"}]}],"member":"311","published-online":{"date-parts":[[2024,7,3]]},"reference":[{"key":"e_1_2_10_2_1","unstructured":"Y.Yamanaka S.Katagiri H.Nabae K.Suzumori G.Endo inIEEE\/SICE Int. Symp. Syst. Integration IEEE Piscataway NJ2020 pp.87\u201392."},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2020.103427"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abg6049"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-control-062322-100607"},{"key":"e_1_2_10_6_1","doi-asserted-by":"crossref","unstructured":"M.Park Y.Ohm D.Kim Y.\u2010L.Park inProc. IEEE Int. Conf. Soft Robot. IEEE Piscataway NJ2019 pp.384\u2013390.","DOI":"10.1109\/ROBOSOFT.2019.8722739"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1089\/soro.2018.0135"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/advs.201800541"},{"key":"e_1_2_10_9_1","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2021.630935"},{"key":"e_1_2_10_10_1","doi-asserted-by":"crossref","unstructured":"J. M.Bern Y.Schnider P.Banzet N.Kumar S.Coros inProc. IEEE Int. Conf. Soft Robot.IEEE Piscataway NJ2020 pp.417\u2013423.","DOI":"10.1109\/RoboSoft48309.2020.9116011"},{"key":"e_1_2_10_11_1","doi-asserted-by":"crossref","unstructured":"A.AlBeladi G.Krishnan M.\u2010A.Belabbas S.Hutchinson inProc. IEEE Int. Conf. Robot. Autom.IEEE Piscataway NJ2021 pp.11753\u201311759.","DOI":"10.1109\/ICRA48506.2021.9560751"},{"key":"e_1_2_10_12_1","doi-asserted-by":"crossref","unstructured":"R. K.Katzschmann C.Della Santina Y.Toshimitsu A.Bicchi D.Rus inProc. IEEE Int. Conf. Soft Robot.IEEE Piscataway NJ2019 pp.454\u2013461.","DOI":"10.1109\/ROBOSOFT.2019.8722799"},{"key":"e_1_2_10_13_1","doi-asserted-by":"crossref","unstructured":"A. D.Marchese K.Komorowski C. D.Onal D.Rus inProc. IEEE Int. Conf. Robot. Autom.IEEE Piscataway NJ2014 pp.2189\u20132196.","DOI":"10.1109\/ICRA.2014.6907161"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2015.1057238"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574704001018"},{"key":"e_1_2_10_16_1","doi-asserted-by":"crossref","unstructured":"D. B.Camarillo K. E.Loewke C. R.Carlson J. K.Salisbury inProc. IEEE Int. Conf. Robot. Automat.IEEE Piscataway NJ2008 pp.2940\u20132947.","DOI":"10.1109\/ROBOT.2008.4543656"},{"key":"e_1_2_10_17_1","doi-asserted-by":"crossref","unstructured":"J. M.Croom D. C.Rucker J. M.Romano R. J.Webster inProc. IEEE Int. Conf. Robot. Automat.IEEE Piscataway NJ2010pp.4591\u20134596.","DOI":"10.1109\/ROBOT.2010.5509461"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0087640"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2017.2690977"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.02.001"},{"key":"e_1_2_10_21_1","doi-asserted-by":"crossref","unstructured":"A.Reiter A.Bajo K.Iliopoulos N.Simaan P. K.Allen inProc. IEEE RAS & EMBS Int. Conf. Biomed. Robot. Biomechatronics IEEE Piscataway NJ2012 pp.829\u2013834.","DOI":"10.1109\/BioRob.2012.6290702"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-018-0209-y"},{"key":"e_1_2_10_23_1","unstructured":"P.Werner M.Hofer C.Sferrazza R.D\u2019Andrea arXiv:1909.09096 2019."},{"key":"e_1_2_10_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2975709"},{"key":"e_1_2_10_25_1","doi-asserted-by":"crossref","unstructured":"Y.She S. Q.Liu P.Yu E.Adelson inProc. IEEE Int. Conf. Robot. Autom.IEEE Piscataway NJ2020 pp.10075\u201310081.","DOI":"10.1109\/ICRA40945.2020.9197369"},{"key":"e_1_2_10_26_1","first-page":"2236121","volume":"4","author":"Bradski G.","year":"2000","journal-title":"Dr. Dobb's J Software Tools."},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364910368147"},{"key":"e_1_2_10_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2967269"},{"key":"e_1_2_10_29_1","unstructured":"K.Simonyan A.Zisserman inProc. Int. Conf. Learn. Representations Computational and Biological Learning Society USA2015 pp.1\u201314."},{"key":"e_1_2_10_30_1","doi-asserted-by":"crossref","unstructured":"K.He X.Zhang S.Ren J.Sun inProc. IEEE Conf. Comput. Vision Pattern Recognit. IEEE Piscataway NJ2016 pp.770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_10_31_1","unstructured":"M.Tan Q.Le inProc. Int. Conf. Mach. Learn. PMLR USA2019 pp.6105\u20136114."},{"key":"e_1_2_10_32_1","unstructured":"M.Tan Q.Le inProc. Int. Conf. Mach. Learn. PMLR USA2021 pp.10096\u201310106."},{"key":"e_1_2_10_33_1","doi-asserted-by":"crossref","unstructured":"E.Olson inProc. IEEE Int. Conf. Robot. Autom. IEEE Piscataway NJ2011 pp.3400\u20133407.","DOI":"10.1109\/ICRA.2011.5979561"},{"key":"e_1_2_10_34_1","doi-asserted-by":"crossref","unstructured":"Y.Toshimitsu K. W.Wong T.Buchner R.Katzschmann inProc. IEEE\/RSJ Int. Conf. Intell. Robots Syst. IEEE Piscataway NJ2021 pp.653\u2013660.","DOI":"10.1109\/IROS51168.2021.9636539"},{"key":"e_1_2_10_35_1","unstructured":"Y.Zhang R. K.Katzschmann arXiv:2201.04098 2022."},{"key":"e_1_2_10_36_1","unstructured":"Y.Pedari A.Parvaresh S. A. A.Moosavian inProc. Int. Conf. Robot. Mechatronics IEEE USA2019 pp.625\u2013630."},{"key":"e_1_2_10_37_1","unstructured":"L.Keselman J.Iselin Woodfill A.Grunnet\u2010Jepsen A.Bhowmik inProc. IEEE Conf. Comput. Vision Pattern Recognit. Workshops IEEE Piscataway NJ2017pp.1\u201310."},{"key":"e_1_2_10_38_1","unstructured":"I.Loshchilov F.Hutter arXiv:1711.05101 2017."},{"key":"e_1_2_10_39_1","doi-asserted-by":"crossref","unstructured":"H.Zheng R. K. Katzschmann.HehuiDatasets Accompanying Paper \u201cVision\u2010Based Online Key Point Estimation of Deformable Robots\u201d.ETH Zurich2024.https:\/\/doi.org\/10.3929\/ETHZ\u2010B\u2010000678014.","DOI":"10.1002\/aisy.202400105"},{"key":"e_1_2_10_40_1","doi-asserted-by":"crossref","unstructured":"H.Zheng S.Pinzello B. G.Cangan Thomas J. K.Buchner Robert K.Katzschmann Code accompanying the paper Vision\u2010Based Online Key Point Estimation of Deformable Robots2024.","DOI":"10.1002\/aisy.202400105"}],"container-title":["Advanced Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/advanced.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/aisy.202400105","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T18:53:18Z","timestamp":1759863198000},"score":1,"resource":{"primary":{"URL":"https:\/\/advanced.onlinelibrary.wiley.com\/doi\/10.1002\/aisy.202400105"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,3]]},"references-count":39,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["10.1002\/aisy.202400105"],"URL":"https:\/\/doi.org\/10.1002\/aisy.202400105","archive":["Portico"],"relation":{},"ISSN":["2640-4567","2640-4567"],"issn-type":[{"value":"2640-4567","type":"print"},{"value":"2640-4567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,3]]},"assertion":[{"value":"2024-02-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"2400105"}}