{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T16:06:19Z","timestamp":1778083579762,"version":"3.51.4"},"reference-count":45,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,10,6]],"date-time":"2023-10-06T00:00:00Z","timestamp":1696550400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Contact-rich robotic manipulation tasks such as assembly are widely studied due to their close relevance with social and manufacturing industries. Although the task is highly related to vision and force, current methods lack a unified mechanism to effectively fuse the two sensors. We consider coordinating multimodality from perception to control and propose a vision-force curriculum policy learning scheme to effectively fuse the features and generate policy. Experiments in simulations indicate the priorities of our method, which could insert pegs with 0.1 mm clearance. Furthermore, the system is generalizable to various initial configurations and unseen shapes, and it can be robustly transferred from simulation to reality without fine-tuning, showing the effectiveness and generalization of our proposed method. The experiment videos and code will be available at <jats:ext-link>https:\/\/sites.google.com\/view\/vf-assembly<\/jats:ext-link>.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1280773","type":"journal-article","created":{"date-parts":[[2023,10,6]],"date-time":"2023-10-06T07:31:22Z","timestamp":1696577482000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Vision-force-fused curriculum learning for robotic contact-rich assembly tasks"],"prefix":"10.3389","volume":"17","author":[{"given":"Piaopiao","family":"Jin","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yinjie","family":"Lin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yaoxian","family":"Song","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tiefeng","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,10,6]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/0278364919887447","article-title":"Learning dexterous in-hand manipulation","volume":"39","author":"Andrychowicz","year":"2020","journal-title":"Int. J. Rob. Res"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1553374.1553380","article-title":"\u201cCurriculum learning,\u201d","author":"Bengio","year":"2009","journal-title":"Proceedings of the 26th Annual International Conference on Machine Learning"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.14401","article-title":"Sim2real for peg-hole insertion with eye-in-hand camera","author":"Bogunowicz","year":"2020","journal-title":"arXiv"},{"key":"B4","first-page":"8973","article-title":"\u201cClosing the sim-to-real loop: adapting simulation randomization with real world experience,\u201d","volume-title":"Proc. IEEE Int. Conf. Robot. Automat","author":"Chebotar","year":"2019"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1465","DOI":"10.1109\/IROS.2001.977187","article-title":"\u201cSearch strategies for peg-in-hole assemblies with position uncertainty,\u201d","author":"Chhatpar","year":"2001","journal-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst"},{"key":"B6","doi-asserted-by":"publisher","first-page":"6437","DOI":"10.1109\/ICRA48506.2021.9561646","article-title":"\u201cTactile-rl for insertion: generalization to objects of unknown geometry,\u201d","author":"Dong","year":"2021","journal-title":"Proc. IEEE Int. Conf. Robot. Automat"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2962","DOI":"10.1109\/LRA.2021.3062315","article-title":"KPAM 2.0: feedback control for category-level robotic manipulation","volume":"6","author":"Gao","year":"2021","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B8","first-page":"1696","article-title":"\u201cFast robust peg-in-hole insertion with continuous visual servoing,\u201d","volume-title":"Proceedings of the 2020 Conference on Robot Learning, Volume 155 of Proceedings of Machine Learning Research","author":"Haugaard","year":"2021"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90","article-title":"\u201cDeep residual learning for image recognition,\u201d","author":"He","year":"2016","journal-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B10","volume-title":"Stable Baselines","author":"Hill","year":"2018"},{"key":"B11","doi-asserted-by":"crossref","first-page":"304","DOI":"10.23919\/ACC.1984.4788393","article-title":"\u201cImpedance control: an approach to manipulation,\u201d","volume-title":"1984 American Control Conference","author":"Hogan","year":"1984"},{"key":"B12","first-page":"1097","article-title":"\u201cAdaptive hybrid visual servoing\/force control in unknown environment,\u201d","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst., Volume","author":"Hosoda","year":"1996"},{"key":"B13","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1109\/IROS.2017.8202244","article-title":"\u201cDeep reinforcement learning for high precision assembly tasks,\u201d","volume-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Inoue","year":"2017"},{"key":"B14","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1109\/JRA.1987.1087068","article-title":"A unified approach for motion and force control of robot manipulators: the operational space formulation","volume":"3","author":"Khatib","year":"1987","journal-title":"IEEE J. Robot. Autom"},{"key":"B15","doi-asserted-by":"publisher","first-page":"10444","DOI":"10.1109\/IROS45743.2020.9341579","article-title":"\u201cMultimodal sensor fusion with differentiable filters,\u201d","author":"Lee","year":"","journal-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst"},{"key":"B16","doi-asserted-by":"publisher","first-page":"582","DOI":"10.1109\/TRO.2019.2959445","article-title":"Making sense of vision and touch: learning multimodal representations for contact-rich tasks","volume":"36","author":"Lee","year":"","journal-title":"IEEE Trans. Robot"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1478","DOI":"10.1109\/LRA.2021.3139675","article-title":"Unified method for task-space motion\/force\/impedance control of manipulator with unknown contact reaction strategy","volume":"7","author":"Lin","year":"2022","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B18","doi-asserted-by":"publisher","first-page":"3080","DOI":"10.1109\/ICRA.2019.8793506","article-title":"\u201cReinforcement learning on variable impedance controller for high-precision robotic assembly,\u201d","author":"Luo","year":"2019","journal-title":"Proc. IEEE Int. Conf. Robot. Automat"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.11512","article-title":"Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: a large-scale study","author":"Luo","year":"2021","journal-title":"arXiv"},{"key":"B20","first-page":"170","article-title":"\u201cExternal hybrid vision\/force control,\u201d","author":"Mezouar","year":"2007","journal-title":"Proc. Int. Conf. Adv. Robot"},{"key":"B21","first-page":"8762","article-title":"\u201cMulti-view picking: next-best-view reaching for improved grasping in clutter,\u201d","volume-title":"Proc. IEEE Int. Conf. Robot. Automat","author":"Morrison","year":"2019"},{"key":"B22","doi-asserted-by":"crossref","first-page":"7154","DOI":"10.1109\/ICRA48891.2023.10161491","article-title":"\u201cLearning on the job: self-rewarding offline-to-online finetuning for industrial insertion of novel connectors from vision,\u201d","volume-title":"2023 IEEE International Conference on Robotics and Automation (ICRA)","author":"Nair","year":"2023"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2737","DOI":"10.1109\/LRA.2021.3060389","article-title":"Reinforcement learning for robotic assembly using non-diagonal stiffness matrix","volume":"6","author":"Oikawa","year":"2021","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B24","doi-asserted-by":"publisher","first-page":"4441","DOI":"10.1109\/LRA.2021.3068911","article-title":"A general visual-impedance framework for effectively combining vision and force sensing in feature space","volume":"6","author":"Oliva","year":"2021","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B25","first-page":"1","article-title":"\u201cSim-to-real transfer of robotic control with dynamics randomization,\u201d","volume-title":"Proc. IEEE Int. Conf. Robot. Automat","author":"Peng","year":"2018"},{"key":"B26","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1115\/1.3139652","article-title":"Hybrid position\/force control of manipulators","volume":"103","author":"Raibert","year":"1981","journal-title":"J. Dyn. Syst. Meas. Control"},{"key":"B27","first-page":"5548","article-title":"\u201cDeep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards,\u201d","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Schoettler","year":"2019"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1707.06347","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017","journal-title":"arXiv"},{"key":"B29","first-page":"156","article-title":"\u201cLearning contact-rich manipulation skills with guided policy search,\u201d","volume-title":"Proc. IEEE Int. Conf. Robot. Automat","author":"Sergey","year":"2015"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1753","DOI":"10.2991\/ijcis.d.210531.001","article-title":"Tactile\u2013visual fusion based robotic grasp detection method with a reproducible sensor","volume":"14","author":"Song","year":"2021","journal-title":"Int. J. Comput. Intell. Syst"},{"key":"B31","doi-asserted-by":"crossref","first-page":"6330","DOI":"10.1109\/ICRA46639.2022.9811798","article-title":"\u201cInsertionnet 2.0: minimal contact multi-step insertion using multimodal multiview sensory input,\u201d","volume-title":"2022 International Conference on Robotics and Automation (ICRA)","author":"Spector","year":"2022"},{"key":"B32","first-page":"6023","article-title":"\u201cLearning contact-rich assembly skills using residual admittance policy,\u201d","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst","author":"Spector","year":"2021"},{"key":"B33","doi-asserted-by":"publisher","first-page":"1159","DOI":"10.1109\/LRA.2020.2967325","article-title":"Learning to assemble: estimating 6D poses for robotic object-object manipulation","volume":"5","author":"[Stev\u015di\u0107","year":"2020","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B34","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"B35","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1109\/COASE.2016.7743375","article-title":"\u201cAutonomous alignment of peg and hole by force\/torque measurement for robotic assembly,\u201d","author":"Tang","year":"2016","journal-title":"Proc. IEEE Int. Conf. Autom. Sci. Eng"},{"key":"B36","doi-asserted-by":"publisher","first-page":"5026","DOI":"10.1109\/IROS.2012.6386109","article-title":"\u201cMuJoCo: a physics engine for model-based control,\u201d","author":"Todorov","year":"2012","journal-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst"},{"key":"B37","first-page":"5792","article-title":"\u201cQuickly inserting pegs into uncertain holes using multi-view images and deep network trained on synthetic data,\u201d","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst","author":"Triyonoputro","year":"2019"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1674","DOI":"10.1109\/TMECH.2022.3224907","article-title":"Peg-in-hole using transient information of force response","volume":"28","author":"Unten","year":"2023","journal-title":"IEEE\/ASME Trans. Mechatron"},{"key":"B39","first-page":"3928","article-title":"\u201cStable reinforcement learning with autoencoders for tactile and visual data,\u201d","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst","author":"Van Hoof","year":"2016"},{"key":"B40","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1115\/1.3149634","article-title":"Quasi-static assembly of compliantly supported rigid parts","volume":"104","author":"Whitney","year":"1982","journal-title":"J. Dyn. Syst. Meas. Control"},{"key":"B41","doi-asserted-by":"crossref","DOI":"10.1109\/TMECH.2023.3313516","article-title":"A bi-objective scheme for kinematic control of mobile robotic arms with manipulability optimization","author":"Xie","year":"2023","journal-title":"IEEE\/ASME Trans. Mechatron"},{"key":"B42","doi-asserted-by":"publisher","first-page":"3478","DOI":"10.1109\/LRA.2023.3261759","article-title":"Geometric-feature representation based pre-training method for reinforcement learning of peg-in-hole tasks","volume":"8","author":"Zang","year":"2023","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B43","first-page":"726","article-title":"\u201cTransporter networks Rearranging the visual world for robotic manipulation,\u201d","volume-title":"Conference on Robot Learning (PMLR)","author":"Zeng","year":"2021"},{"key":"B44","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1109\/ICRA48891.2023.10161116","article-title":"\u201cVision-based six-dimensional peg-in-hole for practical connector insertion,\u201d","volume-title":"2023 IEEE International Conference on Robotics and Automation (ICRA)","author":"Zhang","year":"2023"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2110.04276","article-title":"Offline meta-reinforcement learning for industrial insertion","author":"Zhao","year":"2021","journal-title":"arXiv"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280773\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,6]],"date-time":"2023-10-06T07:31:43Z","timestamp":1696577503000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280773\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,6]]},"references-count":45,"alternative-id":["10.3389\/fnbot.2023.1280773"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1280773","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,6]]},"article-number":"1280773"}}