{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T16:22:33Z","timestamp":1782404553446,"version":"3.54.5"},"reference-count":29,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2024,8,11]],"date-time":"2024-08-11T00:00:00Z","timestamp":1723334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Proyectos Sin\u00e9rgicos de I+D la Comunidad de Madrid","award":["Y2020\/NMT-6660"],"award-info":[{"award-number":["Y2020\/NMT-6660"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Humanoid robots often struggle with tasks such as lifting objects due to the complexity involved in identifying contact points, applying the correct force, and tracking task progress. We propose an integrated solution that leverages the dual-arm capability of humanoids and utilizes sensor fusion from vision and force sensors. Our system employs a computer vision algorithm to detect and characterize object properties (shape, size, position, orientation) and differentiate between parallel and non-parallel bi-manipulation tasks. The controller then identifies optimal contact points for the end effectors, generating trajectories fed into a closed-loop controller using force feedback. For parallel bi-manipulation, momentum cancellation is achieved through sensor fusion. For non-parallel surfaces, a reinforcement learning algorithm determines the appropriate lifting force to prevent slippage using only two contact points. Experimental validation on a real humanoid platform demonstrates the effectiveness of our approach in autonomously lifting objects, regardless of contact surface configuration. This advancement significantly enhances the reliability and versatility of humanoid robots in performing complex manipulation tasks, contributing to their practical deployment in human-oriented environments.<\/jats:p>","DOI":"10.3390\/robotics13080121","type":"journal-article","created":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T05:19:38Z","timestamp":1723439978000},"page":"121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Hybrid Sensor Fusion Mixed with Reinforcement Learning in Autonomous Dual-Arm Lifting Tasks Performed by Humanoid Robots"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9370-9286","authenticated-orcid":false,"given":"Juan","family":"Hernandez-Vicen","sequence":"first","affiliation":[{"name":"RoboticsLab, University Carlos III de Madrid, 28045 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1052-3345","authenticated-orcid":false,"given":"Bartek","family":"\u0141ukawski","sequence":"additional","affiliation":[{"name":"RoboticsLab, University Carlos III de Madrid, 28045 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3539-4583","authenticated-orcid":false,"given":"Santiago","family":"Martinez","sequence":"additional","affiliation":[{"name":"RoboticsLab, University Carlos III de Madrid, 28045 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikos","family":"Tsagarakis","sequence":"additional","affiliation":[{"name":"Instituto Italiano di Tecnologia (IIT), 16163 Genova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carlos","family":"Balaguer","sequence":"additional","affiliation":[{"name":"RoboticsLab, University Carlos III de Madrid, 28045 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,11]]},"reference":[{"key":"ref_1","unstructured":"Karar, A.S., Said, S., and Beyrouthy, T. (2019, January 24\u201326). Pepper humanoid robot as a service robot: A customer approach. Proceedings of the 2019 3rd International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris, France."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"384","DOI":"10.5772\/56629","article-title":"Visual recognition system for cleaning tasks by humanoid robots","volume":"10","author":"Attamimi","year":"2013","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Garcia-Haro, J.M., O\u00f1a, E.D., Hernandez-Vicen, J., Martinez, S., and Balaguer, C. (2021). Service robots in catering applications: A review and future challenges. Electronics, 10.","DOI":"10.3390\/electronics10010047"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1109\/JAS.2017.7510604","article-title":"A survey of human-centered intelligent robots: Issues and challenges","volume":"4","author":"He","year":"2017","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Soroka, A.J., Qiu, R., Noyvirt, A., and Ji, Z. (2012, January 25\u201327). Challenges for service robots operating in non-industrial environments. Proceedings of the IEEE 10th International Conference on Industrial Informatics, Beijing, China.","DOI":"10.1109\/INDIN.2012.6301139"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kimura, N., Ito, K., Fuji, T., Fujimoto, K., Esaki, K., Beniyama, F., and Moriya, T. (2015, January 11\u201313). Mobile dual-arm robot for automated order picking system in warehouse containing various kinds of products. Proceedings of the 2015 IEEE\/SICE International Symposium on System Integration (SII), Nagoya, Japan.","DOI":"10.1109\/SII.2015.7404942"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Yan, L., Mu, Z., Xu, W., and Yang, B. (2016, January 9\u201314). Coordinated compliance control of dual-arm robot for payload manipulation: Master-slave and shared force control. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759419"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1109\/TMECH.2008.2002816","article-title":"Six-dof impedance control of dual-arm cooperative manipulators","volume":"13","author":"Caccavale","year":"2008","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Benali, K., Breth\u00e9, J.F., Gu\u00e9rin, F., and Gorka, M. (2018, January 19\u201322). Dual arm robot manipulator for grasping boxes of different dimensions in a logistics warehouse. Proceedings of the 2018 IEEE International Conference on Industrial Technology (ICIT), Lyon, France.","DOI":"10.1109\/ICIT.2018.8352167"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/S0736-5845(99)00043-5","article-title":"Robust control of planar dual-arm cooperative manipulators","volume":"16","author":"Liu","year":"2000","journal-title":"Robot. Comput. -Integr. Manuf."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Turella, L., and Lingnau, A. (2014). Neural correlates of grasping. Front. Hum. Neurosci., 8.","DOI":"10.3389\/fnhum.2014.00686"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/0010-0277(81)90036-6","article-title":"Specialized channels for cognitive responses","volume":"10","author":"Jeannerod","year":"1981","journal-title":"Cognition"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"129","DOI":"10.4995\/riai.2020.13097","article-title":"Principios b\u00e1sicos para el desarrollo de una aplicaci\u00f3n de bi-manipulaci\u00f3n de cajas por un robot humanoide","volume":"18","author":"Martinez","year":"2021","journal-title":"Rev. Iberoam. Autom\u00e1tica Inform\u00e1tica Ind."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1824","DOI":"10.1109\/ROBOT.2003.1241860","article-title":"Automatic grasp planning using shape primitives","volume":"Volume 2","author":"Miller","year":"2003","journal-title":"Proceedings of the 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422)"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Huebner, K., Ruthotto, S., and Kragic, D. (2008, January 9\u201313). Minimum volume bounding box decomposition for shape approximation in robot grasping. Proceedings of the 2008 IEEE International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ROBOT.2008.4543434"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Huebner, K., and Kragic, D. (2008, January 22\u201326). Selection of robot pre-grasps using box-based shape approximation. Proceedings of the 2008 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Nice, France.","DOI":"10.1109\/IROS.2008.4650722"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1340","DOI":"10.1016\/j.robot.2012.07.005","article-title":"Dual arm manipulation\u2014A survey","volume":"60","author":"Smith","year":"2012","journal-title":"Robot. Auton. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1007\/BF00237997","article-title":"Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects","volume":"56","author":"Johansson","year":"1984","journal-title":"Exp. Brain Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1038\/nrn2621","article-title":"Coding and use of tactile signals from the fingertips in object manipulation tasks","volume":"10","author":"Johansson","year":"2009","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hernandez-Vicen, J., Martinez, S., de Santos-Rico, R., Menendez, E., and Balaguer, C. (2021). Characterization and Study of the Primitive Dynamic Movements Required to Bi-Manipulate a Box. Electronics, 10.","DOI":"10.3390\/electronics10111354"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Surdilovic, D., Yakut, Y., Nguyen, T.M., Pham, X.B., Vick, A., and Martin-Martin, R. (2010, January 6\u20138). Compliance control with dual-arm humanoid robots: Design, planning and programming. Proceedings of the 2010 10th IEEE-RAS International Conference on Humanoid Robots, Nashville, TN, USA.","DOI":"10.1109\/ICHR.2010.5686273"},{"key":"ref_22","unstructured":"Prattichizzo, D., Trinkle, J.C., and Siciliano, B. (2008). Springer Handbook of Robotics, Springer. Chapter 28."},{"key":"ref_23","unstructured":"Chinellato, E. (2002). Robust Strategies for Selecting Vision-Based Planar Grasps of Unknown Objects with a Three-Finger Hand, Springer."},{"key":"ref_24","unstructured":"Su\u00e1rez Feij\u00f3o, R., Cornell\u00e0 Medrano, J., and Roa Garz\u00f3n, M. (2006). Grasp Quality Measures, Springer."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1177\/0278364914555544","article-title":"Robot grasp planning based on demonstrated grasp strategies","volume":"34","author":"Lin","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Honjo, R., and Katsura, S. (2015, January 9\u201312). Calculation of grasping force to adapt mass variation of object considering human motion. Proceedings of the IECON 2015-41st Annual Conference of the IEEE Industrial Electronics Society, Yokohama, Japan.","DOI":"10.1109\/IECON.2015.7392872"},{"key":"ref_27","first-page":"103","article-title":"Reinforcement learning: An introduction","volume":"21","author":"Thrun","year":"2000","journal-title":"AI Mag."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"J. Artif. Intell. Res."},{"key":"ref_29","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/13\/8\/121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:34:54Z","timestamp":1760110494000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/13\/8\/121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,11]]},"references-count":29,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["robotics13080121"],"URL":"https:\/\/doi.org\/10.3390\/robotics13080121","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,11]]}}}