{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T21:48:46Z","timestamp":1769723326280,"version":"3.49.0"},"reference-count":26,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2023,9,7]],"date-time":"2023-09-07T00:00:00Z","timestamp":1694044800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EU Horizon 2020 programme","award":["101016007"],"award-info":[{"award-number":["101016007"]}]},{"name":"EU Horizon 2020 programme","award":["871237"],"award-info":[{"award-number":["871237"]}]},{"name":"EU Horizon 2020 programme","award":["F\/190042\/01-03\/X44 RELAX"],"award-info":[{"award-number":["F\/190042\/01-03\/X44 RELAX"]}]},{"name":"PON I&amp;C 2014\u20132020","award":["101016007"],"award-info":[{"award-number":["101016007"]}]},{"name":"PON I&amp;C 2014\u20132020","award":["871237"],"award-info":[{"award-number":["871237"]}]},{"name":"PON I&amp;C 2014\u20132020","award":["F\/190042\/01-03\/X44 RELAX"],"award-info":[{"award-number":["F\/190042\/01-03\/X44 RELAX"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This manuscript introduces a mobile cobot equipped with a custom-designed high payload arm called RELAX combined with a novel unified multimodal interface that facilitates Human\u2013Robot Collaboration (HRC) tasks requiring high-level interaction forces on a real-world scale. The proposed multimodal framework is capable of combining physical interaction, Ultra Wide-Band (UWB) radio sensing, a Graphical User Interface (GUI), verbal control, and gesture interfaces, combining the benefits of all these different modalities and allowing humans to accurately and efficiently command the RELAX mobile cobot and collaborate with it. The effectiveness of the multimodal interface is evaluated in scenarios where the operator guides RELAX to reach designated locations in the environment while avoiding obstacles and performing high-payload transportation tasks, again in a collaborative fashion. The results demonstrate that a human co-worker can productively complete complex missions and command the RELAX mobile cobot using the proposed multimodal interaction framework.<\/jats:p>","DOI":"10.3390\/s23187735","type":"journal-article","created":{"date-parts":[[2023,9,7]],"date-time":"2023-09-07T10:11:07Z","timestamp":1694081467000},"page":"7735","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Unified Multimodal Interface for the RELAX High-Payload Collaborative Robot"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1265-3370","authenticated-orcid":false,"given":"Luca","family":"Muratore","sequence":"first","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arturo","family":"Laurenzi","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2854-6000","authenticated-orcid":false,"given":"Alessio","family":"De Luca","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"},{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering (DIBRIS), Universit\u00e0 di Genova, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liana","family":"Bertoni","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"},{"name":"Department of Information Engineering (DII), Universit\u00e0 di Pisa, 56122 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Davide","family":"Torielli","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"},{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering (DIBRIS), Universit\u00e0 di Genova, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lorenzo","family":"Baccelliere","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2138-5427","authenticated-orcid":false,"given":"Edoardo","family":"Del Bianco","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"},{"name":"DISI, Universit\u00e0 di Trento, 38123 Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nikos G.","family":"Tsagarakis","sequence":"additional","affiliation":[{"name":"Humanoids and Human-Centered Mechatronics Research Line, Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Colgate, J., Wannasuphoprasit, W., and Peshkin, M. (1996, January 17\u201322). Cobots: Robots for collaboration with human operators. Proceedings of the 1996 ASME International Mechanical Engineering Congress and Exposition, Atlanta, GA, USA.","DOI":"10.1115\/IMECE1996-0367"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Sherwani, F., Asad, M.M., and Ibrahim, B. (2020, January 26\u201327). Collaborative Robots and Industrial Revolution 4.0 (IR 4.0). Proceedings of the 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), Karachi, Pakistan.","DOI":"10.1109\/ICETST49965.2020.9080724"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1007\/s10514-017-9677-2","article-title":"Progress and Prospects of the Human\u2014Robot Collaboration","volume":"42","author":"Ajoudani","year":"2018","journal-title":"Auton. Robot."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Peternel, L., Tsagarakis, N., and Ajoudani, A. (2016, January 9\u201314). Towards multi-modal intention interfaces for human-robot co-manipulation. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea.","DOI":"10.1109\/IROS.2016.7759414"},{"key":"ref_5","unstructured":"Kirchner, E.A., Fairclough, S.H., and Kirchner, F. (2019). The Handbook of Multimodal-Multisensor Interfaces: Language Processing, Software, Commercialization, and Emerging Directions, Association for Computing Machinery and Morgan & Claypool."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1177\/1541931215591280","article-title":"Field Assessment of Multimodal Communication for Dismounted Human-Robot Teams","volume":"59","author":"Barber","year":"2015","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1109\/LRA.2016.2517206","article-title":"Constraint-Based and Sensorless Force Control With an Application to a Lightweight Dual-Arm Robot","volume":"1","author":"Nicolis","year":"2016","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1205","DOI":"10.1177\/0278364918765952","article-title":"A unified framework for coordinated multi-arm motion planning","volume":"37","author":"Salehian","year":"2018","journal-title":"Int. J. Robot. Res."},{"key":"ref_9","unstructured":"Wilhelm, B., Manfred, B., Braun, M., Rally, P., and Scholtz, O. (2022, December 29). Lightweight Robots in Manual Assembly\u2014Best to Start Simply! Examining Companies\u2019 Initial Experiences with Lightweight Robots. Available online: https:\/\/www.edig.nu\/assets\/images\/content\/Studie-Leichtbauroboter-Fraunhofer-IAO-2016-EN.pdf."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Vitolo, F., Rega, A., Di Marino, C., Pasquariello, A., Zanella, A., and Patalano, S. (2022). Mobile Robots and Cobots Integration: A Preliminary Design of a Mechatronic Interface by Using MBSE Approach. Appl. Sci., 12.","DOI":"10.3390\/app12010419"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1108\/01439911311297784","article-title":"Optimizing human-robot teleoperation interfaces for mobile manipulators","volume":"40","author":"Will","year":"2013","journal-title":"Ind. Robot"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hentout, A., Benbouali, M.R., Akli, I., Bouzouia, B., and Melkou, L. (2013, January 6\u20138). A telerobotic Human\/Robot Interface for mobile manipulators: A study of human operator performance. Proceedings of the 2013 International Conference on Control, Decision and Information Technologies (CoDIT), Hammamet, Tunisia.","DOI":"10.1109\/CoDIT.2013.6689618"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1729881417716043","DOI":"10.1177\/1729881417716043","article-title":"Natural multimodal communication for human\u2013robot collaboration","volume":"14","author":"Maurtua","year":"2017","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"De Paolis, L.T., and Mongelli, A. (2016, January 15\u201318). Transparency of a Bilateral Tele-Operation Scheme of a Mobile Manipulator Robot. Proceedings of the Augmented Reality, Virtual Reality, and Computer Graphics, Lecce, Italy.","DOI":"10.1007\/978-3-319-40651-0"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kim, W., Balatti, P., Lamon, E., and Ajoudani, A. (August, January 31). MOCA-MAN: A MObile and reconfigurable Collaborative Robot Assistant for conjoined huMAN-robot actions. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9197115"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2479","DOI":"10.1109\/LRA.2022.3144792","article-title":"TelePhysicalOperation: Remote Robot Control Based on a Virtual \u201cMarionette\u201d Type Interaction Interface","volume":"7","author":"Torielli","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3593","DOI":"10.1109\/LRA.2019.2928757","article-title":"A Teleoperation Interface for Loco-Manipulation Control of Mobile Collaborative Robotic Assistant","volume":"4","author":"Wu","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1595","DOI":"10.1109\/LRA.2019.2896758","article-title":"CENTAURO: A Hybrid Locomotion and High Power Resilient Manipulation Platform","volume":"4","author":"Kashiri","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1109\/MRA.2020.2979954","article-title":"The XBot Real-Time Software Framework for Robotics: From the Developer to the User Perspective","volume":"27","author":"Muratore","year":"2020","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"104379","DOI":"10.1016\/j.robot.2023.104379","article-title":"Tsagarakis, N.G.; Muratore, L.; The XBot2 real-time middleware for robotics","volume":"163","author":"Laurenzi","year":"2023","journal-title":"Robotics and Autonomous Systems"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Laurenzi, A., Hoffman, E.M., Muratore, L., and Tsagarakis, N.G. (2019, January 20\u201324). CartesI\/O: A ROS Based Real-Time Capable Cartesian Control Framework. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794464"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Haluza, M., and Vesely, J. (June, January 31). Analysis of signals from the DecaWave TREK1000 wideband positioning system using AKRS system. Proceedings of the 2017 International Conference on Military Technologies (ICMT), Brno, Czech Republic.","DOI":"10.1109\/MILTECHS.2017.7988797"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sang, C.L., Adams, M., Korthals, T., H\u00f6rmann, T., Hesse, M., and R\u00fcckert, U. (October, January 30). A Bidirectional Object Tracking and Navigation System using a True-Range Multilateration Method. Proceedings of the 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Pisa, Italy.","DOI":"10.1109\/IPIN.2019.8911811"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Siciliano, B., Sciavicco, L., Villani, L., and Oriolo, G. (2009). Robotics: Modelling, Planning and Control, Springer.","DOI":"10.1007\/978-1-84628-642-1"},{"key":"ref_25","unstructured":"Rottmann, N., Studt, N., Ernst, F., and Rueckert, E. (2020). ROS-Mobile: An Android application for the Robot Operating System. arXiv."},{"key":"ref_26","unstructured":"Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., and Lee, J. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7735\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:46:52Z","timestamp":1760129212000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7735"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,7]]},"references-count":26,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23187735"],"URL":"https:\/\/doi.org\/10.3390\/s23187735","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,7]]}}}