{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T02:49:42Z","timestamp":1768531782397,"version":"3.49.0"},"reference-count":62,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T00:00:00Z","timestamp":1758758400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100020963","name":"Moonshot Research and Development Program","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100020963","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>One weakness of human-robot interaction (HRI) research is the lack of reproducible results, due to the lack of standardised benchmarks. In this work we introduce a multi-user multi-robot multi-goal multi-device manipulation benchmark (M4Bench), a flexible HRI platform in which multiple users can direct either a single\u2014or multiple\u2014simulated robots to perform a multi-goal pick-and-place task. Our software exposes a web-based visual interface, with support for mouse, keyboard, gamepad, eye tracker and electromyograph\/electroencephalograph (EMG\/EEG) user inputs. It can be further extended using native browser libraries or WebSocket interfaces, allowing researchers to add support for their own devices. We also provide tracking for several HRI metrics, such as task completion and command selection time, enabling quantitative comparisons between different user interfaces and devices. We demonstrate the utility of our benchmark with a user study (n = 50) conducted to compare five different input devices, and also compare single-vs. multi-user control. In the pick-and-place task, we found that users performed worse when using the eye tracker + EMG device pair, as compared to mouse + keyboard or gamepad + gamepad, over four quantitative metrics (corrected p <jats:inline-formula><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" id=\"m1\"><mml:mrow><mml:mo>&lt;<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:inline-formula> 0.001). Our software is available at <jats:ext-link>https:\/\/github.com\/arayabrain\/m4bench<\/jats:ext-link>.<\/jats:p>","DOI":"10.3389\/frobt.2025.1528754","type":"journal-article","created":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T04:16:45Z","timestamp":1758773805000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A multi-user multi-robot multi-goal multi-device human-robot interaction manipulation benchmark"],"prefix":"10.3389","volume":"12","author":[{"given":"Akito","family":"Yoshida","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rousslan Fernand Julien","family":"Dossa","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marina","family":"Di Vincenzo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shivakanth","family":"Sujit","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hannah","family":"Douglas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kai","family":"Arulkumaran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,9,25]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.apergo.2017.03.008","article-title":"Hri usability evaluation of interaction modes for a teleoperated agricultural robotic sprayer","volume":"62","author":"Adamides","year":"2017","journal-title":"Appl. Ergon."},{"key":"B2","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1007\/s10514-017-9677-2","article-title":"Progress and prospects of the human\u2013robot collaboration","volume":"42","author":"Ajoudani","year":"2018","journal-title":"Aut. robots"},{"key":"B3","doi-asserted-by":"publisher","first-page":"15549","DOI":"10.3390\/s131115549","article-title":"A multimodal emotion detection system during human\u2013robot interaction","volume":"13","author":"Alonso-Martin","year":"2013","journal-title":"Sensors"},{"key":"B4","doi-asserted-by":"publisher","first-page":"181855","DOI":"10.1109\/access.2020.3028740","article-title":"Object detection recognition and robot grasping based on machine learning: a survey","volume":"8","author":"Bai","year":"2020","journal-title":"IEEE access"},{"key":"B5","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/978-3-319-27146-0_9","article-title":"Expressive lights for revealing mobile service robot state","volume":"1","author":"Baraka","year":"2016","journal-title":"Robot 2015 Second Iber. Robotics Conf. Adv. Robotics"},{"key":"B6","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1007\/s12369-010-0064-9","article-title":"Review of human studies methods in hri and recommendations","volume":"2","author":"Bethel","year":"2010","journal-title":"Int. J. Soc. Robotics"},{"key":"B7","first-page":"4","article-title":"Sus-a quick and dirty usability scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. industry"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1522","DOI":"10.1177\/154193120805201946","article-title":"Measurement invariance of the nasa tlx","volume":"52","author":"Bustamante","year":"2008","journal-title":"Proc. Hum. factors ergonomics Soc. Annu. Meet."},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.00081","article-title":"Partnr: a benchmark for planning and reasoning in embodied multi-agent tasks","author":"Chang","year":"2024","journal-title":"arXiv Prepr. arXiv:2411.00081"},{"key":"B10","doi-asserted-by":"crossref","DOI":"10.1016\/j.robot.2022.104335","volume-title":"A survey of multi-agent human-robot interaction systems","author":"Dahiya","year":"2023"},{"key":"B11","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.jneumeth.2003.10.009","article-title":"Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis","volume":"134","author":"Delorme","year":"2004","journal-title":"J. Neurosci. methods"},{"key":"B12","first-page":"549","article-title":"Exploring multimodal interactions with a robot assistant in an assembly task: a human-centered design approach","volume-title":"VISIGRAPP, GRAPP, HUCAPP, IVAPP","author":"D\u2019Attanasio","year":"2024"},{"key":"B13","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1145\/3568294.3580086","article-title":"The warehouse robot interaction sim: an open-source hri research platform","volume-title":"Companion of the 2023 ACM\/IEEE international conference on human-robot interaction","author":"Esterwood","year":"2023"},{"key":"B14","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1007\/978-3-319-91122-9_44","article-title":"Playbook for uas: ux of goal-oriented planning and execution","volume-title":"Engineering psychology and cognitive ergonomics: 15th international conference, EPCE 2018, held as part of HCI international 2018, Las Vegas, NV, USA, July 15-20, 2018, proceedings 15","author":"Gale","year":"2018"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2280","DOI":"10.1016\/j.patcog.2014.01.005","article-title":"Automatic generation and detection of highly reliable fiducial markers under occlusion","volume":"47","author":"Garrido-Jurado","year":"2014","journal-title":"Pattern Recognit."},{"key":"B16","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1561\/1100000005","article-title":"Human\u2013robot interaction: a survey","volume":"1","author":"Goodrich","year":"2008","journal-title":"Found. Trends\u00ae Human\u2013Computer Interact."},{"key":"B17","doi-asserted-by":"publisher","first-page":"160044","DOI":"10.1038\/sdata.2016.44","article-title":"The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments","volume":"3","author":"Gorgolewski","year":"2016","journal-title":"Sci. data"},{"key":"B18","doi-asserted-by":"publisher","first-page":"267","DOI":"10.3389\/fnins.2013.00267","article-title":"Meg and eeg data analysis with mne-python","volume":"267","author":"Gramfort","year":"2013","journal-title":"Front. Neurosci."},{"key":"B19","doi-asserted-by":"publisher","first-page":"2054","DOI":"10.1109\/TNSRE.2017.2703586","article-title":"Toward multimodal human\u2013robot interaction to enhance active participation of users in gait rehabilitation","volume":"25","author":"Gui","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabilitation Eng."},{"key":"B20","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1145\/1357054.1357076","article-title":"Exploring the use of tangible user interfaces for human-robot interaction: a comparative study","author":"Guo","year":"2008","journal-title":"Proc. SIGCHI Conf. Hum. Factors Comput. Syst."},{"key":"B21","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"Nasa-task load index (Nasa-tlx); 20 years later","volume":"50","author":"Hart","year":"2006","journal-title":"Proc. Hum. factors ergonomics Soc. Annu. Meet."},{"key":"B22","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/s0166-4115(08)62386-9","article-title":"Development of nasa-tlx (task load index): results of empirical and theoretical research","author":"Hart","year":"1988","journal-title":"Hum. Ment. workload\/Elsevier"},{"key":"B23","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1109\/thms.2019.2904558","article-title":"Evaluating fluency in human\u2013robot collaboration","volume":"49","author":"Hoffman","year":"2019","journal-title":"IEEE Trans. Human-Machine Syst."},{"key":"B24","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1145\/2696454.2696479","article-title":"When children teach a robot to write: an autonomous teachable humanoid which uses simulated handwriting","volume-title":"Proceedings of the tenth annual ACM\/IEEE international conference on human-robot interaction","author":"Hood","year":"2015"},{"key":"B25","doi-asserted-by":"crossref","first-page":"11633","DOI":"10.1109\/ICRA48891.2023.10161412","article-title":"Interacting with multi-robot systems via mixed reality","volume-title":"2023 IEEE international conference on robotics and automation (ICRA)","author":"Kennel-Maushart","year":"2023"},{"key":"B26","doi-asserted-by":"publisher","first-page":"06828","DOI":"10.48550\/arXiv.2302.06828","article-title":"RoboHive: a unified framework for robot learning","author":"Kumar","year":"2023","journal-title":"arXiv"},{"key":"B27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/SSRR.2015.7443000","article-title":"A supervisory control method for multi-robot task allocation in urban search and rescue","volume-title":"2015 IEEE international symposium on safety, security, and rescue robotics (SSRR)","author":"Liu","year":"2015"},{"key":"B28","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1109\/ISPCC.2017.8269699","article-title":"Comparative empirical usability assessment of two hri input devices for a mobile robot","volume-title":"2017 4th international conference on signal processing, computing and control (ISPCC)","author":"Mallan","year":"2017"},{"key":"B29","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1109\/ICRA57147.2024.10610855","article-title":"Roco: dialectic multi-robot collaboration with large language models","volume-title":"2024 IEEE international conference on robotics and automation (ICRA)","author":"Mandi","year":"2024"},{"key":"B30","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1109\/HRI.2013.6483569","article-title":"Survey of metrics for human-robot interaction","volume-title":"2013 8th ACM\/IEEE international conference on human-robot interaction (HRI)","author":"Murphy","year":"2013"},{"key":"B31","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1145\/2909824.3020253","article-title":"Game-theoretic modeling of human adaptation in human-robot collaboration","volume-title":"Proceedings of the 2017 ACM\/IEEE international conference on human-robot interaction","author":"Nikolaidis","year":"2017"},{"key":"B32","first-page":"4","article-title":"Metrics for evaluating human-robot interactions","volume":"2003","author":"Olsen","year":"2003","journal-title":"Proc. PERMIS (Citeseer)"},{"key":"B33","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1145\/2909824.3020231","article-title":"Windfield: learning wind meteorology with handheld haptic robots","volume-title":"Proceedings of the 2017 ACM\/IEEE international conference on human-robot interaction","author":"\u00d6zg\u00fcr","year":"2017"},{"key":"B34","doi-asserted-by":"publisher","first-page":"2825","DOI":"10.5555\/1953048.2078195","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"B35","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/s11721-016-0124-3","article-title":"Investigating the effect of increasing robot group sizes on the human psychophysiological state in the context of human\u2013swarm interaction","volume":"10","author":"Podevijn","year":"2016","journal-title":"Swarm Intell."},{"key":"B36","doi-asserted-by":"crossref","first-page":"3395","DOI":"10.1109\/IROS.2018.8594231","article-title":"The power of color: a study on the effective use of colored light in human-robot interaction","volume-title":"2018 IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"P\u00f6rtner","year":"2018"},{"key":"B37","first-page":"8494","article-title":"Virtualhome: simulating household activities via programs","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Puig","year":"2018"},{"key":"B38","article-title":"Watch-and-help: a challenge for social perception and human-ai collaboration","author":"Puig","year":"2020","journal-title":"arXiv Prepr. arXiv:2010.09890"},{"key":"B39","doi-asserted-by":"publisher","first-page":"918","DOI":"10.3390\/s17040918","article-title":"Autonomous multi-robot search for a hazardous source in a turbulent environment","volume":"17","author":"Ristic","year":"2017","journal-title":"Sensors"},{"key":"B40","doi-asserted-by":"crossref","first-page":"2702","DOI":"10.1109\/ICASSP.2016.7472168","article-title":"Multimodal human action recognition in assistive human-robot interaction","volume-title":"2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)","author":"Rodomagoulakis","year":"2016"},{"key":"B41","article-title":"Human-robot interaction: final report for darpa\/nsf study on human-robot interaction","author":"Rogers","year":"2002"},{"key":"B42","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1016\/j.artint.2017.08.005","article-title":"Intelligent agent supporting human\u2013multi-robot team collaboration","volume":"252","author":"Rosenfeld","year":"2017","journal-title":"Artif. Intell."},{"key":"B43","doi-asserted-by":"publisher","first-page":"4","DOI":"10.3390\/robotics12010004","article-title":"Measuring performance: metrics for manipulator design, control, and optimization","volume":"12","author":"Russo","year":"2022","journal-title":"Robotics"},{"key":"B44","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1007\/s12193-023-00421-w","article-title":"Comparing alternative modalities in the context of multimodal human-robot interaction","volume":"18","author":"Saren","year":"2024","journal-title":"JMUI"},{"key":"B45","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1109\/ROMAN.2017.8172325","article-title":"Proactive, incremental learning of gesture-action associations for human-robot collaboration","volume-title":"2017 26th IEEE international symposium on robot and human interactive communication (RO-MAN)","author":"Shukla","year":"2017"},{"key":"B46","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1145\/1121241.1121249","article-title":"Common metrics for human-robot interaction","volume-title":"Proceedings of the 1st ACM SIGCHI\/SIGART conference on Human-robot interaction","author":"Steinfeld","year":"2006"},{"key":"B47","doi-asserted-by":"publisher","first-page":"923","DOI":"10.3390\/s22030923","article-title":"Robot system assistant (rosa): towards intuitive multi-modal and multi-device human-robot interaction","volume":"22","author":"Strazdas","year":"2022","journal-title":"Sensors"},{"key":"B48","doi-asserted-by":"publisher","first-page":"1084000","DOI":"10.3389\/fnbot.2023.1084000","article-title":"Recent advancements in multimodal human\u2013robot interaction","volume":"17","author":"Su","year":"2023","journal-title":"Front. Neurorobotics"},{"key":"B49","doi-asserted-by":"publisher","first-page":"17305","DOI":"10.1109\/access.2023.3245833","article-title":"Multimodal multi-user mixed reality human\u2013robot interface for remote operations in hazardous environments","volume":"11","author":"Szczurek","year":"2023","journal-title":"IEEE Access"},{"key":"B50","doi-asserted-by":"publisher","first-page":"103029","DOI":"10.1016\/j.bspc.2021.103029","article-title":"A multimodal emotion recognition method based on facial expressions and electroencephalography","volume":"70","author":"Tan","year":"2021","journal-title":"Biomed. Signal Process. Control"},{"key":"B51","doi-asserted-by":"crossref","first-page":"7405","DOI":"10.1109\/ICRA57147.2024.10610705","article-title":"Human-robot gym: benchmarking reinforcement learning in human-robot collaboration","volume-title":"2024 IEEE international conference on robotics and automation (ICRA)","author":"Thumm","year":"2024"},{"key":"B52","volume-title":"MuJoCo: a physics engine for model-based controlIROS","author":"Todorov","year":"2012"},{"key":"B53","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1016\/j.mechatronics.2018.02.009","article-title":"Survey on human\u2013robot collaboration in industrial settings: safety, intuitive interfaces and applications","volume":"55","author":"Villani","year":"2018","journal-title":"Mechatronics"},{"key":"B54","doi-asserted-by":"publisher","first-page":"601","DOI":"10.1007\/s10514-019-09889-6","article-title":"Humans interacting with multi-robot systems: a natural affect-based approach","volume":"44","author":"Villani","year":"2020","journal-title":"Aut. Robots"},{"key":"B55","doi-asserted-by":"crossref","first-page":"4193","DOI":"10.1109\/IROS.2016.7759617","article-title":"Apriltag 2: efficient and robust fiducial detection","volume-title":"2016 IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"Wang","year":"2016"},{"key":"B56","doi-asserted-by":"publisher","first-page":"102165","DOI":"10.1016\/j.techsoc.2022.102165","article-title":"Multi-actor perspectives on human robotic collaboration implementation in the heavy automotive manufacturing industry-a swedish case study","volume":"72","author":"Wang","year":"2023","journal-title":"Technol. Soc."},{"key":"B57","first-page":"1","article-title":"Lami: large language models for multi-modal human-robot interaction","volume-title":"Extended abstracts of the CHI conference on human factors in computing systems","author":"Wang","year":""},{"key":"B58","doi-asserted-by":"publisher","first-page":"2300359","DOI":"10.1002\/aisy.202300359","article-title":"Multimodal human\u2013robot interaction for human-centric smart manufacturing: a survey","volume":"6","author":"Wang","year":"","journal-title":"Adv. Intell. Syst."},{"key":"B59","article-title":"Hydra - a framework for elegantly configuring complex applications","author":"Yadan","year":"2019","journal-title":"Github"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.02485","article-title":"Building cooperative embodied agents modularly with large language models","author":"Zhang","year":"2023","journal-title":"arXiv Prepr. arXiv:2307.02485"},{"key":"B61","doi-asserted-by":"crossref","DOI":"10.1109\/HRI53351.2022.9889595","article-title":"An analysis of metrics and methods in research from human-robot interaction conferences, 2015-2021","volume-title":"HRI","author":"Zimmerman","year":"2022"},{"key":"B62","doi-asserted-by":"crossref","first-page":"1019","DOI":"10.1109\/ICRA57147.2024.10611462","article-title":"Language and sketching: an llm-driven interactive multimodal multitask robot navigation framework","volume-title":"2024 IEEE international conference on robotics and automation (ICRA)","author":"Zu","year":"2024"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1528754\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T04:16:52Z","timestamp":1758773812000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1528754\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,25]]},"references-count":62,"alternative-id":["10.3389\/frobt.2025.1528754"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2025.1528754","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,25]]},"article-number":"1528754"}}