{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T20:27:09Z","timestamp":1776371229741,"version":"3.51.2"},"reference-count":59,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,1,25]],"date-time":"2022-01-25T00:00:00Z","timestamp":1643068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Federal Ministry of Education and Research","doi-asserted-by":"publisher","award":["03ZZ0448L"],"award-info":[{"award-number":["03ZZ0448L"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Federal Ministry of Education and Research","doi-asserted-by":"publisher","award":["03ZZ04X02B"],"award-info":[{"award-number":["03ZZ04X02B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents an implementation of RoSA, a Robot System Assistant, for safe and intuitive human-machine interaction. The interaction modalities were chosen and previously reviewed using a Wizard of Oz study emphasizing a strong propensity for speech and pointing gestures. Based on these findings, we design and implement a new multi-modal system for contactless human-machine interaction based on speech, facial, and gesture recognition. We evaluate our proposed system in an extensive study with multiple subjects to examine the user experience and interaction efficiency. It reports that our method achieves similar usability scores compared to the entirely human remote-controlled robot interaction in our Wizard of Oz study. Furthermore, our framework\u2019s implementation is based on the Robot Operating System (ROS), allowing modularity and extendability for our multi-device and multi-user method.<\/jats:p>","DOI":"10.3390\/s22030923","type":"journal-article","created":{"date-parts":[[2022,1,25]],"date-time":"2022-01-25T21:07:11Z","timestamp":1643144831000},"page":"923","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":48,"title":["Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0463-6781","authenticated-orcid":false,"given":"Dominykas","family":"Strazdas","sequence":"first","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2910-8202","authenticated-orcid":false,"given":"Jan","family":"Hintz","sequence":"additional","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9179-6061","authenticated-orcid":false,"given":"Aly","family":"Khalifa","sequence":"additional","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1575-0942","authenticated-orcid":false,"given":"Ahmed A.","family":"Abdelrahman","sequence":"additional","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3621-7194","authenticated-orcid":false,"given":"Thorsten","family":"Hempel","sequence":"additional","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3632-2402","authenticated-orcid":false,"given":"Ayoub","family":"Al-Hamadi","sequence":"additional","affiliation":[{"name":"Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,25]]},"reference":[{"key":"ref_1","first-page":"1227","article-title":"Kollaborative Robotikanwendungen an Montagearbeitspl\u00e4tzen","volume":"57","author":"Rusch","year":"2020","journal-title":"HMD Prax. Wirtsch."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hasnain, S.K., Mostafaoui, G., Salesse, R., Marin, L., and Gaussier, P. (2013, January 18\u201322). Intuitive human robot interaction based on unintentional synchrony: A psycho-experimental study. Proceedings of the 2013 IEEE Third Joint International Conference on Development and Learning and Epigenetic Robotics (ICDL), Osaka, Japan.","DOI":"10.1109\/DevLrn.2013.6652569"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Reardon, C.M., Haring, K.S., Gregory, J.M., and Rogers, J.G. (2021, January 25\u201327). Evaluating Human Understanding of a Mixed Reality Interface for Autonomous Robot-Based Change Detection. Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics, SSRR 2021, New York, NY, USA.","DOI":"10.1109\/SSRR53300.2021.9597854"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Szafir, D. (2019). Mediating Human-Robot Interactions with Virtual, Augmented, and Mixed Reality. Virtual, Augmented and Mixed Reality. Applications and Case Studies, Springer. HCII 2019. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-030-21565-1_9"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Al, G.A., Estrela, P., and Martinez-Hernandez, U. (2020, January 14\u201316). Towards an intuitive human-robot interaction based on hand gesture recognition and proximity sensors. Proceedings of the 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Karlsruhe, Germany.","DOI":"10.1109\/MFI49285.2020.9235264"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"207635","DOI":"10.1109\/ACCESS.2020.3037724","article-title":"Robots and Wizards: An Investigation Into Natural Human\u2013Robot Interaction","volume":"8","author":"Strazdas","year":"2020","journal-title":"IEEE Access"},{"key":"ref_7","unstructured":"International Organization for Standardization (2011). ISO 10218-1:2011 Robots and Robotic Devices\u2014Safety Requirements for Industrial Robots, ISO."},{"key":"ref_8","unstructured":"International Organization for Standardization (2016). ISO\/TS 15066:2016 Robots and Robotic Devices\u2014Collaborative Robots, ISO."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Pasinetti, S., Nuzzi, C., Lancini, M., Sansoni, G., Docchio, F., and Fornaser, A. (2018, January 16\u201318). Development and Characterization of a Safety System for Robotic Cells Based on Multiple Time of Flight (TOF) Cameras and Point Cloud Analysis. Proceedings of the 2018 Workshop on Metrology for Industry 4.0 and IoT, Brescia, Italy.","DOI":"10.1109\/METROI4.2018.8439037"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Strazdas, D., Hintz, J., and Al-Hamadi, A. (2021). Robo-HUD: Interaction Concept for Contactless Operation of Industrial Cobotic Systems. Appl. Sci., 11.","DOI":"10.3390\/app11125366"},{"key":"ref_11","unstructured":"Rogalla, O., Ehrenmann, M., Zollner, R., Becher, R., and Dillmann, R. (2002, January 27). Using gesture and speech control for commanding a robot assistant. Proceedings of the 11th IEEE International Workshop on Robot and Human Interactive Communication, Berlin, Germany."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1007\/s12369-017-0408-9","article-title":"Foundations of Visual Linear Human\u2013Robot Interaction via Pointing Gesture Navigation","volume":"9","author":"Dekan","year":"2017","journal-title":"Int. J. Soc. Robot."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1016\/j.jvcir.2013.03.017","article-title":"Gesture-based interaction with voice feedback for a tour-guide robot","volume":"25","author":"Iglesias","year":"2014","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/s12008-013-0191-2","article-title":"A novel augmented reality-based interface for robot path planning","volume":"8","author":"Fang","year":"2014","journal-title":"Int. J. Interact. Des. Manuf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101820","DOI":"10.1016\/j.rcim.2019.101820","article-title":"Augmented reality-assisted robot programming system for industrial applications","volume":"61","author":"Ong","year":"2020","journal-title":"Robot. -Comput.-Integr. Manuf."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Gadre, S.Y., Rosen, E., Chien, G., Phillips, E., Tellex, S., and Konidaris, G. (2019, January 20\u201324). End-user robot programming using mixed reality. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793988"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1429","DOI":"10.1016\/j.procir.2019.04.328","article-title":"Enabling human robot interaction in flexible robotic assembly lines: An augmented reality based software suite","volume":"81","author":"Kousi","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Stetco, C., Muhlbacher-Karrer, S., Lucchi, M., Weyrer, M., Faller, L.M., and Zangl, H. (2020, January 25\u201328). Gesture-based contactless control of mobile manipulators using capacitive sensing. Proceedings of the 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Dubrovnik, Croatia.","DOI":"10.1109\/I2MTC43012.2020.9128751"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1109\/LRA.2016.2632973","article-title":"Contactless Control of a Kinematically Redundant Serial Manipulator Using Tomographic Sensors","volume":"2","author":"Schett","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.rcim.2019.101846","article-title":"Human-robot coexistence and interaction in open industrial cells","volume":"61","author":"Magrini","year":"2020","journal-title":"Robot. -Comput.-Integr. Manuf."},{"key":"ref_21","unstructured":"Irfan, B., Lyubova, N., Garcia Ortiz, M., and Belpaeme, T. (2018, January 5\u20138). Multi-modal open-set person identification in hri. Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction Social, Chicago, IL, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Strazdas, D., Hintz, J., Khalifa, A., and Al-Hamadi, A. (2021, January 8\u201310). Robot System Assistant (RoSA): Concept for an intuitive multi-modal and multi-device interaction system. Proceedings of the 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS), Magdeburg, Germany.","DOI":"10.1109\/ICHMS53169.2021.9582663"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019, January 15\u201320). Arcface: Additive angular margin loss for deep face recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00482"},{"key":"ref_24","unstructured":"Luan, P., Huynh, V., and Tuan Anh, T. (2021, January 10\u201315). Facial Expression Recognition using Residual Masking Network. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Deng, J., Guo, J., Ververas, E., Kotsia, I., and Zafeiriou, S. (2020, January 13\u201319). Retinaface: Single-shot multi-level face localisation in the wild. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00525"},{"key":"ref_26","unstructured":"Picovoice (2021, December 22). Porcupine. Available online: https:\/\/github.com\/Picovoice\/porcupine."},{"key":"ref_27","unstructured":"Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., and Coates, A. (2014). Deepspeech: Scaling up end-to-end speech recognition. arXiv."},{"key":"ref_28","unstructured":"Google (2021, December 22). WebRTC. Available online: https:\/\/webrtc.org."},{"key":"ref_29","unstructured":"RASA (2021, December 22). RASA. Available online: https:\/\/rasa.com\/open-source\/."},{"key":"ref_30","unstructured":"Hempel, T. (2021, December 17). RoSA: Cube Detector. Available online: https:\/\/github.com\/thohemp\/cube_detector\/tree\/v1.0.0."},{"key":"ref_31","unstructured":"Strazdas, D., Khalifa, A., and Hempel, T. (2021, December 15). DoStraTech\/rosa_msgs: Initial Release. Available online: https:\/\/github.com\/DoStraTech\/rosa_msgs\/tree\/v0.1-alpha."},{"key":"ref_32","unstructured":"Bramel, J. (2021, December 22). Alphabet Play Blocks. Available online: https:\/\/www.thingiverse.com\/thing:2368270."},{"key":"ref_33","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, S., Luo, P., Loy, C.C., and Tang, X. (2016, January 27\u201330). Wider face: A face detection benchmark. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.596"},{"key":"ref_35","unstructured":"Guo, X., Li, S., Yu, J., Zhang, J., Ma, J., Ma, L., Liu, W., and Ling, H. (2019). PFLD: A practical facial landmark detector. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Chen, S., Liu, Y., Gao, X., and Han, Z. (2018). Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. Chinese Conference on Biometric Recognition, Springer.","DOI":"10.1007\/978-3-319-97909-0_46"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1007\/s12369-017-0414-y","article-title":"Automatically Classifying User Engagement for Dynamic Multi-party Human\u2013Robot Interaction","volume":"9","author":"Foster","year":"2017","journal-title":"Int. J. Soc. Robot."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.robot.2015.01.004","article-title":"Starting engagement detection towards a companion robot using multimodal features","volume":"75","author":"Vaufreydaz","year":"2016","journal-title":"Robot. Auton. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1007\/s12369-015-0298-7","article-title":"Evaluating the engagement with social robots","volume":"7","author":"Anzalone","year":"2015","journal-title":"Int. J. Soc. Robot."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Li, L., Xu, Q., and Tan, Y.K. (2012). Attention-based addressee selection for service and social robots to interact with multiple persons. Proceedings of the Workshop at SIGGRAPH Asia, Association for Computing Machinery.","DOI":"10.1145\/2425296.2425319"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Richter, V., Carlmeyer, B., Lier, F., Meyer zu Borgsen, S., Schlangen, D., Kummert, F., Wachsmuth, S., and Wrede, B. (2016, January 4\u20137). Are you talking to me? Improving the robustness of dialogue systems in a multi party HRI scenario by incorporating gaze direction and lip movement of attendees. Proceedings of the Fourth International Conference on Human Agent Interaction, Biopolis, Singapore.","DOI":"10.1145\/2974804.2974823"},{"key":"ref_42","unstructured":"Ellgring, J.H. (1986). Nonverbale Kommunikation, Universit\u00e4t W\u00fcrzburg."},{"key":"ref_43","first-page":"607","article-title":"Head pose estimation in computer vision: A survey","volume":"31","author":"Trivedi","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Albiero, V., Chen, X., Yin, X., Pang, G., and Hassner, T. (2021, January 20\u201325). img2pose: Face alignment and detection via 6dof, face pose estimation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00753"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., and Torralba, A. (2019, January 27\u201328). Gaze360: Physically unconstrained gaze estimation in the wild. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00701"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Williams, T., Hirshfield, L., Tran, N., Grant, T., and Woodward, N. (2020). Using augmented reality to better study human-robot interaction. International Conference on Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-030-49695-1_43"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/MSP.2019.2918706","article-title":"Speech Processing for Digital Home Assistants: Combining Signal Processing with Deep-Learning Techniques","volume":"36","author":"Watanabe","year":"2019","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_48","first-page":"1","article-title":"Voice activity detection. fundamentals and speech recognition system robustness","volume":"6","author":"Ramirez","year":"2007","journal-title":"Robust Speech Recognit. Underst."},{"key":"ref_49","unstructured":"Mozzilla (2021, December 22). Deepspeech. Available online: https:\/\/github.com\/mozilla\/DeepSpeech."},{"key":"ref_50","unstructured":"Agarwal, A., and Zesch, T. (2019). German End-to-end Speech Recognition based on DeepSpeech. Preliminary Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, German Society for Computational Linguistics & Language Technology."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1543","DOI":"10.1080\/00140138508963286","article-title":"User feedback requirements with automatic speech recognition","volume":"28","author":"Schurick","year":"1985","journal-title":"Ergonomics"},{"key":"ref_52","unstructured":"Bocklisch, T., Faulkner, J., Pawlowski, N., and Nichol, A. (2017). Rasa: Open source language understanding and dialogue management. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Braun, D., Hernandez Mendez, A., Matthes, F., and Langen, M. (2017). Evaluating Natural Language Understanding Services for Conversational Question Answering Systems. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Association for Computational Linguistics.","DOI":"10.18653\/v1\/W17-5522"},{"key":"ref_54","unstructured":"Microsoft (2021, December 22). Speechsynthesis. Available online: https:\/\/docs.microsoft.com\/de-de\/uwp\/api\/windows.media.speechsynthesis."},{"key":"ref_55","first-page":"4","article-title":"SUS: A quick and dirty usability scale","volume":"189","author":"Brooke","year":"1995","journal-title":"Usability Eval. Ind."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1016\/j.intcom.2010.04.004","article-title":"The usability metric for user experience","volume":"22","author":"Finstad","year":"2010","journal-title":"Interact. Comput."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1207\/S15327590IJHC143&4_11","article-title":"Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies","volume":"14","author":"Lewis","year":"2002","journal-title":"Int. J. -Hum.-Comput. Interact."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/122672.122692","article-title":"Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ","volume":"23","author":"Lewis","year":"1991","journal-title":"SIGCHI Bull."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"9411","DOI":"10.1007\/s11042-020-10073-7","article-title":"Automatic speech recognition: A survey","volume":"80","author":"Malik","year":"2021","journal-title":"Multimed. Tools Appl."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/923\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:07:24Z","timestamp":1760134044000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/923"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,25]]},"references-count":59,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22030923"],"URL":"https:\/\/doi.org\/10.3390\/s22030923","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,25]]}}}