{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T17:10:17Z","timestamp":1779383417408,"version":"3.53.1"},"reference-count":51,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2017,7,27]],"date-time":"2017-07-27T00:00:00Z","timestamp":1501113600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Multi-robot missions are a challenge for operators in terms of workload and situational awareness. These operators have to receive data from the robots, extract information, understand the situation properly, make decisions, generate the adequate commands, and send them to the robots. The consequences of excessive workload and lack of awareness can vary from inefficiencies to accidents. This work focuses on the study of future operator interfaces of multi-robot systems, taking into account relevant issues such as multimodal interactions, immersive devices, predictive capabilities and adaptive displays. Specifically, four interfaces have been designed and developed: a conventional, a predictive conventional, a virtual reality and a predictive virtual reality interface. The four interfaces have been validated by the performance of twenty-four operators that supervised eight multi-robot missions of fire surveillance and extinguishing. The results of the workload and situational awareness tests show that virtual reality improves the situational awareness without increasing the workload of operators, whereas the effects of predictive components are not significant and depend on their implementation.<\/jats:p>","DOI":"10.3390\/s17081720","type":"journal-article","created":{"date-parts":[[2017,7,27]],"date-time":"2017-07-27T11:28:56Z","timestamp":1501154936000},"page":"1720","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["Multi-Robot Interfaces and Operator Situational Awareness: Study of the Impact of Immersion and Prediction"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8863-4419","authenticated-orcid":false,"given":"Juan","family":"Rold\u00e1n","sequence":"first","affiliation":[{"name":"Centre for Automation and Robotics (UPM-CSIC), Universidad Polit\u00e9cnica de Madrid, Jos\u00e9 Guti\u00e9rrez Abascal, 2, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Pe\u00f1a-Tapia","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (UPM-CSIC), Universidad Polit\u00e9cnica de Madrid, Jos\u00e9 Guti\u00e9rrez Abascal, 2, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andr\u00e9s","family":"Mart\u00edn-Barrio","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (UPM-CSIC), Universidad Polit\u00e9cnica de Madrid, Jos\u00e9 Guti\u00e9rrez Abascal, 2, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miguel","family":"Olivares-M\u00e9ndez","sequence":"additional","affiliation":[{"name":"Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Richard Coudenhove-Kalergi, 6, L-1359 Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4893-2571","authenticated-orcid":false,"given":"Jaime","family":"Del Cerro","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (UPM-CSIC), Universidad Polit\u00e9cnica de Madrid, Jos\u00e9 Guti\u00e9rrez Abascal, 2, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1691-3907","authenticated-orcid":false,"given":"Antonio","family":"Barrientos","sequence":"additional","affiliation":[{"name":"Centre for Automation and Robotics (UPM-CSIC), Universidad Polit\u00e9cnica de Madrid, Jos\u00e9 Guti\u00e9rrez Abascal, 2, 28006 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,7,27]]},"reference":[{"key":"ref_1","unstructured":"Cummings, M.L., Bruni, S., Mercier, S., and Mitchell, P.J. (2007). Automation Architecture for Single Operator, Multiple UAV Command and Control, Massachusetts Institute of Technology."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1177\/154193120504900347","article-title":"Up from the rubble: Lessons learned about HRI from search and rescue","volume":"Volume 49","author":"Murphy","year":"2005","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Lysaght, R.J., Hill, S.G., Dick, A.O., Plamondon, B.D., and Linton, P.M. (1989). Operator Workload: Comprehensive Review and Evaluation of Operator Workload Methodologies (No. TR-2075-3), Analytics Inc.","DOI":"10.21236\/ADA212879"},{"key":"ref_4","unstructured":"Moray, N. (2013). Mental Workload: Its Theory and Measurement, Springer."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume":"52","author":"Hart","year":"1988","journal-title":"Adv. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1109\/TSMCA.2010.2046731","article-title":"Modeling workload impact in multiple unmanned vehicle supervisory control","volume":"40","author":"Donmez","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Humans"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"NASA-task load index (NASA-TLX); 20 years later","volume":"Volume 50","author":"Hart","year":"2006","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1177\/154193128803200221","article-title":"Design and evaluation for situation awareness enhancement","volume":"Volume 32","author":"Endsley","year":"1988","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"ref_9","unstructured":"Endsley, M.R. (1988, January 23\u201327). Situation awareness global assessment technique (SAGAT). Proceedings of the IEEE 1988 National Aerospace and Electronics Conference, Dayton, OH, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1007\/s10846-010-9507-7","article-title":"Optical brain imaging to enhance UAV operator training, evaluation, and interface development","volume":"61","author":"Menda","year":"2011","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Haas, E.C., Pillalamarri, K., Stachowiak, C.C., and Fields, M. (August, January 31). Multimodal controls for soldier\/swarm interaction. Proceedings of the 2011 RO-MAN, Atlanta, GA, USA.","DOI":"10.1109\/ROMAN.2011.6005227"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kolling, A., Nunnally, S., and Lewis, M. (2012, January 5\u20138). Towards human control of robot swarms. Proceedings of the Seventh Annual ACM\/IEEE International Conference on Human-Robot Interaction, Boston, MA, USA.","DOI":"10.1145\/2157689.2157704"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1093\/iwc\/iws011","article-title":"Boredom and distraction in multiple unmanned vehicle supervisory control","volume":"25","author":"Cummings","year":"2013","journal-title":"Interact. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Frische, F., and L\u00fcdtke, A. (2013, January 25\u201328). SA\u2013tracer: A tool for assessment of UAV swarm operator SA during mission execution. Proceedings of the 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA), San Diego, CA, USA.","DOI":"10.1109\/CogSIMA.2013.6523849"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1260\/1756-8293.6.4.211","article-title":"An ecological approach to the supervisory control of UAV swarms","volume":"6","author":"Fuchs","year":"2014","journal-title":"Int. J. Micro Air Veh."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ruiz, J.J., Viguria, A., Martinez-de-Dios, J.R., and Ollero, A. (2015, January 9\u201312). Immersive displays for building spatial knowledge in multi-UAV operations. Proceedings of the IEEE 2015 International Conference on Unmanned Aircraft Systems (ICUAS), Denver, CO, USA.","DOI":"10.1109\/ICUAS.2015.7152395"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.robot.2016.03.006","article-title":"Visual feedback with multiple cameras in a UAVs Human\u2013Swarm Interface","volume":"80","author":"Recchiuto","year":"2016","journal-title":"Robot. Auton. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ruano, S., Cuevas, C., Gallego, G., and Garc\u00eda, N. (2017). Augmented Reality Tool for the Situational Awareness Improvement of UAV Operators. Sensors, 17.","DOI":"10.3390\/s17020297"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mortimer, M., Horan, B., and Seyedmahmoudian, M. (2017). Building a Relationship between Robot Characteristics and Teleoperation User Interfaces. Sensors, 17.","DOI":"10.3390\/s17030587"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1109\/TSMCC.2007.905819","article-title":"Human performance issues and user interface design for teleoperated robots","volume":"37","author":"Chen","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_21","first-page":"61","article-title":"Spatial Audio Display Concepts Supporting Situation Awareness for Operators of Unmanned Aerial Vehicles","volume":"Volumes I and II","author":"Simpson","year":"2013","journal-title":"Human Performance, Situation Awareness, and Automation: Current Research and Trends HPSAA II"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Scheggi, S., Aggravi, M., Morbidi, F., and Prattichizzo, D. (June, January 31). Cooperative human-robot haptic navigation. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6907245"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1162\/105474602760204282","article-title":"The effects of operator spatial perception and sensory feedback on human\u2013robot teleoperation performance","volume":"11","author":"Lathan","year":"2002","journal-title":"Presence Teleoper. Virtual Environ."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Monajjemi, V.M., Pourmehr, S., Sadat, S.A., Zhan, F., Wawerla, J., Mori, G., and Vaughan, R. (2014, January 3\u20136). Integrating multi-modal interfaces to command UAVs. Proceedings of the Proceedings of the 2014 ACM\/IEEE International Conference on Human\u2013Robot Interaction, Bielefeld, Germany.","DOI":"10.1145\/2559636.2559646"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kavitha, S., Veena, S., and Kumaraswamy, R. (2015, January 21\u201322). Development of automatic speech recognition system for voice activated Ground Control system. Proceedings of the 2015 International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15), Bangalore, India.","DOI":"10.1109\/ITACT.2015.7492684"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Mantec\u00f3n del Valle, T., Ad\u00e1n, B., Jaureguizar N\u00fa\u00f1ez, F., and Garc\u00eda Santos, N. (2014, January 5\u20139). New generation of human machine interfaces for controlling UAV through depth based gesture recognition. Proceedings of the SPIE Defense, Security and Sensing Conference 2014, Baltimore, MD, USA.","DOI":"10.1117\/12.2053244"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Nagi, J., Giusti, A., Di Caro, G.A., and Gambardella, L.M. (2014, January 3\u20136). Human control of UAVs using face pose estimates and hand gestures. Proceedings of the 2014 ACM\/IEEE International Conference on Human\u2013Robot Interaction, Bielefeld, Germany.","DOI":"10.1145\/2559636.2559833"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1109\/TSMCC.2010.2056682","article-title":"Supervisory control of multiple robots: Human-performance issues and user-interface design","volume":"41","author":"Chen","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_29","unstructured":"Drury, J.L., Richer, J., Rackliffe, N., and Goodrich, M.A. (2006). Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface Approaches, Mitre Corp."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, N., Cartwright, S., Shekhar Nittala, A., Sharlin, E., and Costa Sousa, M. (2015, January 21\u201324). Flying Frustum: A Spatial Interface for Enhancing Human\u2013UAV Awareness. Proceedings of the 3rd International Conference on Human\u2013Agent Interaction, Kyungpook, Korea.","DOI":"10.1145\/2814940.2814956"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hansberger, J.T. (2015). Development of the Next Generation of Adaptive Interfaces (No. ARL-TR-7251), Aberdeen Proving Ground Md Human Research and Engineering Directorate, Army Research Laboratory.","DOI":"10.21236\/ADA614163"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1109\/TSMCC.2010.2052041","article-title":"Optimizing operator-agent interaction in intelligent adaptive interface design: A conceptual framework","volume":"41","author":"Hou","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Larochelle, B., Kruijff, G.J.M., Smets, N., Mioch, T., and Groenewegen, P. (August, January 31). Establishing human situation awareness using a multi-modal operator control unit in an urban search & rescue human\u2013robot team. Proceedigns of the 2011 RO-MAN, Atlanta, GA, USA.","DOI":"10.1109\/ROMAN.2011.6005237"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.ergon.2008.08.008","article-title":"Evaluation of human-agent user interfaces in multi-agent systems","volume":"39","author":"Nam","year":"2009","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.apergo.2016.05.011","article-title":"A meta-analysis of human-system interfaces in unmanned aerial vehicle (UAV) swarm management","volume":"58","author":"Hocraffer","year":"2017","journal-title":"Appl. Ergon."},{"key":"ref_36","unstructured":"Olson, W.A., and Wuennenberg, M.G. (2001, January 14\u201318). Autonomy based human-vehicle interface standards for remotely operated aircraft. Proceedings of the 20th Digital Avionics Systems Conference (DASC) (Cat. No.01CH37219), Daytona Beach, FL, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Scholtz, J., Young, J., Drury, J.L., and Yanco, H.A. (May, January 26). Evaluation of human-robot interaction awareness in search and rescue. Proceedings of the 2004 IEEE International Conference on Robotics and Automation (ICRA\u201904), New Orleans, LA, USA.","DOI":"10.1109\/ROBOT.2004.1307409"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Adams, B., and Suykens, F. (2013, January 12\u201314). Astute: Increased Situational Awareness through proactive decision support and adaptive map-centric user interfaces. Proceedings of the 2013 European Intelligence and Security Informatics Conference, Uppsala, Sweden.","DOI":"10.1109\/EISIC.2013.74"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Rold\u00e1n, J.J., Olivares, M., Miguel, A., del Cerro, J., and Barrientos, A. (2017). Analyzing and Improving Multi-Robot Missions by using Process Mining. Auton. Robots, under review.","DOI":"10.1007\/s10514-017-9686-1"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Krajnik, T., Von\u00e1sek, V., Fiser, D., and Faigl, J. (2011, January 15\u201317). AR-drone as a platform for robotic research and education. Proceedings of the International Conference on Research and Education in Robotics, Prague, Czech Republic.","DOI":"10.1007\/978-3-642-21975-7_16"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Bischoff, R., Huggenberger, U., and Prassler, E. (2011, January 9\u201313). KUKA youBot\u2014A mobile manipulator for research and education. Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5980575"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Dentler, J., Kannan, S.Mendez, and Voos, H. (2016, January 19\u201322). A real-time model predictive position control with collision avoidance for commercial low-cost quadrotors. Proceedings of the 2016 IEEE Conference on Control Applications (CCA), Buenos Aires, Argentina.","DOI":"10.1109\/CCA.2016.7587882"},{"key":"ref_43","first-page":"5","article-title":"ROS: An Open-Source Robot Operating System","volume":"3","author":"Quigley","year":"2009","journal-title":"ICRA Workshop Open Sour. Softw."},{"key":"ref_44","unstructured":"Sanz Mu\u00f1oz, D. (2015). Cognitive Risk Perception System for Obstacle Avoidance in Outdoor mUAV Missions. [Ph.D. Thesis, Technical University of Madrid]."},{"key":"ref_45","unstructured":"Rold\u00e1n, J.J., del Cerro, J., and Barrientos, A. (2017). Using Process Mining to Model Multi-UAV Missions through the Experience. IEEE Intell. Syst."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Rold\u00e1n, J.J., Garcia-Aunon, P., del Cerro, J., and Barrientos, A. (2016, January 9\u201312). Determining mission evolution through UAV telemetry by using decision trees. Proceedings of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary.","DOI":"10.1109\/SMC.2016.7844239"},{"key":"ref_47","unstructured":"Ripton, J., and Prasuethsut, L. (2017, July 25). The VR Race: What You Need to Know about Oculus Rift, HTC Vive and More. Available online: http:\/\/www.techradar.com\/news\/world-of-tech\/future-tech\/the-vr-race-who-s-closest-to-making-vr-a-reality-1266538."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Niehorster, D.C., Li, L., and Lappe, M. (2017). The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research. i-Percept. SAGE J., 8.","DOI":"10.1177\/2041669517708205"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Seo, M.W., Choi, S.W., Lee, S.L., Oh, E.Y., Baek, J.S., and Kang, S.J. (2017). Photosensor-Based Latency Measurement System for Head-Mounted Displays. Sensors, 17.","DOI":"10.3390\/s17051112"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.anl.2007.01.002","article-title":"Autonomic responses during motion sickness induced by virtual reality","volume":"34","author":"Ohyama","year":"2007","journal-title":"Auris Nasus Larynx"},{"key":"ref_51","unstructured":"Fagerholt, E., and Lorentzon, M. (2009). Beyond the HUD-User Interfaces for Increased Player Immersion in FPS Games. [Master\u2019s Thesis, Department of Computer Science and Engineering, Chalmers University of Technology]."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/8\/1720\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:44:09Z","timestamp":1760208249000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/8\/1720"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,27]]},"references-count":51,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2017,8]]}},"alternative-id":["s17081720"],"URL":"https:\/\/doi.org\/10.3390\/s17081720","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,7,27]]}}}