{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:45:25Z","timestamp":1776105925658,"version":"3.50.1"},"reference-count":37,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,1,7]],"date-time":"2021-01-07T00:00:00Z","timestamp":1609977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100005014","name":"Northrop Grumman","doi-asserted-by":"publisher","award":["RE-03163-0200317164"],"award-info":[{"award-number":["RE-03163-0200317164"]}],"id":[{"id":"10.13039\/100005014","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007034","name":"Thales Group","doi-asserted-by":"publisher","award":["RE-02544-0200315666"],"award-info":[{"award-number":["RE-02544-0200315666"]}],"id":[{"id":"10.13039\/501100007034","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Advances in unmanned aircraft systems (UAS) have paved the way for progressively higher levels of intelligence and autonomy, supporting new modes of operation, such as the one-to-many (OTM) concept, where a single human operator is responsible for monitoring and coordinating the tasks of multiple unmanned aerial vehicles (UAVs). This paper presents the development and evaluation of cognitive human-machine interfaces and interactions (CHMI2) supporting adaptive automation in OTM applications. A CHMI2 system comprises a network of neurophysiological sensors and machine-learning based models for inferring user cognitive states, as well as the adaptation engine containing a set of transition logics for control\/display functions and discrete autonomy levels. Models of the user\u2019s cognitive states are trained on past performance and neurophysiological data during an offline calibration phase, and subsequently used in the online adaptation phase for real-time inference of these cognitive states. To investigate adaptive automation in OTM applications, a scenario involving bushfire detection was developed where a single human operator is responsible for tasking multiple UAV platforms to search for and localize bushfires over a wide area. We present the architecture and design of the UAS simulation environment that was developed, together with various human-machine interface (HMI) formats and functions, to evaluate the CHMI2 system\u2019s feasibility through human-in-the-loop (HITL) experiments. The CHMI2 module was subsequently integrated into the simulation environment, providing the sensing, inference, and adaptation capabilities needed to realise adaptive automation. HITL experiments were performed to verify the CHMI2 module\u2019s functionalities in the offline calibration and online adaptation phases. In particular, results from the online adaptation phase showed that the system was able to support real-time inference and human-machine interface and interaction (HMI2) adaptation. However, the accuracy of the inferred workload was variable across the different participants (with a root mean squared error (RMSE) ranging from 0.2 to 0.6), partly due to the reduced number of neurophysiological features available as real-time inputs and also due to limited training stages in the offline calibration phase. To improve the performance of the system, future work will investigate the use of alternative machine learning techniques, additional neurophysiological input features, and a more extensive training stage.<\/jats:p>","DOI":"10.3390\/robotics10010012","type":"journal-article","created":{"date-parts":[[2021,1,10]],"date-time":"2021-01-10T23:03:42Z","timestamp":1610319822000},"page":"12","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles"],"prefix":"10.3390","volume":"10","author":[{"given":"Yixiang","family":"Lim","sequence":"first","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"},{"name":"Saab-NTU Joint Lab, Nanyang Technological University, Singapore 637460, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1460-4764","authenticated-orcid":false,"given":"Nichakorn","family":"Pongsakornsathien","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4995-4166","authenticated-orcid":false,"given":"Alessandro","family":"Gardi","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3399-2291","authenticated-orcid":false,"given":"Roberto","family":"Sabatini","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2110-7021","authenticated-orcid":false,"given":"Trevor","family":"Kistan","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"},{"name":"THALES Australia, WTC North Wharf, Melbourne, VIC 3000, Australia"}]},{"given":"Neta","family":"Ezer","sequence":"additional","affiliation":[{"name":"Northrop Grumman Corporation, 1550 W. Nursery Rd, Linthicum Heights, MD 21090, USA"}]},{"given":"Daniel J.","family":"Bursch","sequence":"additional","affiliation":[{"name":"Northrop Grumman Corporation, 1550 W. Nursery Rd, Linthicum Heights, MD 21090, USA"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1142\/S2301385020500090","article-title":"Multiple UAV Systems: A Survey","volume":"8","author":"Skorobogatov","year":"2020","journal-title":"Unmanned Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/s10846-012-9706-5","article-title":"A Multi-UAS Cooperative Mission Over Non-Segregated Civil Areas","volume":"70","author":"Boccalatte","year":"2012","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"488","DOI":"10.1177\/0018720818799468","article-title":"Vigilance and Automation Dependence in Operation of Multiple Unmanned Aerial Systems (UAS): A Simulation Study","volume":"61","author":"Wohleber","year":"2018","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Eaton, C.M., Chong, E.K.P., and Maciejewski, A.A. (2016). Multiple-Scenario Unmanned Aerial System Control: A Systems Engineering Approach and Review of Existing Control Methods. Aerospace, 3.","DOI":"10.3390\/aerospace3010001"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4852","DOI":"10.1080\/01431161.2018.1490504","article-title":"The impact of small unmanned airborne platforms on passive optical remote sensing: A conceptual perspective","volume":"39","author":"Lippitt","year":"2018","journal-title":"Int. J. Remote. Sens."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Miller, C.A., Mueller, J., Geib, C., Lavergne, D., Walker, P., and Hamell, J. (2019, January 9\u201311). User Interaction Approaches for Managing Multiple UAS in the National Airspace. Proceedings of the 2019 Integrated Communications, Navigation and Surveillance Conference (ICNS), Herndon, VA, USA.","DOI":"10.1109\/ICNSURV.2019.8735205"},{"key":"ref_7","unstructured":"Fern, L., and Shively, J. (2009, January 10\u201313). A Comparison of Varying Levels of Automation on the Supervisory Control of Multiple UASs. Proceedings of the AUVSI\u2019s Unmanned Systems North America, Washington, DC, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"640","DOI":"10.3389\/fpsyg.2019.00640","article-title":"Metacognition, Hardiness, and Grit as Resilience Factors in Unmanned Aerial Systems (UAS) Operations: A Simulation Study","volume":"10","author":"Matthews","year":"2019","journal-title":"Front. Psychol."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Feitshans, G.L., Rowe, A.J., Davis, J.E., Holland, M., and Berger, L. (2008, January 18\u201321). Vigilant spirit control station (VSCS)\u2014\u2018The face of COUNTER\u2019. Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, Honolulu, HI, USA.","DOI":"10.2514\/6.2008-6309"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/978-3-319-41959-6_10","article-title":"Operator-Autonomy Teaming Interfaces to Support Multi-Unmanned Vehicle Missions","volume":"Volume 499","author":"Calhoun","year":"2016","journal-title":"Advances in Intelligent Systems and Computing"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s10846-012-9759-5","article-title":"A Ground Control Station for a Multi-UAV Surveillance System","volume":"69","author":"Perez","year":"2012","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Valenti, M., Bethke, B., Fiore, G., How, J.P., and Feron, E. (2006, January 21\u201324). Indoor Multi-Vehicle Flight Testbed for Fault Detection, Isolation, and Recovery. Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, Keystone, CO, USA.","DOI":"10.2514\/6.2006-6200"},{"key":"ref_13","unstructured":"Patzek, M., Zimmer, D., Feitshans, G., Draper, M., Huges, T., and Flach, J. (2009, January 27\u201330). Multi-UAV Supervisory Control Interface Technology. Proceedings of the International Symposium on Aviation Psychology, Dayton, OH, USA."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Fern, L., Rorie, R.C., Pack, J., Shively, R.J., and Draper, M. (2015, January 22\u201326). An evaluation of Detect and Avoid (DAA) displays for unmanned aircraft systems: The effect of information level and display location on pilot performance. Proceedings of the 15th Annual AIAA Aviation, Technology, Integration and Operations Conference, Dallas, TX, USA.","DOI":"10.2514\/6.2015-3327"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1177\/1541931214581005","article-title":"UAS Sense and Avoid System Interface Design and Evaluation","volume":"Volume 58","author":"Calhoun","year":"2014","journal-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting"},{"key":"ref_16","unstructured":"Ramchurn, S.D., Fischer, J.E., Ikuno, Y., Wu, F., Flann, J., and Waldock, A. (2015, January 25\u201331). A Study of Human-Agent Collaboration for Multi-UAV Task Allocation in Dynamic Environments. Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), Buenos Aires, Argentina."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Berka, C., Levendowski, D.J., Ramsey, C.K., Davis, G., Lumicao, M.N., Stanney, K.M., Reeves, L.M., Regli, S.H., Tremoulet, P.D., and Stibler, K. (2005). Evaluation of an EEG workload model in an Aegis simulation environment. Proceedings SPIE 5797, Biomonitoring for Physiological and Cognitive Performance during Military Operations, International Society for Optics and Photonics.","DOI":"10.1117\/12.598555"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1177\/0018720810368674","article-title":"The Role of Human-Automation Consensus in Multiple Unmanned Vehicle Scheduling","volume":"52","author":"Cummings","year":"2010","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1177\/1555343411410160","article-title":"Adaptive aiding of human-robot teaming effects of imperfect automation on performance, trust, and workload","volume":"5","author":"Parasuraman","year":"2011","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-017-00633-7","article-title":"EEG-Based Cognitive Control Behaviour Assessment: An Ecological study with Professional Air Traffic Controllers","volume":"7","author":"Borghini","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, Y., Cong, W., Dong, B., Wu, F., and Hu, M. (2015, January 23\u201326). Statistical Analysis of Air Traffic Controllers\u2019 Eye Movements. Proceedings of the 11th USA\/Europe Air Traffic Management Research and Development Seminar, ATM 2015, Lisbon, Portugal.","DOI":"10.2514\/atcq.23.1.1"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Toppi, J., Borghini, G., Petti, M., He, E.J., De Giusti, V., He, B., Astolfi, L., and Babiloni, F. (2016). Investigating Cooperative Behavior in Ecological Settings: An EEG Hyperscanning Study. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0154236"},{"key":"ref_23","unstructured":"Harris, J., Wiggins, M., Morrison, B., and Morrison, N. (2013, January 25\u201329). Differentiating Cognitive Complexity and Cognitive Load in High and Low Demand Flight Simulation Tasks. Proceedings of the Australian Computer-Human Interaction Conference, Adelaide, Australia."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Harrison, J., Izzetoglu, K., Ayaz, H., Willems, B., Hah, S., Woo, H., Shewokis, P.A., Bunce, S.C., and Onaral, B. (2013, January 21\u201326). Human performance assessment study in aviation using functional near infrared spectroscopy. Proceedings of the International Conference on Augmented Cognition, Las Vegas, NV, USA.","DOI":"10.1007\/978-3-642-39454-6_46"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s10111-019-00548-5","article-title":"Evaluating the reduced flight deck crew concept using cognitive work analysis and social network analysis: Comparing normal and data-link outage scenarios","volume":"22","author":"Schmid","year":"2019","journal-title":"Cogn. Technol. Work."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MAES.2017.160175","article-title":"Commercial airline single-pilot operations: System design and pathways to certification","volume":"32","author":"Lim","year":"2017","journal-title":"IEEE Aerosp. Electron. Syst. Mag."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lim, Y., Gardi, A., Ezer, N., Kistan, T., and Sabatini, R. (2018, January 20\u201322). Eye-Tracking Sensors for Adaptive Aerospace Human-Machine Interfaces and Interactions. Proceedings of the Metrology for Aerospace (MetroAeroSpace), Rome, Italy.","DOI":"10.1109\/MetroAeroSpace.2018.8453509"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Pongsakornsathien, N., Lim, Y., Gardi, A., Hilton, S., Planke, L.J., Sabatini, R., Kistan, T., and Ezer, N. (2019). Sensor Networks for Aerospace Human-Machine Systems. Sensors, 19.","DOI":"10.3390\/s19163465"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lim, Y., Gardi, A., Ramasamy, S., Vince, J., Pongracic, H., Kistan, T., and Sabatini, R. (2017, January 17\u201321). A Novel Simulation Environment for Cognitive Human Factors Engineering Research. Proceedings of the 36th Digital Avionics Systems Conference, St. Petersburg, FL, USA.","DOI":"10.1109\/DASC.2017.8102126"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1007\/s10846-017-0648-9","article-title":"Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft","volume":"91","author":"Lim","year":"2017","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.measurement.2019.03.032","article-title":"Experimental characterisation of eye-tracking sensors for adaptive human-machine systems","volume":"140","author":"Lim","year":"2019","journal-title":"Measurement"},{"key":"ref_32","unstructured":"Lim, Y., Liu, J., Ramasamy, S., and Sabatini, R. (2016, January 23\u201325). Cognitive Remote Pilot-Aircraft Interface for UAS Operations. Proceedings of the 12th International Conference on Intelligent Unmanned Systems, Xi\u2019an, China."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Planke, L.J., Lim, Y., Gardi, A., Sabatini, R., Kistan, T., and Ezer, N. (2020). A Cyber-Physical-Human System for One-to-Many UAS Operations: Cognitive Load Analysis. Sensors, 20.","DOI":"10.3390\/s20195467"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.apergo.2018.08.028","article-title":"Measuring mental workload using neurophysiological measures: A systematic review","volume":"74","author":"Charles","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1109\/TSMCA.2010.2093888","article-title":"Adaptive automation, level of automation, allocation authority, supervisory control, and adaptive control: Distinctions and modes of adaptation","volume":"41","author":"Sheridan","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part. A Syst. Hum."},{"key":"ref_36","unstructured":"Lim, Y., Ranasinghe, K., Gardi, A., Ezer, N., and Sabatini, R. (2018, January 9\u201314). Human-Machine Interfaces and Interactions for Multi UAS Operations. Proceedings of the 31th Congress of the International Council of the Aeronautical Sciences (ICAS 2018), Belo Horizonte, Brazil."},{"key":"ref_37","unstructured":"Lim, Y., Samreeloy, T., Chantaraviwat, C., Ezer, N., Gardi, A., and Sabatini, R. (2019, January 24\u201326). Cognitive Human-Machine Interfaces and Interactions for Multi-UAV Operations. Proceedings of the 18th Australian International Aerospace Congress, Melbourne, Australia."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/12\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:07:59Z","timestamp":1760159279000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/12"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,7]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["robotics10010012"],"URL":"https:\/\/doi.org\/10.3390\/robotics10010012","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,7]]}}}