{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T17:54:59Z","timestamp":1764784499868,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T00:00:00Z","timestamp":1600819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Thales Australia and Northrop Grumman Corporations","award":["RE-02544-0200315666 and RE-03163-0200317164"],"award-info":[{"award-number":["RE-02544-0200315666 and RE-03163-0200317164"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The continuing development of avionics for Unmanned Aircraft Systems (UASs) is introducing higher levels of intelligence and autonomy both in the flight vehicle and in the ground mission control, allowing new promising operational concepts to emerge. One-to-Many (OTM) UAS operations is one such concept and its implementation will require significant advances in several areas, particularly in the field of Human\u2013Machine Interfaces and Interactions (HMI2). Measuring cognitive load during OTM operations, in particular Mental Workload (MWL), is desirable as it can relieve some of the negative effects of increased automation by providing the ability to dynamically optimize avionics HMI2 to achieve an optimal sharing of tasks between the autonomous flight vehicles and the human operator. The novel Cognitive Human Machine System (CHMS) proposed in this paper is a Cyber-Physical Human (CPH) system that exploits the recent technological developments of affordable physiological sensors. This system focuses on physiological sensing and Artificial Intelligence (AI) techniques that can support a dynamic adaptation of the HMI2 in response to the operators\u2019 cognitive state (including MWL), external\/environmental conditions and mission success criteria. However, significant research gaps still exist, one of which relates to a universally valid method for determining MWL that can be applied to UAS operational scenarios. As such, in this paper we present results from a study on measuring MWL on five participants in an OTM UAS wildfire detection scenario, using Electroencephalogram (EEG) and eye tracking measurements. These physiological data are compared with a subjective measure and a task index collected from mission-specific data, which serves as an objective task performance measure. The results show statistically significant differences for all measures including the subjective, performance and physiological measures performed on the various mission phases. Additionally, a good correlation is found between the two physiological measurements and the task index. Fusing the physiological data and correlating with the task index gave the highest correlation coefficient (CC = 0.726 \u00b1 0.14) across all participants. This demonstrates how fusing different physiological measurements can provide a more accurate representation of the operators\u2019 MWL, whilst also allowing for increased integrity and reliability of the system.<\/jats:p>","DOI":"10.3390\/s20195467","type":"journal-article","created":{"date-parts":[[2020,9,24]],"date-time":"2020-09-24T02:56:43Z","timestamp":1600916203000},"page":"5467","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["A Cyber-Physical-Human System for One-to-Many UAS Operations: Cognitive Load Analysis"],"prefix":"10.3390","volume":"20","author":[{"given":"Lars J.","family":"Planke","sequence":"first","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"given":"Yixiang","family":"Lim","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4995-4166","authenticated-orcid":false,"given":"Alessandro","family":"Gardi","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3399-2291","authenticated-orcid":false,"given":"Roberto","family":"Sabatini","sequence":"additional","affiliation":[{"name":"School of Engineering, RMIT University, Bundoora, VIC 3083, Australia"}]},{"given":"Trevor","family":"Kistan","sequence":"additional","affiliation":[{"name":"THALES Australia\u2014Airspace Mobility Solutions, WTC North Wharf, Melbourne, VIC 3000, Australia"}]},{"given":"Neta","family":"Ezer","sequence":"additional","affiliation":[{"name":"Northrop Grumman Corporation, 1550 W. Nursery Rd, Linthicum Heights, MD 21090, USA"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.paerosci.2018.05.002","article-title":"Avionics Human-Machine Interfaces and Interactions for Manned and Unmanned Aircraft","volume":"102","author":"Lim","year":"2018","journal-title":"Prog. Aerosp. Sci."},{"key":"ref_2","first-page":"31","article-title":"The Effect of Automation on Human Factors in Aviation","volume":"3","author":"Brown","year":"2017","journal-title":"J. Instrum. Autom. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pongsakornsathien, N., Lim, Y., Gardi, A., Hilton, S., Planke, L., Sabatini, R., Kistan, T., and Ezer, N. (2019). Ezer Sensor Networks for Aerospace Human-Machine Systems. Sensors, 19.","DOI":"10.3390\/s19163465"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1007\/s10846-017-0648-9","article-title":"Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft","volume":"91","author":"Lim","year":"2017","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.knosys.2016.08.031","article-title":"Cognitive pilot-aircraft interface for single-pilot operations","volume":"112","author":"Liu","year":"2016","journal-title":"Knowl. Based Syst."},{"key":"ref_6","unstructured":"Alicia, T., Taylor, G., Turpin, T., and Surana, A. (2018). Removing the Bottleneck: Utilizing Autonomy to Manage Multiple UAS Sensors from Inside a Cockpit, SPIE."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MAES.2017.160175","article-title":"Commercial airline single-pilot operations: System design and pathways to certification","volume":"32","author":"Lim","year":"2017","journal-title":"IEEE Aerosp. Electron. Syst. Mag."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Taylor, G., Purman, B., Schermerhorn, P., Garcia-Sampedro, G., Hubal, R., Crabtree, K., Rowe, A., and Spriggs, S. (2015). Multi-Modal Interaction for UAS Control, SPIE.","DOI":"10.1117\/12.2180020"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"31001","DOI":"10.1088\/1741-2560\/12\/3\/031001","article-title":"EEG artifact removal\u2014State-of-the-art and guidelines","volume":"12","year":"2015","journal-title":"J. Neural Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"16495","DOI":"10.1109\/ACCESS.2017.2735633","article-title":"A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms","volume":"5","author":"Kar","year":"2017","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.measurement.2019.03.032","article-title":"Experimental characterisation of eye-tracking sensors for adaptive human-machine systems","volume":"140","author":"Lim","year":"2019","journal-title":"Measurement"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"295","DOI":"10.3389\/fnhum.2019.00295","article-title":"Theories and methods for labeling cognitive workload: Classification and transfer learning","volume":"13","author":"McKendrick","year":"2019","journal-title":"Front. Hum. Neurosci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"359","DOI":"10.3389\/fnhum.2017.00359","article-title":"Measuring mental workload with EEG+fNIRS","volume":"11","author":"Aghajani","year":"2017","journal-title":"Front. Hum. Neurosci."},{"key":"ref_14","first-page":"539","article-title":"Adaptive automation triggered by eeg-based mental workload index: A passive brain-computer interface application in realistic air traffic control environment","volume":"10","author":"Borghini","year":"2016","journal-title":"Front. Hum. Neurosci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1016\/j.neuroimage.2011.07.047","article-title":"Adaptive training using an artificial neural network and eeg metrics for within-and cross-task workload classification","volume":"59","author":"Baldwin","year":"2012","journal-title":"NeuroImage"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1518\/155534307X255618","article-title":"Supporting real-time cognitive state classification on a mobile individual","volume":"1","author":"Dorneich","year":"2007","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Gateau, T., Durantin, G., Lancelot, F., Scannella, S., and Dehais, F. (2015). Real-time state estimation in a flight simulator using fnirs. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0121279"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yuksel, B.F., Oleson, K.B., Harrison, L., Peck, E.M., Afergan, D., Chang, R., and Jacob, R.J. (2016, January 7\u201312). Learn Piano with Bach: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858388"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Tao, D., Tan, H., Wang, H., Zhang, X., Qu, X., and Zhang, T. (2019). A systematic review of physiological measures of mental workload. Int. J. Environ. Res. Public Health, 16.","DOI":"10.3390\/ijerph16152716"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.apergo.2018.08.028","article-title":"Measuring mental workload using physiological measures: A systematic review","volume":"74","author":"Charles","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00140139.2014.956151","article-title":"State of science: Mental workload in ergonomics","volume":"58","author":"Young","year":"2015","journal-title":"Ergonomics"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Longo, L., and Leva, M.C. (2019). Mental Workload Monitoring: New Perspectives from Neuroscience. Human Mental Workload: Models and Applications, Springer International Publishing.","DOI":"10.1007\/978-3-030-32423-0"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1518\/155534308X284417","article-title":"Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs","volume":"2","author":"Parasuraman","year":"2008","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1002\/hfm.20269","article-title":"Brief review on physiological and biochemical evaluations of human mental workload","volume":"22","author":"Lean","year":"2012","journal-title":"Hum. Factors Ergon. Manuf."},{"key":"ref_25","unstructured":"Hilburn, B. (2004). Cognitive complexity in air traffic control: A literature review. EEC Note, Eurocontrol. Tech. Rep. 04\/04."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1080\/02678379608256795","article-title":"The inverted-u relationship between stress and performance: A field study","volume":"10","author":"Westman","year":"1996","journal-title":"Work Stress"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of nasa-tlx (task load index): Results of empirical and theoretical research","volume":"Volume 52","author":"Hancock","year":"1988","journal-title":"Advances in Psychology"},{"key":"ref_28","unstructured":"Hulbert, T. (1989). A Comparison of the \u201cNasa-Tlx\u201d and \u201cIsa\u201d Subjective Workload Rating Techniques\u2014Internal Report, Civil Aviation Authority Air Traffic Control Evaluation Unit."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Peterson, D.A., and Kozhokar, D. (2017, January 9\u201313). Peak-End Effects for Subjective Mental Workload Ratings. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Austin, TX, USA.","DOI":"10.1177\/1541931213601991"},{"key":"ref_30","unstructured":"De Waard, D. (1996). The Measurement of Drivers\u2019 Mental Workload. [Ph.D. Thesis, Groningen University]."},{"key":"ref_31","unstructured":"Council, N.R. (1995). Human Factors in the Design of Tactical Display Systems for the Individual Soldier, The National Academies Press."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1207\/S15326985EP3801_7","article-title":"Direct measurement of cognitive load in multimedia learning","volume":"38","author":"Brunken","year":"2003","journal-title":"Educ. Psychol."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Longo, L., and Leva, M.C. (2019). Real-time mental workload estimation using eeg. Human Mental Workload: Models and Applications, Springer International Publishing.","DOI":"10.1007\/978-3-030-32423-0"},{"key":"ref_34","unstructured":"Noback, C.R., Strominger, N.L., Demarest, R.J., and Ruggiero, D.A. (2005). The Human Nervous System: Structure and Function, Humana Press. [6th ed.]."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"94","DOI":"10.2466\/22.PMS.121c12x5","article-title":"Effective indices for monitoring mental workload while performing multiple tasks 1","volume":"121","author":"Hsu","year":"2015","journal-title":"Percept. Mot. Skills"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"van Erp, J.B., Veltman, H.J., and Grootjen, M. (2010). Brain-based indices for user system symbiosis. Brain-Computer Interfaces, Springer.","DOI":"10.1007\/978-1-84996-272-8_12"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.ijpsycho.2017.10.004","article-title":"Using theta and alpha band power to assess cognitive workload in multitasking environments","volume":"123","author":"Puma","year":"2018","journal-title":"Int. J. Psychophysiol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1518\/001872001775898287","article-title":"Monitoring task loading with multivariate eeg measures during complex forms of human-computer interaction","volume":"43","author":"Smith","year":"2001","journal-title":"Hum. Factors"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/S0165-0173(98)00056-3","article-title":"Eeg alpha and theta oscillations reflect cognitive and memory performance: A review and analysis","volume":"29","author":"Klimesch","year":"1999","journal-title":"Brain Res. Rev."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1093\/cercor\/7.4.374","article-title":"High-resolution eeg mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice","volume":"7","author":"Gevins","year":"1997","journal-title":"Cereb. Cortex"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1518\/001872098779480578","article-title":"Monitoring working memory load during computer-based tasks with eeg pattern recognition methods","volume":"40","author":"Gevins","year":"1998","journal-title":"Hum. Factors"},{"key":"ref_42","unstructured":"Antonenko, P.D. (2007). The Effect of Leads on Cognitive Load and Learning in a Conceptually Rich Hypertext Environment. [Ph.D. Thesis, Iowa State University]."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1504\/IJAACS.2013.050696","article-title":"Using minimal number of electrodes for emotion detection using brain signals produced from a new elicitation technique","volume":"6","author":"Mikhail","year":"2013","journal-title":"Int. J. Auton. Adapt. Commun. Syst."},{"key":"ref_44","unstructured":"Holmqvist, K., Nystr\u00f6m, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de Weijer, J. (2011). Eye Tracking: A Comprehensive Guide to Methods and Measures, Oxford University Press."},{"key":"ref_45","unstructured":"Gilland, J. (2008). Driving, Eye-Tracking and Visual Entropy: Exploration of Age and Task Effects. [Ph.D. Thesis, The University of South Dakota]."},{"key":"ref_46","unstructured":"Scanella, S., Peysakhovich, V., Ehrig, F., and Dehais, F. (2015, January 16\u201321). Can Flight Phase be Inferred Using Eye Movements? Evidence from Real Flight Conditions. Proceedings of the 18th European Conference on Eye Movements, Vienna, Austria."},{"key":"ref_47","unstructured":"Gardi, A., Pongsakornsathien, N., Planke, L., Lim, Y., Kistan, T., and Sabatini, R. (2019). Development of a Cognitive Hmi for Air Traffic Management Systems\u2014Report no. 4: Chmi2 System-Level Implementation and Verification in a Representative Simulation Environment, RMIT University and THALES Australia. RMIT\/SENG\/CPS\/003-2019."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Pongsakornsathien, N., Gardi, A., Sabatini, R., Kistan, T., and Ezer, N. (2020, January 11\u201315). Human-machine interactions in very-low-level uas operations and traffic management. Proceedings of the IEEE\/AIAA 39th Digital Avionics Systems Conference, DASC2020, Piscataway, NJ, USA.","DOI":"10.1109\/DASC50938.2020.9256757"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"025005","DOI":"10.1088\/1741-2560\/8\/2\/025005","article-title":"Towards passive brain-computer interfaces: Applying brain-computer interface technology to human-machine systems in general","volume":"8","author":"Zander","year":"2011","journal-title":"J. Neural Eng."},{"key":"ref_50","unstructured":"Lim, Y., Ranasinghe, K., Gardi, A., Ezer, N., and Sabatini, R. (2018, January 9\u201314). Human-machine interfaces and interactions for multi uas operations. Proceedings of the 31st Congress of the International Council of the Aeronautical Sciences (ICAS 2018), Belo Horizonte, Brazil."},{"key":"ref_51","unstructured":"Lim, Y., Samreeloy, T., Chantaraviwat, C., Ezer, N., Gardi, A., and Sabatini, R. (2019, January 24\u201328). Cognitive human-machine interfaces and interactions for multi-uav operations. Proceedings of the Australian International Aerospace Congress (AIAC18), Melbourne, Australia."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5467\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:13:02Z","timestamp":1760177582000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5467"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,23]]},"references-count":51,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2020,10]]}},"alternative-id":["s20195467"],"URL":"https:\/\/doi.org\/10.3390\/s20195467","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,9,23]]}}}