{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T09:37:46Z","timestamp":1778233066803,"version":"3.51.4"},"reference-count":44,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T00:00:00Z","timestamp":1737072000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union","award":["101006664"],"award-info":[{"award-number":["101006664"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Remote assistance for highly automated vehicles (HAVs), i.e., third-party assistance from support staff outside the vehicle in times of the need for assistance, presents a solution to extend the capabilities of HAVs by integrating a third party for decision making in uncertain situations. Similar to other control center positions, we expect the remote assistance tasks to exert high mental demands on the human operators. Therefore, we assessed impact of elevated mental workload during HAV remote assistance in a controlled environment in a user study (N = 37) with the goal of identifying cues to differentiate workload levels based on eye-tracking-related, skin conductance, and cardiovascular indicators. The results provide evidence that (A) elevated workload induced via a secondary task depreciates performance, and (B) we can identify workload levels person-independently as differences in tonic skin conductance (F(2,72) = 24.538, p &lt; 0.001, partial \u03b7\u00b2 = 0.405) and pupil dilation (F(2,72) = 13.872, p &lt; 0.001, partial \u03b7\u00b2 = 0.278), resulting in a classification accuracy of 58% in a three-class classification task. The results provide evidence that we are able to differentiate operator workload during remote assistance in a time-resolved way with the ultimate goal to provide adaptations to counteract task deficiencies.<\/jats:p>","DOI":"10.3390\/info16010064","type":"journal-article","created":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T11:24:56Z","timestamp":1737113096000},"page":"64","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Multimodal Assessment of Mental Workload During Automated Vehicle Remote Assistance: Modeling of Eye-Tracking-Related, Skin Conductance, and Cardiovascular Indicators"],"prefix":"10.3390","volume":"16","author":[{"given":"Fabian","family":"Walocha","sequence":"first","affiliation":[{"name":"German Aerospace Center, Institute of Transportation Systems, 38108 Braunschweig, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8352-1052","authenticated-orcid":false,"given":"Andreas","family":"Schrank","sequence":"additional","affiliation":[{"name":"German Aerospace Center, Institute of Transportation Systems, 38108 Braunschweig, Germany"}]},{"given":"Hoai Phuong","family":"Nguyen","sequence":"additional","affiliation":[{"name":"German Aerospace Center, Institute of Transportation Systems, 38108 Braunschweig, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7911-3512","authenticated-orcid":false,"given":"Klas","family":"Ihme","sequence":"additional","affiliation":[{"name":"German Aerospace Center, Institute of Transportation Systems, 38108 Braunschweig, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2025,1,17]]},"reference":[{"key":"ref_1","unstructured":"SAE International (2024, November 01). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, J3016_202104, 2021 (J3016). Available online: https:\/\/www.sae.org\/standards\/content\/j3016_202104."},{"key":"ref_2","unstructured":"(2023, October 25). Deutscher Bundestag. Gesetz zur \u00c4nderung des Stra\u00dfenverkehrsgesetzes und des Pflichtversicherungsgesetzes\u2014Gesetz zum Autonomen Fahren, 2021. Bundesgesetzblatt (48). Available online: https:\/\/www.gesetze-im-internet.de\/stvg\/BJNR004370909.html."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kettwich, C., Schrank, A., Avsar, H., and Oehl, M. (2022). A Helping Human Hand: Relevant Scenarios for the Remote Operation of Highly Automated Vehicles in Public Transport. Appl. Sci., 12.","DOI":"10.3390\/app12094350"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kettwich, C., Schrank, A., and Oehl, M. (2021). Teleoperation of Highly Automated Vehicles in Public Transport: User-Centered Design of a Human-Machine Interface for Remote-Operation and Its Expert Usability Evaluation. MTI, 5.","DOI":"10.3390\/mti5050026"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/s10111-024-00753-x","article-title":"Human-centered design and evaluation of a workplace for the remote assistance of highly automated vehicles","volume":"26","author":"Schrank","year":"2024","journal-title":"Cogn. Tech. Work"},{"key":"ref_6","unstructured":"Meyer, G., and Beiker, S. (2024). Human Factors Considerations of Remote Operation Supporting Level 4 Automation. Road Vehicle Automation 11, Springer Nature Switzerland."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00140139.2014.956151","article-title":"State of science: Mental workload in ergonomics","volume":"58","author":"Young","year":"2015","journal-title":"Ergonomics"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1080\/14639220210123806","article-title":"Multiple resources and performance prediction","volume":"3","author":"Wickens","year":"2002","journal-title":"Theor. Issues Ergon. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Migliorini, Y., Imbert, J.-P., Roy, R.N., Lafont, A., and Dehais, F. (2022). Degraded States of Engagement in Air Traffic Control. Safety, 8.","DOI":"10.3390\/safety8010019"},{"key":"ref_10","unstructured":"Gramann, K., and Schandry, R. (2009). Psychophysiologie: K\u00f6rperliche Indikatoren psychischen Geschehens, Beltz PVU. 4., vollst. \u00fcberarb. Aufl."},{"key":"ref_11","unstructured":"Burnett, G., Gabbard, J., Green, P., Osswald, S., Pfleging, B., Kun, A., Eren, A., and Antrobus, V. (2015). Comparing heart rate and pupil size as objective measures of workload in the driving context. Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Nottingham, UK, 1\u20133 September 2015, ACM."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2854","DOI":"10.1016\/j.promfg.2015.07.783","article-title":"Review of Eye-related Measures of Drivers\u2019 Mental Workload","volume":"3","author":"Marquart","year":"2015","journal-title":"Procedia Manuf."},{"key":"ref_13","unstructured":"Mandryk, R., Hancock, M., Perry, M., and Cox, A. (2018). Look into my Eyes. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21\u201326 April 2018, ACM."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"291","DOI":"10.3233\/WOR-205318","article-title":"Subjective and objective assessments of mental workload for UAV operations","volume":"72","author":"Li","year":"2022","journal-title":"Work"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1177\/001872088202400605","article-title":"Cognitive load and the functional field of view","volume":"24","author":"Williams","year":"1982","journal-title":"Hum. Factors"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.chb.2008.12.021","article-title":"Uncovering cognitive processes: Different techniques that can contribute to cognitive load research and instruction","volume":"25","author":"Kester","year":"2009","journal-title":"Comput. Hum. Behav."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1037\/0033-2909.91.2.276","article-title":"Task-evoked pupillary responses, processing load, and the structure of processing resources","volume":"91","author":"Beatty","year":"1982","journal-title":"Psychol. Bull."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ziegler, M.G. (2012). Psychological Stress and the Autonomic Nervous System. Primer on the Autonomic Nervous System, Elsevier.","DOI":"10.1016\/B978-0-12-386525-0.00061-5"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Magjarevic, R., Nagel, J.H., Vander Sloten, J., Verdonck, P., Nyssen, M., and Haueisen, J. (2009). Influence of Mental Stress on Heart Rate and Heart Rate Variability. 4th European Conference of the International Federation for Medical and Biological Engineering, Springer Berlin Heidelberg.","DOI":"10.1007\/978-3-540-89208-3"},{"key":"ref_20","unstructured":"Dinges, D.F., Mallis, M.M., Maislin, G., and Powell, J.W. (2024, November 18). Evaluation of Techniques for Ocular Measurement as An Index of Fatigue and the Basis for Alertness Management, Washington, DC, USA, 1998. Available online: https:\/\/trid.trb.org\/View\/647942."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1177\/001872089403600209","article-title":"Blink rate: A possible measure of fatigue","volume":"36","author":"Stern","year":"1994","journal-title":"Hum. Factors"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1355","DOI":"10.5665\/sleep.2964","article-title":"Cues of fatigue: Effects of sleep deprivation on facial appearance","volume":"36","author":"Sundelin","year":"2013","journal-title":"Sleep"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.apergo.2018.08.028","article-title":"Measuring mental workload using physiological measures: A systematic review","volume":"74","author":"Charles","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_24","unstructured":"Feary, M., Feuerle, T., Rechea, C.G., Saez, F.J., Johnson, C., Martinie, C., Palanque, P., Pasquini, A., van Leeuwen, P., and Winckler, M. (2015). Pupillometry to assess Air Traffic Controller workload through the Mental Workload Model. Proceedings of the 5th International Conference on Application and Theory of Automation in Command and Control Systems, Toulouse, France, 30 September\u20132 October 2015, ACM."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3372","DOI":"10.1021\/acs.iecr.5b03685","article-title":"Pupillometry Based Real-Time Monitoring of Operator\u2019s Cognitive Workload To Prevent Human Error during Abnormal Situations","volume":"55","author":"Bhavsar","year":"2016","journal-title":"Ind. Eng. Chem. Res."},{"key":"ref_26","first-page":"2283","article-title":"Improving pilot mental workload evaluation with combined measures","volume":"24","author":"Wanyan","year":"2014","journal-title":"Biomed. Mater. Eng."},{"key":"ref_27","first-page":"1","article-title":"Electrophysiological and performance variations following driving events involving an increase in mental workload","volume":"11","author":"Berthelon","year":"2019","journal-title":"Eur. Transp. Res. Rev."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.apergo.2018.06.006","article-title":"Mental workload is reflected in driver behaviour, physiology, eye movements and prefrontal cortex activation","volume":"73","author":"Foy","year":"2018","journal-title":"Appl. Ergon."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Unni, A., Ihme, K., Jipp, M., and Rieger, J.W. (2017). Assessing the Driver\u2019s Current Level of Working Memory Load with High Density Functional Near-infrared Spectroscopy: A Realistic Driving Simulator Study. Front. Hum. Neurosci., 11.","DOI":"10.3389\/fnhum.2017.00167"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1037\/h0043688","article-title":"Age differences in short-term retention of rapidly changing information","volume":"55","author":"KIRCHNER","year":"1958","journal-title":"J. Exp. Psychol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1080\/10447318.2018.1456150","article-title":"A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale","volume":"35","author":"Franke","year":"2019","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref_32","unstructured":"Fischer, M., Richter, A., Pl\u00e4ttner, J., Temme, G., Kelsch, J., Assmann, D., and K\u00f6ster, F. (2014, January 4\u20135). Modular and scalable driving simulator hardware and software for the development of future driver assistance and automation systems. Proceedings of the Driving Simulation Conference 2024, Driving Simulation Conference 2024, Paris, France."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1177\/0018720819886765","article-title":"Drivers\u2019 Cue Utilization Predicts Cognitive Resource Consumption During a Simulated Driving Scenario","volume":"63","author":"Sturman","year":"2021","journal-title":"Hum. Factors"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1097\/00006842-196301000-00004","article-title":"The Relationship Between Autonomic Indicators of Psychological Stress: Heart Rate and Skin Conductance","volume":"25","author":"Lazarus","year":"1963","journal-title":"Psychosom. Med."},{"key":"ref_35","unstructured":"Hancock, P.A., and Meshkati, N. (1988). Development of the NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human Mental Workload, Elsevier."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"Nasa-Task Load Index (NASA-TLX); 20 Years Later","volume":"50","author":"Hart","year":"2006","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_37","unstructured":"Krishnapuram, B., Shah, M., Smola, A., Aggarwal, C., Shen, D., and Rastogi, R. (2016). XGBoost. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13\u201317 August 2016, ACM."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Meir, A., Grimberg, E., and Musicant, O. (2024). The human-factors\u2019 challenges of (tele)drivers of Autonomous Vehicles. Ergonomics, 1\u201321.","DOI":"10.1080\/00140139.2024.2346552"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"979","DOI":"10.1080\/10803548.2019.1675259","article-title":"Study of mental workload imposed by different tasks based on teleoperation","volume":"27","author":"Shao","year":"2021","journal-title":"Int. J. Occup. Saf. Ergon."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Tang, L., Si, J., Sun, L., Mao, G., and Yu, S. (2022). Assessment of the mental workload of trainee pilots of remotely operated aircraft using functional near-infrared spectroscopy. BMC Neurol., 22.","DOI":"10.1186\/s12883-022-02683-5"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.bbr.2013.10.042","article-title":"Using near infrared spectroscopy and heart rate variability to detect mental overload","volume":"259","author":"Durantin","year":"2014","journal-title":"Behav. Brain Res."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Uchitel, J., Vidal-Rosas, E.E., Cooper, R.J., and Zhao, H. (2021). Wearable, Integrated EEG-fNIRS Technologies: A Review. Sensors, 21.","DOI":"10.3390\/s21186106"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"e00204","DOI":"10.1016\/j.ohx.2021.e00204","article-title":"A low-cost, wearable, do-it-yourself functional near-infrared spectroscopy (DIY-fNIRS) headband","volume":"10","author":"Tsow","year":"2021","journal-title":"HardwareX"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/1743-0003-9-5","article-title":"Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors","volume":"9","author":"Liao","year":"2012","journal-title":"J. Neuroeng. Rehabil."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/1\/64\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T10:31:07Z","timestamp":1759919467000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/1\/64"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,17]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["info16010064"],"URL":"https:\/\/doi.org\/10.3390\/info16010064","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,17]]}}}