{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T02:02:45Z","timestamp":1770516165242,"version":"3.49.0"},"reference-count":55,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T00:00:00Z","timestamp":1758844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["52402507"],"award-info":[{"award-number":["52402507"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012130","name":"Aeronautical Science Foundation of China","doi-asserted-by":"publisher","award":["201813300002"],"award-info":[{"award-number":["201813300002"]}],"id":[{"id":"10.13039\/501100012130","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["Systems"],"abstract":"<jats:p>An experiment with a two interaction modalities (traditional: touch; novel: touch\u2013speech) \u00d7 two task complexities (low: visual single task; high: audio\u2013visual dual task) within-subjects design was conducted to observe alterations in crew performance (including task performance, subjective workload, and eye responses) in a typical planning task-based on a high-fidelity special vehicle simulation platform. The results revealed that (1) compared to the traditional interaction modality, the novel interaction modality significantly improved task performance, reduced subjective workload, increased mean peak saccade velocity, and decreased fixation entropy; (2) under high task complexity, subjective workload, mean pupil diameter, and the nearest neighbor index showed significant increases, while no significant decline in task performance was observed; (3) no interaction effect of crew performance was observed between interaction modality and task complexity. The foregoing results imply that (1) the novel interaction modality incorporating speech input exhibits advantages over the traditional touch-based modality in terms of enhancing task performance (over 45% improvement) and reducing cognitive workload; (2) leveraging dual-channel audio\u2013visual information processing facilitates the maintenance of task performance under high task complexity and multi-tasking demands; (3) eye movement characteristics may serve as informative indicators for evaluating the benefits of speech-based interaction and the effectiveness of cognitive resource allocation under high-complexity task conditions. The results can provide a basis for the design of the display and control interface in special vehicles.<\/jats:p>","DOI":"10.3390\/systems13100847","type":"journal-article","created":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T12:33:03Z","timestamp":1758889983000},"page":"847","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Effects of Introducing Speech Interaction Modality on Performance of Special Vehicle Crew Under Various Task Complexity Conditions"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9582-8008","authenticated-orcid":false,"given":"Chuanyan","family":"Feng","sequence":"first","affiliation":[{"name":"School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China"},{"name":"Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China"}]},{"given":"Shuang","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China"}]},{"given":"Xiaoru","family":"Wanyan","sequence":"additional","affiliation":[{"name":"School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9454-3549","authenticated-orcid":false,"given":"Chunying","family":"Qian","sequence":"additional","affiliation":[{"name":"School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China"}]},{"given":"Kun","family":"Ji","sequence":"additional","affiliation":[{"name":"School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China"}]},{"given":"Fang","family":"Xie","sequence":"additional","affiliation":[{"name":"China North Vehicle Research Institute, China North Industries Group Corporation Ltd., Beijing 100072, China"}]},{"given":"Yue","family":"Zhou","sequence":"additional","affiliation":[{"name":"China North Vehicle Research Institute, China North Industries Group Corporation Ltd., Beijing 100072, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,26]]},"reference":[{"key":"ref_1","first-page":"7632892","article-title":"Autonomous Vehicles and Intelligent Automation: Applications, Challenges, and Opportunities","volume":"2022","author":"Bathla","year":"2022","journal-title":"Mob. Inf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1177\/0018720819900402","article-title":"Challenges to Human Drivers in Increasingly Automated Vehicles","volume":"62","author":"Hancock","year":"2020","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1115","DOI":"10.1080\/00140139.2021.1905882","article-title":"Why Human Factors Science Is Demonstrably Necessary: Historical and Evolutionary Foundations","volume":"64","author":"Hancock","year":"2021","journal-title":"Ergonomics"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chan, D.W.M., Baghbaderani, A.B., and Sarvari, H. (2022). An Empirical Study of the Human Error-Related Factors Leading to Site Accidents in the Iranian Urban Construction Industry. Buildings, 12.","DOI":"10.3390\/buildings12111858"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Salvendy, G., and Karwowski, W. (2021). Human Errors and Human Reliability. Handbook of Human Factors and Ergonomics, Wiley.","DOI":"10.1002\/9781119636113"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2168","DOI":"10.1109\/TVCG.2020.2970512","article-title":"Touch? Speech? Or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and Analysis","volume":"26","author":"Saktheeswaran","year":"2020","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"106317","DOI":"10.1016\/j.chb.2020.106317","article-title":"Interface and Interaction Design: How Mobile Touch Devices Foster Cognitive Offloading","volume":"108","author":"Grinschgl","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"103256","DOI":"10.1016\/j.ergon.2021.103256","article-title":"The Effect of Systemic Sclerosis on Use of Mobile Touchscreen Interfaces: Design Guidelines and Physio-Rehabilitation","volume":"87","author":"Brunzini","year":"2022","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Detjen, H., Geisler, S., and Schneegass, S. (2020, January 11\u201314). Maneuver-Based Control Interventions During Automated Driving: Comparing Touch, Voice, and Mid-Air Gestures as Input Modalities. Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada.","DOI":"10.1109\/SMC42975.2020.9283431"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dudek, M., and Schulte, A. (2022, January 3). Effects of Tasking Modalities in Manned-Unmanned Teaming Missions. Proceedings of the AIAA SCITECH 2022 Forum, San Diego, CA, USA.","DOI":"10.2514\/6.2022-2478"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cha, M.C. (2025). Switching between Touch and Voice: Factors Influencing Modality Selection in Multimodal Systems. Ergonomics, 1\u201314.","DOI":"10.1080\/00140139.2025.2499200"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kim, Y.-H., Lee, B., Srinivasan, A., and Choe, E.K. (2021, January 6). Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445421"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3460938","article-title":"Affective Automotive User Interfaces\u2013Reviewing the State of Driver Affect Research and Emotion Regulation in the Car","volume":"54","author":"Braun","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Herbig, N., Pal, S., D\u00fcwel, T., Meladaki, K., Monshizadeh, M., Hnatovskiy, V., Kr\u00fcger, A., and van Genabith, J. (2020, January 5\u201310). MMPE: A Multi-Modal Interface Using Handwriting, Touch Reordering, and Speech Commands for Post-Editing Machine Translation. Proceedings of the Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online.","DOI":"10.18653\/v1\/2020.acl-demos.37"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"20416695231160420","DOI":"10.1177\/20416695231160420","article-title":"Touch and Voice Have Different Advantages in Perceiving Positive and Negative Emotions","volume":"14","author":"Oya","year":"2023","journal-title":"i-Perception"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"103958","DOI":"10.1016\/j.apergo.2022.103958","article-title":"Input Modality Matters: A Comparison of Touch, Speech, and Gesture Based in-Vehicle Interaction","volume":"108","author":"Zhang","year":"2023","journal-title":"Appl. Ergon."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"103348","DOI":"10.1016\/j.ergon.2022.103348","article-title":"Driver\u2019s Attitudes and Preferences toward Connected Vehicle Information System","volume":"91","author":"Zhou","year":"2022","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_18","first-page":"15","article-title":"Evaluation of Interface Modality for Control of Multiple Unmanned Vehicles","volume":"Volume 10276","author":"Harris","year":"2017","journal-title":"Engineering Psychology and Cognitive Ergonomics: Cognition and Design"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"103842","DOI":"10.1016\/j.apergo.2022.103842","article-title":"Multimodal Interaction: Input-Output Modality Combinations for Identification Tasks in Augmented Reality","volume":"105","author":"Lazaro","year":"2022","journal-title":"Appl. Ergon."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1177\/0018720818788995","article-title":"Effects of Touch, Voice, and Multimodal Input, and Task Load on Multiple-UAV Monitoring Performance During Simulated Manned-Unmanned Teaming in a Military Helicopter","volume":"60","author":"Levulis","year":"2018","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_21","unstructured":"Franca, A., Achim, E., Andreas, S., and Bernd, H. (2017, January 6\u201311). Virtual Reality Meets Smartwatch: Intuitive, Natural, and Multi-Modal Interaction. Proceedings of the 2017 chi Conference Extended Abstracts on Human Factors in Computing Systems, Denver, CO, USA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2682628","article-title":"ImageSpirit: Verbal Guided Image Parsing","volume":"34","author":"Cheng","year":"2014","journal-title":"ACM Trans. Graph."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zimmerer, C., Krop, P., Fischbach, M., and Latoschik, M.E. (2022, January 29). Reducing the Cognitive Load of Playing a Digital Tabletop Game with a Multimodal Interface. Proceedings of the CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA.","DOI":"10.1145\/3491102.3502062"},{"key":"ref_24","first-page":"16","article-title":"Touch-and-Go: Interior Tactile Communication in Armored Fighting Vehicles","volume":"28","author":"Katzman","year":"2020","journal-title":"Ergon. Des. Q. Hum. Factors Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e2101062118","DOI":"10.1073\/pnas.2101062118","article-title":"Task Complexity Moderates Group Synergy","volume":"118","author":"Almaatouq","year":"2021","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.aap.2019.01.019","article-title":"Effects of Time of Day and Taxi Route Complexity on Navigation Errors: An Experimental Study","volume":"125","author":"Zhang","year":"2019","journal-title":"Accid. Anal. Prev."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.apergo.2019.01.012","article-title":"The Effects of Flight Complexity on Gaze Entropy: An Experimental Study with Fighter Pilots","volume":"77","author":"Rieiro","year":"2019","journal-title":"Appl. Ergon."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1007\/978-3-031-21671-8_7","article-title":"Task Complexity and Performance in Individuals and Groups Without Communication","volume":"Volume 13775","author":"Gurney","year":"2022","journal-title":"Computational Theory of Mind for Human-Machine Teams"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hebbar, P.A., Bhattacharya, K., Prabhakar, G., Pashilkar, A.A., and Biswas, P. (2021). Correlation Between Physiological and Performance-Based Metrics to Estimate Pilots\u2019 Cognitive Workload. Front. Psychol., 12.","DOI":"10.3389\/fpsyg.2021.555446"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.jss.2022.07.010","article-title":"The Measurement of Cognitive Workload in Surgery Using Pupil Metrics: A Systematic Review and Narrative Analysis","volume":"280","author":"Naik","year":"2022","journal-title":"J. Surg. Res."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/j.erap.2013.09.001","article-title":"Saccadic Peak Velocity as an Alternative Index of Operator Attention: A Short Review","volume":"63","author":"Stasi","year":"2013","journal-title":"Eur. Rev. Appl. Psychol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"N62","DOI":"10.1088\/0967-3334\/37\/9\/N62","article-title":"Fatigue in the Military: Towards a Fatigue Detection Test Based on the Saccadic Velocity","volume":"37","author":"Rieiro","year":"2016","journal-title":"Physiol. Meas."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1518\/155534307X255627","article-title":"A Random Glance at the Flight Deck: Pilots\u2019 Scanning Strategies and the Real-Time Assessment of Mental Workload","volume":"1","author":"Nocera","year":"2007","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_34","unstructured":"Dudek, M., and Schulte, A. (2020, January 18\u201319). Meaningful Guidance of Unmanned Aerial Vehicles in Dynamic Environments. Proceedings of the 1st International Conference on Cognitive Aircraft Systems, Toulouse, France."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"103254","DOI":"10.1016\/j.net.2024.10.016","article-title":"Scrutinizing the Effect of Task Complexity on Operators\u2019 Task Performance Time in a Digitalized Main Control Room of a Nuclear Power Plant","volume":"57","author":"Jang","year":"2025","journal-title":"Nucl. Eng. Technol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1016\/j.ergon.2012.09.001","article-title":"Task Complexity: A Review and Conceptualization Framework","volume":"42","author":"Liu","year":"2012","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume":"52","author":"Hart","year":"1988","journal-title":"Adv. Psychol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"413","DOI":"10.3357\/ASEM.2579.2010","article-title":"Saccadic Peak Velocity Sensitivity to Variations in Mental Workload","volume":"81","author":"Stasi","year":"2010","journal-title":"Aviat. Space Environ. Med."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2854","DOI":"10.1016\/j.promfg.2015.07.783","article-title":"de Review of Eye-Related Measures of Drivers\u2019 Mental Workload","volume":"3","author":"Marquart","year":"2015","journal-title":"Procedia Manuf."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"NASA-Task Load Index (NASA-TLX): 20 Years Later","volume":"50","author":"Hart","year":"2006","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"103193","DOI":"10.1016\/j.ergon.2021.103193","article-title":"The Ability of Eye-Tracking Metrics to Classify and Predict the Perceived Driving Workload","volume":"86","author":"Bitkina","year":"2021","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1340","DOI":"10.1080\/08941939.2021.2025282","article-title":"Eye-Tracking Indicators of Workload in Surgery: A Systematic Review","volume":"35","author":"Tolvanen","year":"2022","journal-title":"J. Investig. Surg."},{"key":"ref_43","unstructured":"Field, A. (2024). Discovering Statistics Using IBM SPSS Statistics, Sage Publications Limited. [6th ed.]."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1111\/opo.12399","article-title":"Recommendations for Analysis of Repeated-Measures Designs: Testing and Correcting for Sphericity and Use of MANOVA and Mixed Model Analysis","volume":"37","author":"Armstrong","year":"2017","journal-title":"Ophthalmic Physiol. Opt."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"407","DOI":"10.4103\/aca.ACA_94_19","article-title":"Application of Student\u2019s t-Test, Analysis of Variance, and Covariance","volume":"22","author":"Mishra","year":"2019","journal-title":"Ann. Card. Anaesth."},{"key":"ref_46","unstructured":"Pfleging, B. (December, January 29). SpeeT: A Multimodal Interaction Style Combining Speech and Touch Interaction in Automotive Environments. Proceedings of the Automotive UI, Salzburg, Austria."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.cja.2022.06.011","article-title":"Effects of Input Method and Display Mode of Situation Map on Early Warning Aircraft Reconnaissance Task Performance with Different Information Complexities","volume":"36","author":"Liang","year":"2023","journal-title":"Chin. J. Aeronaut."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Lee, J., Rodriguez, S.S., Natarrajan, R., Chen, J., Deep, H., and Kirlik, A. (2021, January 18). What\u2019s This? A Voice and Touch Multimodal Approach for Ambiguity Resolution in Voice Assistants. Proceedings of the 2021 International Conference on Multimodal Interaction, Montr\u00e9al, QC, Canada.","DOI":"10.1145\/3462244.3479902"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1518\/001872008X288394","article-title":"Multiple Resources and Mental Workload","volume":"50","author":"Wickens","year":"2008","journal-title":"Hum. Factors"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Maggi, P., and Nocera, F.D. (2021). Sensitivity of the Spatial Distribution of Fixations to Variations in the Type of Task Demand and Its Relation to Visual Entropy. Front. Hum. Neurosci., 15.","DOI":"10.3389\/fnhum.2021.642535"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"106640","DOI":"10.1016\/j.aap.2022.106640","article-title":"Multimodal Driver State Modeling through Unsupervised Learning","volume":"170","author":"Tavakoli","year":"2022","journal-title":"Accid. Anal. Prev."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1365","DOI":"10.1177\/0018720819874544","article-title":"Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training","volume":"62","author":"Wu","year":"2020","journal-title":"Hum. Factors J. Hum. Factors Ergon. Soc."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"e08826","DOI":"10.1016\/j.heliyon.2022.e08826","article-title":"Multiple Levels of Mental Attentional Demand Modulate Peak Saccade Velocity and Blink Rate","volume":"8","author":"Bachurina","year":"2022","journal-title":"Heliyon"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"5569","DOI":"10.1007\/s12652-020-01913-3","article-title":"A Novel Muscle-Computer Interface for Hand Gesture Recognition Using Depth Vision","volume":"11","author":"Zhou","year":"2020","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"2425","DOI":"10.1109\/JBHI.2024.3406737","article-title":"A Review of AIoT-Based Human Activity Recognition: From Application to Technique","volume":"29","author":"Qi","year":"2025","journal-title":"IEEE J. Biomed. Health Inform."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/10\/847\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T12:44:23Z","timestamp":1758890663000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/10\/847"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,26]]},"references-count":55,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["systems13100847"],"URL":"https:\/\/doi.org\/10.3390\/systems13100847","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,26]]}}}