{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,8]],"date-time":"2026-03-08T22:18:35Z","timestamp":1773008315032,"version":"3.50.1"},"reference-count":32,"publisher":"Institution of Engineering and Technology (IET)","issue":"1","license":[{"start":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T00:00:00Z","timestamp":1755820800000},"content-version":"vor","delay-in-days":233,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2023YFB2604100"],"award-info":[{"award-number":["2023YFB2604100"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U2233208"],"award-info":[{"award-number":["U2233208"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["ietresearch.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["IET Biometrics"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:p>Fatigue among air traffic controllers is a factor contributing to civil aviation crashes. Existing methods for extracting and fuzing fatigue features encounter two main challenges: (1) the low accuracy of traditional single\u2010mode fatigue recognition methods, and (2) disregarding multimodal data correlations in traditional multimodal methods for feature concatenation and fusion. This paper proposes an interactive algorithm for the fusion and recognition of multimode fatigue features that combines multihead attention (MHA) and cross\u2010attention (XATTN) which are based on an improved speech and facial fatigue recognition model. First, an improved conformer model which combines a convolutional module with a transformer encoder is proposed using the radiotelephony communication data of controllers by employing the filter bank method for extracting profound speech features. Second, facial data of controllers are processed via pointwise convolutions employing a stack of inverted residual layers, which facilitates the extraction of facial features. Third, speech and facial features are fuzed interactively by combining MHA and XATTN, which achieves high accuracy of recognizing the fatigue state of controllers working in complex operational environments. A series of experiments were conducted with audiovisual data sets collected from actual air traffic control (ATC) missions. Comparing with four competing methods for fuzing multimodal features, the experimental results indicate that the proposed method for fuzing multimode features achieved a recognition accuracy of 99.2%, which was 8.9% higher than that for a speech single\u2010mode model and 0.4% higher than that for a facial single\u2010mode model.<\/jats:p>","DOI":"10.1049\/bme2\/7626919","type":"journal-article","created":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T06:49:16Z","timestamp":1755931756000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Dynamic Interactive Fusion Model for Extracting Fatigue Features Based on the Audiovisual Data Flow of Air Traffic Controllers"],"prefix":"10.1049","volume":"2025","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6103-750X","authenticated-orcid":false,"given":"Zhiyuan","family":"Shen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xueyan","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junqi","family":"Bai","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kai","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4098-998X","authenticated-orcid":false,"given":"Yifan","family":"Xu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"265","published-online":{"date-parts":[[2025,8,22]]},"reference":[{"key":"e_1_2_12_1_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2019.2941773"},{"key":"e_1_2_12_2_2","doi-asserted-by":"crossref","unstructured":"WangJ. LiL. WangD. andZhengT. F. Research on Generalization Property of Time-Varying Fbank-Weighted MFCC for I-Vector Based Speaker Verification The 9th International Symposium on Chinese Spoken Language Processing 2014 IEEE 423\u2013423.","DOI":"10.1109\/ISCSLP.2014.6936667"},{"key":"e_1_2_12_3_2","doi-asserted-by":"crossref","unstructured":"DongL. XuS. andXuB. Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) 2018 IEEE 5884\u20135888.","DOI":"10.1109\/ICASSP.2018.8462506"},{"key":"e_1_2_12_4_2","doi-asserted-by":"crossref","unstructured":"TsaiY. H. H. BaiS. LiangP. P. KolterJ. Z. MorencyL. P. andSalakhutdinovR. Multimodal Transformer for Unaligned Multimodal Language Sequences Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics 2019 Association for Computational Linguistics 6558\u20136569.","DOI":"10.18653\/v1\/P19-1656"},{"key":"e_1_2_12_5_2","unstructured":"GulatiA. QinJ. andChiuC. C. et al.Conformer: Convolution-Augmented Transformer for Speech Recognition 2020 arXiv preprint arXiv: 2005.08100."},{"key":"e_1_2_12_6_2","unstructured":"PengY. DalmiaS. LaneI. andWatanabeS. Branchformer: Parallel MPL-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding Proceedings of the 39th International Conference on Machine Learning 2022 Association for Computational Linguistics 17627\u201317643."},{"key":"e_1_2_12_7_2","doi-asserted-by":"crossref","unstructured":"SarmaM. GhahremaniP. PoveyD. GoelN. K. SarmaK. K. andDehakN. Emotion Identification From Raw Speech Signals Using DNNs Proceedings of Interspeech 2018 ISCA Archive 3097\u20133101.","DOI":"10.21437\/Interspeech.2018-1353"},{"key":"e_1_2_12_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2018.2860246"},{"key":"e_1_2_12_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trip.2019.100087"},{"key":"e_1_2_12_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2023.106295"},{"key":"e_1_2_12_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aei.2024.103065"},{"key":"e_1_2_12_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aei.2023.102123"},{"key":"e_1_2_12_13_2","doi-asserted-by":"crossref","unstructured":"HeK. ZhangX. RenS. andSunJ. Deep Residual Learning for Image Recognition IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 IEEE 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_12_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2019.01.008"},{"key":"e_1_2_12_15_2","doi-asserted-by":"crossref","unstructured":"DingX. ZhangX. MaN. HanJ. DingG. andSunJ. RepVGG: Making VGG-Style Convnets Great Again Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition 2021 IEEE 13728\u201313737.","DOI":"10.1109\/CVPR46437.2021.01352"},{"key":"e_1_2_12_16_2","unstructured":"HowardA. G. ZhuM. andChenB. et al.Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications 2017 arXiv preprint arXiv: 1704.04861."},{"key":"e_1_2_12_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3046715"},{"key":"e_1_2_12_18_2","doi-asserted-by":"publisher","DOI":"10.38094\/jastt20291"},{"key":"e_1_2_12_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2019.102447"},{"key":"e_1_2_12_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-77380-3_81"},{"key":"e_1_2_12_21_2","unstructured":"EphratA. MosseriI. andLangO. et al.Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation 2018 arXiv preprint arXiv: 1804.03619."},{"key":"e_1_2_12_22_2","doi-asserted-by":"crossref","unstructured":"NojavanasghariB. GopinathD. KoushikJ. Baltru\u0161aitisT. andMorencyL. Deep Multimodal Fusion for Persuasiveness Prediction Proceedings of the 18th ACM International Conference on Multimodal Interaction 2016 ACM 284\u2013288.","DOI":"10.1145\/2993148.2993176"},{"key":"e_1_2_12_23_2","doi-asserted-by":"crossref","unstructured":"TsaiY. H. H. BaiS. LiangP. P. KolterJ. Z. MorencyL. P. andSalakhutdinovR. Multimodal Transformer for Unaligned Multimodal Language Sequences Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019 Association for Computational Linguistics 6558\u20136569.","DOI":"10.18653\/v1\/P19-1656"},{"key":"e_1_2_12_24_2","doi-asserted-by":"crossref","unstructured":"MaiS. HuH. andXingS. Divide Conquer and Combine: Hierarchical Feature Fusion Network With Local and Global Perspectives for Multimodal Affective Computing Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019 Association for Computational Linguistics 481\u2013492.","DOI":"10.18653\/v1\/P19-1046"},{"key":"e_1_2_12_25_2","unstructured":"ChaoL. TaoJ. YangM. LiY. andWenZ. Audio Visual Emotion Recognition With Temporal Alignment and Perception Attention 2016 arXiv preprint arXiv: 1603.08321."},{"key":"e_1_2_12_26_2","unstructured":"XuH. ZhangH. HanK. WangY. PengY. andLiX. Learning Alignment for Multimodal Emotion Recognition From Speech 2019 arXiv preprint arXiv: 1909.05645."},{"key":"e_1_2_12_27_2","doi-asserted-by":"crossref","unstructured":"YoonS. ByunS. DeyS. andJungK. Speech Emotion Recognition Using Multi-Hop Attention Mechanism IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) 2019 IEEE 2822\u20132826.","DOI":"10.1109\/ICASSP.2019.8683483"},{"key":"e_1_2_12_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIE.2023.3288182"},{"key":"e_1_2_12_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2024.112370"},{"key":"e_1_2_12_30_2","doi-asserted-by":"publisher","DOI":"10.1049\/bme2.12113"},{"key":"e_1_2_12_31_2","doi-asserted-by":"crossref","unstructured":"HuangJ. TaoJ. LiuB. LianZ. andNiuM. Multimodal Transformer Fusion for Continuous Emotion Recognition Proceedings of the 2020 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) 2020 IEEE 3507\u20133511.","DOI":"10.1109\/ICASSP40776.2020.9053762"},{"key":"e_1_2_12_32_2","doi-asserted-by":"crossref","unstructured":"YangK. XuH. andGaoK. Cross-Modal BERT for Text-Audio Sentiment Analysis Proceedings of the 28th ACM International Conference on Multimedia 2020 ACM 521\u2013528.","DOI":"10.1145\/3394171.3413690"}],"container-title":["IET Biometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/bme2\/7626919","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/full-xml\/10.1049\/bme2\/7626919","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/bme2\/7626919","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,8]],"date-time":"2026-03-08T20:19:21Z","timestamp":1773001161000},"score":1,"resource":{"primary":{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/10.1049\/bme2\/7626919"}},"subtitle":[],"editor":[{"given":"Junlin","family":"Hu","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,1]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10.1049\/bme2\/7626919"],"URL":"https:\/\/doi.org\/10.1049\/bme2\/7626919","archive":["Portico"],"relation":{},"ISSN":["2047-4938","2047-4946"],"issn-type":[{"value":"2047-4938","type":"print"},{"value":"2047-4946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1]]},"assertion":[{"value":"2024-12-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"7626919"}}