{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T12:34:57Z","timestamp":1767875697755,"version":"3.49.0"},"reference-count":77,"publisher":"World Scientific Pub Co Pte Ltd","issue":"03","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Neur. Syst."],"published-print":{"date-parts":[[2026,3]]},"abstract":"<jats:p>This study aims to develop a multimodal driver emotion recognition system that accurately identifies a driver\u2019s emotional state during the driving process by integrating facial expressions, ElectroCardioGram (ECG) and ElectroEncephaloGram (EEG) signals. Specifically, this study proposes a model that employs a Conformer for analyzing facial images to extract visual cues related to the driver\u2019s emotions. Additionally, two Autoformers are utilized to process ECG and EEG signals. The embeddings from these three modalities are then fused using a cross-attention mechanism. The integrated features from the cross-attention mechanism are passed through a fully connected layer and classified to determine the driver\u2019s emotional state. The experimental results demonstrate that the fusion of visual, physiological and neurological modalities significantly improves the reliability and accuracy of emotion detection. The proposed approach not only offers insights into the emotional processes critical for driver assistance systems and vehicle safety but also lays the foundation for further advancements in emotion recognition area.<\/jats:p>","DOI":"10.1142\/s0129065725500698","type":"journal-article","created":{"date-parts":[[2025,9,6]],"date-time":"2025-09-06T04:15:47Z","timestamp":1757132147000},"source":"Crossref","is-referenced-by-count":0,"title":["Driver Emotion Recognition Using Multimodal Signals by Combining Conformer and Autoformer"],"prefix":"10.1142","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9652-2351","authenticated-orcid":false,"given":"Weiguang","family":"Wang","sequence":"first","affiliation":[{"name":"School of Intelligence Engineering, Shandong Management University 250357, Jinan, P.\u00a0R.\u00a0China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0305-8454","authenticated-orcid":false,"given":"Jian","family":"Lian","sequence":"additional","affiliation":[{"name":"School of Intelligence Engineering, Shandong Management University 250357, Jinan, P.\u00a0R.\u00a0China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4025-0154","authenticated-orcid":false,"given":"Chuanjie","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Intelligence Engineering, Shandong Management University 250357, Jinan, P.\u00a0R.\u00a0China"}]}],"member":"219","published-online":{"date-parts":[[2025,9,25]]},"reference":[{"key":"S0129065725500698BIB001","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2024.110178"},{"key":"S0129065725500698BIB002","doi-asserted-by":"publisher","DOI":"10.1109\/IC3SE62002.2024.10593521"},{"key":"S0129065725500698BIB003","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-658-40232-7_29"},{"key":"S0129065725500698BIB004","doi-asserted-by":"publisher","DOI":"10.1109\/CoBCom62281.2024.10631203"},{"issue":"5","key":"S0129065725500698BIB005","first-page":"867","volume":"11","author":"Chen Y.","year":"2024","journal-title":"J. Traffic Transp. Eng."},{"key":"S0129065725500698BIB006","doi-asserted-by":"publisher","DOI":"10.1007\/s42154-023-00270-z"},{"key":"S0129065725500698BIB007","doi-asserted-by":"publisher","DOI":"10.1016\/j.trf.2018.08.012"},{"key":"S0129065725500698BIB008","doi-asserted-by":"publisher","DOI":"10.1016\/j.tbs.2017.02.001"},{"key":"S0129065725500698BIB009","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2015.10.021"},{"key":"S0129065725500698BIB010","doi-asserted-by":"publisher","DOI":"10.1016\/j.trf.2017.07.002"},{"key":"S0129065725500698BIB011","doi-asserted-by":"publisher","DOI":"10.1016\/j.tbs.2024.100937"},{"key":"S0129065725500698BIB012","doi-asserted-by":"publisher","DOI":"10.1016\/j.comcom.2022.07.031"},{"key":"S0129065725500698BIB013","doi-asserted-by":"publisher","DOI":"10.1109\/ICSSIT.2018.8748541"},{"key":"S0129065725500698BIB014","doi-asserted-by":"crossref","unstructured":"Z. Kowalczuk, M. Czubenko and T. Merta, Emotion monitoring system for drivers,\n                      IFAC-PapersOnLine\n                      \u00a0\n                      52\n                      (8) (2019) 200\u2013205.","DOI":"10.1016\/j.ifacol.2019.08.071"},{"key":"S0129065725500698BIB015","doi-asserted-by":"crossref","unstructured":"K. Bengler, K. Dietmayer, B. Farber, M. Maurer, C. Stiller and H. Winner, Three decades of driver assistance systems: Review and future perspectives,\n                      Intell. Transp. Syst. Mag.\n                      \u00a0\n                      6\n                      (2014) 6\u201322.","DOI":"10.1109\/MITS.2014.2336271"},{"key":"S0129065725500698BIB016","doi-asserted-by":"publisher","DOI":"10.7708\/ijtte.2018.8(3).06"},{"key":"S0129065725500698BIB017","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.122784"},{"key":"S0129065725500698BIB018","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.115036"},{"key":"S0129065725500698BIB019","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2018.09.002"},{"key":"S0129065725500698BIB020","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-024-20428-z"},{"key":"S0129065725500698BIB021","first-page":"1","volume-title":"Int. Conf. Computer Vision and Machine Intelligence","author":"Saadi I.","year":"2023"},{"key":"S0129065725500698BIB022","doi-asserted-by":"publisher","DOI":"10.1016\/j.measen.2023.100711"},{"key":"S0129065725500698BIB023","first-page":"667","volume":"2015","author":"Hildebrandt A.","year":"2015","journal-title":"Int. Encycl. Social Behav. Sci."},{"key":"S0129065725500698BIB024","doi-asserted-by":"publisher","DOI":"10.1016\/j.jth.2017.11.001"},{"key":"S0129065725500698BIB025","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC57777.2023.10422309"},{"key":"S0129065725500698BIB026","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2024.e26171"},{"key":"S0129065725500698BIB027","doi-asserted-by":"publisher","DOI":"10.1016\/j.proeng.2012.06.298"},{"key":"S0129065725500698BIB028","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065722500241"},{"key":"S0129065725500698BIB029","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2023.104989"},{"key":"S0129065725500698BIB030","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065724500278"},{"key":"S0129065725500698BIB031","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065724500242"},{"key":"S0129065725500698BIB032","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065725500029"},{"key":"S0129065725500698BIB033","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3243048"},{"key":"S0129065725500698BIB034","doi-asserted-by":"crossref","unstructured":"A. Gulati, J. Qin, C.C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu and R. Pang, Conformer: Convolution-augmented transformer for speech recognition, preprint (2020), arXiv:2005. 08100.","DOI":"10.21437\/Interspeech.2020-3015"},{"key":"S0129065725500698BIB035","doi-asserted-by":"crossref","unstructured":"H. Lin, X. Cheng, X. Wu, F. Yang, D. Shen, Z. Wang, Q. Song and W. Yuan, Cat: Cross attention in vision transformer, preprint (2021), arXiv:2106.05786.","DOI":"10.1109\/ICME52920.2022.9859720"},{"key":"S0129065725500698BIB036","doi-asserted-by":"publisher","DOI":"10.1109\/ICOASE.2019.8723728"},{"key":"S0129065725500698BIB037","doi-asserted-by":"publisher","DOI":"10.3390\/s18124270"},{"key":"S0129065725500698BIB038","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-99-8761-0_9"},{"key":"S0129065725500698BIB039","doi-asserted-by":"publisher","DOI":"10.1016\/j.measen.2024.101186"},{"key":"S0129065725500698BIB040","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2020.2981446"},{"key":"S0129065725500698BIB041","doi-asserted-by":"publisher","DOI":"10.1109\/ACPR.2015.7486599"},{"key":"S0129065725500698BIB042","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"S0129065725500698BIB043","doi-asserted-by":"publisher","DOI":"10.1162\/neco_a_01199"},{"key":"S0129065725500698BIB044","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3152247"},{"key":"S0129065725500698BIB045","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-023-01100-9"},{"key":"S0129065725500698BIB046","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2024.121658"},{"key":"S0129065725500698BIB047","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106981"},{"key":"S0129065725500698BIB048","first-page":"1","volume-title":"IEEE 17th India Council Int. Conf. (INDICON)","author":"Murthy Y. Srinivasa","year":"2020"},{"key":"S0129065725500698BIB049","doi-asserted-by":"publisher","DOI":"10.1109\/DTPI52967.2021.9540071"},{"key":"S0129065725500698BIB050","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2015.7364066"},{"key":"S0129065725500698BIB051","doi-asserted-by":"publisher","DOI":"10.1007\/s13246-020-00853-8"},{"key":"S0129065725500698BIB052","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3158658"},{"key":"S0129065725500698BIB053","doi-asserted-by":"publisher","DOI":"10.1111\/exsy.12494"},{"key":"S0129065725500698BIB054","doi-asserted-by":"publisher","DOI":"10.1159\/000512985"},{"key":"S0129065725500698BIB055","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbr.2015.10.036"},{"key":"S0129065725500698BIB056","first-page":"89","volume-title":"IEEE 13th Int. Conf. Industrial and Information Systems","author":"Bhardwaj R.","year":"2018"},{"key":"S0129065725500698BIB057","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2021.3090217"},{"key":"S0129065725500698BIB058","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2024.106204"},{"key":"S0129065725500698BIB059","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2024.106881"},{"key":"S0129065725500698BIB060","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2024.112320"},{"key":"S0129065725500698BIB061","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065724500035"},{"key":"S0129065725500698BIB062","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065724500588"},{"key":"S0129065725500698BIB063","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2023.3250460"},{"key":"S0129065725500698BIB064","first-page":"200171","volume":"17","author":"Ahmed N.","year":"2023","journal-title":"Intell. Syst. Appl."},{"key":"S0129065725500698BIB065","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.107772"},{"key":"S0129065725500698BIB066","doi-asserted-by":"publisher","DOI":"10.1142\/S0129065722500496"},{"issue":"99","key":"S0129065725500698BIB067","first-page":"1","author":"Zhang X.","year":"2024","journal-title":"IEEE Trans. Intell. Vehicles"},{"key":"S0129065725500698BIB068","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-021-01249-8"},{"key":"S0129065725500698BIB069","doi-asserted-by":"publisher","DOI":"10.1109\/TBIOM.2024.3435046"},{"key":"S0129065725500698BIB070","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-024-03222-2"},{"key":"S0129065725500698BIB071","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01722"},{"key":"S0129065725500698BIB072","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW63382.2024.00042"},{"key":"S0129065725500698BIB073","doi-asserted-by":"publisher","DOI":"10.1109\/SPMB50085.2020.9353642"},{"key":"S0129065725500698BIB074","doi-asserted-by":"publisher","DOI":"10.1109\/ICTRC.2015.7156444"},{"key":"S0129065725500698BIB075","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ab909d"},{"key":"S0129065725500698BIB076","doi-asserted-by":"publisher","DOI":"10.1109\/ASPCON49795.2020.9276717"},{"key":"S0129065725500698BIB077","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC46164.2021.9630398"}],"container-title":["International Journal of Neural Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0129065725500698","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T09:25:43Z","timestamp":1767864343000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0129065725500698"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,25]]},"references-count":77,"journal-issue":{"issue":"03","published-print":{"date-parts":[[2026,3]]}},"alternative-id":["10.1142\/S0129065725500698"],"URL":"https:\/\/doi.org\/10.1142\/s0129065725500698","relation":{},"ISSN":["0129-0657","1793-6462"],"issn-type":[{"value":"0129-0657","type":"print"},{"value":"1793-6462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,25]]},"article-number":"2550069"}}