{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T15:52:53Z","timestamp":1772725973447,"version":"3.50.1"},"reference-count":46,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T00:00:00Z","timestamp":1672185600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["61762079"],"award-info":[{"award-number":["61762079"]}]},{"name":"National Natural Science Foundation of China","award":["61662070"],"award-info":[{"award-number":["61662070"]}]},{"name":"National Natural Science Foundation of China","award":["62162056"],"award-info":[{"award-number":["62162056"]}]},{"name":"National Natural Science Foundation of China","award":["2021CYZC06"],"award-info":[{"award-number":["2021CYZC06"]}]},{"name":"National Natural Science Foundation of China","award":["1604FKCA097"],"award-info":[{"award-number":["1604FKCA097"]}]},{"name":"National Natural Science Foundation of China","award":["17YF1GA015"],"award-info":[{"award-number":["17YF1GA015"]}]},{"name":"National Natural Science Foundation of China","award":["20YF8GA048"],"award-info":[{"award-number":["20YF8GA048"]}]},{"name":"Industrial Support Foundations of Gansu","award":["61762079"],"award-info":[{"award-number":["61762079"]}]},{"name":"Industrial Support Foundations of Gansu","award":["61662070"],"award-info":[{"award-number":["61662070"]}]},{"name":"Industrial Support Foundations of Gansu","award":["62162056"],"award-info":[{"award-number":["62162056"]}]},{"name":"Industrial Support Foundations of Gansu","award":["2021CYZC06"],"award-info":[{"award-number":["2021CYZC06"]}]},{"name":"Industrial Support Foundations of Gansu","award":["1604FKCA097"],"award-info":[{"award-number":["1604FKCA097"]}]},{"name":"Industrial Support Foundations of Gansu","award":["17YF1GA015"],"award-info":[{"award-number":["17YF1GA015"]}]},{"name":"Industrial Support Foundations of Gansu","award":["20YF8GA048"],"award-info":[{"award-number":["20YF8GA048"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["61762079"],"award-info":[{"award-number":["61762079"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["61662070"],"award-info":[{"award-number":["61662070"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["62162056"],"award-info":[{"award-number":["62162056"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["2021CYZC06"],"award-info":[{"award-number":["2021CYZC06"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["1604FKCA097"],"award-info":[{"award-number":["1604FKCA097"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["17YF1GA015"],"award-info":[{"award-number":["17YF1GA015"]}]},{"name":"Key Science and Technology Support Program of Gansu Province","award":["20YF8GA048"],"award-info":[{"award-number":["20YF8GA048"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Emotions significantly impact human physical and mental health, and, therefore, emotion recognition has been a popular research area in neuroscience, psychology, and medicine. In this paper, we preprocess the raw signals acquired by millimeter-wave radar to obtain high-quality heartbeat and respiration signals. Then, we propose a deep learning model incorporating a convolutional neural network and gated recurrent unit neural network in combination with human face expression images. The model achieves a recognition accuracy of 84.5% in person-dependent experiments and 74.25% in person-independent experiments. The experiments show that it outperforms a single deep learning model compared to traditional machine learning algorithms.<\/jats:p>","DOI":"10.3390\/s23010338","type":"journal-article","created":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T02:54:42Z","timestamp":1672282482000},"page":"338","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Wireless Sensing Technology Combined with Facial Expression to Realize Multimodal Emotion Recognition"],"prefix":"10.3390","volume":"23","author":[{"given":"Xiaochao","family":"Dang","sequence":"first","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zetong","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9740-0988","authenticated-orcid":false,"given":"Zhanjun","family":"Hao","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Macidan","family":"Ga","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinyu","family":"Han","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaotong","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1007\/s11920-014-0531-3","article-title":"The Role of Emotion and Emotion Regulation in Social Anxiety Disorder","volume":"17","author":"Jazaieri","year":"2015","journal-title":"Curr. Psychiatry Rep."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"11148","DOI":"10.1073\/pnas.1534743100","article-title":"Affective style and in vivo immune response: Neurobehavioral mechanisms","volume":"100","author":"Rosenkranz","year":"2003","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1386","DOI":"10.1177\/0956797620953883","article-title":"Positive Affect Is Associated with Less Memory Decline: Evidence from a 9-Year Longitudinal Study","volume":"31","author":"Hittner","year":"2020","journal-title":"Psychol. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"100306","DOI":"10.1016\/j.iot.2020.100306","article-title":"A ZigBee Intrusion Detection System for IoT using Secure and Efficient Data Collection","volume":"12","author":"Sadikin","year":"2020","journal-title":"Internet Things"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2789","DOI":"10.1109\/TMC.2020.3045635","article-title":"Enabling Fine-Grained Finger Gesture Recognition on Commodity WiFi Devices","volume":"21","author":"Tan","year":"2022","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/j.bspc.2016.12.008","article-title":"Novel signal processing method for vital sign monitoring using FMCW radar","volume":"33","author":"He","year":"2017","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3133","DOI":"10.1109\/JIOT.2020.2965115","article-title":"Kalman-Filter-Based Integration of IMU and UWB for High-Accuracy Indoor Positioning and Navigation","volume":"7","author":"Feng","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5633","DOI":"10.1109\/TGRS.2013.2291573","article-title":"Short-Range Wideband FMCW Radar for Millimetric Displacement Measurements","volume":"52","author":"Anghel","year":"2014","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Qu, K., Zhang, R.F., and Fang, Z.J. (2022). High Accuracy Motion Detection Algorithm via ISM Band FMCW Radar. Remote Sens., 14.","DOI":"10.3390\/rs14010058"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"10808","DOI":"10.1109\/JSEN.2022.3169231","article-title":"ML-HGR-Net: A Meta-Learning Network for FMCW Radar Based Hand Gesture Recognition","volume":"22","author":"Shen","year":"2022","journal-title":"IEEE Sens. J."},{"key":"ref_11","first-page":"374","article-title":"Emotions Recognition Using EEG Signals: A Survey","volume":"10","author":"Fonseca","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1126\/science.22.572.789","article-title":"Principles of physiological psychology","volume":"22","author":"Woodworth","year":"1905","journal-title":"Science"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1017\/S0954579405050340","article-title":"The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology","volume":"17","author":"Posner","year":"2005","journal-title":"Dev. Psychopathol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"104757","DOI":"10.1016\/j.compbiomed.2021.104757","article-title":"EEG Channel Correlation Based Model for Emotion Recognition","volume":"136","author":"Mri","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"94601","DOI":"10.1109\/ACCESS.2021.3091487","article-title":"Emotion Recognition from EEG Signal Focusing on Deep Learning and Shallow Learning Techniques","volume":"9","author":"Islam","year":"2021","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"865","DOI":"10.18280\/ts.380333","article-title":"Online Learning Behavior Analysis Based on Image Emotion Recognition","volume":"38","author":"Wang","year":"2021","journal-title":"Trait. Signal"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1098\/rsos.211412","article-title":"Associations between vocal emotion recognition and socio-emotional adjustment in children","volume":"8","author":"Neves","year":"2021","journal-title":"R. Soc. Open Sci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1016\/j.ins.2021.10.005","article-title":"A survey on facial emotion recognition techniques: A state-of-the-art literature review","volume":"582","author":"Canal","year":"2022","journal-title":"Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hasnul, M.A., Aziz, N.A.A.A., Alelyani, S., Mohana, M., and Aziz, A.A.A. (2021). Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare\u2014A Review. Sensors, 21.","DOI":"10.3390\/s21155015"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"162197","DOI":"10.1109\/ACCESS.2021.3132233","article-title":"A Review of Emotion Recognition Methods From Keystroke, Mouse, and Touchscreen Dynamics","volume":"9","author":"Yang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1109\/TAFFC.2018.2874996","article-title":"Automatic Recognition of Facial Displays of Unfelt Emotions","volume":"12","author":"Kulkarni","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_23","first-page":"9","article-title":"Touch Gesture and Emotion Recognition Using Decomposed Spatiotemporal Convolutions","volume":"71","author":"Li","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.patcog.2021.107868","article-title":"Multi-task learning for gait-based identity recognition and emotion recognition using attention enhanced temporal graph convolutional network","volume":"114","author":"Sheng","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1803","DOI":"10.1109\/TASLP.2022.3171965","article-title":"ISNet: Individual Standardization Network for Speech Emotion Recognition","volume":"30","author":"Fan","year":"2022","journal-title":"IEEE-ACM Trans. Audio Speech Lang. Process."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1109\/TAFFC.2019.2901673","article-title":"Feature Extraction and Selection for Emotion Recognition from Electrodermal Activity","volume":"12","author":"Shukla","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TAFFC.2014.2339834","article-title":"Feature Extraction and Selection for Emotion Recognition from EEG","volume":"5","author":"Jenke","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review","volume":"59","author":"Jz","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s00521-012-0814-8","article-title":"Modular neural-SVM scheme for speech emotion recognition using ANOVA feature selection method","volume":"23","author":"Sheikhan","year":"2013","journal-title":"Neural Comput. Appl."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tiwari, P., Rathod, H., Thakkar, S., and Darji, A.D. (2021). Multimodal emotion recognition using SDA-LDA algorithm in video clips. J. Ambient. Intell. Humaniz. Comput.","DOI":"10.1007\/s12652-021-03529-7"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1109\/TCDS.2020.2976112","article-title":"A Channel-fused Dense Convolutional Network for EEG-based Emotion Recognition","volume":"13","author":"Gao","year":"2020","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"117327","DOI":"10.1109\/ACCESS.2019.2936124","article-title":"Speech Emotion Recognition Using Deep Learning Techniques: A Review","volume":"7","author":"Khalil","year":"2019","journal-title":"IEEE Access"},{"key":"ref_33","first-page":"114177","article-title":"MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach","volume":"167","author":"Kwon","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1259","DOI":"10.1002\/spe.2955","article-title":"Modified deep belief network based human emotion recognition with multiscale features from video sequences","volume":"51","author":"Sreenivas","year":"2021","journal-title":"Softw.-Pract. Exp."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2812","DOI":"10.1109\/TMTT.2014.2358572","article-title":"A Hybrid FMCW-Interferometry Radar for Indoor Precise Positioning and Versatile Life Activity Monitoring","volume":"62","author":"Wang","year":"2014","journal-title":"IEEE Trans. Microw. Theory Tech."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3278","DOI":"10.1109\/JSEN.2018.2808688","article-title":"Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor","volume":"18","author":"Zhang","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1109\/JSEN.2019.2946095","article-title":"Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection","volume":"20","author":"Li","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"54958","DOI":"10.1109\/ACCESS.2019.2912956","article-title":"Remote Monitoring of Human Vital Signs Using mm-Wave FMCW Radar","volume":"7","author":"Alizadeh","year":"2019","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1109\/TMTT.2016.2633352","article-title":"A Portable FMCW Interferometry Radar with Programmable Low-IF Architecture for Localization, ISAR Imaging, and Vital Sign Tracking","volume":"65","author":"Peng","year":"2017","journal-title":"IEEE Trans. Microw. Theory Tech."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1796","DOI":"10.1049\/rsn2.12297","article-title":"Emotion recognition method using millimetre wave radar based on deep learning","volume":"16","author":"Dang","year":"2022","journal-title":"IET Radar Sonar Navig."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1109\/TSP.2013.2288675","article-title":"Variational Mode Decomposition","volume":"62","author":"Dragomiretskiy","year":"2014","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_42","first-page":"9","article-title":"One-dimensional convolutional neural networks for low\/high arousal classification from electrodermal activity","volume":"71","author":"Francisco","year":"2022","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1145\/3236621","article-title":"Emotion Recognition Using Wireless Signals","volume":"61","author":"Zhao","year":"2018","journal-title":"Commun. ACM"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1109\/TETCI.2019.2902438","article-title":"EmoSense: Computational Intelligence Driven Emotion Sensing via Wireless Channel Data","volume":"4","author":"Gu","year":"2020","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Khan, A.N., Ihalage, A.A., Ma, Y., Liu, B., Liu, Y., and Hao, Y. (2021). Deep learning framework for subject-independent emotion detection using wireless signals. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0242946"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.patrec.2018.04.010","article-title":"Hybrid deep neural networks for face emotion recognition","volume":"115","author":"Jain","year":"2018","journal-title":"Pattern Recognit. Lett."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/1\/338\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:54:29Z","timestamp":1760147669000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/1\/338"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,28]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["s23010338"],"URL":"https:\/\/doi.org\/10.3390\/s23010338","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,28]]}}}