{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T06:22:26Z","timestamp":1781763746414,"version":"3.54.5"},"reference-count":44,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T00:00:00Z","timestamp":1617062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mental stress can lead to traffic accidents by reducing a driver\u2019s concentration or increasing fatigue while driving. In recent years, demand for methods to detect drivers\u2019 stress in advance to prevent dangerous situations increased. Thus, we propose a novel method for detecting driving stress using nonlinear representations of short-term (30 s or less) physiological signals for multimodal convolutional neural networks (CNNs). Specifically, from hand\/foot galvanic skin response (HGSR, FGSR) and heart rate (HR) short-term input signals, first, we generate corresponding two-dimensional nonlinear representations called continuous recurrence plots (Cont-RPs). Second, from the Cont-RPs, we use multimodal CNNs to automatically extract FGSR, HGSR, and HR signal representative features that can effectively differentiate between stressed and relaxed states. Lastly, we concatenate the three extracted features into one integrated representation vector, which we feed to a fully connected layer to perform classification. For the evaluation, we use a public stress dataset collected from actual driving environments. Experimental results show that the proposed method demonstrates superior performance for 30-s signals, with an overall accuracy of 95.67%, an approximately 2.5\u20133% improvement compared with that of previous works. Additionally, for 10-s signals, the proposed method achieves 92.33% classification accuracy, which is similar to or better than the performance of other methods using long-term signals (over 100 s).<\/jats:p>","DOI":"10.3390\/s21072381","type":"journal-article","created":{"date-parts":[[2021,3,31]],"date-time":"2021-03-31T00:13:10Z","timestamp":1617149590000},"page":"2381","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":50,"title":["Driving Stress Detection Using Multimodal Convolutional Neural Networks with Nonlinear Representation of Short-Term Physiological Signals"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5681-217X","authenticated-orcid":false,"given":"Jaewon","family":"Lee","sequence":"first","affiliation":[{"name":"Bio-Intelligence &amp; Data Mining Laboratory, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2028-3557","authenticated-orcid":false,"given":"Hyeonjeong","family":"Lee","sequence":"additional","affiliation":[{"name":"Bio-Intelligence &amp; Data Mining Laboratory, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miyoung","family":"Shin","sequence":"additional","affiliation":[{"name":"Bio-Intelligence &amp; Data Mining Laboratory, School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1038\/nrcardio.2017.189","article-title":"Effects of stress on the development and progression of cardiovascular disease","volume":"15","author":"Steptoe","year":"2018","journal-title":"Nat. Rev. Cardiol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"105408","DOI":"10.1016\/j.cmpb.2020.105408","article-title":"Towards an anxiety and stress recognition system for academic environments based on physiological features","volume":"190","year":"2020","journal-title":"Comput. Meth. Prog. Bio."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1049\/htl.2019.0001","article-title":"Influence of mental stress on the pulse wave features of photoplethysmograms","volume":"7","author":"Celka","year":"2020","journal-title":"Healthc. Technol. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"83","DOI":"10.3324\/haematol.2018.211391","article-title":"Mental stress causes vasoconstriction in subjects with sickle cell disease and in normal controls","volume":"105","author":"Shah","year":"2020","journal-title":"Haematologica"},{"key":"ref_5","unstructured":"American Psychological Association (2017). Stress in America: The State of Our Nation, American Psychological Association. Stress in America Survey."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2793","DOI":"10.1007\/s00500-020-05338-0","article-title":"Toward soft real-time stress detection using wrist-worn devices for human workspaces","volume":"25","author":"Khowaja","year":"2020","journal-title":"Soft Comput."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3186585","article-title":"A critical review of proactive detection of driver stress levels based on multimodal measurements","volume":"51","author":"Rastgoo","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Munla, N., Khalil, M., Shahin, A., and Mourad, A. (2015, January 16\u201318). Driver stress level detection using HRV analysis. Proceedings of the 2015 International Conference on Advances in Biomedical Engineering (ICABME), Beirut, Lebanon.","DOI":"10.1109\/ICABME.2015.7323251"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1802","DOI":"10.1109\/TITS.2016.2618424","article-title":"Predicting Upcoming Values of Stress While Driving","volume":"18","year":"2017","journal-title":"IEEE T. Intell. Transp. Syst."},{"key":"ref_10","first-page":"1505","article-title":"How the autonomic nervous system and driving style change with incremental stressing conditions during simulated driving","volume":"16","author":"Valenza","year":"2014","journal-title":"IEEE T. Intell. Transp. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"112793","DOI":"10.1016\/j.eswa.2019.07.010","article-title":"Automatic driver stress level classification using multimodal deep learning","volume":"138","author":"Rastgoo","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1495","DOI":"10.1049\/el.2016.1393","article-title":"Driver state estimation by convolutional neural network using multimodal sensor data","volume":"52","author":"Lim","year":"2016","journal-title":"Electron. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gao, H., Y\u00fcce, A., and Thiran, J.P. (2014, January 27\u201330). Detecting emotional stress from facial expressions for driving safety. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7026203"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"74118","DOI":"10.1109\/ACCESS.2020.2988348","article-title":"A Low-Cost, Portable Solution for Stress and Relaxation Estimation Based on a Real-Time Fuzzy Algorithm","volume":"8","author":"Zalabarria","year":"2020","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, J., Mei, X., Liu, H., Yuan, S., and Qian, T. (2019, January 19\u201321). Detecting Negative Emotional Stress Based on Facial Expression in Real Time. Proceedings of the 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), Wuxi, China.","DOI":"10.1109\/SIPROCESS.2019.8868735"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Soman, K., Alex, V., and Srinivas, C. (2013, January 22\u201323). Analysis of physiological signals in response to stress using ECG and respiratory signals of automobile drivers. Proceedings of the 2013 International Multi-Conference on Automation, Computing, Communication, Control and Compressed Sensing (iMac4s), Kottayam, India.","DOI":"10.1109\/iMac4s.2013.6526476"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/MCE.2016.2590178","article-title":"A survey of affective computing for stress detection: Evaluating technologies in stress detection for better health","volume":"5","author":"Greene","year":"2016","journal-title":"IEEE Consum. Electr. Mag."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"34362","DOI":"10.1109\/ACCESS.2020.2974933","article-title":"Machine Learning Ranks ECG as an Optimal Wearable Biosignal for Assessing Driving Stress","volume":"8","author":"Elgendi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_19","first-page":"1","article-title":"The Body and the Brain: Measuring Skin Conductance Responses to Understand the Emotional Experience","volume":"22","author":"Christopoulos","year":"2016","journal-title":"Organ. Res. Methods"},{"key":"ref_20","unstructured":"Aqajari, S.A.H., Naeini, E.K., Mehrabadi, M.A., Labbaf, S., Rahmani, A.M., and Dutt, N. (2020). GSR Analysis for Stress: Development and Validation of an Open Source Tool for Noisy Naturalistic GSR Data. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3361562","article-title":"Continuous detection of physiological stress with commodity hardware","volume":"1","author":"Mishra","year":"2020","journal-title":"ACM Trans. Comput. Healthc."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zontone, P., Affanni, A., Bernardini, R., Piras, A., and Rinaldo, R. (2019, January 2\u20136). Stress detection through electrodermal activity (EDA) and electrocardiogram (ECG) analysis in car drivers. Proceedings of the 2019 27th European Signal Processing Conference (EUSIPCO), A Coruna, Spain.","DOI":"10.23919\/EUSIPCO.2019.8902631"},{"key":"ref_23","unstructured":"Guardiola, S., Girb\u00e9s, V., Armesto, L., Dols, J., and Tornero, J. (2021, January 25). Physiological Signal Analysis for Driver Stress Detection. Available online: https:\/\/www.researchgate.net\/publication\/320244776_PHYSIOLOGICAL_SIGNAL_ANALYSIS_FOR_DRDRIV_STRESS_DETECTION."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s42452-020-04134-7","article-title":"Stress level classification using statistical analysis of skin conductance signal while driving","volume":"3","author":"Memar","year":"2021","journal-title":"SN Appl. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bianco, S., Napoletano, P., and Schettini, R. (2019, January 20\u201323). Multimodal car driver stress recognition. Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare, Trento, Italy.","DOI":"10.1145\/3329189.3329221"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.eswa.2017.01.040","article-title":"Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers","volume":"85","author":"Chen","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Seo, W., Kim, N., Kim, S., Lee, C., and Park, S.M. (2019). Deep ECG-respiration network (DeepER net) for recognizing mental stress. Sensors, 19.","DOI":"10.3390\/s19133021"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"\u0160alkevicius, J., Dama\u0161evi\u010dius, R., Maskeliunas, R., and Laukien\u0117, I. (2019). Anxiety level recognition for virtual reality therapy system using physiological signals. Electronics, 8.","DOI":"10.3390\/electronics8091039"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1016\/j.bspc.2015.02.012","article-title":"Acute mental stress assessment via short term HRV analysis in healthy adults: A systematic review with meta-analysis","volume":"18","author":"Castaldo","year":"2015","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Jim\u00e9nez-Limas, M.A., Ram\u00edrez-Fuentes, C.A., Tovar-Corona, B., and Garay-Jim\u00e9nez, L.I. (2018, January 5\u20137). Feature selection for stress level classification into a physiologycal signals set. Proceedings of the 2018 15th International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE), Mexico City, Mexico.","DOI":"10.1109\/ICEEE.2018.8533968"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1016\/j.neucom.2011.10.047","article-title":"A k-nearest-neighbor classifier with heart rate variability feature-based transformation algorithm for driving stress recognition","volume":"116","author":"Wang","year":"2013","journal-title":"Neurocomputing"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ghaderi, A., Frounchi, J., and Farnam, A. (2015, January 25\u201327). Machine learning-based signal processing using physiological signals for stress detection. Proceedings of the 2015 22nd Iranian Conference on Biomedical Engineering (ICBME), Tehran, Iran.","DOI":"10.1109\/ICBME.2015.7404123"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"026702","DOI":"10.1103\/PhysRevE.66.026702","article-title":"Recurrence-plot-based measures of complexity and their application to heart-rate-variability data","volume":"66","author":"Marwan","year":"2002","journal-title":"Phys. Rev. E"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1007\/s11517-018-1914-0","article-title":"A unified non-linear approach based on recurrence quantification analysis and approximate entropy: Application to the classification of heart rate variability of age-stratified subjects","volume":"57","author":"Singh","year":"2019","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"40","DOI":"10.3389\/fphys.2020.00040","article-title":"Recurrence Quantification Analysis of Heart Rate during Mental Arithmetic Stress in Young Females","volume":"11","author":"Dimitriev","year":"2020","journal-title":"Front. Physiol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/j.physrep.2006.11.001","article-title":"Recurrence plots for the analysis of complex systems","volume":"438","author":"Marwan","year":"2007","journal-title":"Phys. Rep."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting stress during real-world driving tasks using physiological sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE T. Intell. Transp. Syst."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1089\/tmj.2017.0250","article-title":"Deep ECGNet: An optimal deep learning framework for monitoring mental stress using ultra short-term ECG signals","volume":"24","author":"Hwang","year":"2018","journal-title":"Telemed. J. E-Health"},{"key":"ref_39","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wang, K., Murphey, Y.L., Zhou, Y., Hu, X., and Zhang, X. (2019, January 22\u201325). Detection of driver stress in real-world driving environment using physiological signals. Proceedings of the 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Helsinki, Finland.","DOI":"10.1109\/INDIN41052.2019.8972264"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lopez-Martinez, D., El-Haouij, N., and Picard, R. (2019, January 3\u20136). Detection of Real-world Driving-induced Affective State Using Physiological Signals and Multi-view Multi-task Machine Learning. Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), Cambridge, UK.","DOI":"10.1109\/ACIIW.2019.8925190"},{"key":"ref_42","unstructured":"Wang, K., and Guo, P. (2020). An Ensemble Classification Model with Unsupervised Representation Learning for Driving Stress Recognition Using Physiological Signals. IEEE T. Intell. Transp. Syst., 1\u201313."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1016\/j.bspc.2013.06.014","article-title":"A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals","volume":"8","author":"Singh","year":"2013","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_44","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2381\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:10:26Z","timestamp":1760364626000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/7\/2381"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,30]]},"references-count":44,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21072381"],"URL":"https:\/\/doi.org\/10.3390\/s21072381","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,30]]}}}