{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,7]],"date-time":"2026-01-07T08:07:38Z","timestamp":1767773258560,"version":"3.37.3"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T00:00:00Z","timestamp":1632441600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T00:00:00Z","timestamp":1632441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004885","name":"ume\u00e5 university","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004885","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Umea University"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2022,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Affective computing solutions, in the literature, mainly rely on machine learning methods designed to accurately detect human affective states. Nevertheless, many of the proposed methods are based on handcrafted features, requiring sufficient expert knowledge in the realm of signal processing. With the advent of deep learning methods, attention has turned toward reduced feature engineering and more end-to-end machine learning. However, most of the proposed models rely on late fusion in a multimodal context. Meanwhile, addressing interrelations between modalities for intermediate-level data representation has been largely neglected. In this paper, we propose a novel deep convolutional neural network, called CN-Waterfall, consisting of two modules: <jats:italic>Base<\/jats:italic> and <jats:italic>General<\/jats:italic>. While the <jats:italic>Base<\/jats:italic> module focuses on the low-level representation of data from each single modality, the <jats:italic>General<\/jats:italic> module provides further information, indicating relations between modalities in the intermediate- and high-level data representations. The latter module has been designed based on theoretically grounded concepts in the <jats:italic>Explainable AI (XAI)<\/jats:italic> domain, consisting of four different fusions. These fusions are mainly tailored to <jats:italic>correlation<\/jats:italic>- and <jats:italic>non-correlation<\/jats:italic>-based modalities. To validate our model, we conduct an exhaustive experiment on WESAD and MAHNOB-HCI, two publicly and academically available datasets in the context of multimodal affective computing. We demonstrate that our proposed model significantly improves the performance of physiological-based multimodal affect detection.<\/jats:p>","DOI":"10.1007\/s00521-021-06516-3","type":"journal-article","created":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T20:23:48Z","timestamp":1632515028000},"page":"2157-2176","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["CN-waterfall: a deep convolutional neural network for multimodal physiological affect detection"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9009-0999","authenticated-orcid":false,"given":"Nazanin","family":"Fouladgar","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marjan","family":"Alirezaie","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kary","family":"Fr\u00e4mling","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,9,24]]},"reference":[{"key":"6516_CR1","unstructured":"Picard R (1995) Affective computing. MIT Technical Report"},{"key":"6516_CR2","doi-asserted-by":"crossref","unstructured":"Giannakakis G, Grigoriadis D, Giannakaki K, Olympia S, Alexandros R, Manolis T (2019) Review on psychological stress detection using biosignals. IEEE Trans Affect Comput","DOI":"10.1109\/TAFFC.2019.2927337"},{"key":"6516_CR3","doi-asserted-by":"crossref","unstructured":"Jing C, Liu G, Hao M (2009) The research on emotion recognition from ecg signal. In: 2009 international conference on information technology and computer science, volume\u00a01, pp 497\u2013500","DOI":"10.1109\/ITCS.2009.108"},{"issue":"1","key":"6516_CR4","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1109\/T-AFFC.2011.28","volume":"3","author":"A Foteini","year":"2012","unstructured":"Foteini A, Dimitrios H, Anderson Adam K (2012) Ecg pattern analysis for emotion detection. IEEE Trans Affect Comput 3(1):102\u2013115","journal-title":"IEEE Trans Affect Comput"},{"key":"6516_CR5","doi-asserted-by":"publisher","first-page":"06","DOI":"10.1016\/j.bioelechem.2013.01.005","volume":"16","author":"V Christiaan","year":"2013","unstructured":"Christiaan V, Renske P, Juliane H, Joris V, Klaessens John HGM, Berend O, Cor K (2013) The effect of stress on core and peripheral body temperature in humans. Stress (Amsterdam, Netherlands) 16:06","journal-title":"Stress (Amsterdam, Netherlands)"},{"key":"6516_CR6","doi-asserted-by":"publisher","first-page":"592","DOI":"10.3390\/s20030592","volume":"20","author":"D Andrius","year":"2020","unstructured":"Andrius D, Arturas K, Vytautas B (2020) Human emotion recognition: review of sensors and methods. Sensors 20:592","journal-title":"Sensors"},{"issue":"356\u201365","key":"6516_CR7","first-page":"06","volume":"45","author":"S Lawrence","year":"2008","unstructured":"Lawrence S, Thomas S, Scott K, Jeffrey C, Ronald L, Brad H (2008) Mental stress and trapezius muscle activation under psychomotor challenge: a focus on emg gaps during computer work. Psychophysiology 45(356\u201365):06","journal-title":"Psychophysiology"},{"key":"6516_CR8","doi-asserted-by":"crossref","unstructured":"Sun F-T, Kuo C, Cheng H-T, Buthpitiya S, Collins P, Griss M (2012) Activity-aware mental stress detection using physiological sensors. In: International conference on mobile computing, vol 76. applications, and services. Springer, Berlin, Heidelberg, pp 282\u2013301","DOI":"10.1007\/978-3-642-29336-8_16"},{"key":"6516_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2019.101646","volume":"55","author":"JA Dom\u00ednguez-Jim\u00e9nez","year":"2020","unstructured":"Dom\u00ednguez-Jim\u00e9nez JA, Campo-Landines KC, Mart\u00ednez-Santos JC, Delahoz EJ, Contreras-Ortiz SH (2020) A machine learning model for emotion recognition from physiological signals. Biomed Signal Process Control 55:101646","journal-title":"Biomed Signal Process Control"},{"key":"6516_CR10","doi-asserted-by":"crossref","unstructured":"Girardi D, Lanubile F, Novielli N (2017) Emotion detection using noninvasive low cost sensors. In: 2017 seventh international conference on affective computing and intelligent interaction (ACII), pp 125\u2013130","DOI":"10.1109\/ACII.2017.8273589"},{"key":"6516_CR11","doi-asserted-by":"crossref","unstructured":"Liu M, Di F, Zhang X, Gong X (2016) Human emotion recognition based on galvanic skin response signal feature selection and svm. In: 2016 international conference on smart city and systems engineering (ICSCSE), pp 157\u2013160","DOI":"10.1109\/ICSCSE.2016.0051"},{"key":"6516_CR12","unstructured":"Wang K, Guo P (2020) An ensemble classification model with unsupervised representation learning for driving stress recognition using physiological signals. In: IEEE transactions on intelligent transportation systems, pp 1\u201313"},{"key":"6516_CR13","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1007\/b98229","volume-title":"Affective dialogue systems","author":"H Andreas","year":"2004","unstructured":"Andreas H, Silke G, Peter S, Jason W (2004) Emotion recognition using bio-sensors: first steps towards an automatic system. Affective dialogue systems. Springer, Berlin, pp 36\u201348"},{"key":"6516_CR14","doi-asserted-by":"publisher","first-page":"101824","DOI":"10.1016\/j.artmed.2020.101824","volume":"104","author":"AF Patlar","year":"2020","unstructured":"Patlar AF, Baris I, Aydin A (2020) Wearable sensor-based evaluation of psychosocial stress in patients with metabolic syndrome. Artif Intell Med 104:101824","journal-title":"Artif Intell Med"},{"key":"6516_CR15","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.compbiomed.2019.02.015","volume":"107","author":"A Anum","year":"2019","unstructured":"Anum A, Muhammad M, Muhammad AS (2019) Human stress classification using eeg signals in response to music tracks. Comput Biol Med 107:182\u2013196","journal-title":"Comput Biol Med"},{"key":"6516_CR16","doi-asserted-by":"crossref","unstructured":"Schmidt P, D\u00fcrichen R, Reiss A, Van\u00a0Laerhoven K, Pl\u00f6tz T (2019) Multi-target affect detection in the wild: an exploratory study. In: Proceedings of the 23rd international symposium on wearable computers, ISWC \u201919, pp 211\u2013219","DOI":"10.1145\/3341163.3347741"},{"issue":"12","key":"6516_CR17","doi-asserted-by":"publisher","first-page":"2067","DOI":"10.1109\/TPAMI.2008.26","volume":"30","author":"K Jonghwa","year":"2008","unstructured":"Jonghwa K, Elisabeth A (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067\u20132083","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"6516_CR18","unstructured":"Pritam S, Ali E (2020) Self-supervised ecg representation learning for emotion recognition. IEEE Trans Affect Comput"},{"key":"6516_CR19","doi-asserted-by":"crossref","unstructured":"Chakraborty S, Aich S, Joo M,, Sain M, Kim H-C (2019) A multichannel convolutional neural network architecture for the detection of the state of mind using physiological signals from wearable devices. J Healthc Eng","DOI":"10.1155\/2019\/5397814"},{"key":"6516_CR20","doi-asserted-by":"crossref","unstructured":"Ragav A, Krishna NH, Narayanan N, Thelly K, Vijayaraghavan V (2019) Scalable deep learning for stress and affect detection on resource-constrained devices. In: 18th IEEE international conference on machine learning and applications (ICMLA), pp 1585\u20131592","DOI":"10.1109\/ICMLA.2019.00261"},{"key":"6516_CR21","doi-asserted-by":"crossref","unstructured":"Lin J, Pan S, Lee CS, Oviatt S (2019) An explainable deep fusion network for affect recognition using physiological signals. In: Proceedings of the 28th ACM international conference on information and knowledge management, CIKM \u201919, pp 2069\u20132072. Association for Computing Machinery","DOI":"10.1145\/3357384.3358160"},{"key":"6516_CR22","unstructured":"Fr\u00e4mling K (1996) Explaining results of neural networks by contextual importance and utility. In: Proceedings of the AISB\u201996 conference, Brighton, UK, 1\u20132"},{"key":"6516_CR23","doi-asserted-by":"crossref","unstructured":"Fouladgar N, Alirezaie M, Fr\u00e4mling K (2020) Decision explanation: applying contextual importance and contextual utility in affect detection. In: Italian workshop on explainable artificiale intelligence (AI*AI2020)","DOI":"10.1007\/978-3-030-77091-4_1"},{"key":"6516_CR24","doi-asserted-by":"crossref","unstructured":"Fouladgar N, Alirezaie M, Fr\u00e4mling K (2021) Exploring contextual importance and utility in explaining affect detection. In: AIxIA 2020\u2013advances in artificial intelligence: XIXth international conference of the Italian association for artificial intelligence,, volume 12414, pp 3\u201318","DOI":"10.1007\/978-3-030-77091-4_1"},{"key":"6516_CR25","doi-asserted-by":"crossref","unstructured":"Schmidt P, Reiss A, Duerichen R, Marberger C, Van\u00a0Laerhoven K (2018) Introducing wesad, a multimodal dataset for wearable stress and affect detection. In: Proceedings of the 20th ACM international conference on multimodal interaction, Association for Computing Machinery, pp 400\u2013408","DOI":"10.1145\/3242969.3242985"},{"issue":"1","key":"6516_CR26","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/T-AFFC.2011.25","volume":"3","author":"S Mohammad","year":"2012","unstructured":"Mohammad S, Jeroen L, Thierry P, Maja P (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42\u201355","journal-title":"IEEE Trans Affect Comput"},{"issue":"6","key":"6516_CR27","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1037\/h0077714","volume":"39","author":"A Russell James","year":"1980","unstructured":"Russell James A (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161\u20131178","journal-title":"J Pers Soc Psychol"},{"key":"6516_CR28","first-page":"715","volume":"17","author":"P Jonathan","year":"2005","unstructured":"Jonathan P, James R, Bradley P (2005) The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol 17:715\u201334","journal-title":"Dev Psychopathol"},{"issue":"4","key":"6516_CR29","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/j.ijhcs.2006.11.011","volume":"65","author":"L Mandryk Regan","year":"2007","unstructured":"Mandryk Regan L, Stella AM (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int J Hum Comput Stud 65(4):329\u2013347","journal-title":"Int J Hum Comput Stud"},{"key":"6516_CR30","doi-asserted-by":"crossref","unstructured":"Khalili Z, Moradi MH (2009) Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of EEG. In: International joint conference on neural networks, pp 1571\u20131575","DOI":"10.1109\/IJCNN.2009.5178854"},{"key":"6516_CR31","doi-asserted-by":"publisher","first-page":"105217","DOI":"10.1016\/j.knosys.2019.105217","volume":"190","author":"F Laura","year":"2020","unstructured":"Laura F, Gianmaria M, Francesco S, Hamido F, Filippo C (2020) Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowl Based Syst 190:105217","journal-title":"Knowl Based Syst"},{"issue":"2","key":"6516_CR32","first-page":"230","volume":"11","author":"KB Hyung","year":"2020","unstructured":"Hyung KB, Sungho J (2020) Deep physiological affect network for the recognition of human emotions. IEEE Trans Affect Comput 11(2):230\u2013243","journal-title":"IEEE Trans Affect Comput"},{"key":"6516_CR33","doi-asserted-by":"crossref","unstructured":"Wijsman J, Grundlehner B, Penders J, Hermens H (2010) Trapezius muscle emg as predictor of mental stress. In: ACM transactions on embedded computing systems (TECS), volume\u00a012, pp 155\u2013163, 01","DOI":"10.1145\/1921081.1921100"},{"key":"6516_CR34","doi-asserted-by":"crossref","unstructured":"Wijsman J, Grundlehner B, Liu H, Penders J, Hermens H (2013) Wearable physiological sensors reflect mental stress state in office-like situations. In: 2013 humaine association conference on affective computing and intelligent interaction, pp 600\u2013605","DOI":"10.1109\/ACII.2013.105"},{"issue":"2","key":"6516_CR35","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1109\/TITB.2011.2169804","volume":"16","author":"J Choi","year":"2012","unstructured":"Choi J, Ahmed B, Gutierrez-Osuna R (2012) Development and evaluation of an ambulatory stress monitor based on wearable sensors. IEEE Trans Inf Technol Biomed 16(2):279\u201386","journal-title":"IEEE Trans Inf Technol Biomed"},{"key":"6516_CR36","doi-asserted-by":"crossref","unstructured":"Vila G, Godin C, Charbonnier S, Labyt E, Sakri O, Campagne A (2018) Pressure-specific feature selection for acute stress detection from physiological recordings. In: 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp 2341\u20132346","DOI":"10.1109\/SMC.2018.00402"},{"key":"6516_CR37","doi-asserted-by":"publisher","first-page":"04","DOI":"10.1109\/JSEN.2018.2875160","volume":"19","author":"C Yekta","year":"2019","unstructured":"Yekta C, Niaz C, Deniz E, Cem E (2019) Continuous stress detection using wearable sensors in real life: algorithmic programming contest case study. Sensors 19:04","journal-title":"Sensors"},{"issue":"5","key":"6516_CR38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3186585","volume":"51","author":"RASTGOO Mohammad Naim","year":"2018","unstructured":"Mohammad Naim RASTGOO, Bahareh N, Andry R, Vinod C, Dian T (2018) A critical review of proactive detection of driver stress levels based on multimodal measurements. ACM Comput Surv 51(5):1\u201335","journal-title":"ACM Comput Surv"},{"key":"6516_CR39","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1016\/j.asoc.2019.01.006","volume":"77","author":"C Amandeep","year":"2019","unstructured":"Amandeep C, Mandeep S (2019) An application of phonocardiography signals for psychological stress detection using non-linear entropy based features in empirical mode decomposition domain. Appl Soft Comput 77:24\u201333","journal-title":"Appl Soft Comput"},{"key":"6516_CR40","unstructured":"Plarre K, Raij A, Hossain SM, Ali AA, Nakajima M, al\u2019Absi M, Ertin E, Kamarck T, Kumar S, Scott M, Siewiorek D, Smailagic A, Wittmers LE (2011) Continuous inference of psychological stress from sensory measurements collected in the natural environment. In: Proceedings of the 10th ACM\/IEEE international conference on information processing in sensor networks, pp 97\u2013108"},{"key":"6516_CR41","doi-asserted-by":"crossref","unstructured":"Markovi\u0107 D, Vuji\u010di\u0107 D, Stoji\u0107 D, Jovanovi\u0107 \u017d, Pe\u0161ovi\u0107 U, Ran\u0111i\u0107 Sini\u0161a (2019) Monitoring system based on iot sensor data with complex event processing and artificial neural networks for patients stress detection. In: 2019 18th international symposium INFOTEH-JAHORINA (INFOTEH), pp 1\u20136","DOI":"10.1109\/INFOTEH.2019.8717748"},{"key":"6516_CR42","unstructured":"Fr\u00e4mling K (2020) Explainable ai without interpretable model"},{"key":"6516_CR43","first-page":"1","volume":"126","author":"S Patrick","year":"2018","unstructured":"Patrick S, Christa B, Lothar S (2018) Correlation coefficients: appropriate use and interpretation. Anesthesia Analgesia 126:1","journal-title":"Anesthesia Analgesia"},{"key":"6516_CR44","unstructured":"Werner R, Valev D, Danov D (2009) The pearson\u2019s correlation -a measure for the linear relationships between time series? In: Fundamental space research"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-021-06516-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-021-06516-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-021-06516-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T18:20:02Z","timestamp":1643998802000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-021-06516-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,24]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,2]]}},"alternative-id":["6516"],"URL":"https:\/\/doi.org\/10.1007\/s00521-021-06516-3","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"type":"print","value":"0941-0643"},{"type":"electronic","value":"1433-3058"}],"subject":[],"published":{"date-parts":[[2021,9,24]]},"assertion":[{"value":"12 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}