{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T18:44:50Z","timestamp":1769280290702,"version":"3.49.0"},"reference-count":53,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2020,8,21]],"date-time":"2020-08-21T00:00:00Z","timestamp":1597968000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Xinhua Net Future Media Convergence Institute","award":["S-0003-LX-18"],"award-info":[{"award-number":["S-0003-LX-18"]}]},{"DOI":"10.13039\/501100008530","name":"European Regional Development Fund","doi-asserted-by":"publisher","award":["TIN2017-85409-P"],"award-info":[{"award-number":["TIN2017-85409-P"]}],"id":[{"id":"10.13039\/501100008530","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100015494","name":"Instituto de Telecomunica\u00e7\u00f5es","doi-asserted-by":"publisher","award":["UIDB\/50008\/2020"],"award-info":[{"award-number":["UIDB\/50008\/2020"]}],"id":[{"id":"10.13039\/501100015494","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low\/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal\/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.<\/jats:p>","DOI":"10.3390\/s20174723","type":"journal-article","created":{"date-parts":[[2020,8,21]],"date-time":"2020-08-21T09:21:51Z","timestamp":1598001711000},"page":"4723","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0514-7517","authenticated-orcid":false,"given":"Patr\u00edcia","family":"Bota","sequence":"first","affiliation":[{"name":"Instituto Superior T\u00e9cnico (IST), Department of Bioengineering (DBE) and Instituto de Telecomunica\u00e7\u00f5es (IT), Av. Rovisco Pais n. 1, Torre Norte-Piso 10, 1049-001 Lisbon, Portugal"}]},{"given":"Chen","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Media Convergence Production Technology and Systems, Xinhua News Agency &amp; Future Media Convergence Institute (FMCI), Xinhua Net, Jinxuan Building, No. 129 Xuanwumen West Street, Beijing 100031, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1320-5024","authenticated-orcid":false,"given":"Ana","family":"Fred","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico (IST), Department of Bioengineering (DBE) and Instituto de Telecomunica\u00e7\u00f5es (IT), Av. Rovisco Pais n. 1, Torre Norte-Piso 10, 1049-001 Lisbon, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6764-8432","authenticated-orcid":false,"given":"Hugo","family":"Silva","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico (IST), Department of Bioengineering (DBE) and Instituto de Telecomunica\u00e7\u00f5es (IT), Av. Rovisco Pais n. 1, Torre Norte-Piso 10, 1049-001 Lisbon, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Greenberg, L.S., and Safran, J. (1987). Emotion, Cognition, and Action. Theoretical Foundations of Behavior Therapy, Springer.","DOI":"10.1007\/978-1-4899-0827-8_14"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., and Yang, X. (2018). A Review of Emotion Recognition Using Physiological Signals. Sensors, 18.","DOI":"10.3390\/s18072074"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/02699939208411068","article-title":"An argument for basic emotions","volume":"6","author":"Paul","year":"1992","journal-title":"Cogn. Emot."},{"key":"ref_4","unstructured":"Damasio, A.R. (1994). Descartes\u2019 Error: Emotion, Reason, and the Human Brain, G.P. Putnam."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1037\/0003-066X.50.5.372","article-title":"The emotion probe: Studies of motivation and attention","volume":"50","author":"Lang","year":"1995","journal-title":"Am. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., and Van Laerhoven, K. (2018, January 16\u201320). Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection. Proceedings of the International Conference on Multimodal Interaction, Boulder, CO, USA.","DOI":"10.1145\/3242969.3242985"},{"key":"ref_7","unstructured":"Pinto, J. (2019). Exploring Physiological Multimodality for Emotional Assessment. [Master\u2019s Thesis, Instituto Superior T\u00e9cnico]."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"DEAP: A Database for Emotion Analysis using Physiological Signals","volume":"3","author":"Koelstra","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/34.954607","article-title":"Toward machine emotional intelligence: Analysis of affective physiological state","volume":"23","author":"Picard","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_10","unstructured":"Schmidt, P., Reiss, A., Duerichen, R., and Laerhoven, K.V. (2018). Wearable affect and stress recognition: A review. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"140990","DOI":"10.1109\/ACCESS.2019.2944001","article-title":"A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals","volume":"7","author":"Bota","year":"2019","journal-title":"IEEE Access"},{"key":"ref_12","unstructured":"Liu, C., Rani, P., and Sarkar, N. (2005, January 2\u20136). An empirical study of machine learning techniques for affect recognition in human-robot interaction. Proceedings of the International Conference on Intelligent Robots and Systems, Edmonton, AB, Canada."},{"key":"ref_13","unstructured":"Kim, S.M., Valitutti, A., and Calvo, R.A. (2010, January 5). Evaluation of Unsupervised Emotion Models to Textual Affect Recognition. Proceedings of the NAAL HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"22196","DOI":"10.1109\/ACCESS.2018.2821192","article-title":"Leveraging Unlabeled Data for Emotion Recognition with Enhanced Collaborative Semi-Supervised Learning","volume":"6","author":"Zhang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Alhagry, S., Fahmy, A.A., and El-Khoribi, R.A. (2017). Emotion Recognition based on EEG using LSTM Recurrent Neural Network. Int. J. Adv. Comput. Sci. Appl., 8.","DOI":"10.14569\/IJACSA.2017.081046"},{"key":"ref_16","unstructured":"Zhang, J., Chen, M., Hu, S., Cao, Y., and Kozma, R. (2016, January 9\u201312). PNN for EEG-based Emotion Recognition. Proceedings of the International Conference on Systems, Man, and Cybernetics, Budapest, Hungary."},{"key":"ref_17","unstructured":"Salari, S., Ansarian, A., and Atrianfar, H. (March, January 28). Robust emotion classification using neural network models. Proceedings of the Iranian Joint Congress on Fuzzy and Intelligent Systems, Kerman, Iran."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kim, J.H., Matson, E.T., Myung, H., and Xu, P. (2013). Analysis of Physiological Signals for Emotion Recognition Based on Support Vector Machine. Robot Intelligence Technology and Applications 2012, Springer.","DOI":"10.1007\/978-3-642-37374-9"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Cheng, B. (2012). Emotion Recognition from Physiological Signals Using Support Vector Machine, Springer.","DOI":"10.1007\/978-3-642-03718-4_6"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"He, C., Yao, Y.J., and Ye, X.S. (2017). An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors, Springer.","DOI":"10.1007\/978-981-10-2404-7_2"},{"key":"ref_21","unstructured":"Huang, T., Zeng, Z., Li, C., and Leung, C.S. (2012, January 12\u201315). Emotion Recognition Using KNN Classification for User Modeling and Sharing of Affect States. Proceedings of the Neural Information Processing, Doha, Qatar."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"509","DOI":"10.3233\/THC-174836","article-title":"Emotion recognition from multichannel EEG signals using K-nearest neighbor classification","volume":"26","author":"Li","year":"2018","journal-title":"Technol. Health Care"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"908","DOI":"10.1111\/j.1469-8986.2010.01170.x","article-title":"An affective computing approach to physiological emotion specificity: Toward subject-independent and stimulus-independent classification of film-induced emotions","volume":"48","author":"Kolodyazhniy","year":"2011","journal-title":"Psychophysiology"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, X., Xu, C., Xue, W., Hu, J., He, Y., and Gao, M. (2018). Emotion Recognition Based on Multichannel Physiological Signals with Comprehensive Nonlinear Processing. Sensors, 18.","DOI":"10.3390\/s18113886"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gong, P., Ma, H.T., and Wang, Y. (2016, January 6\u20139). Emotion recognition based on the multiple physiological signals. Proceedings of the International Conference on Real-time Computing and Robotics, Angkor Wat, Cambodia.","DOI":"10.1109\/RCAR.2016.7784015"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/s40846-019-00505-7","article-title":"Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems","volume":"40","author":"Ayata","year":"2020","journal-title":"J. Med. Biol. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chen, J., Hu, B., Wang, Y., Moore, P., Dai, Y., Feng, L., and Ding, Z. (2017). Subject-independent emotion recognition based on physiological signals: A three-stage decision method. BMC Med. Informatics Decis. Mak., 17.","DOI":"10.1186\/s12911-017-0562-x"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"8317357","DOI":"10.1155\/2017\/8317357","article-title":"Emotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain","volume":"2017","author":"Zhuang","year":"2017","journal-title":"BioMed Res. Int."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1016\/j.procs.2015.04.138","article-title":"An Approach to EEG Based Emotion Recognition and Classification Using Kernel Density Estimation","volume":"48","author":"Lahane","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"94160","DOI":"10.1109\/ACCESS.2019.2928691","article-title":"Interpretable Emotion Recognition Using EEG Signals","volume":"7","author":"Qing","year":"2019","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2408","DOI":"10.1016\/j.proeng.2011.08.452","article-title":"Study of Emotion Recognition Based on Electrocardiogram and RBF neural network","volume":"15","author":"Xianhai","year":"2011","journal-title":"Procedia Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"6486","DOI":"10.1038\/s41598-019-42826-2","article-title":"Heart sound signals can be used for emotion recognition","volume":"9","author":"Xiefeng","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dissanayake, T., Rajapaksha, Y., Ragel, R., and Nawinne, I. (2019). An Ensemble Learning Approach for Electrocardiogram Sensor Based Human Emotion Recognition. Sensors, 19.","DOI":"10.3390\/s19204495"},{"key":"ref_34","unstructured":"Shukla, J., Barreda-Angeles, M., Oliver, J., Nandi, G.C., and Puig, D. (2019). Feature Extraction and Selection for Emotion Recognition from Electrodermal Activity. IEEE Trans. Affect. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Udovi\u010di\u0107, G., \u00d0erek, J., Russo, M., and Sikora, M. (2017, January 23\u201327). Wearable Emotion Recognition System Based on GSR and PPG Signals. Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care, Mountain View, CA, USA.","DOI":"10.1145\/3132635.3132641"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Liu, M., Fan, D., Zhang, X., and Gong, X. (2016, January 25\u201326). Human Emotion Recognition Based on Galvanic Skin Response Signal Feature Selection and SVM. Proceedings of the 2016 International Conference on Smart City and Systems Engineering, Hunan, China.","DOI":"10.1109\/ICSCSE.2016.0051"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2018\/5296523","article-title":"Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals","volume":"2018","author":"Wei","year":"2018","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_38","unstructured":"Chen, J., Hu, B., Xu, L., Moore, P., and Su, Y. (2015, January 9\u201312). Feature-level fusion of multimodal physiological signals for emotion recognition. Proceedings of the International Conference on Bioinformatics and Biomedicine (BIBM), Washington, DC, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Canento, F., Fred, A., Silva, H., Gamboa, H., and Louren\u00e7o, A. (2011, January 28\u201331). Multimodal biosignal sensor data handling for emotion recognition. Proceedings of the 2011 IEEE Sensors Conference, Limerick, Ireland.","DOI":"10.1109\/ICSENS.2011.6127029"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Xie, J., Xu, X., and Shu, L. (2018, January 20\u201322). WT Feature Based Emotion Recognition from Multi-channel Physiological Signals with Decision Fusion. Proceedings of the Asian Conference on Affective Computing and Intelligent Interaction, Beijing, China.","DOI":"10.1109\/ACIIAsia.2018.8470381"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1109\/TAFFC.2016.2625250","article-title":"ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors","volume":"9","author":"Subramanian","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Aguileta, A.A., Brena, R.F., Mayora, O., Molino-Minero-Re, E., and Trejo, L.A. (2019). Multi-Sensor Fusion for Activity Recognition\u2014A Survey. Sensors, 19.","DOI":"10.3390\/s19173808"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/j.entcs.2019.04.009","article-title":"Emotion Recognition from Physiological Signal Analysis: A Review","volume":"343","author":"Egger","year":"2019","journal-title":"Electron. Notes Theor. Comput. Sci."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s40537-020-00289-7","article-title":"A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals","volume":"7","author":"Doma","year":"2020","journal-title":"J. Big Data"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Dzedzickis, A., Kaklauskas, A., and Bucinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors, 20.","DOI":"10.3390\/s20030592"},{"key":"ref_46","unstructured":"Marechal, C., Miko\u0142ajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., and W\u0119grzyn-Wolska, K. (2019). High-Performance Modelling and Simulation for Big Data Applications: Selected Results of the COST Action IC1406 cHiPSet, Springer International Publishing."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review","volume":"59","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_48","unstructured":"Duda, R.O., Hart, P.E., and Stork, D.G. (2000). Pattern Classification, Wiley-Interscience. [2nd ed.]."},{"key":"ref_49","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1109\/MPRV.2014.61","article-title":"Biosignals for Everyone","volume":"13","author":"Fred","year":"2014","journal-title":"IEEE Pervasive Comput."},{"key":"ref_51","unstructured":"Alves, A.P., Pl\u00e1cido da Silva, H., Lourenco, A., and Fred, A. (2013, January 11\u201314). BITalino: A Biosignal Acquisition System based on Arduino. Proceedings of the International Conference on Biomedical Electronics and Devices (BIODEVICES), Barcelona, Spain."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/T-AFFC.2011.25","article-title":"A Multimodal Database for Affect Recognition and Implicit Tagging","volume":"3","author":"Soleymani","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Wiem, M., and Lachiri, Z. (2017). Emotion Classification in Arousal Valence Model using MAHNOB-HCI Database. Int. J. Adv. Comput. Sci. Appl., 8.","DOI":"10.14569\/IJACSA.2017.080344"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4723\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:04:39Z","timestamp":1760177079000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4723"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,21]]},"references-count":53,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["s20174723"],"URL":"https:\/\/doi.org\/10.3390\/s20174723","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,21]]}}}