{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T17:29:35Z","timestamp":1774718975383,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,12,4]],"date-time":"2022-12-04T00:00:00Z","timestamp":1670112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Taif University","award":["TURSP-2020\/36"],"award-info":[{"award-number":["TURSP-2020\/36"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Emotion charting using multimodal signals has gained great demand for stroke-affected patients, for psychiatrists while examining patients, and for neuromarketing applications. Multimodal signals for emotion charting include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and galvanic skin response (GSR) signals. EEG, ECG, and GSR are also known as physiological signals, which can be used for identification of human emotions. Due to the unbiased nature of physiological signals, this field has become a great motivation in recent research as physiological signals are generated autonomously from human central nervous system. Researchers have developed multiple methods for the classification of these signals for emotion detection. However, due to the non-linear nature of these signals and the inclusion of noise, while recording, accurate classification of physiological signals is a challenge for emotion charting. Valence and arousal are two important states for emotion detection; therefore, this paper presents a novel ensemble learning method based on deep learning for the classification of four different emotional states including high valence and high arousal (HVHA), low valence and low arousal (LVLA), high valence and low arousal (HVLA) and low valence high arousal (LVHA). In the proposed method, multimodal signals (EEG, ECG, and GSR) are preprocessed using bandpass filtering and independent components analysis (ICA) for noise removal in EEG signals followed by discrete wavelet transform for time domain to frequency domain conversion. Discrete wavelet transform results in spectrograms of the physiological signal and then features are extracted using stacked autoencoders from those spectrograms. A feature vector is obtained from the bottleneck layer of the autoencoder and is fed to three classifiers SVM (support vector machine), RF (random forest), and LSTM (long short-term memory) followed by majority voting as ensemble classification. The proposed system is trained and tested on the AMIGOS dataset with k-fold cross-validation. The proposed system obtained the highest accuracy of 94.5% and shows improved results of the proposed method compared with other state-of-the-art methods.<\/jats:p>","DOI":"10.3390\/s22239480","type":"journal-article","created":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T08:10:57Z","timestamp":1670227857000},"page":"9480","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals"],"prefix":"10.3390","volume":"22","author":[{"given":"Amna Waheed","family":"Awan","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Bahria University, Islamabad 44000, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0504-3558","authenticated-orcid":false,"given":"Syed Muhammad","family":"Usman","sequence":"additional","affiliation":[{"name":"Department of Creative Technologies, Faculty of Computing and AI, Air University, Islamabad 44000, Pakistan"}]},{"given":"Shehzad","family":"Khalid","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Bahria University, Islamabad 44000, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2891-7844","authenticated-orcid":false,"given":"Aamir","family":"Anwar","sequence":"additional","affiliation":[{"name":"School of Computing and Engineering, The University of West London, London W5 5RF, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1585-2962","authenticated-orcid":false,"given":"Roobaea","family":"Alroobaea","sequence":"additional","affiliation":[{"name":"Department of Computer Science, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1523-1330","authenticated-orcid":false,"given":"Saddam","family":"Hussain","sequence":"additional","affiliation":[{"name":"School of Digital Science, Universiti Brunei Darussalam, Jalan Tungku Link, Gadong BE1410, Brunei"}]},{"given":"Jasem","family":"Almotiri","sequence":"additional","affiliation":[{"name":"Department of Computer Science, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5406-0389","authenticated-orcid":false,"given":"Syed Sajid","family":"Ullah","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Technology, University of Agder (UiA), N-4898 Grimstad, Norway"},{"name":"Department of Electrical and Computer Engineering, Villanova University, Villanova, PA 19085, USA"}]},{"given":"Muhammad Usman","family":"Akram","sequence":"additional","affiliation":[{"name":"College of Eletrical and Mechanical Engineering (E & ME), National University of Science and Technology (NUST), Islamabad 44000, Pakistan"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rahim, A., Sagheer, A., Nadeem, K., Da, M.N., Rahim, A., and Akram, U. (2019, January 21\u201322). Emotion Charting Using Real-time Monitoring of Physiological Signals. Proceedings of the International Conference on Robotics and Automation in Industry (ICRAI), Rawalpindi, Pakistan.","DOI":"10.1109\/ICRAI47710.2019.8967398"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Rouse, B.A. (1995). Substance Abuse and Mental Health Statistics Sourcebook, Office, of Applied Studies, Substance Abuse and Mental Health Services Administration, Public Health Service, US Department of Health and Human Services.","DOI":"10.1037\/e481682006-001"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Garg, S., Patro, R.K., Behera, S., Tigga, N.P., and Pandey, R. (2021). An overlapping sliding window and combined features based emotion recognition system for EEG signals. Appl. Comput. Inform.","DOI":"10.1108\/ACI-05-2021-0130"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"De Nadai, S., D\u2019Inc\u00e0, M., Parodi, F., Benza, M., Trotta, A., Zero, E., Zero, L., and Sacile, R. (2016, January 12\u201316). Enhancing safety of transport by road by on-line monitoring of driver emotions. Proceedings of the 2016 11th System of Systems Engineering Conference (SoSE), Kongsberg, Norway.","DOI":"10.1109\/SYSOSE.2016.7542941"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Guo, R., Li, S., He, L., Gao, W., Qi, H., and Owens, G. (2013, January 5\u20138). Pervasive and unobtrusive emotion sensing for human mental health. Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare, Venice, Italy.","DOI":"10.4108\/icst.pervasivehealth.2013.252133"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"99","DOI":"10.5334\/pb-46-1-2-99","article-title":"Psychopathy and Physiological Detection of Concealed Information: A review","volume":"46","author":"Verschuere","year":"2006","journal-title":"Psychol. Belg."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Dar, M.N., Akram, M.U., Khawaja, S.G., and Pujari, A.N. (2020). CNN and LSTM-based emotion charting using physiological signals. Sensors, 20.","DOI":"10.3390\/s20164551"},{"key":"ref_8","first-page":"15","article-title":"Studies of emotion: A theoretical and empirical review of psychophysiological studies of emotion","volume":"1","author":"Niemic","year":"2002","journal-title":"J. Undergrad. Res."},{"key":"ref_9","first-page":"1442","article-title":"Emotion recognition based on EEG feature maps through deep learning network","volume":"24","author":"Topic","year":"2021","journal-title":"Eng. Sci. Technol. Int. J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1109\/TAFFC.2019.2901673","article-title":"Feature extraction and selection for emotion recognition from electrodermal activity","volume":"12","author":"Shukla","year":"2019","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2266","DOI":"10.1109\/JSEN.2018.2883497","article-title":"Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals","volume":"19","author":"Gupta","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"104428","DOI":"10.1016\/j.compbiomed.2021.104428","article-title":"Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals","volume":"134","author":"Maheshwari","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tung, K., Liu, P.K., Chuang, Y.C., Wang, S.H., and Wu, A.Y.A. (2018, January 3\u20136). Entropy-assisted multi-modal emotion recognition framework based on physiological signals. Proceedings of the IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), Sarawak, Malaysia.","DOI":"10.1109\/IECBES.2018.8626634"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1541","DOI":"10.1109\/TAFFC.2020.3014842","article-title":"Self-supervised ECG representation learning for emotion recognition","volume":"13","author":"Sarkar","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_15","unstructured":"Zhao, Y., Cao, X., Lin, J., Yu, D., and Cao, X. (2021). Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model. IEEE Trans. Affect. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez-Tejada, L.A., Maruyama, Y., Yoshimura, N., and Koike, Y. (2020). Analysis of Personality and EEG features in Emotion Recognition using Machine Learning Techniques to classify arousal and valence labels. Mach. Learn. Knowl. Extr., 2.","DOI":"10.3390\/make2020007"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Klados, M.A., Konstantinidi, P., Dacosta-Aguayo, R., Kostaridou, V.D., Vinciarelli, A., and Zervakis, M. (2020). Automatic recognition of personality profiles using EEG functional connectivity during emotional processing. Brain Sci., 10.","DOI":"10.3390\/brainsci10050278"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Yang, J., Lin, J., Yu, D., and Cao, X. (2020, January 19\u201324). A 3D convolutional neural network for emotion recognition based on EEG signals. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207420"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"8659","DOI":"10.1016\/j.eswa.2010.06.065","article-title":"EEG signal classification using PCA, ICA, LDA and support vector machines","volume":"37","author":"Subasi","year":"2010","journal-title":"Expert Syst. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2901","DOI":"10.1109\/TNNLS.2020.3008938","article-title":"Time\u2013frequency representation and convolutional neural network-based emotion recognition","volume":"32","author":"Khare","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Shu, L., Xie, J., Yang, Z.L., Li, Z., Liao, D., and Xu, X. (2018). A review of emotion recognition using physiological signals. Sensors, 18.","DOI":"10.3390\/s18072074"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"106818","DOI":"10.1016\/j.eplepsyres.2021.106818","article-title":"Detection of preictal state in epileptic seizures using ensemble classifier","volume":"178","author":"Usman","year":"2021","journal-title":"Epilepsy Res."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.bbe.2021.01.001","article-title":"Epileptic seizure prediction using scalp electroencephalogram signals","volume":"41","author":"Usman","year":"2021","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"104710","DOI":"10.1016\/j.compbiomed.2021.104710","article-title":"A Deep Learning based Ensemble Learning Method for Epileptic Seizure Prediction","volume":"136","author":"Usman","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Waheed, Z., Waheed, A., and Akram, M.U. (2016, January 12\u201316). A Robust Non-Vascular Retina Recognition System using Structural Features of Retinal Image. Proceedings of the 13th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan.","DOI":"10.1109\/IBCAST.2016.7429862"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Awan, A.W., Awan, Z.W., and Akram, M.U. (2015, January 16\u201318). A robust algorithm for segmentation of blood vessels in the presence of lesions in retinal fundus images. Proceedings of the IEEE International Conference on Imaging Systems and Techniques (IST), Macau, China.","DOI":"10.1109\/IST.2015.7294519"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Waheed, Z., Akram, M.U., Waheed, A., and Shaukat, A. (2015, January 15\u201317). Robust Extraction of Blood Vessels for Retinal Recognition. Proceedings of the Second International Conference on Information Security and Cyber Forensics (InfoSec), Cape Town, South Africa.","DOI":"10.1109\/InfoSec.2015.7435497"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"50","DOI":"10.21833\/ijaas.2019.03.008","article-title":"Principle components analysis for seizures prediction using wavelet transform","volume":"6","author":"Usman","year":"2019","journal-title":"Int. J. Adv. Appl. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.seizure.2019.08.006","article-title":"Using scalp EEG and intracranial EEG signals for predicting epileptic seizures: Review of available methodologies","volume":"71","author":"Usman","year":"2019","journal-title":"Seizure"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"9074759","DOI":"10.1155\/2017\/9074759","article-title":"Epileptic seizures prediction using machine learning methods","volume":"2017","author":"Usman","year":"2017","journal-title":"Comput. Math. Methods Med."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"39998","DOI":"10.1109\/ACCESS.2020.2976866","article-title":"Epileptic seizures prediction using deep learning techniques","volume":"8","author":"Usman","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hamam, H. (2022). Rotation Invariant Parallel Signal Processing Using a Diffractive Phase Element for Image Compression. Appl. Sci., 12.","DOI":"10.3390\/app12010439"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hamam, H. (2022). Modeling Several Optical Components Using Scalar Diffraction Theory. Appl. Sci., 12.","DOI":"10.3390\/app12073580"},{"key":"ref_34","unstructured":"Lepcha, D.C., Dogra, A., Goyal, B., Chohan, J.S., Koundal, D., Zaguia, A., and Hamam, H. (2022). Multimodal Medical Image Fusion Based on Pixel Significance Using Anisotropic Diffusion and Cross Bilateral Filter. Hum. -Cent. Comput. Inf. Sci., 12."},{"key":"ref_35","first-page":"4260804","article-title":"A new V-net convolutional neural network based on four-dimensional hyperchaotic system for medical image encryption","volume":"2022","author":"Wang","year":"2022","journal-title":"Secur. Commun. Netw."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ouichka, O., Echtioui, A., and Hamam, H. (2022). Deep Learning Models for Predicting Epileptic Seizures Using IEEG Signals. Electronics, 11.","DOI":"10.3390\/electronics11040605"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"26931","DOI":"10.1109\/JSEN.2021.3120787","article-title":"A computerized approach for automatic human emotion recognition using sliding mode singular spectrum analysis","volume":"21","author":"Sharma","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_38","first-page":"57","article-title":"Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS)","volume":"7","author":"Abdulhay","year":"2018","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3579","DOI":"10.1109\/JSEN.2020.3027181","article-title":"A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition","volume":"21","author":"Bhattacharyya","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Anwar, A., Rehman, I.U., and Husamaldin, L. (2022, January 26\u201329). Smart Education for People with Disabilities (PwDs): Conceptual Framework for PwDs Emotions Classification from Student Utterances (SUs) during Online Learning. Proceedings of the 2022 IEEE International Smart Cities Conference (ISC2), Pafos, Cyprus.","DOI":"10.1109\/ISC255366.2022.9922083"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Amjad, A., Qaiser, S., Anwar, A., and Ali, R. (2021, January 7\u201310). Analysing Public Sentiments Regarding COVID-19 Vaccines: A Sentiment Analysis Approach. Proceedings of the 2021 IEEE International Smart Cities Conference (ISC2), Manchester, UK.","DOI":"10.1109\/ISC253183.2021.9562904"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9480\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:33:54Z","timestamp":1760146434000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9480"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,4]]},"references-count":41,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["s22239480"],"URL":"https:\/\/doi.org\/10.3390\/s22239480","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,4]]}}}