{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:03:49Z","timestamp":1760148229920,"version":"build-2065373602"},"reference-count":63,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T00:00:00Z","timestamp":1681257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100016714","name":"University of Sharjah in the United Arab Emirates","doi-asserted-by":"publisher","award":["20020403159"],"award-info":[{"award-number":["20020403159"]}],"id":[{"id":"10.13039\/100016714","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Modern developments in machine learning methodology have produced effective approaches to speech emotion recognition. The field of data mining is widely employed in numerous situations where it is possible to predict future outcomes by using the input sequence from previous training data. Since the input feature space and data distribution are the same for both training and testing data in conventional machine learning approaches, they are drawn from the same pool. However, because so many applications require a difference in the distribution of training and testing data, the gathering of training data is becoming more and more expensive. High performance learners that have been trained using similar, already-existing data are needed in these situations. To increase a model\u2019s capacity for learning, transfer learning involves transferring knowledge from one domain to another related domain. To address this scenario, we have extracted ten multi-dimensional features from speech signals using OpenSmile and a transfer learning method to classify the features of various datasets. In this paper, we emphasize the importance of a novel transfer learning system called Task-based Unification and Adaptation (TUA), which bridges the disparity between extensive upstream training and downstream customization. We take advantage of the two components of the TUA, task-challenging unification and task-specific adaptation. Our algorithm is studied using the following speech datasets: the Arabic Emirati-accented speech dataset (ESD), the English Speech Under Simulated and Actual Stress (SUSAS) dataset and the Ryerson Audio-Visual Database of Emotional Speech and Song dataset (RAVDESS). Using the multidimensional features and transfer learning method on the given datasets, we were able to achieve an average speech emotion recognition rate of 91.2% on the ESD, 84.7% on the RAVDESS and 88.5% on the SUSAS datasets, respectively.<\/jats:p>","DOI":"10.3390\/info14040236","type":"journal-article","created":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T02:57:08Z","timestamp":1681268228000},"page":"236","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Novel Task-Based Unification and Adaptation (TUA) Transfer Learning Approach for Bilingual Emotional Speech Data"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7856-9342","authenticated-orcid":false,"given":"Ismail","family":"Shahin","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, University of Sharjah, Abu Dhabi P.O. Box 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1570-0897","authenticated-orcid":false,"given":"Ali Bou","family":"Nassif","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, University of Sharjah, Abu Dhabi P.O. Box 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rameena","family":"Thomas","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, University of Sharjah, Abu Dhabi P.O. Box 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9422-7251","authenticated-orcid":false,"given":"Shibani","family":"Hamsa","sequence":"additional","affiliation":[{"name":"Center for Cyber Physical Systems, Department of Electrical and Computer Engineering, Khalifa University, Abu Dhabi P.O. Box 127788, United Arab Emirates"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Shahin, I. (2020, January 4\u20136). Emotion Recognition Using Speaker Cues. Proceedings of the 2020 Advances in Science and Engineering Technology International Conferences (ASET), Dubai, United Arab Emirates.","DOI":"10.1109\/ASET48392.2020.9118271"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shahin, I. (2017, January 21\u201323). Speaker verification in emotional talking environments based on three-stage frame-work. Proceedings of the 2017 International Conference on Electrical and Computing Technologies and Applications (ICECTA), Ras Al Khaimah, United Arab Emirates.","DOI":"10.1109\/ICECTA.2017.8251961"},{"key":"ref_3","first-page":"10","article-title":"Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels","volume":"2","author":"Wu","year":"2010","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_4","unstructured":"Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv."},{"key":"ref_5","unstructured":"Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. (2018, January 26\u201328). Efficient neural architecture search via parameters sharing. Proceedings of the International Conference on Machine Learning, Macau, China."},{"key":"ref_6","unstructured":"Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. (2019). Once-for-all: Train one network and specialize it for efficient deployment. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Yu, J., Jin, P., Liu, H., Bender, G., Kindermans, P.J., Tan, M., Huang, T., Song, X., Pang, R., and Le, Q. (2020, January 23\u201327). Bignas: Scaling up neural architecture search with big single-stage models. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.","DOI":"10.1007\/978-3-030-58571-6_41"},{"key":"ref_8","unstructured":"Shahin, I. (2008, January 18\u201320). Using emotions to identify speakers. Proceedings of the The 5th International Workshop on Signal Processing and its Applications (WoSPA 2008), Sharjah, United Arab Emirates."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/j.ipm.2008.09.003","article-title":"Acoustic feature selection for automatic emotion recognition from speech","volume":"45","author":"Rong","year":"2009","journal-title":"Inf. Process. Manag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10772-011-9089-1","article-title":"Identifying speakers using their emotion cues","volume":"14","author":"Shahin","year":"2011","journal-title":"Int. J. Speech Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/s12193-011-0082-4","article-title":"Studying and enhancing talking condition recognition in stressful and emotional talking environments based on HMMs, CHMM2s and SPHMMs","volume":"6","author":"Shahin","year":"2012","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1007\/s10772-013-9188-2","article-title":"Employing both gender and emotion cues to enhance speaker identification performance in emotional talking environments","volume":"16","author":"Shahin","year":"2013","journal-title":"Int. J. Speech Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.bspc.2014.10.008","article-title":"Weighted spectral features based on local Hu moments for speech emotion recognition","volume":"18","author":"Sun","year":"2015","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., and Weiss, B. (2005, January 4\u20138). A database of German emotional speech. Proceedings of the Ninth European Conference on Speech Communication and Technology, Lisbon, Portugal.","DOI":"10.21437\/Interspeech.2005-446"},{"key":"ref_15","unstructured":"Haq, S., Jackson, P.J., and Edge, J. (2009, January 10\u201313). Speaker-dependent audio-visual emotion recognition. Proceedings of the AVSP, Norwich, UK."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1007\/s12652-016-0406-z","article-title":"CHEAVD: A Chinese natural emotional audio\u2013visual database","volume":"8","author":"Li","year":"2017","journal-title":"J. Ambient. Intell. Humaniz. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1109\/TAFFC.2015.2392101","article-title":"Speech emotion recognition using Fourier parameters","volume":"6","author":"Wang","year":"2015","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_18","unstructured":"Schuller, B., Rigoll, G., and Lang, M. (2004, January 17\u201321). Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture. Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, QC, Canada."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1007\/s10772-014-9251-7","article-title":"Talking condition recognition in stressful and emotional talking environments based on CSPHMM2s","volume":"18","author":"Shahin","year":"2015","journal-title":"Int. J. Speech Technol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hansen, J.H., and Bou-Ghazale, S.E. (1997, January 22\u201325). Getting started with SUSAS: A speech under simulated and actual stress database. Proceedings of the Fifth European Conference on Speech Communication and Technology, Rhodes, Greece.","DOI":"10.21437\/Eurospeech.1997-494"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1109\/TASLP.2017.2759338","article-title":"Semisupervised autoencoders for speech emotion recognition","volume":"26","author":"Deng","year":"2017","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_22","unstructured":"Huang, A., and Bao, P. (2019). Human vocal sentiment analysis. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0196391"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.bspc.2018.08.035","article-title":"Speech emotion recognition using deep 1D & 2D CNN LSTM networks","volume":"47","author":"Zhao","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"104886","DOI":"10.1016\/j.knosys.2019.104886","article-title":"Bagged support vector machines for emotion recognition from speech","volume":"184","author":"Bhavan","year":"2019","journal-title":"Knowl.-Based Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"125868","DOI":"10.1109\/ACCESS.2019.2938007","article-title":"Speech emotion recognition from 3D log-mel spectrograms with deep learning network","volume":"7","author":"Meng","year":"2019","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"90368","DOI":"10.1109\/ACCESS.2019.2927384","article-title":"Parallelized convolutional recurrent neural network with spectral features for speech emotion recognition","volume":"7","author":"Jiang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Schuller, B., Arsic, D., Rigoll, G., Wimmer, M., and Radig, B. (2007, January 15\u201320). Audiovisual behavior modeling by combined feature spaces. Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP\u201907, Honolulu, HI, USA.","DOI":"10.1109\/ICASSP.2007.366340"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"96994","DOI":"10.1109\/ACCESS.2020.2991811","article-title":"Emotion recognition from speech using wavelet packet transform cochlear filter bank and random forest classifier","volume":"8","author":"Hamsa","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2575","DOI":"10.1007\/s00521-018-3760-2","article-title":"Novel cascaded Gaussian mixture model-deep neural network classifier for speaker identification in emotional talking environments","volume":"32","author":"Shahin","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"26777","DOI":"10.1109\/ACCESS.2019.2901352","article-title":"Emotion recognition using hybrid Gaussian mixture model and deep neural network","volume":"7","author":"Shahin","year":"2019","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"119871","DOI":"10.1016\/j.eswa.2023.119871","article-title":"Speaker identification from emotional and noisy speech using learned voice segregation and speech VGG","volume":"224","author":"Hamsa","year":"2023","journal-title":"Expert Syst. Appl."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"215851","DOI":"10.1109\/ACCESS.2020.3038493","article-title":"Speech Emotion Recognition by Combining a Unified First-Order Attention Network with Data Balance","volume":"8","author":"Chen","year":"2020","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1109\/TAFFC.2016.2553038","article-title":"BAUM-1: A spontaneous audio-visual face database of affective and mental states","volume":"8","author":"Zhalehpour","year":"2016","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Dhall, A., Ramana Murthy, O., Goecke, R., Joshi, J., and Gedeon, T. (2015, January 9\u201313). Video and image based emotion recognition challenges in the wild: Emotiw 2015. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2829994"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Li, Y., Tao, J., Schuller, B., Shan, S., Jiang, D., and Jia, J. (2018, January 20\u201322). Mec 2017: Multimodal emotion recognition challenge. Proceedings of the 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), Beijing, China.","DOI":"10.1109\/ACIIAsia.2018.8470342"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"16560","DOI":"10.1109\/ACCESS.2020.2967791","article-title":"Speech emotion recognition using 3d convolutions and attention-based sliding recurrent networks with auditory front-ends","volume":"8","author":"Peng","year":"2020","journal-title":"IEEE Access"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/TAFFC.2016.2515617","article-title":"MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception","volume":"8","author":"Busso","year":"2016","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"222533","DOI":"10.1109\/ACCESS.2020.3043894","article-title":"Exploration of an Independent Training Framework for Speech Emotion Recognition","volume":"8","author":"Zhong","year":"2020","journal-title":"IEEE Access"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"23496","DOI":"10.1109\/ACCESS.2020.2969032","article-title":"Learning deep binaural representations with deep convolutional neural networks for spontaneous speech emotion recognition","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"116080","DOI":"10.1016\/j.eswa.2021.116080","article-title":"Novel dual-channel long short-term memory compressed capsule networks for emotion recognition","volume":"188","author":"Shahin","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_44","first-page":"1","article-title":"Dynamic routing between capsules","volume":"30","author":"Sabour","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Eyben, F., W\u00f6llmer, M., and Schuller, B. (2010, January 25\u201329). Opensmile: The munich versatile and fast open-source audio feature extractor. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.","DOI":"10.1145\/1873951.1874246"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Latif, S., Rana, R., Younis, S., Qadir, J., and Epps, J. (2018). Transfer learning for improving speech emotion classification accuracy. arXiv.","DOI":"10.21437\/Interspeech.2018-1625"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"9","DOI":"10.3389\/fcomp.2020.00009","article-title":"A review of generalizable transfer learning in automatic emotion recognition","volume":"2","author":"Feng","year":"2020","journal-title":"Front. Comput. Sci."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Schuller, B., Steidl, S., and Batliner, A. (2009, January 6\u201310). The interspeech 2009 emotion challenge. Proceedings of the Tenth Annual Conference of the International Speech Communication Association, Brighton, UK.","DOI":"10.21437\/Interspeech.2009-103"},{"key":"ref_50","unstructured":"Yao, Q. (2016). Multi-Sensory Emotion Recognition with Speech and Facial Expression. [Ph.D. Thesis, The University of Southern Mississippi]."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Ghifary, M., Kleijn, W.B., and Zhang, M. (2014, January 1\u20135). Domain adaptive neural networks for object recognition. Proceedings of the Pacific Rim International Conference on Artificial Intelligence, Gold Coast, QLD, Australia.","DOI":"10.1007\/978-3-319-13560-1_76"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Sawada, Y., and Kozuka, K. (2015, January 18\u201322). Transfer learning method using multi-prediction deep Boltzmann machines for a small scale dataset. Proceedings of the 2015 14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, Japan.","DOI":"10.1109\/MVA.2015.7153145"},{"key":"ref_53","unstructured":"Latif, S., Rana, R., Younis, S., Qadir, J., and Epps, J. (2018). Cross corpus speech emotion classification-an effective transfer learning technique. arXiv."},{"key":"ref_54","first-page":"1","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21\u201326). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.106"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_57","unstructured":"Hogg, R.V., McKean, J., and Craig, A.T. (2005). Introduction to Mathematical Statistics, Pearson Education."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Gao, Y., Li, B., Wang, N., and Zhu, T. (2017, January 16\u201318). Speech emotion recognition using local and global features. Proceedings of the International Conference on Brain Informatics, Beijing, China.","DOI":"10.1007\/978-3-319-70772-3_1"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Zhang, B., Essl, G., and Provost, E.M. (2015, January 21\u201324). Recognizing emotion from singing and speaking using shared models. Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi\u2019an, China.","DOI":"10.1109\/ACII.2015.7344563"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.engappai.2004.08.035","article-title":"A genetic classification method for speaker recognition","volume":"18","author":"Hong","year":"2005","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1109\/TSA.2005.853206","article-title":"Real-time speaker identification and verification","volume":"14","author":"Kinnunen","year":"2005","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"87995","DOI":"10.1109\/ACCESS.2021.3086062","article-title":"An Enhanced Emotion Recognition Algorithm Using Pitch Correlogram, Deep Sparse Matrix Representation and Random Forest Classifier","volume":"9","author":"Hamsa","year":"2021","journal-title":"IEEE Access"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.csl.2005.06.003","article-title":"Support vector machines for speaker and language recognition","volume":"20","author":"Campbell","year":"2006","journal-title":"Comput. Speech Lang."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/4\/236\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:14:24Z","timestamp":1760123664000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/4\/236"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,12]]},"references-count":63,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["info14040236"],"URL":"https:\/\/doi.org\/10.3390\/info14040236","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2023,4,12]]}}}