{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T11:17:27Z","timestamp":1784114247201,"version":"3.55.0"},"reference-count":46,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T00:00:00Z","timestamp":1724544000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of Hubei Province project","award":["2022CFB529"],"award-info":[{"award-number":["2022CFB529"]}]},{"name":"National Natural Science Foundation of Hubei Province project","award":["2022CFB971"],"award-info":[{"award-number":["2022CFB971"]}]},{"name":"National Natural Science Foundation of Hubei Province project","award":["20232BAB212026"],"award-info":[{"award-number":["20232BAB212026"]}]},{"name":"National Natural Science Foundation of Hubei Province project","award":["JXJG-23-27-6"],"award-info":[{"award-number":["JXJG-23-27-6"]}]},{"name":"National Natural Science Foundation of Hubei Province project","award":["JCYJ20230807152900001"],"award-info":[{"award-number":["JCYJ20230807152900001"]}]},{"name":"Jiangxi Provincial Natural Science Foundation","award":["2022CFB529"],"award-info":[{"award-number":["2022CFB529"]}]},{"name":"Jiangxi Provincial Natural Science Foundation","award":["2022CFB971"],"award-info":[{"award-number":["2022CFB971"]}]},{"name":"Jiangxi Provincial Natural Science Foundation","award":["20232BAB212026"],"award-info":[{"award-number":["20232BAB212026"]}]},{"name":"Jiangxi Provincial Natural Science Foundation","award":["JXJG-23-27-6"],"award-info":[{"award-number":["JXJG-23-27-6"]}]},{"name":"Jiangxi Provincial Natural Science Foundation","award":["JCYJ20230807152900001"],"award-info":[{"award-number":["JCYJ20230807152900001"]}]},{"name":"university teaching reform research project of Jiangxi Province","award":["2022CFB529"],"award-info":[{"award-number":["2022CFB529"]}]},{"name":"university teaching reform research project of Jiangxi Province","award":["2022CFB971"],"award-info":[{"award-number":["2022CFB971"]}]},{"name":"university teaching reform research project of Jiangxi Province","award":["20232BAB212026"],"award-info":[{"award-number":["20232BAB212026"]}]},{"name":"university teaching reform research project of Jiangxi Province","award":["JXJG-23-27-6"],"award-info":[{"award-number":["JXJG-23-27-6"]}]},{"name":"university teaching reform research project of Jiangxi Province","award":["JCYJ20230807152900001"],"award-info":[{"award-number":["JCYJ20230807152900001"]}]},{"name":"Shenzhen Science and Technology Program","award":["2022CFB529"],"award-info":[{"award-number":["2022CFB529"]}]},{"name":"Shenzhen Science and Technology Program","award":["2022CFB971"],"award-info":[{"award-number":["2022CFB971"]}]},{"name":"Shenzhen Science and Technology Program","award":["20232BAB212026"],"award-info":[{"award-number":["20232BAB212026"]}]},{"name":"Shenzhen Science and Technology Program","award":["JXJG-23-27-6"],"award-info":[{"award-number":["JXJG-23-27-6"]}]},{"name":"Shenzhen Science and Technology Program","award":["JCYJ20230807152900001"],"award-info":[{"award-number":["JCYJ20230807152900001"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Speech emotion recognition (SER) is not only a ubiquitous aspect of everyday communication, but also a central focus in the field of human\u2013computer interaction. However, SER faces several challenges, including difficulties in detecting subtle emotional nuances and the complicated task of recognizing speech emotions in noisy environments. To effectively address these challenges, we introduce a Transformer-based model called MelTrans, which is designed to distill critical clues from speech data by learning core features and long-range dependencies. At the heart of our approach is a dual-stream framework. Using the Transformer architecture as its foundation, MelTrans deciphers broad dependencies within speech mel-spectrograms, facilitating a nuanced understanding of emotional cues embedded in speech signals. Comprehensive experimental evaluations on the EmoDB (92.52%) and IEMOCAP (76.54%) datasets demonstrate the effectiveness of MelTrans. These results highlight MelTrans\u2019s ability to capture critical cues and long-range dependencies in speech data, setting a new benchmark within the context of these specific datasets. These results highlight the effectiveness of the proposed model in addressing the complex challenges posed by SER tasks.<\/jats:p>","DOI":"10.3390\/s24175506","type":"journal-article","created":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T03:32:01Z","timestamp":1724643121000},"page":"5506","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["MelTrans: Mel-Spectrogram Relationship-Learning for Speech Emotion Recognition via Transformers"],"prefix":"10.3390","volume":"24","author":[{"given":"Hui","family":"Li","sequence":"first","affiliation":[{"name":"School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China"},{"name":"National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiawen","family":"Li","sequence":"additional","affiliation":[{"name":"National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hai","family":"Liu","sequence":"additional","affiliation":[{"name":"National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0348-949X","authenticated-orcid":false,"given":"Tingting","family":"Liu","sequence":"additional","affiliation":[{"name":"National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Chen","sequence":"additional","affiliation":[{"name":"National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinge","family":"You","sequence":"additional","affiliation":[{"name":"School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1080\/07370024.2020.1724790","article-title":"User Representations in Human-Computer Interaction","volume":"36","author":"Seinfeld","year":"2020","journal-title":"Hum. -Comput. Interact."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"121368","DOI":"10.1016\/j.eswa.2023.121368","article-title":"Semi-supervised cross-lingual speech emotion recognition","volume":"237","author":"Agarla","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Gao, R., and Grauman, K. (2021, January 20\u201325). Visualvoice: Audio-visual speech separation with cross-modal consistency. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01524"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/j.ipm.2008.09.003","article-title":"Acoustic feature selection for automatic emotion recognition from speech","volume":"45","author":"Rong","year":"2009","journal-title":"Inf. Process. Manag."},{"key":"ref_5","first-page":"10","article-title":"Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels","volume":"2","author":"Wu","year":"2010","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TMM.2010.2058095","article-title":"Speech emotion analysis: Exploring the role of context","volume":"12","author":"Tawari","year":"2010","journal-title":"IEEE Trans. Multimed."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1023\/A:1023426522496","article-title":"Context-independent multilingual emotion recognition from speech signals","volume":"6","author":"Hozjan","year":"2003","journal-title":"Int. J. Speech Technol."},{"key":"ref_8","unstructured":"Doulamis, N. (2006, January 4\u20138). An adaptable emotionally rich pervasive computing system. Proceedings of the 2006 14th European Signal Processing Conference, Florence, Italy."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"113816","DOI":"10.1016\/j.eswa.2020.113816","article-title":"Self-driving cars: A survey","volume":"165","author":"Badue","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bekmanova, G., Yergesh, B., Sharipbay, A., and Mukanova, A. (2022). Emotional Speech Recognition Method Based on Word Transcription. Sensors, 22.","DOI":"10.3390\/s22051937"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s10772-022-09983-8","article-title":"Hybrid end-to-end model for Kazakh speech recognition","volume":"26","author":"Mamyrbayev","year":"2023","journal-title":"Int. J. Speech Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/JSTSP.2019.2955012","article-title":"Automatic Assessment of Depression From Speech via a Hierarchical Attention Transfer Network and Attention Autoencoders","volume":"14","author":"Zhao","year":"2020","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"127271","DOI":"10.1109\/ACCESS.2023.3329678","article-title":"Deep learning in EEG-based BCIs: A comprehensive review of transformer models, advantages, challenges, and applications","volume":"11","author":"Abibullaev","year":"2023","journal-title":"IEEE Access"},{"key":"ref_14","unstructured":"Liu, J., and Wang, H. (September, January 30). Graph Isomorphism Network for Speech Emotion Recognition. Proceedings of the Interspeech, Brno, Czechia."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1109\/TASLP.2023.3245401","article-title":"A Discriminative Feature Representation Method Based on Cas-caded Attention Network with Adversarial Strategy for Speech Emotion Recognition","volume":"31","author":"Liu","year":"2023","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201323). Squeeze-and-Excitation Networks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1109\/TASLP.2022.3145293","article-title":"MsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech Synthesis","volume":"30","author":"Lei","year":"2022","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Makiuchi, M.R., Uto, K., and Shinoda, K. (2021, January 13\u201317). Multimodal emotion recognition with high-level speech and text features. Proceedings of the 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Cartagena, Colombia.","DOI":"10.1109\/ASRU51503.2021.9688036"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Li, Y., Zhang, S., Wang, Z., Yang, S., Yang, W., Xia, S.-T., and Zhou, E. (2021, January 11\u201317). Tokenpose: Learning keypoint tokens for human pose estimation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.01112"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"118943","DOI":"10.1016\/j.eswa.2022.118943","article-title":"Learning multi-scale features for speech emotion recognition with connection attention mechanism","volume":"214","author":"Chen","year":"2023","journal-title":"Expert Syst. Appl."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1627","DOI":"10.1109\/TAFFC.2021.3109485","article-title":"Few-shot learning in emotion recognition of spontaneous speech using a siamese neural network with adaptive sample pair formation","volume":"14","author":"Feng","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_24","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bekmanova, G., Yelibayeva, G., Yergesh, B., Orynbay, L., Sairanbekova, A., and Kaderkeyeva, Z. (2022, January 17\u201320). Emotional Coloring of Kazakh People\u2019s Names in the Semantic Knowledge Database of \u201cFascinating Onomastics\u201d Mobile Application. Proceedings of the 2022 IEEE\/WIC\/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Niagara Falls, ON, Canada.","DOI":"10.1109\/WI-IAT55865.2022.00105"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"112512","DOI":"10.1016\/j.chaos.2022.112512","article-title":"MFF-SAug: Multi feature fusion with spectrogram augmentation of speech emotion recognition using convolution neural network","volume":"162","author":"Jothimani","year":"2022","journal-title":"Chaos Solitons Fractals"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., and Weiss, B. (2005, January 4\u20138). A database of German emotional speech. Proceedings of the Interspeech, Lisbon, Portugal.","DOI":"10.21437\/Interspeech.2005-446"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1055","DOI":"10.1109\/TAFFC.2019.2916092","article-title":"Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)","volume":"12","author":"Gideon","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1109\/TCSS.2022.3228649","article-title":"RobinNet: A Multimodal Speech Emotion Recognition System With Speaker Recognition for Social Interactions","volume":"11","author":"Khurana","year":"2024","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1109\/TASLP.2024.3363444","article-title":"METTS: Multilingual Emotional Text-to-Speech by Cross-Speaker and Cross-Lingual Emotion Transfer","volume":"32","author":"Zhu","year":"2024","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"6472","DOI":"10.1109\/TCSVT.2022.3163445","article-title":"Temporal Relation Inference Network for Multimodal Speech Emotion Recognition","volume":"32","author":"Dong","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"119511","DOI":"10.1016\/j.eswa.2023.119511","article-title":"MSRANet: Learning discriminative embeddings for speaker verification via channel and spatial attention mechanism in alterable scenarios","volume":"217","author":"Zheng","year":"2023","journal-title":"Expert Syst. Appl."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zou, H., Si, Y., Chen, C., Rajan, D., and Chng, E.S. (2022, January 23\u201327). Speech Emotion Recognition with Co-Attention Based Multi-Level Acoustic Information. Proceedings of the ICASSP 2022\u20142022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.","DOI":"10.1109\/ICASSP43922.2022.9747095"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1109\/TASLP.2023.3235194","article-title":"SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing","volume":"31","author":"Chen","year":"2023","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_36","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"114177","DOI":"10.1016\/j.eswa.2020.114177","article-title":"MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach","volume":"167","author":"Kwon","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Thanh, P.V., Huyen, N.T.T., Quan, P.N., and Trang, N.T.T. (2024, January 14\u201319). A Robust Pitch-Fusion Model for Speech Emotion Recognition in Tonal Languages. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.","DOI":"10.1109\/ICASSP48485.2024.10448373"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2193","DOI":"10.1109\/TASLP.2023.3282092","article-title":"Dual-TBNet: Improving the Robustness of Speech Features via Dual-Transformer-BiLSTM for Speech Emotion Recognition","volume":"31","author":"Liu","year":"2023","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"10745","DOI":"10.1109\/TPAMI.2023.3263585","article-title":"Dawn of the transformer era in speech emotion recognition: Closing the valence gap","volume":"45","author":"Wagner","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/LSP.2022.3219352","article-title":"Speech Emotion Recognition via Multi-Level Attention Network","volume":"29","author":"Liu","year":"2022","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2189217","DOI":"10.1080\/09540091.2023.2189217","article-title":"Multi-branch feature learning based speech emotion recognition using SCAR-NET","volume":"35","author":"Mao","year":"2023","journal-title":"Connect. Sci."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Shen, S., Gao, Y., Liu, F., Wang, H., and Zhou, A. (2024, January 14\u201319). Emotion Neural Transducer for Fine-Grained Speech Emotion Recognition. Proceedings of the ICASSP 2024\u20142024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.","DOI":"10.1109\/ICASSP48485.2024.10446974"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"776","DOI":"10.1109\/TMM.2023.3271019","article-title":"A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations","volume":"26","author":"Ma","year":"2024","journal-title":"IEEE Trans. Multimed."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1564","DOI":"10.1109\/TCDS.2021.3123979","article-title":"Convolutional-Recurrent Neural Networks With Multiple Attention Mechanisms for Speech Emotion Recognition","volume":"14","author":"Jiang","year":"2022","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"108293","DOI":"10.1016\/j.engappai.2024.108293","article-title":"Newman-Watts-Strogatz topology in deep echo state networks for speech emotion recognition","volume":"133","author":"Soltani","year":"2024","journal-title":"Eng. Appl. Artif. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/17\/5506\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:42:33Z","timestamp":1760110953000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/17\/5506"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,25]]},"references-count":46,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["s24175506"],"URL":"https:\/\/doi.org\/10.3390\/s24175506","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,25]]}}}