{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T20:05:53Z","timestamp":1776456353965,"version":"3.51.2"},"reference-count":49,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T00:00:00Z","timestamp":1662595200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>Emotion recognition has become one of the most researched subjects in the scientific community, especially in the human\u2013computer interface field. Decades of scientific research have been conducted on unimodal emotion analysis, whereas recent contributions concentrate on multimodal emotion recognition. These efforts have achieved great success in terms of accuracy in diverse areas of Deep Learning applications. To achieve better performance for multimodal emotion recognition systems, we exploit Meaningful Neural Network Effectiveness to enable emotion prediction during a conversation. Using the text and the audio modalities, we proposed feature extraction methods based on Deep Learning. Then, the bimodal modality that is created following the fusion of the text and audio features is used. The feature vectors from these three modalities are assigned to feed a Meaningful Neural Network to separately learn each characteristic. Its architecture consists of a set of neurons for each component of the input vector before combining them all together in the last layer. Our model was evaluated on a multimodal and multiparty dataset for emotion recognition in conversation MELD. The proposed approach reached an accuracy of 86.69%, which significantly outperforms all current multimodal systems. To sum up, several evaluation techniques applied to our work demonstrate the robustness and superiority of our model over other state-of-the-art MELD models.<\/jats:p>","DOI":"10.3390\/bdcc6030095","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T20:50:27Z","timestamp":1662670227000},"page":"95","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Multimodal Emotional Classification Based on Meaningful Learning"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7623-2113","authenticated-orcid":false,"given":"Hajar","family":"Filali","sequence":"first","affiliation":[{"name":"Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jamal","family":"Riffi","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chafik","family":"Boulealam","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohamed Adnane","family":"Mahraz","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hamid","family":"Tairi","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,8]]},"reference":[{"key":"ref_1","unstructured":"(2012). Definition of \u2018Emotion\u2019. Merriam-Webster Dictionary, Merriam-Webster."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8316","DOI":"10.1109\/TIP.2020.3011846","article-title":"Facial Expression Recognition in Videos Using Dynamic Kernels","volume":"29","author":"Perveen","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chen, L., Ouyang, Y., Zeng, Y., and Li, Y. (2020, January 18\u201322). Dynamic Facial Expression Recognition Model Based on BiLSTM-Attention. Proceedings of the 2020 15th International Conference on Computer Science & Education (ICCSE), IEEE, Delft, The Netherlands.","DOI":"10.1109\/ICCSE49874.2020.9201892"},{"key":"ref_4","first-page":"349","article-title":"Social Media Networks Security Threats, Risks and Recommendation: A Case Study in the Kurdistan Region","volume":"13","author":"Zeebaree","year":"2020","journal-title":"Int. J. Innov. Creat. Change"},{"key":"ref_5","first-page":"498","article-title":"Real Time Implementation of Stegofirewall System","volume":"8","author":"Ameen","year":"2019","journal-title":"Int. J. Comput. Digit. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Baimbetov, Y., Khalil, I., Steinbauer, M., and Anderst-Kotsis, G. (2014, January 25\u201327). Using Big Data for Emotionally Intelligent Mobile Services through Multi-Modal Emotion Recognition. Proceedings of the International Conference on Smart Homes and Health Telematics, Denver, CO, USA.","DOI":"10.1007\/978-3-319-19312-0_11"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1023\/A:1013365332180","article-title":"Modeling Multimodal Expression of User\u2019s Affective Subjective Experience","volume":"12","author":"Lisetti","year":"2002","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"52","DOI":"10.38094\/jastt20291","article-title":"Multimodal Emotion Recognition Using Deep Learning","volume":"2","author":"Abdullah","year":"2021","journal-title":"J. Appl. Sci. Technol. Trends"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"25241","DOI":"10.1007\/s11042-021-10918-9","article-title":"Human Emotion Recognition Based on Facial Expressions via Deep Learning on High-Resolution Images","volume":"80","author":"Said","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s10462-012-9368-5","article-title":"Features and Classifiers for Emotion Recognition from Speech: A Survey from 2000 to 2011","volume":"43","author":"Anagnostopoulos","year":"2015","journal-title":"Artif. Intell. Rev."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1026","DOI":"10.3390\/covid2080076","article-title":"An Exploratory Study of Tweets about the SARS-CoV-2 Omicron Variant: Insights from Sentiment Analysis, Language Interpretation, Source Tracking, Type Classification, and Embedded URL Detection","volume":"2","author":"Thakur","year":"2022","journal-title":"COVID"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1109\/TAFFC.2017.2714671","article-title":"Emotions Recognition Using EEG Signals: A Survey","volume":"10","author":"Alarcao","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_13","unstructured":"Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (August, January 28). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_14","unstructured":"Chen, S.-Y., Hsu, C.-C., Kuo, C.-C., and Ku, L.-W. (2018). EmotionLines: An Emotion Corpus of Multi-Party Conversations. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive Emotional Dyadic Motion Capture Database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref_16","unstructured":"Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P. (2016). MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Choi, W.Y., Song, K.Y., and Lee, C.W. (2018, January 20). Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data. Proceedings of the Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), Melbourne, Australia.","DOI":"10.18653\/v1\/W18-3304"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1109\/JSTSP.2017.2764438","article-title":"End-to-End Multimodal Emotion Recognition Using Deep Neural Networks","volume":"11","author":"Tzirakis","year":"2017","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12\u201315). Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), IEEE, Barcelona, Spain.","DOI":"10.1109\/ICDM.2016.0055"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Maat, L., and Pantic, M. (2007). Gaze-X: Adaptive, Affective, Multimodal Interface for Single-User Office Scenarios. Artifical Intelligence for Human Computing, Proceedings of the 8th International Conference on Multimodal Interfaces, Banff, AB, Canada, 2\u20134 November 2006, Springer.","DOI":"10.1145\/1180995.1181032"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Su, Q., Chen, F., Li, H., Yan, N., and Wang, L. (2018, January 3\u20136). Multimodal Emotion Perception in Children with Autism Spectrum Disorder by Eye Tracking Study. Proceedings of the 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), IEEE, Sarawak, Malaysia.","DOI":"10.1109\/IECBES.2018.8626642"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"172948","DOI":"10.1109\/ACCESS.2019.2955637","article-title":"A Hybrid Latent Space Data Fusion Method for Multimodal Emotion Recognition","volume":"7","author":"Nemati","year":"2019","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Prasad, G., Dikshit, A., and Lalitha, S. (2021, January 26\u201327). Sentiment and Emotion Analysis for Effective Human-Machine Interaction during Covid-19 Pandemic. Proceedings of the 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN), IEEE, Noida, India.","DOI":"10.1109\/SPIN52536.2021.9566147"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Priyasad, D., Fernando, T., Denman, S., Sridharan, S., and Fookes, C. (2020, January 4\u20138). Attention Driven Fusion for Multi-Modal Emotion Recognition. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9054441"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sun, B., Li, L., Zhou, G., Wu, X., He, J., Yu, L., Li, D., and Wei, Q. (2015, January 9\u201313). Combining Multimodal Features within a Fusion Network for Emotion Recognition in the Wild. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830586"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Schuller, B., Valster, M., Eyben, F., Cowie, R., and Pantic, M. (2012, January 22\u201326). Avec 2012: The Continuous Audio\/Visual Emotion Challenge. Proceedings of the 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, USA.","DOI":"10.1145\/2388676.2388776"},{"key":"ref_27","unstructured":"Cevher, D., Zepf, S., and Klinger, R. (2019). Towards Multimodal Emotion Recognition in German Speech Events in Cars Using Transfer Learning. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Georgiou, E., Papaioannou, C., and Potamianos, A. (2019, January 15\u201319). Deep Hierarchical Fusion with Application in Sentiment Analysis. Proceedings of the Interspeech, Graz, Austria.","DOI":"10.21437\/Interspeech.2019-3243"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1080\/10447318.2016.1159799","article-title":"Data Fusion for Real-Time Multimodal Emotion Recognition through Webcams and Microphones in e-Learning","volume":"32","author":"Bahreini","year":"2016","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_30","first-page":"41","article-title":"Towards Emotion Recognition in Texts\u2013a Sound-Symbolic Experiment","volume":"7","author":"Slavova","year":"2019","journal-title":"Int. J. Cogn. Res. Sci. Eng. Educ. (IJCRSEE)"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Pan, Z., Luo, Z., Yang, J., and Li, H. (2020). Multi-Modal Attention for Speech Emotion Recognition. arXiv.","DOI":"10.21437\/Interspeech.2020-1653"},{"key":"ref_32","unstructured":"Krishna, D.N., and Patil, A. (2020, January 25\u201329). Multimodal Emotion Recognition Using Cross-Modal Attention and 1D Convolutional Neural Networks. Proceedings of the Interspeech, Shanghai, China."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Huang, J., Li, Y., Tao, J., Lian, Z., Wen, Z., Yang, M., and Yi, J. (2017, January 23\u201327). Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network. Proceedings of the 7th Annual Workshop on Audio\/Visual Emotion Challenge, Mountain View, CA, USA.","DOI":"10.1145\/3133944.3133946"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1109\/TASLP.2021.3049898","article-title":"CTNet: Conversational Transformer Network for Emotion Recognition","volume":"29","author":"Lian","year":"2021","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"107751","DOI":"10.1016\/j.knosys.2021.107751","article-title":"A Multi-View Network for Real-Time Emotion Recognition in Conversations","volume":"236","author":"Ma","year":"2022","journal-title":"Knowl. Based Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"176274","DOI":"10.1109\/ACCESS.2020.3026823","article-title":"Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature Fusion","volume":"8","author":"Siriwardhana","year":"2020","journal-title":"IEEE Access"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Xie, B., Sidulova, M., and Park, C.H. (2021). Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion. Sensors, 21.","DOI":"10.3390\/s21144913"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-Based Learning Applied to Document Recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning to Forget: Continual Prediction with LSTM","volume":"12","author":"Gers","year":"2000","journal-title":"Neural Comput."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1007\/s11063-021-10636-1","article-title":"Meaningful Learning for Deep Facial Emotional Features","volume":"54","author":"Filali","year":"2022","journal-title":"Neural Process. Lett."},{"key":"ref_42","unstructured":"Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L.-P. (August, January 30). Context-Dependent Sentiment Analysis in User-Generated Videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Eyben, F., W\u00f6llmer, M., and Schuller, B. (2010, January 25\u201329). Opensmile: The Munich Versatile and Fast Open-Source Audio Feature Extractor. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.","DOI":"10.1145\/1873951.1874246"},{"key":"ref_45","first-page":"659","article-title":"Gaussian Mixture Models","volume":"741","author":"Reynolds","year":"2009","journal-title":"Encycl. Biom."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1016\/S0959-440X(96)80056-X","article-title":"Hidden Markov Models","volume":"6","author":"Eddy","year":"1996","journal-title":"Curr. Opin. Struct. Biol."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wang, S.-C. (2003). Artificial Neural Network. Interdisciplinary Computing in Java Programming, Springer.","DOI":"10.1007\/978-1-4615-0377-4"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.1038\/nbt1206-1565","article-title":"What Is a Support Vector Machine?","volume":"24","author":"Noble","year":"2006","journal-title":"Nat. Biotechnol."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Schapire, R.E. (2013). Explaining Adaboost. Empirical Inference, Springer.","DOI":"10.1007\/978-3-642-41136-6_5"}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/6\/3\/95\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:27:49Z","timestamp":1760142469000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/6\/3\/95"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,8]]},"references-count":49,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["bdcc6030095"],"URL":"https:\/\/doi.org\/10.3390\/bdcc6030095","relation":{},"ISSN":["2504-2289"],"issn-type":[{"value":"2504-2289","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,8]]}}}