{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,12]],"date-time":"2026-07-12T03:23:51Z","timestamp":1783826631671,"version":"3.55.0"},"reference-count":113,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,4,12]],"date-time":"2022-04-12T00:00:00Z","timestamp":1649721600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Emotion classification is a research area in which there has been very intensive literature production concerning natural language processing, multimedia data, semantic knowledge discovery, social network mining, and text and multimedia data mining. This paper addresses the issue of emotion classification and proposes a method for classifying the emotions expressed in multimodal data extracted from videos. The proposed method models multimodal data as a sequence of features extracted from facial expressions, speech, gestures, and text, using a linguistic approach. Each sequence of multimodal data is correctly associated with the emotion by a method that models each emotion using a hidden Markov model. The trained model is evaluated on samples of multimodal sentences associated with seven basic emotions. The experimental results demonstrate a good classification rate for emotions.<\/jats:p>","DOI":"10.3390\/mti6040028","type":"journal-article","created":{"date-parts":[[2022,4,12]],"date-time":"2022-04-12T22:48:45Z","timestamp":1649803725000},"page":"28","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Emotion Classification from Speech and Text in Videos Using a Multimodal Approach"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3580-0505","authenticated-orcid":false,"given":"Maria Chiara","family":"Caschera","sequence":"first","affiliation":[{"name":"National Research Council, Institute of Research on Population and Social Policies (CNR-IRPPS), Via Palestro 32, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4298-5437","authenticated-orcid":false,"given":"Patrizia","family":"Grifoni","sequence":"additional","affiliation":[{"name":"National Research Council, Institute of Research on Population and Social Policies (CNR-IRPPS), Via Palestro 32, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fernando","family":"Ferri","sequence":"additional","affiliation":[{"name":"National Research Council, Institute of Research on Population and Social Policies (CNR-IRPPS), Via Palestro 32, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Dalgleish, T., and Power, T. (1999). Basic emotions. Handbook of Cognition and Emotion, John Wiley & Sons.","DOI":"10.1002\/0470013494"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Plutchik, R. (2017, February 22). The Nature of Emotions; American Scientist Vol. 89, No. 4 (JULY-AUGUST 2001); Sigma Xi, The Scientific Research Honor Society. Available online: https:\/\/www.jstor.org\/stable\/27857503.","DOI":"10.1511\/2001.28.739"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1037\/0033-295X.110.1.145","article-title":"Core affect and the psychological construction of emotion","volume":"110","author":"Russell","year":"2003","journal-title":"Psychol. Rev."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1080\/09658210903130764","article-title":"A comparison of dimensional models of emotion: Evidence from emotions, prototypical events, autobiographical memories, and words","volume":"17","author":"Rubin","year":"2009","journal-title":"Memory"},{"key":"ref_5","unstructured":"Chen, D., and Mooney, R. (2011, January 27). Panning for gold: Finding relevant semantic content for grounded language learning. Proceedings of the Symposium Machine Learning in Speech and Language Processing, Bellevue, WA, USA. Available online: www.cs.utexas.edu\/~ml\/papers\/chen.mlslp11.pdf."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Paleari, M., Chellali, R., and Huet, B. (2010, January 1\u20132). Features for multimodal emotion recognition: An extensive study. Proceedings of the 2010 IEEE Conference on Cybernetics and Intelligent Systems (CIS), Berks, UK.","DOI":"10.1109\/ICCIS.2010.5518574"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1504\/IJWGS.2007.012638","article-title":"Multimodal interaction systems: Information and time features","volume":"3","author":"Caschera","year":"2007","journal-title":"Int. J. Web Grid Serv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Caschera, M.C., Ferri, F., and Grifoni, P. (2016, January 1\u20134). Sentiment analysis from textual to multimodal features in digital environments. Proceedings of the 8th International Conference on Management of Digital EcoSystems (MEDES), ACM, New York, NY, USA.","DOI":"10.1145\/3012071.3012089"},{"key":"ref_9","unstructured":"Lee, S., and Narayanan, S. (2008, January 15\u201317). Audio-visual emotion recognition using Gaussian mixture models for face and voice. Proceedings of the IEEE International Symposium on Multimedia, Berkeley, CA, USA."},{"key":"ref_10","unstructured":"Grifoni, P. (2009). Interpretation methods and ambiguity management in multimodal systems. Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services: Evolutionary Techniques for Improving Accessibility, IGI Global."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Tepperman, J., Traum, D., and Narayanan, S. (2006, January 17\u201321). Yeah right: Sarcasm recognition for spoken dialogue systems. Proceedings of the InterSpeech-ICSLP, Pittsburgh, PA, USA.","DOI":"10.21437\/Interspeech.2006-507"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Voeller, J.G. (2009). Human behavior and deception detection. Handbook of Science and Technology for Homeland Security, John Wiley & Sons.","DOI":"10.1002\/9780470087923"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Abouelenien, M., Perez-Rosas, V., Mihalcea, R., and Burzo, M. (2014, January 12\u201316). Deception detection using a multimodal approach. Proceedings of the 16th ACM International Conference on Multimodal Interaction, ICMI 2014, Istanbul, Turkey.","DOI":"10.1145\/2663204.2663229"},{"key":"ref_14","unstructured":"Ma, M.D. (2012). Methods of detecting potential terrorists at airports. Security Dimensions and Socio-Legal Studies, CEEOL."},{"key":"ref_15","first-page":"1449","article-title":"Facial expression recognition for security","volume":"2","author":"Butalia","year":"2012","journal-title":"Int. J. Mod. Eng. Res. (IJMER)"},{"key":"ref_16","unstructured":"Lim, T.B., Husin, M.H., Zaaba, Z.F., and Osman, M.A. (2015, January 11\u201313). Implementation of an automated smart home control for detecting human emotions via facial detection. Proceedings of the 5th International Conference on Computing and Informatics, ICOCI 2015, Istanbul, Turkey."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jocs.2010.12.007","article-title":"Twitter mood predicts the stock market","volume":"2","author":"Bollen","year":"2011","journal-title":"J. Comput. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Williamson, J.R., Quatieri, T.F., Helfer, B.S., Ciccarelli, G., and Mehta, D.D. (2014, January 7). Vocal and facial biomarkers of depression based on motor in coordination and timing. Proceedings of the 4th International Workshop on Audio\/Visual Emotion Challenge, Orlando, FL, USA.","DOI":"10.1145\/2661806.2661809"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/T-AFFC.2012.38","article-title":"Detecting depression severity from vocal prosody","volume":"4","author":"Yang","year":"2013","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sivasangari, A., Ajitha, P., Rajkumar, I., and Poonguzhali, S. (2019). Emotion recognition system for autism disordered people. J. Ambient Intell. Humaniz. Comput., 1\u20137.","DOI":"10.1007\/s12652-019-01492-y"},{"key":"ref_21","unstructured":"De Silva, L.C., Miyasato, T., and Nakatsu, R. (1997, January 12). Facial emotion recognition using multimodal information. Proceedings of the IEEE Int. Conf. on Information, Communications and Signal Processing (ICICS\u201997), Singapore."},{"key":"ref_22","unstructured":"Massaro, D.W. (1998, January 4\u20137). Illusions and issues in bimodal speech perception. Proceedings of the Auditory Visual Speech Perception\u201998, Sydney, Australia."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion recognition in human computer interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., and Narayanan, S. (2004, January 14\u201315). Analysis of emotion recognition using facial expressions, speech and multimodal information. Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI\u201904), State College, PA, USA.","DOI":"10.1145\/1027933.1027968"},{"key":"ref_25","unstructured":"Chen, L.S., Huang, T.S., Miyasato, T., and Nakatsu, R. (1998, January 14\u201316). Multimodal human emotion\/expression recognition. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1370","DOI":"10.1109\/JPROC.2003.817122","article-title":"Toward an affect-sensitive multimodal human-computer interaction","volume":"91","author":"Pantic","year":"2003","journal-title":"Proc. IEEE"},{"key":"ref_27","first-page":"282","article-title":"Sentiment analysis and opinion mining: A survey","volume":"2","author":"Vinodhini","year":"2012","journal-title":"Int. J. Adv. Res. Comput. Sci. Softw. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000011","article-title":"Opinion mining and sentiment analysis","volume":"2","author":"Pang","year":"2008","journal-title":"Found. Trends Inf. Retr."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1016\/j.asej.2014.04.011","article-title":"Sentiment analysis algorithms and applications: A survey","volume":"5","author":"Medhat","year":"2014","journal-title":"Ain Shams Eng. J."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Rustamov, S., Mustafayev, E., and Clements, M.A. (2013, January 4\u20137). Sentiment analysis using neuro-fuzzy and hidden Markov models of text. Proceedings of the IEEE Southeastcon 2013, Jacksonville, FL, USA.","DOI":"10.1109\/SECON.2013.6567382"},{"key":"ref_31","unstructured":"Kamps, J., Marx, M., Mokken, R., and Rijke, M. (2004, January 26\u201328). Using WordNet to measure semantic orientations of adjectives. Proceedings of the Fourth International Conference on Language Resources and Evaluation, Lisbon, Portugal."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wu, C., Shen, L., and Wang, X. (2009, January 7\u20138). A new method of using contextual information to infer the semantic orientations of context dependent opinions. Proceedings of the International Conference on Artificial Intelligence and Computational Intelligence, Shanghai, China.","DOI":"10.1109\/AICI.2009.406"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Peng, T.C., and Shih, C.C. (September, January 31). An unsupervised snippet-based sentiment classification method for Chinese unknown phrases without using reference word pairs. Proceedings of the 2010 IEEE\/WIC\/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Toronto, ON, Canada.","DOI":"10.1109\/WI-IAT.2010.229"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Li, G., and Liu, F. (2010, January 15\u201316). A clustering-based approach on sentiment analysis. Proceedings of the IEEE International Conference on Intelligent System and Knowledge Engineering, Hangzhou, China.","DOI":"10.1109\/ISKE.2010.5680859"},{"key":"ref_35","unstructured":"Adam, A., and Blockeel, H. (2015, January 7). Dealing with overlapping clustering: A constraint-based approach to algorithm selection. Proceedings of the 2015 International Conference on Meta-Learning and Algorithm Selection (MetaSel\u201915), Porto, Portugal."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"178","DOI":"10.22271\/allresearch.2021.v7.i4c.8484","article-title":"Hierarchical clustering: A Survey","volume":"7","author":"Shetty","year":"2021","journal-title":"Int. J. Appl. Res."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Maddah, M., Wells, W.M., Warfield, S.K., Westin, C.F., and Grimson, W.E. (2007, January 2\u20136). Probabilistic clustering and quantitative analysis of white matter fiber tracts. Proceedings of the 2007 Conference on Information Processing in Medical Imaging, Kerkrade, The Netherlands.","DOI":"10.1007\/978-3-540-73273-0_31"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Rodriguez, M.Z., Comin, C.H., Casanova, D., Bruno, O.M., Amancio, D.R., Costa, L.D.F., and Rodrigues, F. (2019). Clustering algorithms: A comparative approach. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0210236"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1109\/21.97458","article-title":"A survey of decision tree classifier methodology","volume":"21","author":"Safavian","year":"1991","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ruiz, M., and Srinivasan, P. (1999, January 15\u201319). Hierarchical neural networks for text categorization. Proceedings of the ACM SIGIR Conference 1999, Berkeley, CA, USA.","DOI":"10.1145\/312624.312700"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Liu, L., and \u00d6zsu, M.T. (2009). Rule-based classification. Encyclopedia of Database Systems, Springer.","DOI":"10.1007\/978-0-387-39940-9"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"De Raedt, L., and Flach, P. (2001). Understanding probabilistic classifiers. Machine Learning: ECML 2001, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/3-540-44795-4"},{"key":"ref_43","unstructured":"Melville, P., and Gryc, W. (July, January 28). Sentiment analysis of blogs by combining lexical knowledge with text classification. Proceedings of the KDD\u201909, Paris, France. ACM 978-1-60558-495-9\/09\/06."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Aggarwal, C.C., and Zhai, C.X. (2012). Mining Text Data, Springer Science + Business Media.","DOI":"10.1007\/978-1-4614-3223-4"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1016\/S1005-8885(09)60606-3","article-title":"Sentiment classification using the theory of ANNs","volume":"17","author":"Jian","year":"2010","journal-title":"J. China Univ. Posts Telecommun."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.eswa.2012.07.059","article-title":"Document-level sentiment classification: An empirical comparison between SVM and ANN","volume":"40","author":"Moraes","year":"2013","journal-title":"Expert Syst. Appl."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"6000","DOI":"10.1016\/j.eswa.2011.11.107","article-title":"Senti-lexicon and improved Na\u00efve Bayes algorithms for sentiment analysis of restaurant reviews","volume":"39","author":"Kang","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"7674","DOI":"10.1016\/j.eswa.2010.12.147","article-title":"Sentiment classification of Internet restaurant reviews written in Cantonese","volume":"38","author":"Zhang","year":"2011","journal-title":"Expert Syst. Appl."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Pang, B., Lee, L., and Vaithyanathan, S. (2002, January 6\u20137). Thumbs up? Sentiment classification using machine learning techniques. Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, Philadelphia, PA, USA.","DOI":"10.3115\/1118693.1118704"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"11","DOI":"10.5121\/ijsc.2014.5102","article-title":"Methodological study of opinion mining and sentiment analysis techniques","volume":"5","author":"Singh","year":"2014","journal-title":"Int. J. Soft Comput. (IJSC)"},{"key":"ref_51","first-page":"523","article-title":"Sentiment analysis levels and techniques: A survey","volume":"6","author":"Patil","year":"2016","journal-title":"Int. J. Innov. Eng. Technol. (IJIET)"},{"key":"ref_52","unstructured":"Stalidis, P., Giatsoglou, M., Diamantarasa, K., Sarigiannidis, G., and Chatzisavvas, K.C. (2015). Machine learning sentiment prediction based on hybrid document representation. arXiv."},{"key":"ref_53","first-page":"55","article-title":"Analysis of emotion recognition system through speech signal using KNN and GMM classifier","volume":"10","author":"Prakash","year":"2015","journal-title":"IOSR J. Electron. Commun. Eng. (IOSR-JECE)"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Schuller, B., Rigoll, G., and Lang, M. (2003, January 6\u201310). Hidden Markov model-based speech emotion recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2003, Hong Kong, China.","DOI":"10.1109\/ICME.2003.1220939"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/S0167-6393(03)00099-2","article-title":"Speech emotion recognition using hidden Markov models","volume":"41","author":"Nwe","year":"2003","journal-title":"Speech Commun."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Hu, H., Xu, M., and Wu, W. (2007, January 15\u201320). GMM supervector based SVM with spectral features for speech emotion recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, Honolulu, HI, USA.","DOI":"10.1109\/ICASSP.2007.366937"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1109\/TASSP.1980.1163420","article-title":"Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences","volume":"28","author":"Davis","year":"1980","journal-title":"IEEE Trans. Acoust. Speech Signal Process."},{"key":"ref_58","unstructured":"Pao, T., Chen, Y., and Yeh, J. (2004, January 15\u201318). Emotion recognition from Mandarin speech signals. Proceedings of the International Symposium on Chinese Spoken Language Processing, Hong Kong, China."},{"key":"ref_59","unstructured":"Waibel, A., and Lee, K.-F. (1990). A tutorial on hidden Markov models and selected applications in speech recognition. Readings in Speech Recognition, Morgan Kaufmann Publishers Inc."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1016\/j.patcog.2010.09.020","article-title":"Survey on speech emotion recognition: Features, classification schemes, and databases","volume":"44","author":"Ayadi","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Lee, C., Yildrim, S., Bulut, M., Kazemzadeh, A., Busso, C., Deng, Z., Lee, S., and Narayanan, S. (2004, January 4\u20138). Emotion recognition based on phoneme classes. Proceedings of the ICSLP 2004, Jeju Island, Korea.","DOI":"10.21437\/Interspeech.2004-322"},{"key":"ref_62","first-page":"2969","article-title":"Speech emotion recognition by Gaussian mixture model","volume":"6","author":"Reshma","year":"2015","journal-title":"Int. J. Comput. Sci. Inf. Technol."},{"key":"ref_63","first-page":"433","article-title":"Emotion recognition using neural network: A comparative study","volume":"7","author":"Hendy","year":"2013","journal-title":"World Acad. Sci. Eng. Technol."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1109\/TASL.2006.876121","article-title":"An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional TTS","volume":"14","author":"Navas","year":"2006","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Atassi, H., and Esposito, A. (2008, January 3\u20135). A speaker independent approach to the classification of emotional vocal expressions. Proceedings of the Twentieth International Conference on Tools with Artificial Intelligence, ICTAI 2008, Dayton, OH, USA.","DOI":"10.1109\/ICTAI.2008.158"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Lugger, M., and Yang, B. (2007, January 15\u201320). The relevance of voice quality features in speaker independent emotion recognition. Proceedings of the ICASSP 2007, Honolulu, HI, USA.","DOI":"10.1109\/ICASSP.2007.367152"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"1787","DOI":"10.1007\/s12652-017-0644-8","article-title":"Feature fusion methods research based on deep belief networks for speech emotion recognition under noise condition","volume":"10","author":"Huang","year":"2019","journal-title":"J. Ambient Intell. Hum. Comput."},{"key":"ref_68","first-page":"1470","article-title":"A survey for multimodal sentiment analysis methods","volume":"5","author":"Sikandar","year":"2014","journal-title":"Int. J. Comput. Technol. Appl."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1146\/annurev.ps.30.020179.002523","article-title":"Facial expressions of emotion","volume":"30","author":"Ekman","year":"1979","journal-title":"Ann. Rev. Psychol."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.neunet.2014.10.005","article-title":"Towards an intelligent framework for multimodal affective data analysis","volume":"63","author":"Poria","year":"2015","journal-title":"Neural Netw."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s12652-011-0087-6","article-title":"Emotional facial sensing and multimodal fusion in a continuous 2D affective space","volume":"3","author":"Cerezo","year":"2012","journal-title":"J. Ambient Intell. Hum. Comput."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Morency, L.-P., Mihalcea, R., and Doshi, P. (2011, January 14\u201318). Towards multimodal sentiment analysis: Harvesting opinions from the web. Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI 2011, Alicante, Spain.","DOI":"10.1145\/2070481.2070509"},{"key":"ref_73","unstructured":"Ramos Pereira, M.H., CardealP\u00e1dua, F.L., Machado Pereira, A.C., Benevenuto, F., and Dalip, D.H. (2016, January 17\u201320). Fusing audio, textual, and visual features for sentiment analysis of news videos. Proceedings of the ICWSM 2016, Cologne, Germany."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s12193-015-0195-2","article-title":"Emonets: Multimodaldeeplearningapproachesforemotionrecognitioninvideo","volume":"10","author":"Kahou","year":"2016","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Wollmer, M., Metallinou, A., Eyben, F., Schuller, B., and Narayanan, S.S. (2010, January 26\u201330). Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSMT modeling. Proceedings of the Interspeech, Makuhari, Japan.","DOI":"10.21437\/Interspeech.2010-646"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.neucom.2015.01.095","article-title":"Fusing audio, visual and textual clues for sentiment analysis from multimodal content","volume":"174","author":"Poria","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_77","unstructured":"Cid, F., Manso, L.J., and N\u00fa\u00f1ez, P. (October, January 28). A novel multimodal emotion recognition approach for affective human robot interaction. Proceedings of the FinE-R 2015 IROS Workshop, Hamburg, Germany."},{"key":"ref_78","unstructured":"Datcu, D., and Rothkrantz, L. (2009, January 2\u20133). Multimodal recognition of emotions in car environments. Proceedings of the Second Driver Car Interaction & Interface Conference (DCI&I-2009), Praag, Czech Republic."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"11","DOI":"10.4018\/jaec.2013070102","article-title":"Multimodal approach for emotion recognition using a formal computational model","volume":"4","author":"Meftah","year":"2013","journal-title":"Int. J. Appl. Evol. Comput. (IJAEC)"},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/TPAMI.2008.52","article-title":"A survey of affect recognition methods: Audio, visual, and spontaneous expressions","volume":"31","author":"Zeng","year":"2009","journal-title":"PAMI"},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1109\/TMM.2008.921737","article-title":"Audio\u2013visual affective expression recognition through multistream fused HMM","volume":"10","author":"Zeng","year":"2008","journal-title":"Trans. Multimed."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1016\/j.neunet.2005.03.006","article-title":"Emotion recognition in human\u2013computer interaction","volume":"18","author":"Fragopanagos","year":"2005","journal-title":"Neural Netw."},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Paouzaiou, A., and Karpouzis, K. (2006, January 2\u20134). Modeling naturalistic affective states via facial and vocal expressions recognition. Proceedings of the 8th International Conference on Multimodal Interfaces (ICMI \u201906), Banff, AB, Canada.","DOI":"10.1145\/1180995.1181029"},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"You, Q., Luo, J., Jin, H., and Yang, J. (2016, January 22\u201325). Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia. Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (WSDM\u201916), San Francisco, CA, USA.","DOI":"10.1145\/2835776.2835779"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Siddiqui, M.F.H., and Javaid, A.Y. (2020). A multimodal facial emotion recognition framework through the fusion of speech with visible and infrared images. Multimodal Technol. Interact., 4.","DOI":"10.3390\/mti4030046"},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Zhou, W., Cheng, J., Lei, X., Benes, B., and Adamo, N. (2020). Deep Learning-Based Emotion Recognition from Real-Time Videos, HCI.","DOI":"10.1007\/978-3-030-49062-1_22"},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Pandeya, Y.R., Bhattarai, B., and Lee, J. (2021). Deep-learning-based multimodal emotion classification for music videos. Sensors, 21.","DOI":"10.3390\/s21144927"},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Khorrami, P., Le Paine, T., Brady, K., Dagli, C., and Huang, T.S. (2016, January 25\u201328). How deep neural networks can improve emotion recognition on video data. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7532431"},{"key":"ref_89","doi-asserted-by":"crossref","unstructured":"Ranganathan, H., Chakraborty, S., and Panchanathan, S. (2016, January 7\u201310). Multimodal emotion recognition using deep learning architectures. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477679"},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"52","DOI":"10.38094\/jastt20291","article-title":"Multimodal emotion recognition using deep learning","volume":"2","author":"Abdullah","year":"2021","journal-title":"J. Appl. Sci. Technol. Trends"},{"key":"ref_91","first-page":"3187","article-title":"Towards emotion recognition from contextual information using machine learning","volume":"11","year":"2019","journal-title":"J. Ambient Intell. Human Comput."},{"key":"ref_92","doi-asserted-by":"crossref","unstructured":"Perifanos, K., and Goutsos, D. (2021). Multimodal hate speech detection in Greek social media. Multimodal Technol. Interact., 5.","DOI":"10.3390\/mti5070034"},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1109\/TSMCA.2012.2210407","article-title":"InteSe: An integrated model for resolvingambiguities in multimodalsentences","volume":"43","author":"Caschera","year":"2013","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_94","first-page":"809","article-title":"Multimodal Database of Emotional Speech, Video and Gestures","volume":"12","author":"Pelikant","year":"2018","journal-title":"World Acad. Sci. Eng. Technol. Int. J. Comput. Inf. Eng."},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Caschera, M.C., D\u2019Ulizia, A., Ferri, F., and Grifoni, P. (2016). MCBF: Multimodal Corpora Building Framework. Human Language Technology: Challenges for Computer Science and Linguistics, Springer International Publishing.","DOI":"10.1007\/978-3-319-43808-5_14"},{"key":"ref_96","unstructured":"(2017, February 22). Available online: https:\/\/cdn.crowdemotion.co.uk."},{"key":"ref_97","unstructured":"(2017, February 22). crowdemotion api. Available online: https:\/\/cdn.crowdemotion.co.uk\/demos\/api-demo\/index.html."},{"key":"ref_98","unstructured":"(2017, February 22). Available online: http:\/\/apidemo.theysay.io\/."},{"key":"ref_99","unstructured":"(2017, February 22). Criptodivisas en Pruebas. Available online: http:\/\/www.theysay.io\/."},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Eyben, F., Weninger, F., Gro\u00df, F., and Schuller, B. (2013, January 21\u201325). Recent developments in opensmile, the munich open-source multimedia feature extractor. Proceedings of the ACMMM\u201913, Barcelona, Spain.","DOI":"10.1145\/2502081.2502224"},{"key":"ref_101","unstructured":"(2017, March 08). Software of the Stanford Natural Language Processing Group. Available online: http:\/\/Nlp.stanford.edu\/software\/."},{"key":"ref_102","first-page":"387","article-title":"An approach for managing ambiguities in multimodal interaction","volume":"Volume 4805","author":"Meersman","year":"2007","journal-title":"OTM-WS 2007, Part I: LNCS"},{"key":"ref_103","first-page":"313","article-title":"Building a large annotated corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1994","journal-title":"Comput. Linguist."},{"key":"ref_104","doi-asserted-by":"crossref","unstructured":"Caschera, M.C., Ferri, F., and Grifoni, P. (2008). Ambiguity detection in multimodal systems. Advanced Visual Interfaces 2008, ACM Press.","DOI":"10.1145\/1385569.1385625"},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"1097","DOI":"10.1121\/1.405558","article-title":"Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion","volume":"93","author":"Murray","year":"1993","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_106","doi-asserted-by":"crossref","first-page":"87","DOI":"10.4156\/jnit.vol4.issue5.10","article-title":"From modal to multimodal ambiguities: A classification approach","volume":"4","author":"Caschera","year":"2013","journal-title":"JNIT"},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1007\/s12530-020-09344-3","article-title":"Evaluation of a dynamic classification method for multimodal ambiguities based on Hidden markov models","volume":"12","author":"Grifoni","year":"2021","journal-title":"Evol. Syst."},{"key":"ref_108","doi-asserted-by":"crossref","first-page":"178","DOI":"10.2991\/ijcis.d.200208.001","article-title":"DAMA: A dynamic classification of multimodal ambiguities","volume":"13","author":"Grifoni","year":"2020","journal-title":"Int. J. Comput. Intell. Syst."},{"key":"ref_109","unstructured":"Yakhnenko, O., Silvescu, A., and Honavar, V. (2005, January 27\u201330). Discriminatively trained Markov model for sequence classification. Proceedings of the ICDM\u201905: Fifth IEEE International Conference on Data Mining, Houston, TX, USA."},{"key":"ref_110","unstructured":"Kempf-Leonard, K. (2005). Type I and Type II Error. Encyclopedia of Social Measurement, Elsevier."},{"key":"ref_111","unstructured":"Manliguez, C. (2017, February 22). Generalized Confusion Matrix for Multiple Classes. Available online: https:\/\/www.researchgate.net\/publication\/310799885_Generalized_Confusion_Matrix_for_Multiple_Classes."},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1080\/10463283.2017.1329107","article-title":"Doing emotions: The role of culture in everyday emotions","volume":"28","author":"Mesquita","year":"2017","journal-title":"Eur. Rev. Soc. Psychol."},{"key":"ref_113","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1007\/s12187-016-9440-9","article-title":"Changes in emotions from childhood to young adulthood","volume":"11","author":"Fenouillet","year":"2018","journal-title":"Child Indic. Res."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/6\/4\/28\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:52:27Z","timestamp":1760136747000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/6\/4\/28"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,12]]},"references-count":113,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["mti6040028"],"URL":"https:\/\/doi.org\/10.3390\/mti6040028","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,12]]}}}