{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T16:25:01Z","timestamp":1792427101589,"version":"4.5.0"},"reference-count":44,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2017,6,1]],"date-time":"2017-06-01T00:00:00Z","timestamp":1496275200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computation"],"abstract":"<jats:p>Emotion recognition from speech may play a crucial role in many applications related to human\u2013computer interaction or understanding the affective state of users in certain tasks, where other modalities such as video or physiological parameters are unavailable. In general, a human\u2019s emotions may be recognized using several modalities such as analyzing facial expressions, speech, physiological parameters (e.g., electroencephalograms, electrocardiograms) etc. However, measuring of these modalities may be difficult, obtrusive or require expensive hardware. In that context, speech may be the best alternative modality in many practical applications. In this work we present an approach that uses a Convolutional Neural Network (CNN) functioning as a visual feature extractor and trained using raw speech information. In contrast to traditional machine learning approaches, CNNs are responsible for identifying the important features of the input thus, making the need of hand-crafted feature engineering optional in many tasks. In this paper no extra features are required other than the spectrogram representations and hand-crafted features were only extracted for validation purposes of our method. Moreover, it does not require any linguistic model and is not specific to any particular language. We compare the proposed approach using cross-language datasets and demonstrate that it is able to provide superior results vs. traditional ones that use hand-crafted features.<\/jats:p>","DOI":"10.3390\/computation5020026","type":"journal-article","created":{"date-parts":[[2017,6,1]],"date-time":"2017-06-01T10:36:36Z","timestamp":1496313396000},"page":"26","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":53,"title":["Deep Visual Attributes vs. Hand-Crafted Audio Features on Multidomain Speech Emotion Recognition"],"prefix":"10.3390","volume":"5","author":[{"given":"Michalis","family":"Papakostas","sequence":"first","affiliation":[{"name":"Computer Science and Engineering Department, University of Texas at Arlington, Arlington, TX 76019, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evaggelos","family":"Spyrou","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, National Center for Scientific Research\u2014\u201cDemokritos\u201d, Athens 15310, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Theodoros","family":"Giannakopoulos","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, National Center for Scientific Research\u2014\u201cDemokritos\u201d, Athens 15310, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giorgos","family":"Siantikos","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, National Center for Scientific Research\u2014\u201cDemokritos\u201d, Athens 15310, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitrios","family":"Sgouropoulos","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telecommunications, National Center for Scientific Research\u2014\u201cDemokritos\u201d, Athens 15310, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6916-3129","authenticated-orcid":false,"given":"Phivos","family":"Mylonas","sequence":"additional","affiliation":[{"name":"Department of Informatics, Ionian University, Corfu 49100, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fillia","family":"Makedon","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering Department, University of Texas at Arlington, Arlington, TX 76019, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,6,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Wang, X., and Petriu, E.M. (2011, January 19\u201321). Facial expression anlysis using eye gaze information. Proceedings of the IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA), Ottava, ON, Canada.","DOI":"10.1109\/CIMSA.2011.6059936"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/j.neunet.2005.03.004","article-title":"Emotion recognition through facial expression analysis based on a neurofuzzy network","volume":"18","author":"Ioannou","year":"2005","journal-title":"Neural Netw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1016\/j.jnca.2006.09.007","article-title":"Bi-modal emotion recognition from expressive face and body gestures","volume":"30","author":"Gunes","year":"2007","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1016\/j.neunet.2008.05.003","article-title":"Recognizing emotions expressed by body pose: A biologically inspired neural model","volume":"21","author":"Schindler","year":"2008","journal-title":"Neural Netw."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1007\/BF02344719","article-title":"Emotion recognition system using short-term monitoring of physiological signals","volume":"42","author":"Kim","year":"2004","journal-title":"Med. Boil. Eng. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s10462-012-9368-5","article-title":"Features and classifiers for emotion recognition from speech: A survey from 2000 to 2011","volume":"43","author":"Anagnostopoulos","year":"2015","journal-title":"Artif. Intell. Rev."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion recognition in human-computer interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1016\/j.patcog.2010.09.020","article-title":"Survey on speech emotion recognition: Features, classification schemes, and databases","volume":"44","author":"Kamel","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Shen, P., Changjun, Z., and Chen, X. (2011, January 12\u201314). Automatic speech emotion recognition using support vector machine. Proceedings of the 2011 International Conference on Electronic and Mechanical Engineering and Information Technology (EMEIT), Harbin, China.","DOI":"10.1109\/EMEIT.2011.6023178"},{"key":"ref_10","unstructured":"Kishore, K.K., and Satish, P.K. (2013, January 22\u201323). Emotion recognition in speech using MFCC and wavelet features. Proceedings of the 2013 IEEE 3rd International Advance Computing Conference (IACC), Ghaziabad, India."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1016\/j.sigpro.2009.09.009","article-title":"Emotion recognition from speech signals using new harmony features","volume":"90","author":"Yang","year":"2010","journal-title":"Signal Process."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1016\/j.specom.2010.08.013","article-title":"Automatic speech emotion recognition using modulation spectral features","volume":"53","author":"Wu","year":"2011","journal-title":"Speech Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1016\/j.specom.2010.02.010","article-title":"Class-level spectral features for emotion recognition","volume":"52","author":"Bitouk","year":"2010","journal-title":"Speech Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1007\/s10772-012-9139-3","article-title":"Emotion recognition from speech using source, system, and prosodic features","volume":"15","author":"Koolagudi","year":"2012","journal-title":"Int. J. Speech Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/T-AFFC.2010.16","article-title":"Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels","volume":"2","author":"Wu","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_16","unstructured":"Giannoulis, P., and Potamianos, G. (2017, June 02). A hierarchical approach with feature selection for emotion recognition from speech. Available online: https:\/\/pdfs.semanticscholar.org\/36ca\/1c7ce51111d635554e54854874cc87cd5731.pdf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1162","DOI":"10.1016\/j.specom.2011.06.004","article-title":"Emotion recognition using a hierarchical binary decision tree approach","volume":"53","author":"Lee","year":"2011","journal-title":"Speech Commun."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1154","DOI":"10.1016\/j.dsp.2012.05.007","article-title":"Speech emotion recognition: Features and classification models","volume":"22","author":"Chen","year":"2012","journal-title":"Digit. Signal Process."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1109\/TETC.2013.2274797","article-title":"Gender-driven emotion recognition through speech signals for ambient intelligence applications","volume":"1","author":"Bisio","year":"2013","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Koolagudi, S.G., Kumar, N., and Rao, K.S. (2011, January 24\u201325). Speech emotion recognition using segmental level prosodic analysis. Proceedings of the 2011 International Conference on Devices and Communications (ICDeCom), Mesra, India.","DOI":"10.1109\/ICDECOM.2011.5738536"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tawari, A., and Trivedi, M.M. (2010, January 23\u201326). Speech emotion analysis in noisy real-world environment. Proceedings of the 2010 20th International Conference on Pattern Recognition (ICPR), Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.1132"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lefter, I., Rothkrantz, L.J., Wiggers, P., and van Leeuwen, D.A. (2010, January 6\u201310). Emotion recognition from speech by combining databases and fusion of classifiers. Proceedings of the International Conference on Text, Speech and Dialogue, Brno, Czech Republic.","DOI":"10.1007\/978-3-642-15760-8_45"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1109\/T-AFFC.2010.8","article-title":"Cross-corpus acoustic emotion recognition: Variances and strategies","volume":"1","author":"Schuller","year":"2010","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., and Schuller, B. (2011, January 22\u201327). Deep neural networks for acoustic emotion recognition: Raising the benchmarks. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.","DOI":"10.1109\/ICASSP.2011.5947651"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Han, K., Yu, D., and Tashev, I. (2014, January 14\u201318). Speech emotion recognition using deep neural network and extreme learning machine. Interspeech, Annual Conference of the International Speech Communication Association, Singapore.","DOI":"10.21437\/Interspeech.2014-57"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Huang, Z., Dong, M., Mao, Q., and Zhan, Y. (2014, January 3\u20137). Speech emotion recognition using CNN. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA.","DOI":"10.1145\/2647868.2654984"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, L., Zhao, Y., Jiang, D., Zhang, Y., Wang, F., Gonzalez, I., Valentin, E., and Sahli, H. (2013, January 2\u20135). Hybrid Deep Neural Network\u2013Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition. Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Italy.","DOI":"10.1109\/ACII.2013.58"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M.A., Schuller, B., and Zafeiriou, S. (2016, January 20\u201325). Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.","DOI":"10.1109\/ICASSP.2016.7472669"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12\u201315). Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.","DOI":"10.1109\/ICDM.2016.0055"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zheng, W.Q., Yu, J.S., and Zou, Y.X. (2015, January 21\u201324). An experimental study of speech emotion recognition based on deep convolutional neural networks. Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), Xian, China.","DOI":"10.1109\/ACII.2015.7344669"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s10772-011-9125-1","article-title":"Emotion recognition from speech: A review","volume":"15","author":"Koolagudi","year":"2012","journal-title":"Int. J. Speech Technol."},{"key":"ref_32","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","article-title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","volume":"29","author":"Hinton","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_34","unstructured":"Lee, H., Pham, P., Largman, Y., and Ng, A.Y. (2009, January 5\u201310). Unsupervised feature learning for audio classification using convolutional deep belief networks. Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada."},{"key":"ref_35","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_36","unstructured":"Ioffe, S., and Szegedy, C. (arXiv, 2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv."},{"key":"ref_37","unstructured":"Costantini, G., Iaderola, I., Paoloni, A., and Todisco, M. (2014, January 26\u201331). EMOVO Corpus: An Italian Emotional Speech Database. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914), Reykjavic, Island."},{"key":"ref_38","unstructured":"Jackson, P., and Haq, S. (2014). Surrey Audio-Visual Expressed Emotion (SAVEE) Database, University of Surrey."},{"key":"ref_39","first-page":"1517","article-title":"A database of German emotional speech","volume":"5","author":"Burkhardt","year":"2005","journal-title":"Interspeech"},{"key":"ref_40","first-page":"958","article-title":"Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis","volume":"3","author":"Simard","year":"2003","journal-title":"ICDAR"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3\u20137). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedi, Orlando, FL, USA.","DOI":"10.1145\/2647868.2654889"},{"key":"ref_42","first-page":"249","article-title":"Understanding the difficulty of training deep feedforward neural networks","volume":"9","author":"Glorot","year":"2010","journal-title":"Aistats"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Giannakopoulos, T. (2015). Pyaudioanalysis: An open-source python library for audio signal analysis. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0144610"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"J. Lang. Resour. Eval."}],"container-title":["Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-3197\/5\/2\/26\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:37:37Z","timestamp":1760207857000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-3197\/5\/2\/26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,1]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2017,6]]}},"alternative-id":["computation5020026"],"URL":"https:\/\/doi.org\/10.3390\/computation5020026","relation":{},"ISSN":["2079-3197"],"issn-type":[{"value":"2079-3197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,1]]}}}