{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T16:41:25Z","timestamp":1775666485041,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,3,2]],"date-time":"2022-03-02T00:00:00Z","timestamp":1646179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The emotional speech recognition method presented in this article was applied to recognize the emotions of students during online exams in distance learning due to COVID-19. The purpose of this method is to recognize emotions in spoken speech through the knowledge base of emotionally charged words, which are stored as a code book. The method analyzes human speech for the presence of emotions. To assess the quality of the method, an experiment was conducted for 420 audio recordings. The accuracy of the proposed method is 79.7% for the Kazakh language. The method can be used for different languages and consists of the following tasks: capturing a signal, detecting speech in it, recognizing speech words in a simplified transcription, determining word boundaries, comparing a simplified transcription with a code book, and constructing a hypothesis about the degree of speech emotionality. In case of the presence of emotions, there occurs complete recognition of words and definitions of emotions in speech. The advantage of this method is the possibility of its widespread use since it is not demanding on computational resources. The described method can be applied when there is a need to recognize positive and negative emotions in a crowd, in public transport, schools, universities, etc. The experiment carried out has shown the effectiveness of this method. The results obtained will make it possible in the future to develop devices that begin to record and recognize a speech signal, for example, in the case of detecting negative emotions in sounding speech and, if necessary, transmitting a message about potential threats or riots.<\/jats:p>","DOI":"10.3390\/s22051937","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T21:25:14Z","timestamp":1646169914000},"page":"1937","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Emotional Speech Recognition Method Based on Word Transcription"],"prefix":"10.3390","volume":"22","author":[{"given":"Gulmira","family":"Bekmanova","sequence":"first","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Nur-Sultan 010008, Kazakhstan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Banu","family":"Yergesh","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Nur-Sultan 010008, Kazakhstan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Altynbek","family":"Sharipbay","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Nur-Sultan 010008, Kazakhstan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8964-3891","authenticated-orcid":false,"given":"Assel","family":"Mukanova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Nur-Sultan 010008, Kazakhstan"},{"name":"Higher School of Information Technology and Engineering, Astana International University, Nur-Sultan 010000, Kazakhstan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3233\/WEB-190395","article-title":"Emotional machines: The next revolution","volume":"17","author":"Franzoni","year":"2019","journal-title":"Web Intell."},{"key":"ref_2","unstructured":"Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., and Cambria, E. (February, January 27). DialogueRNN: An attentive RNN for emotion detection in conversations. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_3","first-page":"719","article-title":"A deep learning semantic approach to emotion recognition using the IBM watson bluemix alchemy language","volume":"Volume 10406","author":"Biondi","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/MIS.2021.3062200","article-title":"Sentiment Analysis and Topic Recognition in Video Transcriptions","volume":"36","author":"Stappen","year":"2021","journal-title":"IEEE Intell. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.procs.2017.12.003","article-title":"An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment","volume":"125","author":"Yang","year":"2018","journal-title":"Procedia Comput. Sci."},{"key":"ref_6","unstructured":"Gupta, O., Raviv, D., and Raskar, R. (2016). Deep video gesture recognition using illumination invariants. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., G\u00fcl\u00e7ehre, \u00c7., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., and Ferrari, R.C. (2013, January 9\u201313). Combining modality specific deep neural networks for emotion recognition in video. Proceedings of the 2013 ACM International Conference on Multimodal Interaction, Sydney, Australia.","DOI":"10.1145\/2522848.2531745"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"\u00d6zdemir, M., Elag\u00f6z, B., Alaybeyoglu, A., and Akan, A. (2020, January 19\u201320). Deep Learning Based Facial Emotion Recognition System (Derin \u00d6\u011frenme Tabanl\u0131 Y\u00fcz Duygular\u0131 Tan\u0131ma Sistemi). Proceedings of the 2020 Medical Technologies Congress (TIPTEKNO), Antalya, Turkey.","DOI":"10.1109\/TIPTEKNO50054.2020.9299256"},{"key":"ref_9","unstructured":"Kahou, S.E., Michalski, V., Konda, K., Memisevic, R., and Pal, C. (2015, January 9\u201313). Recurrent neural networks for emotion recognition in video. Proceedings of the ACM International Conference on Multimodal Interaction, ICMI 2015, Seattle, DC, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.inffus.2018.09.008","article-title":"Emotion recognition using deep learning approach from audio\u2013visual emotional big data","volume":"49","author":"Hossain","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1007\/s11760-013-0522-6","article-title":"Recognition of emotions from video using acoustic and facial features","volume":"9","author":"Rao","year":"2015","journal-title":"Signal Image Video Process."},{"key":"ref_12","unstructured":"Cruz, A., Bhanu, B., and Thakoor, N. (2012, January 11\u201315). Facial emotion recognition in continuous video. Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1007\/978-981-10-3174-8_47","article-title":"Emotion recognition from videos using facial expressions","volume":"517","author":"Vyshnavi","year":"2017","journal-title":"Adv. Intell. Syst. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mehta, D., Siddiqui, M.F.H., and Javaid, A.Y. (2019). Recognition of emotion intensities using machine learning algorithms: A comparative study. Sensors, 19.","DOI":"10.3390\/s19081897"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"36063","DOI":"10.1007\/s11042-020-09428-x","article-title":"Emotional sounds of crowds: Spectrogram-based analysis using deep learning","volume":"79","author":"Franzoni","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3130961","article-title":"Distant Emotion Recognition","volume":"1","author":"Salekin","year":"2017","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Fayek, H.M., Lech, M., and Cavedon, L. (2015, January 14\u201316). Towards real-time speech emotion recognition using deep neural networks. Proceedings of the 9th International Conference on Signal Processing and Communication Systems, ICSPCS 2015, Cairns, Australia.","DOI":"10.1109\/ICSPCS.2015.7391796"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Mirsamadi, S., Barsoum, E., and Zhang, C. (2017, January 5\u20139). Automatic speech emotion recognition using recurrent neural networks with local attention. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7952552"},{"key":"ref_19","first-page":"653","article-title":"A web-based system for emotion vector extraction","volume":"Volume 10406","author":"Franzoni","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Franzoni, V., Li, Y., and Mengoni, P. (2017, January 23\u201326). A path-based model for emotion abstraction on facebook using sentiment analysis and taxonomy knowledge. Proceedings of the 2017 IEEE\/WIC\/ACM International Conference on Web Intelligence, WI 2017, Leipzig, Germany.","DOI":"10.1145\/3106426.3109420"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Canales, L., and Martinez-Barco, P. (2014, January 24\u201326). Emotion detection from text: A survey. Proceedings of the Processing in the 5th Information Systems Research Working Days, JISIC 2014, Hague, The Netherlands.","DOI":"10.3115\/v1\/W14-6905"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"14","DOI":"10.18178\/ijmlc.2019.9.1.759","article-title":"Facial emotion recognition from videos using deep convolutional neural networks","volume":"9","author":"Abdulsalam","year":"2014","journal-title":"Int. J. Mach. Learn. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"17","DOI":"10.3233\/WEB-190397","article-title":"Automating facial emotion recognition","volume":"17","author":"Gervasi","year":"2019","journal-title":"Web Intell."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2331","DOI":"10.1007\/s11042-015-3180-6","article-title":"Audio-visual emotion recognition using FCBF feature selection method and particle swarm optimization for fuzzy ARTMAP neural networks","volume":"76","author":"Gharavian","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sinith, M.S., Aswathi, E., Deepa, T.M., Shameema, C.P., and Rajan, S. (2015, January 10\u201312). Emotion recognition from audio signals using Support Vector Machine. Proceedings of the IEEE Recent Advances in Intelligent Computational Systems, RAICS 2015, Trivandrum, Kerala, India.","DOI":"10.1109\/RAICS.2015.7488403"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kwon, S. (2020). A CNN-assisted enhanced audio signal processing for speech emotion recognition. Sensors, 20.","DOI":"10.3390\/s20010183"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kannadaguli, P., and Bhat, V. (2019, January 15\u201316). Comparison of hidden markov model and artificial neural network based machine learning techniques using DDMFCC vectors for emotion recognition in Kannada. Proceedings of the 5th IEEE International WIE Conference on Electrical and Computer Engineering, WIECON-ECE 2019, Bangalore, India.","DOI":"10.1109\/WIECON-ECE48653.2019.9019936"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Tursunov, A., Choeh, J.Y., and Kwon, S. (2021). Age and gender recognition using a convolutional neural network with a specially designed multi-attention module through speech spectrograms. Sensors, 21.","DOI":"10.3390\/s21175892"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shahin, I. (2019, January 9\u201311). Emotion recognition based on third-order circular suprasegmental hidden markov model. Proceedings of the IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology, JEEIT 2019, Amman, Jordan.","DOI":"10.1109\/JEEIT.2019.8717396"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Abo Absa, A.H., and Deriche, M. (2017, January 8\u201311). A two-stage hierarchical multilingual emotion recognition system using hidden markov models and neural networks. Proceedings of the 9th IEEE-GCC Conference and Exhibition, GCCCE 2017, Manama, Bahrain.","DOI":"10.1109\/IEEEGCC.2017.8448155"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1016\/j.ins.2015.09.050","article-title":"Weighted high-order hidden Markov models for compound emotions recognition in text","volume":"329","author":"Quan","year":"2016","journal-title":"Inf. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"518","DOI":"10.17516\/1997-1397-2016-9-4-518-523","article-title":"Speech-based emotion recognition and speaker identification: Static vs. dynamic mode of speech representation","volume":"9","author":"Sidorov","year":"2016","journal-title":"J. Sib. Fed. Univ.-Math. Phys."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1111\/j.1751-228X.2007.00004.x","article-title":"We feel, therefore we learn: The relevance of affective and social neuroscience to education","volume":"1","author":"Damasio","year":"2007","journal-title":"Mind Brain Educ."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1007\/978-3-030-68198-2_58","article-title":"Emotion Analysis in Distance Learning","volume":"Volume 1328","author":"Auer","year":"2021","journal-title":"Educating Engineers for Future Industrial Revolutions"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Baker, M., Andriessen, J., and J\u00e4rvel\u00e4, S. (2013). Affective Learning Together. Social and Emotional Dimension of Collaborative Learning, Routledge.","DOI":"10.4324\/9780203069684"},{"key":"ref_36","first-page":"301","article-title":"Student Emotion Recognition in Computer Science Education: A Blessing or Curse?","volume":"Volume 11590","year":"2019","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1016\/j.procs.2016.05.264","article-title":"Student Emotion Recognition System (SERS) for e-learning Improvement Based on Learner Concentration Metric","volume":"85","author":"Krithika","year":"2016","journal-title":"Procedia Comput. Sci."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Franzoni, V., Biondi, G., Perri, D., and Gervasi, O. (2020). Enhancing Mouth-Based Emotion Recognition Using Transfer Learning. Sensors, 20.","DOI":"10.20944\/preprints202007.0379.v1"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Luna-Jim\u00e9nez, C., Griol, D., Callejas, Z., Kleinlein, R., Montero, J.M., and Fern\u00e1ndez-Mart\u00ednez, F. (2021). Multimodal emotion recognition on RAVDESS dataset using transfer learning. Sensors, 21.","DOI":"10.3390\/s21227665"},{"key":"ref_40","first-page":"669","article-title":"Ontology-based sentiment analysis of kazakh sentences","volume":"Volume 10406","author":"Yergesh","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"9","DOI":"10.3233\/WEB-190396","article-title":"Sentiment analysis of Kazakh text and their polarity","volume":"17","author":"Yergesh","year":"2019","journal-title":"Web Intell."},{"key":"ref_42","first-page":"538","article-title":"Method of Sentiment Preservation in the Kazakh-Turkish Machine Translation","volume":"Volume 12250","author":"Zhetkenbay","year":"2020","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Yergesh, B., Bekmanova, G., and Sharipbay, A. (2017, January 5\u20138). Sentiment analysis on the hotel reviews in the Kazakh language. Proceedings of the 2nd International Conference on Computer Science and Engineering, UBMK 2017, Antalya, Turkey.","DOI":"10.1109\/UBMK.2017.8093531"},{"key":"ref_44","first-page":"717","article-title":"Methods for Analyzing Polarity of the Kazakh Texts Related to the Terrorist Threats","volume":"Volume 11619","author":"Bekmanova","year":"2019","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_45","first-page":"54","article-title":"Recognition of the continuous-speech russian phrases using their voiceless fragments","volume":"4","author":"Shelepov","year":"2016","journal-title":"Eurasian J. Math. Comput. Appl."},{"key":"ref_46","first-page":"50","article-title":"On the recognition of Russian words using generalized transcription","volume":"1","author":"Shelepov","year":"2018","journal-title":"Probl. Artif. Intell."},{"key":"ref_47","first-page":"633","article-title":"Algorithms for phonemic recognition of words for a given dictionary","volume":"4","author":"Nitsenko","year":"2004","journal-title":"Artif. Intell. [Iskusstv. Intell.]"},{"key":"ref_48","unstructured":"Shelepov, V.Y. (2007). The concept of phonemic recognition of separately pronounced Russian words. Recognition of syn-tactically related phrases. Materials of international scientific-technical conference. Artif. Intell., 162\u2013170. (In Russian)."},{"key":"ref_49","first-page":"662","article-title":"To the problem of phonemic recognition","volume":"4","author":"Shelepov","year":"2005","journal-title":"Artif. Intell. [Iskusstv. Intell.]"},{"key":"ref_50","unstructured":"Sharipbayev, A.A., Bekmanova, G.T., and Shelepov, V.U. (2021, December 29). Formalization of Phonologic Rules of the Kazakh Language for System Automatic Speech Recognition. Available online: http:\/\/dspace.enu.kz\/handle\/data\/1013."},{"key":"ref_51","unstructured":"Bekmanova, G.T., Nitsenko, A.V., Sharipbaev, A.A., and Shelepov, V.Y. (2010). Algorithms for recognition of the Kazakh word as a whole. Structural Classification of Kazakh Language Words, Bulletin of the L.N. Gumilyov Eurasian National University."},{"key":"ref_52","first-page":"70","article-title":"The refined identification of beginning-end of speech; the recognition of the voiceless sounds at the beginning-end of speech. On the recognition of the extra-large vocabularies","volume":"5","author":"Shelepov","year":"2017","journal-title":"Eurasian J. Math. Comput. Appl."},{"key":"ref_53","unstructured":"Kazakh Grammar (2002). Phonetics, Word Formation, Morphology, Syntax, Astana-Poligraphy. (In Kazakh)."},{"key":"ref_54","first-page":"170","article-title":"Sentiment Analysis Model Based on the Word Structural Representation","volume":"Volume 12960","author":"Bekmanova","year":"2021","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Sharipbaev, A.A., Bekmanova, G.T., Buribayeva, A.K., Yergesh, B.Z., Mukanova, A.S., and Kaliyev, A.K. (2012, January 20\u201324). Semantic neural network model of morphological rules of the agglutinative languages. Proceedings of the 6th International Conference on Soft Computing and Intelligent Systems, and 13th International Symposium on Advanced Intelligence Systems, SCIS\/ISIS 2012, Kobe, Japan.","DOI":"10.1109\/SCIS-ISIS.2012.6505337"},{"key":"ref_56","first-page":"627","article-title":"Semantic hyper-graph based representation of nouns in the Kazakh language","volume":"18","author":"Yergesh","year":"2014","journal-title":"Comput. Sist."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Sharipbay, A., Yergesh, B., Razakhova, B., Yelibayeva, G., and Mukanova, A. (2019, January 2\u20135). Syntax parsing model of Kazakh simple sentences. Proceedings of the 2nd International Conference on Data Science, E-Learning and Information Systems, DATA 2019, Dubai, United Arab Emirates.","DOI":"10.1145\/3368691.3368745"},{"key":"ref_58","unstructured":"Razakhova, B.S., and Sharipbaev, A.\u0410. (2012). Formalization of Syntactic Rules of the Kazakh Language, Bulletin of the L.N. Gumilyov Eurasian National University."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Yelibayeva, G., Sharipbay, A., Mukanova, A., and Razakhova, B. (2020, January 9\u201310). Applied ontology for the automatic classification of simple sentences of the kazakh language. Proceedings of the 5th International Conference on Computer Science and Engineering, UBMK 2020, Diyarbakir, Turkey.","DOI":"10.1109\/UBMK50275.2020.9219461"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"65","DOI":"10.18576\/amis\/140108","article-title":"Speech emotion recognition for Kazakh and Russian languages","volume":"14","author":"Kozhakhmet","year":"2020","journal-title":"Appl. Math. Inf. Sci."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1937\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:30:27Z","timestamp":1760135427000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1937"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,2]]},"references-count":60,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22051937"],"URL":"https:\/\/doi.org\/10.3390\/s22051937","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,2]]}}}