{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,24]],"date-time":"2025-08-24T01:29:25Z","timestamp":1755998965240,"version":"3.41.2"},"reference-count":33,"publisher":"World Scientific Pub Co Pte Ltd","issue":"Supp04","funder":[{"name":"nsfc under grant number","award":["61370205"],"award-info":[{"award-number":["61370205"]}]},{"name":"science and technology project of chongqing education commission of china","award":["KJQN201900520"],"award-info":[{"award-number":["KJQN201900520"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Inter. Net."],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:p> In the process of learning, learners will express their emotions through a variety of forms, facial expressions and voice are more obvious, which are most easily obtained through computers. In the previous methods, it is mainly based on a single modal, such as expression, speech, text and so on. Due to the diversity of information, the accuracy rate of multimodal recognition is higher than that of single modal recognition. Therefore, this paper proposed a DNN-based multimodal learning emotion analysis method which combines video and speech to detect students\u2019 learning emotion in real time. We use this method to automatically identify learning emotions in primary school English classroom. According to different learning emotions, the PAD emotion scale was used to correspond learning emotions with learning states. Teachers can judge students\u2019 learning state according to the change of students\u2019 learning emotions, so as to adjust teaching methods and strategies in time. <\/jats:p>","DOI":"10.1142\/s0219265921460117","type":"journal-article","created":{"date-parts":[[2022,2,10]],"date-time":"2022-02-10T09:05:53Z","timestamp":1644483953000},"source":"Crossref","is-referenced-by-count":9,"title":["Multimodal Emotion Recognition and State Analysis of Classroom Video and Audio Based on Deep Neural Network"],"prefix":"10.1142","volume":"22","author":[{"given":"Mingyong","family":"Li","sequence":"first","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Mingyue","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Zheng","family":"Jiang","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Zongwei","family":"Zhao","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Jiayan","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Mingyuan","family":"Ge","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]},{"given":"Huiming","family":"Duan","sequence":"additional","affiliation":[{"name":"The Personnel Department, Chongqing Normal University, Chongqing, P. R. China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6045-4643","authenticated-orcid":false,"given":"Yanxia","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Computer and Information Science, Chongqing Normal University, Chongqing, P. R. China"}]}],"member":"219","published-online":{"date-parts":[[2022,2,9]]},"reference":[{"key":"S0219265921460117BIB001","first-page":"3","volume":"2013","author":"Lerner Matthew D.","year":"2013","journal-title":"Developmental Cognitive Neuroscience"},{"key":"S0219265921460117BIB002","series-title":"IEEE Engineering in Medicine and Biology Society. Annual Conference","first-page":"2012","volume-title":"Conference proceedings: \u2026 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.","author":"Jukka Kortelainen","year":"2012"},{"key":"S0219265921460117BIB003","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-009-0344-2"},{"issue":"01","key":"S0219265921460117BIB004","first-page":"76","volume":"38","author":"Zhang Qi","year":"2020","journal-title":"Journal of Distance Education"},{"key":"S0219265921460117BIB006","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-8655(97)00080-9"},{"volume-title":"Research on Emotion Recognition Based on Speech and Face","year":"2012","author":"Zhang Shi qing","key":"S0219265921460117BIB008"},{"key":"S0219265921460117BIB009","series-title":"University of Chinese Academy of Sciences","volume-title":"Research on Human Action Recognition Method Based on Multimodal Information Fusion","author":"Zhao Qing song","year":"2020"},{"key":"S0219265921460117BIB010","series-title":"Huazhong University of Science and Technology","volume-title":"Analysis of Student Learning Status Based on Outlier Detection","author":"Lu Liu sheng","year":"2016"},{"issue":"11","key":"S0219265921460117BIB011","volume":"40","author":"C Holding Benjamin","year":"2017","journal-title":"Sleep"},{"issue":"11","key":"S0219265921460117BIB013","volume":"40","author":"C Holding Benjamin","year":"2017","journal-title":"Sleep"},{"key":"S0219265921460117BIB014","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/2593036"},{"issue":"4","key":"S0219265921460117BIB015","volume":"22","author":"Junhan Yoon","year":"2018","journal-title":"Institute of Electrical and Electronics"},{"key":"S0219265921460117BIB016","doi-asserted-by":"publisher","DOI":"10.1080\/15622975.2017.1324176"},{"key":"S0219265921460117BIB017","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2018.06.005"},{"key":"S0219265921460117BIB018","series-title":"IEEE Engineering in Medicine and Biology Society. Annual Conference","first-page":"2018","volume-title":"Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society.","author":"Valentina Chaparro","year":"2018"},{"key":"S0219265921460117BIB019","doi-asserted-by":"publisher","DOI":"10.1080\/01691864.2019.1667872"},{"key":"S0219265921460117BIB020","doi-asserted-by":"publisher","DOI":"10.2991\/ijcis.2019.125905651"},{"key":"S0219265921460117BIB021","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-018-01195-9"},{"issue":"1","key":"S0219265921460117BIB022","volume":"43","author":"Rahdari Farhad","year":"2019","journal-title":"Iranian Journal of Science and Technology, Transactions of Electrical Engineering"},{"key":"S0219265921460117BIB023","volume":"14","author":"Mado Proverbio Alice","year":"2020","journal-title":"Frontiers in human neuroscience"},{"issue":"1","key":"S0219265921460117BIB024","volume":"27","author":"Ghaleb Esam","year":"2020","journal-title":"IEEE MultiMedia"},{"issue":"24","key":"S0219265921460117BIB025","volume":"19","author":"V\u00e1lber C\u00e9sar Cavalcanti Roza","year":"2019","journal-title":"Sensors"},{"key":"S0219265921460117BIB026","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-018-03735-0"},{"journal-title":"IEEE transactions on cybernetics","year":"2020","author":"Xiaowei Zhang","key":"S0219265921460117BIB027"},{"issue":"1","key":"S0219265921460117BIB028","volume":"40","author":"Ayata De\u011fer","year":"2020","journal-title":"Journal of Medical and Biological Engineering"},{"volume-title":"The Effectiveness of the Flipped Classroom in Higher Education. 2015 Fifth International Conference on E-Learning (econf)","year":"2015","author":"Elazab S.","key":"S0219265921460117BIB029"},{"key":"S0219265921460117BIB030","doi-asserted-by":"publisher","DOI":"10.1016\/j.comcom.2019.11.022"},{"issue":"3","key":"S0219265921460117BIB031","volume":"11","author":"Wang Zhongmin","year":"2020","journal-title":"International Journal of Machine Learning and Cybernetics"},{"issue":"1","key":"S0219265921460117BIB032","volume":"29","author":"Chen Yuefen","year":"2018","journal-title":"Computer Journal"},{"issue":"2","key":"S0219265921460117BIB033","volume":"22","author":"Xianmo Quan","year":"2016","journal-title":"Korean Institute of Chefs"},{"key":"S0219265921460117BIB035","volume":"2265","author":"Cao Hai Tao","year":"2013","journal-title":"Advanced Materials Research"},{"key":"S0219265921460117BIB036","doi-asserted-by":"publisher","DOI":"10.1109\/NCA.2017.8171361"},{"key":"S0219265921460117BIB037","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2020.3029136"}],"container-title":["Journal of Interconnection Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0219265921460117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,6]],"date-time":"2022-10-06T12:00:01Z","timestamp":1665057601000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0219265921460117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,9]]},"references-count":33,"journal-issue":{"issue":"Supp04","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["10.1142\/S0219265921460117"],"URL":"https:\/\/doi.org\/10.1142\/s0219265921460117","relation":{},"ISSN":["0219-2659","1793-6713"],"issn-type":[{"type":"print","value":"0219-2659"},{"type":"electronic","value":"1793-6713"}],"subject":[],"published":{"date-parts":[[2022,2,9]]},"article-number":"2146011"}}