{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T16:09:21Z","timestamp":1772122161258,"version":"3.50.1"},"reference-count":31,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T00:00:00Z","timestamp":1651017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>It is not an easy task for organizers to observe the engagement level of a video meeting audience. This research was conducted to build an intelligent system to enhance the experience of video conversations such as virtual meetings and online classrooms using convolutional neural network (CNN)- and support vector machine (SVM)-based machine learning models to classify the emotional states and the attention level of the participants to a video conversation. This application visualizes their attention and emotion analytics in a meaningful manner. This proposed system provides an artificial intelligence (AI)-powered analytics system with optimized machine learning models to monitor the audience and prepare insightful reports on the basis of participants\u2019 facial features throughout the video conversation. One of the main objectives of this research is to utilize the neural accelerator chip to enhance emotion and attention detection tasks. A custom CNN developed by Gyrfalcon Technology Inc (GTI) named GnetDet was used in this system to run the trained model on their GTI Lightspeeur 2803 neural accelerator chip.<\/jats:p>","DOI":"10.3390\/a15050150","type":"journal-article","created":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T13:40:57Z","timestamp":1651066857000},"page":"150","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["An Emotion and Attention Recognition System to Classify the Level of Engagement to a Video Conversation by Participants in Real Time Using Machine Learning Models and Utilizing a Neural Accelerator Chip"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8174-1537","authenticated-orcid":false,"given":"Janith","family":"Kodithuwakku","sequence":"first","affiliation":[{"name":"Digital Business and Innovations, Tokyo International University, Saitama 350-1197, Japan"}]},{"given":"Dilki Dandeniya","family":"Arachchi","sequence":"additional","affiliation":[{"name":"Digital Business and Innovations, Tokyo International University, Saitama 350-1197, Japan"}]},{"given":"Jay","family":"Rajasekera","sequence":"additional","affiliation":[{"name":"Digital Business and Innovations, Tokyo International University, Saitama 350-1197, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1080\/01587919.2020.1821607","article-title":"Evaluating videoconferencing systems for the quality of the educational experience","volume":"41","author":"Correia","year":"2020","journal-title":"Distance Educ."},{"key":"ref_2","first-page":"68","article-title":"Students\u2019 E-Learning Experience through a Synchronous Zoom Web Conference System","volume":"5","author":"Rahayu","year":"2020","journal-title":"J. ELT Res. Acad. J. Stud. Engl. Lang. Teach. Learn."},{"key":"ref_3","unstructured":"Zoom (2022, February 01). Web-Native SDKs-Zoom Software Development Kit (Zoom SDK)-Zoom Developer-Technical Documentation and Reference. Available online: htps:\/\/marketplace.zoom.us\/docs."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dalgleish, T., and Power, M.J. (1999). Basic Emotions. Handbook of Cognition and Emotion, John Wiley & Sons Ltd.","DOI":"10.1002\/0470013494"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1109\/TAFFC.2014.2316163","article-title":"The faces of engagement: Automatic recognition of student engagementfrom facial expressions","volume":"5","author":"Whitehill","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Li, Y., Li, X., Ratcliffe, M., Liu, L., Qi, Y., and Liu, Q. (2011, January 18). A real-time EEG-based BCI system for attention recognition in ubiquitous environment. Proceedings of the 2011 International Workshop on Ubiquitous Affective Awareness and Intelligent Interaction, Beijing, China.","DOI":"10.1145\/2030092.2030099"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1007\/s40846-017-0344-z","article-title":"EEG-based detection model for evaluating and improving learning attention","volume":"38","author":"Chiang","year":"2018","journal-title":"J. Med. Biol. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2020.01.011","article-title":"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review","volume":"59","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ekman, P., and Friesen, W.V. (1978). Facial action coding system. Environmental Psychology & Nonverbal Behavior, Consulting Psychologists Press.","DOI":"10.1037\/t27734-000"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1169","DOI":"10.1109\/29.1644","article-title":"Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression","volume":"36","author":"Daugman","year":"1988","journal-title":"IEEE Trans. Acoust. Speech Signal Processing"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Asthana, A., Saragih, J., Wagner, M., and Goecke, R. (2009, January 10\u201312). Evaluating AAM fitting methods for facial expression recognition. Proceedings of the 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, The Netherlands.","DOI":"10.1109\/ACII.2009.5349489"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wu, S., Du, Z., Li, W., Huang, D., and Wang, Y. (2019, January 14\u201318). Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze. Proceedings of the 2019 International Conference on Multimodal Interaction, Suzhou, China.","DOI":"10.1145\/3340555.3353739"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Xia, L. (2014, January 25\u201326). Facial Expression Recognition Based on SVM. Proceedings of the 7th International Conference on Intelligent Computation Technology and Automation, Changsha, China.","DOI":"10.1109\/ICICTA.2014.69"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Fan, S., Shen, Z., Jiang, M., Koenig, B.L., Xu, J., Kankanhalli, M.S., and Zhao, Q. (2018, January 18\u201323). Emotional attention: A study of image sentiment and visual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00785"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cordel, M.O., Fan, S., Shen, Z., and Kankanhalli, M.S. (2019, January 15\u201320). Emotion-aware human attention prediction. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00415"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1109\/TCSS.2021.3059318","article-title":"2021. Collaborative edge computing with FPGA-based CNN accelerators for energy-efficient and time-aware face tracking system","volume":"9","author":"Liu","year":"2021","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1697","DOI":"10.1109\/JPROC.2019.2915983","article-title":"Edge computing for autonomous driving: Opportunities and challenges","volume":"107","author":"Liu","year":"2019","journal-title":"Proc. IEEE"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1109\/TC.2019.2941875","article-title":"Fast and efficient convolutional accelerator for edge computing","volume":"69","author":"Ardakani","year":"2019","journal-title":"IEEE Trans. Comput."},{"key":"ref_19","unstructured":"Yu, W., Yang, K., Bai, Y., Xiao, T., Yao, H., and Rui, Y. (2016, January 19\u201324). Visualizing and comparing AlexNet and VGG using deconvolutional layers. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_20","unstructured":"GTI (2022, January 12). Web-Gyrfalcon Technology Inc Official Website. Available online: https:\/\/www.gyrfalcontech.ai\/."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lu, L., Xie, J., Huang, R., Zhang, J., Lin, W., and Liang, Y. (May, January 28). An efficient hardware accelerator for sparse convolutional neural networks on FPGAs. Proceedings of the 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), San Diego, CA, USA.","DOI":"10.1109\/FCCM.2019.00013"},{"key":"ref_22","unstructured":"Dlib (2022, February 01). Web-Dlib Documentation. Available online: http:\/\/dlib.net\/."},{"key":"ref_23","first-page":"1755","article-title":"Dlib-ml: A machine learning toolkit","volume":"10","author":"King","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_24","unstructured":"OpenCV (2022, February 01). Web-OpenCV Documentation. Available online: https:\/\/docs.opencv.org\/."},{"key":"ref_25","unstructured":"Bradski, G., and Kaehler, A. (2008). Learning OpenCV: Computer Vision with the OpenCV Library, O\u2019Reilly Media, Inc."},{"key":"ref_26","unstructured":"PyImageSearch (2022, February 06). Web\u2013Article by PyImageSearch-Facial Landmarks with Dlib, OpenCV, and Python. Available online: https:\/\/www.pyimagesearch.com\/2017\/04\/03\/facial-landmarks-dlib-opencv-python\/."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sarsenov, A., and Latuta, K. (September, January 31). Face recognition based on facial landmarks. Proceedings of the 2017 IEEE 11th International Conference on Application of Information and Communication Technologies (AICT), Hokkaido, Japan.","DOI":"10.1109\/ICAICT.2017.8687015"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kazemi, V., and Sullivan, J. (2014, January 23\u201328). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.241"},{"key":"ref_29","unstructured":"Scikit-learn (2022, January 15). Web-Support Vector Machines-Scikit-Learn 1.0.1 Documentation. Available online: https:\/\/scikit-learn.org\/stable\/modules\/svm.html."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kramer, O. (2016). Scikit-learn. Machine Learning for Evolution Strategies, Springer.","DOI":"10.1007\/978-3-319-33383-0_5"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1016\/j.ieri.2012.06.171","article-title":"Facial expression recognition based on SVM in E-learning","volume":"2","author":"Chen","year":"2012","journal-title":"Ieri Procedia"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/5\/150\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:01:54Z","timestamp":1760137314000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/5\/150"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,27]]},"references-count":31,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["a15050150"],"URL":"https:\/\/doi.org\/10.3390\/a15050150","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,27]]}}}