{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T10:27:18Z","timestamp":1791196038354,"version":"4.1.0"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T00:00:00Z","timestamp":1716249600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T00:00:00Z","timestamp":1716249600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Intell Syst"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper introduces a deep learning methodology for analyzing audience engagement in online video events. The proposed deep learning framework consists of six layers and starts with keyframe extraction from the video stream and the participants\u2019 face detection. Subsequently, the head pose and emotion per participant are estimated using the HopeNet and JAA-Net deep architectures. Complementary to video analysis, the audio signal is also processed using a neural network that follows the DenseNet-121 architecture. Its purpose is to detect events related to audience engagement, including speech, pauses, and applause. With the combined analysis of video and audio streams, the interest and attention of each participant are inferred more accurately. An experimental evaluation is performed on a newly generated dataset consisting of recordings from online video events, where the proposed framework achieves promising results. Concretely, the <jats:italic>F<\/jats:italic>1 scores were 79.21% for interest estimation according to pose, 65.38% for emotion estimation, and 80% for sound event detection. The proposed framework has applications in online educational events, where it can help tutors assess audience engagement and comprehension while hinting at points in their lectures that may require further clarification. It is effective for video streaming platforms that want to provide video recommendations to online users according to audience engagement.<\/jats:p>","DOI":"10.1007\/s44196-024-00512-w","type":"journal-article","created":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T13:01:42Z","timestamp":1716296502000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["A Deep Learning Framework for Monitoring Audience Engagement in Online Video Events"],"prefix":"10.1007","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9738-1553","authenticated-orcid":false,"given":"Alexandros","family":"Vrochidis","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikolaos","family":"Dimitriou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stelios","family":"Krinidis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Savvas","family":"Panagiotidis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stathis","family":"Parcharidis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitrios","family":"Tzovaras","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,5,21]]},"reference":[{"key":"512_CR1","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1007\/s00138-022-01273-2","volume":"33","author":"Z Jiao","year":"2022","unstructured":"Jiao, Z., Lei, H., Zong, H., Cai, Y., Zhong, Z.: Potential escalator-related injury identification and prevention based on multi-module integrated system for public health. Mach. Vis. Appl. 33, 29 (2022). https:\/\/doi.org\/10.1007\/s00138-022-01273-2","journal-title":"Mach. Vis. Appl."},{"key":"512_CR2","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1007\/s00138-020-01064-7","volume":"31","author":"L Citraro","year":"2020","unstructured":"Citraro, L., M\u00e1rquez-Neila, P., Savar\u00e8, S., Jayaram, V., Dubout, C., Renaut, F., Hasfura, A., Shitrit, B., Fua, P.: Real-time camera pose estimation for sports fields. Mach. Vis. Appl. 31, 16 (2020). https:\/\/doi.org\/10.1007\/s00138-020-01064-7","journal-title":"Mach. Vis. Appl."},{"key":"512_CR3","doi-asserted-by":"publisher","unstructured":"Vrochidis, A., Dimitriou, N., Krinidis, S., Panagiotidis, S., Parcharidis, S., Tzovaras, D.: A multi-modal audience analysis system for predicting popularity of online videos. In: Iliadis, L., Macintyre, J., Jayne, C., Pimenidis, E. (eds.) EANN 2021, 3, 465\u2013476 (2021). https:\/\/doi.org\/10.1007\/978-3-030-80568-5_38","DOI":"10.1007\/978-3-030-80568-5_38"},{"key":"512_CR4","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1007\/s00138-023-01397-z","volume":"34","author":"MLS Kokila","year":"2023","unstructured":"Kokila, M.L.S., Christopher, V.B., Sajan, R.I., Akhila, T.S., Kavitha, M.J.: Efficient abnormality detection using patch-based 3D convolution with recurrent model. Mach. Vis. Appl. 34, 54 (2023). https:\/\/doi.org\/10.1007\/s00138-023-01397-z","journal-title":"Mach. Vis. Appl."},{"key":"512_CR5","doi-asserted-by":"publisher","unstructured":"Vrochidis, A., Dimitriou, N., Krinidis, S., Panagiotidis, S., Parcharidis, S., Tzovaras, D.: Video Popularity prediction through fusing early viewership with video content. In: Vincze, M., Patten, T., Christensen, H., Nalpantidis, L., Liu, M., (eds.) Computer Vision Systems, ICVS 2021, 12899, 159\u2013168 (2021). https:\/\/doi.org\/10.1007\/978-3-030-87156-7_13","DOI":"10.1007\/978-3-030-87156-7_13"},{"key":"512_CR6","doi-asserted-by":"crossref","unstructured":"Chen, R., Zhou, P., Wang, W., Chen, N., Peng, P., Sun, X., Wang, W.: PR-Net: preference reasoning for personalized video highlight detection. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 7980\u20137989 (2021). arXiv:2109.01799","DOI":"10.1109\/ICCV48922.2021.00788"},{"key":"512_CR7","doi-asserted-by":"publisher","first-page":"23157","DOI":"10.1007\/s11042-020-09385-5","volume":"80","author":"R Abed","year":"2021","unstructured":"Abed, R., Bahroun, S., Zagrouba, E.: KeyFrame extraction based on face quality measurement and convolutional neural network for efficient face recognition in videos. Multimed. Tools Appl. 80, 23157\u201323179 (2021). https:\/\/doi.org\/10.1007\/s11042-020-09385-5","journal-title":"Multimed. Tools Appl."},{"issue":"7","key":"512_CR8","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1109\/TCSVT.2006.877419","volume":"16","author":"X Song","year":"2006","unstructured":"Song, X., Fan, G.: Joint key-frame extraction and object segmentation for content-based video analysis. IEEE Trans. Circuits Syst. Video Technol. 16(7), 904\u2013914 (2006)","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"512_CR9","doi-asserted-by":"crossref","unstructured":"Zhang, M., Tian, L., Li, C.: Key frame extraction based on entropy difference and perceptual hash. In: IEEE International Symposium on Multimedia (ISM) 2017, pp. 557\u2013560 (2017).","DOI":"10.1109\/ISM.2017.109"},{"key":"512_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2174\/2213275911666180719111118","volume":"11","author":"KAP Milan","year":"2018","unstructured":"Milan, K.A.P., Jeyaraman, K., Arockia, P.J.R.: Key-frame extraction techniques: a review. Recent Patents Comput. Sci. 11, 1 (2018). https:\/\/doi.org\/10.2174\/2213275911666180719111118","journal-title":"Recent Patents Comput. Sci."},{"issue":"6","key":"512_CR11","doi-asserted-by":"publisher","first-page":"9294","DOI":"10.3934\/mbe.2021457","volume":"18","author":"Y Sun","year":"2021","unstructured":"Sun, Y., Li, P., Jiang, Z., Hu, S.: Feature fusion and clustering for key frame extraction. Math. Biosci. Eng. 18(6), 9294\u20139311 (2021). https:\/\/doi.org\/10.3934\/mbe.2021457","journal-title":"Math. Biosci. Eng."},{"issue":"1","key":"512_CR12","doi-asserted-by":"publisher","first-page":"215000230","DOI":"10.1142\/S021812662150002X","volume":"30","author":"B Tang","year":"2021","unstructured":"Tang, B., Chen, W.: A description scheme for video overview based on scene detection and face clustering. J Circ. Syst. Comput. 30(1), 215000230 (2021). https:\/\/doi.org\/10.1142\/S021812662150002X","journal-title":"J Circ. Syst. Comput."},{"key":"512_CR13","doi-asserted-by":"publisher","first-page":"24663","DOI":"10.1007\/s11042-018-5658-5","volume":"77","author":"D Luo","year":"2018","unstructured":"Luo, D., Wen, G., Li, D., Hu, Y., Huan, E.: Deep-learning-based face detection using iterative bounding-box regression. Multimed. Tools Appl. 77, 24663\u201324680 (2018). https:\/\/doi.org\/10.1007\/s11042-018-5658-5","journal-title":"Multimed. Tools Appl."},{"issue":"11","key":"512_CR14","doi-asserted-by":"publisher","first-page":"4284","DOI":"10.1109\/TCSVT.2019.2955463","volume":"30","author":"H Mo","year":"2020","unstructured":"Mo, H., Liu, L., Zhu, W., Li, Q., Liu, H., Yin, S., Wei, S.: A multi-task hardwired accelerator for face detection and alignment. IEEE Trans. Circuits Syst. Video Technol. 30(11), 4284\u20134298 (2020)","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"issue":"4","key":"512_CR15","doi-asserted-by":"publisher","first-page":"1792","DOI":"10.1109\/TCSVT.2021.3082635","volume":"32","author":"X Li","year":"2022","unstructured":"Li, X., Lai, S., Qian, X.: DBCFace: towards pure convolutional neural network face detection. IEEE Trans. Circuits Syst. Video Technol. 32(4), 1792\u20131804 (2022)","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"512_CR16","doi-asserted-by":"publisher","unstructured":"Jiang, H., Learned-Miller, E.: Face detection with the faster R-CNN. In: 12th IEEE International Conference on Automatic Face & Gesture Recognition 2017, pp. 650\u2013657 (2017). https:\/\/doi.org\/10.1109\/FG.2017.82","DOI":"10.1109\/FG.2017.82"},{"key":"512_CR17","doi-asserted-by":"publisher","unstructured":"Yang, S., Luo, P., Loy, C.C., Tang, X.: WIDER FACE: A face detection benchmark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5525\u20135533 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.596","DOI":"10.1109\/CVPR.2016.596"},{"key":"512_CR18","unstructured":"Jain, V., Learned-Miller, E.: FDDB: A benchmark for face detection in unconstrained settings. University of Massachusetts, Amherst technical report 2, 4 (2010)."},{"key":"512_CR19","doi-asserted-by":"publisher","unstructured":"Klare, B.F., Klein, B., Taborsky, E., Blanton, A., Cheney, J., Allen, K., Gother, P., Mah, A., Burge, M., Jain, A.K.: Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus benchmark A. In: CVPR, 1931\u20131939 (2015). https:\/\/doi.org\/10.1109\/BTAS.2018.8698561","DOI":"10.1109\/BTAS.2018.8698561"},{"key":"512_CR20","doi-asserted-by":"publisher","unstructured":"Shi, X., Shan, S., Kan, M., Wu, S., Chen, X.: Real-time rotation-invariant face detection with progressive calibration networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2295\u20132303 (2018). https:\/\/doi.org\/10.1109\/CVPR.2018.00244","DOI":"10.1109\/CVPR.2018.00244"},{"key":"512_CR21","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.bdr.2017.06.002","volume":"11","author":"D Triantafyllidou","year":"2018","unstructured":"Triantafyllidou, D., Nousi, P., Tefas, A.: Fast deep convolutional face detection in the wild exploiting hard sample mining. Big data Res. 11, 65\u201376 (2018). https:\/\/doi.org\/10.1016\/j.bdr.2017.06.002","journal-title":"Big data Res."},{"key":"512_CR22","doi-asserted-by":"publisher","unstructured":"Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a Dataset of 230,000 3D Facial Landmarks). In: IEEE International Conference on Computer Vision, pp. 1021\u20131030 (2017). https:\/\/doi.org\/10.1109\/ICCV.2017.116","DOI":"10.1109\/ICCV.2017.116"},{"key":"512_CR23","doi-asserted-by":"publisher","unstructured":"Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S. Z.: Face alignment across large poses: a 3d solution. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 146\u2013155 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.23","DOI":"10.1109\/CVPR.2016.23"},{"key":"512_CR24","doi-asserted-by":"publisher","unstructured":"Park, J., Heo, S., Lee, K., Song, H., Lee, S.: Robust facial pose estimation using landmark selection method for binocular stereo vision. In: 25th IEEE International Conference on Image Processing (ICIP), 186\u2013190 (2018). https:\/\/doi.org\/10.1109\/ICIP.2018.8451443","DOI":"10.1109\/ICIP.2018.8451443"},{"key":"512_CR25","doi-asserted-by":"publisher","unstructured":"Paracchini, M., Marcon, M., Tubaro, S.: Fast and reliable facial landmarks localization in non frontal images. In: 8th European Workshop on Visual Information Processing (EUVIP), pp. 88\u201392 (2019). https:\/\/doi.org\/10.1109\/EUVIP47703.2019.8946249","DOI":"10.1109\/EUVIP47703.2019.8946249"},{"key":"512_CR26","doi-asserted-by":"publisher","unstructured":"Derkach, D., Ruiz, A., Sukno, F. M.: Head pose estimation based on 3-D facial landmarks localization and regression. In: 12th IEEE International Conference on Automatic Face & Gesture Recognition 2017, pp. 820\u2013827 (2017). https:\/\/doi.org\/10.1109\/EUVIP47703.2019.8946249","DOI":"10.1109\/EUVIP47703.2019.8946249"},{"key":"512_CR27","doi-asserted-by":"publisher","unstructured":"Fard, A. P., Abdollahi, H., Mahoor, M.: ASMNet: a lightweight deep neural network for face alignment and pose estimation. In: IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1521\u20131530 (2021). https:\/\/doi.org\/10.1109\/CVPRW53098.2021.00168","DOI":"10.1109\/CVPRW53098.2021.00168"},{"issue":"5","key":"512_CR28","doi-asserted-by":"publisher","first-page":"690","DOI":"10.26599\/TST.2020.9010001","volume":"25","author":"X Yang","year":"2020","unstructured":"Yang, X., Jia, X., Yuan, M., Yan, D.M.: Real-time facial pose estimation and tracking by coarse-to-fine iterative optimization. Tsinghua Sci. Technol. 25(5), 690\u2013700 (2020). https:\/\/doi.org\/10.26599\/TST.2020.9010001","journal-title":"Tsinghua Sci. Technol."},{"key":"512_CR29","doi-asserted-by":"publisher","unstructured":"Koujan, M. R., Alhabawee, L., Giannakakis, G., Pugeault, N., Roussos, A.: Real-time facial expression recognition \u201cIn The Wild\u201d by disentangling 3D expression from identity. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 24\u201331 (2020). https:\/\/doi.org\/10.1109\/FG47880.2020.00084","DOI":"10.1109\/FG47880.2020.00084"},{"key":"512_CR30","doi-asserted-by":"publisher","unstructured":"Xi, O., Kawaai, S., Goh, E. G. H., Shen, S., Wan, D., Ming, H., Huang, D. Y.: Audio-visual emotion recognition using deep transfer learning and multiple temporal models. In: 19th ACM International Conference on Multimodal Interaction 2017, 577\u2013582 (2017). https:\/\/doi.org\/10.1145\/3136755.3143012","DOI":"10.1145\/3136755.3143012"},{"key":"512_CR31","unstructured":"Shao, Z., Liu, Z., Cai, J., Wu, Y., Ma, L.: Facial action unit detection using attention and relation learning. In IEEE transactions on affective computing, (2019). arXiv:1808.03457"},{"key":"512_CR32","doi-asserted-by":"publisher","first-page":"3","DOI":"10.32604\/cmc.2022.023328","volume":"71","author":"SA Rizwan","year":"2022","unstructured":"Rizwan, S.A., Ghadi, Y., Jalal, A., Kim, K.: Automated facial expression recognition and age estimation using deep learning. Comput. Mater. Contin. 71, 3 (2022). https:\/\/doi.org\/10.32604\/cmc.2022.023328","journal-title":"Comput. Mater. Contin."},{"key":"512_CR33","doi-asserted-by":"publisher","unstructured":"Niu, X., Han, H., Yang, S., Huang, Y., Shan, S.: Local Relationship Learning With Person-Specific Shape Regularization for Facial Action Unit Detection. IEEE\/CVF Conference on Computer Vision and Pattern Recognition CVPR, 11909\u201311918 (2019). https:\/\/doi.org\/10.1109\/CVPR.2019.01219","DOI":"10.1109\/CVPR.2019.01219"},{"key":"512_CR34","unstructured":"Phan, H., Pham, L., Koch, P., Duong, N. Q. K., Mcloughlin, I., Alfred, M.: Audio Event Detection and Localization with Multitask Regression Network. Technical Report (2020)."},{"key":"512_CR35","doi-asserted-by":"publisher","unstructured":"Greco, A., Saggese, A., Vento, M., Vigilante, V.: SoReNet: a novel deep network for audio surveillance applications. In: IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 546\u2013551 (2019). https:\/\/doi.org\/10.1109\/SMC.2019.8914435","DOI":"10.1109\/SMC.2019.8914435"},{"key":"512_CR36","doi-asserted-by":"publisher","first-page":"3610","DOI":"10.1109\/TIFS.2020.2994740","volume":"15","author":"A Greco","year":"2020","unstructured":"Greco, A., Petkov, N., Saggese, A., Vento, M.: AReN: a deep learning approach for sound event recognition using a brain inspired representation. IEEE Trans. Inf. Forensics Secur. 15, 3610\u20133624 (2020). https:\/\/doi.org\/10.1109\/TIFS.2020.2994740","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"512_CR37","doi-asserted-by":"crossref","unstructured":"Romanov, S.A., Kharkovchuk, N.A., Sinelnikov, M.R., Abrash, M. R., Filinkov, V.: Development of a non-speech audio event detection system. In: IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), 1421\u20131423 (2020).","DOI":"10.1109\/EIConRus49466.2020.9039115"},{"key":"512_CR38","doi-asserted-by":"publisher","unstructured":"Kao, C. C., Wang, W., Sun, M., Wang, C.: R-CRNN: Region-based convolutional recurrent neural network for audio event detection. (2018). https:\/\/doi.org\/10.21437\/Interspeech.2018-2323","DOI":"10.21437\/Interspeech.2018-2323"},{"issue":"10","key":"512_CR39","doi-asserted-by":"publisher","first-page":"1593","DOI":"10.3390\/electronics9101593","volume":"9","author":"I Papadimitriou","year":"2020","unstructured":"Papadimitriou, I., Vafeiadis, A., Lalas, A., Votis, K., Tzovaras, D.: Audio-based event detection at different SNR settings using two-dimensional spectrogram magnitude representations. Electronics 9(10), 1593 (2020). https:\/\/doi.org\/10.3390\/electronics9101593","journal-title":"Electronics"},{"key":"512_CR40","doi-asserted-by":"crossref","unstructured":"Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: Retinaface: Single-stage dense face localisation in the wild. (2019). arXiv:1905.00641","DOI":"10.1109\/CVPR42600.2020.00525"},{"key":"512_CR41","unstructured":"Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. J. Mach. Learn. Res. Proc. Track, 9, 249\u2013256 (2010). https:\/\/proceedings.mlr.press\/v9\/glorot10a.html. Accessed 22 Jan 2024"},{"key":"512_CR42","doi-asserted-by":"publisher","unstructured":"Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 764\u2013773 (2017). https:\/\/doi.org\/10.1109\/ICCV.2017.89","DOI":"10.1109\/ICCV.2017.89"},{"key":"512_CR43","doi-asserted-by":"publisher","unstructured":"Ruiz, N., Chong, E., Rehg, J. M.: Fine-grained head pose estimation without key-points. In: IEEE Computer Vision and Pattern Recognition Workshops, 2155\u2013215509 (2018). https:\/\/doi.org\/10.1109\/CVPRW.2018.00281","DOI":"10.1109\/CVPRW.2018.00281"},{"key":"512_CR44","doi-asserted-by":"publisher","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770\u2013778 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"issue":"2","key":"512_CR45","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1007\/s11263-020-01378-z","volume":"129","author":"Z Shao","year":"2021","unstructured":"Shao, Z., Liu, Z., Cai, J., Ma, L.: JAA-Net: joint facial action unit detection and face alignment via adaptive attention. Int. J. Comput. Vis. 129(2), 321\u2013340 (2021). https:\/\/doi.org\/10.1007\/s11263-020-01378-z","journal-title":"Int. J. Comput. Vis."},{"key":"512_CR46","first-page":"1755","volume":"10","author":"DE King","year":"2009","unstructured":"King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755\u20131758 (2009)","journal-title":"J. Mach. Learn. Res."},{"key":"512_CR47","volume-title":"What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS)","author":"P Ekman","year":"1997","unstructured":"Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press (1997)"},{"key":"512_CR48","doi-asserted-by":"publisher","unstructured":"Hinduja, S., Canavan, S., Kaur, G.: Multimodal fusion of physiological signals and facial action units for pain recognition. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition, 577\u2013581 (2020). https:\/\/doi.org\/10.1109\/FG47880.2020.00060","DOI":"10.1109\/FG47880.2020.00060"},{"key":"512_CR49","unstructured":"Grafsgaard, J., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.: Automatically recognizing facial expression: Predicting engagement and frustration. In: Educational data mining: pp. 43\u201350, (2013)."},{"key":"512_CR50","doi-asserted-by":"publisher","unstructured":"McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., Nieto, O.: librosa: Audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, 8, 18\u201325 (2015). https:\/\/doi.org\/10.25080\/Majora-7b98e3ed-003","DOI":"10.25080\/Majora-7b98e3ed-003"},{"issue":"5","key":"512_CR51","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0196391","volume":"13","author":"SR Livingstone","year":"2018","unstructured":"Livingstone, S.R., Russo, F.A.: The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE 13(5), e0196391 (2018). https:\/\/doi.org\/10.1371\/journal.pone.0196391","journal-title":"PLoS ONE"},{"key":"512_CR52","unstructured":"LiveMedia Platform. INVENTICS A.E., Home Page. https:\/\/www.livemedia.gr (2023). Accessed 18 Jan 2024."},{"key":"512_CR53","doi-asserted-by":"crossref","unstructured":"Vrochidis, A., Tsita, C., Dimitriou, N., Krinidis, S., Panagiotidis, S., Parcharidis, S., Chatzis, V.: User perception and evaluation of a deep learning framework for audience engagement analysis in mass events. In:\u00a0International Conference on Human-Computer Interaction,\u00a0pp. 268\u2013287, (2023).","DOI":"10.1007\/978-3-031-48057-7_17"},{"key":"512_CR54","doi-asserted-by":"publisher","unstructured":"Lin, T., Goyal, P., Girshick, R., He, K., Doll\u00e1r, P.: Focal loss for dense object detection. In: IEEE International Conference on Computer Vision (ICCV), 2999\u20133007 (2017). https:\/\/doi.org\/10.1109\/ICCV.2017.324","DOI":"10.1109\/ICCV.2017.324"},{"key":"512_CR55","unstructured":"Ridnik, T., Ben-Baruch, E., Noy, A.: ImageNet-21K Pretraining for the Masses. (2021). arXiv:2104.10972"},{"key":"512_CR56","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.patrec.2015.06.026","volume":"65","author":"P Foggia","year":"2015","unstructured":"Foggia, P., Petkov, N., Saggese, A., Strisciuglio, N., Vento, M.: Reliable detection of audio events in highly noisy environments. Pattern Recogn. Lett. 65, 22\u201328 (2015). https:\/\/doi.org\/10.1016\/j.patrec.2015.06.026","journal-title":"Pattern Recogn. Lett."},{"issue":"2","key":"512_CR57","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1109\/T-AFFC.2013.4","volume":"4","author":"SM Mavadati","year":"2012","unstructured":"Mavadati, S.M., Mahoor, M.H., Barlett, K., Trinh, P., Cohn, J.F.: DISFA: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151\u2013160 (2012). https:\/\/doi.org\/10.1109\/T-AFFC.2013.4","journal-title":"IEEE Trans. Affect. Comput."},{"key":"512_CR58","unstructured":"Vafeiadis, A., Kalatzis, D., Votis, K., Giakoumis, D., Tzovaras, D., Chen, L., Hamzaoui, R.: Acoustic scene classification: from a hybrid classifier to deep learning. In: Proceedings of the Detection and Classification of Acoustic Scenes and Events 2017 Workshop (DCASE2017) (2017). https:\/\/dora.dmu.ac.uk\/handle\/2086\/15000"},{"key":"512_CR59","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Maaten, L., Weinberger, K.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700\u20134708 (2017). arXiv:1608.06993","DOI":"10.1109\/CVPR.2017.243"},{"issue":"3","key":"512_CR60","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1007\/s11222-017-9746-6","volume":"28","author":"D Hand","year":"2018","unstructured":"Hand, D., Christen, P.: A note on using the F-measure for evaluating record linkage algorithms. Stat. Comput. 28(3), 539\u2013547 (2018). https:\/\/doi.org\/10.1007\/s11222-017-9746-6","journal-title":"Stat. Comput."},{"key":"512_CR61","doi-asserted-by":"publisher","first-page":"692","DOI":"10.1016\/j.imavis.2014.06.002","volume":"32","author":"X Zhang","year":"2014","unstructured":"Zhang, X., Yin, L., Cohn, J., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.: BP4D-Spontaneous: A high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32, 692\u2013706 (2014). https:\/\/doi.org\/10.1016\/j.imavis.2014.06.002","journal-title":"Image Vis. Comput."},{"key":"512_CR62","doi-asserted-by":"publisher","unstructured":"Vrochidis, A., Tsita, C., Dimitriou, N., Krinidis, S., Panagiotidis, S., Parcharidis, S., Tzovaras, D., Chatzis, V.: User Perception and evaluation of a deep learning framework for audience engagement analysis in mass events. In: International Conference on Human-Computer Interaction, 268\u2013287 (2023). https:\/\/doi.org\/10.1007\/978-3-031-48057-7_17","DOI":"10.1007\/978-3-031-48057-7_17"}],"container-title":["International Journal of Computational Intelligence Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-024-00512-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44196-024-00512-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-024-00512-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T13:04:23Z","timestamp":1716296663000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44196-024-00512-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,21]]},"references-count":62,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["512"],"URL":"https:\/\/doi.org\/10.1007\/s44196-024-00512-w","relation":{},"ISSN":["1875-6883"],"issn-type":[{"value":"1875-6883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,21]]},"assertion":[{"value":"26 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; nor in the decision to publish the results.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"There is informed consent for this study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}}],"article-number":"124"}}