{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:28:11Z","timestamp":1740122891563,"version":"3.37.3"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"28","license":[{"start":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T00:00:00Z","timestamp":1707091200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T00:00:00Z","timestamp":1707091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002666","name":"Aalto University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002666","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Active Speaker Detection (ASD) aims at identifying the active speaker among multiple speakers in a video scene. Previous ASD models often seek audio and visual features from long video clips with a complex 3D Convolutional Neural Network (CNN) architecture. However, models based on 3D CNNs can generate discriminative spatial-temporal features, but this comes at the expense of computational complexity, and they frequently face challenges in detecting active speakers in short video clips. This work proposes the <jats:bold>A<\/jats:bold>ctive <jats:bold>S<\/jats:bold>peaker <jats:bold>Net<\/jats:bold>work (AS-Net) model, a simple yet effective ASD method tailored for detecting active speakers in relatively short video clips without relying on 3D CNNs. Instead, it incorporates the Temporal Shift Module (TSM) into 2D CNNs, facilitating the extraction of dense temporal visual features without the need for additional computations. Moreover, self-attention and cross-attention schemes are introduced to enhance long-term temporal audio-visual synchronization, thereby improving ASD performance. Experimental results demonstrate that AS-Net outperforms state-of-the-art 2D CNN-based methods on the AVA-ActiveSpeaker dataset and remains competitive with the methods utilizing more complex architectures.<\/jats:p>","DOI":"10.1007\/s11042-024-18457-9","type":"journal-article","created":{"date-parts":[[2024,2,5]],"date-time":"2024-02-05T06:02:16Z","timestamp":1707112936000},"page":"72027-72042","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["AS-Net: active speaker detection using deep audio-visual attention"],"prefix":"10.1007","volume":"83","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6317-9752","authenticated-orcid":false,"given":"Abduljalil","family":"Radman","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jorma","family":"Laaksonen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,2,5]]},"reference":[{"key":"18457_CR1","doi-asserted-by":"crossref","unstructured":"Chung S-W, Kang HG, Chung JS (2020) Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision. In: INTERSPEECH. pp 3486\u20133490","DOI":"10.21437\/Interspeech.2020-1113"},{"key":"18457_CR2","doi-asserted-by":"publisher","first-page":"942","DOI":"10.1109\/TMM.2021.3061800","volume":"24","author":"X Qian","year":"2021","unstructured":"Qian X, Brutti A, Lanz O, Omologo M, Cavallaro A (2021) Audio-visual tracking of concurrent speakers. IEEE Trans Multimed 24:942\u2013954","journal-title":"IEEE Trans Multimed"},{"issue":"9","key":"18457_CR3","doi-asserted-by":"publisher","first-page":"13667","DOI":"10.1007\/s11042-022-13746-7","volume":"82","author":"L Pibre","year":"2023","unstructured":"Pibre L, Madrigal F, Equoy C, Lerasle F, Pellegrini T, Pinquier J, Ferran\u00e9 I (2023) Audio-video fusion strategies for active speaker detection in meetings. Multimed Tools Appl 82(9):13667\u201313688","journal-title":"Multimed Tools Appl"},{"key":"18457_CR4","doi-asserted-by":"crossref","unstructured":"Wang Q, Downey C, Wan L, Mansfield PA, Moreno IL (2018) Speaker diarization with LSTM. In: ICASSP. pp 5239\u20135243","DOI":"10.1109\/ICASSP.2018.8462628"},{"key":"18457_CR5","doi-asserted-by":"publisher","first-page":"27685","DOI":"10.1007\/s11042-018-5944-2","volume":"77","author":"P Caba\u00f1as-Molero","year":"2018","unstructured":"Caba\u00f1as-Molero P, Lucena M, Fuertes JM, Vera-Candeas P, Ruiz-Reyes N (2018) Multimodal speaker diarization for meetings using volume-evaluated SRP-PHAT and video analysis. Multimed Tools Appl 77:27685\u201327707","journal-title":"Multimed Tools Appl"},{"key":"18457_CR6","doi-asserted-by":"crossref","unstructured":"Chan DY, Wang J-F, Chin H-T (2023) A new speaker-diarization technology with denoising spectral-LSTM for online automatic multi-dialogue recording. Multimed Tools Appl 1\u201316","DOI":"10.1007\/s11042-023-17283-9"},{"key":"18457_CR7","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.cviu.2018.02.001","volume":"173","author":"JS Chung","year":"2018","unstructured":"Chung JS, Zisserman A (2018) Learning to lip read words by watching videos. Comput Vision Image Understand 173:76\u201385","journal-title":"Comput Vision Image Understand"},{"key":"18457_CR8","doi-asserted-by":"crossref","unstructured":"Kumar P, Malik S, Raman B (2023) Interpretable multimodal emotion recognition using hybrid fusion of speech and image data. Multimed Tools Appl 1\u201322","DOI":"10.1007\/s11042-023-16443-1"},{"key":"18457_CR9","doi-asserted-by":"crossref","unstructured":"Chakravarty P, Mirzaei S, Tuytelaars T, Van\u00a0hamme H (2015) Who\u2019s speaking? audio-supervised classification of active speakers in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. pp 87\u201390","DOI":"10.1145\/2818346.2820780"},{"key":"18457_CR10","doi-asserted-by":"crossref","unstructured":"Fujita Y, Kanda N, Horiguchi S, Nagamatsu K, Watanabe S (2019) End-to-End Neural Speaker Diarization with Permutation-free Objectives. In: INTERSPEECH. pp 4300\u20134304","DOI":"10.21437\/Interspeech.2019-2899"},{"issue":"2","key":"18457_CR11","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1109\/TCDS.2019.2927941","volume":"12","author":"K Stefanov","year":"2019","unstructured":"Stefanov K, Beskow J, Salvi G (2019) Self-supervised vision-based detection of the active speaker as support for socially aware language acquisition. IEEE Trans Cognit Develop Syst 12(2):250\u2013259","journal-title":"IEEE Trans Cognit Develop Syst"},{"key":"18457_CR12","doi-asserted-by":"crossref","unstructured":"Prajwal K, Afouras T, Zisserman A (2022) Sub-word level lip reading with visual attention. In: CVPR. pp 5162\u20135172","DOI":"10.1109\/CVPR52688.2022.00510"},{"key":"18457_CR13","doi-asserted-by":"crossref","unstructured":"Chung JS, Zisserman A (2016) Out of time: automated lip sync in the wild. In: ACCV. pp 251\u2013263","DOI":"10.1007\/978-3-319-54427-4_19"},{"issue":"5","key":"18457_CR14","doi-asserted-by":"publisher","first-page":"1086","DOI":"10.1109\/TPAMI.2017.2648793","volume":"40","author":"ID Gebru","year":"2017","unstructured":"Gebru ID, Ba S, Li X, Horaud R (2017) Audio-visual speaker diarization based on spatiotemporal bayesian fusion. IEEE Trans Pattern Anal Mach Intell 40(5):1086\u20131099","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"18457_CR15","doi-asserted-by":"crossref","unstructured":"Alc\u00e1zar JL, Caba F, Mai L, Perazzi F, Lee J-Y, Arbel\u00e1ez P, Ghanem B (2020) Active speakers in context. In: CVPR. pp 12465\u201312474","DOI":"10.1109\/CVPR42600.2020.01248"},{"key":"18457_CR16","doi-asserted-by":"crossref","unstructured":"Roth J, Chaudhuri S, Klejch O, Marvin R, Gallagher A, Kaver L, Ramaswamy S, Stopczynski A, Schmid C, Xi Z et\u00a0al (2020) Ava Active Speaker: An audio-visual dataset for active speaker detection. In: ICASSP. pp 4492\u20134496","DOI":"10.1109\/ICASSP40776.2020.9053900"},{"issue":"4","key":"18457_CR17","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1016\/j.conb.2006.06.008","volume":"16","author":"DA Bulkin","year":"2006","unstructured":"Bulkin DA, Groh JM (2006) Seeing sounds: visual and auditory interactions in the brain. Curr Opinion Neurobiol 16(4):415\u2013419","journal-title":"Curr Opinion Neurobiol"},{"issue":"8","key":"18457_CR18","doi-asserted-by":"publisher","first-page":"11239","DOI":"10.1007\/s11042-022-13557-w","volume":"82","author":"E Ghaleb","year":"2023","unstructured":"Ghaleb E, Niehues J, Asteriadis S (2023) Joint modelling of audio-visual cues using attention mechanisms for emotion recognition. Multimed Tools Appl 82(8):11239\u201311264","journal-title":"Multimed Tools Appl"},{"key":"18457_CR19","doi-asserted-by":"crossref","unstructured":"Alc\u00e1zar JL, Caba F, Thabet AK, Ghanem B (2021) Maas: Multi-modal assignation for active speaker detection. In: ICCV. pp 265\u2013274","DOI":"10.1109\/ICCV48922.2021.00033"},{"key":"18457_CR20","unstructured":"Chung JS (2019) Naver at ActivityNet Challenge 2019\u2013Task B Active Speaker Detection (AVA). arXiv:1906.10555"},{"key":"18457_CR21","doi-asserted-by":"crossref","unstructured":"Carneiro H, Weber C, Wermter S (2021) FaVoA: Face-Voice association favours ambiguous speaker detection. In: ICANN. pp 439\u2013450","DOI":"10.1007\/978-3-030-86362-3_36"},{"key":"18457_CR22","doi-asserted-by":"crossref","unstructured":"Zhang Y, Liang S, Yang S, Liu X, Wu Z, Shan S, Chen X (2021) Unicon: Unified context network for robust active speaker detection. In: Proceedings of the 29th ACM international conference on multimedia. pp 3964\u20133972","DOI":"10.1145\/3474085.3475275"},{"key":"18457_CR23","doi-asserted-by":"crossref","unstructured":"Alc\u00e1zar JL, Cordes M, Zhao C, Ghanem B (2022) End-to-end active speaker detection. In: ECCV. pp 126\u2013143","DOI":"10.1007\/978-3-031-19836-6_8"},{"key":"18457_CR24","doi-asserted-by":"crossref","unstructured":"Min K, Roy S, Tripathi S, Guha T, Majumdar S (2022) Learning long-term spatial-temporal graphs for active speaker detection. In: ECCV. pp 371\u2013387","DOI":"10.1007\/978-3-031-19833-5_22"},{"key":"18457_CR25","doi-asserted-by":"crossref","unstructured":"K\u00f6p\u00fckl\u00fc O, Taseska M, Rigoll G (2021) How to design a three-stage architecture for audio-visual active speaker detection in the wild. In: ICCV. pp 1193\u20131203","DOI":"10.1109\/ICCV48922.2021.00123"},{"key":"18457_CR26","doi-asserted-by":"crossref","unstructured":"Huang C, Koishida K (2020) Improved active speaker detection based on optical flow. In: CVPR workshops. pp 950\u2013951","DOI":"10.1109\/CVPRW50498.2020.00483"},{"key":"18457_CR27","doi-asserted-by":"crossref","unstructured":"Tao R, Pan Z, Das RK, Qian X, Shou MZ, Li H (2021) Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. In: Proceedings of the 29th ACM international conference on multimedia. pp 3927\u20133935","DOI":"10.1145\/3474085.3475587"},{"key":"18457_CR28","doi-asserted-by":"crossref","unstructured":"Datta G, Etchart T, Yadav V, Hedau V, Natarajan P, Chang S-F (2022) Asd-Transformer: Efficient active speaker detection using self and multimodal transformers. In: ICASSP. pp 4568\u20134572","DOI":"10.1109\/ICASSP43922.2022.9746991"},{"key":"18457_CR29","doi-asserted-by":"publisher","first-page":"5800","DOI":"10.1109\/TMM.2022.3199109","volume":"25","author":"J Xiong","year":"2022","unstructured":"Xiong J, Zhou Y, Zhang P, Xie L, Huang W, Zha Y (2022) Look &listen: Multi-modal correlation learning for active speaker detection and speech enhancement. IEEE Trans Multimed 25:5800\u20135812","journal-title":"IEEE Trans Multimed"},{"issue":"64\u201367","key":"18457_CR30","first-page":"2","volume":"5","author":"LR Medsker","year":"2001","unstructured":"Medsker LR, Jain L (2001) Recurrent neural networks. Design Appl 5(64\u201367):2","journal-title":"Recurrent neural networks. Design Appl"},{"key":"18457_CR31","doi-asserted-by":"crossref","unstructured":"Lin J, Gan C, Han S (2019) Tsm: Temporal shift module for efficient video understanding. In: ICCV. pp 7083\u20137093","DOI":"10.1109\/ICCV.2019.00718"},{"issue":"8","key":"18457_CR32","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735\u20131780","journal-title":"Neural Comput"},{"key":"18457_CR33","doi-asserted-by":"crossref","unstructured":"Cho K, Merri\u00ebnboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. In: EMNLP. pp 1724\u20131734","DOI":"10.3115\/v1\/D14-1179"},{"key":"18457_CR34","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30"},{"key":"18457_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2022.108792","volume":"130","author":"H Tang","year":"2022","unstructured":"Tang H, Yuan C, Li Z, Tang J (2022) Learning attention-guided pyramidal features for few-shot fine-grained recognition. Pattern Recognit 130:108792","journal-title":"Pattern Recognit"},{"key":"18457_CR36","doi-asserted-by":"crossref","unstructured":"Tang H, Liu J, Yan S, Yan R, Li Z, Tang J (2023) M3net: Multi-view encoding, matching, and fusion for few-shot fine-grained action recognition. In: Proceedings of the 31st ACM international conference on multimedia. pp 1719\u20131728","DOI":"10.1145\/3581783.3612221"},{"key":"18457_CR37","doi-asserted-by":"crossref","unstructured":"Naik D, CD J (2023) Video captioning using sentence vector-enabled convolutional framework with short-connected LSTM. Multimed Tools Appl 1\u201327","DOI":"10.1007\/s11042-023-15978-7"},{"key":"18457_CR38","doi-asserted-by":"crossref","unstructured":"Martin AF, Greenberg CS (2010) The NIST 2010 speaker recognition evaluation. In: INTERSPEECH. pp 2726\u20132729","DOI":"10.21437\/Interspeech.2010-722"},{"key":"18457_CR39","doi-asserted-by":"crossref","unstructured":"Kim YJ, Heo H-S, Choe S, Chung S-W, Kwon Y, Lee B-J, Kwon Y, Chung JS (2021) Look who\u2019s talking: Active speaker detection in the wild. In: INTERSPEECH. pp 3675\u20133679","DOI":"10.21437\/Interspeech.2021-2041"},{"key":"18457_CR40","unstructured":"Zhang Y-H, Xiao J, Yang S, Shan S (2019) Multi-task learning for audio-visual active speaker detection. The ActivityNet Large-Scale Activity Recognition Challenge 1\u20134"},{"key":"18457_CR41","doi-asserted-by":"crossref","unstructured":"Chung S-W, Chung JS, Kang H-G (2019) Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In: ICASSP. pp 3965\u20133969","DOI":"10.1109\/ICASSP.2019.8682524"},{"key":"18457_CR42","doi-asserted-by":"crossref","unstructured":"Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: Delving deep into convolutional nets. In: BMVC","DOI":"10.5244\/C.28.6"},{"issue":"11","key":"18457_CR43","doi-asserted-by":"publisher","first-page":"2673","DOI":"10.1109\/78.650093","volume":"45","author":"M Schuster","year":"1997","unstructured":"Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673\u20132681","journal-title":"IEEE Trans Signal Process"},{"key":"18457_CR44","unstructured":"Wang X, Cheng F, Bertasius G, Crandall D (2023) Loconet: Long-short context network for active speaker detection. arXiv:2301.08237"},{"key":"18457_CR45","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR. pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"18457_CR46","doi-asserted-by":"crossref","unstructured":"Kim C, Shin HV, Oh T-H, Kaspar A, Elgharib M, Matusik W (2018) On learning associations of faces and voices. In: ACCV. pp 276\u2013292","DOI":"10.1007\/978-3-030-20873-8_18"},{"key":"18457_CR47","unstructured":"Kingma DP, Ba J (2015) Adam: A method for stochastic optimizations. In: ICLR. pp 1\u201315"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-024-18457-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-024-18457-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-024-18457-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T17:10:00Z","timestamp":1722359400000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-024-18457-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,5]]},"references-count":47,"journal-issue":{"issue":"28","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["18457"],"URL":"https:\/\/doi.org\/10.1007\/s11042-024-18457-9","relation":{},"ISSN":["1573-7721"],"issn-type":[{"type":"electronic","value":"1573-7721"}],"subject":[],"published":{"date-parts":[[2024,2,5]]},"assertion":[{"value":"30 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 January 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 January 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 February 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}