{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T17:44:35Z","timestamp":1774719875926,"version":"3.50.1"},"reference-count":51,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2025,5,6]],"date-time":"2025-05-06T00:00:00Z","timestamp":1746489600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Shahid Rajaee Teacher Training University","award":["4891"],"award-info":[{"award-number":["4891"]}]},{"name":"Shahid Rajaee Teacher Training University","award":["11410"],"award-info":[{"award-number":["11410"]}]},{"name":"Cognitive Science and Technology Council","award":["4891"],"award-info":[{"award-number":["4891"]}]},{"name":"Cognitive Science and Technology Council","award":["11410"],"award-info":[{"award-number":["11410"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>Human visual attention is influenced by multiple factors, including visual, auditory, and facial cues. While integrating auditory and visual information enhances prediction accuracy, many existing models rely solely on visual-temporal data. Inspired by cognitive studies, we propose a computational model that combines spatial, temporal, face (low-level and high-level visual cues), and auditory saliency to predict visual attention more effectively. Our approach processes video frames to generate spatial, temporal, and face saliency maps, while an audio branch localizes sound-producing objects. These maps are then integrated to form the final audio-visual saliency map. Experimental results on the audio-visual dataset demonstrate that our model outperforms state-of-the-art image and video saliency models and the basic model and aligns more closely with behavioral and eye-tracking data. Additionally, ablation studies highlight the contribution of each information source to the final prediction.<\/jats:p>","DOI":"10.3390\/bdcc9050120","type":"journal-article","created":{"date-parts":[[2025,5,6]],"date-time":"2025-05-06T09:08:56Z","timestamp":1746522536000},"page":"120","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A Computational\u2013Cognitive Model of Audio-Visual Attention in Dynamic Environments"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-6693-0313","authenticated-orcid":false,"given":"Hamideh","family":"Yazdani","sequence":"first","affiliation":[{"name":"Faculty of Computer Engineering, Shahid Rajaee Teacher Training University, Tehran 16785163, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alireza","family":"Bosaghzadeh","sequence":"additional","affiliation":[{"name":"Faculty of Computer Engineering, Shahid Rajaee Teacher Training University, Tehran 16785163, Iran"},{"name":"Faculty of Information Technology, Ho Chi Minh City Open University, Ho Chi Minh City 700000, Vietnam"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7013-8078","authenticated-orcid":false,"given":"Reza","family":"Ebrahimpour","sequence":"additional","affiliation":[{"name":"Center for Cognitive Science, Institute for Convergence Science and Technology (ICST), Sharif University of Technology, Tehran, 14588-89694, Iran"},{"name":"School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM), Tehran 19395-5746, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fadi","family":"Dornaika","sequence":"additional","affiliation":[{"name":"Faculty of Computer Engineering, University of the Basque Country, 20018 San Sebastian, Spain"},{"name":"IKERBASQUE, Basque Foundation for Science, 48009 Bilbao, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TPAMI.2012.89","article-title":"State-of-the-Art in Visual Attention Modeling","volume":"35","author":"Borji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1130","DOI":"10.1016\/j.imavis.2009.10.006","article-title":"Online learning of task-driven object-based visual attention control","volume":"28","author":"Borji","year":"2010","journal-title":"Image Vis. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Liu, Y., Qiao, M., Xu, M., Li, B., Hu, W., and Borji, A. (2020, January 23\u201328). Learning to predict salient faces: A novel visual-audio saliency model. Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XX 16.","DOI":"10.1007\/978-3-030-58565-5_25"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5706","DOI":"10.1109\/TIP.2015.2487833","article-title":"Salient Object Detection: A Benchmark","volume":"24","author":"Borji","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3805","DOI":"10.1109\/TIP.2020.2966082","article-title":"A Multimodal Saliency Model for Videos with High Audio-Visual Correspondence","volume":"29","author":"Min","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tsiami, A., Koutras, P., and Maragos, P. (2020, January 13\u201319). STAViS: Spatio-Temporal AudioVisual Saliency Network. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00482"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2","DOI":"10.16910\/jemr.5.4.2","article-title":"Influence of soundtrack on eye movements during video exploration","volume":"5","author":"Coutrot","year":"2012","journal-title":"J. Eye Mov. Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.16910\/jemr.6.4.1","article-title":"Different types of sounds influence gaze differently in videos","volume":"6","author":"Song","year":"2013","journal-title":"J. Eye Mov. Res."},{"key":"ref_9","unstructured":"Min, X., Zhai, G., Gao, Z., Hu, C., and Yang, X. (2014, January 18\u201320). Sound influences visual attention discriminately in videos. Proceedings of the 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), Singapore."},{"key":"ref_10","unstructured":"Tavakoli, H.R., Borji, A., Rahtu, E., and Kannala, J. (2019). DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/0010-0285(80)90005-5","article-title":"A feature-integration theory of attention","volume":"12","author":"Treisman","year":"1980","journal-title":"Cogn. Psychol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Koch, C., and Ullman, S. (1987). Chapter 1\u2014Shifts in selective visual attention: Towards the underlying neural circuitry. Matters of Intelligence: Conceptual Structures in Cognitive Neuroscience, Springer.","DOI":"10.1007\/978-94-009-3833-5_5"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/34.730558","article-title":"A model of saliency-based visual attention for rapid scene analysis","volume":"20","author":"Itti","year":"1998","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Harel, J., Koch, C., and Perona, P. (2006, January 4). Graph-Based Visual Saliency. Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS\u201906), Cambridge, MA, USA.","DOI":"10.7551\/mitpress\/7503.003.0073"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Riche, N., and Mancas, M. (2016). Bottom-Up Saliency Models for Still Images: A Practical Review. From Human Attention to Computational Attention: A Multidisciplinary Approach, Springer.","DOI":"10.1007\/978-1-4939-3435-5_9"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hou, X., and Zhang, L. (2007, January 17\u201322). Saliency Detection: A Spectral Residual Approach. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383267"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1167\/8.7.32","article-title":"SUN: A Bayesian framework for saliency using natural statistics","volume":"8","author":"Zhang","year":"2008","journal-title":"J. Vis."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2827","DOI":"10.1007\/s10639-022-11283-2","article-title":"Evaluating the Effects of Educational Multimedia Design Principles on Cognitive Load Using EEG Signal Analysis","volume":"28","author":"Farkish","year":"2023","journal-title":"Educ. Inf. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"104216","DOI":"10.1016\/j.imavis.2021.104216","article-title":"Spatiotemporal module for video saliency prediction based on self-attention","volume":"112","author":"Wang","year":"2021","journal-title":"Image Vis. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3910","DOI":"10.1109\/TIP.2014.2336549","article-title":"Video Saliency Incorporating Spatiotemporal Cues and Uncertainty Weighting","volume":"23","author":"Fang","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4185","DOI":"10.1109\/TIP.2015.2460013","article-title":"Consistent Video Saliency Using Local Gradient Flow Optimization and Global Refinement","volume":"24","author":"Wang","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2527","DOI":"10.1109\/TCSVT.2016.2595324","article-title":"Saliency Detection for Unconstrained Videos Using Superpixel-Level Graph and Spatiotemporal Propagation","volume":"27","author":"Liu","year":"2017","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1515\/phys-2018-0027","article-title":"The visual attention saliency map for movie retrospection","volume":"16","author":"Rogalska","year":"2018","journal-title":"Open Phys."},{"key":"ref_24","first-page":"163","article-title":"A Computational-Cognitive model of Visual Attention in Dynamic Environments","volume":"10","author":"Bosaghzadeh","year":"2021","journal-title":"J. Electr. Comput. Eng. Innov."},{"key":"ref_25","unstructured":"Wang, W., Shen, J., and Porikli, F. (2015, January 7\u201312). Saliency-aware geodesic video object segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1109\/TPAMI.2017.2662005","article-title":"Saliency-Aware Video Object Segmentation","volume":"40","author":"Wang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","unstructured":"Harris, D. (2014). Predicting Eyes\u2019 Fixations in Movie Videos: Visual Saliency Experiments on a New Eye-Tracking Database. Engineering Psychology and Cognitive Ergonomics, Proceedings of the 11th International Conference, EPCE 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, 22\u201327 June 2014, Springer International Publishing."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, G., Chen, C., Fan, D., Hao, A., and Qin, H. (2021, January 19\u201325). From Semantic Categories to Fixations: A Novel Weakly-supervised Visual-auditory Saliency Detection Approach. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.","DOI":"10.1109\/CVPR46437.2021.01487"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1109\/TCSVT.2022.3203421","article-title":"A Comprehensive Survey on Video Saliency Detection with Auditory Information: The Audio-visual Consistency Perceptual is the Key!","volume":"33","author":"Chen","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_30","first-page":"1953","article-title":"A Review on Different Feature Recognition Techniques for Speech Process in Automatic Speech Recognition","volume":"8","author":"Ashwini","year":"2019","journal-title":"Int. J. Sci. Technol. Res."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1943","DOI":"10.1016\/j.cub.2005.09.040","article-title":"Mechanisms for Allocating Auditory Attention: An Auditory Saliency Map","volume":"15","author":"Kayser","year":"2005","journal-title":"Curr. Biol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1109\/TMM.2012.2228476","article-title":"Multimodal Analysis for Identification and Segmentation of Moving-Sounding Objects","volume":"15","author":"Izadinia","year":"2013","journal-title":"IEEE Trans. Multimed."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2639","DOI":"10.1162\/0899766042321814","article-title":"Canonical correlation analysis: An overview with application to learning methods","volume":"16","author":"Hardoon","year":"2004","journal-title":"Neural Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/TIP.2011.2161092","article-title":"A Psychovisual Quality Metric in Free-Energy Principle","volume":"21","author":"Zhai","year":"2012","journal-title":"IEEE Trans. Image Process."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1038\/nrn2787","article-title":"The free-energy principle: A unified brain theory?","volume":"11","author":"Friston","year":"2010","journal-title":"Nat. Reviews Neurosci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1109\/LSP.2015.2413944","article-title":"Visual Saliency Detection With Free Energy Theory","volume":"22","author":"Gu","year":"2015","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1167\/14.8.5","article-title":"How saliency, faces, and sound influence gaze in dynamic social scenes","volume":"14","author":"Coutrot","year":"2014","journal-title":"J. Vis."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","article-title":"Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks","volume":"23","author":"Zhang","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2996463","article-title":"Fixation Prediction through Multimodal Analysis","volume":"13","author":"Min","year":"2016","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1109\/TCSVT.2013.2273613","article-title":"A Video Saliency Detection Model in Compressed Domain","volume":"24","author":"Fang","year":"2014","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_41","unstructured":"Guo, C., Ma, Q., and Zhang, L. (2008, January 23\u201328). Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"996","DOI":"10.1109\/TPAMI.2012.147","article-title":"Visual Saliency Based on Scale-Space Analysis in the Frequency Domain","volume":"35","author":"Li","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Zhang, J., and Sclaroff, S. (2013, January 1\u20138). Saliency Detection: A Boolean Map Approach. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Sydney, Australia.","DOI":"10.1109\/ICCV.2013.26"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Judd, T., Ehinger, K., Durand, F., and Torralba, A. (October, January 29). Learning to predict where humans look. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459462"},{"key":"ref_45","unstructured":"Platt, J., Koller, D., Singer, Y., and Roweis, S. (2007). Predicting human gaze using low-level saliency combined with face detection. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"2552","DOI":"10.1109\/TIP.2015.2425544","article-title":"Spatiotemporal Saliency Detection for Video Sequences Based on Random Walk With Restart","volume":"24","author":"Kim","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1167\/9.12.15","article-title":"Static and space-time visual saliency detection by self-resemblance","volume":"9","author":"Seo","year":"2009","journal-title":"J. Vis."},{"key":"ref_48","unstructured":"Koller, D., Schuurmans, D., Bengio, Y., and Bottou, L. (2008). Dynamic visual attention: Searching for coding length increments. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Riche, N., Duvinage, M., Mancas, M., Gosselin, B., and Dutoit, T. (2013, January 1\u20138). Saliency and Human Fixations: State-of-the-Art and Study of Comparison Metrics. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Sydney, Australia.","DOI":"10.1109\/ICCV.2013.147"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1016\/j.imavis.2013.08.004","article-title":"Selection of a best metric and evaluation of bottom-up visual saliency models","volume":"31","author":"Emami","year":"2013","journal-title":"Image Vis. Comput."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1109\/TPAMI.2018.2815601","article-title":"What Do Different Evaluation Metrics Tell Us About Saliency Models?","volume":"41","author":"Bylinskii","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/5\/120\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:27:53Z","timestamp":1760030873000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/5\/120"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,6]]},"references-count":51,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["bdcc9050120"],"URL":"https:\/\/doi.org\/10.3390\/bdcc9050120","relation":{},"ISSN":["2504-2289"],"issn-type":[{"value":"2504-2289","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,6]]}}}