{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T14:17:30Z","timestamp":1771942650887,"version":"3.50.1"},"reference-count":55,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T00:00:00Z","timestamp":1771200000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Audio\u2013visual learning seeks to achieve holistic scene understanding by integrating auditory and visual cues. Early research focused on fully fine-tuning pre-trained models, incurring high computational costs. Consequently, recent studies have adopted parameter-efficient tuning methods to adapt large-scale vision models to the audio\u2013visual domain. Despite the competitive performance of existing methods, several challenges persist. Firstly, effectively leveraging the complementary semantics between the audio and visual modalities remains difficult, as these two modalities capture fundamentally different aspects of a video. Secondly, comprehending dynamic video context is challenging because both spatial attributes (such as scale) and temporal characteristics (such as motion) of objects co-evolve over time, making semantic comprehension more complex. To address these challenges, we propose a novel framework, named Joint Spatial\u2013Temporal Modeling with Semantic Completion (JMSC). JMSC introduces cross-modal latent reconstruction, which moves beyond shallow correlation by encouraging the model to reconstruct one modality\u2019s complete semantic summary from a masked version of its counterpart. Furthermore, JMSC learns a unified representation of video spatial attributes and temporal changes by jointly modeling them under audio guidance, enabling accurate localization and consistent tracking in dynamic video scenes. Experimental results demonstrate that JMSC achieves state-of-the-art performance across multiple downstream tasks while maintaining high computational efficiency.<\/jats:p>","DOI":"10.3390\/s26041288","type":"journal-article","created":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T10:14:32Z","timestamp":1771236872000},"page":"1288","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["JMSC: Joint Spatial\u2013Temporal Modeling with Semantic Completion for Audio\u2013Visual Learning"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6217-6268","authenticated-orcid":false,"given":"Xinfu","family":"Xu","sequence":"first","affiliation":[{"name":"Faculty of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Yang","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4372-1767","authenticated-orcid":false,"given":"Zhibin","family":"Yu","sequence":"additional","affiliation":[{"name":"Faculty of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"},{"name":"Key Laboratory of Ocean Observation and Information of Hainan Province, Sanya 572000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Hao, D., Mao, Y., He, B., Han, X., Dai, Y., and Zhong, Y. (2024, January 20\u201327). Improving audio-visual segmentation with bidirectional generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i3.27978"},{"key":"ref_2","unstructured":"Gao, S., Chen, Z., Chen, G., Wang, W., and Lu, T. (2024, January 20\u201327). Avsegformer: Audio-visual segmentation with transformer. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cheng, H., Liu, Z., Zhou, H., Qian, C., Wu, W., and Wang, L. (2022). Joint-modal label denoising for weakly-supervised audio-visual video parsing. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23\u201327 October 2022, Springer.","DOI":"10.1007\/978-3-031-19830-4_25"},{"key":"ref_4","first-page":"34722","article-title":"Multi-modal grouping network for weakly-supervised audio-visual video parsing","volume":"35","author":"Mo","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_5","first-page":"9758","article-title":"Self-supervised learning by cross-modal audio-video clustering","volume":"33","author":"Alwassel","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhang, S., Ni, D., Wei, Z., Yang, K., Jin, S., Huang, G., Liang, Z., Zhang, L., and Li, L. (2024). Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data. Sensors, 24.","DOI":"10.3390\/s24123714"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Das, A., Sarma, M.S., Hoque, M.M., Siddique, N., and Dewan, M.A.A. (2024). AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors, 24.","DOI":"10.3390\/s24185862"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chae, H., Lee, J., Kim, J., Lee, S., Lee, J., Chung, Y., and Park, D. (2024). Novel Method for Detecting Coughing Pigs with Audio-Visual Multimodality for Smart Agriculture Monitoring. Sensors, 24.","DOI":"10.3390\/s24227232"},{"key":"ref_9","unstructured":"Li, G., Hou, W., and Hu, D. (November, January 29). Progressive spatio-temporal perception for audio-visual question answering. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Schwartz, I., Schwing, A.G., and Hazan, T. (2019, January 15\u201320). A simple baseline for audio-visual scene-aware dialog. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01283"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Jeon, S., and Kim, M.S. (2022). Noise-Robust Multimodal Audio-Visual Speech Recognition System for Speech-Based Interaction Applications. Sensors, 22.","DOI":"10.3390\/s22207738"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"He, Y., Seng, K.P., and Ang, L.M. (2023). Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild. Sensors, 23.","DOI":"10.3390\/s23041834"},{"key":"ref_13","first-page":"56075","article-title":"Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks","volume":"36","author":"Duan","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","unstructured":"Zhao, Y., Xi, W., Cui, Y., Bai, G., Liu, X., and Zhao, J. (November, January 28). Copl: Parameter-efficient collaborative prompt learning for audio-visual tasks. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia."},{"key":"ref_15","unstructured":"Gong, Y., Rouditchenko, A., Liu, A.H., Harwath, D., Karlinsky, L., Kuehne, H., and Glass, J.R. (2023, January 1\u20135). Contrastive Audio-Visual Masked Autoencoder. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda."},{"key":"ref_16","unstructured":"Zhao, X., Wang, Y., and Jin, P. (March, January 25). Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, K., Tian, Y., and Hatzinakos, D. (2024, January 17\u201321). Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vision Transformers with Cross-Modal Adaptation. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW63382.2024.00190"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","article-title":"Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization","volume":"128","author":"Selvaraju","year":"2019","journal-title":"Int. J. Comput. Vis."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mahmud, T., and Marculescu, D. (2023, January 3\u20137). Ave-clip: Audioclip-based multi-window temporal transformer for audio visual event localization. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV56688.2023.00513"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhou, J., Zheng, L., Zhong, Y., Hao, S., and Wang, M. (2021, January 20\u201325). Positive sample propagation along the audio-visual event line. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00833"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yu, J., Cheng, Y., Zhao, R.W., Feng, R., and Zhang, Y. (2022, January 10\u201314). Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Portugal.","DOI":"10.1145\/3503161.3547869"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., and Zhong, Y. (2022). Audio\u2013visual segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23\u201327 October 2022, Springer.","DOI":"10.1007\/978-3-031-19836-6_22"},{"key":"ref_23","unstructured":"Sun, P., Zhang, H., and Hu, D. (November, January 28). Unveiling and Mitigating Bias in Audio Visual Segmentation. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia."},{"key":"ref_24","unstructured":"Ma, J., Sun, P., Wang, Y., and Hu, D. (October, January 29). Stepping stones: A progressive training strategy for audio-visual semantic segmentation. Proceedings of the European Conference on Computer Vision, Milan, Italy."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.R., and Hu, D. (2022, January 18\u201324). Learning to answer questions in dynamic audio-visual scenarios. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01852"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Yun, H., Yu, Y., Yang, W., Lee, K., and Kim, G. (2021, January 11\u201317). Pano-avqa: Grounded audio-visual question answering on 360deg videos. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00204"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, H., Zeng, R., Wu, Q., Tan, M., and Gan, C. (2020, January 12\u201316). Cross-Modal Relation-Aware Networks for Audio-Visual Event Localization. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA. MM \u201920.","DOI":"10.1145\/3394171.3413581"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xia, Y., and Zhao, Z. (2022, January 18\u201324). Cross-modal background suppression for audio-visual event localization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01936"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"7239","DOI":"10.1109\/TPAMI.2022.3223688","article-title":"Contrastive positive sample propagation along the audio-visual event line","volume":"45","author":"Zhou","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, H. (2022). Tip-adapter: Training-free adaption of clip for few-shot classification. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23\u201327 October 2022, Springer.","DOI":"10.1007\/978-3-031-19833-5_29"},{"key":"ref_31","unstructured":"Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 9\u201315). Parameter-efficient transfer learning for NLP. Proceedings of the International Conference on Machine Learning. PMLR, Long Beach, CA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Liu, M., Li, B., and Yu, Y. (2024). OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning. arXiv.","DOI":"10.3233\/FAIA240499"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Lester, B., Al-Rfou, R., and Constant, N. (2021, January 7\u201311). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"ref_34","first-page":"3","article-title":"Lora: Low-rank adaptation of large language models","volume":"1","author":"Hu","year":"2022","journal-title":"Int. Conf. Learn. Represent."},{"key":"ref_35","first-page":"10088","article-title":"Qlora: Efficient finetuning of quantized llms","volume":"36","author":"Dettmers","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P.H., Zhang, X.P., and Tang, Y. (2023, January 2\u20136). Tem-adapter: Adapting image-text pretraining for video question answer. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.01282"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Li, Y., Quan, R., Zhu, L., and Yang, Y. (2023, January 18\u201322). Efficient Multimodal Fusion via Interactive Prompting. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00256"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Lin, Y.B., Sung, Y.L., Lei, J., Bansal, M., and Bertasius, G. (2023, January 18\u201322). Vision Transformers Are Parameter-Efficient Audio-Visual Learners. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00228"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., and Dong, L. (2022, January 18\u201324). Swin transformer v2: Scaling up capacity and resolution. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01170"},{"key":"ref_40","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"He, K., Chen, X., Xie, S., Li, Y., Doll\u00e1r, P., and Girshick, R. (2022, January 18\u201324). Masked autoencoders are scalable vision learners. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01553"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Tian, Y., Shi, J., Li, B., Duan, Z., and Xu, C. (2018, January 8\u201314). Audio-visual event localization in unconstrained videos. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01216-8_16"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Gao, R., and Grauman, K. (2021, January 20\u201325). Visualvoice: Audio-visual speech separation with cross-modal consistency. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01524"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Cubuk, E.D., Zoph, B., Shlens, J., and Le, Q.V. (2020, January 14\u201319). Randaugment: Practical automated data augmentation with a reduced search space. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00359"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"4617","DOI":"10.1109\/TMM.2023.3324498","article-title":"Leveraging the Video-Level Semantic Consistency of Event for Audio-Visual Event Localization","volume":"26","author":"Jiang","year":"2024","journal-title":"IEEE Trans. Multimed."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"2650","DOI":"10.1109\/TMM.2025.3535359","article-title":"Listen With Seeing: Cross-Modal Contrastive Learning for Audio-Visual Event Localization","volume":"27","author":"Sun","year":"2025","journal-title":"IEEE Trans. Multimed."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"14393","DOI":"10.1109\/JSEN.2025.3545487","article-title":"Cross-Modal Feature Enhancement Networks for Audio-Visual Event Localization","volume":"25","author":"Xu","year":"2025","journal-title":"IEEE Sens. J."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Cheng, Y., Li, Y., He, J., and Feng, R. (2024, January 10\u201315). Mixtures of experts for audio-visual learning. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.","DOI":"10.52202\/079017-0007"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"112925","DOI":"10.1016\/j.knosys.2024.112925","article-title":"Multi-Relation Learning Network for audio-visual event localization","volume":"310","author":"Zhang","year":"2025","journal-title":"Knowl.-Based Syst."},{"key":"ref_50","first-page":"15448","article-title":"Learning generative vision transformer with energy-based latent space for saliency prediction","volume":"34","author":"Zhang","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_51","unstructured":"Li, K., Yang, Z., Chen, L., Yang, Y., and Xiao, J. (November, January 29). Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"4108","DOI":"10.1109\/TIP.2025.3580269","article-title":"Contrastive Conditional Latent Diffusion for Audio-Visual Segmentation","volume":"34","author":"Mao","year":"2025","journal-title":"IEEE Trans. Image Process."},{"key":"ref_53","unstructured":"Bhosale, S., Yang, H., Kanojia, D., Deng, J., and Zhu, X. (March, January 25). Unsupervised Audio-Visual Segmentation with Modality Alignment. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Kim, H., Jung, I., Suh, D., Zhang, Y., Lee, S., and Hong, S. (2025, January 11\u201315). Question-Aware Gaussian Experts for Audio-Visual Question Answering. Proceedings of the Computer Vision and Pattern Recognition Conference, CVPR, Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.01277"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14\u201319). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00975"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/4\/1288\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T13:55:24Z","timestamp":1771941324000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/4\/1288"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,16]]},"references-count":55,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,2]]}},"alternative-id":["s26041288"],"URL":"https:\/\/doi.org\/10.3390\/s26041288","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,16]]}}}