{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T17:19:12Z","timestamp":1790702352839,"version":"4.1.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T00:00:00Z","timestamp":1646352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100007129","name":"Shandong Provincial Natural Science Foundation","doi-asserted-by":"crossref","award":["ZR2019JQ23"],"award-info":[{"award-number":["ZR2019JQ23"]}],"id":[{"id":"10.13039\/501100007129","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2022,11,30]]},"abstract":"<jats:p>Visual attention in Visual Question Answering (VQA) targets at locating the right image regions regarding the answer prediction, offering a powerful technique to promote multi-modal understanding. However, recent studies have pointed out that the highlighted image regions from the visual attention are often irrelevant to the given question and answer, leading to model confusion for correct visual reasoning. To tackle this problem, existing methods mostly resort to aligning the visual attention weights with human attentions. Nevertheless, gathering such human data is laborious and expensive, making it burdensome to adapt well-developed models across datasets. To address this issue, in this article, we devise a novel visual attention regularization approach, namely, AttReg, for better visual grounding in VQA. Specifically, AttReg first identifies the image regions that are essential for question answering yet unexpectedly ignored (i.e., assigned with low attention weights) by the backbone model. And then a mask-guided learning scheme is leveraged to regularize the visual attention to focus more on these ignored key regions. The proposed method is very flexible and model-agnostic, which can be integrated into most visual attention-based VQA models and require no human attention supervision. Extensive experiments over three benchmark datasets, i.e., VQA-CP v2, VQA-CP v1, and VQA v2, have been conducted to evaluate the effectiveness of AttReg. As a by-product, when incorporating AttReg into the strong baseline LMH, our approach can achieve a new state-of-the-art accuracy of 60.00% with an absolute performance gain of 7.01% on the VQA-CP v2 benchmark dataset. In addition to the effectiveness validation, we recognize that the faithfulness of the visual attention in VQA has not been well explored in literature. In the light of this, we propose to empirically validate such property of visual attention and compare it with the prevalent gradient-based approaches.<\/jats:p>","DOI":"10.1145\/3498340","type":"journal-article","created":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T10:31:58Z","timestamp":1646389918000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Answer Questions with Right Image Regions: A Visual Attention Regularization Approach"],"prefix":"10.1145","volume":"18","author":[{"given":"Yibing","family":"Liu","sequence":"first","affiliation":[{"name":"Shandong University, Jimo, Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yangyang","family":"Guo","sequence":"additional","affiliation":[{"name":"Shandong University, Jimo, Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianhua","family":"Yin","sequence":"additional","affiliation":[{"name":"Shandong University, Jimo, Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuemeng","family":"Song","sequence":"additional","affiliation":[{"name":"Shandong University, Jimo, Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weifeng","family":"Liu","sequence":"additional","affiliation":[{"name":"China University of Petroleum (East China), Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liqiang","family":"Nie","sequence":"additional","affiliation":[{"name":"Shandong University, Jimo, Qingdao, Shandong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min","family":"Zhang","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology (Shenzhen), Shenzhen, Guangdong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,4]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00522"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00636"},{"key":"e_1_3_2_4_2","first-page":"39","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Andreas Jacob","year":"2016","unstructured":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Deep compositional question answering with neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 39\u201348."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.279"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00209"},{"key":"e_1_3_2_7_2","first-page":"839","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Cadene Remi","year":"2019","unstructured":"Remi Cadene, Corentin Dancette, Hedi Ben-Younes, Matthieu Cord, and Devi Parikh. 2019. RUBi: Reducing unimodal biases in visual question answering. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 839\u2013850."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01081"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1418"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1092"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3390891"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3282469"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1044"},{"key":"e_1_3_2_15_2","first-page":"3197","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Gat Itai","year":"2020","unstructured":"Itai Gat, Idan Schwartz, Alexander G. Schwing, and Tamir Hazan. 2020. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 3197\u20133208."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.670"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331186"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/98"},{"key":"e_1_3_2_19_2","article-title":"Loss-rescaling VQA: Revisiting language prior problem from a class-imbalance view","author":"Guo Yangyang","year":"2020","unstructured":"Yangyang Guo, Liqiang Nie, Zhiyong Cheng, and Qi Tian. 2020. Loss-rescaling VQA: Revisiting language prior problem from a class-imbalance view. arXiv preprint arXiv:2010.16010 (2020).","journal-title":"arXiv preprint arXiv:2010.16010"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.1212303"},{"key":"e_1_3_2_21_2","first-page":"3543","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Jain Sarthak","year":"2019","unstructured":"Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL, 3543\u20133556."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6776"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2018.2862341"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2017.2759508"},{"issue":"49","key":"e_1_3_2_25_2","first-page":"1","article-title":"Semantic concept network and deep walk-based visual question answering","volume":"15","author":"Li Qun","year":"2019","unstructured":"Qun Li, Fu Xiao, Le An, Xianzhong Long, and Xiaochuan Sun. 2019. Semantic concept network and deep walk-based visual question answering. ACM Trans. Multim. Comput. Commun. Appl. 15, 49 (2019), 1\u201319.","journal-title":"ACM Trans. Multim. Comput. Commun. Appl."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2896494"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2890628"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.265"},{"key":"e_1_3_2_29_2","first-page":"1175","volume-title":"ACM Multimedia","author":"Liu Fei","year":"2019","unstructured":"Fei Liu, Jing Liu, Richang Hong, and Hanqing Lu. 2019. Erasing-based attention learning for visual question answering. In ACM Multimedia. ACM, 1175\u20131183."},{"key":"e_1_3_2_30_2","first-page":"289","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Lu Jiasen","year":"2016","unstructured":"Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.2016. Hierarchical question-image co-attention for visual question answering. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 289\u2013297."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01231-1_1"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6858"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12272"},{"key":"e_1_3_2_35_2","first-page":"1548","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Ramakrishnan Sainandan","year":"2018","unstructured":"Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 1548\u20131558."},{"key":"e_1_3_2_36_2","first-page":"91","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 91\u201399."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/371"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00268"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.499"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.727"},{"key":"e_1_3_2_42_2","first-page":"407","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Teney Damien","year":"2020","unstructured":"Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton van den Hengel. 2020. On the value of out-of-distribution testing: An example of Goodhart\u2019s law. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 407\u2013417."},{"key":"e_1_3_2_43_2","first-page":"8601","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Wu Jialin","year":"2019","unstructured":"Jialin Wu and Raymond J. Mooney. 2019. Self-critical reasoning for robust visual question answering. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 8601\u20138611."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.05.001"},{"key":"e_1_3_2_45_2","first-page":"1645","volume-title":"ACM Multim","author":"Xu Dejing","year":"2017","unstructured":"Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multim. ACM, 1645\u20131653."},{"key":"e_1_3_2_46_2","first-page":"632","volume-title":"ACM Multim","author":"Yang Hui","year":"2003","unstructured":"Hui Yang, Lekha Chaisorn, Yunlong Zhao, Shi-Yong Neo, and Tat-Seng Chua. 2003. VideoQA: Question answering on news video. In ACM Multim. ACM, 632\u2013641."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.10"},{"issue":"51","key":"e_1_3_2_48_2","first-page":"1","article-title":"Multi-source multi-level attention networks for visual question answering","volume":"15","author":"Yu Dongfei","year":"2019","unstructured":"Dongfei Yu, Jianlong Fu, Xinmei Tian, and Tao Mei. 2019. Multi-source multi-level attention networks for visual question answering. ACM Trans. Multim. Comput. Commun. Appl. 15, 51 (2019), 1\u201320.","journal-title":"ACM Trans. Multim. Comput. Commun. Appl."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.542"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2019.00043"},{"key":"e_1_3_2_51_2","first-page":"1050","volume-title":"ACM Multim","author":"Zhao Zhou","year":"2017","unstructured":"Zhou Zhao, Jinghao Lin, Xinghua Jiang, Deng Cai, Xiaofei He, and Yueting Zhuang. 2017. Video question answering via hierarchical dual-level attention network learning. In ACM Multim. ACM, 1050\u20131058."},{"key":"e_1_3_2_52_2","first-page":"305","volume-title":"ACM Multim","author":"Zhou Luowei","year":"2017","unstructured":"Luowei Zhou, Chenliang Xu, Parker A. Koch, and Jason J. Corso. 2017. Watch what you just said: Image captioning with text-conditional attention. In ACM Multim. ACM, 305\u2013313."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/151"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.540"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3498340","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3498340","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:05Z","timestamp":1750188605000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3498340"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,4]]},"references-count":53,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,11,30]]}},"alternative-id":["10.1145\/3498340"],"URL":"https:\/\/doi.org\/10.1145\/3498340","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,4]]},"assertion":[{"value":"2021-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}