{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T07:58:18Z","timestamp":1777103898597,"version":"3.51.4"},"reference-count":36,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T00:00:00Z","timestamp":1643328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2017YFC0820700-3"],"award-info":[{"award-number":["2017YFC0820700-3"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Science and Technology Innovation 2030 - major project of &quot;new generation artificial intelligence&quot;","award":["ZDI135-96"],"award-info":[{"award-number":["ZDI135-96"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61433012"],"award-info":[{"award-number":["61433012"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>VQA (Visual Question Answering) is a multi-model task. Given a picture and a question related to the image, it will determine the correct answer. The attention mechanism has become a de facto component of almost all VQA models. Most recent VQA approaches use dot-product to calculate the intra-modality and inter-modality attention between visual and language features. In this paper, the BAN (Bilinear Attention Network) method was used to calculate attention. We propose a deep multimodality bilinear attention network (DMBA-NET) framework with two basic attention units (BAN-GA and BAN-SA) to construct inter-modality and intra-modality relations. The two basic attention units are the core of the whole network framework and can be cascaded in depth. In addition, we encode the question based on the dynamic word vector of BERT(Bidirectional Encoder Representations from Transformers), then use self-attention to process the question features further. Then we sum them with the features obtained by BAN-GA and BAN-SA before the final classification. Without using the Visual Genome datasets for augmentation, the accuracy of our model reaches 70.85% on the test-std dataset of VQA 2.0.<\/jats:p>","DOI":"10.3390\/s22031045","type":"journal-article","created":{"date-parts":[[2022,1,29]],"date-time":"2022-01-29T01:43:27Z","timestamp":1643420607000},"page":"1045","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Deep Modular Bilinear Attention Network for Visual Question Answering"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8615-139X","authenticated-orcid":false,"given":"Feng","family":"Yan","sequence":"first","affiliation":[{"name":"School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wushouer","family":"Silamu","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China"},{"name":"Laboratory of Multi-Lingual Information Technology, Xinjiang University, Urumqi 830046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanbing","family":"Li","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7\u201313). VQA: Visual Question Answering. Proceedings of the International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.279"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016, January 1\u20135). Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1044"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yu, Z., Yu, J., Fan, J., and Tao, D. (2017, January 22\u201329). Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.202"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ben-younes, H., Cadene, R., Cord, M., and Thome, N. (2017, January 22\u201329). MUTAN: Multimodal Tucker Fusion for Visual Question Answering. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.285"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Xu, H., and Saenko, K. (2016). Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering. Computer Vision\u2014ECCV 2016, Springer International Publishing.","DOI":"10.1007\/978-3-319-46478-7_28"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sun, Q., and Fu, Y. (2019, January 10\u201313). Stacked Self-Attention Networks for Visual Question Answering. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada.","DOI":"10.1145\/3323873.3325044"},{"key":"ref_7","unstructured":"Chowdhury, M.I.H., Nguyen, K., Sridharan, S., and Fookes, C. (2018, January 7\u201310). Hierarchical Relational Attention for Video Question Answering. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yu, D., Fu, J., Mei, T., and Rui, Y. (2017, January 21\u201326). Multi-level Attention Networks for Visual Question Answering. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.446"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18\u201323). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00636"},{"key":"ref_10","unstructured":"Kim, J.-H., Jun, J., and Zhang, B.-T. (2018, January 3\u20138). Bilinear attention networks. Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, QC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yu, Z., Yu, J., Cui, Y., Tao, D., and Tian, Q. (2019, January 15\u201320). Deep modular co-attention networks for visual question answering. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00644"},{"key":"ref_12","unstructured":"Peng, G., Jiang, Z., You, H., Lu, P., Hoi, S., Wang, X., and Li, H. (2019, January 15\u201320). Dynamic fusion with intra-and inter-modality attention flow for visual question answering. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5411","DOI":"10.1007\/s00500-020-05539-7","article-title":"Cross-modality co-attention networks for visual question answering","volume":"25","author":"Han","year":"2021","journal-title":"Soft Comput."},{"key":"ref_14","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wu, Q., Wang, P., Shen, C., Dick, A., and Hengel, A.V.D. (2016, January 27\u201330). Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.500"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, P., Wu, Q., Shen, C., Dick, A., and van den Hengel, A. (2017, January 19\u201325). Explicit Knowledge-based Reasoning for Visual Question Answering. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/179"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2413","DOI":"10.1109\/TPAMI.2017.2754246","article-title":"FVQA: Fact-Based Visual Question Answering","volume":"40","author":"Wang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1367","DOI":"10.1109\/TPAMI.2017.2708709","article-title":"Image Captioning and Visual Question Answering Based on Attributes and External Knowledge","volume":"40","author":"Wu","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","unstructured":"Jia, D., Wei, D., Socher, R., Li-Jia, L., Kai, L., and Li, F. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","article-title":"Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations","volume":"123","author":"Krishna","year":"2017","journal-title":"Int. J. Comput. Vis."},{"key":"ref_21","unstructured":"Lu, J., Batra, D., Parikh, D., and Lee, S. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. arXiv."},{"key":"ref_22","unstructured":"Li, L.H., Yatskar, M., Yin, D., Hsieh, C., and Chang, K. (2019). VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tan, H., and Bansal, M. (2019). LXMERT: Learning Cross-Modality Encoder Representations from Transformers. arXiv.","DOI":"10.18653\/v1\/D19-1514"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., and Gao, J. (2020, January 7\u201312). Unified Vision-Language Pre-Training for Image Captioning and VQA. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.7005"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"5947","DOI":"10.1109\/TNNLS.2018.2817340","article-title":"Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering","volume":"29","author":"Yu","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_26","unstructured":"Kim, J., On, K., Kim, J., Ha, J., and Zhang, B. (2016). Hadamard Product for Low-rank Bilinear Pooling. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1017\/S1351324916000334","article-title":"Word2Vec","volume":"23","author":"Church","year":"2017","journal-title":"Nat. Lang. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.","DOI":"10.18653\/v1\/N18-1202"},{"key":"ref_31","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_32","unstructured":"Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv."},{"key":"ref_33","unstructured":"Jiang, Y., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., and Parikh, D. (2018). Pythia v0. 1: The winning entry to the vqa challenge 2018. arXiv."},{"key":"ref_34","unstructured":"Zhang, Y., Hare, J., and Prugel-Bennett, A. (2018). Learning to count objects in natural images for visual question answering. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Cadene, R., Ben-Younes, H., Cord, M., and Thome, N. (2019, January 16\u201317). Murel: Multimodal relational reasoning for visual question answering. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00209"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2020.3004830","article-title":"Mra-net: Improving vqa via multi-modal relation attention network","volume":"44","author":"Peng","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/1045\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:10:23Z","timestamp":1760134223000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/1045"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,28]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22031045"],"URL":"https:\/\/doi.org\/10.3390\/s22031045","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,28]]}}}