{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T12:01:46Z","timestamp":1788955306506,"version":"build-2803163510"},"reference-count":156,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,3,29]],"date-time":"2024-03-29T00:00:00Z","timestamp":1711670400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,29]],"date-time":"2024-03-29T00:00:00Z","timestamp":1711670400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Vis Comput"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>2D image understanding is a complex problem within computer vision, but it holds the key to providing human-level scene comprehension. It goes further than identifying the objects in an image, and instead, it attempts to <jats:italic>understand<\/jats:italic> the scene. Solutions to this problem form the underpinning of a range of tasks, including image captioning, visual question answering (VQA), and image retrieval. Graphs provide a natural way to represent the relational arrangement between objects in an image, and thus, in recent years graph neural networks (GNNs) have become a standard component of many 2D image understanding pipelines, becoming a core architectural component, especially in the VQA group of tasks. In this survey, we review this rapidly evolving field and we provide a taxonomy of graph types used in 2D image understanding approaches, a comprehensive list of the GNN models used in this domain, and a roadmap of future potential developments. To the best of our knowledge, this is the first comprehensive survey that covers image captioning, visual question answering, and image retrieval techniques that focus on using GNNs as the main part of their architecture.\n<\/jats:p>","DOI":"10.1007\/s00371-024-03343-0","type":"journal-article","created":{"date-parts":[[2024,3,29]],"date-time":"2024-03-29T13:01:37Z","timestamp":1711717297000},"page":"491-516","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Graph neural networks in vision-language image understanding: a survey"],"prefix":"10.1007","volume":"41","author":[{"given":"Henry","family":"Senior","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Slabaugh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shanxin","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luca","family":"Rossi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,3,29]]},"reference":[{"key":"3343_CR1","unstructured":"Chamberlain, B.P., Shirobokov, S., Rossi, E., Frasca, F., Markovich, T., Hammerla, N., Bronstein, M.M., Hansmire, M.: Graph neural networks for link prediction with subgraph sketching. arXiv preprint (2022)"},{"key":"3343_CR2","unstructured":"Barbero, F., Bodnar, C., Oc\u00e1riz\u00a0Borde, H.S., Bronstein, M., Veli\u010dkovi\u0107, P., Li\u00f2, P.: Sheaf neural networks with connection Laplacians. In: Topological, Algebraic and Geometric Learning Workshops, pp. 28\u201336. PMLR (2022)"},{"key":"3343_CR3","unstructured":"Frasca, F., Bevilacqua, B., Bronstein, M.M., Maron, H.: Understanding and extending subgraph gnns by rethinking their symmetries. arXiv preprint (2022)"},{"key":"3343_CR4","doi-asserted-by":"crossref","first-page":"2713","DOI":"10.1007\/s00371-021-02245-9","volume":"37","author":"M Shopon","year":"2021","unstructured":"Shopon, M., Bari, A.H., Gavrilova, M.L.: Residual connection-based graph convolutional neural networks for gait recognition. Vis. Comput. 37, 2713\u20132724 (2021)","journal-title":"Vis. Comput."},{"issue":"7","key":"3343_CR5","doi-asserted-by":"crossref","first-page":"2881","DOI":"10.1007\/s00371-022-02499-x","volume":"39","author":"Z-Y Liu","year":"2023","unstructured":"Liu, Z.-Y., Liu, J.-W.: Hypergraph attentional convolutional neural network for salient object detection. Vis. Comput. 39(7), 2881\u20132907 (2023)","journal-title":"Vis. Comput."},{"key":"3343_CR6","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1007\/s00371-019-01644-3","volume":"36","author":"Y Qin","year":"2020","unstructured":"Qin, Y., Mo, L., Li, C., Luo, J.: Skeleton-based action recognition by part-aware graph convolutional networks. Vis. Comput. 36, 621\u2013631 (2020)","journal-title":"Vis. Comput."},{"key":"3343_CR7","first-page":"110210","volume":"8","author":"A Bicciato","year":"2023","unstructured":"Bicciato, A., Cosmo, L., Minello, G., Rossi, L., Torsello, A.: Gnn-lofi: a novel graph neural network through localized feature-based histogram intersection. Pattern Recogn. 8, 110210 (2023)","journal-title":"Pattern Recogn."},{"issue":"9","key":"3343_CR8","doi-asserted-by":"crossref","first-page":"2579","DOI":"10.1093\/bioinformatics\/btac112","volume":"38","author":"Z Wang","year":"2022","unstructured":"Wang, Z., Liu, M., Luo, Y., Xu, Z., Xie, Y., Wang, L., Cai, L., Qi, Q., Yuan, Z., Yang, T., et al.: Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics 38(9), 2579\u20132586 (2022)","journal-title":"Bioinformatics"},{"issue":"42","key":"3343_CR9","doi-asserted-by":"crossref","first-page":"0158","DOI":"10.1126\/sciadv.abf0158","volume":"8","author":"SJ Clipman","year":"2022","unstructured":"Clipman, S.J., Mehta, S.H., Mohapatra, S., Srikrishnan, A.K., Zook, K.J., Duggal, P., Saravanan, S., Nandagopal, P., Kumar, M.S., Lucas, G.M., et al.: Deep learning and social network analysis elucidate drivers of hiv transmission in a high-incidence cohort of people who inject drugs. Sci. Adv. 8(42), 0158 (2022)","journal-title":"Sci. Adv."},{"key":"3343_CR10","doi-asserted-by":"crossref","unstructured":"Shi, W., Rajkumar, R.: Point-gnn: graph neural network for 3d object detection in a point cloud. In: CVPR, pp. 1711\u20131719 (2020)","DOI":"10.1109\/CVPR42600.2020.00178"},{"key":"3343_CR11","doi-asserted-by":"crossref","unstructured":"Cosmo, L., Minello, G., Bronstein, M., Rodol\u00e0, E., Rossi, L., Torsello, A.: 3d shape analysis through a quantum lens: the average mixing kernel signature, pp. 1\u201320. IJCV (2022)","DOI":"10.1007\/s11263-022-01610-y"},{"key":"3343_CR12","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.aiopen.2021.01.001","volume":"1","author":"J Zhou","year":"2020","unstructured":"Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., Sun, M.: Graph neural networks: a review of methods and applications. AI Open 1, 57\u201381 (2020)","journal-title":"AI Open"},{"issue":"1","key":"3343_CR13","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1093\/bib\/bbab340","volume":"23","author":"H-C Yi","year":"2022","unstructured":"Yi, H.-C., You, Z.-H., Huang, D.-S., Kwoh, C.K.: Graph representation learning in bioinformatics: trends, methods and applications. Brief. Bioinform. 23(1), 340 (2022)","journal-title":"Brief. Bioinform."},{"key":"3343_CR14","doi-asserted-by":"crossref","unstructured":"Thomas, J.J., Tran, T.H.N., Lechuga, G.P., Belaton, B.: Convolutional graph neural networks: a review and applications of graph autoencoder in chemoinformatics. In: Deep Learning Techniques and Optimization Strategies in Big Data Analytics, pp. 107\u2013123 (2020)","DOI":"10.4018\/978-1-7998-1192-3.ch007"},{"key":"3343_CR15","unstructured":"Chen, C., Wu, Y., Dai, Q., Zhou, H.-Y., Xu, M., Yang, S., Han, X., Yu, Y.: A survey on graph neural networks and graph transformers in computer vision: a task-oriented perspective. arXiv preprint (2022)"},{"key":"3343_CR16","first-page":"5834","volume":"34","author":"S Chen","year":"2021","unstructured":"Chen, S., Guhur, P.-L., Schmid, C., Laptev, I.: History aware multimodal transformer for vision-and-language navigation. Adv. Neural. Inf. Process. Syst. 34, 5834\u20135847 (2021)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"3343_CR17","doi-asserted-by":"crossref","unstructured":"Wang, H., Wang, W., Liang, W., Xiong, C., Shen, J.: Structured scene memory for vision-language navigation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 8455\u20138464 (2021)","DOI":"10.1109\/CVPR46437.2021.00835"},{"key":"3343_CR18","doi-asserted-by":"crossref","unstructured":"An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., Wang, L.: Etpnav: evolving topological planning for vision-language navigation in continuous environments (2023). arXiv preprint arXiv:2304.03047","DOI":"10.1109\/TPAMI.2024.3386695"},{"key":"3343_CR19","first-page":"20660","volume":"33","author":"Z Deng","year":"2020","unstructured":"Deng, Z., Narasimhan, K., Russakovsky, O.: Evolving graphical planner: contextual global planning for vision-and-language navigation. Adv. Neural. Inf. Process. Syst. 33, 20660\u201320672 (2020)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"3343_CR20","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Chen, J., Gao, C., Wang, W., Yang, L., Ren, H., Xia, H., Liu, S.: Target-driven structured transformer planner for vision-language navigation. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 4194\u20134203 (2022)","DOI":"10.1145\/3503161.3548281"},{"key":"3343_CR21","doi-asserted-by":"crossref","unstructured":"Zhu, F., Liang, X., Zhu, Y., Yu, Q., Chang, X., Liang, X.: Soon: scenario oriented object navigation with graph-based exploration. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 12689\u201312699 (2021)","DOI":"10.1109\/CVPR46437.2021.01250"},{"key":"3343_CR22","first-page":"1","volume":"8","author":"W Wu","year":"2023","unstructured":"Wu, W., Chang, T., Li, X., Yin, Q., Hu, Y.: Vision-language navigation: a survey and taxonomy. Neural Comput. Appl. 8, 1\u201326 (2023)","journal-title":"Neural Comput. Appl."},{"key":"3343_CR23","doi-asserted-by":"crossref","unstructured":"Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., Chang, X.: Dynamic graph enhanced contrastive learning for chest x-ray report generation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 3334\u20133343 (2023)","DOI":"10.1109\/CVPR52729.2023.00325"},{"key":"3343_CR24","doi-asserted-by":"crossref","unstructured":"Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., Chang, X.: Cross-modal clinical graph transformer for ophthalmic report generation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 20656\u201320665 (2022)","DOI":"10.1109\/CVPR52688.2022.02000"},{"key":"3343_CR25","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D.: When radiology report generation meets knowledge graph. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12910\u201312917 (2020)","DOI":"10.1609\/aaai.v34i07.6989"},{"key":"3343_CR26","first-page":"1","volume":"6","author":"AA Yusuf","year":"2023","unstructured":"Yusuf, A.A., Feng, C., Mao, X., Ally Duma, R., Abood, M.S., Chukkol, A.H.A.: Graph neural networks for visual question answering: a systematic review. Multimed. Tools Appl. 6, 1\u201338 (2023)","journal-title":"Multimed. Tools Appl."},{"key":"3343_CR27","unstructured":"Tang, Z., Sun, Z.-H., Wu, E.Q., Wei, C.-F., Ming, D., Chen, S.: MRCG: a MRI retrieval system with convolutional and graph neural networks for secure and private IOMT. IEEE J. Biomed. Health Inf. (2021)"},{"key":"3343_CR28","doi-asserted-by":"crossref","unstructured":"Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: Every picture tells a story: generating sentences from images. In: ECCV, pp. 15\u201329. Springer (2010)","DOI":"10.1007\/978-3-642-15561-1_2"},{"key":"3343_CR29","doi-asserted-by":"crossref","unstructured":"Laina, I., Rupprecht, C., Navab, N.: Towards unsupervised image captioning with shared multimodal embeddings. In: ICCV, pp. 7414\u20137424 (2019)","DOI":"10.1109\/ICCV.2019.00751"},{"key":"3343_CR30","doi-asserted-by":"crossref","unstructured":"Li, G., Zhu, L., Liu, P., Yang, Y.: Entangled transformer for image captioning. In: ICCV, pp. 8928\u20138937 (2019)","DOI":"10.1109\/ICCV.2019.00902"},{"key":"3343_CR31","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.1007\/s00371-020-01867-9","volume":"37","author":"G Barlas","year":"2021","unstructured":"Barlas, G., Veinidis, C., Arampatzis, A.: What we see in a photograph: content selection for image captioning. Vis. Comput. 37, 1309\u20131326 (2021)","journal-title":"Vis. Comput."},{"key":"3343_CR32","first-page":"1","volume":"6","author":"K Bayoudh","year":"2021","unstructured":"Bayoudh, K., Knani, R., Hamdaoui, F., Mtibaa, A.: A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. Vis. Comput. 6, 1\u201332 (2021)","journal-title":"Vis. Comput."},{"issue":"3","key":"3343_CR33","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1007\/s00371-018-1566-y","volume":"35","author":"X Liu","year":"2019","unstructured":"Liu, X., Xu, Q., Wang, N.: A survey on deep neural network-based image captioning. Vis. Comput. 35(3), 445\u2013470 (2019)","journal-title":"Vis. Comput."},{"key":"3343_CR34","first-page":"1","volume":"6","author":"J Zhong","year":"2022","unstructured":"Zhong, J., Cao, Y., Zhu, Y., Gong, J., Chen, Q.: Multi-channel weighted fusion for image captioning. Vis. Comput. 6, 1\u201318 (2022)","journal-title":"Vis. Comput."},{"key":"3343_CR35","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s00371-019-01786-4","volume":"37","author":"S Hashemi Hosseinabad","year":"2021","unstructured":"Hashemi Hosseinabad, S., Safayani, M., Mirzaei, A.: Multiple answers to a question: a new approach for visual question answering. Vis. Comput. 37, 119\u2013131 (2021)","journal-title":"Vis. Comput."},{"key":"3343_CR36","first-page":"1","volume":"6","author":"Z Guo","year":"2022","unstructured":"Guo, Z., Han, D.: Multi-modal co-attention relation networks for visual question answering. Vis. Comput. 6, 1\u201313 (2022)","journal-title":"Vis. Comput."},{"issue":"9","key":"3343_CR37","doi-asserted-by":"crossref","first-page":"1847","DOI":"10.1007\/s00371-019-01773-9","volume":"36","author":"J Pradhan","year":"2020","unstructured":"Pradhan, J., Ajad, A., Pal, A.K., Banka, H.: Multi-level colored directional motif histograms for content-based image retrieval. Vis. Comput. 36(9), 1847\u20131868 (2020)","journal-title":"Vis. Comput."},{"key":"3343_CR38","doi-asserted-by":"crossref","unstructured":"Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D., Bernstein, M., Fei-Fei, L.: Image retrieval using scene graphs. In: CVPR, pp. 3668\u20133678 (2015)","DOI":"10.1109\/CVPR.2015.7298990"},{"key":"3343_CR39","doi-asserted-by":"crossref","unstructured":"Yao, T., Pan, Y., Li, Y., Mei, T.: Exploring visual relationship for image captioning. In: ECCV, pp. 684\u2013699 (2018)","DOI":"10.1007\/978-3-030-01264-9_42"},{"key":"3343_CR40","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., Polosukhin, I.: Attention is all you need. In: NeurIPS, vol. 30 (2017)"},{"key":"3343_CR41","unstructured":"Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.: Relational inductive biases, deep learning, and graph networks. arXiv preprint (2018)"},{"issue":"6","key":"3343_CR42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3295748","volume":"51","author":"MZ Hossain","year":"2019","unstructured":"Hossain, M.Z., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. 51(6), 1\u201336 (2019)","journal-title":"ACM Comput. Surv."},{"key":"3343_CR43","doi-asserted-by":"crossref","unstructured":"Zou, Y., Xie, Q.: A survey on vqa: datasets and approaches. In: ITCA, pp. 289\u2013297. IEEE (2020)","DOI":"10.1109\/ITCA52113.2020.00069"},{"key":"3343_CR44","first-page":"1","volume":"5","author":"AA Yusuf","year":"2022","unstructured":"Yusuf, A.A., Chong, F., Xianling, M.: An analysis of graph convolutional networks and recent datasets for visual question answering. Artif. Intell. Rev. 5, 1\u201324 (2022)","journal-title":"Artif. Intell. Rev."},{"key":"3343_CR45","unstructured":"Alexander, M., Gunasekaran, S.: A survey on image retrieval methods. Preprint (2014)"},{"issue":"1","key":"3343_CR46","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R Krishna","year":"2017","unstructured":"Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D.A., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. IJCV 123(1), 32\u201373 (2017)","journal-title":"IJCV"},{"key":"3343_CR47","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: ECCV, pp. 740\u2013755. Springer (2014)","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"3343_CR48","first-page":"67","volume":"2","author":"P Young","year":"2014","unstructured":"Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans. ACL 2, 67\u201378 (2014)","journal-title":"Trans. ACL"},{"key":"3343_CR49","doi-asserted-by":"crossref","unstructured":"Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: visual question answering. In: ICCV, pp. 2425\u20132433 (2015)","DOI":"10.1109\/ICCV.2015.279"},{"issue":"10","key":"3343_CR50","doi-asserted-by":"crossref","first-page":"2413","DOI":"10.1109\/TPAMI.2017.2754246","volume":"40","author":"P Wang","year":"2017","unstructured":"Wang, P., Wu, Q., Shen, C., Dick, A., Van Den Hengel, A.: Fvqa: fact-based visual question answering. PAMI 40(10), 2413\u20132427 (2017)","journal-title":"PAMI"},{"key":"3343_CR51","doi-asserted-by":"crossref","unstructured":"Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: a visual question answering benchmark requiring external knowledge. In: CVPR (2019)","DOI":"10.1109\/CVPR.2019.00331"},{"key":"3343_CR52","doi-asserted-by":"crossref","unstructured":"Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: CVPR, pp. 8317\u20138326 (2019)","DOI":"10.1109\/CVPR.2019.00851"},{"key":"3343_CR53","doi-asserted-by":"crossref","unstructured":"Singh, A.K., Mishra, A., Shekhar, S., Chakraborty, A.: From strings to things: knowledge-enabled vqa model that can read and reason. In: ICCV, pp. 4602\u20134612 (2019)","DOI":"10.1109\/ICCV.2019.00470"},{"key":"3343_CR54","doi-asserted-by":"crossref","unstructured":"Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D.: Graph r-cnn for scene graph generation. In: ECCV (2018)","DOI":"10.1007\/978-3-030-01246-5_41"},{"key":"3343_CR55","unstructured":"Zhang, C., Chao, W.-L., Xuan, D.: An empirical study on leveraging scene graphs for visual question answering. In: Sidorov, K., Hicks, Y. (eds.) BMVC, pp. 151\u2013115114. BMVA Press (2019). https:\/\/dx.doi.org\/10.5244\/C.33.151"},{"key":"3343_CR56","doi-asserted-by":"crossref","unstructured":"Wu, S., Wieland, J., Farivar, O., Schiller, J.: Automatic alt-text: computer-generated image descriptions for blind users on a social network service. In: ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 1180\u20131192 (2017)","DOI":"10.1145\/2998181.2998364"},{"key":"3343_CR57","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR, pp. 3128\u20133137 (2015)","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"3343_CR58","doi-asserted-by":"crossref","unstructured":"Yang, X., Tang, K., Zhang, H., Cai, J.: Auto-encoding scene graphs for image captioning. In: CVPR, pp. 10685\u201310694 (2019)","DOI":"10.1109\/CVPR.2019.01094"},{"key":"3343_CR59","doi-asserted-by":"crossref","unstructured":"Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: CVPR, pp. 10578\u201310587 (2020)","DOI":"10.1109\/CVPR42600.2020.01059"},{"key":"3343_CR60","doi-asserted-by":"crossref","unstructured":"He, S., Liao, W., Tavakoli, H.R., Yang, M., Rosenhahn, B., Pugeault, N.: Image captioning through image transformer. In: ACCV (2020)","DOI":"10.1007\/978-3-030-69538-5_10"},{"key":"3343_CR61","doi-asserted-by":"crossref","unstructured":"Yang, X., Gao, C., Zhang, H., Cai, J.: Hierarchical scene graph encoder\u2013decoder for image paragraph captioning. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 4181\u20134189 (2020)","DOI":"10.1145\/3394171.3413859"},{"key":"3343_CR62","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.neucom.2020.02.041","volume":"396","author":"R Li","year":"2020","unstructured":"Li, R., Liang, H., Shi, Y., Feng, F., Wang, X.: Dual-cnn: a convolutional language decoder for paragraph image captioning. Neurocomputing 396, 92\u2013101 (2020)","journal-title":"Neurocomputing"},{"issue":"25","key":"3343_CR63","doi-asserted-by":"crossref","first-page":"35619","DOI":"10.1007\/s11042-021-11878-w","volume":"81","author":"V Jain","year":"2022","unstructured":"Jain, V., Al-Turjman, F., Chaudhary, G., Nayar, D., Gupta, V., Kumar, A.: Video captioning: a review of theory, techniques and practices. Multimed. Tools Appl. 81(25), 35619\u201335653 (2022)","journal-title":"Multimed. Tools Appl."},{"key":"3343_CR64","doi-asserted-by":"crossref","unstructured":"Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: elevating the role of image understanding in visual question answering. In: CVPR (2017)","DOI":"10.1109\/CVPR.2017.670"},{"key":"3343_CR65","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248\u2013255. IEEE (2009)","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"3343_CR66","doi-asserted-by":"crossref","unstructured":"Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: Dbpedia: A nucleus for a web of open data. In: The Semantic Web, pp. 722\u2013735. Springer (2007)","DOI":"10.1007\/978-3-540-76298-0_52"},{"issue":"4","key":"3343_CR67","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1023\/B:BTTJ.0000047600.45421.6d","volume":"22","author":"H Liu","year":"2004","unstructured":"Liu, H., Singh, P.: Conceptnet\u2014a practical commonsense reasoning tool-kit. BT Technol. J. 22(4), 211\u2013226 (2004)","journal-title":"BT Technol. J."},{"key":"3343_CR68","doi-asserted-by":"crossref","unstructured":"Tandon, N., Melo, G., Weikum, G.: Acquiring comparative commonsense knowledge from the web. In: AAAI, vol. 28 (2014)","DOI":"10.1609\/aaai.v28i1.8735"},{"key":"3343_CR69","unstructured":"Iwana, B.K., Rizvi, S.T.R., Ahmed, S., Dengel, A., Uchida, S.: Judging a book by its cover. arXiv preprint (2016)"},{"issue":"10","key":"3343_CR70","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2629489","volume":"57","author":"D Vrande\u010di\u0107","year":"2014","unstructured":"Vrande\u010di\u0107, D., Kr\u00f6tzsch, M.: Wikidata: a free collaborative knowledgebase. Commun. ACM 57(10), 78\u201385 (2014)","journal-title":"Commun. ACM"},{"key":"3343_CR71","doi-asserted-by":"crossref","unstructured":"Han, X., Wu, Z., Huang, P.X., Zhang, X., Zhu, M., Li, Y., Zhao, Y., Davis, L.S.: Automatic spatially-aware fashion concept discovery. In: ICCV, pp. 1463\u20131471 (2017)","DOI":"10.1109\/ICCV.2017.163"},{"key":"3343_CR72","first-page":"1","volume":"6","author":"Z Cui","year":"2022","unstructured":"Cui, Z., Hu, Y., Sun, Y., Gao, J., Yin, B.: Cross-modal alignment with graph reasoning for image-text retrieval. Multimed. Tools Appl. 6, 1\u201318 (2022)","journal-title":"Multimed. Tools Appl."},{"key":"3343_CR73","doi-asserted-by":"crossref","unstructured":"Yoon, S., Kang, W.Y., Jeon, S., Lee, S., Han, C., Park, J., Kim, E.-S.: Image-to-image retrieval by learning similarity between scene graphs. In: AAAI, vol. 35, pp. 10718\u201310726 (2021)","DOI":"10.1609\/aaai.v35i12.17281"},{"key":"3343_CR74","unstructured":"Misraa, A.K., Kale, A., Aggarwal, P., Aminian, A.: Multi-modal retrieval using graph neural networks. arXiv preprint (2020)"},{"key":"3343_CR75","doi-asserted-by":"crossref","unstructured":"Zhao, D., Wang, A., Russakovsky, O.: Understanding and evaluating racial biases in image captioning. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 14830\u201314840 (2021)","DOI":"10.1109\/ICCV48922.2021.01456"},{"key":"3343_CR76","doi-asserted-by":"crossref","unstructured":"Hirota, Y., Nakashima, Y., Garcia, N.: Gender and racial bias in visual question answering datasets. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1280\u20131292 (2022)","DOI":"10.1145\/3531146.3533184"},{"key":"3343_CR77","doi-asserted-by":"crossref","first-page":"107563","DOI":"10.1016\/j.patcog.2020.107563","volume":"108","author":"J Yu","year":"2020","unstructured":"Yu, J., Zhu, Z., Wang, Y., Zhang, W., Hu, Y., Tan, J.: Cross-modal knowledge reasoning for knowledge-based visual question answering. Pattern Recogn. 108, 107563 (2020)","journal-title":"Pattern Recogn."},{"key":"3343_CR78","unstructured":"Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: NeurIPS, vol. 28 (2015)"},{"key":"3343_CR79","doi-asserted-by":"crossref","unstructured":"Zhong, Y., Wang, L., Chen, J., Yu, D., Li, Y.: Comprehensive image captioning via scene graph decomposition. In: ECCV, pp. 211\u2013229. Springer (2020)","DOI":"10.1007\/978-3-030-58568-6_13"},{"key":"3343_CR80","doi-asserted-by":"crossref","unstructured":"Guo, L., Liu, J., Tang, J., Li, J., Luo, W., Lu, H.: Aligning linguistic words and visual semantic units for image captioning. In: ACM International Conference on Multimedia, pp. 765\u2013773 (2019)","DOI":"10.1145\/3343031.3350943"},{"key":"3343_CR81","doi-asserted-by":"crossref","unstructured":"Zhou, D., Yang, J., Zhang, C., Tang, Y.: Joint scence network and attention-guided for image captioning. In: ICDM, pp. 1535\u20131540. IEEE (2021)","DOI":"10.1109\/ICDM51629.2021.00201"},{"key":"3343_CR82","doi-asserted-by":"crossref","unstructured":"Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., Manning, C.D.: Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In: Proceedings of the Fourth Workshop on Vision and Language, pp. 70\u201380 (2015)","DOI":"10.18653\/v1\/W15-2812"},{"key":"3343_CR83","doi-asserted-by":"crossref","unstructured":"De\u00a0Marneffe, M.-C., Manning, C.D.: The stanford typed dependencies representation. In: Coling 2008: Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation, pp. 1\u20138 (2008)","DOI":"10.3115\/1608858.1608859"},{"key":"3343_CR84","doi-asserted-by":"crossref","unstructured":"Teney, D., Liu, L., Den\u00a0Hengel, A.: Graph-structured representations for visual question answering. In: CVPR, pp. 1\u20139 (2017)","DOI":"10.1109\/CVPR.2017.344"},{"key":"3343_CR85","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.neucom.2022.03.071","volume":"492","author":"H Pan","year":"2022","unstructured":"Pan, H., Huang, J.: Multimodal high-order relational network for vision-and-language tasks. Neurocomputing 492, 62\u201375 (2022)","journal-title":"Neurocomputing"},{"key":"3343_CR86","doi-asserted-by":"crossref","unstructured":"Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: semantic propositional image caption evaluation. In: ECCV, pp. 382\u2013398 (2016)","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"3343_CR87","doi-asserted-by":"crossref","unstructured":"Yao, T., Pan, Y., Li, Y., Mei, T.: Hierarchy parsing for image captioning. In: ICCV, pp. 2621\u20132629 (2019)","DOI":"10.1109\/ICCV.2019.00271"},{"key":"3343_CR88","doi-asserted-by":"crossref","unstructured":"Kan, J., Hu, K., Wang, Z., Wu, Q., Hagenbuchner, M., Tsoi, A.C.: Topic-guided local-global graph neural network for image captioning. In: ICME, pp. 1\u20136. IEEE (2021)","DOI":"10.1109\/ICMEW53276.2021.9455991"},{"key":"3343_CR89","doi-asserted-by":"crossref","unstructured":"Dong, X., Long, C., Xu, W., Xiao, C.: Dual graph convolutional networks with transformer and curriculum learning for image captioning. In: ICME, pp. 2615\u20132624 (2021)","DOI":"10.1145\/3474085.3475439"},{"key":"3343_CR90","unstructured":"Liu, C., Yu, G., Volkovs, M., Chang, C., Rai, H., Ma, J., Gorti, S.K.: Guided similarity separation for image retrieval. In: NeurIPS, vol. 32 (2019)"},{"key":"3343_CR91","doi-asserted-by":"crossref","unstructured":"Li, C., Wang, H., Zhang, Z., Sun, A., Ma, Z.: Topic modeling for short texts with auxiliary word embeddings. In: ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 165\u2013174 (2016)","DOI":"10.1145\/2911451.2911499"},{"key":"3343_CR92","doi-asserted-by":"crossref","unstructured":"Sui, J., Yu, H., Liang, X., Ping, P.: Image caption method based on graph attention network with global context. In: 2022 7th International Conference on Image, Vision and Computing (ICIVC), pp. 480\u2013487. IEEE (2022)","DOI":"10.1109\/ICIVC55077.2022.9886239"},{"key":"3343_CR93","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.cviu.2019.04.004","volume":"184","author":"U Chaudhuri","year":"2019","unstructured":"Chaudhuri, U., Banerjee, B., Bhattacharya, A.: Siamese graph convolutional network for content based remote sensing image retrieval. Comput. Vis. Image Underst. 184, 22\u201330 (2019)","journal-title":"Comput. Vis. Image Underst."},{"issue":"4","key":"3343_CR94","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3447772","volume":"54","author":"A Hogan","year":"2021","unstructured":"Hogan, A., Blomqvist, E., Cochez, M., d\u2019Amato, C., Melo, G.D., Gutierrez, C., Kirrane, S., Gayo, J.E.L., Navigli, R., Neumaier, S., et al.: Knowledge graphs. ACM Comput. Surv. 54(4), 1\u201337 (2021)","journal-title":"ACM Comput. Surv."},{"issue":"1","key":"3343_CR95","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/TNNLS.2020.2978386","volume":"32","author":"Z Wu","year":"2020","unstructured":"Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32(1), 4\u201324 (2020)","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"3343_CR96","doi-asserted-by":"crossref","unstructured":"Yang, S., Zhang, M., Dong, W., Li, D.: Betty: enabling large-scale GNN training with batch-level graph partitioning. In: Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, vol. 2, pp. 103\u2013117 (2023)","DOI":"10.1145\/3575693.3575725"},{"key":"3343_CR97","unstructured":"Kong, K., Chen, J., Kirchenbauer, J., Ni, R., Bruss, C.B., Goldstein, T.: Goat: a global transformer on large-scale graphs. In: International Conference on Machine Learning, pp. 17375\u201317390. PMLR (2023)"},{"key":"3343_CR98","doi-asserted-by":"crossref","unstructured":"Morris, C., Ritzert, M., Fey, M., Hamilton, W.L., Lenssen, J.E., Rattan, G., Grohe, M.: Weisfeiler and leman go neural: higher-order graph neural networks. In: AAAI, vol. 33, pp. 4602\u20134609 (2019)","DOI":"10.1609\/aaai.v33i01.33014602"},{"key":"3343_CR99","unstructured":"Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: NeurIPS, pp. 1025\u20131035 (2017)"},{"key":"3343_CR100","unstructured":"Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.S.: Gated graph sequence neural networks (2015). CoRR abs\/1511.05493"},{"key":"3343_CR101","doi-asserted-by":"crossref","unstructured":"Cho, K., Merrienboer, B., G\u00fcl\u00e7ehre, Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. In: Conference on Empirical Methods in Natural Language Processing (2014)","DOI":"10.3115\/v1\/D14-1179"},{"key":"3343_CR102","unstructured":"Veli\u010dkovi\u0107, P., Cucurull, G., Casanova, A., Romero, A., Li\u00f2, P., Bengio, Y.: Graph attention networks. In: ICLR. (Accepted as poster) (2018)"},{"key":"3343_CR103","doi-asserted-by":"crossref","unstructured":"Khademi, M.: Multimodal neural graph memory networks for visual question answering. In: Proceedings of the 58th Annual Meeting of the ACL, pp. 7177\u20137188 (2020)","DOI":"10.18653\/v1\/2020.acl-main.643"},{"key":"3343_CR104","doi-asserted-by":"crossref","unstructured":"Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., Sun, X.: Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In: AAAI, vol. 34, pp. 3438\u20133445 (2020)","DOI":"10.1609\/aaai.v34i04.5747"},{"key":"3343_CR105","unstructured":"Bodnar, C., Giovanni, F.D., Chamberlain, B.P., Li\u00f2, P., Bronstein, M.M.: Neural sheaf diffusion: a topological perspective on heterophily and oversmoothing in GNNs. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) NeurIPS (2022). https:\/\/openreview.net\/forum?id=vbPsD-BhOZ"},{"key":"3343_CR106","unstructured":"Abu-El-Haija, S., Perozzi, B., Kapoor, A., Alipourfard, N., Lerman, K., Harutyunyan, H., Ver\u00a0Steeg, G., Galstyan, A.: Mixhop: higher-order graph convolutional architectures via sparsified neighborhood mixing. In: ICML, pp. 21\u201329. PMLR (2019)"},{"key":"3343_CR107","first-page":"2625","volume":"34","author":"C Bodnar","year":"2021","unstructured":"Bodnar, C., Frasca, F., Otter, N., Wang, Y., Lio, P., Montufar, G.F., Bronstein, M.: Weisfeiler and lehman go cellular: Cw networks. NeurIPS 34, 2625\u20132640 (2021)","journal-title":"NeurIPS"},{"key":"3343_CR108","unstructured":"Chamberlain, B., Rowbottom, J., Gorinova, M.I., Bronstein, M., Webb, S., Rossi, E.: Grand: graph neural diffusion. In: ICML, pp. 1407\u20131418. PMLR (2021)"},{"issue":"9","key":"3343_CR109","first-page":"7239","volume":"13","author":"DS Lakshminarasimhan Srinivasan","year":"2018","unstructured":"Lakshminarasimhan Srinivasan, D.S., Amutha, A.: Image captioning-a deep learning approach. Int. J. Appl. Eng. Res. 13(9), 7239\u20137242 (2018)","journal-title":"Int. J. Appl. Eng. Res."},{"key":"3343_CR110","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the ACL, pp. 311\u2013318 (2002)","DOI":"10.3115\/1073083.1073135"},{"key":"3343_CR111","unstructured":"Lin, C.-Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74\u201381 (2004)"},{"key":"3343_CR112","unstructured":"Banerjee, S., Lavie, A.: Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In: ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And\/or Summarization, pp. 65\u201372 (2005)"},{"key":"3343_CR113","doi-asserted-by":"crossref","unstructured":"Vedantam, R., Lawrence\u00a0Zitnick, C., Parikh, D.: Cider: consensus-based image description evaluation. In: CVPR, pp. 4566\u20134575 (2015)","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"3343_CR114","doi-asserted-by":"crossref","unstructured":"Monti, F., Otness, K., Bronstein, M.M.: Motifnet: a motif-based graph convolutional network for directed graphs. In: 2018 IEEE Data Science Workshop (DSW), pp. 225\u2013228. IEEE (2018)","DOI":"10.1109\/DSW.2018.8439897"},{"key":"3343_CR115","doi-asserted-by":"crossref","unstructured":"Song, Z., Zhou, X.: Exploring explicit and implicit visual relationships for image captioning. In: ICME, pp. 1\u20136. IEEE (2021)","DOI":"10.1109\/ICME51207.2021.9428310"},{"key":"3343_CR116","doi-asserted-by":"crossref","unstructured":"Yang, X., Peng, J., Wang, Z., Xu, H., Ye, Q., Li, C., Yan, M., Huang, F., Li, Z., Zhang, Y.: Transforming visual scene graphs to image captions (2023). arXiv preprint arXiv:2305.02177","DOI":"10.18653\/v1\/2023.acl-long.694"},{"key":"3343_CR117","doi-asserted-by":"crossref","first-page":"107075","DOI":"10.1016\/j.patcog.2019.107075","volume":"98","author":"J Wang","year":"2020","unstructured":"Wang, J., Wang, W., Wang, L., Wang, Z., Feng, D.D., Tan, T.: Learning visual relationship and context-aware attention for image captioning. Pattern Recogn. 98, 107075 (2020)","journal-title":"Pattern Recogn."},{"key":"3343_CR118","doi-asserted-by":"crossref","unstructured":"Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint (2015)","DOI":"10.3115\/v1\/P15-1150"},{"key":"3343_CR119","first-page":"55","volume":"32","author":"S Herdade","year":"2019","unstructured":"Herdade, S., Kappeler, A., Boakye, K., Soares, J.: Image captioning: transforming objects into words. Adv. Neural Inf. Process. Syst. 32, 55 (2019)","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"3343_CR120","doi-asserted-by":"crossref","first-page":"104165","DOI":"10.1016\/j.imavis.2021.104165","volume":"110","author":"H Sharma","year":"2021","unstructured":"Sharma, H., Jalal, A.S.: Visual question answering model based on graph neural network and contextual attention. Image Vis. Comput. 110, 104165 (2021)","journal-title":"Image Vis. Comput."},{"key":"3343_CR121","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP, pp. 1532\u20131543 (2014)","DOI":"10.3115\/v1\/D14-1162"},{"key":"3343_CR122","doi-asserted-by":"crossref","unstructured":"Li, L., Gan, Z., Cheng, Y., Liu, J.: Relation-aware graph attention network for visual question answering. In: ICCV, pp. 10313\u201310322 (2019)","DOI":"10.1109\/ICCV.2019.01041"},{"key":"3343_CR123","doi-asserted-by":"crossref","unstructured":"Nuthalapati, S.V., Chandradevan, R., Giunchiglia, E., Li, B., Kayser, M., Lukasiewicz, T., Yang, C.: Lightweight visual question answering using scene graphs. In: ACM International Conference on Information and Knowledge Management, pp. 3353\u20133357 (2021)","DOI":"10.1145\/3459637.3482218"},{"key":"3343_CR124","doi-asserted-by":"crossref","unstructured":"Zhu, Z., Yu, J., Sun, Y., Hu, Y., Wang, Y., Wu, Q.: Mucko: multi-layer cross-modal knowledge reasoning for fact-based visual question answering. In: IJCAI (2020)","DOI":"10.24963\/ijcai.2020\/153"},{"key":"3343_CR125","unstructured":"Narasimhan, M., Lazebnik, S., Schwing, A.: Out of the box: reasoning with graph convolution nets for factual visual question answering. NeurIPS, vol. 31 (2018)"},{"key":"3343_CR126","doi-asserted-by":"crossref","first-page":"107650","DOI":"10.1016\/j.knosys.2021.107650","volume":"237","author":"L Liu","year":"2022","unstructured":"Liu, L., Wang, M., He, X., Qing, L., Chen, H.: Fact-based visual question answering via dual-process system. Knowl. Based Syst. 237, 107650 (2022)","journal-title":"Knowl. Based Syst."},{"issue":"5","key":"3343_CR127","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1017\/S0140525X00223431","volume":"23","author":"KE Stanovich","year":"2000","unstructured":"Stanovich, K.E., West, R.F.: 24 individual differences in reasoning: implications for the rationality debate? Behav. Brain Sci. 23(5), 665\u2013726 (2000)","journal-title":"Behav. Brain Sci."},{"key":"3343_CR128","doi-asserted-by":"crossref","unstructured":"Wang, Y., Yasunaga, M., Ren, H., Wada, S., Leskovec, J.: VQA-GNN: reasoning with multimodal knowledge via graph neural networks for visual question answering. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 21582\u201321592 (2023)","DOI":"10.1109\/ICCV51070.2023.01973"},{"key":"3343_CR129","doi-asserted-by":"crossref","unstructured":"Wang, Z., You, H., Li, L.H., Zareian, A., Park, S., Liang, Y., Chang, K.-W., Chang, S.-F.: Sgeitl: scene graph enhanced image-text learning for visual commonsense reasoning. In: AAAI, vol. 36, pp. 5914\u20135922 (2022)","DOI":"10.1609\/aaai.v36i5.20536"},{"key":"3343_CR130","doi-asserted-by":"crossref","unstructured":"Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y.N., Natarajan, P.: A thousand words are worth more than a picture: natural language-centric outside-knowledge visual question answering. arXiv preprint (2022)","DOI":"10.1109\/CVPR52688.2022.00501"},{"issue":"140","key":"3343_CR131","first-page":"1","volume":"21","author":"C Raffel","year":"2020","unstructured":"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 21(140), 1\u201367 (2020)","journal-title":"JMLR"},{"key":"3343_CR132","doi-asserted-by":"crossref","unstructured":"Gao, D., Li, K., Wang, R., Shan, S., Chen, X.: Multi-modal graph neural network for joint reasoning on vision and scene text. In: CVPR, pp. 12746\u201312756 (2020)","DOI":"10.1109\/CVPR42600.2020.01276"},{"key":"3343_CR133","doi-asserted-by":"crossref","unstructured":"Liang, Y., Wang, X., Duan, X., Zhu, W.: Multi-modal contextual graph neural network for text visual question answering. In: ICPR, pp. 3491\u20133498 (2021). DOIurlhttps:\/\/doi.org\/10.1109\/ICPR48806.2021.9412891","DOI":"10.1109\/ICPR48806.2021.9412891"},{"key":"3343_CR134","unstructured":"Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017). https:\/\/openreview.net\/forum?id=SJU4ayYgl"},{"key":"3343_CR135","unstructured":"Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint (2018)"},{"key":"3343_CR136","unstructured":"Zhang, X., Jiang, M., Zheng, Z., Tan, X., Ding, E., Yang, Y.: Understanding image retrieval re-ranking: a graph neural network perspective. arXiv preprint (2020)"},{"key":"3343_CR137","doi-asserted-by":"crossref","unstructured":"Wang, M., Zhou, W., Tian, Q., Li, H.: Deep graph convolutional quantization networks for image retrieval. IEEE Trans. Multimed. (2022)","DOI":"10.1109\/TMM.2022.3143694"},{"key":"3343_CR138","doi-asserted-by":"crossref","unstructured":"Zhang, F., Xu, M., Mao, Q., Xu, C.: Joint attribute manipulation and modality alignment learning for composing text and image to image retrieval. In: ACM International Conference on Multimedia, pp. 3367\u20133376 (2020)","DOI":"10.1145\/3394171.3413917"},{"key":"3343_CR139","first-page":"1","volume":"19","author":"U Chaudhuri","year":"2021","unstructured":"Chaudhuri, U., Banerjee, B., Bhattacharya, A., Datcu, M.: Attention-driven graph convolution network for remote sensing image retrieval. IEEE Geosci. Remote Sens. Lett. 19, 1\u20135 (2021)","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"3343_CR140","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhang, Y., Feng, R., Zhang, T., Fan, W.: Zero-shot sketch-based image retrieval via graph convolution network. In: AAAI, vol. 34, pp. 12943\u201312950 (2020)","DOI":"10.1609\/aaai.v34i07.6993"},{"key":"3343_CR141","doi-asserted-by":"publisher","unstructured":"Zhang, B., Xiong, D., Su, J., Duan, H., Zhang, M.: Variational neural machine translation. In: Conference on Empirical Methods in Natural Language Processing, pp. 521\u2013530. ACL, Austin, Texas (2016). https:\/\/doi.org\/10.18653\/v1\/D16-1050","DOI":"10.18653\/v1\/D16-1050"},{"key":"3343_CR142","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint (2020)"},{"key":"3343_CR143","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV, pp. 10012\u201310022 (2021)","DOI":"10.1109\/ICCV48922.2021.00986"},{"issue":"1","key":"3343_CR144","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TPAMI.2021.3137605","volume":"45","author":"X Chang","year":"2021","unstructured":"Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., Hauptmann, A.: A comprehensive survey of scene graphs: generation and application. IEEE Trans. Pattern Anal. Mach. Intell. 45(1), 1\u201326 (2021)","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"3343_CR145","doi-asserted-by":"crossref","unstructured":"Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., Gao, J.: Vinvl: revisiting visual representations in vision-language models. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579\u20135588 (2021)","DOI":"10.1109\/CVPR46437.2021.00553"},{"key":"3343_CR146","unstructured":"Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al.: Improving image generation with better captions. Computer Science. https:\/\/cdn.openai.com\/papers\/dall-e-3.pdf (2023)"},{"key":"3343_CR147","unstructured":"Xu, S.: Clip-diffusion-lm: apply diffusion model on image captioning. arXiv preprint (2022)"},{"key":"3343_CR148","doi-asserted-by":"crossref","unstructured":"Li, H., Gu, J., Koner, R., Sharifzadeh, S., Tresp, V.: Do dall-e and flamingo understand each other? arXiv preprint (2022)","DOI":"10.1109\/ICCV51070.2023.00191"},{"key":"3343_CR149","doi-asserted-by":"crossref","unstructured":"Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2021)","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"3343_CR150","unstructured":"Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint (2022)"},{"key":"3343_CR151","unstructured":"Conwell, C., Ullman, T.: Testing relational understanding in text-guided image generation. arXiv preprint (2022)"},{"key":"3343_CR152","doi-asserted-by":"crossref","unstructured":"Wei, C., Liu, C., Qiao, S., Zhang, Z., Yuille, A., Yu, J.: De-diffusion makes text a strong cross-modal interface (2023). arXiv preprint arXiv:2311.00618","DOI":"10.1109\/CVPR52733.2024.01281"},{"key":"3343_CR153","doi-asserted-by":"crossref","unstructured":"Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., et al.: Vizwiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 333\u2013342 (2010)","DOI":"10.1145\/1866029.1866080"},{"key":"3343_CR154","unstructured":"Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.: Pali: a jointly-scaled multilingual language-image model. arXiv preprint (2022)"},{"key":"3343_CR155","unstructured":"Zeng, Y., Zhang, X., Li, H., Wang, J., Zhang, J., Zhou, W.: X$$^2$$-vlm: all-in-one pre-trained model for vision-language tasks. arXiv preprint (2022)"},{"key":"3343_CR156","doi-asserted-by":"crossref","unstructured":"Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al.: MPLUG: effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint (2022)","DOI":"10.18653\/v1\/2022.emnlp-main.488"}],"container-title":["The Visual Computer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00371-024-03343-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00371-024-03343-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00371-024-03343-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,24]],"date-time":"2025-01-24T13:02:31Z","timestamp":1737723751000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00371-024-03343-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,29]]},"references-count":156,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["3343"],"URL":"https:\/\/doi.org\/10.1007\/s00371-024-03343-0","relation":{},"ISSN":["0178-2789","1432-2315"],"issn-type":[{"value":"0178-2789","type":"print"},{"value":"1432-2315","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,29]]},"assertion":[{"value":"26 February 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 March 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}