{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T19:37:27Z","timestamp":1782934647889,"version":"3.54.5"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T00:00:00Z","timestamp":1780272000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T00:00:00Z","timestamp":1781222400000},"content-version":"vor","delay-in-days":11,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SIViP"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Scene Graph Generation (SGG) aims to represent an image\u2019s objects and their pairwise relationships in a structured graph for downstream visual reasoning. However, conventional SGG models struggle with long-tail predicate distributions and closed-world vocabularies, resulting in poor generalization to rare or unseen relationships. We propose a neurosymbolic framework for zero-shot relationship retrieval that addresses these challenges by integrating deep visual features with external commonsense knowledge. Our model first detects objects and refines them via positional overlap and semantic similarity. It then retrieves candidate predicates through two complementary channels: (1) a visual-textual prototype retrieval that aligns subject-object representations with a broad predicate embedding space, and (2) a knowledge graph constrained retrieval that ranks relationships using heterogeneous commonsense graphs. A calibration and late-fusion module combines these channels, balancing confidence between head and tail classes. Evaluations on the Visual Genome (VG) and GQA benchmarks under zero-shot and open-vocabulary settings show strong strict zero-shot performance. On the reported VG split, ZeroRel reaches zR@100 = 37.1%, improving on the strongest prior zero-shot baseline in our comparison table (KnowZRel, 35.7%) while maintaining competitive overall recall and improved mean recall on rare predicates. The model also generalizes to GQA without retraining, demonstrating robust cross-dataset transfer. Ablations on knowledge sources and embedding models show that a heterogeneous Common Sense Knowledge Graph (CSKG) with ComplEx embeddings yields the best performance. These results indicate that combining visual prototype retrieval with structured knowledge retrieval improves coverage of rare and unseen relationships without sacrificing scene-graph quality on frequent predicates.<\/jats:p>","DOI":"10.1007\/s11760-026-05439-7","type":"journal-article","created":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T08:11:52Z","timestamp":1781251912000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ZeroRel: Multimodal Transformer-Guided Zero-Shot Relationship Retrieval for Generalized Scene Graph Generation"],"prefix":"10.1007","volume":"20","author":[{"given":"Muhammad Junaid","family":"Khan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adil Masood","family":"Siddiqui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maryam","family":"Rasool","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hussain","family":"Ali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Umar","family":"Ghafoor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaleed","family":"Khan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,12]]},"reference":[{"key":"5439_CR1","doi-asserted-by":"crossref","unstructured":"Khan, M.J., Ilievski, F., Breslin, J.G., Curry, E.: A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge. Neurosymbolic Artificial Intelligence (Pre-press), 1\u201324 (2024)","DOI":"10.3233\/NAI-240719"},{"key":"5439_CR2","unstructured":"Cohen, W.W., Sun, H., Hofer, R.A., Siegler, M.: Scalable neural methods for reasoning with a symbolic knowledge base (2020). arXiv preprint arXiv:2002.06115"},{"key":"5439_CR3","doi-asserted-by":"crossref","unstructured":"Kim, K., Yoon, K., Jeon, J., In, Y., Moon, J., Kim, D., Park, C.: Llm4sgg: Large language model for weakly supervised scene graph generation. arXiv e-prints, 2310 (2023)","DOI":"10.1109\/CVPR52733.2024.02674"},{"issue":"1","key":"5439_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TPAMI.2021.3137605","volume":"45","author":"X Chang","year":"2021","unstructured":"Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., Hauptmann, A.: A comprehensive survey of scene graphs: generation and application. IEEE Trans. Pattern Anal. Mach. Intell. 45(1), 1\u201326 (2021)","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"5439_CR5","doi-asserted-by":"crossref","unstructured":"Zhong, Y., Wang, L., Chen, J., Yu, D., Li, Y.: Comprehensive image captioning via scene graph decomposition. In: European Conference on Computer Vision, pp. 211\u2013229. Springer (2020)","DOI":"10.1007\/978-3-030-58568-6_13"},{"key":"5439_CR6","doi-asserted-by":"crossref","unstructured":"Koner, R., Li, H., Hildebrandt, M., Das, D., Tresp, V., G\u00fcnnemann, S.: Graphhopper: Multi-hop scene graph reasoning for visual question answering. In: International Semantic Web Conference, pp. 111\u2013127. Springer (2021)","DOI":"10.1007\/978-3-030-88361-4_7"},{"key":"5439_CR7","doi-asserted-by":"crossref","unstructured":"Im, J., Nam, J., Park, N., Lee, H., Park, S.: Egtr: Extracting graph from transformer for scene graph generation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 24229\u201324238 (2024)","DOI":"10.1109\/CVPR52733.2024.02287"},{"key":"5439_CR8","doi-asserted-by":"crossref","unstructured":"Chen, Z., Wu, J., Lei, Z., Zhang, Z., Chen, C.W.: Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention. In: European Conference on Computer Vision, pp. 108\u2013124. Springer (2024)","DOI":"10.1007\/978-3-031-72848-8_7"},{"key":"5439_CR9","doi-asserted-by":"crossref","unstructured":"Ilievski, F., Szekely, P., Zhang, B.: Cskg: The commonsense knowledge graph. In: European Semantic Web Conference, pp. 680\u2013696. Springer (2021)","DOI":"10.1007\/978-3-030-77385-4_41"},{"key":"5439_CR10","doi-asserted-by":"crossref","unstructured":"Ji, J., Krishna, R., Fei-Fei, L., Niebles, J.C.: Action genome: Actions as compositions of spatio-temporal scene graphs. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236\u201310247 (2020)","DOI":"10.1109\/CVPR42600.2020.01025"},{"key":"5439_CR11","doi-asserted-by":"crossref","unstructured":"Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., Ling, M.: Scene graph generation with external knowledge and image reconstruction. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969\u20131978 (2019)","DOI":"10.1109\/CVPR.2019.00207"},{"key":"5439_CR12","doi-asserted-by":"crossref","unstructured":"Ilievski, F., Garijo, D., Chalupsky, H., Divvala, N.T., Yao, Y., Rogers, C., Li, R., Liu, J., Singh, A., Schwabe, D., et al.: Kgtk: a toolkit for large knowledge graph manipulation and analysis. In: International Semantic Web Conference, pp. 278\u2013293. Springer (2020)","DOI":"10.1007\/978-3-030-62466-8_18"},{"key":"5439_CR13","doi-asserted-by":"crossref","unstructured":"Kan, X., Cui, H., Yang, C.: Zero-shot scene graph relation prediction through commonsense knowledge integration. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 466\u2013482. Springer (2021)","DOI":"10.1007\/978-3-030-86520-7_29"},{"key":"5439_CR14","unstructured":"Trouillon, T., Welbl, J., Riedel, S., Gaussier, \u00c9., Bouchard, G.: Complex embeddings for simple link prediction. In: International Conference on Machine Learning (ICML), pp. 2071\u20132080 (2016). PMLR"},{"issue":"4","key":"5439_CR15","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1109\/MIC.2022.3176500","volume":"26","author":"MJ Khan","year":"2022","unstructured":"Khan, M.J., Breslin, J.G., Curry, E.: Common sense knowledge infusion for visual understanding and reasoning: Approaches, challenges, and applications. IEEE Internet Comput. 26(4), 21\u201327 (2022)","journal-title":"IEEE Internet Comput."},{"issue":"1","key":"5439_CR16","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R Krishna","year":"2017","unstructured":"Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123(1), 32\u201373 (2017)","journal-title":"Int. J. Comput. Vision"},{"key":"5439_CR17","doi-asserted-by":"crossref","unstructured":"Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700\u20136709 (2019)","DOI":"10.1109\/CVPR.2019.00686"},{"key":"5439_CR18","doi-asserted-by":"crossref","unstructured":"Khan, M.J., Breslin, J.G., Curry, E.: Knowzrel: Common sense knowledge-based zero-shot relationship retrieval for generalised scene graph generation. IEEE Transactions on Artificial Intelligence (2025)","DOI":"10.1109\/TAI.2025.3544177"},{"key":"5439_CR19","doi-asserted-by":"crossref","unstructured":"Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural motifs: Scene graph parsing with global context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831\u20135840 (2018)","DOI":"10.1109\/CVPR.2018.00611"},{"key":"5439_CR20","doi-asserted-by":"crossref","unstructured":"Tang, K., Zhang, H., Wu, B., Luo, W., Liu, W.: Learning to compose dynamic tree structures for visual contexts. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 6619\u20136628 (2019)","DOI":"10.1109\/CVPR.2019.00678"},{"key":"5439_CR21","doi-asserted-by":"crossref","unstructured":"Tang, K., Niu, Y., Huang, J., Shi, J., Zhang, H.: Unbiased scene graph generation from biased training. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 3716\u20133725 (2020)","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"5439_CR22","unstructured":"Zhou, L., Zhou, Y., Lam, T.L., Xu, Y.: Context-aware mixture-of-experts for unbiased scene graph generation. (2022) arXiv preprint arXiv:2208.07109"},{"key":"5439_CR23","doi-asserted-by":"crossref","unstructured":"Ma, K., Ilievski, F., Francis, J., Bisk, Y., Nyberg, E., Oltramari, A.: Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. In: 35th AAAI Conference on Artificial Intelligence, (2021)","DOI":"10.1609\/aaai.v35i15.17593"},{"key":"5439_CR24","doi-asserted-by":"crossref","unstructured":"Khan, M.J., Breslin, J.G., Curry, E.: Expressive scene graph generation using commonsense knowledge infusion for visual understanding and reasoning. In: European Semantic Web Conference, pp. 93\u2013112. Springer (2022)","DOI":"10.1007\/978-3-031-06981-9_6"},{"key":"5439_CR25","doi-asserted-by":"crossref","unstructured":"Li, R., Zhang, S., Lin, D., Chen, K., He, X.: From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 28076\u201328086 (2024)","DOI":"10.1109\/CVPR52733.2024.02652"},{"key":"5439_CR26","doi-asserted-by":"crossref","unstructured":"Ali, M., Khan, S.: Clip-decoder: Zeroshot multilabel classification using multimodal clip aligned representations. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 4675\u20134679. (2023)","DOI":"10.1109\/ICCVW60793.2023.00505"},{"key":"5439_CR27","doi-asserted-by":"publisher","first-page":"132238","DOI":"10.52202\/079017-4203","volume":"37","author":"G Chen","year":"2024","unstructured":"Chen, G., Li, J., Wang, W.: Scene graph generation with role-playing large language models. Adv. Neural. Inf. Process. Syst. 37, 132238\u2013132266 (2024)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"5439_CR28","doi-asserted-by":"crossref","unstructured":"Yang, D., Kim, M., Mac\u00a0Kim, S., Kwak, B.-w., Park, M., Hong, J., Woo, W., Yeo, J.: Llm meets scene graph: Can large language models understand and generate scene graphs? a benchmark and empirical study. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 21335\u201321360 (2025)","DOI":"10.18653\/v1\/2025.acl-long.1036"},{"key":"5439_CR29","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., Chen, C.-W.: Boosting scene graph generation with visual relation saliency. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) (2022)","DOI":"10.1145\/3514041"},{"key":"5439_CR30","doi-asserted-by":"crossref","unstructured":"Huang, W., JI, Y., Zhu, G., Ying, L., Liu, C.: Navigating the unseen: Zero-shot scene graph generation via capsule-based equivariant features. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29448\u201329457 (2025)","DOI":"10.1109\/CVPR52734.2025.02742"},{"key":"5439_CR31","doi-asserted-by":"crossref","unstructured":"Yu, X., Li, J., Yuan, S., Wang, C., Wu, C.: Zero-shot scene graph generation with relational graph neural networks. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 1894\u20131900 (2022). IEEE","DOI":"10.1109\/ICPR56361.2022.9956712"},{"key":"5439_CR32","doi-asserted-by":"crossref","unstructured":"Yu, X., Chen, R., Li, J., Sun, J., Yuan, S., Ji, H., Lu, X., Wu, C.: Zero-shot scene graph generation with knowledge graph completion. In: 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1\u20136 (2022). IEEE","DOI":"10.1109\/ICME52920.2022.9859944"},{"key":"5439_CR33","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp. 2117\u20132125 (2017)","DOI":"10.1109\/CVPR.2017.106"},{"key":"5439_CR34","doi-asserted-by":"crossref","unstructured":"Chen, T., Yu, W., Chen, R., Lin, L.: Knowledge-embedded routing network for scene graph generation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 6163\u20136171 (2019)","DOI":"10.1109\/CVPR.2019.00632"},{"key":"5439_CR35","unstructured":"Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Adv. Neural. Inf. Process. Syst. 26, (2013)"},{"key":"5439_CR36","doi-asserted-by":"crossref","unstructured":"Zareian, A., Karaman, S., Chang, S.-F.: Bridging knowledge graphs to generate scene graphs. In: European Conference on Computer Vision, pp. 606\u2013623. Springer (2020)","DOI":"10.1007\/978-3-030-58592-1_36"},{"key":"5439_CR37","unstructured":"Khan, M.J., Siddiqui, A.M., Khan, H.S., Akram, F., Khan, J.: Murelsgg: Multimodal relationship prediction for neurosymbolic scene graph generation. IEEE Access (2025)"},{"key":"5439_CR38","doi-asserted-by":"crossref","unstructured":"Jiang, B., Zhuang, Z., Shivakumar, S.S., Taylor, C.J.: Enhancing scene graph generation with hierarchical relationships and commonsense knowledge. In: 2025 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 8883\u20138894. IEEE (2025)","DOI":"10.1109\/WACV61041.2025.00861"},{"key":"5439_CR39","doi-asserted-by":"crossref","unstructured":"Zhu, X., Xing, Y., Wang, R., Wang, Y., Lan, X.: Calibration for long-tailed scene graph generation. In: Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3037\u20133046 (2024)","DOI":"10.1145\/3664647.3680818"},{"issue":"6","key":"5439_CR40","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","volume":"39","author":"S Ren","year":"2016","unstructured":"Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137\u20131149 (2016)","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"issue":"14","key":"5439_CR41","doi-asserted-by":"publisher","first-page":"8073","DOI":"10.1007\/s00521-024-09505-4","volume":"36","author":"SA Balanya","year":"2024","unstructured":"Balanya, S.A., Maronas, J., Ramos, D.: Adaptive temperature scaling for robust calibration of deep neural networks. Neural Comput. Appl. 36(14), 8073\u20138095 (2024)","journal-title":"Neural Comput. Appl."},{"key":"5439_CR42","unstructured":"Rusak, E., Reizinger, P., Juhos, A., Bringmann, O., Zimmermann, R.S., Brendel, W.: Infonce: Identifying the gap between theory and practice (2024). arXiv preprint arXiv:2407.00143"},{"key":"5439_CR43","unstructured":"Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 1321\u20131330 (2017). PMLR"},{"issue":"13","key":"5439_CR44","doi-asserted-by":"publisher","first-page":"3952","DOI":"10.3390\/s25133952","volume":"25","author":"L Yu","year":"2025","unstructured":"Yu, L., Tang, L., Mu, L.: A review of detection transformer: From basic architecture to advanced developments and visual perception applications. Sensors 25(13), 3952 (2025)","journal-title":"Sensors"},{"key":"5439_CR45","doi-asserted-by":"crossref","unstructured":"Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: An open multilingual graph of general knowledge. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4444\u20134451 (2017)","DOI":"10.1609\/aaai.v31i1.11164"},{"key":"5439_CR46","unstructured":"Yang, B., Yih, W.-T., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases (2014). arXiv preprint arXiv:1412.6575"}],"container-title":["Signal, Image and Video Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11760-026-05439-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11760-026-05439-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11760-026-05439-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T17:50:23Z","timestamp":1782928223000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11760-026-05439-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6]]},"references-count":46,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["5439"],"URL":"https:\/\/doi.org\/10.1007\/s11760-026-05439-7","relation":{},"ISSN":["1863-1703","1863-1711"],"issn-type":[{"value":"1863-1703","type":"print"},{"value":"1863-1711","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6]]},"assertion":[{"value":"24 November 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 April 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 June 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"411"}}