{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T19:26:36Z","timestamp":1781292396893,"version":"3.54.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T00:00:00Z","timestamp":1672876800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2020AAA0108600"],"award-info":[{"award-number":["2020AAA0108600"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2023,1,31]]},"abstract":"<jats:p>The scene graph is a symbolic data structure that comprehensively describes the objects and visual relations in a visual scene, while ignoring the inherent perceptual saliency of each visual relation (i.e., relation saliency). However, humans often quickly allocate attention to important\/salient visual relations in a scene. To align with such human perception of a scene, we explicitly model the perceptual saliency of visual relation in scene graph by upgrading each graph edge (i.e., visual relation) with an attribute of relation saliency. We present a new design, named as Saliency-guided Message Passing (SMP), that boosts the generation of such scene graph structure with the guidance from the visual relation saliency. Technically, an object interaction encoder is first utilized to strengthen object relation representations by jointly exploiting the appearance, semantic, and spatial relations in between. A branch is further leveraged to estimate the relation saliency of each visual relation by ordinal regression. Next, conditioned on the object and relation features (coupled with the estimated relation saliency), our SMP enhances scene graph generation by performing message passing over the objects and the most salient relations. Extensive experiments on VG-KR and VG150 datasets demonstrate the superiority of SMP for the scene graph generation. Moreover, we empirically validate the compelling generalizability of the learned scene graphs via SMP on downstream tasks like cross-model retrieval and image captioning.<\/jats:p>","DOI":"10.1145\/3514041","type":"journal-article","created":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T13:36:53Z","timestamp":1647524213000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Boosting Scene Graph Generation with Visual Relation Saliency"],"prefix":"10.1145","volume":"19","author":[{"given":"Yong","family":"Zhang","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Longgang District Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingwei","family":"Pan","sequence":"additional","affiliation":[{"name":"JD AI Research, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting","family":"Yao","sequence":"additional","affiliation":[{"name":"JD AI Research, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Huang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Longgang District Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Mei","sequence":"additional","affiliation":[{"name":"JD AI Research, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang-Wen","family":"Chen","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,1,5]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00636"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2015.2487833"},{"issue":"6","key":"e_1_3_2_4_2","first-page":"13","article-title":"Rank-consistent ordinal regression for neural networks","volume":"1","author":"Cao Wenzhi","year":"2019","unstructured":"Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. 2019. Rank-consistent ordinal regression for neural networks. arXiv preprint arXiv:1901.07884 1, 6 (2019), 13.","journal-title":"arXiv preprint arXiv:1901.07884"},{"key":"e_1_3_2_5_2","first-page":"4613","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Chen Long","year":"2019","unstructured":"Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. 2019. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 4613\u20134623."},{"key":"e_1_3_2_6_2","first-page":"6163","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Chen Tianshui","year":"2019","unstructured":"Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. 2019. Knowledge-embedded routing network for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 6163\u20136171."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3177745"},{"key":"e_1_3_2_8_2","unstructured":"Fartash Faghri David J. Fleet Jamie Ryan Kiros and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference ."},{"key":"e_1_3_2_9_2","article-title":"Correspondence autoencoders for cross-modal retrieval","author":"Feng Fangxiang","year":"2015","unstructured":"Fangxiang Feng, Xiaojie Wang, Ruifan Li, and Ibrar Ahmad. 2015. Correspondence autoencoders for cross-modal retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 26 (2015), 1\u201322.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-44795-4_13"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.322"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298990"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00898"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0981-7"},{"key":"e_1_3_2_17_2","first-page":"11109","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Li Rongjie","year":"2021","unstructured":"Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. 2021. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11109\u201311119."},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Yehao Li Jiahao Fan Yingwei Pan Ting Yao Weiyao Lin and Tao Mei. 2022. Uni-EDEN: Universal encoder-decoder network by multi-granular vision-language pre-training. ACM Transactions on Multimedia Computing Communications and Applications (TOMM) 18 2 Article 48 (2022) 16.","DOI":"10.1145\/3473140"},{"key":"e_1_3_2_19_2","first-page":"335","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Li Yikang","year":"2018","unstructured":"Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. 2018. Factorizable net: An efficient subgraph-based framework for scene graph generation. In Proceedings of the European Conference on Computer Vision. 335\u2013351."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.142"},{"key":"e_1_3_2_21_2","first-page":"3799","volume-title":"Proceedings of the 29th ACM International Conference on Multimedia","author":"Li Yehao","year":"2021","unstructured":"Yehao Li, Yingwei Pan, Jingwen Chen, Ting Yao, and Tao Mei. 2021. X-modaler: A versatile and high-performance codebase for cross-modal analytics. In Proceedings of the 29th ACM International Conference on Multimedia. 3799\u20133802."},{"key":"e_1_3_2_22_2","first-page":"8518","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence.","author":"Li Yehao","year":"2021","unstructured":"Yehao Li, Yingwei Pan, Ting Yao, Jingwen Chen, and Tao Mei. 2021. Scheduled sampling in vision-language pretraining with decoupled encoder-decoder network. In Proceedings of the AAAI Conference on Artificial Intelligence.8518\u20138526."},{"key":"e_1_3_2_23_2","first-page":"12497","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Li Yehao","year":"2019","unstructured":"Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. 2019. Pointing novel objects in image captioning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12497\u201312506."},{"key":"e_1_3_2_24_2","article-title":"Ordinal regression by extended binary classifications","author":"Lin H. T.","year":"2007","unstructured":"H. T. Lin and L. Li. 2007. Ordinal regression by extended binary classifications. In Advances in Neural Information Processing Systems, B. Sch\u00f6lkopf, J. Platt, and T. Hoffman (Eds.). Vol. 19. MIT Press. https:\/\/proceedings.neurips.cc\/paper\/2006\/file\/019f8b946a256d9357eadc5ace2c8678-Paper.pdf.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_25_2","first-page":"3746","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Lin Xin","year":"2020","unstructured":"Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. 2020. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3746\u20133753."},{"key":"e_1_3_2_26_2","first-page":"852","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Lu Cewu","year":"2016","unstructured":"Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In Proceedings of the European Conference on Computer Vision. Springer, 852\u2013869."},{"key":"e_1_3_2_27_2","first-page":"5600","volume-title":"Proceedings of the 29th ACM International Conference on Multimedia","author":"Luo Jianjie","year":"2021","unstructured":"Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. 2021. CoCo-BERT: Improving video-language pre-training with contrastive cross-modal matching and denoising. In Proceedings of the 29th ACM International Conference on Multimedia. 5600\u20135608."},{"key":"e_1_3_2_28_2","unstructured":"Jianming Lv Qinzhe Xiao and Jiajie Zhong. 2020. AVR: Attention based salient visual relationship detection. arXiv:2003.07012. Retrieved from https:\/\/arxiv.org\/abs\/2003.07012."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.532"},{"key":"e_1_3_2_30_2","unstructured":"Yingwei Pan Yehao Li Jianjie Luo Jun Xu Ting Yao and Tao Mei. 2020. Auto-captions on GIF: A large-scale video-sentence dataset for vision-language pre-training. arXiv:2007.02375. Retrieved from https:\/\/arxiv.org\/abs\/2007.02375."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01098"},{"key":"e_1_3_2_32_2","first-page":"1532","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.","author":"Pennington Jeffrey","year":"2014","unstructured":"Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.1532\u20131543."},{"key":"e_1_3_2_33_2","first-page":"3957","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Qi Mengshi","year":"2019","unstructured":"Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. 2019. Attentive relational networks for mapping images to scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3957\u20133966."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2577031"},{"key":"e_1_3_2_35_2","unstructured":"Kaihua Tang. 2020. A Scene Graph Generation Codebase in PyTorch. Retrieved 11 March 2020 from https:\/\/github.com\/KaihuaTang\/Scene-Graph-Benchmark.pytorch."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"e_1_3_2_37_2","first-page":"6619","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Tang Kaihua","year":"2019","unstructured":"Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. 2019. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 6619\u20136628."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.344"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3226037"},{"key":"e_1_3_2_40_2","first-page":"8188","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang Wenbin","year":"2019","unstructured":"Wenbin Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2019. Exploring context and visual pattern of relationship for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8188\u20138197."},{"key":"e_1_3_2_41_2","first-page":"222","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Wang Wenbin","year":"2020","unstructured":"Wenbin Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2020. Sketching image gist: Human-mimetic hierarchical scene graph generation. In Proceedings of the European Conference on Computer Vision. Springer, 222\u2013239."},{"issue":"4","key":"e_1_3_2_42_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3271485","article-title":"Image captioning via semantic guidance attention and consensus selection strategy","volume":"14","author":"Wu Jie","year":"2018","unstructured":"Jie Wu, Haifeng Hu, and Yi Wu. 2018. Image captioning via semantic guidance attention and consensus selection strategy. ACM Transactions on Multimedia Computing, Communications, and Applications 14, 4 (2018), 1\u201319.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.634"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.330"},{"issue":"1","key":"e_1_3_2_45_2","first-page":"1","article-title":"Exploring image enhancement for salient object detection in low light images","volume":"17","author":"Xu Xin","year":"2021","unstructured":"Xin Xu, Shiqin Wang, Zheng Wang, Xiaolong Zhang, and Ruimin Hu. 2021. Exploring image enhancement for salient object detection in low light images. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1\u201319.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_2_46_2","first-page":"670","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Yang Jianwei","year":"2018","unstructured":"Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Graph r-cnn for scene graph generation. In Proceedings of the European Conference on Computer Vision. 670\u2013685."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01264-9_42"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00271"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.524"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00166"},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of the ACM International Conference on Multimedia","author":"Yu Fan","year":"2020","unstructured":"Fan Yu, Haonan Wang, Tongwei Ren, Jinhui Tang, and Gangshan Wu. 2020. Visual relation of interest detection. In Proceedings of the ACM International Conference on Multimedia."},{"key":"e_1_3_2_52_2","first-page":"606","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Zareian Alireza","year":"2020","unstructured":"Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. 2020. Bridging knowledge graphs to generate scene graphs. In Proceedings of the European Conference on Computer Vision. Springer, 606\u2013623."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00611"},{"issue":"1","key":"e_1_3_2_54_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3412847","article-title":"HCMSL: Hybrid cross-modal similarity learning for cross-modal retrieval","volume":"17","author":"Zhang Chengyuan","year":"2021","unstructured":"Chengyuan Zhang, Jiayu Song, Xiaofeng Zhu, Lei Zhu, and Shichao Zhang. 2021. HCMSL: Hybrid cross-modal similarity learning for cross-modal retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1\u201322.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_2_55_2","first-page":"11535","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhang Ji","year":"2019","unstructured":"Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. 2019. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11535\u201311543."},{"key":"e_1_3_2_56_2","first-page":"13041","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence.","author":"Zhou Luowei","year":"2020","unstructured":"Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence.13041\u201313049."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514041","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3514041","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:35Z","timestamp":1750186955000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514041"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,5]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,31]]}},"alternative-id":["10.1145\/3514041"],"URL":"https:\/\/doi.org\/10.1145\/3514041","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,5]]},"assertion":[{"value":"2021-06-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}