{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,22]],"date-time":"2025-11-22T11:33:24Z","timestamp":1763811204300,"version":"3.41.0"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T00:00:00Z","timestamp":1697587200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2018YFB1800804"],"award-info":[{"award-number":["2018YFB1800804"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["NSFC 61925105, 62171257 and 62101307"],"award-info":[{"award-number":["NSFC 61925105, 62171257 and 62101307"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Tsinghua University-China Mobile Communications Group Co., Ltd"},{"name":"Tencent Foundation through the XPLORER PRIZE"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>Scene graph generation (SGG) has been developed to detect objects and their relationships from the visual data and has attracted increasing attention in recent years. Existing works have focused on extracting object context for SGG. However, very few works have attempted to exploit implicit contextual correlations among relationships of the objects. Furthermore, most existing SGG schemes rely on high-level features to predict the predicates while overlooking the potential inherent association of low-level features with the object relationships. We present in this article a novel scheme to capture enhanced contextual information for both objects and relationships. We design a Dual-branch Context Analysis Transformer (DCAT) architecture to extract both object context and relationship context from the visual data with dual transformer branches and then effectively fuse both high-level and low-level features by an adaptive approach to facilitate relationship prediction. Specifically, we first conduct feature representation learning to enrich relation representations by the visual, spatial, and linguistic feature extractors. Next, two transformer branches are designed to leverage the modeling of global associative interaction and mine the hidden association among objects and relationships. Then, we devise a novel feature disentangling method to decouple contextualized high-level features with guidance from the visual semantics. Finally, we develop a refined attention module to perform low-level feature recalibration for the refinement of the final predicate prediction. Experiments on Visual Genome and Action Genome datasets demonstrate the effectiveness of DCAT for both image and video SGG settings. Moreover, we also test the quality of the generated image scene graphs to verify the generalizability on downstream tasks like sentence-to-graph retrieval and image retrieval.<\/jats:p>","DOI":"10.1145\/3615868","type":"journal-article","created":{"date-parts":[[2023,8,15]],"date-time":"2023-08-15T10:19:31Z","timestamp":1692094771000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Boosting Scene Graph Generation with Contextual Information"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8483-0110","authenticated-orcid":false,"given":"Shiqi","family":"Sun","sequence":"first","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7719-4219","authenticated-orcid":false,"given":"Danlan","family":"Huang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8763-9338","authenticated-orcid":false,"given":"Xiaoming","family":"Tao","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7461-2016","authenticated-orcid":false,"given":"Chengkang","family":"Pan","sequence":"additional","affiliation":[{"name":"China Mobile Research Institute, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8656-1946","authenticated-orcid":false,"given":"Guangyi","family":"Liu","sequence":"additional","affiliation":[{"name":"China Mobile Research Institute, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6720-234X","authenticated-orcid":false,"given":"Changwen","family":"Chen","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,18]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Yunsheng Bai Hao Ding Yang Qiao Agustin Marinovic Ken Gu Ting Chen Yizhou Sun and Wei Wang. 2019. Unsupervised inductive graph-level representation learning via graph-graph proximity. In Proceedings of the 28th International Joint Conference on Artificial Intelligence . 1988\u20131994.","DOI":"10.24963\/ijcai.2019\/275"},{"key":"e_1_3_1_3_2","first-page":"423","volume-title":"Proceedings of the 8th International Conference on Machine Vision","author":"Burnaev Evgeny","year":"2015","unstructured":"Evgeny Burnaev, Pavel Erofeev, and Artem Papanov. 2015. Influence of resampling on accuracy of imbalanced classification. In Proceedings of the 8th International Conference on Machine Vision. SPIE, 423\u2013427."},{"key":"e_1_3_1_4_2","first-page":"4613","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Chen Long","year":"2019","unstructured":"Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. 2019. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 4613\u20134623."},{"key":"e_1_3_1_5_2","first-page":"4086","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Chen Xinlei","year":"2017","unstructured":"Xinlei Chen and Abhinav Gupta. 2017. Spatial memory for context reasoning in object detection. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 4086\u20134096."},{"key":"e_1_3_1_6_2","first-page":"508","volume-title":"Proceedings of the IEEE International Conference on Multimedia and Expo","author":"Chen Yunian","year":"2019","unstructured":"Yunian Chen, Yanjie Wang, Yang Zhang, and Yanwen Guo. 2019. Panet: A context-based predicate association network for scene graph generation. In Proceedings of the IEEE International Conference on Multimedia and Expo. 508\u2013513."},{"key":"e_1_3_1_7_2","first-page":"16372","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Cong Yuren","year":"2021","unstructured":"Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 16372\u201316382."},{"key":"e_1_3_1_8_2","first-page":"1271","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Divvala Santosh K.","year":"2009","unstructured":"Santosh K. Divvala, Derek Hoiem, James H. Hays, Alexei A. Efros, and Martial Hebert. 2009. An empirical study of context in object detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1271\u20131278."},{"key":"e_1_3_1_9_2","first-page":"0","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops","author":"Gkanatsios Nikolaos","year":"2019","unstructured":"Nikolaos Gkanatsios, Vassilis Pitsikalis, Petros Koutras, and Petros Maragos. 2019. Attention-translation-relation network for scalable scene graph generation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops. 0\u20130."},{"key":"e_1_3_1_10_2","first-page":"1840","volume-title":"Proceedings of the IEEE International Conference on Image Processing","author":"Gkanatsios Nikolaos","year":"2019","unstructured":"Nikolaos Gkanatsios, Vassilis Pitsikalis, Petros Koutras, Athanasia Zlatintsi, and Petros Maragos. 2019. Deeply supervised multimodal attentional translation embeddings for visual relationship detection. In Proceedings of the IEEE International Conference on Image Processing. 1840\u20131844."},{"key":"e_1_3_1_11_2","first-page":"10323","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Gu Jiuxiang","year":"2019","unstructured":"Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. 2019. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 10323\u201310332."},{"key":"e_1_3_1_12_2","first-page":"1969","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Gu Jiuxiang","year":"2019","unstructured":"Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. 2019. Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1969\u20131978."},{"key":"e_1_3_1_13_2","first-page":"2961","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"He Kaiming","year":"2017","unstructured":"Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 2961\u20132969."},{"key":"e_1_3_1_14_2","first-page":"210","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Herzig Roei","year":"2020","unstructured":"Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. 2020. Learning canonical representations for scene graph to image generation. In Proceedings of the European Conference on Computer Vision. Springer, 210\u2013227."},{"key":"e_1_3_1_15_2","first-page":"7132","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Hu Jie","year":"2018","unstructured":"Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 7132\u20137141."},{"key":"e_1_3_1_16_2","first-page":"4700","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Huang Gao","year":"2017","unstructured":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 4700\u20134708."},{"issue":"11","key":"e_1_3_1_17_2","doi-asserted-by":"crossref","first-page":"3820","DOI":"10.1109\/TPAMI.2020.2992222","article-title":"Contextual translation embedding for visual relationship detection and scene graph generation","volume":"43","author":"Hung Zih-Siou","year":"2020","unstructured":"Zih-Siou Hung, Arun Mallya, and Svetlana Lazebnik. 2020. Contextual translation embedding for visual relationship detection and scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3820\u20133832.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_18_2","first-page":"10236","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ji Jingwei","year":"2020","unstructured":"Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2020. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10236\u201310247."},{"key":"e_1_3_1_19_2","first-page":"1219","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Johnson Justin","year":"2018","unstructured":"Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018. Image generation from scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1219\u20131228."},{"key":"e_1_3_1_20_2","first-page":"3668","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Johnson Justin","year":"2015","unstructured":"Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3668\u20133678."},{"key":"e_1_3_1_21_2","first-page":"3128","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Karpathy Andrej","year":"2015","unstructured":"Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3128\u20133137."},{"key":"e_1_3_1_22_2","unstructured":"Jin-Hwa Kim Jaehyun Jun and Byoung-Tak Zhang. 2018. Bilinear attention networks. In Proceedings of the Advances in Neural Information Processing Systems 31 (2018). 1571\u20131581."},{"issue":"1","key":"e_1_3_1_23_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","article-title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","volume":"123","author":"Krishna Ranjay","year":"2017","unstructured":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, 1 (2017), 32\u201373.","journal-title":"International Journal of Computer Vision"},{"key":"e_1_3_1_24_2","unstructured":"Hanchao Li Pengfei Xiong Jie An and Lingxue Wang. 2018. Pyramid attention network for semantic segmentation. In Proceedings of the British Machine Vision Conference (2018)."},{"key":"e_1_3_1_25_2","unstructured":"Li-Jia Li Hao Su Li Fei-Fei and Eric Xing. 2010. Object bank: A high-level image representation for scene classification and semantic feature sparsification. In Proceedings of the Advances in Neural Information Processing Systems 23 (2010). 1378\u20131386."},{"key":"e_1_3_1_26_2","first-page":"11109","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Li Rongjie","year":"2021","unstructured":"Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. 2021. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11109\u201311119."},{"key":"e_1_3_1_27_2","unstructured":"Yikang Li Tao Ma Yeqi Bai Nan Duan Sining Wei and Xiaogang Wang. 2019. Pastegan: A semi-parametric method to generate image from scene graph. In Proceedings of the Advances in Neural Information Processing Systems 32 (2019). 3948\u20133958."},{"key":"e_1_3_1_28_2","first-page":"335","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Li Yikang","year":"2018","unstructured":"Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. 2018. Factorizable net: An efficient subgraph-based framework for scene graph generation. In Proceedings of the European Conference on Computer Vision. Springer, 335\u2013351."},{"key":"e_1_3_1_29_2","first-page":"1261","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Li Yikang","year":"2017","unstructured":"Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. 2017. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 1261\u20131270."},{"key":"e_1_3_1_30_2","first-page":"13874","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Li Yiming","year":"2022","unstructured":"Yiming Li, Xiaoshan Yang, and Changsheng Xu. 2022. Dynamic scene graph generation via anticipatory pre-training. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 13874\u201313883."},{"key":"e_1_3_1_31_2","first-page":"2117","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Lin Tsung-Yi","year":"2017","unstructured":"Tsung-Yi Lin, Piotr Doll\u00e1r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2117\u20132125."},{"key":"e_1_3_1_32_2","first-page":"3746","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Lin Xin","year":"2020","unstructured":"Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. 2020. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3746\u20133753."},{"key":"e_1_3_1_33_2","first-page":"6985","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Liu Yong","year":"2018","unstructured":"Yong Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2018. Structure inference net: Object detection using scene-level context and instance-level relationships. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 6985\u20136994."},{"key":"e_1_3_1_34_2","first-page":"852","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Lu Cewu","year":"2016","unstructured":"Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In Proceedings of the European Conference on Computer Vision. Springer, 852\u2013869."},{"key":"e_1_3_1_35_2","first-page":"15931","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Lu Yichao","year":"2021","unstructured":"Yichao Lu, Himanshu Rai, Jason Chang, Boris Knyazev, Guangwei Yu, Shashank Shekhar, Graham W. Taylor, and Maksims Volkovs. 2021. Context-aware scene graph generation with seq2seq transformers. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 15931\u201315941."},{"key":"e_1_3_1_36_2","first-page":"19467","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Lyu Xinyu","year":"2022","unstructured":"Xinyu Lyu, Lianli Gao, Yuyu Guo, Zhou Zhao, Hao Huang, Heng Tao Shen, and Jingkuan Song. 2022. Fine-grained predicates learning for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 19467\u201319475."},{"key":"e_1_3_1_37_2","first-page":"2328","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Maheshwari Paridhi","year":"2021","unstructured":"Paridhi Maheshwari, Ritwick Chaudhry, and Vishwa Vinay. 2021. Scene graph embeddings using relative similarity supervision. In Proceedings of the AAAI Conference on Artificial Intelligence. 2328\u20132336."},{"key":"e_1_3_1_38_2","first-page":"2929","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Marszalek Marcin","year":"2009","unstructured":"Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. 2009. Actions in context. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2929\u20132936."},{"issue":"12","key":"e_1_3_1_39_2","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1016\/j.tics.2007.09.009","article-title":"The role of context in object recognition","volume":"11","author":"Oliva Aude","year":"2007","unstructured":"Aude Oliva and Antonio Torralba. 2007. The role of context in object recognition. Trends in Cognitive Sciences 11, 12 (2007), 520\u2013527.","journal-title":"Trends in Cognitive Sciences"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Pennington Jeffrey","year":"2014","unstructured":"Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1532\u20131543."},{"issue":"3","key":"e_1_3_1_41_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MWC.013.2200553","article-title":"A generalized semantic communication system: From sources to channels","volume":"30","author":"Qin Zhijin","year":"2023","unstructured":"Zhijin Qin, Feifei Gao, Bo Lin, Xiaoming Tao, Guangyi Liu, and Chengkang Pan. 2023. A generalized semantic communication system: From sources to channels. IEEE Wireless Communications 30, 3 (2023), 18\u201326.","journal-title":"IEEE Wireless Communications"},{"key":"e_1_3_1_42_2","unstructured":"Zhijin Qin Xiaoming Tao Jianhua Lu Wen Tong and Geoffrey Ye Li. 2021. Semantic communications: Principles and challenges. arXiv:2201.01389. Retrieved from https:\/\/arxiv.org\/abs\/2201.01389"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","unstructured":"Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 3982\u20133992.","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_3_1_44_2","first-page":"73","volume-title":"Proceedings of the IEEE International Symposium on Multimedia","author":"Ren Li","year":"2018","unstructured":"Li Ren and Kien Hua. 2018. Improved image description via embedded object structure graph and semantic feature matching. In Proceedings of the IEEE International Symposium on Multimedia. 73\u201380."},{"key":"e_1_3_1_45_2","unstructured":"Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems 28 (2015). 91\u201399."},{"key":"e_1_3_1_46_2","first-page":"178","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops","author":"Schroeder Brigit","year":"2020","unstructured":"Brigit Schroeder and Subarna Tripathi. 2020. Structured query-based image retrieval using scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops. 178\u2013179."},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","first-page":"70","DOI":"10.18653\/v1\/W15-2812","volume-title":"Proceedings of the 4th Workshop on Vision and Language","author":"Schuster Sebastian","year":"2015","unstructured":"Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the 4th Workshop on Vision and Language. 70\u201380."},{"key":"e_1_3_1_48_2","first-page":"13936","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Suhail Mohammed","year":"2021","unstructured":"Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. 2021. Energy-based learning for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 13936\u201313945."},{"key":"e_1_3_1_49_2","first-page":"6105","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Tan Mingxing","year":"2019","unstructured":"Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 6105\u20136114."},{"key":"e_1_3_1_50_2","unstructured":"Kaihua Tang. 2020. A Scene Graph Generation Codebase in PyTorch. (2020). Retrieved July 1 2022 from https:\/\/github.com\/KaihuaTang\/Scene-Graph-Benchmark.pytorch"},{"key":"e_1_3_1_51_2","first-page":"3716","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Tang Kaihua","year":"2020","unstructured":"Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased scene graph generation from biased training. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 3716\u20133725."},{"key":"e_1_3_1_52_2","first-page":"6619","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Tang Kaihua","year":"2019","unstructured":"Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. 2019. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 6619\u20136628."},{"key":"e_1_3_1_53_2","first-page":"13688","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Teng Yao","year":"2021","unstructured":"Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu. 2021. Target adaptive context aggregation for video scene graph generation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 13688\u201313697."},{"key":"e_1_3_1_54_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems 30 (2017). 6000\u20136010."},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Melissa Le-Hoa V\u00f5 Sage E. P. Boettcher and Dejan Draschkow. 2019. Reading scenes: How scene grammar guides attention and aids perception in real-world environments. Current Opinion in Psychology 29 (2019) 205\u2013210.","DOI":"10.1016\/j.copsyc.2019.03.009"},{"key":"e_1_3_1_56_2","first-page":"9695","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang Jiaqi","year":"2021","unstructured":"Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. 2021. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9695\u20139704."},{"issue":"10","key":"e_1_3_1_57_2","doi-asserted-by":"crossref","first-page":"2413","DOI":"10.1109\/TPAMI.2017.2754246","article-title":"Fvqa: Fact-based visual question answering","volume":"40","author":"Wang Peng","year":"2017","unstructured":"Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 10 (2017), 2413\u20132427.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_58_2","first-page":"1508","volume-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision","author":"Wang Sijin","year":"2020","unstructured":"Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. 2020. Cross-modal scene graph matching for relationship-aware image-text retrieval. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision. 1508\u20131517."},{"key":"e_1_3_1_59_2","first-page":"8188","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang Wenbin","year":"2019","unstructured":"Wenbin Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2019. Exploring context and visual pattern of relationship for scene graph generation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8188\u20138197."},{"issue":"9","key":"e_1_3_1_60_2","first-page":"R303\u2013R304","article-title":"Visual memory: What do you know about what you saw?","volume":"8","author":"Wolfe Jeremy M.","year":"1998","unstructured":"Jeremy M. Wolfe. 1998. Visual memory: What do you know about what you saw? Current Biology 8, 9 (1998), R303\u2013R304.","journal-title":"Current Biology"},{"key":"e_1_3_1_61_2","first-page":"3","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Woo Sanghyun","year":"2018","unstructured":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision. Springer, 3\u201319."},{"issue":"6","key":"e_1_3_1_62_2","first-page":"1367","article-title":"Image captioning and visual question answering based on attributes and external knowledge","volume":"40","author":"Wu Qi","year":"2017","unstructured":"Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, and Anton Van Den Hengel. 2017. Image captioning and visual question answering based on attributes and external knowledge. IEEE Transactions on Pattern analysis and Machine Intelligence 40, 6 (2017), 1367\u20131381.","journal-title":"IEEE Transactions on Pattern analysis and Machine Intelligence"},{"key":"e_1_3_1_63_2","first-page":"1492","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Xie Saining","year":"2017","unstructured":"Saining Xie, Ross Girshick, Piotr Doll\u00e1r, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1492\u20131500."},{"key":"e_1_3_1_64_2","first-page":"5410","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Xu Danfei","year":"2017","unstructured":"Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. 2017. Scene graph generation by iterative message passing. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5410\u20135419."},{"key":"e_1_3_1_65_2","first-page":"374","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Xu Li","year":"2022","unstructured":"Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu. 2022. Meta spatio-temporal debiasing for video scene graph generation. In Proceedings of the European Conference on Computer Vision. Springer, 374\u2013390."},{"key":"e_1_3_1_66_2","first-page":"265","volume-title":"Proceedings of the 28th ACM International Conference on Multimedia","author":"Yan Shaotian","year":"2020","unstructured":"Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian-Sheng Hua. 2020. Pcpl: Predicate-correlation perception learning for unbiased scene graph generation. In Proceedings of the 28th ACM International Conference on Multimedia. 265\u2013273."},{"key":"e_1_3_1_67_2","first-page":"670","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Yang Jianwei","year":"2018","unstructured":"Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Graph r-cnn for scene graph generation. In Proceedings of the European Conference on Computer Vision. Springer, 670\u2013685."},{"key":"e_1_3_1_68_2","first-page":"10685","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Yang Xu","year":"2019","unstructured":"Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10685\u201310694."},{"key":"e_1_3_1_69_2","first-page":"17","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Yao Bangpeng","year":"2010","unstructured":"Bangpeng Yao and Li Fei-Fei. 2010. Modeling mutual context of object and human pose in human-object interaction activities. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 17\u201324."},{"key":"e_1_3_1_70_2","first-page":"684","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Yao Ting","year":"2018","unstructured":"Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision. Springer, 684\u2013699."},{"key":"e_1_3_1_71_2","first-page":"10718","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Yoon Sangwoong","year":"2021","unstructured":"Sangwoong Yoon, Woo Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. 2021. Image-to-image retrieval by learning similarity between scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence. 10718\u201310726."},{"key":"e_1_3_1_72_2","doi-asserted-by":"crossref","unstructured":"Peter Young Alice Lai Micah Hodosh and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014) 67\u201378.","DOI":"10.1162\/tacl_a_00166"},{"key":"e_1_3_1_73_2","doi-asserted-by":"crossref","unstructured":"Jing Yu Yuan Chai Yujing Wang Yue Hu and Qi Wu. 2020. Cogtree: Cognition tree loss for unbiased scene graph generation. In Proceedings of the 29th International Joint Conference on Artificial Intelligence . 1274\u20131280.","DOI":"10.24963\/ijcai.2021\/176"},{"key":"e_1_3_1_74_2","first-page":"606","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Zareian Alireza","year":"2020","unstructured":"Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. 2020. Bridging knowledge graphs to generate scene graphs. In Proceedings of the European Conference on Computer Vision. Springer, 606\u2013623."},{"key":"e_1_3_1_75_2","first-page":"5831","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zellers Rowan","year":"2018","unstructured":"Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5831\u20135840."},{"key":"e_1_3_1_76_2","first-page":"5532","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhang Hanwang","year":"2017","unstructured":"Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. 2017. Visual translation embedding network for visual relation detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5532\u20135540."},{"key":"e_1_3_1_77_2","first-page":"11535","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhang Ji","year":"2019","unstructured":"Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. 2019. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11535\u201311543."},{"key":"e_1_3_1_78_2","first-page":"15465","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhang Xuying","year":"2021","unstructured":"Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2021. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 15465\u201315474."},{"issue":"1","key":"e_1_3_1_79_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3514041","article-title":"Boosting scene graph generation with visual relation saliency","volume":"19","author":"Zhang Yong","year":"2023","unstructured":"Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. 2023. Boosting scene graph generation with visual relation saliency. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 1 (2023), 1\u201317.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_1_80_2","first-page":"9259","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Zhao Qijie","year":"2019","unstructured":"Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. 2019. M2det: A single-shot object detector based on multi-level feature pyramid network. In Proceedings of the AAAI Conference on Artificial Intelligence. 9259\u20139266."},{"key":"e_1_3_1_81_2","first-page":"1823","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Zhong Yiwu","year":"2021","unstructured":"Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. 2021. Learning to generate scene graph from natural language supervision. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 1823\u20131834."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3615868","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3615868","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:10:18Z","timestamp":1750295418000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3615868"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,18]]},"references-count":80,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3615868"],"URL":"https:\/\/doi.org\/10.1145\/3615868","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2023,10,18]]},"assertion":[{"value":"2022-11-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}