{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:59:04Z","timestamp":1750309144319,"version":"3.41.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,2,28]],"date-time":"2024-02-28T00:00:00Z","timestamp":1709078400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62102187, 62372243"],"award-info":[{"award-number":["62102187, 62372243"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"crossref","award":["BK20210639"],"award-info":[{"award-number":["BK20210639"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>\n            <jats:italic>Object detection<\/jats:italic>\n            is a widely studied problem in existing works. However, in this paper, we turn to a more challenging problem of \u201c\n            <jats:italic>Covered Object Reasoning<\/jats:italic>\n            \u201d, aimed at reasoning the category label of target object in the given image particularly when it has been totally\n            <jats:italic>covered<\/jats:italic>\n            (or\n            <jats:italic>invisible<\/jats:italic>\n            ). To resolve this problem, we propose\n            <jats:italic>CoBjeason<\/jats:italic>\n            to seize the opportunity when visual reasoning meets the knowledge graph, where \u201c\n            <jats:italic>empirical cognition<\/jats:italic>\n            \u201d on common visual contexts have been incorporated as knowledge graph to conduct reinforced multi-hop reasoning via two collaborative agents. Such two agents, for one thing, stand at the covered object (or\n            <jats:italic>unknown entity<\/jats:italic>\n            ) to observe the surrounding visual cues in the given image and gradually select\n            <jats:italic>entities<\/jats:italic>\n            and\n            <jats:italic>relations<\/jats:italic>\n            from the global\n            <jats:italic>gallery-level<\/jats:italic>\n            knowledge graph which contains entity-pairs frequently occurring across the entire image-collection, so as to\n            <jats:italic>infer<\/jats:italic>\n            the main structure of image-level knowledge graph\n            <jats:italic>forward<\/jats:italic>\n            expanded from the\n            <jats:italic>unknown entity<\/jats:italic>\n            . In turn, for another, based on the\n            <jats:italic>reasoned<\/jats:italic>\n            image-level knowledge graph, the semantic context among\n            <jats:italic>entities<\/jats:italic>\n            will be aggregated backward into\n            <jats:italic>unknown entity<\/jats:italic>\n            to select an appropriate entity from the global\n            <jats:italic>gallery-level<\/jats:italic>\n            knowledge graph as the reasoning result. Moreover, such two agents will collaborate with each other, securing that the above\n            <jats:italic>Forward<\/jats:italic>\n            &amp;\n            <jats:italic>Backward Reasoning<\/jats:italic>\n            will step towards the same destination of the higher performance on covered object reasoning. To our best knowledge, this is the first work on\n            <jats:italic>Covered Object Reasoning<\/jats:italic>\n            with Knowledge Graphs and reinforced Multi-Agent collaboration. Particularly, our study on\n            <jats:italic>Covered Object Reasoning<\/jats:italic>\n            and the proposed model\n            <jats:italic>CoBjeason<\/jats:italic>\n            could offer novel insights into more basic Computer Vision (CV) tasks, such as\n            <jats:italic>Semantic Segmentation<\/jats:italic>\n            with better understanding on the current scene when some objects are blurred or covered,\n            <jats:italic>Visual Question Answering<\/jats:italic>\n            with enhancement on the inference in more complicated visual context when some objects are covered or invisible, and\n            <jats:italic>Image Caption Generation<\/jats:italic>\n            with the augmentation on the richness of visual context for images containing partially visible objects. The improvement on the above basic CV tasks can further refine more complicated ones involved with nuanced visual interpretation like Autonomous Driving, where the recognition and reasoning on partially visible or covered object are critical. According to the experimental results, our proposed\n            <jats:italic>CoBjeason<\/jats:italic>\n            can achieve the best overall ranking performance on covered object reasoning compared with other models, meanwhile enjoying the advantage of lower \u201c\n            <jats:italic>exploration cost<\/jats:italic>\n            \u201d, with the insensitivity against the long-tail covered objects and the acceptable time complexity.\n          <\/jats:p>","DOI":"10.1145\/3643565","type":"journal-article","created":{"date-parts":[[2024,1,26]],"date-time":"2024-01-26T10:03:15Z","timestamp":1706263395000},"page":"1-56","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["CoBjeason: Reasoning Covered Object in Image by Multi-Agent Collaboration Based on Informed Knowledge Graph"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0542-1827","authenticated-orcid":false,"given":"Huan","family":"Rong","sequence":"first","affiliation":[{"name":"Nanjing University of Information Science &amp; Technology, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8970-2649","authenticated-orcid":false,"given":"Minfeng","family":"Qian","sequence":"additional","affiliation":[{"name":"Nanjing University of Information Science &amp; Technology, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2320-1692","authenticated-orcid":false,"given":"Tinghuai","family":"Ma","sequence":"additional","affiliation":[{"name":"Nanjing University of Information Science &amp; Technology, Nanjing, China and Jiangsu Ocean University, Lianyungang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7445-9936","authenticated-orcid":false,"given":"Di","family":"Jin","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4960-174X","authenticated-orcid":false,"given":"Victor S.","family":"Sheng","sequence":"additional","affiliation":[{"name":"Texas Tech University, Lubbock, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,2,28]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-022-01109-2"},{"key":"e_1_3_2_3_2","article-title":"Translating embeddings for modeling multi-relational data","volume":"26","author":"Bordes Antoine","year":"2013","unstructured":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems 26 (2013).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561086"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6630"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2020.3025128"},{"key":"e_1_3_2_7_2","volume-title":"International Conference on Learning Representations","author":"Das Rajarshi","year":"2018","unstructured":"Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, and Andrew McCallum. 2018. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In International Conference on Learning Representations."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3082568"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00433-9"},{"key":"e_1_3_2_10_2","first-page":"1661","volume-title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence: Melbourne, Australia, August 19","volume":"25","author":"Fang Yuan","unstructured":"Yuan Fang, Kingsley Kuan, Jie Lin, Cheston Tan, and Vijay Chandrasekhar. 2017. Object detection meets knowledge graphs. (2017). In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence: Melbourne, Australia, August 19, (2017), Vol. 25. 1661\u20131667."},{"key":"e_1_3_2_11_2","first-page":"3061","volume-title":"International Conference on Machine Learning","author":"Fedus William","year":"2020","unstructured":"William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. 2020. Revisiting fundamentals of experience replay. In International Conference on Machine Learning. PMLR, 3061\u20133071."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1269"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33018303"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3070843"},{"key":"e_1_3_2_15_2","first-page":"5436","volume-title":"International Conference on Machine Learning","author":"Kristiadi Agustinus","year":"2020","unstructured":"Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. 2020. Being Bayesian, even just a bit, fixes overconfidence in ReLU networks. In International Conference on Machine Learning. PMLR, 5436\u20135446."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359554"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM51629.2021.00044"},{"issue":"3","key":"e_1_3_2_18_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3576921","article-title":"Preference-aware graph attention networks for cross-domain recommendations with collaborative knowledge graph","volume":"41","author":"Li Yakun","year":"2023","unstructured":"Yakun Li, Lei Hou, and Juanzi Li. 2023. Preference-aware graph attention networks for cross-domain recommendations with collaborative knowledge graph. ACM Transactions on Information Systems 41, 3 (2023), 1\u201326.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3444685.3446312"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2018.00135"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449986"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3502720"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511019"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.10"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.2981890"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6214"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-021-01145-6"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599839"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00900"},{"key":"e_1_3_2_30_2","first-page":"2071","volume-title":"International Conference on Machine Learning","author":"Trouillon Th\u00e9o","year":"2016","unstructured":"Th\u00e9o Trouillon, Johannes Welbl, Sebastian Riedel, \u00c9ric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning. PMLR, 2071\u20132080."},{"issue":"1","key":"e_1_3_2_31_2","first-page":"614","article-title":"Informed machine learning\u2013a taxonomy and survey of integrating prior knowledge into learning systems","volume":"35","author":"Von Rueden Laura","year":"2021","unstructured":"Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, and Jannis Schuecker. 2021. Informed machine learning\u2013a taxonomy and survey of integrating prior knowledge into learning systems. IEEE Transactions on Knowledge and Data Engineering 35, 1 (2021), 614\u2013633.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_3_2_32_2","first-page":"4393","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Wan Guojia","year":"2021","unstructured":"Guojia Wan and Bo Du. 2021. GaussianPath: A Bayesian multi-hop reasoning framework for knowledge graph reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4393\u20134401."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.6073"},{"key":"e_1_3_2_34_2","first-page":"1926","volume-title":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","author":"Wan Guojia","year":"2021","unstructured":"Guojia Wan, Shirui Pan, Chen Gong, Chuan Zhou, and Gholamreza Haffari. 2021. Reasoning like human: Hierarchical reinforcement learning for knowledge graph reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 1926\u20131932."},{"issue":"1","key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3533017","article-title":"Multi-concept representation learning for knowledge graph completion","volume":"17","author":"Wang Jiapu","year":"2023","unstructured":"Jiapu Wang, Boyue Wang, Junbin Gao, Yongli Hu, and Baocai Yin. 2023. Multi-concept representation learning for knowledge graph completion. ACM Transactions on Knowledge Discovery from Data 17, 1 (2023), 1\u201319.","journal-title":"ACM Transactions on Knowledge Discovery from Data"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475470"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512083"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3126648"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3006080"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3042943"},{"key":"e_1_3_2_41_2","first-page":"16628","article-title":"Finite-time analysis for double Q-learning","volume":"33","author":"Xiong Huaqing","year":"2020","unstructured":"Huaqing Xiong, Lin Zhao, Yingbin Liang, and Wei Zhang. 2020. Finite-time analysis for double Q-learning. Advances in Neural Information Processing Systems 33 (2020), 16628\u201316638.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00382"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00952"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00658"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2020.3011807"},{"key":"e_1_3_2_46_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR) 2015","author":"Yang Bishan","year":"2015","unstructured":"Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR) 2015."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446428"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i5.20538"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401171"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6999"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00867"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2023.3238524"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643565","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643565","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:28Z","timestamp":1750287028000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,28]]},"references-count":51,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3643565"],"URL":"https:\/\/doi.org\/10.1145\/3643565","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"type":"print","value":"1556-4681"},{"type":"electronic","value":"1556-472X"}],"subject":[],"published":{"date-parts":[[2024,2,28]]},"assertion":[{"value":"2023-07-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}