{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T03:12:29Z","timestamp":1774926749193,"version":"3.50.1"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T00:00:00Z","timestamp":1740441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62106223"],"award-info":[{"award-number":["62106223"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Central Universities","award":["226-2024-00048"],"award-info":[{"award-number":["226-2024-00048"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>\n            Recommender systems (RSs) are designed to provide personalized recommendations to users. Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). We propose an evaluation framework\n            <jats:sc>KG4RecEval<\/jats:sc>\n            to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric\n            <jats:italic>KG utilization efficiency in recommendation<\/jats:italic>\n            (KGER). We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. The code and supplementary material of this article are available at:\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/HotBento\/KG4RecEval\">https:\/\/github.com\/HotBento\/KG4RecEval<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3713071","type":"journal-article","created":{"date-parts":[[2025,1,21]],"date-time":"2025-01-21T10:03:35Z","timestamp":1737453815000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["<scp>KG4RecEval<\/scp>\n            : Does Knowledge Graph Really Matter for Recommender Systems?"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-9782-4121","authenticated-orcid":false,"given":"Haonan","family":"Zhang","sequence":"first","affiliation":[{"name":"College of Control Science and Engineering, Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9812-3911","authenticated-orcid":false,"given":"Dongxia","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Control Science and Engineering, Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3350-7022","authenticated-orcid":false,"given":"Zhu","family":"Sun","sequence":"additional","affiliation":[{"name":"Singapore University of Technology and Design, Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2180-9157","authenticated-orcid":false,"given":"Yanhui","family":"Li","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1893-6259","authenticated-orcid":false,"given":"Youcheng","family":"Sun","sequence":"additional","affiliation":[{"name":"The University of Manchester, Manchester, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4408-4528","authenticated-orcid":false,"given":"Huizhi","family":"Liang","sequence":"additional","affiliation":[{"name":"Newcastle University, Newcastle upon Tyne, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1936-2840","authenticated-orcid":false,"given":"Wenhai","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Control Science and Engineering, Zhejiang University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,25]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.99"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.3390\/a11090137"},{"key":"e_1_3_2_4_2","first-page":"115","article-title":"Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures","author":"Bergstra James","year":"2013","unstructured":"James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the International Conference on Machine Learning. PMLR, 115\u2013123.","journal-title":"Proceedings of the International Conference on Machine Learning."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2011.07.021"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2010.03.009"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021240730564"},{"key":"e_1_3_2_8_2","volume-title":"Proceedings of the 5th ACM Conference on Recommender Systems (RecSys \u201911)","author":"Cantador Iv\u00e1n","year":"2011","unstructured":"Iv\u00e1n Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2011. 2nd workshop on information heterogeneity and fusion in recommender systems (HetRec 2011). In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys \u201911). ACM, New York, NY. Retrieved from https:\/\/grouplens.org\/datasets\/hetrec-2011\/"},{"key":"e_1_3_2_9_2","first-page":"151","article-title":"Unifying knowledge graph learning and recommendation: Towards a better understanding of user preferences","author":"Cao Yixin","year":"2019","unstructured":"Yixin Cao, Xiang Wang, Xiangnan He, Zikun Hu, and Tat-Seng Chua. 2019. Unifying knowledge graph learning and recommendation: Towards a better understanding of user preferences. In Proceedings of the world Wide Web Conference, 151\u2013161.","journal-title":"Proceedings of the world Wide Web Conference"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2009.12.040"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2012.10.037"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498419"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963776"},{"key":"e_1_3_2_14_2","first-page":"1390","volume-title":"Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Du Yuntao","year":"2022","unstructured":"Yuntao Du, Xinjun Zhu, Lu Chen, Baihua Zheng, and Yunjun Gao. 2022. HAKG: Hierarchy-aware knowledge gated network for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1390\u20131400."},{"key":"e_1_3_2_15_2","first-page":"1","article-title":"Towards a definition of knowledge graphs","volume":"48","author":"Ehrlinger Lisa","year":"2016","unstructured":"Lisa Ehrlinger and Wolfram W\u00f6\u00df. 2016. Towards a definition of knowledge graphs. SEMANTiCS (Posters, Demos, SuCCESS) 48, 1\u20134 (2016), 2.","journal-title":"SEMANTiCS (Posters, Demos, SuCCESS)"},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1007\/978-3-031-06981-9_22","volume-title":"Proceedings of the European Semantic Web Conference","author":"Ettorre Antonia","year":"2022","unstructured":"Antonia Ettorre, Anna Bobasheva, Franck Michel, and Catherine Faron. 2022. Stunning doodle: A tool for joint visualization and analysis of knowledge graphs and graph embeddings. In Proceedings of the European Semantic Web Conference. Springer, 370\u2013386."},{"key":"e_1_3_2_17_2","unstructured":"Lukas Faber Yifan Lu and Roger Wattenhofer. 2021. Should graph neural networks use features edges or both? arXiv:2103.06857. Retrieved from https:\/\/arxiv.org\/abs\/2103.06857"},{"key":"e_1_3_2_18_2","unstructured":"Yang Gao Yi-Fan Li Yu Lin Hang Gao and Latifur Khan. 2020. Deep learning on knowledge graph for recommender system: A survey. arXiv:2004.00387. Retrieved from https:\/\/arxiv.org\/abs\/2004.00387"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.3028705"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"F. Maxwell Harper and Joseph A. Konstan. 2015. The Movielens Datasets: History and Context. Retrieved from https:\/\/grouplens.org\/datasets\/movielens\/","DOI":"10.1145\/2827872"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219965"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"issue":"3","key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"274","DOI":"10.3991\/ijet.v16i03.18851","article-title":"A review of content-based and context-based recommendation systems","volume":"16","author":"Javed Umair","year":"2021","unstructured":"Umair Javed, Kamran Shaukat, Ibrahim A. Hameed, Farhat Iqbal, Talha Mahboob Alam, and Suhuai Luo. 2021. A review of content-based and context-based recommendation systems. International Journal of Emerging Technologies in Learning 16, 3 (2021), 274\u2013306.","journal-title":"International Journal of Emerging Technologies in Learning"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3616855.3635850"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401944"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557338"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.108050"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09231-w"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467350"},{"key":"e_1_3_2_30_2","first-page":"1210","article-title":"Jointly learning explainable rules for recommendation with knowledge graph","author":"Ma Weizhi","year":"2019","unstructured":"Weizhi Ma, Min Zhang, Yue Cao, Woojeong Jin, Chenyang Wang, Yiqun Liu, Shaoping Ma, and Xiang Ren. 2019. Jointly learning explainable rules for recommendation with knowledge graph. In Proceedings of the World Wide Web Conference, 1210\u20131221.","journal-title":"Proceedings of the World Wide Web Conference"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2019.06.019"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1018"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2009.10.005"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3502192"},{"key":"e_1_3_2_35_2","first-page":"4303","volume-title":"Proceedings of the IEEE International Conference on Big Data (Big Data \u201922).","author":"Pugazhenthi Thamizhiniyan","year":"2022","unstructured":"Thamizhiniyan Pugazhenthi and Huizhi Liang. 2022. Improving conversational recommender systems via knowledge graph-based semantic fusion with historical interaction data. In Proceedings of the IEEE International Conference on Big Data (Big Data \u201922). IEEE, 4303\u20134312."},{"key":"e_1_3_2_36_2","unstructured":"Steffen Rendle Christoph Freudenthaler Zeno Gantner and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv:1205.2618. Retrieved from https:\/\/arxiv.org\/abs\/1205.2618"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517887"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-022-00592-5"},{"key":"e_1_3_2_39_2","article-title":"A survey of collaborative filtering techniques","author":"Su Xiaoyuan","year":"2009","unstructured":"Xiaoyuan Su and Taghi M. Khoshgoftaar. 2009. A survey of collaborative filtering techniques. In Proceedings of the Advances in Artificial Intelligence, Vol. 2009.","journal-title":"Proceedings of the Advances in Artificial Intelligence,"},{"key":"e_1_3_2_40_2","first-page":"77","article-title":"The TREC-8 question answering track report","volume":"99","author":"Voorhees Ellen M.","year":"1999","unstructured":"Ellen M. Voorhees. 1999. The TREC-8 question answering track report. In Proceedings of the Text REtrieval Conference (TREC \u201999), Vol. 99, 77\u201382.","journal-title":"Proceedings of the Text REtrieval Conference (TREC \u201999)"},{"key":"e_1_3_2_41_2","first-page":"6127","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"34","author":"Wang Chao","year":"2020","unstructured":"Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, and Hui Xiong. 2020. Setrank: A setwise Bayesian approach for collaborative ranking from implicit feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 6127\u20136136."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3570483"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271739"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330836"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"3307","DOI":"10.1145\/3308558.3313417","article-title":"Knowledge graph convolutional networks for recommender systems","author":"Wang Hongwei","year":"2019","unstructured":"Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. 2019. Knowledge graph convolutional networks for recommender systems. In Proceedings of the World Wide Web Conference, 3307\u20133313.","journal-title":"Proceedings of the World Wide Web Conference"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330989"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450133"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482092"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-021-06573-8"},{"key":"e_1_3_2_50_2","unstructured":"Lanling Xu Zhen Tian Gaowei Zhang Lei Wang Junjie Zhang Bowen Zheng Yifan Li Yupeng Hou Xingyu Pan Yushuo Chen et\u00a0al. 2022. Recent advances in RecBole: Extensions with more practical considerations. arXiv: 2211.15148. Retrieved from https:\/\/arxiv.org\/abs\/2211.15148"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599400"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532009"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507230"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939673"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","unstructured":"Hengtong Zhang Tianhang Zheng Jing Gao Chenglin Miao Lu Su Yaliang Li and Kui Ren. 2019. Data poisoning attack against knowledge graph embedding. arXiv:1904.12052. Retrieved from https:\/\/arxiv.org\/abs\/1904.12052","DOI":"10.24963\/ijcai.2019\/674"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2020.2971942"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1162\/dint_a_00008"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557680"},{"key":"e_1_3_2_59_2","first-page":"4653","article-title":"RecBole: Towards a unified, comprehensive and efficient framework for recommendation algorithms","author":"Zhao Wayne Xin","year":"2021","unstructured":"Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et\u00a0al. 2021. RecBole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM \u201921). ACM, 4653\u20134664.","journal-title":"Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM \u201921)."},{"key":"e_1_3_2_60_2","unstructured":"Cai-Nicolas Ziegler. 2004. Book-Crossing Dataset. Retrieved from http:\/\/www2.informatik.uni-freiburg.de\/cziegler\/BX\/"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3713071","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3713071","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:09Z","timestamp":1750295889000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3713071"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,25]]},"references-count":59,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3713071"],"URL":"https:\/\/doi.org\/10.1145\/3713071","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,25]]},"assertion":[{"value":"2024-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}