{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T07:41:06Z","timestamp":1765438866472,"version":"3.37.3"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>User preference information plays an important role in knowledge graph-based recommender systems, which is reflected in users having different preferences for each entity\u2013relation pair in the knowledge graph. Existing approaches have not modeled this fine-grained user preference feature well, as affecting the performance of recommender systems. In this paper, we propose a deep knowledge preference-aware reinforcement learning network (KPRLN) for the recommendation, which builds paths between user\u2019s historical interaction items in the knowledge graph, learns the preference features of each user\u2013entity\u2013relation and generates the weighted knowledge graph with fine-grained preference features. First, we proposed a hierarchical propagation path construction method to address the problems of the pendant entity and long path exploration in the knowledge graph. The method expands outward to form clusters centered on items and uses them to represent the starting and target states in reinforcement learning. With the iteration of clusters, we can better learn the pendant entity preference and explore farther paths. Besides, we design an attention graph convolutional network, which focuses on more influential entity\u2013relation pairs, to aggregate user and item higher order representations that contain fine-grained preference features. Finally, extensive experiments on two real-world datasets demonstrate that our method outperforms other state-of-the-art baselines.<\/jats:p>","DOI":"10.1007\/s40747-023-01083-7","type":"journal-article","created":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T07:01:49Z","timestamp":1685084509000},"page":"6645-6659","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["KPRLN: deep knowledge preference-aware reinforcement learning network for recommendation"],"prefix":"10.1007","volume":"9","author":[{"given":"Di","family":"Wu","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0435-8854","authenticated-orcid":false,"given":"Mingjing","family":"Tang","sequence":"additional","affiliation":[]},{"given":"Shu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Ao","family":"You","sequence":"additional","affiliation":[]},{"given":"Wei","family":"Gao","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,5,26]]},"reference":[{"key":"1083_CR1","doi-asserted-by":"crossref","unstructured":"Wang S, Hu L, Wang Y, He X, Sheng QZ, Orgun MA, Cao L, Ricci F, Yu PS (2021) Graph learning based recommender systems: a review. arXiv:2105.06339","DOI":"10.24963\/ijcai.2021\/630"},{"key":"1083_CR2","doi-asserted-by":"publisher","first-page":"2724","DOI":"10.1109\/TKDE.2017.2754499","volume":"29","author":"Q Wang","year":"2017","unstructured":"Wang Q, Mao Z, Wang B, Guo L (2017) Knowledge graph embedding: a survey of approaches and applications. IEEE Trans Knowl Data Eng 29:2724\u20132743","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"1083_CR3","doi-asserted-by":"crossref","unstructured":"Liu J, Duan L (2021) A survey on knowledge graph-based recommender systems. In: 2021 IEEE 5th advanced information technology, electronic and automation control conference (IAEAC), Chongqing, vol 5. pp 2450\u20132453","DOI":"10.1109\/IAEAC50856.2021.9390863"},{"key":"1083_CR4","doi-asserted-by":"crossref","unstructured":"Wang X, Wang D, Xu C, He X, Cao Y, Chua T-S (2019) Explainable reasoning over knowledge graphs for recommendation. In: Proceedings of the AAAI conference on artificial intelligence, Hawaii, vol 33. pp 5329\u20135336","DOI":"10.1609\/aaai.v33i01.33015329"},{"key":"1083_CR5","unstructured":"Veli\u010dkovi\u0107 P, Cucurull G, Casanova A, Romero A, Li\u00f2 P, Bengio Y (2018) Graph attention networks. In: International conference on learning representations. https:\/\/arxiv.org\/abs\/1710.10903"},{"key":"1083_CR6","doi-asserted-by":"publisher","unstructured":"Wang X, He X, Cao Y, Liu M, Chua T-S (2019) Kgat: knowledge graph attention network for recommendation. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining, Anchorage. pp 950\u2013958. https:\/\/doi.org\/10.1145\/3292500.3330989","DOI":"10.1145\/3292500.3330989"},{"key":"1083_CR7","doi-asserted-by":"publisher","unstructured":"Wang H, Zhao M, Xie X, Li W, Guo M (2019) Knowledge graph convolutional networks for recommender systems. In: The world wide web conference, San Francisco. pp 3307\u20133313. https:\/\/doi.org\/10.1145\/3308558.3313417","DOI":"10.1145\/3308558.3313417"},{"key":"1083_CR8","doi-asserted-by":"crossref","unstructured":"Wang H, Zhang F, Zhang M, Leskovec J, Zhao M, Li W, Wang Z (2019) Knowledge-aware graph neural networks with label smoothness regularization for recommender systems. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining, Anchorage. pp 68\u2013977","DOI":"10.1145\/3292500.3330836"},{"key":"1083_CR9","unstructured":"Huai Z, Tao J, Che F, Yang G, Zhang D (2021) Knowledge graph enhanced recommender system. arXiv preprint arXiv:2112.09425"},{"key":"1083_CR10","doi-asserted-by":"crossref","unstructured":"Hui B, Zhang L, Zhou X, Wen X, Nian Y (2022) Personalized recommendation system based on knowledge embedding and historical behavior. Appl Intell 52:954\u2013966","DOI":"10.1007\/s10489-021-02363-w"},{"key":"1083_CR11","doi-asserted-by":"publisher","DOI":"10.1145\/3543846","author":"MM Afsar","year":"2022","unstructured":"Afsar MM, Crump T, Far B (2022) Reinforcement learning based recommender systems: a survey. ACM Comput Surv. https:\/\/doi.org\/10.1145\/3543846","journal-title":"ACM Comput Surv"},{"key":"1083_CR12","unstructured":"Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. Adv Neural Inf Process Syst 2:2787\u20132795. https:\/\/proceedings.neurips.cc\/paper\/2013\/file\/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf"},{"key":"1083_CR13","doi-asserted-by":"crossref","unstructured":"Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI conference on artificial intelligence, Qu\u00e9bec City, vol 28","DOI":"10.1609\/aaai.v28i1.8870"},{"key":"1083_CR14","unstructured":"Balazevic I, Allen C, Hospedales T (2019) Multi-relational poincar\u00e9 graph embeddings. Adv Neural Inf Process Syst 4460\u20134470"},{"key":"1083_CR15","doi-asserted-by":"crossref","unstructured":"Xu C, Li R (2019) Relation embedding with dihedral group in knowledge graph. In: Proceedings of the 57th annual meeting of the association for computational linguistics, Florence. pp 263\u2013272","DOI":"10.18653\/v1\/P19-1026"},{"key":"1083_CR16","doi-asserted-by":"crossref","unstructured":"Zhang F, Yuan NJ, Lian D, Xie X, Ma W-Y (2016) Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. KDD\u201916, San Francisco. pp 353\u2013362","DOI":"10.1145\/2939672.2939673"},{"key":"1083_CR17","doi-asserted-by":"crossref","unstructured":"Wang H, Zhang F, Xie X, Guo M (2018) Dkn: Deep knowledge aware network for news recommendation. In: Proceedings of the 2018 World Wide Web Conference. WWW \u201918, Lyon. pp 1835\u20131844","DOI":"10.1145\/3178876.3186175"},{"key":"1083_CR18","doi-asserted-by":"crossref","unstructured":"Yu X, Ren X, Sun Y, Gu Q, Sturt B, Khandelwal U, Norick B, Han J (2014) Personalized entity recommendation: A heterogeneous information network approach. In: Proceedings of the 7th ACM international conference on web search and data mining, New York. pp 283\u2013292","DOI":"10.1145\/2556195.2556259"},{"key":"1083_CR19","doi-asserted-by":"crossref","unstructured":"Zhao H, Yao Q, Li J, Song Y, Lee DL (2017) Meta-graph based recommendation fusion over heterogeneous information networks. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, Halifax. pp 635\u2013644","DOI":"10.1145\/3097983.3098063"},{"key":"1083_CR20","doi-asserted-by":"crossref","unstructured":"Wang H, Zhang F, Wang J, Zhao M, Li W, Xie X, Guo M (2018) Ripplenet: propagating user preferences on the knowledge graph for recommender systems. In: Proceedings of the 27th ACM international conference on information and knowledge management, Torino. pp 417\u2013426","DOI":"10.1145\/3269206.3271739"},{"key":"1083_CR21","doi-asserted-by":"crossref","unstructured":"Wang X, Wang S, Liang X, Zhao D, Huang J, Xu X, Dai B, Miao Q (2022) Deep reinforcement learning: a survey. IEEE Trans Neural Netw Learn Syst 1\u201315","DOI":"10.1109\/TNNLS.2022.3207346"},{"issue":"7540","key":"1083_CR22","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529\u2013533. https:\/\/doi.org\/10.1038\/nature14236","journal-title":"Nature"},{"issue":"7587","key":"1083_CR23","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484\u2013489. https:\/\/doi.org\/10.1038\/nature16961","journal-title":"Nature"},{"key":"1083_CR24","doi-asserted-by":"crossref","unstructured":"Codevilla F, M\u00fcller M, L\u00f3pez A, Koltun V, Dosovitskiy A (2018) End-to-end driving via conditional imitation learning. In: 2018 IEEE international conference on robotics and automation (ICRA), Brisbane. pp 4693\u20134700","DOI":"10.1109\/ICRA.2018.8460487"},{"key":"1083_CR25","first-page":"1265","volume":"6","author":"G Shani","year":"2005","unstructured":"Shani G, Heckerman D, Brafman RI (2005) An mdp-based recommender system. J Mach Learn Res 6:1265-1295","journal-title":"J Mach Learn Res"},{"key":"1083_CR26","doi-asserted-by":"crossref","unstructured":"Hu B, Shi C, Liu J (2017) Playlist recommendation based on reinforcement learning. In: Intelligence Science I: Second IFIP TC 12 International Conference (ICIS), Shanghai. pp 172\u2013182","DOI":"10.1007\/978-3-319-68121-4_18"},{"key":"1083_CR27","doi-asserted-by":"crossref","unstructured":"Zheng G, Zhang F, Zheng Z, Xiang Y, Yuan NJ, Xie X, Li Z (2018) Drn: a deep reinforcement learning framework for news recommendation. In: Proceedings of the 2018 world wide web conference, Lyon. pp 167\u2013176","DOI":"10.1145\/3178876.3185994"},{"key":"1083_CR28","doi-asserted-by":"crossref","unstructured":"Zhao X, Xia L, Zhang L, Ding Z, Yin D, Tang J (2018) Deep reinforcement learning for page-wise recommendations. In: Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver. pp 95\u2013103","DOI":"10.1145\/3240323.3240374"},{"issue":"6","key":"1083_CR29","doi-asserted-by":"publisher","first-page":"1203","DOI":"10.1016\/j.ipm.2018.04.008","volume":"54","author":"M Karimi","year":"2018","unstructured":"Karimi M, Jannach D, Jugovac M (2018) News recommender systems: survey and roads ahead. Inf Process Manag 54(6):1203\u20131227. https:\/\/doi.org\/10.1016\/j.ipm.2018.04.008","journal-title":"Inf Process Manag"},{"key":"1083_CR30","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106421","volume":"209","author":"Q Wang","year":"2020","unstructured":"Wang Q, Ji Y, Hao Y, Cao J (2020) Grl: knowledge graph completion with gan-based reinforcement learning. Knowl Based Syst 209:106421","journal-title":"Knowl Based Syst"},{"key":"1083_CR31","doi-asserted-by":"crossref","unstructured":"Xiong W, Hoang T, Wang WY (2017) Deeppath: a reinforcement learning method for knowledge graph reasoning. arXiv preprint arXiv:1707.06690","DOI":"10.18653\/v1\/D17-1060"},{"key":"1083_CR32","unstructured":"Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2018) Go for a walk and arrive at the answer: reasoning over paths in knowledge bases using reinforcement learning. In: International conference on learning representations. https:\/\/arxiv.org\/abs\/1711.05851"},{"key":"1083_CR33","doi-asserted-by":"crossref","unstructured":"Lin XV, Socher R, Xiong C (2018) Multi-hop knowledge graph reasoning with reward shaping. In: EMNLP. https:\/\/arxiv.org\/abs\/1711.05851","DOI":"10.18653\/v1\/D18-1362"},{"key":"1083_CR34","doi-asserted-by":"crossref","unstructured":"Xian Y, Fu Z, Muthukrishnan S, De\u00a0Melo G, Zhang Y (2019) Reinforcement knowledge graph reasoning for explainable recommendation. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, Paris. pp 285\u2013294","DOI":"10.1145\/3331184.3331203"},{"key":"1083_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.107217","volume":"227","author":"S Tao","year":"2021","unstructured":"Tao S, Qiu R, Ping Y, Ma H (2021) Multi-modal knowledge-aware reinforcement learning network for explainable recommendation. Knowl Based Syst 227:107217","journal-title":"Knowl Based Syst"},{"key":"1083_CR36","doi-asserted-by":"crossref","unstructured":"Grover A, Leskovec J (2016) node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco. pp 855\u2013864","DOI":"10.1145\/2939672.2939754"},{"key":"1083_CR37","doi-asserted-by":"crossref","unstructured":"Van\u00a0Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, Phoenix, vol 30","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"1083_CR38","unstructured":"Hamilton W, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. Adv Neural Inf Process Syst 1025\u20131035"},{"key":"1083_CR39","doi-asserted-by":"crossref","unstructured":"Wang H, Zhang F, Zhao M, Li W, Xie X, Guo M (2019) Multi-task feature learning for knowledge graph enhanced recommendation. In: The world wide web conference, WWW \u201919. San Francisco. pp 2000\u20132010","DOI":"10.1145\/3308558.3313411"},{"issue":"3","key":"1083_CR40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2168752.2168771","volume":"3","author":"S Rendle","year":"2012","unstructured":"Rendle S (2012) Factorization machines with libfm. ACM Trans Intell Syst Technol (TIST) 3(3):1\u201322. https:\/\/doi.org\/10.1145\/2168752.2168771","journal-title":"ACM Trans Intell Syst Technol (TIST)"},{"key":"1083_CR41","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106194","volume":"204","author":"Z Yang","year":"2020","unstructured":"Yang Z, Dong S (2020) Hagerec: hierarchical attention graph convolutional network incorporating knowledge graph for explainable recommendation. Knowl Based Syst 204:106194. https:\/\/doi.org\/10.1016\/j.knosys.2020.106194","journal-title":"Knowl Based Syst"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01083-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-01083-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01083-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T19:20:18Z","timestamp":1698434418000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-01083-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":41,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["1083"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-01083-7","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2023,5,26]]},"assertion":[{"value":"25 October 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 April 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 May 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}