{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T01:56:23Z","timestamp":1769824583387,"version":"3.49.0"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2021,7,26]],"date-time":"2021-07-26T00:00:00Z","timestamp":1627257600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,26]],"date-time":"2021-07-26T00:00:00Z","timestamp":1627257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61872446"],"award-info":[{"award-number":["61872446"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["World Wide Web"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the proliferation of large-scale knowledge graphs (KGs), multi-hop knowledge graph reasoning has been a capstone that enables machines to be able to handle intelligent tasks, especially where some explicit reasoning path is appreciated for decision making. To train a KG reasoner, supervised learning-based methods suffer from false-negative issues, i.e., unseen paths during training are not to be found in prediction; in contrast, reinforcement learning (RL)-based methods do not require labeled paths, and can explore to cover many appropriate reasoning paths. In this connection, efforts have been dedicated to investigating several RL formulations for multi-hop KG reasoning. Particularly, current RL-based methods generate rewards at the very end of the reasoning process, due to which short paths of hops less than a given threshold are likely to be overlooked, and the overall performance is impaired. To address the problem, we propose , a revised RL formulation of multi-hop KG reasoning that is characterized by two novel designs\u2014the stop signal and the worth-trying signal. The stop signal instructs the agent of RL to stay at the entity after finding the answer, preventing from hopping further even if the threshold is not reached; meanwhile, the worth-trying signal encourages the agent to try to learn some partial patterns from the paths that fail to lead to the answer. To validate the design of our model , comprehensive experiments are carried out on three benchmark knowledge graphs, and the results and analysis suggest the superiority of  over state-of-the-art methods.<\/jats:p>","DOI":"10.1007\/s11280-021-00911-5","type":"journal-article","created":{"date-parts":[[2021,7,26]],"date-time":"2021-07-26T11:02:37Z","timestamp":1627297357000},"page":"1837-1856","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["To hop or not, that is the question: Towards effective multi-hop reasoning over knowledge graphs"],"prefix":"10.1007","volume":"24","author":[{"given":"Jinzhi","family":"Liao","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6339-0219","authenticated-orcid":false,"given":"Xiang","family":"Zhao","sequence":"additional","affiliation":[]},{"given":"Jiuyang","family":"Tang","sequence":"additional","affiliation":[]},{"given":"Weixin","family":"Zeng","sequence":"additional","affiliation":[]},{"given":"Zhen","family":"Tan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,7,26]]},"reference":[{"key":"911_CR1","doi-asserted-by":"crossref","unstructured":"Arras, L., Montavon, G., M\u00fcller, K., Samek, W.: Explaining recurrent neural network predictions in sentiment analysis. In: WASSA@EMNLP 2017, September 8 2017, pp 159\u2013168, Copenhagen, Denmark (2017)","DOI":"10.18653\/v1\/W17-5221"},{"key":"911_CR2","doi-asserted-by":"crossref","unstructured":"Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.G.: Dbpedia: A nucleus for a web of open data. In: ISWC 2007 + ASWC 2007, November 11-15, 2007, pp 722\u2013735, Busan, Korea (2007)","DOI":"10.1007\/978-3-540-76298-0_52"},{"key":"911_CR3","doi-asserted-by":"crossref","unstructured":"Balazevic, I., Allen, C., Hospedales, T.M.: Hypernetwork knowledge graph embeddings. In: ICANN 2019, September 17-19, 2019, pp 553\u2013565, Munich, Germany (2019)","DOI":"10.1007\/978-3-030-30493-5_52"},{"issue":"2","key":"911_CR4","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1007\/s10994-013-5363-6","volume":"94","author":"A Bordes","year":"2014","unstructured":"Bordes, A., Glorot, X., Weston, J., Bengio, Y.: A semantic matching energy function for learning with multi-relational data - application to word-sense disambiguation. Mach. Learn. 94(2), 233\u2013259 (2014)","journal-title":"Mach. Learn."},{"key":"911_CR5","unstructured":"Bordes, A., Usunier, N., Garc\u00eda-Dur\u00e1n, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: NeurIPS 2013, December 5-8, 2013, pp 2787\u20132795, Lake Tahoe, Nevada, United States (2013)"},{"key":"911_CR6","unstructured":"Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, Jr. E., Mitchell, T.M.: Toward an architecture for never-ending language learning. In: AAAI 2010, July 11\u201315, 2010, Atlanta, Georgia, USA (2010)"},{"key":"911_CR7","unstructured":"Cohen, W.W.: Tensorlog: A differentiable deductive database. arXiv:1605.06523 (2016)"},{"key":"911_CR8","unstructured":"Das, R., Dhuliawala, S., Zaheer, M., Vilnis, L., Durugkar, I., Krishnamurthy, A., Smola, A., McCallum, A.: Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: ICLR 2018, April 30 - May 3, 2018 Conference Track Proceedings, Vancouver, BC, Canada (2018)"},{"key":"911_CR9","unstructured":"Das, R., Neelakantan, A., Belanger, D., McCallum, A.: Chains of reasoning over entities, relations, and text using recurrent neural networks. In: EACL 2017, April 3-7, 2017, pp 132\u2013141, Valencia, Spain (2017)"},{"key":"911_CR10","unstructured":"Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2d knowledge graph embeddings. In: AAAI-18, February 2-7, 2018, pp 1811\u20131818, New Orleans, Louisiana, USA (2018)"},{"key":"911_CR11","doi-asserted-by":"crossref","unstructured":"Gardner, M., Talukdar, P.P., Krishnamurthy, J., Mitchell, T.M.: Incorporating vector space similarity in random walk inference over knowledge bases. In: EMNLP 2014, October 25-29, 2014, pp 397\u2013406, Doha, Qatar (2014)","DOI":"10.3115\/v1\/D14-1044"},{"key":"911_CR12","unstructured":"Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS 2010, May, 13-15, 2010, pp 249\u2013256, Chia Laguna Resort, Sardinia, Italy (2010)"},{"key":"911_CR13","doi-asserted-by":"crossref","unstructured":"Godin, F., Kumar, A., Mittal, A.: Learning when not to answer: a ternary reward structure for reinforcement learning based question answering. In: NAACL-HLT 2019, June, 2-7, 2019, pp 122\u2013129, Minneapolis, MN, USA (2019)","DOI":"10.18653\/v1\/N19-2016"},{"key":"911_CR14","doi-asserted-by":"crossref","unstructured":"Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., Yang, G.: XAI - explainable artificial intelligence. Sci. Robot. vol 4(37) (2019)","DOI":"10.1126\/scirobotics.aay7120"},{"issue":"8","key":"911_CR15","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735\u20131780 (1997)","journal-title":"Neural Comput."},{"key":"911_CR16","unstructured":"Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR 2015, May 7-9, 2015 Conference Track Proceedings, San Diego, CA, USA (2015)"},{"key":"911_CR17","doi-asserted-by":"crossref","unstructured":"Kok, S., Domingos, P.M.: Statistical predicate invention. In: ICML 2007, June, 20-24, 2007, pp 433\u2013440, Corvallis, Oregon, USA (2007)","DOI":"10.1145\/1273496.1273551"},{"issue":"1","key":"911_CR18","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1007\/s10994-010-5205-8","volume":"81","author":"N Lao","year":"2010","unstructured":"Lao, N., Cohen, W.W.: Relational retrieval using a combination of path-constrained random walks. Mach. Learn. 81(1), 53\u201367 (2010)","journal-title":"Mach. Learn."},{"key":"911_CR19","unstructured":"Lao, N., Mitchell, T.M., Cohen, W.W.: Random walk inference and learning in A large scale knowledge base. In: EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, pp 529\u2013539 (2011)"},{"issue":"2","key":"911_CR20","doi-asserted-by":"publisher","first-page":"167","DOI":"10.3233\/SW-140134","volume":"6","author":"J Lehmann","year":"2015","unstructured":"Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., Bizer, C.: Dbpedia - A large-scale, multilingual knowledge base extracted from wikipedia. Semant. Web 6(2), 167\u2013195 (2015)","journal-title":"Semant. Web"},{"key":"911_CR21","doi-asserted-by":"crossref","unstructured":"Lin, X.V., Socher, R., Xiong, C.: Multi-hop knowledge graph reasoning with reward shaping. In: EMNLP 2018, October 31 - November 4, 2018, pp 3243\u20133253, Brussels, Belgium (2018)","DOI":"10.18653\/v1\/D18-1362"},{"key":"911_CR22","doi-asserted-by":"crossref","unstructured":"Meilicke, C., Chekol, M.W., Ruffinelli, D., Stuckenschmidt, H.: Anytime bottom-up rule learning for knowledge graph completion. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, August 10-16, 2019, pp 3137\u20133143, Macao, China (2019)","DOI":"10.24963\/ijcai.2019\/435"},{"key":"911_CR23","unstructured":"Muggleton, S.: Developments in inductive logic programming, panel position paper. In: FGCS 1992, June 1-5, pp 1071\u20131073, Tokyo, Japan (1992)"},{"issue":"3&4","key":"911_CR24","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1007\/BF03037227","volume":"13","author":"S Muggleton","year":"1995","unstructured":"Muggleton, S.: Inverse entailment and progol. New Gener. Comput. 13(3&4), 245\u2013286 (1995)","journal-title":"New Gener. Comput."},{"key":"911_CR25","unstructured":"Murdoch, W.J., Szlam, A.: Automatic rule extraction from long short term memory networks. In: ICLR 2017, Toulon, France, April 24-26, 2017 Conference Track Proceedings (2017)"},{"key":"911_CR26","unstructured":"Qu, M., Tang, J.: Probabilistic logic neural networks for reasoning. In: NeurIPS 2019, 8-14 December 2019, pp 7710\u20137720, Vancouver, Canada (2019)"},{"key":"911_CR27","first-page":"239","volume":"5","author":"JR Quinlan","year":"1990","unstructured":"Quinlan, J.R.: Learning logical definitions from relations. Mach. Learn. 5, 239\u2013266 (1990)","journal-title":"Mach. Learn."},{"key":"911_CR28","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., Guestrin, C.: \u201cwhy should I trust you?\u201d: Explaining the predictions of any classifier. In: SIGKDD 2016, August, 13-17, 2016, pp 1135\u20131144, San Francisco, CA, USA (2016)","DOI":"10.18653\/v1\/N16-3020"},{"issue":"1-2","key":"911_CR29","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s10994-006-5833-1","volume":"62","author":"M Richardson","year":"2006","unstructured":"Richardson, M., Domingos, P.M.: Markov logic networks. Mach. Learn. 62(1-2), 107\u2013136 (2006)","journal-title":"Mach. Learn."},{"key":"911_CR30","unstructured":"Rieger, L., Singh, C., Murdoch, W.J., Yu, B.: Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. arXiv:1909.13584 (2019)"},{"key":"911_CR31","unstructured":"Rockt\u00e4schel, T., Riedel, S.: End-to-end differentiable proving. In: NeurIPS 2017, 4-9 December 2017, pp 3788\u20133800, Long Beach, CA, USA (2017)"},{"key":"911_CR32","unstructured":"Shen, Y., Chen, J., Huang, P., Guo, Y., Gao, J.: M-walk: Learning to walk over graphs using monte carlo tree search. In: NeurIPS 2018, 3-8 December 2018, pp 6787\u20136798, Montr\u00e9al, Canada (2018)"},{"key":"911_CR33","unstructured":"Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: WWW 2007, May 8-12, 2007, pp 697\u2013706, Banff, Alberta, Canada (2007)"},{"key":"911_CR34","unstructured":"Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: End-to-end memory networks. In: NeurIPS 2015, December 7-12, 2015, pp 2440\u20132448, Montreal, Quebec, Canada (2015)"},{"key":"911_CR35","unstructured":"Sun, Z., Deng, Z., Nie, J., Tang, J.: Rotate: Knowledge graph embedding by relational rotation in complex space. In: ICLR 2019, May 6\u20139, 2019, New Orleans, LA, USA (2019)"},{"key":"911_CR36","doi-asserted-by":"crossref","unstructured":"Toutanova, K., Chen, D., Pantel, P., Poon, H., Choudhury, P., Gamon, M.: Representing text for joint embedding of text and knowledge bases. In: EMNLP 2015, September 17-21, 2015, pp 1499\u20131509, Lisbon, Portugal (2015)","DOI":"10.18653\/v1\/D15-1174"},{"key":"911_CR37","unstructured":"Trouillon, T., Welbl, J., Riedel, S., Gaussier, \u00c9., Bouchard, G.: Complex embeddings for simple link prediction. In: ICML 2016, June 19-24, 2016, pp 2071\u20132080, New York City, NY, USA (2016)"},{"key":"911_CR38","unstructured":"Wang, W.Y., Mazaitis, K., Cohen, W.W.: Programming with personalized pagerank: a locally groundable first-order probabilistic logic. In: CIKM 2013, October 27 - November 1, 2013, pp 2129\u20132138, San Francisco, CA, USA (2013)"},{"key":"911_CR39","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1162\/tacl_a_00021","volume":"6","author":"J Welbl","year":"2018","unstructured":"Welbl, J., Stenetorp, P., Riedel, S.: Constructing datasets for multi-hop reading comprehension across documents. Trans. Assoc. Comput. Linguist. 6, 287\u2013302 (2018)","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"911_CR40","first-page":"229","volume":"8","author":"RJ Williams","year":"1992","unstructured":"Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229\u2013256 (1992)","journal-title":"Mach. Learn."},{"key":"911_CR41","doi-asserted-by":"crossref","unstructured":"Xiong, W., Hoang, T., Wang, W.Y.: Deeppath: A reinforcement learning method for knowledge graph reasoning. In: EMNLP 2017, September 9-11, 2017, pp 564\u2013573, Copenhagen, Denmark (2017)","DOI":"10.18653\/v1\/D17-1060"},{"key":"911_CR42","unstructured":"Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: ICML 2015, July 6-11, 2015, pp 2048\u20132057, Lille France (2015)"},{"key":"911_CR43","unstructured":"Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. In: ICLR 2015, May 7-9, 2015 Conference Track Proceedings, San Diego, CA, USA (2015)"},{"key":"911_CR44","unstructured":"Yang, F., Yang, Z., Cohen, W.W.: Differentiable learning of logical rules for knowledge base reasoning. In: NeurIPS 2017, December 4-9, 2017, pp 2319\u20132328, Long Beach, CA, USA (2017)"},{"key":"911_CR45","unstructured":"Zhang, S., Tay, Y., Yao, L., Liu, Q.: Quaternion knowledge graph embeddings. In: NeurIPS 2019, December 8-14, 2019, pp 2731\u20132741, Vancouver, BC, Canada (2019)"}],"container-title":["World Wide Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11280-021-00911-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11280-021-00911-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11280-021-00911-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T20:57:56Z","timestamp":1634677076000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11280-021-00911-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,26]]},"references-count":45,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["911"],"URL":"https:\/\/doi.org\/10.1007\/s11280-021-00911-5","relation":{},"ISSN":["1386-145X","1573-1413"],"issn-type":[{"value":"1386-145X","type":"print"},{"value":"1573-1413","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,26]]},"assertion":[{"value":"21 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 March 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 June 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}