{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,19]],"date-time":"2026-04-19T08:45:07Z","timestamp":1776588307396,"version":"3.51.2"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T00:00:00Z","timestamp":1719878400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T00:00:00Z","timestamp":1719878400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Shenyang Science and Technology Plan Project","award":["RC210469"],"award-info":[{"award-number":["RC210469"]}]},{"name":"Liaoning Provincial Science and Technology Innovation Project in the Field of Artificial Intelligence","award":["2023JH26\/10100005"],"award-info":[{"award-number":["2023JH26\/10100005"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Large language models have achieved outstanding performance on various downstream tasks with their advanced understanding of natural language and zero-shot capability. However, they struggle with knowledge constraints, particularly in tasks requiring complex reasoning or extended logical sequences. These limitations can affect their performance in question answering by leading to inaccuracies and hallucinations. This paper proposes a novel framework called KnowledgeNavigator that leverages large language models on knowledge graphs to achieve accurate and interpretable multi-hop reasoning. Especially with an analysis-retrieval-reasoning process, KnowledgeNavigator searches the optimal path iteratively to retrieve external knowledge and guide the reasoning to reliable answers. KnowledgeNavigator treats knowledge graphs and large language models as flexible components that can be switched between different tasks without additional costs. Experiments on three benchmarks demonstrate that KnowledgeNavigator significantly improves the performance of large language models in question answering and outperforms all large language models-based baselines.<\/jats:p>","DOI":"10.1007\/s40747-024-01527-8","type":"journal-article","created":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T15:21:21Z","timestamp":1719933681000},"page":"7063-7076","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["KnowledgeNavigator: leveraging large language models for enhanced reasoning over knowledge graph"],"prefix":"10.1007","volume":"10","author":[{"given":"Tiezheng","family":"Guo","sequence":"first","affiliation":[]},{"given":"Qingwen","family":"Yang","sequence":"additional","affiliation":[]},{"given":"Chen","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Yanyi","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Pan","family":"Li","sequence":"additional","affiliation":[]},{"given":"Jiawei","family":"Tang","sequence":"additional","affiliation":[]},{"given":"Dapeng","family":"Li","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6659-1785","authenticated-orcid":false,"given":"Yingyou","family":"Wen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,7,2]]},"reference":[{"key":"1527_CR1","unstructured":"Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M-A, Lacroix T, Rozi\u00e8re B, Goyal N, Hambro E, Azhar F et al (2023) Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Accessed 09 June 2023"},{"key":"1527_CR2","unstructured":"Anil R, Dai AM, Firat O, Johnson M, Lepikhin D, Passos A, Shakeri S, Taropa E, Bailey P, Chen Z et al (2023) Palm 2 technical report. arXiv preprint arXiv:2305.10403. Accessed 03 July 2023"},{"key":"1527_CR3","unstructured":"Bai J, Bai S, Chu Y, Cui Z, Dang K, Deng X, Fan Y, Ge W, Han Y, Huang F et al (2023) Qwen technical report. arXiv preprint arXiv:2309.16609. Accessed 07 Dec 2023"},{"key":"1527_CR4","unstructured":"Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, Huang X, Zhao E, Zhang Y, Chen Y et al (2023) Siren\u2019s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219. Accessed 08 Aug 2023"},{"key":"1527_CR5","first-page":"182","volume-title":"European Semantic Web Conference","author":"A Martino","year":"2023","unstructured":"Martino A, Iannelli M, Truong C (2023) Knowledge injection to counter large language model (llm) hallucination. European Semantic Web Conference. Springer, New York, pp 182\u2013185"},{"key":"1527_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2023.111165","volume":"151","author":"W Chen","year":"2024","unstructured":"Chen W, Yan-yi L, Tie-zheng G, Da-peng L, Tao H, Zhi L, Qing-wen Y, Hui-han W, Ying-you W (2024) Systems engineering issues for industry applications of large language model. Appl Soft Comput 151:111165","journal-title":"Appl Soft Comput"},{"key":"1527_CR7","unstructured":"Creswell A, Shanahan M, Higgins I (2022) Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. Accessed 10 June 2023"},{"issue":"02","key":"1527_CR8","doi-asserted-by":"publisher","first-page":"715","DOI":"10.1109\/TKDE.2022.3224228","volume":"36","author":"X Zhu","year":"2024","unstructured":"Zhu X, Li Z, Wang X, Jiang X, Sun P, Wang X, Xiao Y, Yuan NJ (2024) Multi-modal knowledge graph construction and application: a survey. IEEE Trans Knowl Data Eng 36(02):715\u2013735","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"1527_CR9","doi-asserted-by":"crossref","unstructured":"Cai B, Xiang Y, Gao L, Zhang H, Li Y, Li J (2022) Temporal knowledge graph completion: A survey. arXiv preprint arXiv:2201.08236. Accessed 11 June 2023","DOI":"10.24963\/ijcai.2023\/734"},{"key":"1527_CR10","first-page":"682","volume-title":"CCF International Conference on Natural Language Processing and Chinese Computing","author":"G Dong","year":"2023","unstructured":"Dong G, Zhao J, Hui T, Guo D, Wang W, Feng B, Qiu Y, Gongque Z, He K, Wang Z et al (2023) Revisit input perturbation problems for llms: a unified robustness evaluation framework for noisy slot filling task. CCF International Conference on Natural Language Processing and Chinese Computing. Springer, New York, pp 682\u2013694"},{"key":"1527_CR11","doi-asserted-by":"crossref","unstructured":"Moiseev F, Dong Z, Alfonseca E, Jaggi M (2022) Skill: structured knowledge infusion for large language models. arXiv preprint arXiv:2205.08184. Accessed 10 Oct 2023","DOI":"10.18653\/v1\/2022.naacl-main.113"},{"issue":"2","key":"1527_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnlest.2022.100159","volume":"20","author":"L Tian","year":"2022","unstructured":"Tian L, Zhou X, Wu Y-P, Zhou W-T, Zhang J-H, Zhang T-S (2022) Knowledge graph and knowledge reasoning: A systematic review. Journal of Electronic Science and Technology 20(2):100159","journal-title":"Journal of Electronic Science and Technology"},{"issue":"3","key":"1527_CR13","doi-asserted-by":"publisher","first-page":"1389","DOI":"10.1002\/widm.1389","volume":"11","author":"N Chakraborty","year":"2021","unstructured":"Chakraborty N, Lukovnikov D, Maheshwari G, Trivedi P, Lehmann J, Fischer A (2021) Introduction to neural network-based question answering over knowledge graphs. Wiley Interdiscip Rev Data Min Knowl Discov 11(3):1389","journal-title":"Wiley Interdiscip Rev Data Min Knowl Discov"},{"key":"1527_CR14","unstructured":"Berant J, Chou A, Frostig R, Liang P (2013) Semantic parsing on freebase from question-answer pairs. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1533\u20131544"},{"key":"1527_CR15","doi-asserted-by":"crossref","unstructured":"Yih W-t, He X, Meek C (2014) Semantic parsing for single-relation question answering. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 643\u2013648","DOI":"10.3115\/v1\/P14-2105"},{"key":"1527_CR16","doi-asserted-by":"crossref","unstructured":"Kr\u00f6tzsch M, Marx M, Ozaki A, Thost V (2018) Attributed description logics: Reasoning on knowledge graphs. In: IJCAI, pp. 5309\u20135313","DOI":"10.24963\/ijcai.2018\/743"},{"issue":"4","key":"1527_CR17","doi-asserted-by":"publisher","first-page":"2192","DOI":"10.1109\/TSMC.2023.3342640","volume":"54","author":"Z Xiao","year":"2024","unstructured":"Xiao Z, Xing H, Qu R, Feng L, Luo S, Dai P, Zhao B, Dai Y (2024) Densely knowledge-aware network for multivariate time series classification. IEEE Trans Syst, Man, Cybern Syst 54(4):2192\u20132204","journal-title":"IEEE Trans Syst, Man, Cybern Syst"},{"issue":"1","key":"1527_CR18","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1109\/TETCI.2023.3304948","volume":"8","author":"Z Xiao","year":"2023","unstructured":"Xiao Z, Xing H, Zhao B, Qu R, Luo S, Dai P, Li K, Zhu Z (2023) Deep contrastive representation learning with self-distillation. IEEE Trans Emerg Topics Comput Intell 8(1):3\u201315","journal-title":"IEEE Trans Emerg Topics Comput Intell"},{"key":"1527_CR19","doi-asserted-by":"crossref","unstructured":"Huang X, Zhang J, Li D, Li P (2019) Knowledge graph embedding based question answering. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. pp. 105\u2013113","DOI":"10.1145\/3289600.3290956"},{"key":"1527_CR20","unstructured":"Yang B, Yih W-t, He X, Gao J, Deng L (2014) Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. Accessed 22 July 2023"},{"issue":"11","key":"1527_CR21","doi-asserted-by":"publisher","first-page":"2168","DOI":"10.1071\/RD16498","volume":"29","author":"W Liu","year":"2017","unstructured":"Liu W, Zhao Q, Piao S, Wang C, Kong Q, An T (2017) Endo-sirna deficiency results in oocyte maturation failure and apoptosis in porcine oocytes. Reprod Fertil Dev 29(11):2168\u20132174","journal-title":"Reprod Fertil Dev"},{"key":"1527_CR22","doi-asserted-by":"crossref","unstructured":"Xiong W, Yu M, Chang S, Guo X, Wang WY (2019) Improving question answering over incomplete kbs with knowledge-aware reader. arXiv preprint arXiv:1905.07098. Accessed 07 Jan 2023","DOI":"10.18653\/v1\/P19-1417"},{"key":"1527_CR23","unstructured":"Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2017) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. arXiv preprint arXiv:1711.05851. Accessed 09 June 2023"},{"key":"1527_CR24","doi-asserted-by":"publisher","first-page":"26719","DOI":"10.1109\/ACCESS.2024.3367588","volume":"12","author":"J Cincovic","year":"2024","unstructured":"Cincovic J, Jovanovic L, Nikolic B, Bacanin N (2024) Neurodegenerative condition detection using modified metaheuristic for attention based recurrent neural networks and extreme gradient boosting tuning. IEEE Access 12:26719\u201326734","journal-title":"IEEE Access"},{"issue":"3","key":"1527_CR25","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1007\/s10462-023-10678-y","volume":"57","author":"M Pavlov-Kagadejev","year":"2024","unstructured":"Pavlov-Kagadejev M, Jovanovic L, Bacanin N, Deveci M, Zivkovic M, Tuba M, Strumberger I, Pedrycz W (2024) Optimizing long-short-term memory models via metaheuristics for decomposition aided wind energy generation forecasting. Artif Intell Rev 57(3):45","journal-title":"Artif Intell Rev"},{"issue":"2","key":"1527_CR26","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1109\/TNNLS.2021.3070843","volume":"33","author":"S Ji","year":"2021","unstructured":"Ji S, Pan S, Cambria E, Marttinen P, Philip SY (2021) A survey on knowledge graphs: representation, acquisition, and applications. IEEE Trans Neural Netw Learn Syst 33(2):494\u2013514","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"1527_CR27","unstructured":"Pan S, Luo L, Wang Y, Chen C, Wang J, Wu X (2023) Unifying large language models and knowledge graphs: a roadmap. arXiv preprint arXiv:2306.08302. Accessed 14 July 2023"},{"key":"1527_CR28","doi-asserted-by":"crossref","unstructured":"Zhang Z, Han X, Liu Z, Jiang X, Sun M, Liu Q (2019) Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129","DOI":"10.18653\/v1\/P19-1139"},{"key":"1527_CR29","doi-asserted-by":"crossref","unstructured":"Feng S, Balachandran V, Bai Y, Tsvetkov Y (2023) Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. arXiv preprint arXiv:2305.08281","DOI":"10.18653\/v1\/2023.emnlp-main.59"},{"key":"1527_CR30","first-page":"11630","volume":"36","author":"D Yu","year":"2022","unstructured":"Yu D, Zhu C, Yang Y, Zeng M (2022) Jaket: joint pre-training of knowledge graph and language understanding. Proc AAAI Conf Artif Intell. 36:11630\u201311638","journal-title":"Proc AAAI Conf Artif Intell."},{"key":"1527_CR31","doi-asserted-by":"crossref","unstructured":"Baek J, Aji AF, Saffari A (2023) Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136. Accessed 10 Aug 2023","DOI":"10.18653\/v1\/2023.matching-1.7"},{"key":"1527_CR32","doi-asserted-by":"crossref","unstructured":"Ji Z, Liu Z, Lee N, Yu T, Wilie B, Zeng M, Fung P (2022) Rho ($$\\rho $$): Reducing hallucination in open-domain dialogues with knowledge grounding. arXiv preprint arXiv:2212.01588. Accessed 26 July 2023","DOI":"10.18653\/v1\/2023.findings-acl.275"},{"key":"1527_CR33","doi-asserted-by":"crossref","unstructured":"Zhang Y, Dai H, Kozareva Z, Smola A, Song L (2018) Variational reasoning for question answering with knowledge graph. Proc AAAI Conf Artif Intell. 32: 6069\u20136076","DOI":"10.1609\/aaai.v32i1.12057"},{"key":"1527_CR34","doi-asserted-by":"crossref","unstructured":"Yih W-t, Richardson M, Meek C, Chang M-W, Suh J (2016) The value of semantic parse labeling for knowledge base question answering. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 201\u2013206","DOI":"10.18653\/v1\/P16-2033"},{"key":"1527_CR35","doi-asserted-by":"crossref","unstructured":"Xu K, Lai Y, Feng Y, Wang Z (2019) Enhancing key-value memory neural networks for knowledge based question answering. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp 2937\u20132947","DOI":"10.18653\/v1\/N19-1301"},{"key":"1527_CR36","doi-asserted-by":"crossref","unstructured":"Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen WW (2018) Open domain question answering using early fusion of knowledge bases and text. arXiv preprint arXiv:1809.00782. Accessed 09 Sept 2023","DOI":"10.18653\/v1\/D18-1455"},{"key":"1527_CR37","doi-asserted-by":"crossref","unstructured":"Baek J, Aji AF, Saffari A (2023) Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136. Accessed 10 Aug 2023","DOI":"10.18653\/v1\/2023.matching-1.7"},{"key":"1527_CR38","unstructured":"Sun J, Xu C, Tang L, Wang S, Lin C, Gong Y, Shum H-Y, Guo J (2023) Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697. Accessed 11 Oct 2023"},{"key":"1527_CR39","unstructured":"Google: Freebase Data Dumps. https:\/\/developers.google.com\/freebase\/data (2023). Accessed 15 Aug 2023"},{"key":"1527_CR40","doi-asserted-by":"crossref","unstructured":"Saxena A, Tripathi A, Talukdar P (2020) Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4498\u20134507","DOI":"10.18653\/v1\/2020.acl-main.412"},{"key":"1527_CR41","doi-asserted-by":"crossref","unstructured":"He G, Lan Y, Jiang J, Zhao WX, Wen J-R (2021) Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. pp. 553\u2013561","DOI":"10.1145\/3437963.3441753"},{"key":"1527_CR42","doi-asserted-by":"crossref","unstructured":"Shi J, Cao S, Hou L, Li J, Zhang H (2021) Transfernet: An effective and transparent framework for multi-hop question answering over relation graph. arXiv preprint arXiv:2104.07302","DOI":"10.18653\/v1\/2021.emnlp-main.341"},{"key":"1527_CR43","unstructured":"Das R, Godbole A, Naik A, Tower E, Zaheer M, Hajishirzi H, Jia R, McCallum A (2022) Knowledge base question answering by case-based reasoning over subgraphs. In: International Conference on Machine Learning, pp. 4777\u20134793. PMLR. Accessed 21 Oct 2023"},{"key":"1527_CR44","unstructured":"Jiang J, Zhou K, Zhao WX, Wen J-R (2022) Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. arXiv preprint arXiv:2212.00959"},{"key":"1527_CR45","doi-asserted-by":"crossref","unstructured":"Jiang J, Zhou K, Dong Z, Ye K, Zhao WX, Wen J-R (2023) Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645","DOI":"10.18653\/v1\/2023.emnlp-main.574"},{"key":"1527_CR46","unstructured":"Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, et al (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01527-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-024-01527-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01527-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,14]],"date-time":"2024-09-14T15:20:26Z","timestamp":1726327226000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-024-01527-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,2]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["1527"],"URL":"https:\/\/doi.org\/10.1007\/s40747-024-01527-8","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,2]]},"assertion":[{"value":"1 March 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}