{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T05:43:43Z","timestamp":1776750223782,"version":"3.51.2"},"reference-count":33,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T00:00:00Z","timestamp":1754006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Scientific and Technological Research Council of Turkey (TUBITAK)\u2019s Industrial PhD program","award":["118C056"],"award-info":[{"award-number":["118C056"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Large language models (LLMs) often tend to hallucinate, especially in domain-specific tasks and tasks that require reasoning. Previously, we introduced SubGraph Retrieval Augmented Generation (SG-RAG) as a novel Graph RAG method for multi-hop question answering. SG-RAG leverages Cypher queries to search a given knowledge graph and retrieve the subgraph necessary to answer the question. The results from our previous work showed the higher performance of our method compared to the traditional Retrieval Augmented Generation (RAG). In this work, we further enhanced SG-RAG by proposing an additional step called Merging and Ordering Triplets (MOT). The new MOT step seeks to decrease the redundancy in the retrieved triplets by applying hierarchical merging to the retrieved subgraphs. Moreover, it provides an ordering among the triplets using the Breadth-First Search (BFS) traversal algorithm. We conducted experiments on the MetaQA benchmark, which was proposed for multi-hop question-answering in the movies domain. Our experiments showed that SG-RAG MOT provided more accurate answers than Chain-of-Thought and Graph Chain-of-Thought. We also found that merging (up to a certain point) highly overlapping subgraphs and defining an order among the triplets helped the LLM to generate more precise answers.<\/jats:p>","DOI":"10.3390\/make7030074","type":"journal-article","created":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T15:56:16Z","timestamp":1754409376000},"page":"74","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["SG-RAG MOT: SubGraph Retrieval Augmented Generation with Merging and Ordering Triplets for Knowledge Graph Multi-Hop Question Answering"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0916-2578","authenticated-orcid":false,"given":"Ahmmad O. M.","family":"Saleh","sequence":"first","affiliation":[{"name":"Computer Science and Engineering, Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul 34956, Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gokhan","family":"Tur","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science, The Grainger College of Engineering, University of Illinois Urbana-Champaign, Champaign, IL 61820, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9243-7776","authenticated-orcid":false,"given":"Yucel","family":"Saygin","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul 34956, Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,1]]},"reference":[{"key":"ref_1","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_2","unstructured":"Tonmoy, S., Zaman, S., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. (2024). A comprehensive survey of hallucination mitigation techniques in large language models. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, J., Chen, J., Ren, R., Cheng, X., Zhao, W.X., Nie, J.Y., and Wen, J.R. (2024). The dawn after the dark: An empirical study on factuality hallucination in large language models. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.586"},{"key":"ref_4","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_5","unstructured":"Setty, S., Jijo, K., Chung, E., and Vidra, N. (2024). Improving Retrieval for RAG based Question Answering Models on Financial Documents. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"AIoa2300068","DOI":"10.1056\/AIoa2300068","article-title":"Almanac\u2014Retrieval-augmented language models for clinical medicine","volume":"1","author":"Zakka","year":"2024","journal-title":"Nejm AI"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Alan, A.Y., Karaarslan, E., and Aydin, O. (2024). A RAG-based Question Answering System Proposal for Understanding Islam: MufassirQAS LLM. arXiv.","DOI":"10.2139\/ssrn.4707470"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv.","DOI":"10.1162\/tacl_a_00638"},{"key":"ref_9","unstructured":"Baker, G.A., Raut, A., Shaier, S., Hunter, L.E., and von der Wense, K. (2024). Lost in the Middle, and In-Between: Enhancing Language Models\u2019 Ability to Reason Over Long Contexts in Multi-Hop QA. arXiv."},{"key":"ref_10","unstructured":"Peysakhovich, A., and Lerer, A. (2023). Attention sorting combats recency bias in long context language models. arXiv."},{"key":"ref_11","unstructured":"Zhang, Q., Chen, S., Bei, Y., Yuan, Z., Zhou, H., Hong, Z., Dong, J., Chen, H., Chang, Y., and Huang, X. (2025). A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models. arXiv."},{"key":"ref_12","unstructured":"Larson, J., and Truitt, S. (2024, June 25). GraphRAG: Unlocking LLM Discovery on Narrative Private Data. Available online: https:\/\/www.microsoft.com\/en-us\/research\/blog\/graphrag-unlocking-llm-discovery-on-narrative-private-data\/."},{"key":"ref_13","unstructured":"Saleh, A.O., T\u00fcr, G., and Saygin, Y. (2024, January 19\u201320). SG-RAG: Multi-Hop Question Answering With Large Language Models Through Knowledge Graphs. Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), Trento, Italy."},{"key":"ref_14","unstructured":"Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.b., Soricut, R., Lazaridou, A., Firat, O., and Schrittwieser, J. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv."},{"key":"ref_15","unstructured":"Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv."},{"key":"ref_16","unstructured":"Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., and Han, J. (2023). Large language models on graphs: A comprehensive survey. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1145\/3655103.3655110","article-title":"Exploring the potential of large language models (llms) in learning on graphs","volume":"25","author":"Chen","year":"2024","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_18","unstructured":"Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., and Larson, J. (2024). From local to global: A graph rag approach to query-focused summarization. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Jin, B., Xie, C., Zhang, J., Roy, K.K., Zhang, Y., Li, Z., Li, R., Tang, X., Wang, S., and Meng, Y. (2024). Graph chain-of-thought: Augmenting large language models by reasoning on graphs. arXiv.","DOI":"10.18653\/v1\/2024.findings-acl.11"},{"key":"ref_20","unstructured":"Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.Y., and Guo, J. (2023). Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.21307\/joss-2019-011","article-title":"A multigraph approach to social network analysis","volume":"16","author":"Shafie","year":"2015","journal-title":"J. Soc. Struct. Carnegie Mellon"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Francis, N., Green, A., Guagliardo, P., Libkin, L., Lindaaker, T., Marsault, V., Plantikow, S., Rydberg, M., Selmer, P., and Taylor, A. (2018, January 10\u201315). Cypher: An evolving query language for property graphs. Proceedings of the 2018 International Conference on Management of Data, Houston, TX, USA.","DOI":"10.1145\/3183713.3190657"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Dai, H., Kozareva, Z., Smola, A.J., and Song, L. (2018, January 2\u20137). Variational Reasoning for Question Answering with Knowledge Graph. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12057"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_25","unstructured":"Lin, C.Y. (2004, January 25\u201326). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out, Barcelona, Spain."},{"key":"ref_26","unstructured":"Lapata, M., Blunsom, P., and Koller, A. (2017, January 3\u20137). A Network-based End-to-End Trainable Task-oriented Dialogue System. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain. Long Papers."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dror, R., Baumer, G., Shlomov, S., and Reichart, R. (2018, January 15\u201320). The hitchhiker\u2019s guide to testing statistical significance in natural language processing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia. Long papers.","DOI":"10.18653\/v1\/P18-1128"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"144","DOI":"10.4097\/kjae.2017.70.2.144","article-title":"Central limit theorem: The cornerstone of modern statistics","volume":"70","author":"Kwak","year":"2017","journal-title":"Korean J. Anesthesiol."},{"key":"ref_29","unstructured":"(2024, May 01). AI@Meta. Llama 3 Model Card. Available online: https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md."},{"key":"ref_30","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Tang, R., Zhang, X., Ma, X., Lin, J., and Ture, F. (2023). Found in the middle: Permutation self-consistency improves listwise ranking in large language models. arXiv.","DOI":"10.18653\/v1\/2024.naacl-long.129"},{"key":"ref_32","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","unstructured":"Team, Q. (2025, April 01). Qwen2.5: A Party of Foundation Models. Available online: https:\/\/qwenlm.github.io\/blog\/qwen2.5\/."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/74\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:20:59Z","timestamp":1760034059000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/74"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,1]]},"references-count":33,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["make7030074"],"URL":"https:\/\/doi.org\/10.3390\/make7030074","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,1]]}}}