{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T11:16:43Z","timestamp":1784805403620,"version":"3.55.0"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["NSF_2033569"],"award-info":[{"award-number":["NSF_2033569"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Large language models (LLMs) are being adopted at an unprecedented rate, yet still face challenges in knowledge-intensive domains such as biomedicine. Solutions such as pretraining and domain-specific fine-tuning add substantial computational overhead, requiring further domain-expertise. Here, we introduce a token-optimized and robust Knowledge Graph-based Retrieval Augmented Generation (KG-RAG) framework by leveraging a massive biomedical KG (SPOKE) with LLMs such as Llama-2-13b, GPT-3.5-Turbo, and GPT-4, to generate meaningful biomedical text rooted in established knowledge.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Compared to the existing RAG technique for Knowledge Graphs, the proposed method utilizes minimal graph schema for context extraction and uses embedding methods for context pruning. This optimization in context extraction results in more than 50% reduction in token consumption without compromising the accuracy, making a cost-effective and robust RAG implementation on proprietary LLMs. KG-RAG consistently enhanced the performance of LLMs across diverse biomedical prompts by generating responses rooted in established knowledge, accompanied by accurate provenance and statistical evidence (if available) to substantiate the claims. Further benchmarking on human curated datasets, such as biomedical true\/false and multiple-choice questions (MCQ), showed a remarkable 71% boost in the performance of the Llama-2 model on the challenging MCQ dataset, demonstrating the framework\u2019s capacity to empower open-source models with fewer parameters for domain-specific questions. Furthermore, KG-RAG enhanced the performance of proprietary GPT models, such as GPT-3.5 and GPT-4. In summary, the proposed framework combines explicit and implicit knowledge of KG and LLM in a token optimized fashion, thus enhancing the adaptability of general-purpose LLMs to tackle domain-specific questions in a cost-effective fashion.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>SPOKE KG can be accessed at https:\/\/spoke.rbvi.ucsf.edu\/neighborhood.html. It can also be accessed using REST-API (https:\/\/spoke.rbvi.ucsf.edu\/swagger\/). KG-RAG code is made available at https:\/\/github.com\/BaranziniLab\/KG_RAG. Biomedical benchmark datasets used in this study are made available to the research community in the same GitHub repository.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae560","type":"journal-article","created":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T19:35:48Z","timestamp":1726601748000},"source":"Crossref","is-referenced-by-count":117,"title":["Biomedical knowledge graph-optimized prompt generation for large language models"],"prefix":"10.1093","volume":"40","author":[{"given":"Karthik","family":"Soman","sequence":"first","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter W","family":"Rose","sequence":"additional","affiliation":[{"name":"San Diego Supercomputer Center, University of California , San Diego, CA 92093,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0290-7979","authenticated-orcid":false,"given":"John H","family":"Morris","sequence":"additional","affiliation":[{"name":"Department of Pharmaceutical Chemistry, School of Pharmacy, University of California , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rabia E","family":"Akbas","sequence":"additional","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brett","family":"Smith","sequence":"additional","affiliation":[{"name":"Institute for Systems Biology , Seattle, WA 98109,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Braian","family":"Peetoom","sequence":"additional","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Catalina","family":"Villouta-Reyes","sequence":"additional","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriel","family":"Cerono","sequence":"additional","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yongmei","family":"Shi","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela","family":"Rizk-Jackson","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sharat","family":"Israni","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotte A","family":"Nelson","sequence":"additional","affiliation":[{"name":"Mate Bioservices, Inc. Swallowtail Ct. , Brisbane, CA 94005,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3545-4665","authenticated-orcid":false,"given":"Sui","family":"Huang","sequence":"additional","affiliation":[{"name":"Institute for Systems Biology , Seattle, WA 98109,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0067-194X","authenticated-orcid":false,"given":"Sergio E","family":"Baranzini","sequence":"additional","affiliation":[{"name":"Department of Neurology, Weill Institute for Neurosciences, University of California, San Francisco , San Francisco, CA 94158,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,9,17]]},"reference":[{"key":"2024093015202020700_btae560-B1","first-page":"46","article-title":"A biomedical open knowledge network harnesses the power of AI to understand deep human biology","volume":"43","author":"Baranzini","year":"2022","journal-title":"AI Mag"},{"key":"2024093015202020700_btae560-B2","author":"Beltagy","year":"2019"},{"key":"2024093015202020700_btae560-B3","doi-asserted-by":"crossref","first-page":"4968","DOI":"10.1021\/acs.jcim.9b00683","article-title":"ROBOKOP KG and KGB: integrated knowledge graphs from federated sources","volume":"59","author":"Bizon","year":"2019","journal-title":"J Chem Inf Model"},{"key":"2024093015202020700_btae560-B4","author":"Chen","year":"2023"},{"key":"2024093015202020700_btae560-B5","doi-asserted-by":"crossref","first-page":"474","DOI":"10.26421\/JDI3.4-5","article-title":"Improved methods to aid unsupervised evidence-based fact checking for online health news","volume":"3","author":"Deka","year":"2022","journal-title":"JDI"},{"key":"2024093015202020700_btae560-B6","author":"Feng","year":"2020"},{"key":"2024093015202020700_btae560-B7","first-page":"1","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM TransComputHealthcare"},{"key":"2024093015202020700_btae560-B8","doi-asserted-by":"crossref","first-page":"e1004259","DOI":"10.1371\/journal.pcbi.1004259","article-title":"Heterogeneous network edge prediction: a data integration approach to prioritize disease-associated genes","volume":"11","author":"Himmelstein","year":"2015","journal-title":"PLoS Comput Biol"},{"key":"2024093015202020700_btae560-B9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.7554\/eLife.26726","article-title":"Systematic integration of biomedical knowledge prioritizes drugs for repurposing","volume":"6","author":"Himmelstein","year":"2017","journal-title":"Elife"},{"key":"2024093015202020700_btae560-B10","author":"Huang","year":"2019"},{"key":"2024093015202020700_btae560-B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024093015202020700_btae560-B12","first-page":"2","author":"Kenton","year":"2019"},{"key":"2024093015202020700_btae560-B13","doi-asserted-by":"crossref","first-page":"3158","DOI":"10.1093\/bioinformatics\/bts591","article-title":"SemMedDB: a PubMed-scale repository of biomedical semantic predications","volume":"28","author":"Kilicoglu","year":"2012","journal-title":"Bioinformatics"},{"key":"2024093015202020700_btae560-B14","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima","year":"2022","journal-title":"Adv Neural Inform Process Syst"},{"key":"2024093015202020700_btae560-B15","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024093015202020700_btae560-B16","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"key":"2024093015202020700_btae560-B17","author":"Lin","year":"2019"},{"key":"2024093015202020700_btae560-B18","author":"Liu","year":"2019"},{"key":"2024093015202020700_btae560-B151","author":"Lu"},{"key":"2024093015202020700_btae560-B19","doi-asserted-by":"crossref","first-page":"bbac409","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024093015202020700_btae560-B20","author":"Luo","year":"2023"},{"key":"2024093015202020700_btae560-B21","doi-asserted-by":"crossref","first-page":"8449","DOI":"10.1609\/aaai.v34i05.6364","article-title":"Graph-based reasoning over heterogeneous external knowledge for commonsense question answering","volume":"34","author":"Lv","year":"2020","journal-title":"AAAI"},{"key":"2024093015202020700_btae560-B22","author":"Maynez","year":"2020"},{"key":"2024093015202020700_btae560-B152","author":"Min"},{"key":"2024093015202020700_btae560-B23","author":"Moiseev","year":"2022"},{"key":"2024093015202020700_btae560-B24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/bioinformatics\/btad080","article-title":"The scalable precision medicine open knowledge engine (SPOKE): a massive knowledge graph of biomedical information","volume":"39","author":"Morris","year":"2023","journal-title":"Bioinformatics"},{"key":"2024093015202020700_btae560-B25","doi-asserted-by":"crossref","first-page":"D712","DOI":"10.1093\/nar\/gkw1128","article-title":"The monarch initiative: an integrative data and analytic platform connecting phenotypes to genotypes across species","volume":"45","author":"Mungall","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2024093015202020700_btae560-B26","first-page":"42","article-title":"Knowledge network embedding of transcriptomic data from spaceflown mice uncovers signs and symptoms associated with terrestrial diseases","volume":"11","author":"Nelson","year":"2021","journal-title":"Life (Basel)"},{"key":"2024093015202020700_btae560-B27","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1093\/jamia\/ocab270","article-title":"Embedding electronic health records onto a knowledge network recognizes prodromal features of multiple sclerosis and predicts diagnosis","volume":"29","author":"Nelson","year":"2021","journal-title":"JAm Med Inform Assoc"},{"key":"2024093015202020700_btae560-B28","doi-asserted-by":"crossref","first-page":"3045","DOI":"10.1038\/s41467-019-11069-0","article-title":"Integrating biomedical research and electronic health records to create knowledge-based biologically meaningful machine-readable embeddings","volume":"10","author":"Nelson","year":"2019","journal-title":"Nat Commun"},{"key":"2024093015202020700_btae560-B29","author":"Pan","year":"2023"},{"key":"2024093015202020700_btae560-B30","author":"Peng","year":"2019"},{"key":"2024093015202020700_btae560-B31","doi-asserted-by":"crossref","first-page":"D833","DOI":"10.1093\/nar\/gkw943","article-title":"DisGeNET: a comprehensive platform integrating information on human disease-associated genes and variants","volume":"45","author":"Pi\u00f1ero","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2024093015202020700_btae560-B32","author":"Raunak","year":"2021"},{"key":"2024093015202020700_btae560-B33","author":"Reimers","year":"2019"},{"key":"2024093015202020700_btae560-B34","author":"Singhal","year":"2022"},{"key":"2024093015202020700_btae560-B35","first-page":"97","article-title":"Time-aware embeddings of clinical data using a knowledge graph","volume":"28","author":"Soman","year":"2023","journal-title":"Pac Symp Biocomput"},{"key":"2024093015202020700_btae560-B36","doi-asserted-by":"crossref","first-page":"1081087","DOI":"10.3389\/fmed.2023.1081087","article-title":"Early detection of Parkinson\u2019s disease through enriching the electronic health record using a biomedical knowledge graph","volume":"10","author":"Soman","year":"2023","journal-title":"Front Med (Lausanne)"},{"key":"2024093015202020700_btae560-B37","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1038\/s43587-024-00573-8","article-title":"Leveraging electronic health records and knowledge networks for Alzheimer\u2019s disease prediction and sex-specific biological insights","volume":"4","author":"Tang","year":"2024","journal-title":"Nat Aging"},{"key":"2024093015202020700_btae560-B38","author":"Touvron","year":"2023"},{"key":"2024093015202020700_btae560-B39","article-title":"Mondo: unifying diseases for the world, by the world","author":"Vasilevsky","year":"2022","journal-title":"MedRxiv"},{"key":"2024093015202020700_btae560-B40","first-page":"7208","year":"2019"},{"key":"2024093015202020700_btae560-B41","author":"Wei","year":"2022"},{"key":"2024093015202020700_btae560-B42","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"AdvNeural Inform Process Syst"},{"key":"2024093015202020700_btae560-B43","first-page":"2346","author":"Yang","year":"2019"},{"key":"2024093015202020700_btae560-B44","first-page":"37309","article-title":"Deep bidirectional language-knowledge graph pretraining","volume":"35","author":"Yasunaga","year":"2022","journal-title":"AdvNeural Inform Process Syst"},{"key":"2024093015202020700_btae560-B45","author":"Yasunaga","year":"2021"},{"key":"2024093015202020700_btae560-B46","author":"Zhao","year":"2023"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae560\/59157142\/btae560.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae560\/59459865\/btae560.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae560\/59459865\/btae560.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T15:20:41Z","timestamp":1727709641000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae560\/7759620"}},"subtitle":[],"editor":[{"given":"Peter","family":"Robinson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,9]]},"references-count":48,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2024,9,2]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae560","relation":{},"ISSN":["1367-4811"],"issn-type":[{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,9]]},"article-number":"btae560"}}