{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T22:45:50Z","timestamp":1775083550729,"version":"3.50.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"2","funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>While large language models (LLMs) have been successfully applied to various tasks, they still face challenges with hallucinations. Augmenting LLMs with domain-specific tools such as database utilities can facilitate easier and more precise access to specialized knowledge. In this article, we present GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National Center for Biotechnology Information (NCBI) for answering genomics questions. Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs by in-context learning and an augmented decoding algorithm that can detect and execute API calls.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Experimental results show that GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, largely surpassing retrieval-augmented LLMs such as the new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: First, API demonstrations have good cross-task generalizability and are more useful than documentations for in-context learning; second, GeneGPT can generalize to longer chains of API calls and answer multi-hop questions in GeneHop, a novel dataset introduced in this work; finally, different types of errors are enriched in different tasks, providing valuable insights for future improvements.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>The GeneGPT code and data are publicly available at https:\/\/github.com\/ncbi\/GeneGPT.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae075","type":"journal-article","created":{"date-parts":[[2024,2,11]],"date-time":"2024-02-11T10:20:06Z","timestamp":1707646806000},"source":"Crossref","is-referenced-by-count":126,"title":["GeneGPT: augmenting large language models with domain tools for improved access to biomedical information"],"prefix":"10.1093","volume":"40","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1268-7239","authenticated-orcid":false,"given":"Qiao","family":"Jin","sequence":"first","affiliation":[{"name":"National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health , Bethesda, MD 20894, United States"}]},{"given":"Yifan","family":"Yang","sequence":"additional","affiliation":[{"name":"National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health , Bethesda, MD 20894, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6036-1516","authenticated-orcid":false,"given":"Qingyu","family":"Chen","sequence":"additional","affiliation":[{"name":"National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health , Bethesda, MD 20894, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9998-916X","authenticated-orcid":false,"given":"Zhiyong","family":"Lu","sequence":"additional","affiliation":[{"name":"National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health , Bethesda, MD 20894, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,2,10]]},"reference":[{"key":"2024022922523873700_btae075-B1","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/S0022-2836(05)80360-2","article-title":"Basic local alignment search tool","volume":"215","author":"Altschul","year":"1990","journal-title":"J Mol Biol"},{"key":"2024022922523873700_btae075-B2","doi-asserted-by":"crossref","first-page":"W29","DOI":"10.1093\/nar\/gkt282","article-title":"Blast: a more efficient report with usability improvements","volume":"41","author":"Boratyn","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2024022922523873700_btae075-B3","first-page":"2206","author":"Borgeaud","year":"2022"},{"key":"2024022922523873700_btae075-B4","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024022922523873700_btae075-B5","author":"Chen","year":"2021"},{"key":"2024022922523873700_btae075-B6","author":"Chowdhery","year":"2022"},{"key":"2024022922523873700_btae075-B7","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1197\/jamia.M1608","article-title":"Answering physicians\u2019 clinical questions: obstacles and potential solutions","volume":"12","author":"Ely","year":"2005","journal-title":"J Am Med Inform Assoc"},{"key":"2024022922523873700_btae075-B8","author":"Gao","year":"2022"},{"key":"2024022922523873700_btae075-B9","first-page":"3929","author":"Guu","year":"2020"},{"key":"2024022922523873700_btae075-B10","first-page":"2023","article-title":"Geneturing tests gpt models in genomics","author":"Hou","year":"2023"},{"key":"2024022922523873700_btae075-B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024022922523873700_btae075-B12","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1681\/ASN.0000000000000166","article-title":"Retrieve, summarize, and verify: how will chatgpt impact information seeking from the medical literature?","volume":"34","author":"Jin","year":"2023","journal-title":"J Am Soc Nephrol"},{"key":"2024022922523873700_btae075-B13","author":"Jin","year":"2023"},{"key":"2024022922523873700_btae075-B14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3490238","article-title":"Biomedical question answering: a survey of approaches and challenges","volume":"55","author":"Jin","year":"2022","journal-title":"ACM Comput Surv"},{"key":"2024022922523873700_btae075-B15","author":"Kaplan","year":"2020"},{"key":"2024022922523873700_btae075-B16","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"key":"2024022922523873700_btae075-B17","author":"Li\u00e9vin","year":"2022"},{"key":"2024022922523873700_btae075-B18","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbac409","article-title":"Biogpt: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024022922523873700_btae075-B19","author":"Mialon","year":"2023"},{"key":"2024022922523873700_btae075-B20","author":"Nori","year":"2023"},{"key":"2024022922523873700_btae075-B21","author":"OpenAI","year":"2023"},{"key":"2024022922523873700_btae075-B22","author":"Parisi","year":"2022"},{"key":"2024022922523873700_btae075-B23","author":"Qin","year":"2023"},{"key":"2024022922523873700_btae075-B24","author":"Radford","year":"2018"},{"key":"2024022922523873700_btae075-B25","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2024022922523873700_btae075-B26","doi-asserted-by":"crossref","first-page":"D23","DOI":"10.1093\/nar\/gky1069","article-title":"Database resources of the national center for biotechnology information","volume":"47","author":"Sayers","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2024022922523873700_btae075-B27","author":"Schick","year":"2023"},{"key":"2024022922523873700_btae075-B28","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/S0076-6879(96)66012-1","article-title":"Entrez: molecular biology database and retrieval system","volume":"266","author":"Schuler","year":"1996","journal-title":"Methods Enzymol"},{"key":"2024022922523873700_btae075-B29","author":"Singhal","year":"2022"},{"issue":"1","key":"2024022922523873700_btae075-B30","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbad493","article-title":"Opportunities and challenges for chatgpt and large language models in biomedicine and health","volume":"25","author":"Tian","year":"2024","journal-title":"Brief Bioinform"},{"key":"2024022922523873700_btae075-B31","author":"Wei","year":"2022"},{"key":"2024022922523873700_btae075-B32","author":"Wei","year":"2022"},{"key":"2024022922523873700_btae075-B33","author":"Wong","year":"2023"},{"key":"2024022922523873700_btae075-B34","author":"Yao","year":"2022"},{"key":"2024022922523873700_btae075-B35","author":"Yuan","year":"2023"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae075\/56659521\/btae075.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/2\/btae075\/56803364\/btae075.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/2\/btae075\/56803364\/btae075.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,29]],"date-time":"2024-02-29T22:53:00Z","timestamp":1709247180000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae075\/7606338"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2024,2,1]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae075","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,2,1]]},"published":{"date-parts":[[2024,2,1]]},"article-number":"btae075"}}