{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T08:33:17Z","timestamp":1777969997613,"version":"3.51.4"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,6,10]],"date-time":"2024-06-10T00:00:00Z","timestamp":1717977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01AI130460"],"award-info":[{"award-number":["R01AI130460"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24AI171008"],"award-info":[{"award-number":["U24AI171008"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000968","name":"American Heart Association","doi-asserted-by":"publisher","award":["19GPSGC35180031"],"award-info":[{"award-number":["19GPSGC35180031"]}],"id":[{"id":"10.13039\/100000968","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004917","name":"Cancer Prevention and Research Institute of Texas","doi-asserted-by":"publisher","award":["RP220244"],"award-info":[{"award-number":["RP220244"]}],"id":[{"id":"10.13039\/100004917","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Precise literature recommendation and summarization are crucial for biomedical professionals. While the latest iteration of generative pretrained transformer (GPT) incorporates 2 distinct modes\u2014real-time search and pretrained model utilization\u2014it encounters challenges in dealing with these tasks. Specifically, the real-time search can pinpoint some relevant articles but occasionally provides fabricated papers, whereas the pretrained model excels in generating well-structured summaries but struggles to cite specific sources. In response, this study introduces RefAI, an innovative retrieval-augmented generative tool designed to synergize the strengths of large language models (LLMs) while overcoming their limitations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>RefAI utilized PubMed for systematic literature retrieval, employed a novel multivariable algorithm for article recommendation, and leveraged GPT-4 turbo for summarization. Ten queries under 2 prevalent topics (\u201ccancer immunotherapy and target therapy\u201d and \u201cLLMs in medicine\u201d) were chosen as use cases and 3 established counterparts (ChatGPT-4, ScholarAI, and Gemini) as our baselines. The evaluation was conducted by 10 domain experts through standard statistical analyses for performance comparison.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The overall performance of RefAI surpassed that of the baselines across 5 evaluated dimensions\u2014relevance and quality for literature recommendation, accuracy, comprehensiveness, and reference integration for summarization, with the majority exhibiting statistically significant improvements (P-values &amp;lt;.05).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>RefAI demonstrated substantial improvements in literature recommendation and summarization over existing tools, addressing issues like fabricated papers, metadata inaccuracies, restricted recommendations, and poor reference integration.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>By augmenting LLM with external resources and a novel ranking algorithm, RefAI is uniquely capable of recommending high-quality literature and generating well-structured summaries, holding the potential to meet the critical needs of biomedical professionals in navigating and synthesizing vast amounts of scientific literature.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocae129","type":"journal-article","created":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T17:13:04Z","timestamp":1716311584000},"page":"2030-2039","source":"Crossref","is-referenced-by-count":53,"title":["RefAI: a GPT-powered retrieval-augmented generative tool for biomedical literature recommendation and summarization"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8784-1745","authenticated-orcid":false,"given":"Yiming","family":"Li","sequence":"first","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Jeff","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, College of Natural Sciences, University of Texas at Austin , Austin, TX 78712,","place":["United States"]}]},{"given":"Manqi","family":"Li","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]},{"name":"Department of Biostatistics & Data Science, School of Public Health, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4014-2957","authenticated-orcid":false,"given":"Yifang","family":"Dang","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Evan","family":"Yu","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Jianfu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1577-874X","authenticated-orcid":false,"given":"Zenan","family":"Sun","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Usama","family":"Hussein","sequence":"additional","affiliation":[{"name":"Department of Lymphoma and Myeloma, The University of Texas MD Anderson Cancer Center , Houston, TX 77030,","place":["United States"]}]},{"given":"Jianguo","family":"Wen","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Ahmed M","family":"Abdelhameed","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Junhua","family":"Mai","sequence":"additional","affiliation":[{"name":"Department of Nanomedicine, Houston Methodist Academic Institute , Houston, TX 77030,","place":["United States"]}]},{"given":"Shenduo","family":"Li","sequence":"additional","affiliation":[{"name":"Division of Hematology and Oncology, Department of Medicine, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Yue","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Quantitative Health Sciences, Mayo Clinic , Rochester, MN 55905,","place":["United States"]}]},{"given":"Xinyue","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Daowei","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Translational Molecular Pathology, The University of Texas MD Anderson Cancer Center , Houston, TX 77030,","place":["United States"]}]},{"given":"Jingna","family":"Feng","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Zehan","family":"Li","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Jianping","family":"He","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Wei","family":"Tao","sequence":"additional","affiliation":[{"name":"Department of Biostatistics & Data Science, School of Public Health, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Tiehang","family":"Duan","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Yanyan","family":"Lou","sequence":"additional","affiliation":[{"name":"Division of Hematology and Oncology, Department of Medicine, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8865-7717","authenticated-orcid":false,"given":"Fang","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4267-1924","authenticated-orcid":false,"given":"Cui","family":"Tao","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,6,10]]},"reference":[{"key":"2026041420160596100_ocae129-B1","author":"Gonz\u00e1lez-M\u00e1rquez","year":"2023"},{"key":"2026041420160596100_ocae129-B2","author":"About"},{"issue":"16","key":"2026041420160596100_ocae129-B3","doi-asserted-by":"crossref","first-page":"5773","DOI":"10.1073\/pnas.1404402111","article-title":"Rescuing US biomedical research from its systemic flaws","volume":"111","author":"Alberts","year":"2014","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2026041420160596100_ocae129-B4","doi-asserted-by":"crossref","first-page":"e1403","DOI":"10.6061\/clinics\/2019\/e1403","article-title":"Approaching literature review for academic purposes: The Literature Review Checklist","volume":"74","author":"Leite","year":"2019","journal-title":"Clinics (Sao Paulo, Brazil)"},{"issue":"1","key":"2026041420160596100_ocae129-B5","first-page":"13","article-title":"A guide to writing the dissertation literature review","volume":"14","author":"Randolph","year":"2009","journal-title":"Pract Assess Res Eval"},{"key":"2026041420160596100_ocae129-B6","doi-asserted-by":"crossref","first-page":"102274","DOI":"10.1016\/j.lindif.2023.102274","article-title":"ChatGPT for good? On opportunities and challenges of large language models for education","volume":"103","author":"Kasneci","year":"2023","journal-title":"Learn Individ Differ"},{"key":"2026041420160596100_ocae129-B7","author":"Hu","year":"2023"},{"issue":"3","key":"2026041420160596100_ocae129-B8","doi-asserted-by":"crossref","first-page":"e0300919","DOI":"10.1371\/journal.pone.0300919","article-title":"AE-GPT: using large language models to extract adverse events from surveillance reports-a use case with influenza vaccine adverse events","volume":"19","author":"Li","year":"2024","journal-title":"PLoS One"},{"key":"2026041420160596100_ocae129-B9","doi-asserted-by":"crossref","first-page":"104621","DOI":"10.1016\/j.jbi.2024.104621","article-title":"Artificial intelligence-powered pharmacovigilance: a review of machine and deep learning in clinical text-based adverse drug event detection for benchmark datasets","volume":"152","author":"Li","year":"2024","journal-title":"J Biomed Inform"},{"key":"2026041420160596100_ocae129-B10","author":"Li","year":"2024"},{"issue":"8","key":"2026041420160596100_ocae129-B11","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1681\/ASN.0000000000000166","article-title":"Retrieve, summarize, and verify: how will ChatGPT affect information seeking from the medical literature?","volume":"34","author":"Jin","year":"2023","journal-title":"J Am Soc Nephrol"},{"issue":"6","key":"2026041420160596100_ocae129-B12","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1080\/00330124.2023.2190373","article-title":"A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT","volume":"75","author":"Day","year":"2023","journal-title":"Prof Geogr"},{"issue":"2","key":"2026041420160596100_ocae129-B13","first-page":"e35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"issue":"10","key":"2026041420160596100_ocae129-B14","doi-asserted-by":"crossref","first-page":"5275","DOI":"10.1097\/MS9.0000000000001228","article-title":"ChatGPT and artificial hallucinations in stem cell research: assessing the accuracy of generated references\u2014a preliminary study","volume":"85","author":"Sharun","year":"2023","journal-title":"Ann Med Surg (Lond)"},{"issue":"10","key":"2026041420160596100_ocae129-B15","doi-asserted-by":"crossref","first-page":"1457","DOI":"10.3390\/jpm13101457","article-title":"Navigating the landscape of personalized medicine: the relevance of ChatGPT, BingChat, and Bard AI in nephrology literature searches","volume":"13","author":"Aiumtrakul","year":"2023","journal-title":"J Pers Med"},{"key":"2026041420160596100_ocae129-B16","doi-asserted-by":"crossref","first-page":"115334","DOI":"10.1016\/j.psychres.2023.115334","article-title":"ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search","volume":"326","author":"McGowan","year":"2023","journal-title":"Psychiatry Res"},{"key":"2026041420160596100_ocae129-B17"},{"key":"2026041420160596100_ocae129-B18"},{"key":"2026041420160596100_ocae129-B19","year":"2023"},{"key":"2026041420160596100_ocae129-B20","year":"2023"},{"key":"2026041420160596100_ocae129-B21","first-page":"9459","author":"Lewis","year":"2020"},{"key":"2026041420160596100_ocae129-B22","author":"Gao"},{"issue":"2","key":"2026041420160596100_ocae129-B23","doi-asserted-by":"crossref","first-page":"AIoa2300068","DOI":"10.1056\/AIoa2300068","article-title":"Almanac\u2013retrieval-augmented language models for clinical medicine","volume":"1","author":"Zakka","year":"2024","journal-title":"NEJM AI"},{"key":"2026041420160596100_ocae129-B24","author":"Ge","year":"2023"},{"key":"2026041420160596100_ocae129-B25","author":"Unlu","year":"2024"},{"issue":"10","key":"2026041420160596100_ocae129-B26","doi-asserted-by":"crossref","first-page":"1444","DOI":"10.1016\/j.mayocp.2023.07.009","article-title":"Revolution or peril? The controversial role of large language models in medical manuscript writing","volume":"98","author":"Milian","year":"2023","journal-title":"Mayo Clin Proc"},{"key":"2026041420160596100_ocae129-B27","first-page":"1877","author":"Brown","year":"2020"},{"key":"2026041420160596100_ocae129-B28","author":"SerpApi"},{"key":"2026041420160596100_ocae129-B29","author":"Clarivate"},{"key":"2026041420160596100_ocae129-B30","year":"2022"},{"key":"2026041420160596100_ocae129-B31","author":"Weisstein"},{"key":"2026041420160596100_ocae129-B32","author":"GPT-4 architecture, datasets, costs and more leaked;","year":"2023"},{"issue":"2","key":"2026041420160596100_ocae129-B33","doi-asserted-by":"crossref","first-page":"100022","DOI":"10.1016\/j.metrad.2023.100022","article-title":"A comprehensive survey of ChatGPT: advancements, applications, prospects, and challenges","volume":"1","author":"Nazir","year":"2023","journal-title":"Meta Radiol"},{"key":"2026041420160596100_ocae129-B34","author":"ScholarAI"},{"issue":"4","key":"2026041420160596100_ocae129-B35","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1038\/s41433-023-02760-0","article-title":"Google\u2019s AI chatbot \u201cBard\u201d: a side-by-side comparison with ChatGPT and its utilization in ophthalmology","volume":"38","author":"Waisberg","year":"2023","journal-title":"Eye"},{"issue":"2","key":"2026041420160596100_ocae129-B36","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1038\/nrg1768","article-title":"Literature mining for the biologist: from information retrieval to biological discovery","volume":"7","author":"Jensen","year":"2006","journal-title":"Nat Rev Genet"},{"issue":"2","key":"2026041420160596100_ocae129-B37","doi-asserted-by":"crossref","first-page":"189068","DOI":"10.1016\/j.bbcan.2023.189068","article-title":"Advances and prospects of mRNA vaccines in cancer immunotherapy","volume":"1879","author":"Liu","year":"2024","journal-title":"Biochim Biophys Acta Rev Cancer"},{"issue":"2","key":"2026041420160596100_ocae129-B38","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/s12032-023-02280-7","article-title":"Revolutionizing cancer treatment: comprehensive insights into immunotherapeutic strategies","volume":"41","author":"Raghani","year":"2024","journal-title":"Med Oncol"},{"issue":"8","key":"2026041420160596100_ocae129-B39","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1038\/nrd3500","article-title":"Cancer immunotherapy\u2014revisited","volume":"10","author":"Lesterhuis","year":"2011","journal-title":"Nat Rev Drug Discov"},{"issue":"5","key":"2026041420160596100_ocae129-B40","doi-asserted-by":"crossref","first-page":"413","DOI":"10.2217\/pme-2023-0083","article-title":"Personalized medicine, the inevitable future of cancer immunotherapy","volume":"20","author":"Basiri","year":"2023","journal-title":"Per Med"},{"issue":"8","key":"2026041420160596100_ocae129-B41","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2026041420160596100_ocae129-B42","doi-asserted-by":"crossref","first-page":"104630","DOI":"10.1016\/j.jbi.2024.104630","article-title":"Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction","volume":"153","author":"Peng","year":"2024","journal-title":"J Biomed Inform"},{"key":"2026041420160596100_ocae129-B43","author":"Wecker"},{"issue":"7","key":"2026041420160596100_ocae129-B44","doi-asserted-by":"crossref","first-page":"e0271949","DOI":"10.1371\/journal.pone.0271949","article-title":"Information loss and bias in Likert survey responses","volume":"17","author":"Westland","year":"2022","journal-title":"PLoS One"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2030\/58868075\/ocae129.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2030\/58868075\/ocae129.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T00:28:46Z","timestamp":1776212926000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2030\/7690757"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,10]]},"references-count":44,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,6,10]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae129","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,6,10]]}}}