{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T20:09:22Z","timestamp":1786738162961,"version":"build-2736575974"},"posted":{"date-parts":[[2026,8,12]]},"group-title":"Health Informatics","reference-count":38,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T00:00:00Z","timestamp":1786492800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"crossref","award":["UL1TR000445"],"award-info":[{"award-number":["UL1TR000445"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,8,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Large language models (LLMs) that use retrieval-augmented generation (RAG) are increasingly used to answer clinical questions, although the evaluation of these systems remains limited. Building on previous studies conducted by our team, this case report aimed to improve upon this knowledge gap by applying a reusable methodology to compare the performance of eight LLMs that utilize RAG techniques for evidence synthesis.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Case Presentation<\/jats:title>\n                  <jats:p>Eight commercially available RAG LLM tools (OpenEvidence, Undermind, Consensus, SciSpace, Elicit, MediSearch, EvidenceHunt, and Scite) were evaluated using twelve ChatGPT-generated clinical questions on the topics of treatment, etiology, and prognosis. To enable comparison, we prompted ChatGPT to identify all key unique medical concepts from the full set of LLM responses to each question. Concepts were categorized as critical (\u201cmust-have\u201d) or non-critical (\u201cnice-to-have\u201d) for answering the clinical question. Experienced information scientists were consulted at each step for their expertise. Descriptive statistics and Kruskal-Wallis tests were used to compare performance across tools and question categories. No significant differences were found among the eight RAG LLMs in their coverage of \u201cmust-have\u201d (p=0.95) or \u201cnice-to-have\u201d (p=0.16) key unique medical concepts, and no single tool consistently captured all identified concepts.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>These findings suggest that RAG LLMs may be supplementary tools for evidence retrieval and synthesis but cannot, at this time, fully replace comprehensive expert review of the medical literature. The evaluation framework presented here may be a useful model for future comparative assessments of rapidly evolving AI evidence synthesis tools.<\/jats:p>\n                <\/jats:sec>","DOI":"10.64898\/2026.08.10.26360108","type":"posted-content","created":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T22:40:19Z","timestamp":1786574419000},"source":"Crossref","is-referenced-by-count":0,"title":["Evaluating Eight Retrieval-Augmented Generation (RAG) Large Language Models\u2019 Responses to Clinical Questions: A Comparative Study"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3081-6487","authenticated-orcid":false,"given":"Poppy A.","family":"Krump","sequence":"first","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0356-9481","authenticated-orcid":false,"given":"Mallory N.","family":"Blasingame","sequence":"additional","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4014-467X","authenticated-orcid":false,"given":"Taneya Y.","family":"Koonce","sequence":"additional","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2526-3857","authenticated-orcid":false,"given":"Annette M.","family":"Williams","sequence":"additional","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6699-6806","authenticated-orcid":false,"given":"Jing","family":"Su","sequence":"additional","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7644-9803","authenticated-orcid":false,"given":"Nunzia B.","family":"Giuse","sequence":"additional","affiliation":[{"name":"Center for Knowledge Management (CKM), Vanderbilt Health, Nashville, TN"},{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN"},{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026081412101226000_2026.08.10.26360108v1.1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-025-02644-9"},{"key":"2026081412101226000_2026.08.10.26360108v1.2","doi-asserted-by":"publisher","DOI":"10.1016\/j.puhe.2026.106220"},{"key":"2026081412101226000_2026.08.10.26360108v1.3","unstructured":"Center for Digital Health and AI. 2026 Physician Survey on Augmented Intelligence [Internet]. American Medical Association; [cited 10 Aug 2026]. <https:\/\/www.ama-assn.org\/practice-management\/digital-health\/physician-survey-augmented-intelligence>."},{"key":"2026081412101226000_2026.08.10.26360108v1.4","unstructured":"Perlo J. Most U.S. doctors are quietly using this AI tool. Few patients know about it. NBC News [Internet]. 2026 May 13 [cited 10 Aug 2026]. <https:\/\/www.nbcnews.com\/tech\/tech-news\/openevidence-ai-doctor-medical-physician-login-app-what-npi-uptodate-rcna341064>."},{"key":"2026081412101226000_2026.08.10.26360108v1.5","doi-asserted-by":"publisher","DOI":"10.1177\/20552076251348850"},{"key":"2026081412101226000_2026.08.10.26360108v1.6","doi-asserted-by":"publisher","DOI":"10.1136\/bmjebm-2024-113199"},{"key":"2026081412101226000_2026.08.10.26360108v1.7","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324924000044"},{"key":"2026081412101226000_2026.08.10.26360108v1.8","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf008"},{"key":"2026081412101226000_2026.08.10.26360108v1.9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000877"},{"key":"2026081412101226000_2026.08.10.26360108v1.10","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC58623.2025.11253463"},{"key":"2026081412101226000_2026.08.10.26360108v1.11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01519-z"},{"key":"2026081412101226000_2026.08.10.26360108v1.12","doi-asserted-by":"publisher","DOI":"10.1111\/codi.70436"},{"key":"2026081412101226000_2026.08.10.26360108v1.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2024.104662"},{"key":"2026081412101226000_2026.08.10.26360108v1.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01377-1"},{"key":"2026081412101226000_2026.08.10.26360108v1.15","doi-asserted-by":"publisher","DOI":"10.4258\/hir.2025.31.3.218"},{"key":"2026081412101226000_2026.08.10.26360108v1.16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpi.2025.100520"},{"key":"2026081412101226000_2026.08.10.26360108v1.17","doi-asserted-by":"publisher","DOI":"10.2196\/90139"},{"key":"2026081412101226000_2026.08.10.26360108v1.18","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2025.106205"},{"key":"2026081412101226000_2026.08.10.26360108v1.19","doi-asserted-by":"publisher","DOI":"10.1177\/20552076251337177"},{"key":"2026081412101226000_2026.08.10.26360108v1.20","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-026-04431-5"},{"key":"2026081412101226000_2026.08.10.26360108v1.21","doi-asserted-by":"publisher","DOI":"10.1038\/s44401-026-00115-x"},{"key":"2026081412101226000_2026.08.10.26360108v1.22","doi-asserted-by":"publisher","DOI":"10.2196\/88766"},{"key":"2026081412101226000_2026.08.10.26360108v1.23","doi-asserted-by":"publisher","DOI":"10.1097\/GOX.0000000000007176"},{"key":"2026081412101226000_2026.08.10.26360108v1.24","doi-asserted-by":"publisher","DOI":"10.1177\/01939459261451723"},{"key":"2026081412101226000_2026.08.10.26360108v1.25","doi-asserted-by":"publisher","DOI":"10.5195\/jmla.2025.1985"},{"key":"2026081412101226000_2026.08.10.26360108v1.26","doi-asserted-by":"publisher","DOI":"10.5195\/jmla.2026.2333"},{"key":"2026081412101226000_2026.08.10.26360108v1.27","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2024-083305"},{"key":"2026081412101226000_2026.08.10.26360108v1.28","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.30220"},{"key":"2026081412101226000_2026.08.10.26360108v1.29","unstructured":"Westrick J , Juarez L , Hilton S. PubMed vs artificial intelligence: comparison of search results [Internet]. Medical Library Association; 2025 [cited 10 Aug 2026]. <https:\/\/library.rush.edu\/librarian-scholarly-research\/mla-2025-ai>."},{"key":"2026081412101226000_2026.08.10.26360108v1.30","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-025-06294-7"},{"key":"2026081412101226000_2026.08.10.26360108v1.31","doi-asserted-by":"publisher","DOI":"10.1590\/1806-9282.20230560"},{"key":"2026081412101226000_2026.08.10.26360108v1.32","doi-asserted-by":"crossref","unstructured":"Van IJzendoorn DGP , Habets PC , Vinkers CH , Otte WM . Clinical study type classification, validation, and PubMed filter comparison with natural language processing and active learning. medRxiv [Preprint]; 2022 [cited 10 Aug 2026]. Available from: http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2022.11.01.22281685.","DOI":"10.1101\/2022.11.01.22281685"},{"issue":"2","key":"2026081412101226000_2026.08.10.26360108v1.33","first-page":"177","article-title":"Information needs of clinical teams: analysis of questions received by the Clinical Informatics Consult Service","volume":"89","year":"2001","journal-title":"Bull Med Libr Assoc"},{"key":"2026081412101226000_2026.08.10.26360108v1.34","unstructured":"GovTech Data Science & AI Division. Prompt engineering playbook (Beta v3) [Internet]. Government of Singapore; 30 Aug 2023 [cited 10 Aug 2026]. <https:\/\/www.developer.tech.gov.sg\/products\/collections\/data-science-and-artificial-intelligence\/playbooks\/prompt-engineering-playbook-beta-v3.pdf>."},{"key":"2026081412101226000_2026.08.10.26360108v1.35","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2008.08.010"},{"key":"2026081412101226000_2026.08.10.26360108v1.36","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103208"},{"key":"2026081412101226000_2026.08.10.26360108v1.37","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaa012"},{"issue":"4","key":"2026081412101226000_2026.08.10.26360108v1.38","first-page":"445","article-title":"A study of clinical questions posed by hospital clinicians","volume":"92","year":"2004","journal-title":"J Med Libr Assoc JMLA"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.08.10.26360108","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T19:10:17Z","timestamp":1786734617000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.08.10.26360108"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,12]]},"references-count":38,"URL":"https:\/\/doi.org\/10.64898\/2026.08.10.26360108","relation":{},"subject":[],"published":{"date-parts":[[2026,8,12]]},"subtype":"preprint"}}