{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T13:39:58Z","timestamp":1794577198377,"version":"4.9.1"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T00:00:00Z","timestamp":1741046400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T00:00:00Z","timestamp":1741046400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["493659010"],"award-info":[{"award-number":["493659010"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Effectively managing evidence-based information is increasingly challenging. This study tested large language models (LLMs), including document- and online-enabled retrieval-augmented generation (RAG) systems, using 13 recent neurology guidelines across 130 questions. Results showed substantial variability. RAG improved accuracy compared to base models but still produced potentially harmful answers. RAG-based systems performed worse on case-based than knowledge-based questions. Further refinement and improved regulation is needed for safe clinical integration of RAG-enhanced LLMs.<\/jats:p>","DOI":"10.1038\/s41746-025-01536-y","type":"journal-article","created":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T07:53:16Z","timestamp":1741074796000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":56,"title":["Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology"],"prefix":"10.1038","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2496-1415","authenticated-orcid":false,"given":"Lars","family":"Masanneck","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sven G.","family":"Meuth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3080-2277","authenticated-orcid":false,"given":"Marc","family":"Pawlitzki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,3,4]]},"reference":[{"key":"1536_CR1","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1016\/S1474-4422(24)00038-3","volume":"23","author":"JD Steinmetz","year":"2024","unstructured":"Steinmetz, J. D. et al. Global, regional, and national burden of disorders affecting the nervous system, 1990\u20132021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet Neurol. 23, 344\u2013381 (2024).","journal-title":"Lancet Neurol."},{"key":"1536_CR2","doi-asserted-by":"publisher","first-page":"e184852","DOI":"10.1001\/jamanetworkopen.2018.4852","volume":"1","author":"M Tonelli","year":"2018","unstructured":"Tonelli, M. et al. Comparison of the Complexity of Patients Seen by Different Medical Subspecialists in a Universal Health Care System. JAMA Netw. Open 1, e184852 (2018).","journal-title":"JAMA Netw. Open"},{"key":"1536_CR3","doi-asserted-by":"publisher","DOI":"10.1111\/ene.16237","volume":"31","author":"CLA Bassetti","year":"2024","unstructured":"Bassetti, C. L. A. et al. General neurology: Current challenges and future implications. Eur. J. Neurol. 31, e16237 (2024).","journal-title":"Eur. J. Neurol."},{"key":"1536_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s43856-023-00370-1","volume":"3","author":"J Clusmann","year":"2023","unstructured":"Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Med 3, 1\u20138 (2023).","journal-title":"Commun. Med"},{"key":"1536_CR5","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health 2, e0000198 (2023).","journal-title":"PLOS Digital Health"},{"key":"1536_CR6","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","volume":"183","author":"JW Ayers","year":"2023","unstructured":"Ayers, J. W. et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern. Med. 183, 589\u2013596 (2023).","journal-title":"JAMA Intern. Med."},{"key":"1536_CR7","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2300145","volume":"1","author":"FN Mirza","year":"2024","unstructured":"Mirza, F. N. et al. Using ChatGPT to Facilitate Truly Informed Medical Consent. NEJM AI 1, AIcs2300145 (2024).","journal-title":"NEJM AI"},{"key":"1536_CR8","first-page":"AIp2300031","volume":"1","author":"AV Eriksen","year":"2023","unstructured":"Eriksen, A. V., M\u00f6ller, S. & Ryg, J. Use of GPT-4 to Diagnose Complex Clinical Cases. NEJM AI 1, AIp2300031 (2023).","journal-title":"NEJM AI"},{"key":"1536_CR9","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1001\/jamapediatrics.2023.5750","volume":"178","author":"J Barile","year":"2024","unstructured":"Barile, J. et al. Diagnostic Accuracy of a Large Language Model in Pediatric Case Studies. JAMA Pediatrics 178, 313\u2013315 (2024).","journal-title":"JAMA Pediatrics"},{"key":"1536_CR10","doi-asserted-by":"publisher","first-page":"e53297","DOI":"10.2196\/53297","volume":"26","author":"L Masanneck","year":"2024","unstructured":"Masanneck, L. et al. Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study. J. Med. Internet Res. 26, e53297 (2024).","journal-title":"J. Med. Internet Res."},{"key":"1536_CR11","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-46411-8","volume":"15","author":"S Sandmann","year":"2024","unstructured":"Sandmann, S., Riepenhausen, S., Plagwitz, L. & Varghese, J. Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks. Nat. Commun. 15, 2050 (2024).","journal-title":"Nat. Commun."},{"key":"1536_CR12","doi-asserted-by":"publisher","unstructured":"Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2005.11401 (2021).","DOI":"10.48550\/arXiv.2005.11401"},{"key":"1536_CR13","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2300068","volume":"1","author":"C Zakka","year":"2024","unstructured":"Zakka, C. et al. Almanac \u2014 Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI 1, AIoa2300068 (2024).","journal-title":"NEJM AI"},{"key":"1536_CR14","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2300235","volume":"1","author":"D Ferber","year":"2024","unstructured":"Ferber, D. et al. GPT-4 for Information Retrieval and Comparison of Medical Oncology Guidelines. NEJM AI 1, AIcs2300235 (2024).","journal-title":"NEJM AI"},{"key":"1536_CR15","unstructured":"Chatbot Arena (formerly LMSYS): Free AI Chat to Compare & Test Best AI Chatbots. https:\/\/lmarena.ai (2024)."},{"key":"1536_CR16","unstructured":"The Federation of State Medical Boards. Recommendations on the Responsible and Ethical Incorporation of AI into Clinical Practice. https:\/\/www.fsmb.org\/advocacy\/news-releases\/fsmb-releases-recommendations-on-the-responsible-and-ethical-incorporation-of-ai-into-clinical-practice\/ (2024)."},{"key":"1536_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-024-01116-6","volume":"7","author":"S Gilbert","year":"2024","unstructured":"Gilbert, S. The EU passes the AI Act and its implications for digital medicine are unclear. npj Digit. Med. 7, 1\u20133 (2024).","journal-title":"npj Digit. Med."},{"key":"1536_CR18","unstructured":"Home. Perplexity https:\/\/docs.perplexity.ai\/home (2024)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01536-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01536-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01536-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T07:53:19Z","timestamp":1741074799000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01536-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,4]]},"references-count":18,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["1536"],"URL":"https:\/\/doi.org\/10.1038\/s41746-025-01536-y","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,4]]},"assertion":[{"value":"25 June 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 February 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 March 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"L.M. reports no conflicts of interest related to this study. He reports honoraria for lecturing, consulting and travel expenses for attending meetings from Biogen, Merck Serono, argenX, Roche, Alexion and Novartis, all outside the scope of this work. His research is funded by the by the German Multiple Sclerosis Foundation (DMSG) and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) \u2013 493659010. S.G.M. reports no conflicts of interest related to this study. He reports honoraria for lecturing and travel expenses for attending meetings from Almirall, Amicus Therapeutics Germany, Bayer Health Care, Biogen, Celgene, Diamed, Genzyme, MedDay Pharmaceuticals, Merck Serono, Novartis, Novo Nordisk, ONO Pharma, Roche, Sanofi-Aventis, Chugai Pharma, QuintilesIMS, and Teva. His research is funded by the German Ministry for Education and Research (BMBF), Bundesinstitut f\u00fcr Risikobewertung (BfR), Deutsche Forschungsgemeinschaft (DFG), Else Kr\u00f6ner Fresenius Foundation, Gemeinsamer Bundesausschuss (G-BA), German Academic Exchange Service, Hertie Foundation, Interdisciplinary Center for Clinical Studies (IZKF) Muenster, German Foundation Neurology and by Alexion, Almirall, Amicus Therapeutics Germany, Biogen, Diamed, Fresenius Medical Care, Genzyme, HERZ Burgdorf, Merck Serono, Novartis, ONO Pharma, Roche, and Teva, all outside the scope of this study. M.P. reports no conflicts of interest related to this study. He received honoraria for lecturing and travel expenses for attending meetings from Alexion, ArgenX, Bayer Health Care, Biogen, Hexal, Merck Serono, Novartis, Roche, Sanofi-Aventis, Takedaand Teva. His research is funded by the by ArgenX, Biogen, Hexal and Novartis, all outside the scope of this study.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"137"}}