{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T17:17:11Z","timestamp":1778260631643,"version":"3.51.4"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T00:00:00Z","timestamp":1753747200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T00:00:00Z","timestamp":1753747200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Yushan Fellow Program by Ministry of Education (MOE), Taiwan : John Tayu Lee","award":["NTU-114V1044-2"],"award-info":[{"award-number":["NTU-114V1044-2"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Stroke is a leading cause of global morbidity and mortality, disproportionately impacting lower socioeconomic groups. In this study, we evaluated three generative LLMs\u2014GPT, Claude, and Gemini\u2014across four stages of stroke care: prevention, diagnosis, treatment, and rehabilitation. Using three prompt engineering techniques\u2014Zero-Shot Learning (ZSL), Chain of Thought (COT), and Talking Out Your Thoughts (TOT)\u2014we applied each to realistic stroke scenarios. Clinical experts assessed the outputs across five domains: (1) accuracy; (2) hallucinations; (3) specificity; (4) empathy; and (5) actionability, based on clinical competency benchmarks. Overall, the LLMs demonstrated suboptimal performance with inconsistent scores across domains. Each prompt engineering method showed strengths in specific areas: TOT does well in empathy and actionability, COT was strong in structured reasoning during diagnosis, and ZSL provided concise, accurate responses with fewer hallucinations, especially in the Treatment stage. However, none consistently met high clinical standards across all stroke care stages.<\/jats:p>","DOI":"10.1038\/s41746-025-01830-9","type":"journal-article","created":{"date-parts":[[2025,7,29]],"date-time":"2025-07-29T09:02:59Z","timestamp":1753779779000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Evaluation of performance of generative large language models for stroke care"],"prefix":"10.1038","volume":"8","author":[{"given":"John Tayu","family":"Lee","sequence":"first","affiliation":[]},{"given":"Vincent Cheng-Sheng","family":"Li","sequence":"additional","affiliation":[]},{"given":"Jia-Jyun","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Hsiao-Hui","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Sophia Sin-Yu","family":"Su","sequence":"additional","affiliation":[]},{"given":"Brian Pin-Hsuan","family":"Chang","sequence":"additional","affiliation":[]},{"given":"Richard Lee","family":"Lai","sequence":"additional","affiliation":[]},{"given":"Chi-Hung","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Chung-Ting","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Valis","family":"Tanapima","sequence":"additional","affiliation":[]},{"given":"Toby Kai-Bo","family":"Shen","sequence":"additional","affiliation":[]},{"given":"Rifat","family":"Atun","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,7,29]]},"reference":[{"key":"1830_CR1","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1016\/S1474-4422(24)00369-7","volume":"23","author":"GBD 2021 Stroke Risk Factor Collaborators.","year":"2024","unstructured":"GBD 2021 Stroke Risk Factor Collaborators. Global, regional, and national burden of stroke and its risk factors, 1990-2021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet Neurol. 23, 973\u20131003 (2024).","journal-title":"Lancet Neurol."},{"key":"1830_CR2","unstructured":"WHO. Tackling NCDs: Best Buys and Other Recommended Interventions for the Prevention and Control of Noncommunicable Diseases 2nd edn (WHO, 2024)."},{"key":"1830_CR3","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2023.102028","volume":"59","author":"eClinicalMedicine.","year":"2023","unstructured":"eClinicalMedicine. The rising global burden of stroke. EClinicalMedicine 59, 102028 (2023).","journal-title":"EClinicalMedicine"},{"key":"1830_CR4","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1161\/STROKEAHA.117.019358","volume":"49","author":"RV Krishnamurthi","year":"2018","unstructured":"Krishnamurthi, R. V. et al. Stroke incidence by major pathological type and ischemic subtypes in the Auckland Regional Community Stroke Studies: changes between 2002 and 2011. Stroke 49, 3\u201310 (2018).","journal-title":"Stroke"},{"key":"1830_CR5","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1161\/01.STR.24.1.35","volume":"24","author":"HP Adams Jr.","year":"1993","unstructured":"Adams, H. P. Jr. et al. Classification of subtype of acute ischemic stroke. Definitions for use in a multicenter clinical trial. TOAST. Trial of Org 10172 in acute stroke treatment. Stroke 24, 35\u201341 (1993).","journal-title":"Stroke"},{"key":"1830_CR6","doi-asserted-by":"publisher","first-page":"1186","DOI":"10.1161\/STROKEAHA.111.639732","volume":"43","author":"J Addo","year":"2012","unstructured":"Addo, J. et al. Socioeconomic status and stroke: an updated review. Stroke 43, 1186\u20131191 (2012).","journal-title":"Stroke"},{"key":"1830_CR7","doi-asserted-by":"publisher","first-page":"e185","DOI":"10.1016\/S2468-2667(18)30030-6","volume":"3","author":"BD Bray","year":"2018","unstructured":"Bray, B. D. et al. Socioeconomic disparities in first stroke incidence, quality of care, and survival: a nationwide registry-based cohort study of 44 million adults in England. Lancet Public Health 3, e185\u2013e193 (2018).","journal-title":"Lancet Public Health"},{"key":"1830_CR8","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/S1474-4422(06)70351-9","volume":"5","author":"AM Cox","year":"2006","unstructured":"Cox, A. M., McKevitt, C., Rudd, A. G. & Wolfe, C. D. Socioeconomic status and stroke. Lancet Neurol. 5, 181\u2013188 (2006).","journal-title":"Lancet Neurol."},{"key":"1830_CR9","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1038\/s41746-023-00988-4","volume":"7","author":"MM Raza","year":"2024","unstructured":"Raza, M. M., Venkatesh, K. P. & Kvedar, J. C. Generative AI and large language models in health care: pathways to implementation. NPJ Digit. Med. 7, 62 (2024).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR10","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1093\/postmj\/qgae122","volume":"101","author":"Z Strika","year":"2024","unstructured":"Strika, Z., Petkovic, K., Likic, R. & Batenburg, R. Bridging healthcare gaps: a scoping review on the role of artificial intelligence, deep learning, and large language models in alleviating problems in medical deserts. Postgrad. Med. J. 101, 4\u201316 (2024).","journal-title":"Postgrad. Med. J."},{"key":"1830_CR11","doi-asserted-by":"publisher","first-page":"148","DOI":"10.3928\/01913913-20240124-02","volume":"61","author":"JS Chen","year":"2024","unstructured":"Chen, J. S. & Granet, D. B. Prompt engineering: helping ChatGPT respond better to patients and parents. J. Pedia. Ophthalmol. Strabismus 61, 148\u2013150 (2024).","journal-title":"J. Pedia. Ophthalmol. Strabismus"},{"key":"1830_CR12","doi-asserted-by":"publisher","first-page":"104770","DOI":"10.1016\/j.ebiom.2023.104770","volume":"95","author":"ZW Lim","year":"2023","unstructured":"Lim, Z. W. et al. Benchmarking large language models\u2019 performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine 95, 104770 (2023).","journal-title":"EBioMedicine"},{"key":"1830_CR13","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"1830_CR14","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","volume":"9","author":"A Gilson","year":"2023","unstructured":"Gilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9, e45312 (2023).","journal-title":"JMIR Med. Educ."},{"key":"1830_CR15","doi-asserted-by":"publisher","first-page":"829","DOI":"10.1111\/aos.15661","volume":"101","author":"I Potapenko","year":"2023","unstructured":"Potapenko, I. et al. Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol. 101, 829\u2013831 (2023).","journal-title":"Acta Ophthalmol."},{"key":"1830_CR16","doi-asserted-by":"publisher","DOI":"10.1016\/j.xops.2023.100324","volume":"3","author":"F Antaki","year":"2023","unstructured":"Antaki, F., Touma, S., Milad, D., El-Khoury, J. & Duval, R. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol. Sci. 3, 100324 (2023).","journal-title":"Ophthalmol. Sci."},{"key":"1830_CR17","doi-asserted-by":"publisher","first-page":"e60501","DOI":"10.2196\/60501","volume":"26","author":"J Zaghir","year":"2024","unstructured":"Zaghir, J. et al. Prompt engineering paradigms for medical applications: scoping review. J. Med. Internet Res. 26, e60501 (2024).","journal-title":"J. Med. Internet Res."},{"key":"1830_CR18","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1038\/s41746-024-01029-4","volume":"7","author":"L Wang","year":"2024","unstructured":"Wang, L. et al. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. NPJ Digit. Med. 7, 41 (2024).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR19","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2023.1161098","volume":"5","author":"J Au Yeung","year":"2023","unstructured":"Au Yeung, J. et al. AI chatbots not yet ready for clinical use. Front. Digit. Health 5, 1161098 (2023).","journal-title":"Front. Digit. Health"},{"key":"1830_CR20","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","volume":"5","author":"X Yang","year":"2022","unstructured":"Yang, X. et al. A large language model for electronic health records. NPJ Digit. Med. 5, 194 (2022).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR21","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s41746-024-01010-1","volume":"7","author":"T Savage","year":"2024","unstructured":"Savage, T., Nayak, A., Gallo, R., Rangan, E. & Chen, J. H. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digit. Med. 7, 20 (2024).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR22","doi-asserted-by":"publisher","first-page":"e0000341","DOI":"10.1371\/journal.pdig.0000341","volume":"3","author":"AJ Thirunavukarasu","year":"2024","unstructured":"Thirunavukarasu, A. J. et al. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: a head-to-head cross-sectional study. PLoS Digit. Health 3, e0000341 (2024).","journal-title":"PLoS Digit. Health"},{"key":"1830_CR23","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1038\/s41746-025-01604-3","volume":"8","author":"J Kim","year":"2025","unstructured":"Kim, J. et al. Artificial intelligence tools in supporting healthcare professionals for tailored patient care. NPJ Digit. Med. 8, 210 (2025).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR24","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41746-025-01472-x","volume":"8","author":"H Ding","year":"2025","unstructured":"Ding, H. et al. Evaluation and practical application of prompt-driven ChatGPTs for EMR generation. NPJ Digit. Med. 8, 77 (2025).","journal-title":"NPJ Digit. Med."},{"key":"1830_CR25","doi-asserted-by":"publisher","first-page":"e428","DOI":"10.1016\/S2589-7500(24)00061-X","volume":"6","author":"JCL Ong","year":"2024","unstructured":"Ong, J. C. L. et al. Ethical and regulatory challenges of large language models in medicine. Lancet Digit. Health 6, e428\u2013e432 (2024).","journal-title":"Lancet Digit. Health"},{"key":"1830_CR26","doi-asserted-by":"publisher","first-page":"e364","DOI":"10.1161\/STR.0000000000000375","volume":"52","author":"DO Kleindorfer","year":"2021","unstructured":"Kleindorfer, D. O. et al. 2021 Guideline for the prevention of stroke in patients with stroke and transient ischemic attack: a guideline from the American Heart Association\/American Stroke Association. Stroke 52, e364\u2013e467 (2021).","journal-title":"Stroke"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01830-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01830-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01830-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,8]],"date-time":"2025-09-08T02:23:32Z","timestamp":1757298212000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01830-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,29]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["1830"],"URL":"https:\/\/doi.org\/10.1038\/s41746-025-01830-9","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,29]]},"assertion":[{"value":"15 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"481"}}