{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T22:53:58Z","timestamp":1791327238001,"version":"4.3.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T00:00:00Z","timestamp":1781049600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:00:00Z","timestamp":1791331200000},"content-version":"vor","delay-in-days":119,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"The InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government."}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Vaccines"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Large language models (LLMs) are increasingly used by clinicians and the public for vaccine information, yet their factual accuracy across languages and vaccine domains remains insufficiently characterized. We evaluated 13 LLMs using VaxEval, a multilingual vaccine-knowledge benchmark of 1886 vaccine-related multiple-choice questions spanning 14 vaccines in English (71%), Spanish (13%), and Chinese (16%). All items underwent quality control, with reference answers verified against authoritative guidance and peer-reviewed sources. Model performance was evaluated under zero-shot, few-shot, and chain-of-thought (CoT) prompting, with exact-match accuracy defined as selecting the pre-specified reference option. We used mixed-effect logistic regression to estimate associations between model group (newer flagship models vs earlier models), prompting strategy, language, and vaccine type, and answer correctness. Mean accuracy across models was 86.0% in English, 83.7% in Spanish, and 80.0% in Chinese. Flagship models had higher odds of correctness than earlier versions (OR 1.57; 95% CI 1.50\u20131.65;\n                    <jats:italic>P<\/jats:italic>\n                    \u2009&lt;\u2009.001). Few-shot prompting was associated with higher correctness (OR 1.17;\n                    <jats:italic>P<\/jats:italic>\n                    \u2009&lt;\u2009.001), whereas CoT prompting was associated with lower correctness (OR 0.79;\n                    <jats:italic>P<\/jats:italic>\n                    \u2009&lt;\u2009.001). Performance varied by vaccine type and question category, underscoring the need for rigorous evaluation, structured guardrails, and targeted refinement before using LLMs for vaccine communication.\n                  <\/jats:p>","DOI":"10.1038\/s41541-026-01500-1","type":"journal-article","created":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T20:13:18Z","timestamp":1781122398000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating large language models on multilingual vaccine knowledge: a benchmark study"],"prefix":"10.1038","volume":"11","author":[{"given":"Siyuan","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lily","family":"Wass","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengdong","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucas","family":"Garay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9","family":"Vizoso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathy","family":"Leung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leesa","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,10]]},"reference":[{"key":"1500_CR1","doi-asserted-by":"publisher","unstructured":"Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2303.13375 (2023).","DOI":"10.48550\/arXiv.2303.13375"},{"key":"1500_CR2","doi-asserted-by":"publisher","first-page":"e31","DOI":"10.1016\/j.bja.2023.04.017","volume":"131","author":"D Shay","year":"2023","unstructured":"Shay, D. et al. Assessment of ChatGPT success with specialty medical knowledge using anaesthesiology board examination practice questions. Br. J. Anaesth. 131, e31\u2013e34 (2023).","journal-title":"Br. J. Anaesth."},{"key":"1500_CR3","first-page":"86","volume":"26","author":"DJ Ozeri","year":"2024","unstructured":"Ozeri, D. J. et al. Performance of ChatGPT in Israeli Hebrew Internal Medicine National Residency Exam. Isr. Med. Assoc. J. 26, 86\u201388 (2024).","journal-title":"Isr. Med. Assoc. J."},{"key":"1500_CR4","doi-asserted-by":"publisher","first-page":"113","DOI":"10.3390\/bioengineering13010113","volume":"13","author":"L Lian","year":"2026","unstructured":"Lian, L. et al. Large language models evaluation of medical licensing examination using GPT-4.0, ERNIE Bot 4.0, and GPT-4o. Bioengineering 13, 113 (2026).","journal-title":"Bioengineering"},{"key":"1500_CR5","doi-asserted-by":"crossref","unstructured":"Roy, S. et al. Beyond accuracy: investigating error types in GPT-4 responses to USMLE questions. In Proc. 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM, 2024).","DOI":"10.1145\/3626772.3657882"},{"key":"1500_CR6","unstructured":"UnitedHealth Group. Improving health literacy could prevent nearly 1 million hospital visits and save over $25 billion a year. UnitedHealth Group. https:\/\/www.unitedhealthgroup.com\/newsroom\/research-reports\/research-library.html (2020)"},{"key":"1500_CR7","doi-asserted-by":"crossref","unstructured":"Nazi, Z. A. & Peng, W. Large language models in healthcare and medical domain: a review. Informatics 11, 57 (2024).","DOI":"10.3390\/informatics11030057"},{"key":"1500_CR8","unstructured":"Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. (Anthropic, 2024)."},{"key":"1500_CR9","doi-asserted-by":"crossref","unstructured":"Islam, R. & Moushi, O. M. GPT-4o: The Cutting-Edge Advancement in Multimodal LLM. (Authorea Preprints, 2024).","DOI":"10.36227\/techrxiv.171986596.65533294\/v1"},{"key":"1500_CR10","doi-asserted-by":"crossref","unstructured":"DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. Nature 645, 633\u2013638 (2025).","DOI":"10.1038\/s41586-025-09422-z"},{"key":"1500_CR11","doi-asserted-by":"publisher","unstructured":"Liu, A. et al. Deepseek-v3 technical report. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2412.19437 (2024).","DOI":"10.48550\/arXiv.2412.19437"},{"key":"1500_CR12","unstructured":"World Health Organization. Ten Threats to Global Health in 2019. (World Health Organization, 2019)."},{"key":"1500_CR13","unstructured":"Sidoti, O. & McClain, C. 34% of U.S. Adults Have Used ChatGPT, About Double the Share in 2023. (Pew Research Center, 2025)."},{"key":"1500_CR14","unstructured":"KFF. KFF Health Misinformation Tracking Poll: Artificial Intelligence and Health Information. (KFF, 2024)."},{"key":"1500_CR15","doi-asserted-by":"publisher","first-page":"e60939","DOI":"10.2196\/60939","volume":"8","author":"S Joshi","year":"2024","unstructured":"Joshi, S. et al. Ensuring accuracy and equity in vaccination information from ChatGPT and CDC: mixed-methods cross-language evaluation. JMIR Form. Res. 8, e60939 (2024).","journal-title":"JMIR Form. Res."},{"key":"1500_CR16","doi-asserted-by":"crossref","unstructured":"Deiana, G. et al. Artificial intelligence and public health: evaluating ChatGPT responses to vaccination myths and misconceptions. Vaccines 11, 1217 (2023).","DOI":"10.3390\/vaccines11071217"},{"key":"1500_CR17","doi-asserted-by":"publisher","first-page":"1360597","DOI":"10.3389\/fpubh.2024.1360597","volume":"12","author":"SA Meo","year":"2024","unstructured":"Meo, S. A. et al. Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance. Front. Public Health 12, 1360597 (2024).","journal-title":"Front. Public Health"},{"key":"1500_CR18","doi-asserted-by":"publisher","unstructured":"Wu, J. et al. Why chain of thought fails in clinical text understanding. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2509.21933 (2025).","DOI":"10.48550\/arXiv.2509.21933"},{"key":"1500_CR19","doi-asserted-by":"publisher","unstructured":"Meincke, L., Mollick, E., Mollick, L. & Shapiro, D. Prompting Science Report 2: the decreasing value of chain of thought in prompting. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2506.07142 (2025).","DOI":"10.48550\/arXiv.2506.07142"},{"key":"1500_CR20","doi-asserted-by":"publisher","unstructured":"Son, G. et al. MM-Eval: a multilingual meta-evaluation benchmark for LLM-as-a-judge and reward models. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2410.17578 (2024).","DOI":"10.48550\/arXiv.2410.17578"},{"key":"1500_CR21","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1515\/jtc-2023-0019","volume":"3","author":"Z Liu","year":"2023","unstructured":"Liu, Z. Cultural bias in large language models: a comprehensive analysis and mitigation strategies. J. Transcult. Commun. 3, 224\u2013244 (2023).","journal-title":"J. Transcult. Commun."},{"key":"1500_CR22","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1093\/pnasnexus\/pgae346","volume":"3","author":"Y Tao","year":"2024","unstructured":"Tao, Y. et al. Cultural bias and cultural alignment of large language models. PNAS Nexus 3, 346 (2024).","journal-title":"PNAS Nexus"},{"key":"1500_CR23","doi-asserted-by":"crossref","unstructured":"Baik, R. L. et al. Adapting communication styles in health chatbot using large language models to support family caregivers from multicultural backgrounds. In Proc. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (ACM, 2025).","DOI":"10.1145\/3706599.3719711"},{"key":"1500_CR24","doi-asserted-by":"publisher","first-page":"e51421","DOI":"10.2196\/51421","volume":"9","author":"F Baglivo","year":"2023","unstructured":"Baglivo, F. et al. Exploring the possible use of AI chatbots in public health education: feasibility study. JMIR Med. Educ. 9, e51421 (2023).","journal-title":"JMIR Med. Educ."},{"key":"1500_CR25","doi-asserted-by":"publisher","first-page":"126559","DOI":"10.1016\/j.vaccine.2024.126559","volume":"44","author":"C Cosma","year":"2025","unstructured":"Cosma, C. et al. Exploring Chatbot contributions to enhancing vaccine literacy and uptake: a scoping review of the literature. Vaccine 44, 126559 (2025).","journal-title":"Vaccine"},{"key":"1500_CR26","doi-asserted-by":"crossref","unstructured":"Hou, Z. et al. A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial. Nat. Med. 31, 1855\u20131862 (2025).","DOI":"10.1038\/s41591-025-03618-6"},{"key":"1500_CR27","doi-asserted-by":"publisher","unstructured":"Team, G. et al. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2403.05530 (2024).","DOI":"10.48550\/arXiv.2403.05530"},{"key":"1500_CR28","doi-asserted-by":"publisher","unstructured":"Achiam, J. et al. Gpt-4 technical report. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2303.08774 (2023).","DOI":"10.48550\/arXiv.2303.08774"},{"key":"1500_CR29","doi-asserted-by":"publisher","first-page":"192","DOI":"10.3390\/fi15060192","volume":"15","author":"KI Roumeliotis","year":"2023","unstructured":"Roumeliotis, K. I. & Tselikas, N. D. ChatGPT and open-AI models: a preliminary review. Future Internet 15, 192 (2023).","journal-title":"Future Internet"},{"key":"1500_CR30","doi-asserted-by":"publisher","unstructured":"Young, A. et al. Yi: open foundation models by 01. AI. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2403.04652 (2024).","DOI":"10.48550\/arXiv.2403.04652"},{"key":"1500_CR31","doi-asserted-by":"publisher","unstructured":"Ormazabal, A. et al. Reka core, flash, and edge: a series of powerful multimodal language models. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2404.12387 (2024).","DOI":"10.48550\/arXiv.2404.12387"},{"key":"1500_CR32","doi-asserted-by":"publisher","unstructured":"Yang, A. et al. Qwen3 technical report. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2505.09388 (2025).","DOI":"10.48550\/arXiv.2505.09388"},{"key":"1500_CR33","doi-asserted-by":"publisher","unstructured":"Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2302.13971 (2023).","DOI":"10.48550\/arXiv.2302.13971"},{"key":"1500_CR34","doi-asserted-by":"publisher","unstructured":"Zeng, A. et al. GLM-4-Voice: towards intelligent and human-like end-to-end spoken chatbot. Preprint at arXiv https:\/\/doi.org\/10.48550\/arXiv.2412.02612 (2024).","DOI":"10.48550\/arXiv.2412.02612"},{"key":"1500_CR35","doi-asserted-by":"crossref","unstructured":"Arlson, K. W. Large Language Models Pass the Turing Test. SuperIntelligence-Robotics-Safety & Alignment, 2 (2025).","DOI":"10.70777\/si.v2i2.14697"},{"key":"1500_CR36","doi-asserted-by":"publisher","first-page":"e76925","DOI":"10.2196\/76925","volume":"11","author":"H Sato","year":"2025","unstructured":"Sato, H., Ogasawara, K. & Sakurai, H. Performance evaluation of 18 generative AI models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: comparative study. JMIR Med. Educ. 11, e76925 (2025).","journal-title":"JMIR Med. Educ."},{"key":"1500_CR37","unstructured":"Adila, D., Shin, C., Cai, L., & Sala, F. Zero-Shot Robustification of Zero-Shot Models with Auxiliary Foundation Models."},{"key":"1500_CR38","unstructured":"Brown, T. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems Vol. 33, 1877\u20131901 (Curran Associates, 2020)."},{"key":"1500_CR39","doi-asserted-by":"crossref","unstructured":"Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In Proc. Advances in Neural Information Processing Systems Vol. 35, 24824\u201324837 (Curran Associates, 2022).","DOI":"10.52202\/068431-1800"},{"key":"1500_CR40","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-025-04151-2","volume":"32","author":"S Bedi","year":"2026","unstructured":"Bedi, S. et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nat. Med. 32, 943\u2013951 (2026).","journal-title":"Nat. Med."},{"key":"1500_CR41","doi-asserted-by":"publisher","first-page":"31907","DOI":"10.1007\/s11042-021-11222-2","volume":"80","author":"B Das","year":"2021","unstructured":"Das, B. et al. Multiple-choice question generation with auto-generated distractors for computer-assisted educational assessment. Multimed. Tools Appl. 80, 31907\u201331925 (2021).","journal-title":"Multimed. Tools Appl."},{"key":"1500_CR42","doi-asserted-by":"crossref","unstructured":"Doughty, J. et al. A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. In Proc. 26th Australasian Computing Education Conference (ACM, 2024).","DOI":"10.1145\/3636243.3636256"}],"container-title":["npj Vaccines"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41541-026-01500-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41541-026-01500-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41541-026-01500-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T22:48:40Z","timestamp":1791326920000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41541-026-01500-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,10]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["1500"],"URL":"https:\/\/doi.org\/10.1038\/s41541-026-01500-1","relation":{},"ISSN":["2059-0105"],"issn-type":[{"value":"2059-0105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,10]]},"assertion":[{"value":"29 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Ethics"}}],"article-number":"187"}}