{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T12:18:22Z","timestamp":1793449102766,"version":"4.5.0"},"reference-count":58,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T00:00:00Z","timestamp":1768867200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T00:00:00Z","timestamp":1768867200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Med"],"published-print":{"date-parts":[[2026,3]]},"DOI":"10.1038\/s41591-025-04151-2","type":"journal-article","created":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T10:02:36Z","timestamp":1768903356000},"page":"943-951","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":73,"title":["Holistic evaluation of large language models for medical tasks with MedHELM"],"prefix":"10.1038","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4502-4191","authenticated-orcid":false,"given":"Suhana","family":"Bedi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hejie","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1930-9529","authenticated-orcid":false,"given":"Miguel","family":"Fuentes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alyssa","family":"Unell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Wornow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8499-824X","authenticated-orcid":false,"given":"Juan M.","family":"Banda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikesh","family":"Kotecha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0423-9679","authenticated-orcid":false,"given":"Timothy","family":"Keyes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7270-2607","authenticated-orcid":false,"given":"Yifan","family":"Mai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mert","family":"Oez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shrey","family":"Jain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8373-7140","authenticated-orcid":false,"given":"Leonardo","family":"Schettini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehr","family":"Kashyap","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9316-5768","authenticated-orcid":false,"given":"Jason Alan","family":"Fries","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akshay","family":"Swaminathan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1194-7510","authenticated-orcid":false,"given":"Philip","family":"Chung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8874-8835","authenticated-orcid":false,"given":"Fateme Nateghi","family":"Haredasht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0246-2180","authenticated-orcid":false,"given":"Ivan","family":"Lopez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2120-5722","authenticated-orcid":false,"given":"Asad","family":"Aali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriel","family":"Tse","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashwin","family":"Nayak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5034-0699","authenticated-orcid":false,"given":"Shivam","family":"Vedak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0592-1453","authenticated-orcid":false,"given":"Sneha S.","family":"Jain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Birju","family":"Patel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oluseyi","family":"Fayanju","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shreya","family":"Shah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7491-4257","authenticated-orcid":false,"given":"Ethan","family":"Goh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dong-han","family":"Yao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Soetikno","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eduardo","family":"Reis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sergios","family":"Gatidis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vasu","family":"Divi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2645-1793","authenticated-orcid":false,"given":"Robson","family":"Capasso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rachna","family":"Saralkar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7802-7172","authenticated-orcid":false,"given":"Chia-Chun","family":"Chiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3959-5748","authenticated-orcid":false,"given":"Jenelle","family":"Jindal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tho","family":"Pham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1487-3237","authenticated-orcid":false,"given":"Faraz","family":"Ghoddusi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8436-0008","authenticated-orcid":false,"given":"Albert S.","family":"Chiou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyo Jung","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohana","family":"Roy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4897-3843","authenticated-orcid":false,"given":"Michael F.","family":"Gensheimer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7779-2910","authenticated-orcid":false,"given":"Hinesh","family":"Patel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8926-5085","authenticated-orcid":false,"given":"Kevin","family":"Schulman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dev","family":"Dash","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danton","family":"Char","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lance","family":"Downing","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francois","family":"Grolleau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3696-4050","authenticated-orcid":false,"given":"Kameron","family":"Black","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bethel","family":"Mieso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aydin","family":"Zahedivash","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wen-wai","family":"Yim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4683-2606","authenticated-orcid":false,"given":"Harshita","family":"Sharma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tony","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hannah","family":"Kirsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9878-1824","authenticated-orcid":false,"given":"Nerissa","family":"Ambers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carlene","family":"Lugtu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5351-9271","authenticated-orcid":false,"given":"Aditya","family":"Sharma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bilal","family":"Mawji","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Alekseyev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vicky","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikas","family":"Kakkar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jarrod","family":"Helzer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anurang","family":"Revri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5134-3300","authenticated-orcid":false,"given":"Yair","family":"Bannett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7988-9356","authenticated-orcid":false,"given":"Roxana","family":"Daneshjou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4387-8740","authenticated-orcid":false,"given":"Jonathan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily","family":"Alsentzer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keith","family":"Morse","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5146-2569","authenticated-orcid":false,"given":"Nirmal","family":"Ravi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6117-8764","authenticated-orcid":false,"given":"Nima","family":"Aghaeepour","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vanessa","family":"Kennedy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3667-6796","authenticated-orcid":false,"given":"Akshay","family":"Chaudhari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1686-4178","authenticated-orcid":false,"given":"Thomas","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4023-419X","authenticated-orcid":false,"given":"Sanmi","family":"Koyejo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew P.","family":"Lungren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8823-0614","authenticated-orcid":false,"given":"Eric","family":"Horvitz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Percy","family":"Liang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8958-0843","authenticated-orcid":false,"given":"Michael A.","family":"Pfeffer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9385-7158","authenticated-orcid":false,"given":"Nigam H.","family":"Shah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,1,20]]},"reference":[{"key":"4151_CR1","unstructured":"Papers with Code. Question answering on MedQA (USMLE). https:\/\/paperswithcode.com\/sota\/question-answering-on-medqa-usmle (2024)."},{"key":"4151_CR2","first-page":"233339282412348","volume":"11","author":"M Khosravi","year":"2024","unstructured":"Khosravi, M., Zare, Z., Mojtabaeian, S. M. & Izadi, R. Artificial intelligence and decision-making in healthcare: a thematic analysis of a systematic review of reviews. Health Serv. Res. Manag. Epidemiol. 11, 23333928241234863 (2024).","journal-title":"Health Serv. Res. Manag. Epidemiol."},{"key":"4151_CR3","unstructured":"Nath, D. Artificial intelligence (AI) will transform the clinical workflow with the next-generation technology. HealthTech Magazines https:\/\/www.healthtechmagazines.com\/artificial-intelligence-ai-will-transform-the-clinical-workflow-with-the-next-generation-technology\/ (2024)."},{"key":"4151_CR4","doi-asserted-by":"publisher","first-page":"1010","DOI":"10.1111\/bju.16676","volume":"135","author":"N Carl","year":"2025","unstructured":"Carl, N. et al. Evaluating interactions of patients with large language models for medical information. BJU Int. 135, 1010\u20131017 (2025).","journal-title":"BJU Int."},{"key":"4151_CR5","unstructured":"Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at https:\/\/arxiv.org\/abs\/2303.13375 (2023)."},{"key":"4151_CR6","doi-asserted-by":"publisher","first-page":"AIe2401235","DOI":"10.1056\/AIe2401235","volume":"2","author":"ID Raji","year":"2025","unstructured":"Raji, I. D., Daneshjou, R. & Alsentzer, E. It\u2019s time to bench the medical exam benchmark. NEJM AI 2, AIe2401235 (2025).","journal-title":"NEJM AI"},{"key":"4151_CR7","unstructured":"Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning 174, 248\u2013260 (PMLR, 2022)."},{"key":"4151_CR8","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","volume":"333","author":"S Bedi","year":"2025","unstructured":"Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333, 319\u2013328 (2025).","journal-title":"JAMA"},{"key":"4151_CR9","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","volume":"30","author":"P Hager","year":"2024","unstructured":"Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613\u20132622 (2024).","journal-title":"Nat. Med."},{"key":"4151_CR10","unstructured":"Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. https:\/\/cdn.openai.com\/pdf\/bd7a39d5-9e9f-47b3-903c-8b847ca650c7\/healthbench_paper.pdf (2025)."},{"key":"4151_CR11","unstructured":"Liang, P. et al. Holistic evaluation of language models. In Transactions on Machine Learning https:\/\/openreview.net\/pdf?id=iO4LZibEqW (2023)."},{"key":"4151_CR12","unstructured":"Leaderboard overview. LM Arena https:\/\/lmarena.ai\/leaderboard (2025)."},{"key":"4151_CR13","unstructured":"Wu, J. et al. BRIDGE: benchmarking large language models for understanding real-world clinical practice text. Preprint at https:\/\/arxiv.org\/abs\/2504.19467 (2025)."},{"key":"4151_CR14","unstructured":"Fries, J. A. et al. BigBio: a framework for data-centric biomedical natural language processing. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=8lQDn9zTQlW (2022)."},{"key":"4151_CR15","doi-asserted-by":"publisher","unstructured":"Croxford, E. et al. Automating evaluation of AI text generation in healthcare with a large language model (LLM)-as-a-judge. Preprint at medRxiv https:\/\/doi.org\/10.1101\/2025.04.22.25326219 (2025).","DOI":"10.1101\/2025.04.22.25326219"},{"key":"4151_CR16","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","volume":"11","author":"D Jin","year":"2021","unstructured":"Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021).","journal-title":"Appl. Sci."},{"key":"4151_CR17","unstructured":"Wornow, M. et al. Context clues: evaluating long context models for clinical prediction tasks on EHRs. In Proc. 13th International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=zg3ec1TdAP (ICLR, 2025)."},{"key":"4151_CR18","unstructured":"Liu, F. et al. Large language models in the clinic: a comprehensive benchmark. Preprint at https:\/\/arxiv.org\/abs\/2405.00716 (2024)."},{"key":"4151_CR19","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1038\/s41746-024-01390-4","volume":"8","author":"C Wu","year":"2025","unstructured":"Wu, C. et al. Towards evaluating and building versatile large language models for medicine. NPJ Digit. Med. 8, 58 (2025).","journal-title":"NPJ Digit. Med."},{"key":"4151_CR20","doi-asserted-by":"crossref","unstructured":"Ouyang, Z. et al. CliMedBench: a large-scale Chinese benchmark for evaluating medical large language models in clinical scenarios. In Proc. 2024 Conference on Empircal Methods in Natural Language Processing 8428\u20138438 (EMNLP, 2024).","DOI":"10.18653\/v1\/2024.emnlp-main.480"},{"key":"4151_CR21","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"4151_CR22","doi-asserted-by":"publisher","first-page":"2546","DOI":"10.1038\/s41591-025-03727-2","volume":"31","author":"S Sandmann","year":"2025","unstructured":"Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31, 2546\u20132549 (2025).","journal-title":"Nat. Med."},{"key":"4151_CR23","doi-asserted-by":"crossref","unstructured":"Cai, Y. et al. MedBench: a large-scale Chinese benchmark for evaluating medical large language models. In Proc. 38th AAAI Conference on Artificial Intelligence 38, 17709\u201317717 (AAAI, 2024).","DOI":"10.1609\/aaai.v38i16.29723"},{"key":"4151_CR24","doi-asserted-by":"crossref","unstructured":"Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. Conference on Computational Natural Language Learning (CoNLL) 314\u2013334 (CoNLL, 2023).","DOI":"10.18653\/v1\/2023.conll-1.21"},{"key":"4151_CR25","unstructured":"Han, T., Kumar, A., Agarwal, C. & Lakkaraju, H. MedSafetyBench: evaluating and improving the medical safety of large language models. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=cFyagd2Yh4 (2024)."},{"key":"4151_CR26","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1038\/s44222-025-00279-5","volume":"3","author":"F Liu","year":"2025","unstructured":"Liu, F. et al. Application of large language models in medicine. Nat. Rev. Bioeng. 3, 85\u2013104 (2025).","journal-title":"Nat. Rev. Bioeng."},{"key":"4151_CR27","doi-asserted-by":"crossref","unstructured":"Magar, I. & Schwartz, R. Data contamination: from memorization to exploitation. In Proc. 60th Annual Meeting of the Association of Computatonal Linguistics (Vol. 2: Short Papers) 157\u2013165 (ACL, 2022).","DOI":"10.18653\/v1\/2022.acl-short.18"},{"key":"4151_CR28","doi-asserted-by":"crossref","unstructured":"Deng, J. et al. ImageNet: a large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 248\u2013255 (IEEE, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"4151_CR29","unstructured":"Gu, J. et al. A survey on LLM-as-a-judge. Preprint at https:\/\/arxiv.org\/abs\/2411.15594 (2025)."},{"key":"4151_CR30","unstructured":"Madaan, L. et al. Quantifying variance in evaluation benchmarks. Preprint at https:\/\/arxiv.org\/abs\/2406.10229 (2024)."},{"key":"4151_CR31","doi-asserted-by":"crossref","unstructured":"Manakul, P., Liusie, A. & Gales, M. J. F. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proc. 2023 Conference on Empirical Methods Natural Language Processing 9004\u20139017 (EMNLP, 2023).","DOI":"10.18653\/v1\/2023.emnlp-main.557"},{"key":"4151_CR32","unstructured":"Guha, B. Secret ballots and costly information gathering: the jury size problem revisited. MPRA Paper no. 73048 (2016)."},{"key":"4151_CR33","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","volume":"30","author":"D Van Veen","year":"2024","unstructured":"Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134\u20131142 (2024).","journal-title":"Nat. Med."},{"key":"4151_CR34","unstructured":"Khandekar, N. et al. MedCalc-Bench: evaluating large language models for medical calculations. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=VXohja0vrQ (2024)."},{"key":"4151_CR35","unstructured":"MT Samples: collection of transcribed medical transcription sample reports and examples. MT Samples https:\/\/www.mtsamples.com\/ (2023)."},{"key":"4151_CR36","doi-asserted-by":"crossref","unstructured":"Ben Abacha, A. et al. MEDEC: a benchmark for medical error detection and correction in clinical notes. in Findings of the Association for Computational Linguistics 22539\u201322550 (ACL, 2025).","DOI":"10.18653\/v1\/2025.findings-acl.1159"},{"key":"4151_CR37","doi-asserted-by":"crossref","unstructured":"Vilares, D. & G\u00f3mez-Rodr\u00edguez, C. HEAD-QA: a healthcare dataset for complex reasoning. In Proc. 57th Annual Meeting of the Association of Computational Linguistics 960\u2013966 (ACL, 2019).","DOI":"10.18653\/v1\/P19-1092"},{"key":"4151_CR38","doi-asserted-by":"crossref","unstructured":"Chen, H., Fang, Z., Singla, Y. & Dredze, M. Benchmarking large language models on answering and explaining challenging medical questions. In Proc. 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) 3563\u20133599 (NAACL, 2025).","DOI":"10.18653\/v1\/2025.naacl-long.182"},{"key":"4151_CR39","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-023-02487-3","volume":"10","author":"W-W Yim","year":"2023","unstructured":"Yim, W. -W. et al. ACI-BENCH: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Sci. Data 10, 586 (2023).","journal-title":"Sci. Data"},{"key":"4151_CR40","unstructured":"Ben Abacha, A. et al. Bridging the gap between consumers\u2019 medication questions and trusted answers. In MEDINFO 2019: Health and Wellbeing e-Networks for All 25\u201329 (IOS Press, 2019)."},{"key":"4151_CR41","doi-asserted-by":"publisher","unstructured":"Zeng, G. et al. MedDialog: large-scale medical dialogue datasets. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (eds Webber, B. et al.) https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.743 (Association for Computational Linguistics, 2020).","DOI":"10.18653\/v1\/2020.emnlp-main.743"},{"key":"4151_CR42","unstructured":"Abacha, A. B., Shivade, C. & Demner-Fushman, D. Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering. In Proc. 18th BioNLP Workshop Shared Task 16\u201325 (ACL, 2019)."},{"key":"4151_CR43","doi-asserted-by":"crossref","unstructured":"Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2567\u20132577 (EMNLP, 2019).","DOI":"10.18653\/v1\/D19-1259"},{"key":"4151_CR44","unstructured":"Lee, G. et al. EHRSQL: a practical text-to-SQL benchmark for electronic health records. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=B2W8Vy0rarw (NeurIPS, 2022)."},{"key":"4151_CR45","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/s41746-023-00939-z","volume":"6","author":"JA Omiye","year":"2023","unstructured":"Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023).","journal-title":"NPJ Digit. Med."},{"key":"4151_CR46","doi-asserted-by":"crossref","unstructured":"Pandit, S. et al. MedHallu: a comprehensive benchmark for detecting medical hallucinations in large language models. Preprint at https:\/\/arxiv.org\/abs\/2502.14302 (2025).","DOI":"10.18653\/v1\/2025.emnlp-main.143"},{"key":"4151_CR47","unstructured":"Wornow, M., Thapa, R., Steinberg, E., Fries, J. A. & Shah, N. H. EHRSHOT: an EHR benchmark for few-shot evaluation of foundation models. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks https:\/\/openreview.net\/pdf?id=CsXC6IcdwI (2023)."},{"key":"4151_CR48","doi-asserted-by":"crossref","unstructured":"Fleming, S. L. et al. MedAlign: a clinician-generated dataset for instruction following with electronic medical records. In Proc. Thirty-Eighth AAAI Conf. Artif. Intell. 38, 21545\u201321555 (AAAI, 2024).","DOI":"10.1609\/aaai.v38i20.30205"},{"key":"4151_CR49","doi-asserted-by":"publisher","unstructured":"Xu, J. Discharge me: BioNLP ACL\u201924 shared task on streamlining discharge documentation (version 1.3). PhysioNet https:\/\/doi.org\/10.13026\/0zf5-fx50 (2024).","DOI":"10.13026\/0zf5-fx50"},{"key":"4151_CR50","doi-asserted-by":"crossref","unstructured":"Chen, Z., Varma, M., Wan, X., Langlotz, C. & Delbrouck, J.-B. Toward expanding the scope of radiology report summarization to multiple anatomies and modalities. In Proc. 61st Annual Meeting of the Association of Computational Linguistics (Vol. 2: Short Papers) 469\u2013484 (ACL, 2023).","DOI":"10.18653\/v1\/2023.acl-short.41"},{"key":"4151_CR51","doi-asserted-by":"publisher","first-page":"470","DOI":"10.1093\/jamia\/ocae312","volume":"32","author":"A Aali","year":"2025","unstructured":"Aali, A. et al. A dataset and benchmark for hospital course summarization with adapted large language models. J. Am. Med. Inform. Assoc. 32, 470\u2013479 (2025).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"4151_CR52","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/jamia\/ocz166","volume":"27","author":"S Henry","year":"2020","unstructured":"Henry, S., Buchan, K., Filannino, M., Stubbs, A. & Uzuner, O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27, 3\u201312 (2020).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"4151_CR53","doi-asserted-by":"crossref","unstructured":"Edin, J. et al. Automated medical coding on MIMIC-III and MIMIC-IV: a critical review and replicability study. In Proc. 46th Int. ACM SIGIR Conf. Research and Development in Information Retrieval 2572\u20132582 (SIGIR, 2023).","DOI":"10.1145\/3539618.3591918"},{"key":"4151_CR54","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1038\/s41746-024-01377-1","volume":"8","author":"I Lopez","year":"2025","unstructured":"Lopez, I. et al. Clinical entity augmented retrieval for clinical information extraction. NPJ Digit. Med. 8, 45 (2025).","journal-title":"NPJ Digit. Med."},{"key":"4151_CR55","doi-asserted-by":"publisher","first-page":"949","DOI":"10.1093\/jamia\/ocae001","volume":"31","author":"M Pillai","year":"2024","unstructured":"Pillai, M., Posada, J., Gardner, R. M., Hernandez-Boussard, T. & Bannett, Y. Measuring quality-of-care in treatment of young children with attention deficit\/hyperactivity disorder using pre-trained language models. J. Am. Med. Inform. Assoc. 31, 949\u2013957 (2024).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"4151_CR56","doi-asserted-by":"publisher","first-page":"e2024067223","DOI":"10.1542\/peds.2024-067223","volume":"155","author":"Y Bannett","year":"2025","unstructured":"Bannett, Y. et al. Applying large language models to assess quality of care: monitoring ADHD medication side effects. Pediatrics 155, e2024067223 (2025).","journal-title":"Pediatrics"},{"key":"4151_CR57","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1001\/jamapediatrics.2023.6032","volume":"178","author":"N Rabbani","year":"2024","unstructured":"Rabbani, N. et al. Evaluation of a large language model to identify confidential content in adolescent encounter notes. JAMA Pediatr. 178, 308\u2013310 (2024).","journal-title":"JAMA Pediatr."},{"key":"4151_CR58","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1001\/jamapediatrics.2024.4438","volume":"179","author":"G Tse","year":"2025","unstructured":"Tse, G. et al. Large language model responses to adolescent patient and proxy messages. JAMA Pediatr. 179, 93\u201394 (2025).","journal-title":"JAMA Pediatr."}],"container-title":["Nature Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41591-025-04151-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41591-025-04151-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41591-025-04151-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T15:03:03Z","timestamp":1774018983000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41591-025-04151-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,20]]},"references-count":58,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3]]}},"alternative-id":["4151"],"URL":"https:\/\/doi.org\/10.1038\/s41591-025-04151-2","relation":{},"ISSN":["1078-8956","1546-170X"],"issn-type":[{"value":"1078-8956","type":"print"},{"value":"1546-170X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,20]]},"assertion":[{"value":"3 June 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 November 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 January 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.O., H.Q., S.J. and L.S. are employees of Microsoft. J.A.F. reports healthcare AI consulting for Snorkel AI and holds stock options in Snorkel AI. This entity had no role in study design, data collection or analysis, decision to publish, or manuscript preparation. A.S. reports employment and owning stock options in Benchmark Health and Daybreak Health; work is unrelated to the research topics explored in MedHELM. P.C. is a Healthcare AI Data Specialist consultant working with OpenAI; this work is unrelated to the research topics explored in MedHELM. A.N. reports cofounding and owning stock in UpDoc. This entity was not involved in any way with the work presented. S.V. reports previously working as an independent contractor for OpenAI\u2019s health and safety initiatives; this work was unrelated to the research topics explored in MedHELM. J.J. is the founder of Jindal Neurology and a per diem physician with Kaiser Permanente. D.Y. reports previously working as an independent contractor for OpenAI\u2019s health and safety initiatives; this work was unrelated to the research topics explored in MedHELM. R.C. reports research funding from Samsung Electronics and serving as advisor to XP Health. None of these entities were involved in any way with the work presented. F.G. is Executive Director of Healthy Oregon and Chief Medical Information Officer at Empathia.ai; none of these entities were involved in any way with the work presented. S.L. serves as an advisor to Google, Codex Health, Gaia Health and Added Health. None of these entities were involved in any way with the work presented. M.F.G. reports institutional grant funding from Xrad Therapeutics. K.S. is a shareholder and advisor to UpDoc, Salud Now, Medeloop and Alethia. None of these entities were involved in any way with the work presented. J.H.C. has received research funding support from: NIH\/National Institute of Allergy and Infectious Diseases (1R01AI17812101); NIH-NCATS Clinical and Translational Science Award (UM1TR004921); Stanford Bio-X Interdisciplinary Initiatives Seed Grants Program (IIP); NIH\/Center for Undiagnosed Diseases at Stanford (U01 NS134358); Stanford RAISE Health Seed Grant 2024; and Josiah Macy Jr. Foundation (AI in Medical Education). J.H.C. is also cofounder of Reaction Explorer, which develops and licenses organic chemistry education software, and has received paid medical expert witness fees (Sutton Pierce, Younker Hyde MacFarlane, Sykes McAllister, Elite Experts), consulting fees from ISHI Health, and one-time honoraria or travel expenses for invited presentations from insitro, General Reinsurance Corporation, AASCIF, Cozeva and other industry conferences, academic institutions and health systems. E.A. reports consulting fees and stock from Fourier Health. N.A. reports serving on Scientific Advisory Boards for January AI, Parallel Bio and Medeloop, and cofounding Takeoff41. None of these entities were involved in any way with the work presented. A.C. reports cofounding and owning stock in Cognita Imaging; owning stock in Subtle Medical, BrainKey and LVIS Corp; and providing consulting services to Elucid Bioimaging and Patient Square Capital. None of these entities were involved in any way with the work presented. S.K. reports cofounding and owning stock options in Virtue AI; this work is unrelated to the research topics explored in MedHELM. E.H. reports serving in a senior role at Microsoft and owning stock in Microsoft, which has efforts in the healthcare domain. M.A.P. is an advisor to Akasa and Surgical Safety Technologies. None of these entities were involved in any way with the work presented. N.H.S. reports being a cofounder of Prealize Health and Atropos Health, serving on the Board of the Coalition for Healthcare AI, and serving as an advisor to Opala, Curai Health, JnJ Innovative Medicines and AbbVie pharmaceuticals. None of these entities were involved in any way with the work presented. The other authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}