{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T10:45:22Z","timestamp":1794307522962,"version":"4.9.0"},"reference-count":52,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T00:00:00Z","timestamp":1777248000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T00:00:00Z","timestamp":1782950400000},"content-version":"vor","delay-in-days":66,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"National Science Foundation,United States","award":["DGE 2139757"],"award-info":[{"award-number":["DGE 2139757"]}],"id":[{"id":"https:\/\/ror.org\/021nxhr62","id-type":"ROR","asserted-by":"crossref"}]},{"name":"National Science Foundation,United States","award":["DGE 2139757"],"award-info":[{"award-number":["DGE 2139757"]}],"id":[{"id":"https:\/\/ror.org\/021nxhr62","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs\u2019 ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore patient-centric metrics that this interactive environment enables.<\/jats:p>","DOI":"10.1038\/s41746-026-02674-7","type":"journal-article","created":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T09:46:44Z","timestamp":1777283204000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["AgentClinic: a multimodal benchmark for tool-using clinical AI agents"],"prefix":"10.1038","volume":"9","author":[{"given":"Samuel","family":"Schmidgall","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rojin","family":"Ziaei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carl","family":"Harris","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ji Woong","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eduardo Pontes","family":"Reis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey","family":"Jopling","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Moor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,4,27]]},"reference":[{"key":"2674_CR1","doi-asserted-by":"crossref","unstructured":"Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930\u20131940 (2023).","DOI":"10.1038\/s41591-023-02448-8"},{"key":"2674_CR2","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. (HEALTH) 3, 1\u201323 (2021).","journal-title":"ACM Trans. Comput. Healthc. (HEALTH)"},{"key":"2674_CR3","doi-asserted-by":"publisher","unstructured":"Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2311.16452 (2023).","DOI":"10.48550\/arXiv.2311.16452"},{"key":"2674_CR4","doi-asserted-by":"crossref","unstructured":"Li\u00e9vin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions? Patterns 5, (2023).","DOI":"10.1016\/j.patter.2024.100943"},{"key":"2674_CR5","unstructured":"Organization, W. H. et al. Health workforce requirements for universal health coverage and the sustainable development goals. World Health Organization (Human resources for health observer, 17) (2016)."},{"key":"2674_CR6","doi-asserted-by":"publisher","first-page":"004695802091030","DOI":"10.1177\/0046958020910305","volume":"57","author":"D McIntyre","year":"2020","unstructured":"McIntyre, D. & Chow, C. K. Waiting time as an indicator for health services under strain: a narrative review. INQUIRY: J. Health Care Organ. Provis. Financing 57, 0046958020910305 (2020).","journal-title":"INQUIRY: J. Health Care Organ. Provis. Financing"},{"key":"2674_CR7","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"2674_CR8","doi-asserted-by":"publisher","first-page":"e855","DOI":"10.1016\/S2589-7500(23)00202-9","volume":"5","author":"A Vaid","year":"2023","unstructured":"Vaid, A., Landi, I., Nadkarni, G. & Nabeel, I. Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders. Lancet Digital Health 5, e855\u2013e858 (2023).","journal-title":"Lancet Digital Health"},{"key":"2674_CR9","doi-asserted-by":"crossref","unstructured":"Xiong, G., Jin, Q., Lu, Z., & Zhang, A. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 6233\u20136251) (2024).","DOI":"10.18653\/v1\/2024.findings-acl.372"},{"key":"2674_CR10","doi-asserted-by":"publisher","unstructured":"Li\u00e9vin, V., Hother, C. E. & Winther, O. Can large language models reason about medical questions? Preprint at https:\/\/doi.org\/10.48550\/arXiv.2207.08143 (2022).","DOI":"10.48550\/arXiv.2207.08143"},{"key":"2674_CR11","doi-asserted-by":"crossref","unstructured":"Wu, C. et al. Pmc-llama: towards building open-source language models for medicine. J. Am. Med. Inform. Assoc. 31, 1833\u20131843 (2023).","DOI":"10.1093\/jamia\/ocae045"},{"key":"2674_CR12","doi-asserted-by":"publisher","unstructured":"Chen, Z. et al. Meditron-70b: scaling medical pretraining for large language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2311.16079 (2023).","DOI":"10.48550\/arXiv.2311.16079"},{"key":"2674_CR13","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00939-z","volume":"6","author":"JA Omiye","year":"2023","unstructured":"Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023).","journal-title":"NPJ Digit. Med."},{"key":"2674_CR14","unstructured":"Ziaei, R. & Schmidgall, S. Language models are susceptible to incorrect patient self-diagnosis in medical applications. In Deep Generative Models for Health Workshop NeurIPS 2023 (2023)."},{"key":"2674_CR15","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1038\/s41746-024-01283-6","volume":"7.1","author":"S Schmidgall","year":"2024","unstructured":"Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digit. Med. 7.1, 295 (2024).","journal-title":"npj Digit. Med."},{"key":"2674_CR16","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-022-01899-x","volume":"10","author":"AE Johnson","year":"2023","unstructured":"Johnson, A. E. et al. Mimic-iv, a freely accessible electronic health record dataset. Sci. Data 10, 1 (2023).","journal-title":"Sci. Data"},{"key":"2674_CR17","unstructured":"Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, 248\u2013260 (PMLR, 2022)."},{"key":"2674_CR18","doi-asserted-by":"publisher","first-page":"68772","DOI":"10.52202\/079017-2197","volume":"37","author":"A Panickssery","year":"2024","unstructured":"Panickssery, A., Bowman, S. R. & Feng, S. Llm evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772\u201368802 (2024).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"2674_CR19","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-023-04530-8","volume":"23","author":"CV Lombardi","year":"2023","unstructured":"Lombardi, C. V., Chidiac, N. T., Record, B. C. & Laukka, J. J. Usmle step 1 and step 2 CK as indicators of resident performance. BMC Med. Educ. 23, 543 (2023).","journal-title":"BMC Med. Educ."},{"key":"2674_CR20","doi-asserted-by":"crossref","unstructured":"Ma, Z., Huang, W., Zhang, J., Gupta, T. & Krishna, R. m&m\u2019s: a benchmark to evaluate tool-use for multi-step multi-modal tasks. In Synthetic Data for Computer Vision Workshop@ CVPR 2024 (2024).","DOI":"10.1007\/978-3-031-72684-2_2"},{"key":"2674_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3704435","volume":"57","author":"Y Qin","year":"2024","unstructured":"Qin, Y. et al. Tool learning with foundation models. ACM Comput. Surv. 57, 1\u201340 (2024).","journal-title":"ACM Comput. Surv."},{"key":"2674_CR22","doi-asserted-by":"publisher","first-page":"2400","DOI":"10.1001\/jama.2012.5960","volume":"307","author":"R Smith-Bindman","year":"2012","unstructured":"Smith-Bindman, R. et al. Use of diagnostic imaging studies and associated radiation exposure for patients enrolled in large integrated health care systems, 1996-2010. Jama 307, 2400\u20132409 (2012).","journal-title":"Jama"},{"key":"2674_CR23","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1038\/s41586-025-08866-7","volume":"642.8067","author":"T Tu","year":"2025","unstructured":"Tu, T. et al. Towards conversational diagnostic artificial intelligence. Nature 642.8067, 442\u2013450 (2025).","journal-title":"Nature"},{"key":"2674_CR24","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1136\/bmj.319.7206.358","volume":"319","author":"JW Ely","year":"1999","unstructured":"Ely, J. W. et al. Analysis of questions asked by family doctors regarding patient care. Bmj 319, 358\u2013361 (1999).","journal-title":"Bmj"},{"key":"2674_CR25","doi-asserted-by":"publisher","first-page":"46595","DOI":"10.52202\/075280-2020","volume":"36","author":"L Zheng","year":"2023","unstructured":"Zheng, L. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Adv. Neural Inf. Process. Syst. 36, 46595\u201346623 (2023).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"2674_CR26","doi-asserted-by":"publisher","unstructured":"Park, J. S. et al. Generative agents: Interactive simulacra of human behavior. In Proc. 36th ACM Symposium on User Interface Software and Technology. 2023. Association for Computing Machinery, New York, NY, USA, Article 2, 1\u201322. https:\/\/doi.org\/10.1145\/3586183.3606763 (2023).","DOI":"10.1145\/3586183.3606763"},{"key":"2674_CR27","doi-asserted-by":"publisher","unstructured":"Li, J. et al. Agent hospital: a simulacrum of hospital with evolvable medical agents. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2405.02957 (2024).","DOI":"10.48550\/arXiv.2405.02957"},{"key":"2674_CR28","doi-asserted-by":"publisher","unstructured":"Jiang, A. Q. et al. Mixtral of experts. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2401.04088 (2024).","DOI":"10.48550\/arXiv.2401.04088"},{"key":"2674_CR29","doi-asserted-by":"publisher","unstructured":"Touvron, H. et al. Llama: open and efficient foundation language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2302.13971 (2023).","DOI":"10.48550\/arXiv.2302.13971"},{"key":"2674_CR30","doi-asserted-by":"publisher","unstructured":"Liao, Y. et al. Automatic interactive evaluation for large language models with state aware patient simulator. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2403.08495 (2024).","DOI":"10.48550\/arXiv.2403.08495"},{"key":"2674_CR31","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","volume":"31.1","author":"S Johri","year":"2025","unstructured":"Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 31.1, 77\u201386 (2025).","journal-title":"Nat. Med."},{"key":"2674_CR32","doi-asserted-by":"crossref","unstructured":"Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Findings of the Association for Computational Linguistics: ACL 2024. 2024.","DOI":"10.18653\/v1\/2024.findings-acl.33"},{"key":"2674_CR33","doi-asserted-by":"publisher","unstructured":"Gu, J. et al. A survey on llm-as-a-judge. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2411.15594 (2024).","DOI":"10.48550\/arXiv.2411.15594"},{"key":"2674_CR34","doi-asserted-by":"crossref","unstructured":"Chen, J. & Mueller, J. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). (2024).","DOI":"10.18653\/v1\/2024.acl-long.283"},{"key":"2674_CR35","doi-asserted-by":"crossref","unstructured":"Chen, G. H., Chen, S., Liu, Z., Jiang, F. & Wang, B. Humans or LLMs as the judge? a study on judgement bias. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing, 8301\u20138327 https:\/\/aclanthology.org\/2024.emnlp-main.474 (Association for Computational Linguistics, 2024).","DOI":"10.18653\/v1\/2024.emnlp-main.474"},{"key":"2674_CR36","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1177\/0272989X14547740","volume":"35","author":"JS Blumenthal-Barby","year":"2015","unstructured":"Blumenthal-Barby, J. S. & Krieger, H. Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy. Med. Decis. Mak. 35, 539\u2013557 (2015).","journal-title":"Med. Decis. Mak."},{"key":"2674_CR37","first-page":"78","volume":"6","author":"MEH Hammond","year":"2021","unstructured":"Hammond, M. E. H., Stehlik, J., Drakos, S. G. & Kfoury, A. G. Bias in medicine: lessons learned and mitigation strategies. Basic Transl. Sci. 6, 78\u201385 (2021).","journal-title":"Basic Transl. Sci."},{"key":"2674_CR38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12910-017-0179-8","volume":"18","author":"C FitzGerald","year":"2017","unstructured":"FitzGerald, C. & Hurst, S. Implicit bias in healthcare professionals: a systematic review. BMC Med. Ethics 18, 1\u201318 (2017).","journal-title":"BMC Med. Ethics"},{"key":"2674_CR39","doi-asserted-by":"publisher","first-page":"40","DOI":"10.7861\/fhj.2020-0233","volume":"8","author":"DP Gopal","year":"2021","unstructured":"Gopal, D. P., Chetty, U., O\u2019Donnell, P., Gajria, C. & Blackadder-Weinstein, J. Implicit bias in healthcare: clinical practice, research and decision making. Future Healthc. J. 8, 40 (2021).","journal-title":"Future Healthc. J."},{"key":"2674_CR40","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1056\/NEJMp2201180","volume":"387","author":"JA Sabin","year":"2022","unstructured":"Sabin, J. A. Tackling implicit bias in health care. N. Engl. J. Med. 387, 105\u2013107 (2022).","journal-title":"N. Engl. J. Med."},{"key":"2674_CR41","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T. et al. Language models are few-shot learners. Adv. neural Inf. Process. Syst. 33, 1877\u20131901 (2020).","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"2674_CR42","unstructured":"Christiano, P. F. et al. Deep reinforcement learning from human preferences. Adv. Neural Inform. Process. Syst. 30, (2017)."},{"key":"2674_CR43","unstructured":"OpenAI et al. Gpt-4 technical report 2303.08774 (2023)."},{"key":"2674_CR44","unstructured":"Ankit Pal, M. S. Openbiollms: advancing open-source large language models for healthcare and life sciences. https:\/\/huggingface.co\/aaditya\/OpenBioLLM-Llama3-70B (2024)."},{"key":"2674_CR45","doi-asserted-by":"publisher","unstructured":"Xie, Q. et al. Me llama: foundation large language models for medical applications. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2402.12749 (2024).","DOI":"10.48550\/arXiv.2402.12749"},{"key":"2674_CR46","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.52202\/068431-1800","volume":"35","author":"J Wei","year":"2022","unstructured":"Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 35, 24824\u201324837 (2022).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"2674_CR47","doi-asserted-by":"publisher","first-page":"22199","DOI":"10.52202\/068431-1613","volume":"35","author":"T Kojima","year":"2022","unstructured":"Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. Adv. Neural Inf. Process. Syst. 35, 22199\u201322213 (2022).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"2674_CR48","doi-asserted-by":"publisher","first-page":"186345","DOI":"10.1007\/s11704-024-40231-1","volume":"18","author":"L Wang","year":"2024","unstructured":"Wang, L. et al. A survey on large language model based autonomous agents. Front. Comput. Sci. 18, 186345 (2024).","journal-title":"Front. Comput. Sci."},{"key":"2674_CR49","doi-asserted-by":"publisher","unstructured":"Gao, Y. et al. Retrieval-augmented generation for large language models: a survey. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2312.10997 (2023).","DOI":"10.48550\/arXiv.2312.10997"},{"key":"2674_CR50","unstructured":"Zhengbao, J. et al. Active retrieval augmented generation. Proceedings of the 2023 conference on empirical methods in natural language processing. (2023)."},{"key":"2674_CR51","unstructured":"Asai, A. et al. Self-rag: Learning to retrieve, generate, and critique through self-reflection. The Twelfth International Conference on Learning Representations. (2023)."},{"key":"2674_CR52","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","volume":"11","author":"D Jin","year":"2021","unstructured":"Jin, D. et al. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021).","journal-title":"Appl. Sci."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02674-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02674-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02674-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T23:18:39Z","timestamp":1782947919000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02674-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,27]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2674"],"URL":"https:\/\/doi.org\/10.1038\/s41746-026-02674-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,27]]},"assertion":[{"value":"19 April 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 April 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"499"}}