{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T07:18:09Z","timestamp":1793431089749,"version":"4.5.0"},"posted":{"date-parts":[[2025,8,26]]},"group-title":"Health Informatics","reference-count":82,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T00:00:00Z","timestamp":1756166400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,8,26]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>AI agents built on large language models (LLMs) can plan tasks, use external tools, and coordinate with other agents. Unlike standard LLMs, agents can execute multi-step processes, access real-time clinical information, and integrate multiple data sources. There has been interest in using such agents for clinical and administrative tasks, however, there is limited knowledge on their performance and whether multi-agent systems function better than a single agent for healthcare tasks.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Purpose<\/jats:title>\n                  <jats:p>To evaluate the performance of AI agents in healthcare, compare AI agent systems vs. standard LLMs and catalog the tools used for task completion<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Data Sources<\/jats:title>\n                  <jats:p>PubMed, Web of Science, and Scopus from October 1, 2022, through August 5, 2025.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Study Selection<\/jats:title>\n                  <jats:p>Peer-reviewed studies implementing AI agents for clinical tasks with quantitative performance comparisons.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Data Extraction<\/jats:title>\n                  <jats:p>Two reviewers (A.G., M.O.) independently extracted data on architectures, performance metrics, and clinical applications. Discrepancies were resolved by discussion, with a third reviewer (E.K.) consulted when consensus could not be reached.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Data Synthesis<\/jats:title>\n                  <jats:p>Twenty studies met inclusion criteria. Across studies, all agent systems outperformed their baseline LLMs in accuracy performance. Improvements ranged from small gains to increases of over 60 percentage points, with a median improvement of 53 percentage points in single-agent tool-calling studies. These systems were particularly effective for discrete tasks such as medication dosing and evidence retrieval. Multi-agent systems showed optimal performance with up to 5 agents, and their effectiveness was particularly pronounced when dealing with highly complex tasks. The highest performance boost occurred when the complexity of the AI agent framework aligned with that of the task.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Limitations<\/jats:title>\n                  <jats:p>Heterogeneous outcomes precluded quantitative meta-analysis. Several studies relied on synthetic data, limiting generalizability.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>AI agents consistently improve clinical task performance of Base-LLMs when architecture matches task complexity. Our analysis indicates a step-change over base-LLMs, with AI agents opening previously inaccessible domains. Future efforts should be based on prospective, multi-center trials using real-world data to determine safety, task matched and cost-effectiveness.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Primary Funding Source<\/jats:title>\n                  <jats:p>This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai and supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Registration<\/jats:title>\n                  <jats:p>PROSPERO CRD420251120318<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2025.08.22.25334232","type":"posted-content","created":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T22:50:13Z","timestamp":1756248613000},"source":"Crossref","is-referenced-by-count":51,"title":["AI Agents in Clinical Medicine: A Systematic Review"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7542-8608","authenticated-orcid":false,"given":"Alon","family":"Gorenshtein","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0438-0827","authenticated-orcid":false,"given":"Mahmud","family":"Omar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4515-8090","authenticated-orcid":false,"given":"Benjamin S","family":"Glicksberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6319-4314","authenticated-orcid":false,"given":"Girish N","family":"Nadkarni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4567-3108","authenticated-orcid":false,"given":"Eyal","family":"Klang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025083000300700000_2025.08.22.25334232v1.1","unstructured":"Health C for D and R. Artificial Intelligence-Enabled Medical Devices. FDA. Published online July 10, 2025. Accessed August 8, 2025. https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-enabled-medical-devices"},{"key":"2025083000300700000_2025.08.22.25334232v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03423-7"},{"key":"2025083000300700000_2025.08.22.25334232v1.3","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2300192"},{"key":"2025083000300700000_2025.08.22.25334232v1.4","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03456-y"},{"key":"2025083000300700000_2025.08.22.25334232v1.5","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.40969"},{"key":"2025083000300700000_2025.08.22.25334232v1.6","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-03626-6"},{"key":"2025083000300700000_2025.08.22.25334232v1.7","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03097-1"},{"key":"2025083000300700000_2025.08.22.25334232v1.8","unstructured":"Zhao YF , Bove A , Thompson D , et al. Generative AI Is Not Ready for Clinical Use in Patient Education for Lower Back Pain Patients, Even With Retrieval-Augmented Generation. AMIA Summits Transl Sci Proc. 2025;2025:644\u2013653."},{"key":"2025083000300700000_2025.08.22.25334232v1.9","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2023.1161098"},{"key":"2025083000300700000_2025.08.22.25334232v1.10","doi-asserted-by":"publisher","DOI":"10.2196\/59823"},{"key":"2025083000300700000_2025.08.22.25334232v1.11","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3"},{"key":"2025083000300700000_2025.08.22.25334232v1.12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7"},{"key":"2025083000300700000_2025.08.22.25334232v1.13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.05777"},{"key":"2025083000300700000_2025.08.22.25334232v1.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03445-1"},{"key":"2025083000300700000_2025.08.22.25334232v1.15","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-023-00370-1"},{"key":"2025083000300700000_2025.08.22.25334232v1.16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.12687"},{"key":"2025083000300700000_2025.08.22.25334232v1.17","unstructured":"AI agents \u2014 what they are, and how they\u2019ll change the way we work. Source. Accessed August 10, 2025. https:\/\/news.microsoft.com\/source\/features\/ai\/ai-agents-what-they-are-and-how-theyll-change-the-way-we-work\/"},{"key":"2025083000300700000_2025.08.22.25334232v1.18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.16736"},{"key":"2025083000300700000_2025.08.22.25334232v1.19","unstructured":"What Are AI Agents? | IBM. July 3, 2024. Accessed August 10, 2025. https:\/\/www.ibm.com\/think\/topics\/ai-agents"},{"key":"2025083000300700000_2025.08.22.25334232v1.20","unstructured":"What are AI Agents?- Agents in Artificial Intelligence Explained - AWS. Amazon Web Services, Inc. Accessed August 10, 2025. https:\/\/aws.amazon.com\/what-is\/ai-agents\/"},{"key":"2025083000300700000_2025.08.22.25334232v1.21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.10997"},{"key":"2025083000300700000_2025.08.22.25334232v1.22","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000877"},{"key":"2025083000300700000_2025.08.22.25334232v1.23","doi-asserted-by":"publisher","DOI":"10.3348\/kjr.2025.0360"},{"key":"2025083000300700000_2025.08.22.25334232v1.24","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.22405"},{"key":"2025083000300700000_2025.08.22.25334232v1.25","doi-asserted-by":"publisher","DOI":"10.1186\/s41077-025-00357-z"},{"key":"2025083000300700000_2025.08.22.25334232v1.26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.01680"},{"key":"2025083000300700000_2025.08.22.25334232v1.27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2505.10468"},{"key":"2025083000300700000_2025.08.22.25334232v1.28","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2024.1422344"},{"key":"2025083000300700000_2025.08.22.25334232v1.29","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(25)00202-8"},{"key":"2025083000300700000_2025.08.22.25334232v1.30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.16833"},{"key":"2025083000300700000_2025.08.22.25334232v1.31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2502.11211"},{"key":"2025083000300700000_2025.08.22.25334232v1.32","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"2025083000300700000_2025.08.22.25334232v1.33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2508.10146"},{"key":"2025083000300700000_2025.08.22.25334232v1.34","doi-asserted-by":"publisher","DOI":"10.2196\/58202"},{"key":"2025083000300700000_2025.08.22.25334232v1.35","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinph.2025.2110792"},{"key":"2025083000300700000_2025.08.22.25334232v1.36","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2025.110363"},{"key":"2025083000300700000_2025.08.22.25334232v1.37","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2025.110731"},{"key":"2025083000300700000_2025.08.22.25334232v1.38","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01550-0"},{"key":"2025083000300700000_2025.08.22.25334232v1.39","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01475-8"},{"key":"2025083000300700000_2025.08.22.25334232v1.40","doi-asserted-by":"publisher","DOI":"10.1038\/s43018-025-00991-6"},{"key":"2025083000300700000_2025.08.22.25334232v1.41","doi-asserted-by":"publisher","DOI":"10.1038\/s41551-025-01463-z"},{"key":"2025083000300700000_2025.08.22.25334232v1.42","doi-asserted-by":"crossref","unstructured":"Automatic biomarker discovery and enrichment with BRAD | Bioinformatics | Oxford Academic. Accessed August 8, 2025. https:\/\/academic.oup.com\/bioinformatics\/article\/41\/5\/btaf159\/8125018","DOI":"10.1093\/bioinformatics\/btaf159"},{"key":"2025083000300700000_2025.08.22.25334232v1.43","doi-asserted-by":"publisher","DOI":"10.1016\/j.arthro.2024.10.042"},{"key":"2025083000300700000_2025.08.22.25334232v1.44","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-025-09442-9"},{"key":"2025083000300700000_2025.08.22.25334232v1.45","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13141649"},{"key":"2025083000300700000_2025.08.22.25334232v1.46","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-025-02748-6"},{"key":"2025083000300700000_2025.08.22.25334232v1.47","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btaf359"},{"key":"2025083000300700000_2025.08.22.25334232v1.48","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07250-3"},{"key":"2025083000300700000_2025.08.22.25334232v1.49","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbaf140"},{"key":"2025083000300700000_2025.08.22.25334232v1.50","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/adbff1"},{"key":"2025083000300700000_2025.08.22.25334232v1.51","doi-asserted-by":"publisher","DOI":"10.2196\/59439"},{"key":"2025083000300700000_2025.08.22.25334232v1.52","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2024.3464555"},{"key":"2025083000300700000_2025.08.22.25334232v1.53","unstructured":"Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems - Yen Sia Low, Michael L Jackson , Rebecca J Hyde , Robert E Brown , Neil M Sanghavi , Julian D Baldwin , C William Pike , Jananee Muralidharan , Gavin Hui , Natasha Alexander , Hadeel Hassan , Rahul V Nene , Morgan Pike , Courtney J Pokrzywa , Shivam Vedak , Adam Paul Yan , Dong-han Yao , Amy R Zipursky , Christina Dinh , Philip Ballentine , Dan C Derieg , Vladimir Polony , Rehan N Chawdry , Jordan Davies , Brigham B Hyde , Nigam H Shah , Saurabh Gombar , 2025. Accessed August 8, 2025. https:\/\/journals.sagepub.com\/doi\/10.1177\/20552076251348850"},{"key":"2025083000300700000_2025.08.22.25334232v1.54","doi-asserted-by":"publisher","DOI":"10.1007\/s00415-025-13261-3"},{"key":"2025083000300700000_2025.08.22.25334232v1.55","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.00344"},{"key":"2025083000300700000_2025.08.22.25334232v1.56","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.07128"},{"key":"2025083000300700000_2025.08.22.25334232v1.57","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2405.02957"},{"key":"2025083000300700000_2025.08.22.25334232v1.58","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.14142"},{"key":"2025083000300700000_2025.08.22.25334232v1.59","doi-asserted-by":"publisher","DOI":"10.1128\/jcm.00689-24"},{"key":"2025083000300700000_2025.08.22.25334232v1.60","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01807-8"},{"key":"2025083000300700000_2025.08.22.25334232v1.61","unstructured":"On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models. Accessed August 9, 2025. https:\/\/arxiv.org\/html\/2405.13966v1"},{"key":"2025083000300700000_2025.08.22.25334232v1.62","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.07324"},{"key":"2025083000300700000_2025.08.22.25334232v1.63","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2505.23450"},{"key":"2025083000300700000_2025.08.22.25334232v1.64","doi-asserted-by":"publisher","DOI":"10.2478\/kbo-2024-0061"},{"key":"2025083000300700000_2025.08.22.25334232v1.65","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.05120"},{"key":"2025083000300700000_2025.08.22.25334232v1.66","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-acl.606"},{"key":"2025083000300700000_2025.08.22.25334232v1.67","unstructured":"CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering. Accessed August 9, 2025. https:\/\/arxiv.org\/html\/2505.16229v1"},{"key":"2025083000300700000_2025.08.22.25334232v1.68","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2300138"},{"key":"2025083000300700000_2025.08.22.25334232v1.69","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.40969"},{"key":"2025083000300700000_2025.08.22.25334232v1.70","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03456-y"},{"key":"2025083000300700000_2025.08.22.25334232v1.71","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.10970"},{"key":"2025083000300700000_2025.08.22.25334232v1.72","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.22358"},{"key":"2025083000300700000_2025.08.22.25334232v1.73","doi-asserted-by":"publisher","DOI":"10.15441\/ceem.25.125"},{"key":"2025083000300700000_2025.08.22.25334232v1.74","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.18374"},{"key":"2025083000300700000_2025.08.22.25334232v1.75","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2411.08881"},{"key":"2025083000300700000_2025.08.22.25334232v1.76","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-02454-5"},{"key":"2025083000300700000_2025.08.22.25334232v1.77","unstructured":"Armitage H. Clinicians can \u2018chat\u2019 with medical records through new AI software, ChatEHR. News Center. June 5, 2025. Accessed August 9, 2025. https:\/\/med.stanford.edu\/news\/all-news\/2025\/06\/chatehr.html"},{"key":"2025083000300700000_2025.08.22.25334232v1.78","unstructured":"Tsinghua University holds Tsinghua AI Agent Hospital Inauguration and 2025 Tsinghua Medicine Townhall Meeting-Tsinghua University. Accessed August 10, 2025. https:\/\/www.tsinghua.edu.cn\/en\/info\/1245\/14224.htm"},{"key":"2025083000300700000_2025.08.22.25334232v1.79","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2022.0372"},{"key":"2025083000300700000_2025.08.22.25334232v1.80","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-024-08956-8"},{"key":"2025083000300700000_2025.08.22.25334232v1.81","doi-asserted-by":"publisher","DOI":"10.1101\/2025.08.13.25333529"},{"key":"2025083000300700000_2025.08.22.25334232v1.82","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2401165"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.08.22.25334232","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:44:05Z","timestamp":1768509845000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2025.08.22.25334232"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,26]]},"references-count":82,"URL":"https:\/\/doi.org\/10.1101\/2025.08.22.25334232","relation":{},"subject":[],"published":{"date-parts":[[2025,8,26]]},"subtype":"preprint"}}