{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T15:04:52Z","timestamp":1790262292322,"version":"4.1.0"},"reference-count":42,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Acute ischemic stroke (AIS) treatment selection requires rapid, guideline-concordant integration of clinical, imaging, and laboratory data, including therapeutic windows, contraindications, stroke severity, and imaging eligibility. This process is complex, expertise-dependent, and vulnerable to safety-critical errors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to develop and validate a structured multiagent large language model (LLM) framework for AIS decision support using real-world cases and to assess its accuracy, safety, auditability, and impact on physician decision-making, particularly among junior physicians and nonspecialists.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We developed a multiagent LLM framework that used structured outputs and guideline-based reasoning to generate treatment recommendations (intravenous thrombolysis, endovascular thrombectomy, standard medical therapy, or non-AIS, or nonstroke) and Trial of ORG 10172 in Acute Stroke Treatment (TOAST) classification. The framework was evaluated using multicenter retrospective real-world cases from 2 hospitals collected between January 2018 and March 2025, prospective cases from February to May 2025, and literature-derived challenging cases from PubMed between January 2024 and January 2025. Performance was assessed against clinical reference standards. Safety was assessed using omission and hallucination event rates, instruction adherence, and 5-point clinical safety ratings. In a prospective physician study, physicians with different seniority and specialty backgrounds made AIS treatment and TOAST classification decisions with and without LLM support. Physician-case decision-level outcomes were analyzed using a binomial generalized linear mixed-effects model accounting for physician and case effects.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      The final analysis included 1055 group A cases, 721 group B cases, 144 literature-derived group C cases, and 161 prospectively collected group D cases. Across representative Baichuan, Qwen, DeepSeek, and GPT models, the multiagent framework consistently improved treatment recommendation accuracy. Model-level accuracy ranges increased from 0.546\u20100.737 to 0.687\u20100.851 in group A, from 0.587\u20100.698 to 0.671\u20100.813 in group B, and from 0.507\u20100.646 to 0.667\u20100.750 in group C. TOAST classification improved overall, with cohort-level variation. Across evaluated models, the multiagent framework increased the mean clinical safety score from 3.70 to 4.01 and reduced mean hallucination and omission rates from 33.6% to 20.6% and from 38.5% to 24.5%, respectively. In the prospective physician study, LLM support increased treatment decision accuracy from 73.1% to 88.6% (odds ratio 2.86, 95% CI, 2.27\u20103.60;\n                      <jats:italic>P<\/jats:italic>\n                      &lt;.001). Accuracy gains were largest among junior and nonspecialist physicians, including junior specialists (0.667 to 0.833), junior nonspecialists (0.600 to 0.846), and senior nonspecialists (0.667 to 0.850). TOAST classification performance also improved (odds ratio 3.63, 95% CI 2.85\u20104.64;\n                      <jats:italic>P<\/jats:italic>\n                      &lt;.001).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>A structured multiagent framework improved LLM performance with average improvements of 18.9% in AIS treatment recommendation and TOAST classification, while producing more structured, auditable outputs with higher safety ratings. It was associated with higher physician decision accuracy, with larger gains among less-experienced physicians, suggesting the potential to narrow expertise-related decision accuracy gaps. Prospective multicenter studies are needed to assess effects on workflow and clinical outcomes.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/96304","type":"journal-article","created":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T17:00:10Z","timestamp":1782234010000},"page":"e96304-e96304","source":"Crossref","is-referenced-by-count":1,"title":["A Multiagent Large Language Model Framework for Emergency Treatment Recommendation in Acute Ischemic Stroke: Development and Validation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2003-3120","authenticated-orcid":false,"given":"Bicong","family":"Yan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4372-4987","authenticated-orcid":false,"given":"Ruipeng","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9971-9653","authenticated-orcid":false,"given":"Li","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3539-2492","authenticated-orcid":false,"given":"Xinyu","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7885-2467","authenticated-orcid":false,"given":"Zhongzheng","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8028-248X","authenticated-orcid":false,"given":"Yuehua","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,30]]},"reference":[{"issue":"10440","key":"R1","doi-asserted-by":"publisher","first-page":"2204","DOI":"10.1016\/S0140-6736(24)00685-8","article-title":"Burden of disease scenarios for 204 countries and territories, 2022-2050: a forecasting analysis for the Global Burden of Disease Study 2021","volume":"403","author":"GBD 2021 Forecasting Collaborators","journal-title":"Lancet"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1177\/17474930241308142","article-title":"World Stroke Organization: Global Stroke Fact Sheet 2025","volume":"20","author":"Feigin","journal-title":"Int J Stroke"},{"issue":"8","key":"R3","doi-asserted-by":"publisher","first-page":"777","DOI":"10.1001\/jamaneurol.2022.1621","article-title":"Structural inequities for historically underserved communities in the adoption of stroke certification in the United States","volume":"79","author":"Shen","journal-title":"JAMA Neurol"},{"issue":"10260","key":"R4","doi-asserted-by":"publisher","first-page":"1443","DOI":"10.1016\/S0140-6736(20)31374-X","article-title":"Stroke systems of care in low-income and middle-income countries: challenges and opportunities","volume":"396","author":"Pandian","journal-title":"Lancet"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1017\/S109285291700013X","article-title":"Optimization of acute stroke care in the emergency department: a call for better utilization of healthcare resources amid growing shortage of neurologists in the United States","volume":"23","author":"Avasarala","journal-title":"CNS Spectr"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.lanwpc.2024.101195","article-title":"Burden of cardiovascular disease among the Western Pacific region and its association with human resources for health, 1990-2021: a systematic analysis of the Global Burden of Disease Study 2021","volume":"51","author":"Wang","journal-title":"Lancet Reg Health West Pac"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.3389\/fneur.2022.1070523","article-title":"Pre- and in-hospital delays in the use of thrombolytic therapy for patients with acute ischemic stroke in rural and urban Egypt","volume":"13","author":"Nasreldein","journal-title":"Front Neurol"},{"issue":"9","key":"R8","doi-asserted-by":"publisher","first-page":"913","DOI":"10.1016\/S1474-4422(16)30073-4","article-title":"Global burden of stroke and risk factors in 188 countries, during 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013","volume":"15","author":"Feigin","journal-title":"Lancet Neurol"},{"key":"R9","unstructured":"Global declaration on stroke. World Stroke Organization. 2023. URL: https:\/\/www.world-stroke.org\/news-and-blog\/news\/global-declaration-on-stroke-commitments-for-facing-stroke [Accessed 18-09-2025]"},{"issue":"3","key":"R10","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1038\/s41591-024-03416-6","article-title":"A generalist medical language model for disease diagnosis assistance","volume":"31","author":"Liu","journal-title":"Nat Med"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a generative artificial intelligence model in a complex diagnostic challenge","volume":"330","author":"Kanjee","journal-title":"JAMA"},{"issue":"5","key":"R12","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1001\/jamainternmed.2024.0295","article-title":"Clinical reasoning of a generative artificial intelligence model compared with physicians","volume":"184","author":"Cabral","journal-title":"JAMA Intern Med"},{"issue":"8067","key":"R13","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1038\/s41586-025-08866-7","article-title":"Towards conversational diagnostic artificial intelligence","volume":"642","author":"Tu","journal-title":"Nature"},{"issue":"8067","key":"R14","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1038\/s41586-025-08869-4","article-title":"Towards accurate differential diagnosis with large language models","volume":"642","author":"McDuff","journal-title":"Nature"},{"issue":"10","key":"R15","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.40969","article-title":"Large language model influence on diagnostic reasoning: a randomized clinical trial","volume":"7","author":"Goh","journal-title":"JAMA Netw Open"},{"issue":"4","key":"R16","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1038\/s41591-024-03456-y","article-title":"GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial","volume":"31","author":"Goh","journal-title":"Nat Med"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/48328","article-title":"Large language models-supported thrombectomy decision-making in acute ischemic stroke based on radiology reports: feasibility qualitative study","volume":"27","author":"Kottlors","journal-title":"J Med Internet Res"},{"key":"R18","unstructured":"Wang B Zhao H Zhou H Song L Xu M Cheng W et al. Baichuan-M1: pushing the medical capability of large language models. arXiv. Preprint posted online on Feb 18, 2025. [doi: 10.48550\/arXiv.2502.12671]"},{"key":"R19","unstructured":"Agarwal S Ahmad L Ai J Altman S Applebaum A Arbus E et al. gpt-oss-120b & gpt-oss-20b model card. arXiv. Preprint posted online on Aug 8, 2025. [doi: 10.48550\/arXiv.2508.10925]"},{"key":"R20","unstructured":"Yang A Yang B Zhang B Hui B Zheng B Yu B et al. Qwen2.5 technical report. arXiv. Preprint posted online on Dec 19, 2024. [doi: 10.48550\/arXiv.2412.15115]"},{"key":"R21","unstructured":"Guo D Yang D Zhang H Song J Wang P Zhu Q et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv. Preprint posted online on Jan 22, 2025. [doi: 10.48550\/arXiv.2501.12948]"},{"key":"R22","unstructured":"Hurst A Lerer A Goucher AP Perelman A Ramesh A Clark A et al. GPT-4o system card. arXiv. Preprint posted online on Oct 25, 2024. [doi: 10.48550\/arXiv.2410.21276]"},{"issue":"12","key":"R23","doi-asserted-by":"publisher","first-page":"e344","DOI":"10.1161\/STR.0000000000000211","article-title":"Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association\/American Stroke Association","volume":"50","author":"Powers","journal-title":"Stroke"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","journal-title":"Commun Med (Lond)"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01940-4","article-title":"CARE-AD: a multi-agent large language model framework for Alzheimer\u2019s disease prediction using longitudinal clinical notes","volume":"8","author":"Li","journal-title":"NPJ Digit Med"},{"issue":"9","key":"R27","doi-asserted-by":"publisher","first-page":"1433","DOI":"10.1002\/sim.1522","article-title":"A mixed-effects multinomial logistic regression model","volume":"22","author":"Hedeker","journal-title":"Stat Med"},{"key":"R28","unstructured":"Li T Zhang G Do QD Yue X Chen W . Long-context LLMs struggle with long in-context learning. arXiv. Preprint posted online on Apr 2, 2024. [doi: 10.48550\/arXiv.2404.02060]"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01651-w","article-title":"Leveraging long context in retrieval augmented language models for medical question answering","volume":"8","author":"Zhang","journal-title":"NPJ Digit Med"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.2196\/71613","article-title":"Patient triage and guidance in emergency departments using large language models: multimetric study","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","article-title":"An evaluation framework for clinical use of large language models in patient interaction tasks","volume":"31","author":"Johri","journal-title":"Nat Med"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.2196\/69929","article-title":"Performance of ChatGPT-4o and four open-source large language models in generating diagnoses based on China's rare disease catalog: comparative study","volume":"27","author":"Zhong","journal-title":"J Med Internet Res"},{"issue":"10","key":"R33","doi-asserted-by":"publisher","first-page":"2886","DOI":"10.1038\/s41591-024-03139-8","article-title":"Integrated image-based deep learning and language models for primary diabetes care","volume":"30","author":"Li","journal-title":"Nat Med"},{"issue":"8","key":"R34","doi-asserted-by":"publisher","first-page":"1040","DOI":"10.1001\/jamainternmed.2021.3333","article-title":"The Epic Sepsis Model falls short-the importance of external validation","volume":"181","author":"Habib","journal-title":"JAMA Intern Med"},{"issue":"8","key":"R35","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","journal-title":"JAMA Intern Med"},{"issue":"10","key":"R36","doi-asserted-by":"publisher","first-page":"2966","DOI":"10.1161\/STROKEAHA.125.051993","article-title":"Accuracy of large language models to identify stroke subtypes within unstructured electronic health record data","volume":"56","author":"Owens","journal-title":"Stroke"},{"issue":"4","key":"R37","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","journal-title":"Nat Med"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01830-9","article-title":"Evaluation of performance of generative large language models for stroke care","volume":"8","author":"Lee","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/s13054-023-04425-6","article-title":"Artificial hallucination: GPT on LSD?","volume":"27","author":"Beutel","journal-title":"Crit Care"},{"issue":"8017","key":"R40","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"Farquhar","journal-title":"Nature"},{"issue":"6","key":"R41","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000899","article-title":"Evaluating large language models for drafting emergency department encounter summaries","volume":"4","author":"Williams","journal-title":"PLOS Digit Health"},{"key":"R42","unstructured":"Enhancing clinical decision-making in acute ischemic stroke with a hybrid-reasoning framework based on large language models. Anonymous GitHub. URL: https:\/\/anonymous.4open.science\/r\/HR-LLM-Stroke\/README.md [Accessed 27-07-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T17:42:22Z","timestamp":1785433342000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e96304"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,30]]},"references-count":42,"URL":"https:\/\/doi.org\/10.2196\/96304","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,30]]},"article-number":"v28i10440e96304"}}