{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T19:02:19Z","timestamp":1789671739037,"version":"4.0.1"},"reference-count":27,"publisher":"Oxford University Press (OUP)","license":[{"start":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T00:00:00Z","timestamp":1789603200000},"content-version":"am","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To develop and validate an open-source evaluation framework that assesses clinical AI diagnostic systems across multiple clinically relevant dimensions (accuracy, cost, time, invasiveness, physician effort, and safety behaviors), addressing the limitations of accuracy-only benchmarks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>We developed DiagnosticXchange, a dynamic clinical simulation platform where AI systems interact with a simulated hospital by ordering tests, requesting imaging, and performing procedures, with each action mapped to Current Procedural Terminology (CPT) codes capturing cost, time, work relative value units, and invasiveness. We validated the framework using 8 large language models on 216 peer-reviewed cases across 19 specialties (1728 sessions). Analyses included competing-risk survival analysis, unsupervised clustering of reasoning strategies, diagnostic contribution scoring, and a pilot comparison with 14 neurologists.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Three systems achieved near-identical accuracy (93.5%-94.0%) yet differed significantly in cost (P &amp;lt;.001; 1.75-fold between the most and least expensive top-3 systems) and 2.1-fold in physician oversight. Competing-risk analysis showed the most efficient system solved 86% of cases within $5000, while the most resource-intensive required $10\u00a0000 for equivalent accuracy. Unsupervised clustering identified 3 reasoning strategies; behavioral profiles predicted resource consumption better than system identity (AIC: 4683 vs 5056). Safety analysis revealed premature diagnosis (up to 9.3%), noncontributory invasive procedures (8.7%-29.9%), and futile invasive procedures on failed cases (3.7%-18.1%). Test-retest analysis confirmed reproducible accuracy (84.6% concordance) but substantial process variability (cost CV: 86.6%). A pilot comparison captured multi-dimensional performance of 14 neurologists alongside AI systems.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Accuracy-only evaluation is insufficient for safe clinical AI deployment. DiagnosticXchange provides an open-source, reproducible framework for multi-dimensional assessment that reveals clinically consequential differences invisible to existing benchmarks.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag143","type":"journal-article","created":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T11:37:58Z","timestamp":1786448278000},"source":"Crossref","is-referenced-by-count":0,"title":["DiagnosticXchange: an open-source framework for evaluating safety, efficiency, and diagnostic reasoning in clinical AI systems"],"prefix":"10.1093","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4964-8943","authenticated-orcid":false,"given":"Moran","family":"Sorka","sequence":"first","affiliation":[{"name":"Faculty of Biology, Technion\u2014Israel Institute of Technology , Haifa 3200003,","place":["Israel"]},{"name":"Ruth and Bruce Rappaport Faculty of Medicine, Technion\u2014Israel Institute of Technology , Haifa 3525433,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7542-8608","authenticated-orcid":false,"given":"Alon","family":"Gorenshtein","sequence":"additional","affiliation":[{"name":"Ruth and Bruce Rappaport Faculty of Medicine, Technion\u2014Israel Institute of Technology , Haifa 3525433,","place":["Israel"]},{"name":"Department of Neurology, Rambam Medical Center , Haifa 3109601,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hillel","family":"Abramovitch","sequence":"additional","affiliation":[{"name":"Department of Neurology, Rambam Medical Center , Haifa 3109601,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1285-0021","authenticated-orcid":false,"given":"Pannathat","family":"Soontrapa","sequence":"additional","affiliation":[{"name":"Division of Neurology, Department of Medicine, Faculty of Medicine Siriraj Hospital, Mahidol University , Bangkok 10700,","place":["Thailand"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3585-1687","authenticated-orcid":false,"given":"Shahar","family":"Shelly","sequence":"additional","affiliation":[{"name":"Ruth and Bruce Rappaport Faculty of Medicine, Technion\u2014Israel Institute of Technology , Haifa 3525433,","place":["Israel"]},{"name":"Department of Neurology, Rambam Medical Center , Haifa 3109601,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6334-5039","authenticated-orcid":false,"given":"Dvir","family":"Aran","sequence":"additional","affiliation":[{"name":"Faculty of Biology, Technion\u2014Israel Institute of Technology , Haifa 3200003,","place":["Israel"]},{"name":"The Taub Faculty of Computer Science, Technion\u2014Israel Institute of Technology , Haifa 3200003,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"2026091714051241700_ocag143-B1","author":"Nori","year":"2023"},{"key":"2026091714051241700_ocag143-B2","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2026091714051241700_ocag143-B3","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1038\/s41586-025-08869-4","article-title":"Towards accurate differential diagnosis with large language models","volume":"642","author":"McDuff","year":"2025","journal-title":"Nature"},{"key":"2026091714051241700_ocag143-B4","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1038\/s41586-025-08866-7","article-title":"Towards conversational diagnostic artificial intelligence","volume":"642","author":"Tu","year":"2025","journal-title":"Nature"},{"key":"2026091714051241700_ocag143-B5","doi-asserted-by":"crossref","first-page":"e60391","DOI":"10.2196\/60391","article-title":"GPT-4 as a clinical decision support tool in ischemic stroke management: evaluation study","volume":"4","author":"Shmilovitch","year":"2025","journal-title":"JMIR AI"},{"key":"2026091714051241700_ocag143-B6","doi-asserted-by":"crossref","first-page":"AIe2401235","DOI":"10.1056\/AIe2401235","article-title":"It\u2019s time to bench the medical exam benchmark","volume":"2","author":"Raji","year":"2025","journal-title":"NEJM AI"},{"key":"2026091714051241700_ocag143-B7","doi-asserted-by":"crossref","first-page":"1501","DOI":"10.1001\/jama.2019.13978","article-title":"Waste in the US health care system: estimated costs and potential for savings","volume":"322","author":"Shrank","year":"2019","journal-title":"JAMA"},{"key":"2026091714051241700_ocag143-B8","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/S0140-6736(16)32585-5","article-title":"Evidence for overuse of medical services around the world","volume":"390","author":"Brownlee","year":"2017","journal-title":"Lancet"},{"key":"2026091714051241700_ocag143-B9","doi-asserted-by":"crossref","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","year":"2021","journal-title":"Appl Sci"},{"key":"2026091714051241700_ocag143-B10","doi-asserted-by":"publisher","first-page":"2019:2567","DOI":"10.18653\/v1\/D19-1259","author":"Jin","year":"2019"},{"key":"2026091714051241700_ocag143-B11","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2026091714051241700_ocag143-B12","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"2026091714051241700_ocag143-B13","doi-asserted-by":"publisher","author":"Arora","year":"2025","DOI":"10.48550\/arXiv.2505.08775"},{"key":"2026091714051241700_ocag143-B14","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2500144","article-title":"MedAgentBench: a virtual EHR environment to benchmark medical LLM agents","volume":"2","author":"Jiang","year":"2025","journal-title":"NEJM AI"},{"key":"2026091714051241700_ocag143-B15","doi-asserted-by":"publisher","author":"Schmidgall","year":"2024","DOI":"10.48550\/arXiv.2405.07960"},{"key":"2026091714051241700_ocag143-B16","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","article-title":"An evaluation framework for clinical use of large language models in patient interaction tasks","volume":"31","author":"Johri","year":"2025","journal-title":"Nat Med"},{"key":"2026091714051241700_ocag143-B17","author":"Nori","year":"2025"},{"key":"2026091714051241700_ocag143-B18","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-025-04151-2","article-title":"Holistic evaluation of large language models for medical tasks with MedHELM","volume":"32","author":"Bedi","year":"2026","journal-title":"Nat Med"},{"key":"2026091714051241700_ocag143-B19","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1038\/s41591-022-01772-9","article-title":"Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"28","author":"Vasey","year":"2022","journal-title":"Nat Med"},{"key":"2026091714051241700_ocag143-B20","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1093\/jamia\/ocae301","article-title":"AI as an intervention: improving clinical outcomes relies on a causal approach","volume":"32","author":"Joshi","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"2026091714051241700_ocag143-B21","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1080\/01621459.1999.10474144","article-title":"A proportional hazards model for the subdistribution of a competing risk","volume":"94","author":"Fine","year":"1999","journal-title":"J Am Stat Assoc"},{"key":"2026091714051241700_ocag143-B22","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1161\/CIRCULATIONAHA.115.017719","article-title":"Introduction to the analysis of survival data in the presence of competing risks","volume":"133","author":"Austin","year":"2016","journal-title":"Circulation"},{"key":"2026091714051241700_ocag143-B23","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.1214\/aos\/1176350951","article-title":"A class of K-sample tests for comparing the cumulative incidence of a competing risk","volume":"16","author":"Gray","year":"1988","journal-title":"Ann Stat"},{"key":"2026091714051241700_ocag143-B24","author":"Cheng","year":"2025"},{"key":"2026091714051241700_ocag143-B25","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1001\/jamainternmed.2024.0295","article-title":"Clinical reasoning of a generative artificial intelligence model compared with physicians","volume":"184","author":"Cabral","year":"2024","journal-title":"JAMA Intern Med"},{"key":"2026091714051241700_ocag143-B26","doi-asserted-by":"crossref","first-page":"e64963","DOI":"10.2196\/64963","article-title":"Comparing diagnostic accuracy of clinical professionals and large language models: a systematic review and meta-analysis","volume":"13","author":"Shan","year":"2025","journal-title":"JMIR Med Inform."},{"key":"2026091714051241700_ocag143-B27","author":"U.S. Food and Drug Administration","year":"2025."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag143\/71191995\/ocag143.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag143\/71191995\/ocag143.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T18:05:19Z","timestamp":1789668319000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article\/doi\/10.1093\/jamia\/ocag143\/8812673"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,17]]},"references-count":27,"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag143","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,17]]},"article-number":"ocag143"}}