{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:52:42Z","timestamp":1792792362854,"version":"4.5.0"},"reference-count":67,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T00:00:00Z","timestamp":1761004800000},"content-version":"vor","delay-in-days":49,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,9,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate predictive diagnostic performance of open- and closed-source large language models (LLMs) in emergency medicine, addressing the urgent need for innovative clinical decision support tools amid rising patient volumes and staffing shortages.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We generated 2370 AI-driven diagnostic predictions (Top-5 diagnoses from each of 6 model pipelines per patient), using data from 79 real-world emergency department cases collected consecutively during a 24-hour peak influx period at a tertiary care center. Pipelines combined open- and closed-source embedding models (text-embedding-ada-002, MXBAI) with foundational models (GPT-4, Llama3, and Qwen2) grounded via retrieval-augmented generation using emergency medicine textbooks. Models\u2019 predictions were assessed against reference diagnoses established by expert consensus.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>All pipelines achieved comparable diagnostic match rates (62.03%-72.15%). Diagnostic performance was significantly influenced by case characteristics: match rates were notably higher for specific versus unspecific diagnoses (85.53% vs 31.41%, P\u2009&amp;lt;\u2009.001) and surgical versus medical cases (79.49% vs 56.25%, P\u2009&amp;lt;\u2009.001). Open-source models demonstrated markedly superior sourcing capabilities compared to GPT-4-based combinations (P\u2009&amp;lt;\u20091.4e-12), with MBXAI\/Qwen2 pipeline achieving perfect citation verification.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Diagnostic accuracy primarily depended on case characteristics rather than the choice of model pipeline, highlighting fundamental AI alignment challenges in clinical reasoning. Low performance in unspecific diagnoses underscores inherent complexities in clinical definitions rather than technological shortcomings alone.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Open-source LLM pipelines provide enhanced sourcing capabilities, crucial for transparent clinical decision-making and interpretability. Further research should expand knowledge bases to include hospital guidelines and regional epidemiology, while exploring on-premises solutions to better align with privacy regulations and clinical integration.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooaf119","type":"journal-article","created":{"date-parts":[[2025,10,2]],"date-time":"2025-10-02T12:00:33Z","timestamp":1759406433000},"source":"Crossref","is-referenced-by-count":12,"title":["Grounded large language models for diagnostic prediction in real-world emergency department settings"],"prefix":"10.1093","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8341-9338","authenticated-orcid":false,"given":"Alexandre","family":"Niset","sequence":"first","affiliation":[{"name":"Cliniques Universitaires Saint-Luc Emergency Department, , Brussels, 1200,","place":["Belgium"]},{"name":"Cliniques Universitaires Saint-Luc Pediatric Intensive Care Unit, , Brussels, 1200,","place":["Belgium"]},{"name":"Sciense New York, NY, 10004,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ines","family":"Melot","sequence":"additional","affiliation":[{"name":"CHU Helora\u2014Jolimont Emergency Department, , La Louvi\u00e8re, 7100,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margaux","family":"Pireau","sequence":"additional","affiliation":[{"name":"Cliniques Universitaires Saint-Luc Emergency Department, , Brussels, 1200,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandre","family":"Englebert","sequence":"additional","affiliation":[{"name":"Sciense New York, NY, 10004,","place":["United States"]},{"name":"UCLouvain Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), , Louvain-la-Neuve, 1348,","place":["Belgium"]},{"name":"Orthopedic Department, CHU UCL Namur Yvoir, 5530,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan","family":"Scius","sequence":"additional","affiliation":[{"name":"CHU UCL Namur Emergency Department, , Yvoir, 5530,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julien","family":"Flament","sequence":"additional","affiliation":[{"name":"CHU UCL Namur Emergency Department, , Yvoir, 5530,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salim","family":"El Hadwe","sequence":"additional","affiliation":[{"name":"Sciense New York, NY, 10004,","place":["United States"]},{"name":"University of Cambridge Department of Clinical Neurosciences, , Cambridge, CB2 0QQ,","place":["United Kingdom"]},{"name":"University of Cambridge Bioelectronics Laboratory, Department of Electrical Engineering, , Cambridge, CB2 0QQ,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mejdeddine","family":"Al Barajraji","sequence":"additional","affiliation":[{"name":"Sciense New York, NY, 10004,","place":["United States"]},{"name":"CHR Citadelle Department of Neurosurgery, , Liege, 4000,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henri","family":"Thonon","sequence":"additional","affiliation":[{"name":"CHU UCL Namur Emergency Department, , Yvoir, 5530,","place":["Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sami","family":"Barrit","sequence":"additional","affiliation":[{"name":"Sciense New York, NY, 10004,","place":["United States"]},{"name":"University of California Computational Precision Health, , Berkeley, CA, 94720-2284,","place":["United States"]},{"name":"Department of Neurological Surgery, University of California San Francisco, CA, 90095,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,10,21]]},"reference":[{"key":"2025102109462324600_ooaf119-B1","doi-asserted-by":"publisher","first-page":"E851","DOI":"10.1503\/cmaj.230719","article-title":"Emergency departments are in crisis now and for the foreseeable future","volume":"195","author":"Varner","year":"2023","journal-title":"CMAJ"},{"key":"2025102109462324600_ooaf119-B2","author":"American College of Emergency Physicians","year":"2022"},{"key":"2025102109462324600_ooaf119-B3","doi-asserted-by":"publisher","first-page":"1300","DOI":"10.1056\/NEJMp068194","article-title":"Crisis in the emergency department","volume":"355","author":"Kellermann","year":"2006","journal-title":"N Engl J Med"},{"key":"2025102109462324600_ooaf119-B4","doi-asserted-by":"publisher","first-page":"xiii","DOI":"10.1111\/j.1475-6773.2006.00692.x","article-title":"Emergency departments in crisis: opportunities for research","volume":"42","author":"Clancy","year":"2007","journal-title":"Health Serv Res"},{"key":"2025102109462324600_ooaf119-B5","doi-asserted-by":"publisher","first-page":"947","DOI":"10.1111\/j.1553-2712.1999.tb01246.x","article-title":"Clinical decision making: an emergency medicine perspective","volume":"6","author":"Kovacs","year":"1999","journal-title":"Acad Emerg Med"},{"key":"2025102109462324600_ooaf119-B6","doi-asserted-by":"publisher","first-page":"e40031","DOI":"10.2196\/40031","article-title":"Artificial intelligence in emergency medicine: viewpoint of current applications and foreseeable opportunities and challenges","volume":"25","author":"Chenais","year":"2023","journal-title":"J Med Internet Res."},{"key":"2025102109462324600_ooaf119-B7","author":"Brown","year":"2020"},{"key":"2025102109462324600_ooaf119-B8","author":"Liu","year":"2024"},{"key":"2025102109462324600_ooaf119-B9","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1080\/13645706.2019.1575882","article-title":"Introduction to artificial intelligence in medicine","volume":"28","author":"Mintz","year":"2019","journal-title":"Minim Invasive Ther Allied Technol"},{"key":"2025102109462324600_ooaf119-B10","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesko","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2025102109462324600_ooaf119-B11","author":"Bommasani","year":"2021"},{"key":"2025102109462324600_ooaf119-B12","doi-asserted-by":"publisher","first-page":"108013","DOI":"10.1016\/j.cmpb.2024.108013","article-title":"ChatGPT in healthcare: a taxonomy and systematic review","volume":"245","author":"Li","year":"2024","journal-title":"Comput Methods Programs Biomed"},{"key":"2025102109462324600_ooaf119-B13","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2025102109462324600_ooaf119-B14","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2025102109462324600_ooaf119-B15","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"key":"2025102109462324600_ooaf119-B16","author":"European Commission","year":"2021"},{"key":"2025102109462324600_ooaf119-B17","author":"European Union","year":"2016"},{"key":"2025102109462324600_ooaf119-B18","author":"Services Dohah","year":"2013"},{"key":"2025102109462324600_ooaf119-B19","author":"OpenAI","year":"2023"},{"key":"2025102109462324600_ooaf119-B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2012.07805,"},{"key":"2025102109462324600_ooaf119-B21","doi-asserted-by":"publisher","first-page":"80218","DOI":"10.1109\/access.2023.3300381","article-title":"From ChatGPT to ThreatGPT: impact of generative AI in cybersecurity and privacy","volume":"11","author":"Gupta","year":"2023","journal-title":"IEEE Access"},{"key":"2025102109462324600_ooaf119-B22","doi-asserted-by":"publisher","first-page":"1275","DOI":"10.1038\/s42256-023-00741-2","article-title":"Protein\u2013protein contact prediction by geometric triangle-aware protein language models","volume":"5","author":"Lin","year":"2023","journal-title":"Nat Mach Intell"},{"key":"2025102109462324600_ooaf119-B23","author":"Lewis","year":"2020"},{"key":"2025102109462324600_ooaf119-B24","doi-asserted-by":"publisher","author":"Barrit","DOI":"10.1101\/2024.02.11.24302658"},{"key":"2025102109462324600_ooaf119-B25","volume-title":"Tintinalli\u2019s Emergency Medicine: A Comprehensive Study Guide, 8e","author":"Tintinalli","year":"2016"},{"key":"2025102109462324600_ooaf119-B26","volume-title":"Rosen\u2019s Emergency Medicine: Concepts and Clinical Practice","author":"Walls","year":"2022"},{"key":"2025102109462324600_ooaf119-B27","doi-asserted-by":"crossref","DOI":"10.1093\/med\/9780198784197.001.0001","volume-title":"Oxford Handbook of Emergency Medicine","author":"Wyatt","year":"2020"},{"key":"2025102109462324600_ooaf119-B28","volume-title":"Textbook of Critical Care E-Book","author":"Vincent","year":"2022"},{"key":"2025102109462324600_ooaf119-B29","volume-title":"ICD-10: International Statistical Classification of Diseases and Related Health Problems: Tenth Revision","author":"World Health Organization","year":"2004","edition":"2nd ed"},{"key":"2025102109462324600_ooaf119-B30","doi-asserted-by":"publisher","first-page":"751","DOI":"10.7326\/0003-4819-78-5-751","article-title":"The problems of the \"problem-oriented medical record","volume":"78","author":"Feinstein","year":"1973","journal-title":"Ann Intern Med"},{"key":"2025102109462324600_ooaf119-B31","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1056\/NEJM196803142781105","article-title":"Medical records that guide and teach","volume":"278","author":"Weed","year":"1968","journal-title":"N Engl J Med"},{"key":"2025102109462324600_ooaf119-B32","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1055\/s-0038-1634558","article-title":"Desiderata for controlled medical vocabularies in the twenty-first century","volume":"37","author":"Cimino","year":"1998","journal-title":"Methods Inf Med"},{"key":"2025102109462324600_ooaf119-B33","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1055\/s-0038-1634569","article-title":"A terminology server for medical language and medical information systems","volume":"34","author":"Rector","year":"1995","journal-title":"Methods Inf Med"},{"key":"2025102109462324600_ooaf119-B34","doi-asserted-by":"publisher","author":"Johnson","year":"2025","DOI":"10.1101\/2024.12.03.24318322"},{"key":"2025102109462324600_ooaf119-B35","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1080\/17470218.2017.1322622","article-title":"Hick\u2019s law for choice reaction time: a review","volume":"71","author":"Proctor","year":"2018","journal-title":"Q J Exp Psychol (Hove)"},{"key":"2025102109462324600_ooaf119-B36","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1186\/s12911-017-0430-8","article-title":"Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system","volume":"17","author":"Ancker","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"key":"2025102109462324600_ooaf119-B37","doi-asserted-by":"publisher","first-page":"1489","DOI":"10.1056\/NEJM198906013202211","article-title":"Our stubborn quest for diagnostic certainty. A cause of excessive testing","volume":"320","author":"Kassirer","year":"1989","journal-title":"N Engl J Med"},{"key":"2025102109462324600_ooaf119-B38","doi-asserted-by":"publisher","first-page":"1223","DOI":"10.1111\/j.1553-2712.2000.tb00467.x","article-title":"The cognitive imperative: thinking about how we think","volume":"7","author":"Croskerry","year":"2000","journal-title":"Acad Emerg Med"},{"key":"2025102109462324600_ooaf119-B39","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1111\/j.1365-2923.2009.03507.x","article-title":"Diagnostic error and clinical reasoning","volume":"44","author":"Norman","year":"2010","journal-title":"Med Educ"},{"key":"2025102109462324600_ooaf119-B40","doi-asserted-by":"publisher","first-page":"893","DOI":"10.7326\/0003-4819-110-11-893","article-title":"Diagnostic reasoning","volume":"110","author":"Kassirer","year":"1989","journal-title":"Ann Intern Med"},{"key":"2025102109462324600_ooaf119-B41","doi-asserted-by":"publisher","first-page":"2377","DOI":"10.1001\/jama.2019.18058","article-title":"Addressing bias in artificial intelligence in health care","volume":"322","author":"Parikh","year":"2019","journal-title":"JAMA"},{"key":"2025102109462324600_ooaf119-B42","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care\u2014addressing ethical challenges","volume":"378","author":"Char","year":"2018","journal-title":"N Engl J Med"},{"key":"2025102109462324600_ooaf119-B43","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1038\/s41746-023-00844-5","article-title":"Harnessing big data for health equity through a comprehensive public database and data collection framework","volume":"6","author":"Sabet","year":"2023","journal-title":"NPJ Digit Med"},{"key":"2025102109462324600_ooaf119-B44","doi-asserted-by":"publisher","first-page":"e2440969","DOI":"10.1001\/jamanetworkopen.2024.40969","article-title":"Large language model influence on diagnostic reasoning: a randomized clinical trial","volume":"7","author":"Goh","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2025102109462324600_ooaf119-B45","author":"Torroba Hennigen","year":"2023"},{"key":"2025102109462324600_ooaf119-B46","volume-title":"LLatrieval: LLM-Verified Retrieval for Verifiable Generation","author":"Association for Computational Linguistics","year":"2024"},{"key":"2025102109462324600_ooaf119-B47","author":"Hoffmann","year":"2022"},{"key":"2025102109462324600_ooaf119-B48","author":"Gupta","year":"2024"},{"key":"2025102109462324600_ooaf119-B49","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2300031","article-title":"Use of GPT-4 to diagnose complex clinical cases","volume":"1","author":"Eriksen","year":"2024","journal-title":"NEJM AI"},{"key":"2025102109462324600_ooaf119-B50","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J Mach Learn Res"},{"key":"2025102109462324600_ooaf119-B51","doi-asserted-by":"publisher","first-page":"e10438","DOI":"10.1002\/lrh2.10438","article-title":"Diagnostic accuracy of GPT-4 on common clinical scenarios and challenging cases","volume":"8","author":"Rutledge","year":"2024","journal-title":"Learn Health Syst"},{"key":"2025102109462324600_ooaf119-B52","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1038\/s41746-025-01486-5","article-title":"Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms","volume":"8","author":"Yang","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2025102109462324600_ooaf119-B53","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"2025102109462324600_ooaf119-B54","doi-asserted-by":"publisher","DOI":"10.1101\/2024.03.12.24303785","article-title":"Influence of a large language model on diagnostic reasoning: a randomized clinical vignette study","author":"Goh","year":"2024","journal-title":"medRxiv"},{"key":"2025102109462324600_ooaf119-B55","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1067\/mem.2002.124900","article-title":"Human factors and ergonomics in the emergency department","volume":"40","author":"Wears","year":"2002","journal-title":"Ann Emerg Med"},{"key":"2025102109462324600_ooaf119-B56","doi-asserted-by":"publisher","first-page":"e78","DOI":"10.1016\/S2589-7500(20)30274-0","article-title":"Rapid triage for COVID-19 using routine clinical data for patients attending hospital: development and prospective validation of an artificial intelligence screening test","volume":"3","author":"Soltan","year":"2021","journal-title":"Lancet Digit Health"},{"key":"2025102109462324600_ooaf119-B57","doi-asserted-by":"publisher","first-page":"e012555","DOI":"10.1136\/bmjopen-2016-012555","article-title":"Evidence-based interventions to reduce adverse events in hospitals: a systematic review of systematic reviews","volume":"6","author":"Zegers","year":"2016","journal-title":"BMJ Open"},{"key":"2025102109462324600_ooaf119-B58","author":"Li","year":"2024"},{"key":"2025102109462324600_ooaf119-B59","author":"Han","year":"2024"},{"key":"2025102109462324600_ooaf119-B60","author":"Chen","year":"2023"},{"key":"2025102109462324600_ooaf119-B61","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1017\/s1481803500014482","article-title":"Prospective time study derivation of emergency physician workload predictors","volume":"7","author":"Innes","year":"2005","journal-title":"CJEM"},{"key":"2025102109462324600_ooaf119-B62","author":"Wang","year":"2022"},{"key":"2025102109462324600_ooaf119-B63","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025102109462324600_ooaf119-B64","author":"Wu","year":"2023"},{"key":"2025102109462324600_ooaf119-B65","author":"Mistral AI","year":"2023"},{"key":"2025102109462324600_ooaf119-B66","author":"Lepikhin","year":"2020"},{"key":"2025102109462324600_ooaf119-B67","author":"Shazeer","year":"2017"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/8\/5\/ooaf119\/64825916\/ooaf119.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/8\/5\/ooaf119\/64825916\/ooaf119.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T13:46:32Z","timestamp":1761054392000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooaf119\/8294036"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,2]]},"references-count":67,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9,2]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooaf119","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,10]]},"published":{"date-parts":[[2025,9,2]]},"article-number":"ooaf119"}}