{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T00:54:20Z","timestamp":1790297660887,"version":"4.1.0"},"reference-count":44,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIRx Med"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Studies have shown that large language models (LLMs) are promising in therapeutic decision-making, with findings comparable to those of medical experts, but these studies used highly curated patient data.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to determine if LLMs can make guideline-concordant treatment decisions based on patient data as typically present in clinical practice (lengthy, unstructured medical text).<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a retrospective study of 80 patients with severe aortic stenosis who were scheduled for either surgical (SAVR; n=24) or transcatheter aortic valve replacement (TAVR; n=56) by our institutional heart team in 2022. Various LLMs (BioGPT, GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4o, LLaMA-2, Mistral, PaLM 2, and DeepSeek-R1) were queried using either anonymized original medical reports or manually generated case summaries to determine the most guideline-concordant treatment. We measured agreement with the heart team using Cohen \u03ba coefficients, reliability using intraclass correlation coefficients (ICCs), and fairness using the frequency bias index (FBI; FBI >1 indicated bias toward TAVR).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n When presented with original medical reports, LLMs showed poor performance (Cohen \u03ba coefficient: \u22120.47 to 0.22; ICC: 0.0\u20101.0; FBI: 0.95\u20101.51). The LLMs\u2019 performance improved substantially when case summaries were used as input and additional guideline knowledge was added to the prompt (Cohen \u03ba coefficient: \u22120.02 to 0.63; ICC: 0.01\u20101.0; FBI: 0.46\u20101.23). Qualitative analysis revealed instances of hallucinations in all LLMs tested.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Even advanced LLMs require extensively curated input for informed treatment decisions. Unreliable responses, bias, and hallucinations pose significant health risks and highlight the need for caution in applying LLMs to real-world clinical decision-making.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/74899","type":"journal-article","created":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T14:15:09Z","timestamp":1762352109000},"page":"e74899-e74899","source":"Crossref","is-referenced-by-count":11,"title":["Assessing the Limitations of Large Language Models in Clinical Practice Guideline\u2013Concordant Treatment Decision-Making on Real-World Data: Retrospective Study"],"prefix":"10.2196","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8376-3366","authenticated-orcid":false,"given":"Tobias","family":"Roeschl","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9807-8292","authenticated-orcid":false,"given":"Marie","family":"Hoffmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3933-7092","authenticated-orcid":false,"given":"Djawid","family":"Hashemi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5337-977X","authenticated-orcid":false,"given":"Felix","family":"Rarreck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1989-6156","authenticated-orcid":false,"given":"Nils","family":"Hinrichs","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8575-0019","authenticated-orcid":false,"given":"Tobias Daniel","family":"Trippel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2509-3034","authenticated-orcid":false,"given":"Matthias I","family":"Gr\u00f6schel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2851-695X","authenticated-orcid":false,"given":"Axel","family":"Unbehaun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1016-7971","authenticated-orcid":false,"given":"Christoph","family":"Klein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2009-6375","authenticated-orcid":false,"given":"J\u00f6rg","family":"Kempfert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1909-4329","authenticated-orcid":false,"given":"Henryk","family":"Dreger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7024-2337","authenticated-orcid":false,"given":"Benjamin","family":"O'Brien","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0132-0773","authenticated-orcid":false,"given":"Gerhard","family":"Hindricks","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1575-2056","authenticated-orcid":false,"given":"Felix","family":"Balzer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7911-8620","authenticated-orcid":false,"given":"Volkmar","family":"Falk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6944-2478","authenticated-orcid":false,"given":"Alexander","family":"Meyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,11,3]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a generative artificial intelligence model in a complex diagnostic challenge","volume":"330","author":"Kanjee","journal-title":"JAMA"},{"key":"R3","unstructured":"Tu T Palepu A Schaekermann M et al. Towards conversational diagnostic AI. arXiv. Preprint posted online on Jan 11, 2024. URL: https:\/\/arxiv.org\/abs\/2401.05654 [Accessed 01-10-2025]"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41523-023-00557-8","article-title":"Large language model (ChatGPT) as a support tool for breast tumor board","volume":"9","author":"Sorin","journal-title":"NPJ Breast Cancer"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejca.2024.114100","article-title":"ChatGPT\u2019s gastrointestinal tumor board tango: a limping dance partner?","volume":"205","author":"Aghamaliyev","journal-title":"Eur J Cancer"},{"issue":"108238","key":"R6","doi-asserted-by":"publisher","first-page":"108238","DOI":"10.1016\/j.clineuro.2024.108238","article-title":"Chat-GPT on brain tumors: an examination of artificial intelligence\/machine learning\u2019s ability to provide diagnoses and treatment plans for example neuro-oncology cases","volume":"239","author":"Kozel","journal-title":"Clin Neurol Neurosurg"},{"issue":"6","key":"R7","doi-asserted-by":"publisher","first-page":"1831","DOI":"10.1007\/s00404-023-07130-5","article-title":"Evaluating ChatGPT as an adjunct for the multidisciplinary tumor board decision-making in primary breast cancer cases","volume":"308","author":"Lukac","journal-title":"Arch Gynecol Obstet"},{"issue":"8","key":"R8","doi-asserted-by":"publisher","first-page":"e496","DOI":"10.4244\/EIJ-D-23-00643","article-title":"A study of ChatGPT in facilitating heart team decisions on severe aortic stenosis","volume":"20","author":"Salihu","journal-title":"EuroIntervention"},{"issue":"25","key":"R9","doi-asserted-by":"publisher","first-page":"2982","DOI":"10.1016\/j.jacc.2020.11.010","article-title":"Global burden of cardiovascular diseases and risk factors, 1990-2019: update from the GBD 2019 study","volume":"76","author":"Roth","journal-title":"J Am Coll Cardiol"},{"issue":"7","key":"R10","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1093\/eurheartj\/ehab395","article-title":"2021 ESC\/EACTS Guidelines for the management of valvular heart disease","volume":"43","author":"Vahanian","journal-title":"Eur Heart J"},{"key":"R11","unstructured":"Ye J Chen X Xu N et al. A comprehensive capability analysis of GPT-3 and GPT-3.5 series models. arXiv. Preprint posted online on Dec 23, 2023. URL: https:\/\/arxiv.org\/abs\/2303.10420 [Accessed 01-10-2025]"},{"key":"R12","unstructured":"Achiam J Adler S Agarwal S et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 4, 2024. URL: https:\/\/arxiv.org\/abs\/2303.08774 [Accessed 01-10-2025]"},{"key":"R13","unstructured":"Anil R Dai AM Firat O et al. PaLM 2 technical report. arXiv. Preprint posted online on Sep 13, 2023. URL: https:\/\/arxiv.org\/abs\/2305.10403 [Accessed 01-10-2025]"},{"key":"R14","unstructured":"Guo D Yang D Zhang H et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv. Preprint posted online on Jan 22, 2025. URL: https:\/\/arxiv.org\/abs\/2501.12948 [Accessed 01-10-2025]"},{"key":"R15","unstructured":"Jiang AQ Sablayrolles A Mensch A et al. Mistral 7B. arXiv. Preprint posted online on Oct 10, 2023. URL: https:\/\/arxiv.org\/abs\/2310.06825 [Accessed 01-10-2025]"},{"key":"R16","unstructured":"Touvron H Martin L Stone K et al. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online on Jul 19, 2023. URL: https:\/\/arxiv.org\/abs\/2307.09288 [Accessed 01-10-2025]"},{"issue":"6","key":"R17","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","journal-title":"Brief Bioinformatics"},{"key":"R18","unstructured":"Li\u00e9vin V Hother CE Motzfeldt AG Winther O . Can large language models reason about medical questions? arXiv. Preprint posted online on Dec 24, 2022. URL: https:\/\/arxiv.org\/abs\/2207.08143 [Accessed 01-10-2025]"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","journal-title":"J Chiropr Med"},{"issue":"6","key":"R20","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.5742","article-title":"Comparison of outcomes after transcatheter aortic valve replacement vs surgical aortic valve replacement among patients with aortic stenosis at low operative risk","volume":"2","author":"Virtanen","journal-title":"JAMA Netw Open"},{"key":"R21","unstructured":"Huang L Yu W Ma W et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. arXiv. Preprint posted online on Nov 19, 2024. URL: https:\/\/arxiv.org\/abs\/2311.05232 [Accessed 01-10-2025]"},{"key":"R22","unstructured":"Cai Y Wang L Wang Y et al. MedBench: a large-scale chinese benchmark for evaluating medical large language models. arXiv. Preprint posted online on Dec 20, 2023. URL: https:\/\/arxiv.org\/abs\/2312.12806 [Accessed 01-10-2025]"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2300031","article-title":"Use of GPT-4 to diagnose complex clinical cases","volume":"1","author":"Eriksen","journal-title":"NEJM AI"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Novak A Zeljkovi\u0107 I Rode F et al. The pulse of artificial intelligence in cardiology: a comprehensive evaluation of state-of-the-art large language models for potential use in clinical cardiology. medRxiv. Preprint posted online on Dec 7, 2024. URL: https:\/\/www.medrxiv.org\/content\/10.1101\/2023.08.08.23293689v3 [Accessed 01-10-2025] [doi: 10.1101\/2023.08.08.23293689]","DOI":"10.1101\/2023.08.08.23293689"},{"key":"R26","unstructured":"Liu NF Lin K Hewitt J et al. Lost in the middle: how language models use long contexts. arXiv. Preprint posted online on Nov 20, 2023. URL: https:\/\/arxiv.org\/abs\/2307.03172 [Accessed 01-10-2025]"},{"issue":"3","key":"R27","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1162\/neco_a_01563","article-title":"Large language models and the reverse turing test","volume":"35","author":"Sejnowski","journal-title":"Neural Comput"},{"key":"R28","unstructured":"Wang B Wei C Liu Z Lin G Chen NF . Resilience of large language models for noisy instructions. arXiv. Preprint posted online on Oct 3, 2024. URL: https:\/\/arxiv.org\/abs\/2404.09754 [Accessed 01-10-2025]"},{"key":"R29","unstructured":"Levy M Jacoby A Goldberg Y . Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv. Preprint posted online on Jul 10, 2024. URL: https:\/\/arxiv.org\/abs\/2402.14848 [Accessed 01-10-2025]"},{"issue":"9","key":"R30","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","journal-title":"Nat Med"},{"issue":"7972","key":"R31","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature New Biol"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.2196\/59823","article-title":"The clinicians\u2019 guide to large language models: a general perspective with a focus on hallucinations","volume":"14","author":"Roustan","journal-title":"Interact J Med Res"},{"issue":"1\u20136","key":"R33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10796-023-10412-7","article-title":"Responsible artificial intelligence (AI) for digital health and medical analytics","volume":"5","author":"Sivarajah","journal-title":"Inf Syst Front"},{"key":"R34","unstructured":"Luo H Specia L . From understanding to utilization: a survey on explainability for large language models. arXiv. Preprint posted online on Feb 22, 2024. URL: https:\/\/arxiv.org\/abs\/2401.12874 [Accessed 01-10-2025]"},{"key":"R35","unstructured":"Liu L Pan Y Li X Chen G . Uncertainty estimation and quantification for LLMs: a simple supervised approach. arXiv. Preprint posted online on Oct 23, 2024. URL: https:\/\/arxiv.org\/abs\/2404.15993 [Accessed 01-10-2025]"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.2196\/51834","article-title":"Cost, usability, credibility, fairness, accountability, transparency, and explainability framework for safe and effective large language models in medical education: narrative review and qualitative study","volume":"3","author":"Quttainah","journal-title":"JMIR AI"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/51319","article-title":"Assessing the current limitations of large language models in advancing health care education","volume":"9","author":"Kim","journal-title":"JMIR Form Res"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured information extraction from scientific text with large language models","volume":"15","author":"Dagdelen","journal-title":"Nat Commun"},{"issue":"4","key":"R39","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1001\/jamaophthalmol.2023.6917","article-title":"Assessment of a large language model\u2019s responses to questions and cases about glaucoma and retina management","volume":"142","author":"Huang","journal-title":"JAMA Ophthalmol"},{"key":"R40","unstructured":"Wei J Wang X Schuurmans D et al. Chain-of-thought prompting elicits reasoning in large language models. Presented at: NIPS \u201922: Proceedings of the 36th International Conference on Neural Information Processing Systems; Nov 28 to Dec 9, 2022; New Orleans, LA, USA. [doi: 10.5555\/3600270.3602070]"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-46995-z","article-title":"Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination","volume":"13","author":"Roso\u0142","journal-title":"Sci Rep"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajem.2024.03.017","article-title":"Comparison of emergency medicine specialist, cardiologist, and chat-GPT in electrocardiography assessment","volume":"80","author":"G\u00fcnay","journal-title":"Am J Emerg Med"},{"issue":"7","key":"R43","doi-asserted-by":"publisher","first-page":"1406","DOI":"10.1016\/j.cjca.2025.01.036","article-title":"Beyond text: the impact of clinical context on GPT-4\u2019s 12-lead electrocardiogram interpretation accuracy","volume":"41","author":"Zeljkovic","journal-title":"Can J Cardiol"},{"issue":"6","key":"R44","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1001\/jama.2024.0018","article-title":"AI\u2019s threat to the medical profession","volume":"331","author":"Fogo","journal-title":"JAMA"}],"container-title":["JMIRx Med"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/xmed.jmir.org\/2025\/1\/e74899","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:05:39Z","timestamp":1782842739000},"score":1,"resource":{"primary":{"URL":"https:\/\/xmed.jmir.org\/2025\/1\/e74899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,3]]},"references-count":44,"URL":"https:\/\/doi.org\/10.2196\/74899","relation":{},"ISSN":["2563-6316"],"issn-type":[{"value":"2563-6316","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,3]]},"article-number":"v6i2e74899"}}