{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T15:57:46Z","timestamp":1792684666814,"version":"4.5.0"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) such as ChatGPT-4, LLaMA-3.1, Gemini-1.5, DeepSeek-R1, and OpenAI-O3 have shown promising potential in health care, particularly for clinical reasoning and decision support. However, their reliability across critical tasks like diagnosis, medical coding, and risk prediction has received mixed reviews, especially in real-world settings without task-specific training.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n \n This study aims to evaluate and compare the zero-shot performance of reasoning and nonreasoning LLMs in three essential clinical tasks: (1) primary diagnosis generation, (2)\n ICD-9<\/jats:italic>\n (\n International Classification of Diseases, Ninth Revision<\/jats:italic>\n ) medical code prediction, and (3) hospital readmission risk stratification. The goal is to assess whether these models can serve as general-purpose clinical decision support tools and to identify gaps in current capabilities.\n <\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n Using the Medical Information Mart for Intensive Care-IV dataset, we selected a random cohort of 300 hospital discharge summaries. Prompts were engineered to include structured clinical content from 5 note sections: chief complaints, past medical history, surgical history, laboratories, and imaging. Prompts were standardized and zero-shot, with no model fine-tuning or repetition across runs. All model interactions were conducted through publicly available web user interfaces, without using application programming interfaces, to simulate real-world accessibility for nontechnical users. We incorporated rationale elicitation into prompts to evaluate model transparency, especially in reasoning models. Ground-truth labels were derived from the primary diagnosis documented in clinical notes, structured\n ICD-9<\/jats:italic>\n codes from diagnosis, and hospital-recorded readmission frequencies for risk stratification. Performance was measured using\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores and correctness percentages, and comparative performance was analyzed statistically.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Among nonreasoning models, LLaMA-3.1 achieved the highest primary diagnosis accuracy (n=255, 85%), followed by ChatGPT-4 (n=254, 84.7%) and Gemini-1.5 (n=237, 79%). For\n ICD-9<\/jats:italic>\n prediction, correctness dropped significantly across all models: LLaMA-3.1 (n=128, 42.6%), ChatGPT-4 (n=122, 40.6%), and Gemini-1.5 (n=44, 14.6%). Hospital readmission risk prediction showed low performance in nonreasoning models: LLaMA-3.1 (n=124, 41.3%), Gemini-1.5 (n=122, 40.7%), and ChatGPT-4 (n=99, 33%). Among reasoning models, OpenAI-O3 outperformed in diagnosis (n=270, 90%) and\n ICD-9<\/jats:italic>\n coding (n=136, 45.3%), while DeepSeek-R1 performed slightly better in the readmission risk prediction (n=218, 72.6% vs O3\u2019s n=212, 70.6%). Despite improved explainability, reasoning models generated verbose responses. None of the models met clinical standards across all tasks, and performance in medical coding remained the weakest area across all models.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n \n Current LLMs exhibit moderate success in zero-shot diagnosis and risk prediction but underperform in\n ICD-9<\/jats:italic>\n code generation, reinforcing findings from prior studies. Reasoning models offer marginally better performance and increased interpretability, with limited reliability. Overall, statistical analysis between the models revealed that OpenAI-O3 outperformed the other models. These results highlight the need for task-specific fine-tuning and need human-in-the-loop checking. Future work will explore fine-tuning, stability through repeated trials, and evaluation on a different subset of deidentified real-world data with a larger sample size.\n <\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/74142","type":"journal-article","created":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T06:45:08Z","timestamp":1749537908000},"page":"e74142-e74142","source":"Crossref","is-referenced-by-count":29,"title":["Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk Stratification: Zero-Shot Prompting Approach"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-5917-4558","authenticated-orcid":false,"given":"Parvati","family":"Naliyatthaliyazchayil","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6196-3116","authenticated-orcid":false,"given":"Raajitha","family":"Muthyala","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1097-316X","authenticated-orcid":false,"given":"Judy Wawira","family":"Gichoya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3625-534X","authenticated-orcid":false,"given":"Saptarshi","family":"Purkayastha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,30]]},"reference":[{"key":"R1","unstructured":"Minaee S Mikolov T Nikzad N et al. Large language models: a survey. arXiv. Preprint posted online on Mar 23, 2025. [doi: 10.48550\/arXiv.2402.06196]"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1002\/hcs2.61","article-title":"Large language models in health care: development, applications, and challenges","volume":"2","author":"Yang","journal-title":"Health Care Sci"},{"key":"R3","unstructured":"Kwan K . Large language models are good medical coders, if provided with tools. arXiv. Preprint posted online on Jul 6, 2024. [doi: 10.48550\/arXiv.2407.12849]"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2300040","article-title":"Large language models are poor medical coders\u2014benchmarking of medical code querying","volume":"1","author":"Soroush","journal-title":"NEJM AI"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1016\/j.thorsurg.2011.04.003","article-title":"Billing, coding, and credentialing in the thoracic surgery practice","volume":"21","author":"Cooke","journal-title":"Thorac Surg Clin"},{"key":"R7","unstructured":"Diagnostic code descriptions: ICD-9. Government of British Columbia. URL: https:\/\/www2.gov.bc.ca\/gov\/content\/health\/practitioner-professional-resources\/msp\/physicians\/diagnostic-code-descriptions-icd-9 [Accessed 09-11-2024]"},{"key":"R8","unstructured":"Johnson A Bulgarelli L Pollard T Horng S Celi LA Mark R . MIMIC-IV (version 2.0). PhysioNet. 2022. URL: https:\/\/physionet.org\/content\/mimiciv\/2.0\/ [Accessed 22-07-2025]"},{"issue":"13","key":"R9","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","journal-title":"N Engl J Med"},{"key":"R10","unstructured":"Zhu Y Wang Z Gao J et al. Prompting large language models for zero-shot clinical prediction with structured longitudinal electronic health record data. arXiv. Preprint posted online on Feb 10, 2024. [doi: 10.48550\/arXiv.2402.01713]"},{"key":"R11","unstructured":"Xie W Xiao Q Zheng Y et al. LLMs for doctors: leveraging medical LLMs to assist doctors, not replace them. arXiv. Preprint posted online on Jun 26, 2024. [doi: 10.48550\/arXiv.2406.18034]"},{"key":"R12","unstructured":"Johnson A Bulgarelli L Pollard T et al. MIMIC-IV (version 3.1). PhysioNet. 2024. URL: https:\/\/physionet.org\/content\/mimiciv\/3.1\/ [Accessed 22-07-2025]"},{"key":"R13","unstructured":"Aali A Veen DV Arefeen Y et al. MIMIC-IV-Ext-BHC: labeled clinical notes dataset for hospital course summarization (version 1.1.0). PhysioNet. 2024. URL: https:\/\/physionet.org\/content\/labelled-notes-hospital-course\/1.2.0\/ [Accessed 22-07-2025]"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Beltagy I Lo K Cohan A . SciBERT: a pretrained language model for scientific text. In: Inui K Jiang J Ng V Wan X , editors. Presented at: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); 2019:3615-3620; Hong Kong, China. [doi: 10.18653\/v1\/D19-1371]","DOI":"10.18653\/v1\/D19-1371"},{"key":"R15","unstructured":"scibert\u2011scivocab\u2011uncased model. PromptLayer. 2025. URL: https:\/\/www.promptlayer.com\/models\/scibertscivocabuncased [Accessed 10-11-2024]"},{"key":"R16","unstructured":"Unified Medical Language System (UMLS). National Library of Medicine. URL: https:\/\/www.nlm.nih.gov\/research\/umls\/index.html [Accessed 12-12-2024]"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1038\/s44387-025-00011-z","article-title":"Large language models for disease diagnosis: a scoping review","volume":"1","author":"Zhou","journal-title":"npj Artif Intell"},{"key":"R18","unstructured":"Nuthakki S Neela S Gichoya JW Purkayastha S . Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks. arXiv. Preprint posted online on Dec 28, 2019. [doi: 10.48550\/arXiv.1912.12397]"},{"key":"R19","unstructured":"Lovon-Melgarejo J Ben-Haddi T Scala J Moreno JG Tamine L . Revisiting the MIMIC-IV benchmark: experiments using language models for electronic health records. In: Demner-Fushman D Ananiadou S Thompson P Ondov B , editors. Presented at: Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC-COLING 2024; May 20, 2024:189-196; Torino, Italy."},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/j.smhl.2025.100564","article-title":"Metamorphic testing for robustness and fairness evaluation of LLM-based automated ICD coding applications","volume":"36","author":"Jaganathan","journal-title":"Smart Health"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-022-01899-x","article-title":"MIMIC-IV, a freely accessible electronic health record dataset","volume":"10","author":"Johnson","journal-title":"Sci Data"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/53724","article-title":"Evaluating the diagnostic performance of large language models on complex multimodal medical cases","volume":"26","author":"Chiu","journal-title":"J Med Internet Res"},{"key":"R23","unstructured":"Zero-shot prompting. DataCamp. URL: https:\/\/www.datacamp.com\/tutorial\/zero-shot-prompting [Accessed 20-12-2024]"},{"key":"R24","unstructured":"Zero-shot prompting: a benchmarking framework for AI prompts. Symbio6. URL: https:\/\/symbio6.nl\/en\/blog\/zero-shot-prompting-benchmarking [Accessed 20-12-2024]"},{"key":"R25","unstructured":"Does GPT-35 or GPT-4 API get minor updates? OpenAI Developer Community. 2024. URL: https:\/\/community.openai.com\/t\/does-gpt-3-5-or-gpt-4-api-get-minor-updates\/380885 [Accessed 25-05-2025]"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Xie Q Schenck EJ Yang HS Chen Y Peng Y Wang F . Faithful AI in medicine: a systematic review with large language models and beyond. medRxiv. Preprint posted online on Jul 1, 2023. [doi: 10.1101\/2023.04.18.23288752]","DOI":"10.21203\/rs.3.rs-3661764\/v1"},{"key":"R27","unstructured":"Li W Wu W Chen M Liu J Xiao X Wu H . Faithfulness in natural language generation: a systematic survey of analysis, evaluation, and optimization methods. arXiv. Preprint posted online on Mar 10, 2022. [doi: 10.48550\/arXiv.2203.05227]"},{"key":"R28","unstructured":"Cascella LM . Artificial intelligence risks: automation bias. MedPro Group. 2023. URL: https:\/\/www.medpro.com\/artificial-intelligence-risks-automationbias [Accessed 20-12-2024]"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Carberry J Xu H . A hierarchical fine-grained deep learning model for automated medical coding. Presented at: 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI); Apr 13-14, 2024:1-6; Mt Pleasant, MI, United States. [doi: 10.1109\/ICMI60790.2024.10585710]","DOI":"10.1109\/ICMI60790.2024.10585710"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Caralt MH Ng CBL Rei M . Continuous predictive modeling of clinical notes and ICD codes in patient health records. In: Demner-Fushman D Ananiadou S Miwa M Roberts K Tsujii J , editors. Presented at: Proceedings of the 23rd Workshop on Biomedical Natural Language Processing; Aug 16, 2024; Bangkok, Thailand. [doi: 10.18653\/v1\/2024.bionlp-1.19]","DOI":"10.18653\/v1\/2024.bionlp-1.19"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1197\/jamia.M1608","article-title":"Answering physicians\u2019 clinical questions: obstacles and potential solutions","volume":"12","author":"Ely","journal-title":"J Am Med Inform Assoc"},{"key":"R32","unstructured":"Tambwekar P Gombolay M . Towards reconciling usability and usefulness of explainable AI methodologies. arXiv. Preprint posted online on Jan 13, 2023. [doi: 10.48550\/arXiv.2301.05347]"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","article-title":"Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study","volume":"6","author":"Zack","journal-title":"Lancet Digit Health"},{"issue":"9","key":"R34","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1016\/j.jacr.2023.06.025","article-title":"\u201cShortcuts\u201d causing bias in radiology artificial intelligence: causes, evaluation, and mitigation","volume":"20","author":"Banerjee","journal-title":"J Am Coll Radiol"},{"issue":"6","key":"R35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","journal-title":"ACM Comput Surv"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1038\/s41591-018-0272-7","article-title":"Privacy in the age of medical big data","volume":"25","author":"Price","journal-title":"Nat Med"},{"key":"R37","unstructured":"pnaliyatthaliyazchayil. GitHub. URL: https:\/\/github.com\/pnaliyatthaliyazchayil\/evaluate_chatbot_llms_for_healthcare [Accessed 22-07-2025]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2025\/1\/e74142","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:05:11Z","timestamp":1782842711000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e74142"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,30]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/74142","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,30]]},"article-number":"v27i4e74142"}}