{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T03:51:06Z","timestamp":1792727466820,"version":"4.5.0"},"reference-count":25,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) are increasingly being explored for clinical decision support. However, whether inference-only LLM outputs can be interpreted as reliable quantitative risk estimates in structured clinical prediction remains unclear.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to evaluate the predictive performance and decision-making behavior of inference-only LLMs in a structured clinical prediction task and compare their outputs with those of an outcome-trained machine learning model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted a controlled benchmarking study using identical structured clinical inputs from patients admitted to intensive care units with spontaneous intracerebral hemorrhage. An outcome-trained extreme gradient boosting model was compared with predictions generated by a general-purpose LLM using four prompting strategies: zero-shot, few-shot, chain-of-thought, and combined few-shot plus chain-of-thought prompting. Performance was evaluated using discrimination metrics, threshold-dependent classification behavior, and concordance between Shapley Additive Explanations\u2013derived feature importance rankings and LLM-derived feature prioritization. The independent testing cohort included 435 patients, of whom 86 (19.7%) experienced in-hospital mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The outcome-trained machine learning model demonstrated superior discriminative performance compared with all LLM-based approaches. LLM predictions achieved moderate discrimination but exhibited substantial variability in threshold-dependent classification behavior across prompting strategies. At a fixed probability threshold of 0.5, LLM approaches consistently demonstrated high sensitivity and lower specificity, whereas operating characteristics varied considerably when thresholds were optimized using the Youden index. The optimal thresholds for LLM-based approaches ranged from 0.74 to 0.88, compared with 0.1555 for the extreme gradient boosting model. Concordance between Shapley Additive Explanations\u2013derived attribution and LLM-derived feature prioritization was modest, suggesting only partial alignment between empirically learned predictor structure and language-based reasoning patterns.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In this structured clinical prediction setting, inference-only LLM outputs demonstrated prompt-sensitive decision behavior despite moderate discriminative performance. These findings suggest that LLM-generated probability outputs should be interpreted cautiously when used for quantitative clinical risk estimation. A complementary framework integrating outcome-trained predictive models with LLM-assisted reasoning may provide a more reliable direction for future clinical decision support systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/29701","type":"journal-article","created":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T08:54:31Z","timestamp":1783587271000},"page":"e29701-e29701","source":"Crossref","is-referenced-by-count":1,"title":["Prompt-Sensitive Decision Behavior of Large Language Models in Intensive Care Unit Mortality Prediction for Spontaneous Intracerebral Hemorrhage: Comparative Benchmarking Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7820-8050","authenticated-orcid":false,"given":"Jinn-Rung","family":"Kuo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4155-9250","authenticated-orcid":false,"given":"Guan-Yu","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2476-4427","authenticated-orcid":false,"given":"Xiao-Han Vivian","family":"Yap","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5345-1373","authenticated-orcid":false,"given":"Chao-Chien","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7701-4260","authenticated-orcid":false,"given":"Yung-De","family":"Kuo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6698-0273","authenticated-orcid":false,"given":"Chung-Feng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,8]]},"reference":[{"issue":"6","key":"R1","doi-asserted-by":"publisher","first-page":"660","DOI":"10.1136\/jnnp-2013-306476","article-title":"Long-term prognosis after intracerebral haemorrhage: systematic review and meta-analysis","volume":"85","author":"Poon","journal-title":"J Neurol Neurosurg Psychiatry"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1055\/s-0038-1649503","article-title":"Global burden of stroke","volume":"38","author":"Katan","journal-title":"Semin Neurol"},{"issue":"7","key":"R3","doi-asserted-by":"publisher","first-page":"e282","DOI":"10.1161\/STR.0000000000000407","article-title":"2022 guideline for the management of patients with spontaneous intracerebral hemorrhage: a guideline from the American Heart Association\/American Stroke Association","volume":"53","author":"Greenberg","journal-title":"Stroke"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/s42466-021-00120-5","article-title":"Prognostication after intracerebral hemorrhage: a review","volume":"3","author":"Witsch","journal-title":"Neurol Res Pract"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-018-0613-8","article-title":"Prognostic models for intracerebral hemorrhage: systematic review and meta-analysis","volume":"18","author":"Greg\u00f3rio","journal-title":"BMC Med Res Methodol"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Chen T Guestrin C . XGBoost: a scalable tree boosting system. Presented at: KDD \u201916: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Aug 13-17, 2016; San Francisco, California, USA. [doi: 10.1145\/2939672.2939785]","DOI":"10.1145\/2939672.2939785"},{"issue":"14","key":"R7","doi-asserted-by":"publisher","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"Rajkomar","journal-title":"N Engl J Med"},{"issue":"5","key":"R8","doi-asserted-by":"publisher","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","article-title":"Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis","volume":"22","author":"Shickel","journal-title":"IEEE J Biomed Health Inform"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-026-03377-5","article-title":"Correction to: developing a high-performance AI model for spontaneous intracerebral hemorrhage mortality prediction using machine learning in ICU settings","volume":"26","author":"Yap","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","journal-title":"Nat Mach Intell"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster","journal-title":"BMC Med"},{"issue":"7972","key":"R12","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"7956","key":"R13","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","journal-title":"Nature"},{"issue":"8","key":"R14","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","journal-title":"Nat Med"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-074820","article-title":"Evaluation of clinical prediction models (part 2): how to undertake an external validation study","volume":"384","author":"Riley","journal-title":"BMJ"},{"issue":"11","key":"R17","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","journal-title":"Lancet Digit Health"},{"key":"R18","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/arXiv.2303.13375]"},{"key":"R19","unstructured":"Lundberg SM Lee SI . A unified approach to interpreting model predictions. Presented at: 31st International Conference on Neural Information Processing Systems; Dec 4-9, 2017; Long Beach, CA, USA. URL: https:\/\/dl.acm.org\/doi\/10.5555\/3295222.3295230 [Accessed 16-06-2026]"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","journal-title":"Nat Mach Intell"},{"issue":"10","key":"R21","doi-asserted-by":"publisher","first-page":"735","DOI":"10.7326\/L15-5093-2","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD)","volume":"162","author":"Collins","journal-title":"Ann Intern Med"},{"key":"R22","unstructured":"Wei J Wang X Schuurmans D et al. Chain-of-thought prompting elicits reasoning in large language models. Presented at: NIPS\u201922: 36th International Conference on Neural Information Processing Systems; Nov 28 to Dec 9, 2022; New Orleans LA, USA. URL: https:\/\/dl.acm.org\/doi\/10.5555\/3600270.3602070 [Accessed 16-06-2026]"},{"key":"R23","unstructured":"Kojima T Gu SS Reid M Matsuo Y Iwasawa Y . Large language models are zero-shot reasoners. Presented at: NIPS\u201922: 36th International Conference on Neural Information Processing Systems; Nov 28 to Dec 9, 2022; New Orleans LA, USA. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.5555\/3600270 [Accessed 16-06-2026]"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3","article-title":"Index for rating diagnostic tests","volume":"3","author":"Youden","journal-title":"Cancer"},{"issue":"3","key":"R25","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","volume":"44","author":"DeLong","journal-title":"Biometrics"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e29701","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:32:49Z","timestamp":1785537169000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e29701"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,8]]},"references-count":25,"URL":"https:\/\/doi.org\/10.2196\/29701","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,8]]},"article-number":"v28i6e29701"}}