{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T12:30:29Z","timestamp":1783600229835,"version":"3.55.0"},"reference-count":25,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T00:00:00Z","timestamp":1783555200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T00:00:00Z","timestamp":1783555200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["DEN Open"],"published-print":{"date-parts":[[2027,4]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate large language models (LLMs) for pre\u2010endoscopy (PE) risk stratification and prediction of endoscopic findings in upper gastrointestinal bleeding (UGIB), and compare their performance with clinical risk scores, conventional machine learning (ML) models, and hybrid approaches.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This multicenter retrospective study included 384 patients with UGIB who underwent endoscopy. Five LLMs (GPT\u20105, Gemini\u20102.5\u2010Flash, Llama 4, Grok, and DeepSeek R1) were tested using structured zero\u2010shot prompts based on PE clinical and laboratory data. Their performance in identifying high\u2010risk patients was compared with the Glasgow\u2010Blatchford Score (GBS), AIMS65, PE Rockall score, and ML models. Hybrid LLM\u2013score models were also assessed. Two gastroenterologists evaluated the quality of LLM\u2010generated justifications.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>GBS showed the best discriminative performance among clinical scores (area under the receiver operating characteristic curve [AUROC] 0.73). Among LLMs, GPT\u20105 achieved the highest accuracy (0.66), while Grok showed the best\u2010balanced performance (0.59; F1 0.47). Gemini\u20102.5\u2010Flash had the highest sensitivity (0.89) but low specificity (0.21). Endoscopic prediction performance was modest, with Gemini\u20102.5\u2010Flash achieving the highest exact\u2010match accuracy (0.34) and micro\u2010F1 (0.38). Hybrid models improved performance over standalone LLMs but did not outperform GBS alone (best: GBS+GPT\u20105, AUROC 0.670. LLMs showed higher numerical performance than conventional ML models, but a statistical comparison was not possible due to unavailable instance\u2010level data. Grok received the highest human evaluation score for explanation quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLMs showed moderate performance in UGIB risk stratification and endoscopic prediction but were inferior to clinical scores, especially GBS. Hybrid models modestly improved over standalone LLMs but not GBS, supporting their use as adjunct tools rather than clinical decision\u2010support systems.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Trial Registration<\/jats:title>\n                    <jats:p>N\/A<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/deo2.70373","type":"journal-article","created":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T11:57:04Z","timestamp":1783598224000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation of Large Language Models in the Clinical Management of Patients With Upper Gastrointestinal Bleeding: Insights From Real\u2010World Patient Data"],"prefix":"10.1002","volume":"7","author":[{"given":"Mohsen","family":"Rajabnia","sequence":"first","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farbod","family":"Davoodi","sequence":"additional","affiliation":[{"name":"Department of Computer Science Missouri University of Science and Technology  Rolla Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahsa","family":"Hajisafarali","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shadi","family":"Asadinia","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahsa","family":"Shiri","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fatemeh","family":"Shirzad","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peyman","family":"Saeedi","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahsa","family":"Mohammadi","sequence":"additional","affiliation":[{"name":"Department of Medicine Alborz University of Medical Sciences  Karaj Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,7,9]]},"reference":[{"issue":"1","key":"e_1_2_13_2_1","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1186\/s12876-022-02413-8","article-title":"Prediction of the Need for Emergency Endoscopic Treatment for Upper Gastrointestinal Bleeding and New Score Model: A Retrospective Study","volume":"22","author":"Sasaki Y.","year":"2022","journal-title":"BMC Gastroenterology"},{"issue":"1","key":"e_1_2_13_3_1","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1053\/j.gastro.2019.09.009","article-title":"Validation of a Machine Learning Model That Outperforms Clinical Risk Scoring Systems for Upper Gastrointestinal Bleeding","volume":"158","author":"Shung D. L.","year":"2020","journal-title":"Gastroenterology"},{"issue":"10","key":"e_1_2_13_4_1","doi-asserted-by":"crossref","first-page":"1657","DOI":"10.14309\/ajg.0000000000000632","article-title":"Explainable Machine Learning Model for Predicting GI Bleed Mortality in the Intensive Care Unit","volume":"115","author":"Deshmukh F.","year":"2020","journal-title":"Official journal of the American College of Gastroenterology | ACG"},{"issue":"1","key":"e_1_2_13_5_1","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1038\/s41746-025-01486-5","article-title":"Multiple Large Language Models versus Experienced Physicians in Diagnosing Challenging Cases With Gastrointestinal Symptoms","volume":"8","author":"Yang X.","year":"2025","journal-title":"NPJ Digital Medicine"},{"issue":"11","key":"e_1_2_13_6_1","doi-asserted-by":"crossref","DOI":"10.14309\/ctg.0000000000000765","article-title":"Digesting Digital Health: A Study of Appropriateness and Readability of ChatGPT\u2010Generated Gastroenterological Information","volume":"15","author":"Toiv A.","year":"2024","journal-title":"Clinical and Translational Gastroenterology"},{"issue":"10","key":"e_1_2_13_7_1","article-title":"Reliability and Usefulness of ChatGPT for Inflammatory Bowel Diseases: An Analysis for Patients and Healthcare Professionals","volume":"15","author":"Cankurtaran R. E.","year":"2023","journal-title":"Cureus"},{"issue":"1","key":"e_1_2_13_8_1","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1053\/j.gastro.2024.09.014","article-title":"Detection of Gastrointestinal Bleeding with Large Language Models to Aid Quality Improvement and Appropriate Reimbursement","volume":"168","author":"Zheng N. S.","year":"2025","journal-title":"Gastroenterology"},{"issue":"1","key":"e_1_2_13_9_1","doi-asserted-by":"crossref","DOI":"10.1136\/bmjgast-2025-001896","article-title":"Large Language Models for Extracting Histopathologic Diagnoses of Colorectal Cancer and Dysplasia From Electronic Health Records","volume":"12","author":"Johnson B.","year":"2025","journal-title":"BMJ Open Gastroenterology"},{"issue":"4","key":"e_1_2_13_10_1","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1016\/j.igie.2023.09.003","article-title":"ChatGPT for Advice on Common GI Endoscopic Procedures: The Promise and the Peril","volume":"2","author":"Lim D. Y. Z.","year":"2023","journal-title":"iGIE"},{"key":"e_1_2_13_11_1","doi-asserted-by":"crossref","DOI":"10.1016\/j.jbi.2025.104885","article-title":"GutGPT: A Multidimensional Knowledge\u2010Enhanced Large Language Model for Gastrointestinal Medicine","volume":"169","author":"Zhang R.\u2010Y.","year":"2025","journal-title":"Journal of Biomedical Informatics"},{"key":"e_1_2_13_12_1","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2024.emnlp-main.64","volume-title":"A Survey on In\u2010Context Learning","author":"Dong Q.","year":"2024"},{"key":"e_1_2_13_13_1","doi-asserted-by":"crossref","first-page":"24824","DOI":"10.52202\/068431-1800","article-title":"Chain\u2010of\u2010Thought Prompting Elicits Reasoning in Large Language Models","volume":"35","author":"Wei J.","year":"2022","journal-title":"Advances in neural information processing systems"},{"key":"e_1_2_13_14_1","unstructured":"S.Yao D.Yu J.Zhao et\u00a0al. \u201cTree of thoughts: Deliberate Problem Solving With Large Language Models \u201d3(2023):1 https:\/\/arxivorg\/abs\/230510601."},{"key":"e_1_2_13_15_1","unstructured":"A.Sellergren S.Kazemzadeh T.Jaroensri et\u00a0al. \u201cMedgemma Technical Report \u201d arXiv preprint arXiv:250705201 (2025)."},{"key":"e_1_2_13_16_1","unstructured":"T.Han L. C.Adams J.\u2010M.Papaioannou et\u00a0al. \u201cMedAlpaca\u2013an Open\u2010source Collection of Medical Conversational AI Models and Training Data \u201d arXiv preprint arXiv:230408247 (2023)."},{"key":"e_1_2_13_17_1","unstructured":"Z.Chen A. H.Cano A.Romanou et\u00a0al. \u201cMeditron\u201070b: Scaling Medical Pretraining for Large Language Models \u201d arXiv preprint arXiv:231116079 (2023)."},{"issue":"1","key":"e_1_2_13_18_1","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1038\/s41746-021-00455-y","article-title":"Med\u2010BERT: Pretrained Contextualized Embeddings on Large\u2010Scale Structured Electronic Health Records for Disease Prediction","volume":"4","author":"Rasmy L.","year":"2021","journal-title":"NPJ Digital Medicine"},{"key":"e_1_2_13_19_1","unstructured":"K.Huang J.Altosaar andR.Ranganath \u201cClinicalbert: Modeling Clinical Notes and Predicting Hospital Readmission \u201d arXiv preprint arXiv:190405342 (2019)."},{"issue":"4","key":"e_1_2_13_20_1","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A Pre\u2010Trained Biomedical Language Representation Model for Biomedical Text Mining","volume":"36","author":"Lee J.","year":"2020","journal-title":"Bioinformatics"},{"key":"e_1_2_13_21_1","doi-asserted-by":"crossref","unstructured":"S.Wang M.Hu Q.Li M.Safari andX.Yang \u201cCapabilities of GPT\u20105 on Multimodal Medical Reasoning \u201d arXiv preprint arXiv:250808224 (2025).","DOI":"10.1117\/12.3086794"},{"key":"e_1_2_13_22_1","unstructured":"G.Team R.Anil S.Borgeaud et\u00a0al. \u201cGemini: A family of Highly Capable Multimodal Models \u201d arXiv preprint arXiv:231211805 (2023)."},{"key":"e_1_2_13_23_1","unstructured":"H.Touvron T.Lavril G.Izacard et\u00a0al. \u201cLlama: Open and Efficient Foundation Language Models \u201d arXiv preprint arXiv:230213971 (2023)."},{"key":"e_1_2_13_24_1","unstructured":"A.Liu B.Feng B.Xue et\u00a0al. \u201cDeepseek\u2010v3 Technical Report \u201d arXiv preprint arXiv:241219437 (2024)."},{"key":"e_1_2_13_25_1","unstructured":"D.Guo D.Yang H.Zhang et\u00a0al. \u201cDeepseek\u2010r1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning \u201d arXiv preprint arXiv:250112948 (2025)."},{"key":"e_1_2_13_26_1","unstructured":"A.Lv R.Xie X.Sun Z.Kang andR.Yan \u201cLanguage Models \u201cGrok\u201d to Copy \u201d arXiv preprint arXiv:240909281 (2024)."}],"container-title":["DEN Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/deo2.70373","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/deo2.70373","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/deo2.70373","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T11:57:11Z","timestamp":1783598231000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/deo2.70373"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,9]]},"references-count":25,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2027,4]]}},"alternative-id":["10.1002\/deo2.70373"],"URL":"https:\/\/doi.org\/10.1002\/deo2.70373","archive":["Portico"],"relation":{},"ISSN":["2692-4609","2692-4609"],"issn-type":[{"value":"2692-4609","type":"print"},{"value":"2692-4609","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,9]]},"assertion":[{"value":"2026-02-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70373"}}