{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T21:03:14Z","timestamp":1789678994620,"version":"4.0.1"},"reference-count":25,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Hepatopancreatobiliary (HPB) malignancies require complex treatment planning that often relies on multidisciplinary team (MDT) discussions. Large language models (LLMs) have recently been explored for clinical decision support, but their performance within real-world multidisciplinary decision environments remains unclear. In particular, the stability of LLM-generated recommendations\u2014that is, whether a model produces the same answer when given the same clinical input\u2014has rarely been examined.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the stability of treatment recommendations generated by contemporary LLMs when identical HPB cases are queried repeatedly, and their concordance with the treatment decisions reached at an institutional MDT conference.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This retrospective study included consecutive cases discussed at a single-center HPB MDT conference between September 1, 2024, and August 31, 2025. Standardized clinical case summaries derived from preconference documentation were provided to 4 LLMs (GPT-4o, GPT-5.2, Gemini 3 Pro, and Claude Sonnet 4.5) through their consumer web interfaces. Each model recommended a treatment among predefined MDT treatment options, and identical queries were repeated 4 times in separate sessions. Stability was quantified as the discordance rate relative to the initial response and, without privileging any single query, as the mean pairwise agreement and Fleiss \u03ba across the 4 iterations. Concordance with MDT decisions was assessed using both the initial and modal responses, together with Cohen \u03ba and class-wise\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n A total of 107 MDT cases were analyzed. Stability differed significantly across models (\n P<\/jats:italic>\n =.01). Gemini 3 Pro showed the lowest discordance rate (mean 12.8%, SD 2.3%) and the highest reference-free agreement (Fleiss \u03ba=0.737), whereas GPT-4o showed the highest discordance rate (mean 30.2%, SD 6.5%) and the lowest agreement (Fleiss \u03ba=0.430). Concordance with MDT decisions ranged from 48.6% to 72.9% using the initial response and from 66.3% to 74.5% using the modal response, and the highest-performing model differed between the 2 definitions. Class-wise\n F<\/jats:italic>\n 1<\/jats:sub>\n -score was consistently lower for surgery (0.400\u20100.520) than for chemotherapy (0.621\u20100.836). Complete discordance occurred in 17 of 107 (15.9%) cases and in none of the 31 anatomically unresectable cases (Fisher exact test,\n P<\/jats:italic>\n =.003). Recurrent or on-treatment disease (adjusted odds ratio [OR] 5.40, 95% CI 1.65-17.68;\n P<\/jats:italic>\n =.005), pancreatic tumor location (adjusted OR 7.37, 95% CI 2.03-26.78;\n P<\/jats:italic>\n =.002), and low MDT agreement level (adjusted OR 10.33, 95% CI 1.54-69.38;\n P<\/jats:italic>\n =.016) were independently associated with complete discordance.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLM-generated treatment recommendations demonstrated moderate alignment with MDT decisions in HPB oncology. Importantly, response stability varied substantially across models, indicating that concordance alone is insufficient for evaluating LLMs as clinical decision support tools. These findings suggest that LLMs may serve as a reasoning-support layer in MDT-like decision environments, but their response stability must be systematically characterized before clinical integration.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/101137","type":"journal-article","created":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T20:13:43Z","timestamp":1789676023000},"page":"e101137-e101137","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4638-652X","authenticated-orcid":false,"given":"Sung Jun","family":"Jo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8998-7811","authenticated-orcid":false,"given":"Eui Hyuk","family":"Chong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4236-5094","authenticated-orcid":false,"given":"Incheon","family":"Kang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6930-5978","authenticated-orcid":false,"given":"Seok Jeong","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5177-8937","authenticated-orcid":false,"given":"Beodeul","family":"Kang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jung Sun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6979-5812","authenticated-orcid":false,"given":"Hong Jae","family":"Chon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang-il","family":"Kwon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min Je","family":"Sung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suk Pyo","family":"Shin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0484-6658","authenticated-orcid":false,"given":"Chansik","family":"An","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9325-2300","authenticated-orcid":false,"given":"Ho Yeong","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeong-Sik","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sujin","family":"Jang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3217-6444","authenticated-orcid":false,"given":"Jung Ho","family":"Im","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kwang Hyun","family":"Ko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3365-0096","authenticated-orcid":false,"given":"Sung Hwan","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1077605","article-title":"Multidisciplinary team diagnosis and treatment of pancreatic cancer: current landscape and future prospects","volume":"13","author":"Yao","journal-title":"Front Oncol"},{"issue":"8","key":"R2","doi-asserted-by":"publisher","first-page":"2081","DOI":"10.1245\/s10434-008-9929-7","article-title":"Evaluating the impact of a single-day multidisciplinary clinic on the management of pancreatic cancer","volume":"15","author":"Pawlik","journal-title":"Ann Surg Oncol"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.3389\/fsurg.2023.1119557","article-title":"The role of the multidisciplinary tumor board (MDTB) in the assessment of pancreatic cancer diagnosis and resectability: a tertiary referral center experience","volume":"10","author":"Quero","journal-title":"Front Surg"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.esmoop.2021.100377","article-title":"Setup of multidisciplinary team discussions for patients with cholangiocarcinoma: current practice and recommendations from the European Network for the Study of Cholangiocarcinoma (ENS-CCA)","volume":"7","author":"Casadio","journal-title":"ESMO Open"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"1239","DOI":"10.1016\/j.athoracsur.2011.05.057","article-title":"The effect of a multidisciplinary thoracic malignancy conference on the treatment of patients with esophageal cancer","volume":"92","author":"Freeman","journal-title":"Ann Thorac Surg"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.e2718","article-title":"Effects of multidisciplinary team working on breast cancer survival: retrospective, comparative, interventional cohort study of 13 722 women","volume":"344","author":"Kesson","journal-title":"BMJ"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.ctrv.2015.11.007","article-title":"The impact of multidisciplinary team meetings on patient assessment, management and outcomes in oncology settings: a systematic review of the literature","volume":"42","author":"Pillay","journal-title":"Cancer Treat Rev"},{"issue":"2","key":"R8","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1016\/j.hpb.2016.11.002","article-title":"Utility of a multidisciplinary tumor board in the management of pancreatic and upper gastrointestinal diseases: an observational study","volume":"19","author":"Brauer","journal-title":"HPB (Oxford)"},{"issue":"7956","key":"R9","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","journal-title":"Nature"},{"issue":"11","key":"R10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.43689","article-title":"Leveraging large language models for decision support in personalized oncology","volume":"6","author":"Benary","journal-title":"JAMA Netw Open"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.surg.2025.109267","article-title":"Use of large language models as clinical decision support tools for management of pancreatic adenocarcinoma using National Comprehensive Cancer Network guidelines","volume":"182","author":"Kaiser","journal-title":"Surgery"},{"issue":"8","key":"R12","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1200\/CCI-24-00230","article-title":"Large language models as decision-making tools in oncology: comparing artificial intelligence suggestions and expert recommendations","volume":"9","author":"Ah-Thiane","journal-title":"JCO Clin Cancer Inform"},{"issue":"9","key":"R14","doi-asserted-by":"publisher","first-page":"4984","DOI":"10.3390\/curroncol31090369","article-title":"The role of artificial intelligence on tumor boards: perspectives from surgeons, medical oncologists and radiation oncologists","volume":"31","author":"Nardone","journal-title":"Curr Oncol"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/65984","article-title":"Large language model applications for health information extraction in oncology: scoping review","volume":"11","author":"Chen","journal-title":"JMIR Cancer"},{"issue":"8","key":"R16","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","journal-title":"JAMA Intern Med"},{"issue":"8","key":"R17","doi-asserted-by":"publisher","first-page":"1040","DOI":"10.1001\/jamainternmed.2021.3333","article-title":"The Epic Sepsis Model falls short\u2014the importance of external validation","volume":"181","author":"Habib","journal-title":"JAMA Intern Med"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1007\/s00404-024-07565-4","article-title":"Evolution of publicly available large language models for complex decision-making in breast cancer care","volume":"310","author":"Griewing","journal-title":"Arch Gynecol Obstet"},{"issue":"13","key":"R19","doi-asserted-by":"publisher","DOI":"10.3390\/cancers18132175","article-title":"Large language models for multidisciplinary tumor board decision-making in primary liver tumors: a retrospective single-center study","volume":"18","author":"Palzer","journal-title":"Cancers (Basel)"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/j.esmoop.2020.100010","article-title":"The impact of the multidisciplinary tumor board (MDTB) on the management of pancreatic diseases in a tertiary referral center","volume":"6","author":"Quero","journal-title":"ESMO Open"},{"issue":"6245","key":"R22","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1126\/science.aac4520","article-title":"Policy forum. Data, privacy, and the greater good","volume":"349","author":"Horvitz","journal-title":"Science"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.52202\/075280-3275","article-title":"Language models don\u2019t always say what they think: unfaithful explanations in chain-of-thought prompting","author":"Turpin","journal-title":"Adv Neural Inf Process Syst"},{"issue":"6","key":"R24","doi-asserted-by":"publisher","first-page":"977","DOI":"10.1016\/j.surg.2014.02.001","article-title":"Borderline resectable pancreatic cancer: a consensus statement by the International Study Group of Pancreatic Surgery (ISGPS)","volume":"155","author":"Bockhorn","journal-title":"Surgery"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s12885-019-6148-5","article-title":"Consensus in determining the resectability of locally progressed pancreatic ductal adenocarcinoma - results of the CONKO-007 multicenter trial","volume":"19","author":"Wittel","journal-title":"BMC Cancer"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T20:13:44Z","timestamp":1789676024000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e101137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,17]]},"references-count":25,"URL":"https:\/\/doi.org\/10.2196\/101137","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,17]]},"article-number":"v28i8e101137"}}