{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T11:29:29Z","timestamp":1787138969972,"version":"build-2736575974"},"reference-count":30,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T00:00:00Z","timestamp":1787097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: Acute-on-chronic pancreatitis (ACP) is a fibro-inflammatory syndrome that stands at the border between acute pancreatitis (AP) and chronic pancreatitis (CP). It can be broadly defined as an acute exacerbation of CP. Large language models (LLMs) are artificial intelligence (AI) machine learning tools that can be employed in clinical decision systems (CDSs). Methods: This is an exploratory benchmark study. Six LLM chatbots were evaluated on a Likert scale by a panel of 13 experts in pancreatology based on the similarity of responses to a position statement (PS) in ACP. Results: ClaudeAI had the best average Likert score, while Alice had the lowest. There was a positive correlation between the length of the response and expert-perceived similarity. There was no correlation between seniority and Likert score. Conclusions: In this exploratory benchmark, some LLMs generated answers that were often judged as similar to the position statement, suggesting that they might be helpful as educational support tools and may assist in the refinement of research questions or in the identification of research gaps in this particular pathology.<\/jats:p>","DOI":"10.3390\/diagnostics16162639","type":"journal-article","created":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T11:04:12Z","timestamp":1787137452000},"page":"2639","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking Large Language Models in Acute-on-Chronic Pancreatitis: An Exploratory Study"],"prefix":"10.3390","volume":"16","author":[{"given":"Alina Florentina","family":"Pistri\u021bu","sequence":"first","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Internal Medicine and Gastroenterology Department, University Emergency Hospital of Bucharest, 050098 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7052-4294","authenticated-orcid":false,"given":"Mihai Radu","family":"Pahomeanu","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Internal Medicine and Gastroenterology Department, University Emergency Hospital of Bucharest, 050098 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8753-9924","authenticated-orcid":false,"given":"Andreea Irina","family":"Ghi\u021b\u0103","sequence":"additional","affiliation":[{"name":"Gastroenterology Department, Emergency Clinical Hospital of Bucharest, 014461 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-1899-7262","authenticated-orcid":false,"given":"Cosmin Mihail","family":"Andrian","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9288-5309","authenticated-orcid":false,"given":"Arina Ilinca","family":"Gheorghe","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6274-2771","authenticated-orcid":false,"given":"Elena","family":"Alupei","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3950-7610","authenticated-orcid":false,"given":"Dana","family":"Bilous","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Gastroenterology Department, \u201cProf. Dr. Agrippa Ionescu\u201d Hospital, 011356 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2984-4727","authenticated-orcid":false,"given":"Deniz","family":"Gunsahin","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Gastroenterology Department, Emergency Clinical Hospital of Bucharest, 014461 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"C\u0103t\u0103lina","family":"Vl\u0103du\u021b","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Gastroenterology Department, \u201cProf. Dr. Agrippa Ionescu\u201d Hospital, 011356 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Costache","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4624-0443","authenticated-orcid":false,"given":"Petru\u021ba Violeta","family":"Filip","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Internal Medicine and Gastroenterology Department, University Emergency Hospital of Bucharest, 050098 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristina","family":"Tocia","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cOvidius\u201d University of Constanta, 900470 Constanta, Romania"},{"name":"Gastroenterology Department, \u201cSf. Apostol Andrei\u201d Emergency County Hospital, 900591 Constanta, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6436-9961","authenticated-orcid":false,"given":"R\u0103zvan C\u0103t\u0103lin","family":"Popescu","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cOvidius\u201d University of Constanta, 900470 Constanta, Romania"},{"name":"Abdominal Surgery Department, \u201cSf. Apostol Andrei\u201d Emergency County Hospital, 900591 Constanta, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5348-0341","authenticated-orcid":false,"given":"Cristian George","family":"\u021aieranu","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Gastroenterology Department, Elias Emergency Clinical Hospital of Bucharest, 011461 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7268-3723","authenticated-orcid":false,"given":"Eugen","family":"Dumitru","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cOvidius\u201d University of Constanta, 900470 Constanta, Romania"},{"name":"Gastroenterology Department, \u201cSf. Apostol Andrei\u201d Emergency County Hospital, 900591 Constanta, Romania"},{"name":"Research and Development Center for Morphological and Genetic Studies in Malignant Pathology (CEDMOG), \u201cOvidius\u201d University of Constanta, 900470 Constanta, Romania"},{"name":"Academy of Romanian Scientists, 050044 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mihai","family":"Cioc\u00eerlan","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Gastroenterology Department, \u201cProf. Dr. Agrippa Ionescu\u201d Hospital, 011356 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Corina Silvia","family":"Pop","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Internal Medicine and Gastroenterology Department, University Emergency Hospital of Bucharest, 050098 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3042-0754","authenticated-orcid":false,"given":"Lucian","family":"Negreanu","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Internal Medicine and Gastroenterology Department, University Emergency Hospital of Bucharest, 050098 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"17060","DOI":"10.1038\/nrdp.2017.60","article-title":"Chronic Pancreatitis","volume":"3","author":"Kleeff","year":"2017","journal-title":"Nat. Rev. Dis. Primers"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.pan.2023.01.010","article-title":"Position Statement on the Definition, Incidence, Diagnosis and Outcome of Acute on Chronic Pancreatitis","volume":"23","author":"Bouwense","year":"2023","journal-title":"Pancreatology"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1080\/00365521.2022.2035811","article-title":"Confusion with the Definition and Diagnostic Criteria for Acute on Chronic Pancreatitis: Review and Recommendations","volume":"57","author":"Pedersen","year":"2022","journal-title":"Scand. J. Gastroenterol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liu, J., Wang, C., Chen, Z., Dai, Q., Bai, J., and Cui, Y. (2024). Analysis of Risk Factors Related to Acute Exacerbation in Patients with Chronic Pancreatitis: A Retrospective Study of 313 Cases from a Single Center in China. BMC Gastroenterol., 24.","DOI":"10.1186\/s12876-024-03528-w"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1111\/bjop.12764","article-title":"Artificial Intelligence Chatbots Mimic Human Collective Behaviour","volume":"117","author":"He","year":"2026","journal-title":"Br. J. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., and Biancone, P. (2021). The Role of Artificial Intelligence in Healthcare: A Structured Literature Review. BMC Med. Inform. Decis. Mak., 21.","DOI":"10.1186\/s12911-021-01488-9"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"100030","DOI":"10.1016\/j.medp.2024.100030","article-title":"Large Language Models Illuminate a Progressive Pathway to Artificial Intelligent Healthcare Assistant","volume":"1","author":"Yuan","year":"2024","journal-title":"Med. Plus"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Qiu, J., Luo, L., and Zhou, Y. (2024). Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe Acute Pancreatitis. BMC Gastroenterol., 24.","DOI":"10.1186\/s12876-024-03348-y"},{"key":"ref_9","first-page":"55","article-title":"A Technique for the Measurement of Attitudes","volume":"22","author":"Likert","year":"1932","journal-title":"Arch. Psychol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"16569","DOI":"10.1073\/pnas.0507655102","article-title":"An Index to Quantify an Individual\u2019s Scientific Research Output","volume":"102","author":"Hirsch","year":"2005","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.jclinepi.2010.03.002","article-title":"Guidelines for Reporting Reliability and Agreement Studies (GRRAS) Were Proposed","volume":"64","author":"Kottner","year":"2011","journal-title":"J. Clin. Epidemiol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"457","DOI":"10.20517\/ir.2024.27","article-title":"A Survey of Datasets in Medicine for Large Language Models","volume":"4","author":"Zhang","year":"2024","journal-title":"Intell. Robot."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"17562848241227031","DOI":"10.1177\/17562848241227031","article-title":"Large Language Models: A Primer and Gastroenterology Applications","volume":"17","author":"Shahab","year":"2024","journal-title":"Ther. Adv. Gastroenterol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"29","DOI":"10.21608\/erji.2023.306718","article-title":"Knowledge and Attitude of Ain Shams University Medical Students towards Artificial Intelligence and Its Application in Medical Education and Practice","volume":"3","author":"Khater","year":"2023","journal-title":"Educ. Res. Innov. J."},{"key":"ref_15","first-page":"20552076221089099","article-title":"Artificial Intelligence and Medical Education: A Global Mixed-Methods Study of Medical Students\u2019 Perspectives","volume":"8","author":"Ejaz","year":"2022","journal-title":"Digit. Health"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1111\/ans.18666","article-title":"Investigating the Impact of Innovative AI Chatbot on Post-Pandemic Medical Education and Clinical Assistance: A Comprehensive Analysis","volume":"94","author":"Xie","year":"2024","journal-title":"ANZ J. Surg."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1080\/08963568.2024.2420300","article-title":"A \u201cDelve\u201d into the Evidence of AI in Production of Academic Business Literature","volume":"30","author":"Deike","year":"2025","journal-title":"J. Bus. Financ. Librariansh."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.ejim.2024.07.012","article-title":"Artificial Intelligence in Scientific Medical Writing: Legitimate and Deceptive Uses and Ethical Concerns","volume":"127","author":"Ramoni","year":"2024","journal-title":"Eur. J. Intern. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1001\/archinte.163.12.1409","article-title":"Effects of Computerized Physician Order Entry and Clinical Decision Support Systems on Medication Safety: A Systematic Review","volume":"163","author":"Kaushal","year":"2003","journal-title":"Arch. Intern. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","article-title":"Artificial Intelligence, Bias and Clinical Safety","volume":"28","author":"Challen","year":"2019","journal-title":"BMJ Qual. Saf."},{"key":"ref_21","first-page":"e44658","article-title":"Reimagining Healthcare: Unleashing the Power of Artificial Intelligence in Medicine","volume":"15","author":"Iqbal","year":"2023","journal-title":"Cureus"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1136\/flgastro-2021-101994","article-title":"Survey on the Perceptions of UK Gastroenterologists and Endoscopists to Artificial Intelligence","volume":"13","author":"Kader","year":"2022","journal-title":"Frontline Gastroenterol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1016\/j.gie.2020.10.029","article-title":"Assessing Perspectives on Artificial Intelligence Applications to Gastroenterology","volume":"93","author":"Kochhar","year":"2021","journal-title":"Gastrointest. Endosc."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Stan-Ilie, M., Sandru, V., Constantinescu, G., Plotogea, O.-M., Rinja, E.M., Tincu, I.F., Jichitu, A., Carasel, A.E., Butuc, A.C., and Popa, B. (2023). Artificial Intelligence\u2014The Rising Star in the Field of Gastroenterology and Hepatology. Diagnostics, 13.","DOI":"10.3390\/diagnostics13040662"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yang, M., Ye, Q., Zhang, J., Huang, F., and Zhang, S. (2024). Hallucination Augmented Contrastive Learning for Multimodal Large Language Model. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Soc.","DOI":"10.1109\/CVPR52733.2024.02553"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1002\/ueg2.12723","article-title":"Artificial Intelligence in Pancreatic Imaging: A Systematic Review","volume":"13","author":"Gheorghe","year":"2025","journal-title":"United Eur. Gastroenterol. J."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1014","DOI":"10.3390\/gastroent15040070","article-title":"The Role of Artificial Intelligence in Endoscopic Ultrasound for Pancreatic Diseases","volume":"15","author":"Constantinescu","year":"2024","journal-title":"Gastroenterol. Insights"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1159\/000518322","article-title":"Guidelines in Gastroenterology: Careful Interpretation Is Essential","volume":"29","author":"Correia","year":"2021","journal-title":"GE-Port. J. Gastroenterol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1186\/s12967-024-05302-8","article-title":"Exploring the Performance of ChatGPT on Acute Pancreatitis-Related Questions","volume":"22","author":"Du","year":"2024","journal-title":"J. Transl. Med."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"\u00c7amur, E., Cesur, T., G\u00fcne\u015f, Y.C., G\u00f6khan, M.B., and \u00d6kten, R.S. (2026). Prospective quantitative analysis of hyperparameter and input optimization in GPT-5: Comparative contribution to radiologist performance in abdominal radiology. Diagn. Interv. Radiol., Online ahead of print.","DOI":"10.4274\/dir.2026.263762"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/16\/2639\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T11:19:42Z","timestamp":1787138382000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/16\/2639"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,19]]},"references-count":30,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["diagnostics16162639"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16162639","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,19]]}}}