{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T22:19:48Z","timestamp":1784326788754,"version":"3.55.0"},"reference-count":20,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T00:00:00Z","timestamp":1775088000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: While large language models (LLMs) have demonstrated proficiency in medical examinations, their comparative performance on European gastroenterology assessments remains underexplored, particularly regarding architectural differences between reasoning and non-reasoning models. This study benchmarks five state-of-the-art LLMs\u2014DeepSeek-R1, ChatGPT-o1, ChatGPT-4o, Gemini-1.5-Pro, and Llama-3.1-405B (All versions January 2025)\u2014using 203 board-style questions from validated ESEGH preparation materials. Methods: Questions from two commercial ESEGH preparation banks were administered five times per model using standardized prompts. Accuracy, consistency, and domain-specific performance across clinical, diagnostic, and therapeutic questions were analyzed. Four practicing gastroenterologists validated human performance under uniform conditions. Results: ChatGPT-o1 achieved the highest overall accuracy at 84.0% (95% CI: 81.8\u201386.3), followed closely by ChatGPT-4o (81.7%), DeepSeek-R1 (79.0%), and Llama-3.1-405B (77.2%), while Gemini-1.5-Pro significantly underperformed with 68.5% accuracy (difference vs. ChatGPT-o1: 15.5 percentage points, 95% CI: 11.9 to 19.1, p &lt; 0.01). Although all models exhibited high internal consistency \u226598.4% average agreement across repeated attempts, with 94.6\u201398.0% of questions answered identically in all five attempts), greater consistency did not necessarily correspond to higher accuracy. Domain-specific analysis revealed that diagnostic questions were answered most accurately, whereas clinical examination questions posed considerable challenges. Topic analysis demonstrated that questions on small intestine disorders were answered with the highest accuracy, in contrast to the lower performance observed in bariatric and pancreatic disorders. Notably, reasoning models, which employed explicit chain-of-thought strategies, outperformed non-reasoning counterparts (81.5% vs. 75.8%, difference: 5.7 percentage points, 95% CI: 3.4 to 8.0, p &lt; 0.001), particularly on therapy questions and complex bait-and-switch formats. Practicing gastroenterologists achieved substantially lower accuracy (mean: 50.9%, range: 37.9\u201369.0%) compared to all LLMs. All models exceeded the current ESEGH passing threshold of 61.5%, with the top four models surpassing this benchmark by 15.7\u201322.5 percentage points. Conclusions: This benchmarking study demonstrates that current LLMs, particularly those with reasoning architectures, achieve high accuracy on European gastroenterology board-style questions. However, significant performance gaps in specific domains highlight limitations that must be addressed before clinical application. These findings provide a baseline for evaluating LLM capabilities in European medical contexts.<\/jats:p>","DOI":"10.3390\/jcm15072692","type":"journal-article","created":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T11:51:03Z","timestamp":1775130663000},"page":"2692","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Comparative Performance of Large Language Models on European Gastroenterology Board-Style Questions: Analysis of Reasoning Versus Non-Reasoning Architectures"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7037-5233","authenticated-orcid":false,"given":"Cem","family":"Simsek","sequence":"first","affiliation":[{"name":"Division of Gastroenterology, Hacettepe University, 06230 Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Petr","family":"Vanek","sequence":"additional","affiliation":[{"name":"Faculty of Medicine and Dentistry, Palack\u00fd University Olomouc, 77515 Olomouc, Czech Republic"},{"name":"Department of Gastroenterology and Digestive Endoscopy, Masaryk Memorial Cancer Institute, 65653 Brno, Czech Republic"},{"name":"Hepatology and Nutrition, Division of Gastroenterology, University of Minnesota, Minneapolis, MN 55455, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hakan","family":"Aydinli","sequence":"additional","affiliation":[{"name":"Division of Gastroenterology, Hacettepe University, 06230 Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"Krivinka","sequence":"additional","affiliation":[{"name":"2nd Department of Internal Medicine\u2014Gastroenterology and Geriatrics, University Hospital Olomouc, Faculty of Medicine and Dentistry, Palack\u00fd University Olomouc, 77900 Olomouc, Czech Republic"},{"name":"Department of Surgery, University Hospital Brno, Faculty of Medicine, Masaryk University, 62500 Brno, Czech Republic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manuel","family":"Lehner","sequence":"additional","affiliation":[{"name":"Gastroenterology, Clarunis Universit\u00e4res Bauchzentrum, 4058 Basel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2910-8559","authenticated-orcid":false,"given":"Sara","family":"Schiavone","sequence":"additional","affiliation":[{"name":"IRCCS Humanitas Research Hospital, 20089 Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7167-1459","authenticated-orcid":false,"given":"Cesare","family":"Hassan","sequence":"additional","affiliation":[{"name":"IRCCS Humanitas Research Hospital, 20089 Milan, Italy"},{"name":"Department of Biomedical Sciences, Humanitas University, 20072 Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6355-0921","authenticated-orcid":false,"given":"Henriette H.","family":"Heinrich","sequence":"additional","affiliation":[{"name":"Gastroenterology, Clarunis Universit\u00e4res Bauchzentrum, 4058 Basel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kung, T., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepa\u00f1o, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., and Maningo, J. (2022). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit Health, 2.","DOI":"10.1101\/2022.12.19.22283643"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Liu, M., Okuhara, T., Chang, X., Shirabe, R., Nishiie, Y., Okada, H., and Kiuchi, T. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis (Preprint), JMIR Publications Inc.","DOI":"10.2196\/preprints.60807"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"105673","DOI":"10.1016\/j.ijmedinf.2024.105673","article-title":"Evaluating the Effectiveness of advanced large language models in medical Knowledge: A Comparative study using Japanese national medical examination","volume":"193","author":"Liu","year":"2025","journal-title":"Int. J. Med. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/17562848231218618","article-title":"Evaluating the role of ChatGPT in gastroenterology: A comprehensive systematic review of applications, benefits, and limitations","volume":"16","author":"Klang","year":"2023","journal-title":"Ther. Adv. Gastroenterol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2280","DOI":"10.14309\/ajg.0000000000002320","article-title":"Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test","volume":"118","author":"Suchman","year":"2023","journal-title":"Am. J. Gastroenterol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1304","DOI":"10.1016\/j.dld.2024.02.019","article-title":"Charting new AI education in gastroenterology: Cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam","volume":"56","author":"Gravina","year":"2024","journal-title":"Dig. Liver Dis."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1002\/ueg2.12480","article-title":"Young GI angle: European Specialty Examination in Gastroenterology and Hepatology exam: Tips and tricks","volume":"11","author":"Domislovic","year":"2023","journal-title":"United Eur. Gastroenterol. J."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ali, S., Shahab, O., Shabeeb, R.A., Ladak, F., Yang, J., Nadkarni, G., Echavarr\u00eda, J., Babar, S., Shaukat, A., and Soroush, A. (2023). General purpose large language models match human performance on gastroenterology board exam self-assessments. medRxiv.","DOI":"10.1101\/2023.09.21.23295918"},{"key":"ref_9","unstructured":"Safavi-Naini, S.A.A., Ali, S., Shahab, O., Shahhoseini, Z., Savage, T., Rafiee, S., Samaan, J., Shabeeb, R.A., Ladak, F., and Yang, J. (2024). Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Samaan, J.S., Margolis, S., Srinivasan, N., Srinivasan, A., Yeo, Y.H., Anand, R., Samaan, F.S., Mirocha, J., Safavi-Naini, S.A., and El Kurdi, B. (2024). Multimodal Large Language Model Passes Specialty Board Examination and Surpasses Human Test-Taker Scores: A Comparative Analysis Examining the Stepwise Impact of Model Prompting Strategies on Performance. medRxiv.","DOI":"10.1101\/2024.07.27.24310809"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med. Educ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1090","DOI":"10.1227\/neu.0000000000002551","article-title":"Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank","volume":"93","author":"Ali","year":"2023","journal-title":"Neurosurgery"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1308\/rcsann.2024.0023","article-title":"The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination","volume":"106","author":"Chan","year":"2024","journal-title":"Ann. R. Coll. Surg. Engl."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Angel, M., Patel, A., Xing, H., Balsz, D., Arbuckle, C., Bruyette, D., and Baldi, P. (2023, January 21\u201324). AI and Veterinary Medicine: Performance of Large Language Models on the North American Licensing Examination. Proceedings of the 2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/SNAMS60348.2023.10375414"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1200\/OP.2023.19.11_suppl.511","article-title":"Large language models encode medical oncology knowledge: Performance on the ASCO and ESMO examination questions","volume":"19","author":"Longwell","year":"2023","journal-title":"JCO Oncol. Pract."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e2346721","DOI":"10.1001\/jamanetworkopen.2023.46721","article-title":"Performance of Large Language Models on a Neurology Board\u2013Style Examination","volume":"6","author":"Schubert","year":"2023","journal-title":"JAMA Netw. Open"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Tarabanis, C., Zahid, S., Mamalis, M., Zhang, K., Kalampokis, E., and Jankelson, L. (2024). Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions. PLoS Digit Health, 3.","DOI":"10.1371\/journal.pdig.0000604"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"618","DOI":"10.1111\/pin.13382","article-title":"Exploring the pitfalls of large language models: Inconsistency and inaccuracy in answering pathology board examination-style questions","volume":"73","author":"Koga","year":"2023","journal-title":"Pathol. Int."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1104","DOI":"10.1002\/jso.27821","article-title":"Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer","volume":"130","author":"Kaiser","year":"2024","journal-title":"J. Surg. Oncol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1272\/jnms.JNMS.2024_91-205","article-title":"Performance of a large language model on Japanese emergency medicine board certification examinations","volume":"91","author":"Igarashi","year":"2024","journal-title":"J. Nippon Med. Sch."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/7\/2692\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T12:11:18Z","timestamp":1775131878000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/7\/2692"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,2]]},"references-count":20,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["jcm15072692"],"URL":"https:\/\/doi.org\/10.3390\/jcm15072692","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,2]]}}}