{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T16:13:28Z","timestamp":1788279208314,"version":"build-2803163510"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) have shown promising performance on medical examinations across specialties. However, comparative evaluations of current-generation LLMs across multiple European anesthesiology examinations, alongside structured assessment of hallucinations vs question-related confusion, remain lacking.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to compare the performance of 4 state-of-the-art LLMs on anesthesiology and intensive medicine examination questions and assess their hallucination rates.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This computational comparative study analyzed 437 multiple-choice questions (1748 queries) from 3 sources: nurse anesthetist school examinations (infirmier anesth\u00e9siste dipl\u00f4m\u00e9 d\u2019\u00c9tat [registered nurse anesthetist]; n=100, 22.9%), European Diploma in Anaesthesiology and Intensive Care (EDAIC; n=219, 50.1%), and EDAIC On-Line Assessment (n=118, 27.0%). Each question was submitted to 4 LLMs (Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5, and Grok 4) using standardized prompts via default web interface settings. Responses were evaluated through structured consensus review by 2 examiners for accuracy, hallucinations, and question-related confusion. Statistical analysis included Friedman and Wilcoxon signed-rank tests with Holm-Bonferroni correction, the Cochran\n Q<\/jats:italic>\n test, and generalized estimating equations.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Average success rates ranged from 86% (SD 18%) to 94% (SD 10%) across LLMs and examination types, exceeding the EDAIC part I passing threshold, representing substantial improvement over previously reported GPT-3.5 performance. For the EDAIC, overall intermodel differences were significant (Friedman\n \u03c7<\/jats:italic>\n 2<\/jats:sup>\n 3<\/jats:sub>\n =13.9;\n P<\/jats:italic>\n =.003;\n W<\/jats:italic>\n =0.02), with Gemini outperforming GPT-5 as the only pairwise difference. Hallucination rates ranged from 11% (11\/100) to 20.1% (44\/219) without significant intermodel differences. All models exceeded the EDAIC passing threshold.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Current-generation LLMs demonstrated consistently high performance across multiple European anesthesiology examinations but continue to produce clinically relevant hallucinations, supporting their role as supervised educational tools rather than autonomous learning resources. These findings underscore the need for structured integration frameworks and systematic verification when deploying LLMs as learning tools in medical education.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/95859","type":"journal-article","created":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T15:13:51Z","timestamp":1788275631000},"page":"e95859-e95859","source":"Crossref","is-referenced-by-count":0,"title":["Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0216-1380","authenticated-orcid":false,"given":"Stefan","family":"Andrei","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2791-8514","authenticated-orcid":false,"given":"Thibault","family":"Giet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1467-4596","authenticated-orcid":false,"given":"Alexis","family":"Belouard","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9151-3474","authenticated-orcid":false,"given":"Mihai","family":"Stefan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mihai","family":"Popescu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3858-9319","authenticated-orcid":false,"given":"S\u00e9bastien","family":"Tanaka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3422-5705","authenticated-orcid":false,"given":"Philippe","family":"Montravers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2648-1693","authenticated-orcid":false,"given":"Aur\u00e9lie","family":"Gouel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,1]]},"reference":[{"issue":"4","key":"R1","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1097\/EJA.0000000000001969","article-title":"Chat-based generative pretrained transformers sits the European Diploma in Anaesthesiology and Intensive Care part I examination: a computational study","volume":"41","author":"Andrei","journal-title":"Eur J Anaesthesiol"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/75452","article-title":"GPT-4o and OpenAI o1 performance on the 2024 Spanish competitive medical specialty access examination: cross-sectional quantitative evaluation study","volume":"12","author":"Benito","journal-title":"JMIR Med Educ"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-31010-4","article-title":"Benchmarking large language models on the United States Medical Licensing Examination for clinical reasoning and medical licensing scenarios","volume":"16","author":"Siam","journal-title":"Sci Rep"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"2145","DOI":"10.1007\/s40670-025-02426-4","article-title":"Factors associated with lower performance of artificial intelligence on answering undergraduate medical education multiple-choice questions","volume":"35","author":"Ferretti","journal-title":"Med Sci Educ"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","first-page":"282","DOI":"10.34197\/ats-scholar.2023-0036PS","article-title":"Large language models and medical education: preparing for a rapid transformation in how trainees will learn to be doctors","volume":"4","author":"Ravi","journal-title":"ATS Sch"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1622292","article-title":"Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior","volume":"8","author":"Anh-Hoang","journal-title":"Front Artif Intell"},{"key":"R8","unstructured":"Metheniti E Bhar S Asher N . Une revue sur les hallucinations des LLM. In: Bechet F Chifu AG Pinel-Sauvagnat K Favre B Maes E Nurbakova D , editors. Actes des 32\u00e8me Conf\u00e9rence sur le Traitement Automatique des Langues Naturelles (TALN). ATALA & ARIA; 2025:756-779. URL: https:\/\/aclanthology.org\/2025.jeptalnrecital-taln.46\/ [Accessed 13-08-2026]"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"12","DOI":"10.3390\/ai6010012","article-title":"Beyond text generation: assessing large language models\u2019 ability to reason logically and follow strict rules","volume":"6","author":"Han","journal-title":"AI"},{"issue":"2","key":"R10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3703155","article-title":"A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions","volume":"43","author":"Huang","journal-title":"ACM Trans Inf Syst"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-08084-9","article-title":"Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam","volume":"25","author":"Altermatt","journal-title":"BMC Med Educ"},{"issue":"7972","key":"R12","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"2","key":"R13","first-page":"137","volume":"20","author":"Ateleanu","journal-title":"Jurnalul Rom\u00e2n de Anestezie Terapie Intensiv\u0103"},{"issue":"8","key":"R14","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1016\/j.redare.2022.08.006","article-title":"On-Line Assessment (OLA) as a preparation for the European Diploma in Anaesthesiology and Intensive Care (EDAIC), a retrospective observational study on the results and the potential impact on the part-I examination","volume":"69","author":"Brogly","journal-title":"Rev Esp Anestesiol Reanim (Engl Ed)"},{"key":"R15","unstructured":"LOI n\u00b0 2012-300 du 5 mars 2012 relative aux recherches impliquant la personne humaine (1). R\u00e9publique Fran\u00e7aise. URL: https:\/\/www.legifrance.gouv.fr\/loda\/id\/JORFTEXT000025441587\/ [Accessed 18-08-2026]"},{"issue":"13","key":"R16","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","journal-title":"N Engl J Med"},{"issue":"11","key":"R17","doi-asserted-by":"publisher","first-page":"1276","DOI":"10.1111\/medu.15402","article-title":"A systematic review of large language models and their implications in medical education","volume":"58","author":"Lucas","journal-title":"Med Educ"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/56859","article-title":"Performance of ChatGPT in the in-training examination for anesthesiology and pain medicine residents in South Korea: observational study","volume":"10","author":"Yoon","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1080\/10872981.2025.2542807","article-title":"Advancements in artificial intelligence transforming medical education: a comprehensive overview","volume":"30","author":"Khakpaki","journal-title":"Med Educ Online"},{"issue":"6","key":"R21","doi-asserted-by":"publisher","first-page":"1253","DOI":"10.1213\/ANE.0000000000006875","article-title":"An exploratory analysis of ChatGPT compared to human performance with the anesthesiology oral board examination: initial insights and implications","volume":"140","author":"Blacker","journal-title":"Anesth Analg"},{"issue":"9","key":"R22","doi-asserted-by":"publisher","first-page":"2511","DOI":"10.1053\/j.jvca.2025.05.033","article-title":"Evaluating large language models on American Board of Anesthesiology-style anesthesiology questions: accuracy, domain consistency, and clinical implications","volume":"39","author":"Patel","journal-title":"J Cardiothorac Vasc Anesth"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","first-page":"2307","DOI":"10.1016\/j.jds.2025.05.012","article-title":"Performance of artificial intelligence chatbots in national dental licensing examination","volume":"20","author":"Chan-Chia Lin","journal-title":"J Dent Sci"},{"issue":"8","key":"R24","doi-asserted-by":"publisher","first-page":"103519","DOI":"10.1016\/j.jsurg.2025.103519","article-title":"The current landscape of artificial intelligence in plastic surgery education and training: a systematic review","volume":"82","author":"Genovese","journal-title":"J Surg Educ"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-08388-w","article-title":"Medical undergraduate students\u2019 readiness and anxiety toward artificial intelligence: a systematic review and meta-analysis","volume":"26","author":"Luo","journal-title":"BMC Med Educ"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/66156","article-title":"Knowledge and perceptions of AI among medical students in Morocco: cross-sectional study","volume":"9","author":"Chakri","journal-title":"JMIR Form Res"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1186\/s40561-024-00313-w","article-title":"Large language models and medical education: a paradigm shift in educator roles","volume":"11","author":"Li","journal-title":"Smart Learn Environ"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","first-page":"227","DOI":"10.3390\/ai6090227","article-title":"Artificial intelligence in medical education: a narrative review on implementation, evaluation, and methodological challenges","volume":"6","author":"Roveta","journal-title":"AI"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Jacovi A Caciularu A Goldman O Goldberg Y . Stop uploading test data in plain text: practical strategies for mitigating data contamination by evaluation benchmarks. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2023:5075-5084. [doi: 10.18653\/v1\/2023.emnlp-main.308]","DOI":"10.18653\/v1\/2023.emnlp-main.308"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Sainz O Campos JA Garc\u00eda-Ferrero I Etxaniz J de Lacalle OL Agirre E . NLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark. arXiv. Preprint posted online on Oct 27, 2023. [doi: 10.48550\/arXiv.2310.18018]","DOI":"10.18653\/v1\/2023.findings-emnlp.722"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T15:13:53Z","timestamp":1788275633000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e95859"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/95859","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,1]]},"article-number":"v10i4e95859"}}