{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T07:33:14Z","timestamp":1794036794930,"version":"4.8.0"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100007397","name":"Charles University in Prague","doi-asserted-by":"crossref","award":["Cooperatio, Medical Diagnostics and Basic Medical Sciences"],"award-info":[{"award-number":["Cooperatio, Medical Diagnostics and Basic Medical Sciences"]}],"id":[{"id":"10.13039\/100007397","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003243","name":"Ministerstvo Zdravotnictv\u00ed Cesk\u00e9 Republiky","doi-asserted-by":"publisher","award":["MH CZ-DRO, Motol University Hospital, 00064203"],"award-info":[{"award-number":["MH CZ-DRO, Motol University Hospital, 00064203"]}],"id":[{"id":"10.13039\/501100003243","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol Exp"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Background<\/jats:title>\n            <jats:p>We compared the performance, confidence, and response consistency of five chatbots powered by large language models in solving European Diploma in Radiology (EDiR) text-based multiple-response questions.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>ChatGPT-4o, ChatGPT-4o-mini, Copilot, Gemini, and Claude 3.5 Sonnet were tested using 52 text-based multiple-response questions from two previous EDiR sessions in two iterations. Chatbots were prompted to evaluate each answer as correct or incorrect and grade its confidence level on a scale of 0 (not confident at all) to 10 (most confident). Scores per question were calculated using a weighted formula that accounted for correct and incorrect answers (range 0.0\u20131.0).<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>Claude 3.5 Sonnet achieved the highest score per question (0.84\u2009\u00b1\u20090.26, mean\u2009\u00b1\u2009standard deviation) compared to ChatGPT-4o (0.76\u2009\u00b1\u20090.31), ChatGPT-4o-mini (0.64\u2009\u00b1\u20090.35), Copilot (0.62\u2009\u00b1\u20090.37), and Gemini (0.54\u2009\u00b1\u20090.39) (<jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). A self-reported confidence in answering the questions was 9.0\u2009\u00b1\u20090.9 for Claude 3.5 Sonnet followed by ChatGPT-4o (8.7\u2009\u00b1\u20091.1), compared to ChatGPT-4o-mini (8.2\u2009\u00b1\u20091.3), Copilot (8.2\u2009\u00b1\u20092.2), and Gemini (8.2\u2009\u00b1\u20091.6, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). Claude 3.5 Sonnet demonstrated superior consistency, changing responses in 5.4% of cases between the two iterations, compared to ChatGPT-4o (6.5%), ChatGPT-4o-mini (8.8%), Copilot (13.8%), and Gemini (18.5%). All chatbots outperformed human candidates from previous EDiR sessions, achieving a passing grade from this part of the examination.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>Claude 3.5 Sonnet exhibited superior accuracy, confidence, and consistency, with ChatGPT-4o performing nearly as well. The variation in performance among the evaluated models was substantial.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Relevance statement<\/jats:title>\n            <jats:p>Variation in performance, consistency, and confidence among chatbots in solving EDiR test-based questions highlights the need for cautious deployment, particularly in high-stakes clinical and educational settings.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Key Points<\/jats:title>\n            <jats:p>\n              <jats:list list-type=\"bullet\">\n                <jats:list-item>\n                  <jats:p>Claude 3.5 Sonnet outperformed other chatbots in accuracy and response consistency.<\/jats:p>\n                <\/jats:list-item>\n                <jats:list-item>\n                  <jats:p>ChatGPT-4o ranked second, showing strong but slightly less reliable performance.<\/jats:p>\n                <\/jats:list-item>\n                <jats:list-item>\n                  <jats:p>All chatbots surpassed EDiR candidates in text-based EDiR questions.<\/jats:p>\n                <\/jats:list-item>\n              <\/jats:list>\n            <\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Graphical Abstract<\/jats:title>\n          <\/jats:sec>","DOI":"10.1186\/s41747-025-00591-0","type":"journal-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T17:19:51Z","timestamp":1755623991000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Five advanced chatbots solving European Diploma in Radiology (EDiR) text-based questions: differences in performance and consistency"],"prefix":"10.1186","volume":"9","author":[{"given":"Jakub","family":"Pristoupil","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Oleaga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vanesa","family":"Junquero","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristina","family":"Merino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suha Sureyya","family":"Ozbek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2299-4707","authenticated-orcid":false,"given":"Lukas","family":"Lambert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"European Society of Radiology (ESR)","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,8,19]]},"reference":[{"key":"591_CR1","doi-asserted-by":"publisher","first-page":"57","DOI":"10.3390\/informatics11030057","volume":"11","author":"ZA Nazi","year":"2024","unstructured":"Nazi ZA, Peng W (2024) Large language models in healthcare and medical domain: a review. Informatics 11:57. https:\/\/doi.org\/10.3390\/informatics11030057","journal-title":"Informatics"},{"key":"591_CR2","doi-asserted-by":"publisher","first-page":"e232756","DOI":"10.1148\/radiol.232756","volume":"310","author":"R Bhayana","year":"2024","unstructured":"Bhayana R (2024) Chatbots and large language models in radiology: a practical primer for clinical and research applications. Radiology 310:e232756. https:\/\/doi.org\/10.1148\/radiol.232756","journal-title":"Radiology"},{"key":"591_CR3","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1186\/s13244-023-01507-5","volume":"14","author":"F Stoehr","year":"2023","unstructured":"Stoehr F, K\u00e4mpgen B, M\u00fcller L et al (2023) Natural language processing for automatic evaluation of free-text answers\u2014a feasibility study based on the European Diploma in Radiology examination. Insights Imaging 14:150. https:\/\/doi.org\/10.1186\/s13244-023-01507-5","journal-title":"Insights Imaging"},{"key":"591_CR4","doi-asserted-by":"publisher","unstructured":"Busch F, Hoffmann L, Santos D et al (2024) Large language models for structured reporting in radiology: past, present, and future. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-024-11107-6","DOI":"10.1007\/s00330-024-11107-6"},{"key":"591_CR5","doi-asserted-by":"publisher","first-page":"e241736","DOI":"10.1148\/radiol.241736","volume":"313","author":"A Meddeb","year":"2024","unstructured":"Meddeb A, L\u00fcken S, Busch F et al (2024) Large language model ability to translate CT and MRI free-text radiology reports into multiple languages. Radiology 313:e241736. https:\/\/doi.org\/10.1148\/radiol.241736","journal-title":"Radiology"},{"key":"591_CR6","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02709-7","volume":"24","author":"J Lee","year":"2024","unstructured":"Lee J, Park S, Shin J, Cho B (2024) Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Med Inform Decis Mak 24:366. https:\/\/doi.org\/10.1186\/s12911-024-02709-7","journal-title":"BMC Med Inform Decis Mak"},{"key":"591_CR7","doi-asserted-by":"publisher","first-page":"e230582","DOI":"10.1148\/radiol.230582","volume":"307","author":"R Bhayana","year":"2023","unstructured":"Bhayana R, Krishna S, Bleakney RR (2023) Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology 307:e230582. https:\/\/doi.org\/10.1148\/radiol.230582","journal-title":"Radiology"},{"key":"591_CR8","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1186\/s13244-025-01941-7","volume":"16","author":"J Pristoupil","year":"2025","unstructured":"Pristoupil J, Oleaga L, Junquero V et al (2025) Generative pre-trained transformer 4o (GPT-4o) in solving text-based multiple response questions for European Diploma in Radiology (EDiR): a comparative study with radiologists. Insights Imaging 16:66. https:\/\/doi.org\/10.1186\/s13244-025-01941-7","journal-title":"Insights Imaging"},{"key":"591_CR9","doi-asserted-by":"publisher","first-page":"905","DOI":"10.1007\/s13244-018-0665-7","volume":"9","author":"European Board of Radiology","year":"2018","unstructured":"European Board of Radiology (2018) The European Diploma in Radiology (EDiR): investing in the future of the new generations of radiologists. Insights Imaging 9:905\u2013909. https:\/\/doi.org\/10.1007\/s13244-018-0665-7","journal-title":"Insights Imaging"},{"key":"591_CR10","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.rxeng.2023.01.007","volume":"65","author":"L Oleaga Zufir\u00eda","year":"2023","unstructured":"Oleaga Zufir\u00eda L (2023) European Diploma in Radiology (EDiR). Radiologia 65:193\u2013194. https:\/\/doi.org\/10.1016\/j.rxeng.2023.01.007","journal-title":"Radiologia"},{"key":"591_CR11","doi-asserted-by":"publisher","first-page":"e240153","DOI":"10.1148\/radiol.240153","volume":"312","author":"N Hayden","year":"2024","unstructured":"Hayden N, Gilbert S, Poisson LM et al (2024) Performance of GPT-4 with vision on text- and image-based ACR diagnostic radiology in-training examination questions. Radiology 312:e240153. https:\/\/doi.org\/10.1148\/radiol.240153","journal-title":"Radiology"},{"key":"591_CR12","doi-asserted-by":"publisher","unstructured":"Brin D, Sorin V, Barash Y et al (2024) Assessing GPT-4 multimodal performance in radiological image analysis. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-024-11035-5","DOI":"10.1007\/s00330-024-11035-5"},{"key":"591_CR13","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1186\/s13244-020-00844-z","volume":"11","author":"European Board of Radiology","year":"2020","unstructured":"European Board of Radiology (2020) Implementation of the clinically oriented reasoning evaluation: impact on the European Diploma in Radiology (EDiR) exam. Insights Imaging 11:45. https:\/\/doi.org\/10.1186\/s13244-020-00844-z","journal-title":"Insights Imaging"},{"key":"591_CR14","doi-asserted-by":"publisher","unstructured":"Savage T, Wang J, Gallo R et al (2025) Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. J Am Med Inform Assoc 32:139\u2013149. https:\/\/doi.org\/10.1093\/jamia\/ocae254","DOI":"10.1093\/jamia\/ocae254"},{"key":"591_CR15","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1016\/j.diii.2024.04.003","volume":"105","author":"P Keshavarz","year":"2024","unstructured":"Keshavarz P, Bagherieh S, Nabipoorashrafi SA et al (2024) ChatGPT in radiology: a systematic review of performance, pitfalls, and future perspectives. Diagn Interv Imaging 105:251\u2013256. https:\/\/doi.org\/10.1016\/j.diii.2024.04.003","journal-title":"Diagn Interv Imaging"},{"key":"591_CR16","doi-asserted-by":"publisher","unstructured":"Oc Y, Hassen H (2024) Comparing the effectiveness of multiple-answer and single-answer multiple-choice questions in assessing student learning. Mark Educ Rev 35:44\u201357. https:\/\/doi.org\/10.1080\/10528008.2024.2417106","DOI":"10.1080\/10528008.2024.2417106"},{"key":"591_CR17","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1007\/s11604-023-01491-2","volume":"42","author":"Y Toyama","year":"2024","unstructured":"Toyama Y, Harigai A, Abe M et al (2024) Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol 42:201\u2013207. https:\/\/doi.org\/10.1007\/s11604-023-01491-2","journal-title":"Jpn J Radiol"},{"key":"591_CR18","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05630-9","volume":"24","author":"G Rossettini","year":"2024","unstructured":"Rossettini G, Rodeghiero L, Corradi F et al (2024) Comparative accuracy of ChatGPT-4, Microsoft Copilot and Google Gemini in the Italian entrance test for healthcare sciences degrees: a cross-sectional study. BMC Med Educ 24:694. https:\/\/doi.org\/10.1186\/s12909-024-05630-9","journal-title":"BMC Med Educ"},{"key":"591_CR19","doi-asserted-by":"publisher","first-page":"e59960","DOI":"10.7759\/cureus.59960","volume":"16","author":"M Tepe","year":"2024","unstructured":"Tepe M, Emekli E (2024) Assessing the responses of large language models (ChatGPT-4, Gemini, and Microsoft Copilot) to frequently asked questions in breast imaging: a study on readability and accuracy. Cureus 16:e59960. https:\/\/doi.org\/10.7759\/cureus.59960","journal-title":"Cureus"},{"key":"591_CR20","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1152\/advan.00093.2024","volume":"49","author":"V Mavrych","year":"2025","unstructured":"Mavrych V, Yaqinuddin A, Bolgova O (2025) Claude, ChatGPT, Copilot, and Gemini performance versus students in different topics of neuroscience. Adv Physiol Educ 49:430\u2013437. https:\/\/doi.org\/10.1152\/advan.00093.2024","journal-title":"Adv Physiol Educ"},{"key":"591_CR21","doi-asserted-by":"publisher","unstructured":"Omar M, Agbareia R, Glicksberg BS, Nadkarni GN, Klang E (2024) Benchmarking the confidence of large language models in clinical questions. MedRxiv 2024.08.11.24311810. https:\/\/doi.org\/10.1101\/2024.08.11.24311810","DOI":"10.1101\/2024.08.11.24311810"},{"key":"591_CR22","doi-asserted-by":"publisher","first-page":"6652","DOI":"10.1007\/s00330-024-10727-2","volume":"34","author":"R Siepmann","year":"2024","unstructured":"Siepmann R, Huppertz M, Rastkhiz A et al (2024) The virtual reference radiologist: comprehensive AI assistance for clinical image reading and interpretation. Eur Radiol 34:6652\u20136666. https:\/\/doi.org\/10.1007\/s00330-024-10727-2","journal-title":"Eur Radiol"},{"key":"591_CR23","doi-asserted-by":"publisher","first-page":"657","DOI":"10.3390\/ejihpe14030043","volume":"14","author":"PF Funk","year":"2024","unstructured":"Funk PF, Hoch CC, Knoedler S et al (2024) ChatGPT\u2019s response consistency: a study on repeated queries of medical examination questions. Eur J Invest Health Psychol Educ 14:657. https:\/\/doi.org\/10.3390\/ejihpe14030043","journal-title":"Eur J Invest Health Psychol Educ"},{"key":"591_CR24","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1186\/s13244-024-01801-w","volume":"15","author":"M Zanardo","year":"2024","unstructured":"Zanardo M, Visser JJ, Colarieti A et al (2024) Impact of AI on radiology: a EuroAIM\/EuSoMII 2024 survey among members of the European Society of Radiology. Insights Imaging 15:240. https:\/\/doi.org\/10.1186\/s13244-024-01801-w","journal-title":"Insights Imaging"}],"container-title":["European Radiology Experimental"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41747-025-00591-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41747-025-00591-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41747-025-00591-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T17:19:53Z","timestamp":1755623993000},"score":1,"resource":{"primary":{"URL":"https:\/\/eurradiolexp.springeropen.com\/articles\/10.1186\/s41747-025-00591-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,19]]},"references-count":24,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["591"],"URL":"https:\/\/doi.org\/10.1186\/s41747-025-00591-0","relation":{},"ISSN":["2509-9280"],"issn-type":[{"value":"2509-9280","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,19]]},"assertion":[{"value":"20 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 April 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 August 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethical approval for this study was obtained from the Ethics Committee of the General University Hospital in Prague (79\/24 S-IV Grant). Informed consent was not required.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Informed consent was not required.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Vanesa Junquero and Cristina Merino are employees of the European Board of Radiology. The remaining authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"79"}}