{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T06:47:22Z","timestamp":1782456442542,"version":"3.54.5"},"reference-count":15,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T00:00:00Z","timestamp":1782432000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T00:00:00Z","timestamp":1782432000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002839","name":"Charit\u00e9 - Universit\u00e4tsmedizin Berlin","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002839","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Med Syst"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Objective: To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions. Materials and Methods: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence. Results: Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties (\u201cSkin\u201d best: 0.041; \u201cSocial &amp; Preventive Medicine\u201d worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties. Discussion: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains. Conclusion: We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved \u201cself-knowledge\u201d, but uneven overconfidence persists.<\/jats:p>","DOI":"10.1007\/s10916-026-02430-0","type":"journal-article","created":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T06:08:31Z","timestamp":1782454111000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering"],"prefix":"10.1007","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7993-7901","authenticated-orcid":false,"given":"Sebastian Daniel","family":"Boie","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florian","family":"Reis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Frey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elias","family":"Gr\u00fcnewald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Balzer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,26]]},"reference":[{"issue":"2","key":"2430_CR1","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1038\/s42256-024-00976-7","volume":"7","author":"M Steyvers","year":"2025","unstructured":"Steyvers M, Tejeda H, Kumar A, Belem C, Karny S, Hu X, et al. What large language models know and what people think they know. Nat Mach Intell. 2025;7(2):221\u201331.","journal-title":"Nat Mach Intell"},{"issue":"14","key":"2430_CR2","doi-asserted-by":"publisher","first-page":"1377","DOI":"10.1001\/jama.2017.12126","volume":"318","author":"AC Alba","year":"2017","unstructured":"Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, et al. Discrimination and Calibration of Clinical Prediction Models: Users\u2019 Guides to the Medical Literature. JAMA. 2017;318(14):1377.","journal-title":"JAMA"},{"issue":"1","key":"2430_CR3","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1038\/s41746-020-00367-3","volume":"4","author":"B Kompa","year":"2021","unstructured":"Kompa B, Snoek J, Beam AL. Second opinion needed: communicating uncertainty in medical machine learning. npj Digit Med. 2021;4(1):4.","journal-title":"npj Digit Med"},{"key":"2430_CR4","doi-asserted-by":"crossref","unstructured":"Ehsan U, Riedl MO. Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models. In: Proceedings of the Halfway to the Future Symposium [Internet]. 2024 [cited 2025 Nov 24]. p. 1\u20138. Available from: http:\/\/arxiv.org\/abs\/2408.05345","DOI":"10.1145\/3686169.3686185"},{"key":"2430_CR5","unstructured":"Regulation (EU) 2024\/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300\/2008, (EU) No 167\/2013, (EU) No 168\/2013, (EU) 2018\/858, (EU) 2018\/1139 and (EU) 2019\/2144 and Directives 2014\/90\/EU, (EU) 2016\/797 and (EU) 2020\/1828 (Artificial Intelligence Act). 2024 July 12; Available from: http:\/\/data.europa.eu\/eli\/reg\/2024\/1689\/oj"},{"issue":"1","key":"2430_CR6","doi-asserted-by":"publisher","first-page":"ooae154","DOI":"10.1093\/jamiaopen\/ooae154","volume":"8","author":"Y Gao","year":"2024","unstructured":"Gao Y, Myers S, Chen S, Dligach D, Miller T, Bitterman DS, et al. Uncertainty estimation in diagnosis generation from large language models: next-word probability is not pre-test probability. JAMIA Open. 2024;8(1):ooae154.","journal-title":"JAMIA Open"},{"key":"2430_CR7","doi-asserted-by":"crossref","unstructured":"De Oliveira R, Garber M, Gwinnutt JM, Rashidi E, Hwang JH (Shantina), Gilmour W, et al. A study of calibration as a measurement of trustworthiness of large language models in biomedical natural language processing. JAMIA Open. 2025 July 3;8(4):ooaf058.","DOI":"10.1093\/jamiaopen\/ooaf058"},{"key":"2430_CR8","doi-asserted-by":"publisher","first-page":"e64348","DOI":"10.2196\/64348","volume":"27","author":"R Bentegeac","year":"2025","unstructured":"Bentegeac R, Le Guellec B, Kuchcinski G, Amouyel P, Hamroun A. Token Probabilities to Mitigate Large Language Models Overconfidence in Answering Medical Questions: Quantitative Study. J Med Internet Res. 2025;27:e64348\u2013e64348.","journal-title":"J Med Internet Res"},{"issue":"3","key":"2430_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3744238","volume":"58","author":"O Shorinwa","year":"2026","unstructured":"Shorinwa O, Mei Z, Lidard J, Ren AZ, Majumdar A. A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions. ACM Comput Surv. 2026;58(3):1\u201338.","journal-title":"ACM Comput Surv"},{"key":"2430_CR10","doi-asserted-by":"crossref","unstructured":"Bakman YF, Yaldiz DN, Buyukates B, Tao C, Dimitriadis D, Avestimehr S. Mars: Meaning-aware response scoring for uncertainty estimation in generative llms. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024. p. 7752\u201367.","DOI":"10.18653\/v1\/2024.acl-long.419"},{"key":"2430_CR11","doi-asserted-by":"publisher","first-page":"e66917","DOI":"10.2196\/66917","volume":"13","author":"M Omar","year":"2025","unstructured":"Omar M, Agbareia R, Glicksberg BS, Nadkarni GN, Klang E. Benchmarking the Confidence of Large Language Models in Answering Clinical Questions: Cross-Sectional Evaluation Study. JMIR Med Inform. 2025;13:e66917\u2013e66917.","journal-title":"JMIR Med Inform"},{"key":"2430_CR12","unstructured":"Pal A, Umapathi LK, Sankarasubbu M. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In: Proceedings of the Conference on Health, Inference, and Learning. 2022. p. 248\u201360."},{"issue":"1","key":"2430_CR13","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1093\/jamia\/ocae254","volume":"32","author":"T Savage","year":"2025","unstructured":"Savage T, Wang J, Gallo R, Boukil A, Patel V, Safavi-Naini SAA, et al. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics Association. 2025;32(1):139\u201349.","journal-title":"Journal of the American Medical Informatics Association"},{"key":"2430_CR14","doi-asserted-by":"publisher","first-page":"100030","DOI":"10.1016\/j.ejrai.2025.100030","volume":"3","author":"B Kocak","year":"2025","unstructured":"Kocak B, Klontzas ME, Stanzione A, Meddeb A, Demircio\u011flu A, Bluethgen C, et al. Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations. European Journal of Radiology Artificial Intelligence. 2025 Sept;3:100030.","journal-title":"European Journal of Radiology Artificial Intelligence"},{"key":"2430_CR15","doi-asserted-by":"publisher","unstructured":"Kim SSY, Liao QV, Vorvoreanu M, Ballard S, Vaughan JW. \u201cI\u2019m Not Sure, But\u2026 Examining the Impact of Large Language Models\u2019 Uncertainty Expression on User Reliance and Trust. In: The 2024 ACM Conference on Fairness Accountability and Transparency [Internet]. Rio de Janeiro Brazil: ACM; 2024 [cited 2025 Dec 18]. p. 822\u201335. Available from: https:\/\/doi.org\/10.1145\/3630106.3658941","DOI":"10.1145\/3630106.3658941"}],"container-title":["Journal of Medical Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-026-02430-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10916-026-02430-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-026-02430-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T06:08:39Z","timestamp":1782454119000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10916-026-02430-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,26]]},"references-count":15,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2430"],"URL":"https:\/\/doi.org\/10.1007\/s10916-026-02430-0","relation":{},"ISSN":["1573-689X"],"issn-type":[{"value":"1573-689X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,26]]},"assertion":[{"value":"12 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 June 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 June 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics and Consent to Participate"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}},{"value":"not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Clinical Trial Number"}},{"value":"the authors declare that generative AI (GPT-5) was used in the preparation of this manuscript solely for language editing to improve grammar and clarity. No generative AI tools were used to develop the study design, analyze the data, interpret the findings, or draft the scientific conclusions. All manuscript edits were reviewed and approved by the authors, who take full responsibility for the content.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Generative AI Statement"}}],"article-number":"103"}}