{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T08:15:09Z","timestamp":1788164109855,"version":"build-2803163510"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T00:00:00Z","timestamp":1788134400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T00:00:00Z","timestamp":1788134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We evaluated proprietary and open-weight foundation models on 24 German medical licensing examinations (2019\u20132024), including 7485 items and response data from 119,878 sittings. For fair comparison, the eight vision-capable models were evaluated on the full benchmark and all thirteen on a shared text-only subset. On the full benchmark, Gemini 3.1 Pro achieved the highest overall accuracy, reaching 99.31% on the first (M1) and 98.37% on the second (M2) examination. On the shared text-only subset, proprietary frontier models performed at near-ceiling levels, several open-weight models (including GLM-5 and DeepSeek V3.2-Thinking) were highly competitive, and even compact ones exceeded mean student performance. Image-present items were more difficult for both students and models, but the associated decline was disproportionately larger for models than for students. Human- and model-defined difficulty subsets showed limited overlap, and model-hard subsets revealed residual differences among top systems. These findings underscore the rapid progress of these models, particularly open-weight systems, and the value of official German medical licensing examinations as a restricted-access benchmark with reduced public exposure. They carry implications for high-stakes assessment and AI-assisted medical education, notably multimodal assessment, human-aligned educational tools, and privacy-preserving local deployment.<\/jats:p>","DOI":"10.1038\/s41746-026-03082-7","type":"journal-article","created":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T06:34:53Z","timestamp":1788158093000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating foundation models on official German medical licensing examinations: Implications for high-stakes assessment and AI-assisted medical education"],"prefix":"10.1038","volume":"9","author":[{"given":"Lasse","family":"Cirkel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johannes","family":"Knitza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Volker","family":"Schillings","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Oksche","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan Carl","family":"Becker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastian","family":"Kuhn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,8,31]]},"reference":[{"key":"3082_CR1","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2, e0000198 (2023).","journal-title":"PLOS Digit Health"},{"key":"3082_CR2","doi-asserted-by":"publisher","unstructured":"Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems. https:\/\/doi.org\/10.48550\/arXiv.2303.13375 (2023).","DOI":"10.48550\/arXiv.2303.13375"},{"key":"3082_CR3","doi-asserted-by":"publisher","first-page":"e66114","DOI":"10.2196\/66114","volume":"26","author":"H Zong","year":"2024","unstructured":"Zong, H. et al. Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. J. Med. Internet Res. 26, e66114 (2024).","journal-title":"J. Med. Internet Res."},{"key":"3082_CR4","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","volume":"30","author":"D Van Veen","year":"2024","unstructured":"Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134\u20131142 (2024).","journal-title":"Nat. Med."},{"key":"3082_CR5","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","volume":"31","author":"K Singhal","year":"2025","unstructured":"Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med. 31, 943\u2013950 (2025).","journal-title":"Nat. Med."},{"key":"3082_CR6","doi-asserted-by":"publisher","unstructured":"Phan, L. et al. Humanity\u2019s Last Exam https:\/\/doi.org\/10.48550\/arXiv.2501.14249 (2025).","DOI":"10.48550\/arXiv.2501.14249"},{"key":"3082_CR7","doi-asserted-by":"publisher","unstructured":"Snell, C., Lee, J., Xu, K. & Kumar, A. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters https:\/\/doi.org\/10.48550\/arXiv.2408.03314 (2024).","DOI":"10.48550\/arXiv.2408.03314"},{"key":"3082_CR8","unstructured":"Open Life Science AI. Open Medical LLM Leaderboard. https:\/\/huggingface.co\/spaces\/openlifescienceai\/open_medical_llm_leaderboard. Accessed March 23, 2026."},{"key":"3082_CR9","doi-asserted-by":"publisher","unstructured":"Nori, H. et al. From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond https:\/\/doi.org\/10.48550\/arXiv.2411.03590 (2024).","DOI":"10.48550\/arXiv.2411.03590"},{"key":"3082_CR10","doi-asserted-by":"crossref","unstructured":"Gallifant, J. et al. Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks 2406.12066 (2024).","DOI":"10.18653\/v1\/2024.findings-emnlp.726"},{"key":"3082_CR11","doi-asserted-by":"publisher","unstructured":"Ravaut, M. et al. A Comprehensive Survey of Contamination Detection Methods in Large Language Models https:\/\/doi.org\/10.48550\/arXiv.2404.00699 (2024).","DOI":"10.48550\/arXiv.2404.00699"},{"key":"3082_CR12","doi-asserted-by":"publisher","unstructured":"Zuo, Y. et al. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding https:\/\/doi.org\/10.48550\/arXiv.2501.18362 (2025)","DOI":"10.48550\/arXiv.2501.18362"},{"key":"3082_CR13","doi-asserted-by":"publisher","unstructured":"Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E. & Stoica, I. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples https:\/\/doi.org\/10.48550\/arXiv.2311.04850 (2023).","DOI":"10.48550\/arXiv.2311.04850"},{"key":"3082_CR14","first-page":"373","volume":"120","author":"LB Jung","year":"2023","unstructured":"Jung, L. B. et al. ChatGPT Passes German State Examination in Medicine With Picture Questions Omitted. Dtsch Arztebl Int. 120, 373\u2013374 (2023).","journal-title":"Dtsch Arztebl Int."},{"key":"3082_CR15","doi-asserted-by":"publisher","first-page":"e46482","DOI":"10.2196\/46482","volume":"9","author":"J Roos","year":"2023","unstructured":"Roos, J., Kasapovic, A., Jansen, T. & Kaczmarczyk, R. Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Medical Students in Germany. JMIR Med. Educ. 9, e46482 (2023).","journal-title":"JMIR Med. Educ."},{"key":"3082_CR16","doi-asserted-by":"publisher","first-page":"e50965","DOI":"10.2196\/50965","volume":"10","author":"A Meyer","year":"2024","unstructured":"Meyer, A., Riese, J. & Streichert, T. Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study. JMIR Med. Educ. 10, e50965 (2024).","journal-title":"JMIR Med. Educ."},{"key":"3082_CR17","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1007\/s00101-024-01403-7","volume":"73","author":"P Humbsch","year":"2024","unstructured":"Humbsch, P., Horn, E., Bohm, K. & Gintrowicz, R. ChatGPT im Einsatz f\u00fcr technology-enhanced learning in An\u00e4sthesiologie und Notfallmedizin und potenzielle klinische Anwendung von KI-Sprachmodellen. Die Anaesthesiologie 73, 324\u2013335 (2024).","journal-title":"Die Anaesthesiologie"},{"key":"3082_CR18","first-page":"888","volume":"121","author":"ME Geissler","year":"2024","unstructured":"Geissler, M. E. et al. The Performance of Artificial Intelligence on a National Medical Licensing Examination. Dtsch Arztebl Int. 121, 888\u2013889 (2024).","journal-title":"Dtsch Arztebl Int."},{"key":"3082_CR19","doi-asserted-by":"publisher","first-page":"e58375","DOI":"10.2196\/58375","volume":"11","author":"J Madrid","year":"2025","unstructured":"Madrid, J. et al. Performance of Plug-In Augmented ChatGPT and Its Ability to Quantify Uncertainty: Simulation Study on the German Medical Board Examination. JMIR Med. Educ. 11, e58375 (2025).","journal-title":"JMIR Med. Educ."},{"key":"3082_CR20","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","volume":"616","author":"M Moor","year":"2023","unstructured":"Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259\u2013265 (2023).","journal-title":"Nature"},{"key":"3082_CR21","doi-asserted-by":"crossref","unstructured":"Matos, J. et al. WorldMedQA-V: A Multilingual, Multimodal Medical Examination Dataset for Multimodal Language Models Evaluation. Findings of the Association for Computational Linguistics: NAACL 2025 7218\u20137231 (2025).","DOI":"10.18653\/v1\/2025.findings-naacl.402"},{"key":"3082_CR22","doi-asserted-by":"publisher","unstructured":"Choi, B., Bae, S., Kweon, S. & Choi, E. KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination. https:\/\/doi.org\/10.48550\/arXiv.2602.13650 (2026).","DOI":"10.48550\/arXiv.2602.13650"},{"key":"3082_CR23","doi-asserted-by":"publisher","first-page":"102938","DOI":"10.1016\/j.artmed.2024.102938","volume":"155","author":"I Alonso","year":"2024","unstructured":"Alonso, I., Oronoz, M. & Agerri, R. MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering. Artif. Intell. Med. 155, 102938 (2024).","journal-title":"Artif. Intell. Med."},{"key":"3082_CR24","doi-asserted-by":"publisher","first-page":"e59505","DOI":"10.2196\/59505","volume":"26","author":"R AlSaad","year":"2024","unstructured":"AlSaad, R. et al. Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. J. Med. Internet Res 26, e59505 (2024).","journal-title":"J. Med. Internet Res"},{"key":"3082_CR25","doi-asserted-by":"publisher","first-page":"888","DOI":"10.1038\/s41586-025-08675-y","volume":"639","author":"VM Rao","year":"2025","unstructured":"Rao, V. M. et al. Multimodal generative AI for medical image interpretation. Nature 639, 888\u2013896 (2025).","journal-title":"Nature"},{"key":"3082_CR26","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01185-7","volume":"7","author":"Q Jin","year":"2024","unstructured":"Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. npj Digital Med. 7, 190 (2024).","journal-title":"npj Digital Med."},{"key":"3082_CR27","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01963-x","volume":"8","author":"M Agrawal","year":"2025","unstructured":"Agrawal, M., Chen, I. Y., Gulamali, F. & Joshi, S. The evaluation illusion of large language models in medicine. npj Digital Med. 8, 600 (2025).","journal-title":"npj Digital Med."},{"key":"3082_CR28","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1038\/s41591-024-03456-y","volume":"31","author":"E Goh","year":"2025","unstructured":"Goh, E. et al. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat. Med. 31, 1233\u20131238 (2025).","journal-title":"Nat. Med."},{"key":"3082_CR29","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1038\/s41746-026-02545-1","volume":"9","author":"SS Everett","year":"2026","unstructured":"Everett, S. S. et al. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. npj Digit. Med. 9, 409 (2026).","journal-title":"npj Digit. Med."},{"key":"3082_CR30","unstructured":"Institut f\u00fcr medizinische und pharmazeutische Pr\u00fcfungsfragen (IMPP). Fachbereich Medizin. https:\/\/www.impp.de\/ueber-uns\/organisation\/fachbereiche\/medizin.html. Accessed March 23, 2026."},{"key":"3082_CR31","unstructured":"Institut f\u00fcr medizinische und pharmazeutische Pr\u00fcfungsfragen (IMPP). Sachverst\u00e4ndige. https:\/\/www.impp.de\/informationen\/partner\/sachverstaendige.html. Accessed March 23, 2026."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-03082-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-03082-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-03082-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T07:26:53Z","timestamp":1788161213000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-03082-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,31]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["3082"],"URL":"https:\/\/doi.org\/10.1038\/s41746-026-03082-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,31]]},"assertion":[{"value":"23 April 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 August 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"670"}}