{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T05:03:26Z","timestamp":1779858206050,"version":"3.53.1"},"reference-count":48,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Behavioral Sciences"],"abstract":"<jats:p>Artificial intelligence (AI) is increasingly integrated into assessment contexts, yet evidence on the reliability and measurement properties of large language models (LLMs) in high-stakes evaluation settings remains limited. This study examines the performance and reproducibility of contemporary multimodal LLMs in a structured medical assessment environment. A cross-sectional dual-setup design was applied using a complete national medical licensing examination (240 multiple-choice items, including image-based questions). Setup 1 evaluated ten models in a single run to characterize overall performance. Setup 2 assessed six models across five independent runs each to quantify measurement stability. Accuracy with 95% confidence intervals, inter-run agreement using Cohen\u2019s kappa, and paired comparisons using McNemar\u2019s test were analyzed. Accuracy ranged from 72.08% to 92.92%. All models demonstrated near-perfect inter-run agreement (mean \u03ba \u2265 0.96) with minimal variability. After correction, only a small number of pairwise comparisons remained significant, indicating convergence among leading systems. In an exploratory submodule, performance on the small set of image-based items was comparable to or slightly higher than performance on text-only items. These findings demonstrate that multimodal LLMs achieve high accuracy and high inter-run reproducibility on a large-scale assessment, supporting their use as objects of AI-based assessment research while leaving questions of cognitive equivalence with human examinees beyond the scope of accuracy-based evaluation.<\/jats:p>","DOI":"10.3390\/bs16050822","type":"journal-article","created":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T05:33:54Z","timestamp":1779687234000},"page":"822","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Artificial Intelligence in Medical Assessment: Reliability and Performance of Multimodal Large Language Models in a High-Stakes Licensing Examination"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6395-6670","authenticated-orcid":false,"given":"Ibrahim","family":"G\u00fcler","sequence":"first","affiliation":[{"name":"Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany"},{"name":"Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelh\u00f6he, Kladower Damm 221, 14089 Berlin, Germany"},{"name":"Department of Health Management, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), Lange Gasse 20, 90403 N\u00fcrnberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7302-210X","authenticated-orcid":false,"given":"Gerrit","family":"Grieb","sequence":"additional","affiliation":[{"name":"Department of Plastic Surgery and Hand Surgery, Gemeinschaftskrankenhaus Havelh\u00f6he, Kladower Damm 221, 14089 Berlin, Germany"},{"name":"Department of Plastic Surgery and Hand Surgery, RWTH Aachen University, Pauwelsstrasse 30, 52074 Aachen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6557-2163","authenticated-orcid":false,"given":"Armin","family":"Kraus","sequence":"additional","affiliation":[{"name":"Department of Plastic, Aesthetic and Hand Surgery, Otto-von-Guericke University, 39120 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philipp","family":"Moog","sequence":"additional","affiliation":[{"name":"Department of Plastic Surgery and Hand Surgery, Klinikum Rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675 Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Uzay","family":"Cambaz","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, Eberhard Karls University of T\u00fcbingen, Geschwister-Scholl-Platz, 72074 T\u00fcbingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-1177-2345","authenticated-orcid":false,"given":"Ezgi","family":"Yavasca","sequence":"additional","affiliation":[{"name":"Department of Nuclear Medicine, Klinikum Ernst von Bergmann, 14467 Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9391-3918","authenticated-orcid":false,"given":"Henrik","family":"Stelling","sequence":"additional","affiliation":[{"name":"Department of Health Management, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), Lange Gasse 20, 90403 N\u00fcrnberg, Germany"},{"name":"Department of Nuclear Medicine, Klinikum Ernst von Bergmann, 14467 Potsdam, Germany"},{"name":"Practices for Nuclear Medicine, Rubensstra\u00dfe 125, 12157 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,19]]},"reference":[{"key":"ref_1","first-page":"e55991","article-title":"Comparing the performance of popular large language models on the national board of medical examiners sample questions","volume":"16","author":"Abbas","year":"2024","journal-title":"Cureus"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1111\/opo.12131","article-title":"When to use the Bonferroni correction","volume":"34","author":"Armstrong","year":"2014","journal-title":"Ophthalmic & Physiological Optics: The Journal of the British College of Ophthalmic Opticians (Optometrists)"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"42","DOI":"10.25282\/ted.1059067","article-title":"How much does the medical specialization exam measure basic medicine?","volume":"21","author":"Keskin","year":"2022","journal-title":"T\u0131p E\u011fitimi D\u00fcnyas\u0131"},{"key":"ref_4","unstructured":"Ayg\u00fcl, Y., Olucoglu, M., and Alpkocak, A. (2024). T\u0131pta uzmanl\u0131k s\u0131nav\u0131nda b\u00fcy\u00fck dil modelleri insanlardan daha m\u0131 ba\u015far\u0131l\u0131?. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1645","DOI":"10.1080\/0142159X.2025.2478872","article-title":"Evaluating the value of AI-generated questions for USMLE step 1 preparation: A study using ChatGPT-3.5","volume":"47","author":"Balu","year":"2025","journal-title":"Medical Teacher"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e63430","DOI":"10.2196\/63430","article-title":"ChatGPT-4 Omni performance in USMLE disciplines and clinical skills: Comparative analysis","volume":"10","author":"Bicknell","year":"2024","journal-title":"JMIR Medical Education"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1214\/ss\/1009213286","article-title":"Interval estimation for a binomial proportion","volume":"16","author":"Brown","year":"2001","journal-title":"Statistical Science"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3","DOI":"10.59863\/MIQL7785","article-title":"The rise of artificial intelligence in educational measurement: Opportunities and ethical challenges","volume":"5","author":"Bulut","year":"2024","journal-title":"Chinese\/English Journal of Educational Measurement and Evaluation"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educational and Psychological Measurement"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1007\/s40670-023-01956-z","article-title":"ChatGPT-4 performance on USMLE step 1 style questions and its implications for medical education: A comparative study across systems and disciplines","volume":"34","author":"Garabet","year":"2023","journal-title":"Medical Science Educator"},{"key":"ref_11","first-page":"e66517","article-title":"A comparative analysis of ChatGPT and medical faculty graduates in medical specialization exams: Uncovering the potential of artificial intelligence in medical education","volume":"16","author":"Gencer","year":"2024","journal-title":"Cureus"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Medical Education"},{"key":"ref_13","first-page":"373","article-title":"ChatGPT passes German state examination in medicine with picture questions omitted","volume":"120","author":"Jung","year":"2023","journal-title":"Deutsches Arzteblatt International"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kipp, M. (2024). From GPT-3.5 to GPT-4.o: A leap in AI\u2019s medical exam performance. Information, 15, (9).","DOI":"10.3390\/info15090543"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"K\u0131l\u0131\u00e7, M. E. (2023). AI in medical education: A comparative analysis of GPT-4 and GPT-3.5 on Turkish medical specialization exam performance. medRxiv.","DOI":"10.1101\/2023.07.12.23292564"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e51148","DOI":"10.2196\/51148","article-title":"Pure wisdom or potemkin villages? A comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE step 3 style questions: Quantitative analysis","volume":"10","author":"Knoedler","year":"2024","journal-title":"JMIR Medical Education"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e63626","DOI":"10.2196\/63626","article-title":"Performance evaluation of large language models in cervical cancer management based on a standardized questionnaire: Comparative study","volume":"27","author":"Kuerbanjiang","year":"2025","journal-title":"Journal of Medical Internet Research"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepa\u00f1o, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., and Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digital Health, 2, (2).","DOI":"10.1371\/journal.pdig.0000198"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_20","unstructured":"Li, P., Tang, X., Chen, S., Cheng, Y., Metoyer, R., Hua, T., and Chawla, N. V. (2025). Adaptive testing for LLM evaluation: A psychometric alternative to static benchmarks. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"976","DOI":"10.1080\/0142159X.2019.1595557","article-title":"Artificial intelligence in medical education","volume":"41","author":"Masters","year":"2019","journal-title":"Medical Teacher"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNemar","year":"1947","journal-title":"Psychometrika"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mendon\u00e7a, P. C., Quintal, F., and Mendon\u00e7a, F. (2025). Evaluating LLMs for automated scoring in formative assessments. Applied Sciences, 15, (5).","DOI":"10.3390\/app15052787"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e50965","DOI":"10.2196\/50965","article-title":"Comparison of the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination: Observational study","volume":"10","author":"Meyer","year":"2024","journal-title":"JMIR Medical Education"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1001\/jamaophthalmol.2023.2754","article-title":"Performance of an upgraded artificial intelligence chatbot for ophthalmic knowledge assessment","volume":"141","author":"Mihalache","year":"2023","journal-title":"JAMA Ophthalmology"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1080\/0142159X.2023.2249588","article-title":"ChatGPT-4: An assessment of an upgraded artificial intelligence chatbot in the United States Medical Licensing Examination","volume":"46","author":"Mihalache","year":"2024","journal-title":"Medical Teacher"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1044","DOI":"10.1093\/beheco\/arh107","article-title":"A farewell to Bonferroni: The problems of low statistical power and publication bias","volume":"15","author":"Nakagawa","year":"2004","journal-title":"Behavioral Ecology"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E","article-title":"Two-sided confidence intervals for the single proportion: Comparison of seven methods","volume":"17","author":"Newcombe","year":"1998","journal-title":"Statistics in Medicine"},{"key":"ref_29","unstructured":"Nori, H., King, N., Mayer McKinney, S., Carignan, D., and Horvitz, E. (2023). Capabilities of GPT-4 on medical challenge problems. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"em2307","DOI":"10.29333\/ejmste\/13428","article-title":"Exploring the potential of artificial intelligence tools in educational measurement and assessment","volume":"19","author":"Owan","year":"2023","journal-title":"Eurasia Journal of Mathematics, Science and Technology Education"},{"key":"ref_31","unstructured":"\u00d6l\u00e7me Se\u00e7me ve Yerle\u015ftirme Merkezi (\u00d6SYM) (2026, March 30). Hakk\u0131nda [About], Available online: https:\/\/www.osym.gov.tr\/TR,8854\/hakkinda.html."},{"key":"ref_32","unstructured":"\u00d6l\u00e7me Se\u00e7me ve Yerle\u015ftirme Merkezi (\u00d6SYM) (2026, March 30). TUS D\u00f6nem 1\u20132 De\u011ferlendirme Raporu, Available online: https:\/\/dokuman.osym.gov.tr\/pdfdokuman\/2021\/GENEL\/tusdonem1-2degrapor_29122021.pdf."},{"key":"ref_33","unstructured":"\u00d6l\u00e7me Se\u00e7me ve Yerle\u015ftirme Merkezi (\u00d6SYM) (2026, March 30). TUS D\u00f6nem 2 TTBT, Available online: https:\/\/dokuman.osym.gov.tr\/pdfdokuman\/2021\/TUSDONEM2\/CS\/tusdonem_2_TTBT.pdf."},{"key":"ref_34","unstructured":"\u00d6l\u00e7me Se\u00e7me ve Yerle\u015ftirme Merkezi (\u00d6SYM) (2026, March 30). TUS D\u00f6nem 2 KTBT, Available online: https:\/\/dokuman.osym.gov.tr\/pdfdokuman\/2021\/TUSDONEM2\/CS\/tusdonem_2_KTBT.pdf."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"310","DOI":"10.14744\/tjtes.2025.07724","article-title":"Can ChatGPT pass the Turkish orthopedics and traumatology board examination? Turkish orthopedic surgeons versus artificial intelligence. ChatGPT T\u00fcrk ortopedi ve travmatoloji yeterlilik s\u0131nav\u0131n\u0131 ge\u00e7ebilir mi? T\u00fcrk Ortopedi Cerrahlar\u0131 yapay zeka\u2019ya kar\u015f\u0131","volume":"31","author":"Pamuk","year":"2025","journal-title":"Ulusal travma ve acil cerrahi dergisi = Turkish Journal of Trauma & Emergency Surgery: TJTES"},{"key":"ref_36","first-page":"e76309","article-title":"Advancements in AI medical education: Assessing ChatGPT\u2019s performance on USMLE-style questions across topics and difficulty levels","volume":"16","author":"Penny","year":"2024","journal-title":"Cureus"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e46482","DOI":"10.2196\/46482","article-title":"Artificial intelligence in medical education: Comparative analysis of ChatGPT, Bing, and medical students in Germany","volume":"9","author":"Roos","year":"2023","journal-title":"JMIR Medical Education"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"177","DOI":"10.4274\/tjo.galenos.2025.27895","article-title":"Performance of ChatGPT-4 Omni and Gemini 1.5 Pro on ophthalmology-related questions in the Turkish medical specialty exam","volume":"55","author":"Sabaner","year":"2025","journal-title":"Turkish Journal of Ophthalmology"},{"key":"ref_39","first-page":"g6284","article-title":"Multiple hypothesis testing and Bonferroni\u2019s correction","volume":"349","author":"Sedgwick","year":"2014","journal-title":"BMJ (Clinical Research Ed.)"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Sismanoglu, S., and Capan, B. S. (2025). Performance of artificial intelligence on Turkish dental specialization exam: Can ChatGPT-4.0 and gemini advanced achieve comparable results to humans?. BMC Medical Education, 25, (1).","DOI":"10.1186\/s12909-024-06389-9"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"e55318","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: Algorithm development and validation study","volume":"12","author":"Sivarajkumar","year":"2024","journal-title":"JMIR Medical Informatics"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"597","DOI":"10.54005\/geneltip.1627508","article-title":"Cross-linguistic evaluation of artificial intelligence chatbots: Performance of ChatGPT-3.5, Copilot and Gemini in neuro-ophthalmologic evaluation in English and Turkish","volume":"35","year":"2025","journal-title":"Genel T\u0131p Dergisi"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"101052","DOI":"10.1016\/j.iheduc.2025.101052","article-title":"Exploring undergraduate students\u2019 perceptions of AI vs. human scoring and feedback","volume":"68","author":"Thomas","year":"2026","journal-title":"The Internet and Higher Education"},{"key":"ref_44","unstructured":"(2026, March 30). Orijinal TUS Sorular\u0131\u2013Sonbahar 2021, Available online: https:\/\/www.tusdata.com\/Content\/Upload\/Resimler\/nasil-soru-yakaliyoruz\/sonbahar-2021\/Materyaller\/orijinal-tus-sorulari\/1.pdf."},{"key":"ref_45","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1111\/bjet.13370","article-title":"Practical and ethical challenges of large language models in education: A systematic scoping review","volume":"55","author":"Yan","year":"2023","journal-title":"British Journal of Educational Technology"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Yang, X., and Chen, W. (2025). The performance of ChatGPT on medical image-based assessments and implications for medical education. BMC Medical Education, 25, (1).","DOI":"10.1186\/s12909-025-07752-0"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e60501","DOI":"10.2196\/60501","article-title":"Prompt engineering paradigms for medical applications: Scoping review","volume":"26","author":"Zaghir","year":"2024","journal-title":"Journal of Medical Internet Research"}],"container-title":["Behavioral Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/5\/822\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T04:25:33Z","timestamp":1779855933000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/5\/822"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,19]]},"references-count":48,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5]]}},"alternative-id":["bs16050822"],"URL":"https:\/\/doi.org\/10.3390\/bs16050822","relation":{},"ISSN":["2076-328X"],"issn-type":[{"value":"2076-328X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,19]]}}}