{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,25]],"date-time":"2026-10-25T20:16:39Z","timestamp":1792959399508,"version":"4.5.0"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T00:00:00Z","timestamp":1745539200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T00:00:00Z","timestamp":1745539200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Educ"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate the performance of advanced large language models (LLMs)\u2014OpenAI-ChatGPT 4, Google AI-Gemini 1.5 Pro, Cohere-Command R\u2009+\u2009and Meta AI-Llama 3 70B on questions from the Turkish Medical Specialty Training Entrance Exam (2021, 1st semester) and analyze their answers for user interpretability in languages other than English.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>The study used questions from the Basic Medical Sciences and Clinical Medical Sciences exams of the Turkish Medical Specialty Training Entrance Exam held on March 21, 2021. The 240 questions were presented to the LLMs in Turkish, and their responses were evaluated based on the official answers published by the Student Selection and Placement Centre.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>ChatGPT 4 was the best-performing model with an overall accuracy of 88.75%. Llama 3 70B followed closely with 79.17% accuracy. Gemini 1.5 Pro achieved 78.13% accuracy, while Command R\u2009+\u2009lagged with 50% accuracy. ChatGPT 4 demonstrated strengths in both basic and clinical medical science questions. Performance varied across question difficulties, with ChatGPT 4 maintaining high accuracy even on the most challenging questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>GPT-4 and Llama 3 70B achieved satisfactory results on the Turkish Medical Specialty Training Entrance Exam, demonstrating their potential as safe sources for basic medical sciences and clinical medical sciences knowledge in languages other than English. These LLMs could be valuable resources for medical education and clinical support in non-English speaking areas. However, Gemini 1.5 Pro and Command R\u2009+\u2009show potential but need significant improvement to compete with the best-performing models.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12909-025-07148-0","type":"journal-article","created":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T03:22:30Z","timestamp":1745551350000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["The role of artificial intelligence in medical education: an evaluation of Large Language Models (LLMs) on the Turkish Medical Specialty Training Entrance Exam"],"prefix":"10.1186","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6510-3666","authenticated-orcid":false,"given":"Murat","family":"Ko\u00e7ak","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3939-2924","authenticated-orcid":false,"given":"Ali Kemal","family":"O\u011fuz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2473-4431","authenticated-orcid":false,"given":"Zafer","family":"Ak\u00e7al\u0131","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,4,25]]},"reference":[{"key":"7148_CR1","unstructured":"ChatGPT-4 is OpenAI's most advanced system, producing safer and more useful responses. URL: https:\/\/openai.com\/product\/gpt-4.\u00a0Accessed 2024\u201304\u201310."},{"key":"7148_CR2","doi-asserted-by":"publisher","unstructured":"Beaulieu-Jones BR, Shah S, Berrigan MT, Marwaha JS, Lai SL, Brat GA. Evaluating capabilities of large language models: performance of GPT4 on surgical knowledge assessments.\u00a0medRxiv : the preprint server for health sciences. 2023.\u00a02023.07.16.23292743. https:\/\/doi.org\/10.1101\/2023.07.16.23292743.","DOI":"10.1101\/2023.07.16.23292743"},{"key":"7148_CR3","doi-asserted-by":"publisher","unstructured":"Li Q, Min X. Unleashing the power of language models in clinical settings: a trailblazing evaluation unveiling novel test design.\u00a0medRxiv.\u00a02023:2023\u201307. https:\/\/doi.org\/10.1101\/2023.07.11.23292512.","DOI":"10.1101\/2023.07.11.23292512"},{"key":"7148_CR4","unstructured":"Sharma P, Thapa K, Dhakal P, Upadhaya MD, Adhikari S, Khanal SR. Performance of chatgpt on usmle: unlocking the potential of large language models for ai-assisted medical education. 2023. arXiv preprint arXiv:2307.00112."},{"key":"7148_CR5","doi-asserted-by":"crossref","unstructured":"Shetty M, Ettlinger M, Lynch M. GPT-4, an artificial intelligence large language model, exhibits high levels of accuracy on dermatology specialty certificate exam questions.\u00a0medRxiv. 2023:2023\u201307.","DOI":"10.1101\/2023.07.13.23292418"},{"key":"7148_CR6","unstructured":"Wu CK, Chen WL, Chen HH. Large language models perform diagnostic reasoning. 2023.\u00a0arXiv preprint arXiv:2307.08922."},{"key":"7148_CR7","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","volume":"9","author":"A Gilson","year":"2023","unstructured":"Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR medical education. 2023;9:e45312. https:\/\/doi.org\/10.2196\/45312.","journal-title":"JMIR medical education"},{"key":"7148_CR8","doi-asserted-by":"crossref","unstructured":"Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study.\u00a0JMIR Med Educ.\u00a02023;9(1):e48002.","DOI":"10.2196\/48002"},{"key":"7148_CR9","doi-asserted-by":"publisher","first-page":"e48039","DOI":"10.2196\/48039","volume":"9","author":"JA Flores-Cohaila","year":"2023","unstructured":"Flores-Cohaila JA, Garc\u00eda-Vicente A, Vizcarra-Jim\u00e9nez SF, De la Cruz-Gal\u00e1n JP, Guti\u00e9rrez-Arratia JD, Quiroga Torres BG, Taype-Rondan A. Performance of ChatGPT on the Peruvian national licensing medical examination: cross-sectional study. JMIR Med Educ. 2023;9:e48039. https:\/\/doi.org\/10.2196\/48039.","journal-title":"JMIR Med Educ"},{"issue":"3","key":"7148_CR10","doi-asserted-by":"publisher","first-page":"442","DOI":"10.5603\/cj.97517","volume":"31","author":"S W\u00f3jcik","year":"2024","unstructured":"W\u00f3jcik S, Rulkiewicz A, Pruszczyk P, Lisik W, Pobo\u017cy M, Domienik-Kar\u0142owicz J. Reshaping medical education: performance of ChatGPT on a PES medical examination. Cardiol J. 2024;31(3):442\u201350. https:\/\/doi.org\/10.5603\/cj.97517.","journal-title":"Cardiol J"},{"key":"7148_CR11","doi-asserted-by":"publisher","first-page":"1265024","DOI":"10.3389\/fonc.2023.1265024","volume":"13","author":"Y Huang","year":"2023","unstructured":"Huang Y, Gomaa A, Semrau S, Haderlein M, Lettmaier S, Weissmann T, Grigo J, Tkhayat HB, Frey B, Gaipl U, Distel L, Maier A, Fietkau R, Bert C, Putz F. Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology. Front Oncol. 2023;13:1265024. https:\/\/doi.org\/10.3389\/fonc.2023.1265024.","journal-title":"Front Oncol"},{"key":"7148_CR12","unstructured":"Gemini 1.5 Pro is a largest and most capable Google AI Studio. URL: https:\/\/googlegemini.co\/. Accessed 2024\u201304\u201310."},{"key":"7148_CR13","unstructured":"Command R+ is Cohere's newest large language model. URL: https:\/\/docs.cohere.com\/docs\/command-r-plus. Accessed 2024\u201304\u201310."},{"key":"7148_CR14","unstructured":"Meta AI. (n.d.). LLaMA 3: open and efficient foundation language model. Retrieved from https:\/\/llama.meta.com\/llama3\/. Accessed 2024\u201304\u201310."},{"key":"7148_CR15","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1007\/s12672-024-01415-0","volume":"15","author":"M Ko\u00e7ak","year":"2024","unstructured":"Ko\u00e7ak M, Ak\u00e7al\u0131 Z. Development trends and knowledge framework of artificial intelligence (AI) applications in oncology by years: a bibliometric analysis from 1992 to 2022. Discov Onc. 2024;15:566. https:\/\/doi.org\/10.1007\/s12672-024-01415-0.","journal-title":"Discov Onc."},{"key":"7148_CR16","unstructured":"Student Selection and Placement Centre. Turkish medical specialty training entrance exam questions. 2021. URL: https:\/\/www.osym.gov.tr\/TR,15072\/tus-cikmis-sorular.html."},{"issue":"1","key":"7148_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.21031\/epod.998592","volume":"13","author":"DB Kalayc\u0131o\u011flu","year":"2022","unstructured":"Kalayc\u0131o\u011flu DB. Gender-based differential item functioning analysis of the medical specialization education entrance examination. J Meas Eval Educ Psychol. 2022;13(1):1\u201314. https:\/\/doi.org\/10.21031\/epod.998592.","journal-title":"J Meas Eval Educ Psychol"},{"issue":"64","key":"7148_CR18","doi-asserted-by":"publisher","first-page":"42","DOI":"10.25282\/ted.1059067","volume":"21","author":"S Aydo\u011fan","year":"2022","unstructured":"Aydo\u011fan S, Keskin A. How much does the medical specialization exam measure basic medicine? T\u0131p E\u011fitimi D\u00fcnyas\u0131. 2022;21(64):42\u20139. https:\/\/doi.org\/10.25282\/ted.1059067.","journal-title":"T\u0131p E\u011fitimi D\u00fcnyas\u0131"},{"issue":"4","key":"7148_CR19","first-page":"2001","volume":"11","author":"R Ozen Kutanis","year":"2011","unstructured":"Ozen Kutanis R, Tunc T, Tun\u00e7 M. How do medical specialty training educators and trainees perceive medical specialty selection examination. Kuram Ve Uygulamada Egitim Bilimleri. 2011;11(4):2001\u20134.","journal-title":"Kuram Ve Uygulamada Egitim Bilimleri"},{"key":"7148_CR20","doi-asserted-by":"publisher","unstructured":"Kilic ME. AI in medical education: a comparative analysis of GPT-4 and GPT-3.5 on Turkish medical specialization exam performance. medRxiv. 2023.\u00a0https:\/\/doi.org\/10.1101\/2023.07.12.23292564.","DOI":"10.1101\/2023.07.12.23292564"},{"key":"7148_CR21","doi-asserted-by":"publisher","unstructured":"Beaulieu-Jones BR, Shah S, Berrigan MT, Marwaha JS, Lai SL, Brat GA. Evaluating capabilities of large language models: performance of GPT4 on American Board of Surgery Qualifying Exam Question Banks. medRxiv. 2023.\u00a0https:\/\/doi.org\/10.1101\/2023.07.16.23292743.","DOI":"10.1101\/2023.07.16.23292743"},{"key":"7148_CR22","doi-asserted-by":"publisher","unstructured":"Davies NP, Wilson R, Winder MS, Tunster SJ, McVicar K, Thakrar S, Williams J, Reid A. ChatGPT sits the DFPH exam: large language model performance and potential to support public health learning. medRxiv. 2023.\u00a0https:\/\/doi.org\/10.1101\/2023.07.04.23291894.","DOI":"10.1101\/2023.07.04.23291894"}],"container-title":["BMC Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12909-025-07148-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12909-025-07148-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12909-025-07148-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T03:22:37Z","timestamp":1745551357000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmededuc.biomedcentral.com\/articles\/10.1186\/s12909-025-07148-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,25]]},"references-count":22,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["7148"],"URL":"https:\/\/doi.org\/10.1186\/s12909-025-07148-0","relation":{},"ISSN":["1472-6920"],"issn-type":[{"value":"1472-6920","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,25]]},"assertion":[{"value":"13 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 April 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 April 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All methods were carried out in accordance with relevant guidelines and regulations. Informed consent to participate was not obtained from all of the participants in the study. Need for consent to participate was waived by an Institutional Review Board (IRB).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"609"}}