{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T02:56:05Z","timestamp":1788317765613,"version":"build-2803163510"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n In recent years, generative artificial intelligence and large language models (LLMs) have rapidly advanced, offering significant potential to transform medical education. Several studies have evaluated the performance of chatbots on multiple-choice medical examinations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n \n The study aims to assess the performance of two LLMs\u2014GPT-4o and OpenAI o1\u2014on the\n M\u00e9dico Interno Residente<\/jats:italic>\n (MIR) 2024 examination, the Spanish national medical test that determines eligibility for competitive medical specialist training positions.\n <\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A total of 176 questions from the MIR 2024 examination were analyzed. Each question was presented individually to the chatbots to ensure independence and prevent memory retention bias. No additional prompts were introduced to minimize potential bias. For each LLM, response consistency under verification prompting was assessed by systematically asking, \u201cAre you sure?\u201d after each response. Accuracy was defined as the percentage of correct responses compared to the official answers provided by the Spanish Ministry of Health. It was assessed for GPT-4o, OpenAI o1, and, as a benchmark, for a consensus of medical specialists and for the average MIR candidate. Subanalyses included performance across different medical subjects, question difficulty (quintiles based on the percentage of examinees correctly answering each question), and question types (clinical cases vs theoretical questions; positive vs negative questions).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Overall accuracy was 89.8% (158\/176) for GPT-4o and 90% (160\/176) after verification prompting, 92.6% (163\/176) for OpenAI o1 and 93.2% (164\/176) after verification prompting, 94.3% (166\/176) for the consensus of medical specialists, and 56.6% (100\/176) for the average MIR candidate. Both LLMs and the consensus of medical specialists outperformed the average MIR candidate across all 20 medical subjects analyzed, with \u226580% LLMs\u2019 accuracy in most domains. A performance gradient was observed: LLMs\u2019 accuracy gradually declined as question difficulty increased. Slightly higher accuracy was observed for clinical cases compared to theoretical questions, as well as for positive questions compared to negative ones. Both models demonstrated high response consistency, with near-perfect agreement between initial responses and those after the verification prompting.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n These findings highlight the excellent performance of GPT-4o and OpenAI o1 on the MIR 2024 examination, demonstrating consistent accuracy across medical subjects and question types. The integration of LLMs into medical education presents promising opportunities and is likely to reshape how students prepare for licensing examinations and change our understanding of medical education. Further research should explore how the wording, language, prompting techniques, and image-based questions can influence LLMs\u2019 accuracy, as well as evaluate the performance of emerging artificial intelligence models in similar assessments.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/75452","type":"journal-article","created":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T21:51:02Z","timestamp":1768254662000},"page":"e75452-e75452","source":"Crossref","is-referenced-by-count":5,"title":["GPT-4o and OpenAI o1 Performance on the 2024 Spanish Competitive Medical Specialty Access Examination: Cross-Sectional Quantitative Evaluation Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2480-9133","authenticated-orcid":false,"given":"Pau","family":"Benito","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2843-7110","authenticated-orcid":false,"given":"Mikel","family":"Isla-Jover","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0077-126X","authenticated-orcid":false,"given":"Pablo","family":"Gonz\u00e1lez-Castro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7335-777X","authenticated-orcid":false,"given":"Pedro Jos\u00e9","family":"Fern\u00e1ndez Esparcia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8772-4565","authenticated-orcid":false,"given":"Manuel","family":"Carpio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-4725-7300","authenticated-orcid":false,"given":"Iv\u00e1n","family":"Blay-Sim\u00f3n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4583-562X","authenticated-orcid":false,"given":"Pablo","family":"Guti\u00e9rrez-Bedia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4480-0416","authenticated-orcid":false,"given":"Maria J","family":"Lapastora","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1751-0048","authenticated-orcid":false,"given":"Beatriz","family":"Carratal\u00e1","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7568-7140","authenticated-orcid":false,"given":"Carlos","family":"Carazo-Casas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,1,12]]},"reference":[{"key":"R1","unstructured":"Zhao WX Zhou K Li J Tang T Wang X Hou Y et al. A survey of large language models. arXiv. Preprint posted online on Mar 31, 2023. [doi: 10.48550\/arXiv.2303.18223]"},{"key":"R2","unstructured":"Radford A Narasimhan K Salimans T Sutskever I . Improving language understanding by generative pre-training. OpenAI; 2018. URL: https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf [Accessed 10-12-2025]"},{"key":"R3","unstructured":"OpenAIAchiam J Adler S Agarwal S Ahmad L Akkaya I et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 15, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R4","unstructured":"OpenAIJaech A Kalai A Lerer A Richardson A El-Kishky A et al. OpenAI o1 system card. arXiv. Preprint posted online on Dec 21, 2024. [doi: 10.48550\/arXiv.2412.16720]"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1145\/3771933","article-title":"Do advanced language models eliminate the need for prompt engineering in software engineering?","author":"Wang","journal-title":"ACM Trans Softw Eng Methodol"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"41","DOI":"10.3390\/info13010041","article-title":"A literature survey of recent advances in chatbots","volume":"13","author":"Caldarini","journal-title":"Information"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.79878","article-title":"Applications of artificial intelligence in medical education: a systematic review","volume":"17","author":"Hallquist","journal-title":"Cureus"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-63997-7","article-title":"In-depth analysis of ChatGPT\u2019s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions","volume":"14","author":"Knoedler","journal-title":"Sci Rep"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/51148","article-title":"Pure wisdom or Potemkin villages? A comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE step 3 style questions: quantitative analysis","volume":"10","author":"Knoedler","journal-title":"JMIR Med Educ"},{"key":"R11","unstructured":"Li Z Shi Y Liu Z Yang F Liu N Du M . Quantifying multilingual performance of large language models across languages. arXiv. Preprint posted online on Apr 17, 2024. [doi: 10.48550\/arXiv.2404.11553]"},{"key":"R12","unstructured":"Gao Y Xiong Y Gao X Jia K Pan J Bi Y et al. Retrieval-augmented generation for large language models: a survey. arXiv. Preprint posted online on Dec 18, 2023. [doi: 10.48550\/arXiv.2312.10997]"},{"issue":"6","key":"R13","doi-asserted-by":"publisher","first-page":"1460","DOI":"10.3390\/clinpract13060130","article-title":"Evaluating the efficacy of ChatGPT in navigating the Spanish medical residency entrance examination (MIR): promising horizons for AI in clinical medicine","volume":"13","author":"Guillen-Grima","journal-title":"Clin Pract"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/48039","article-title":"Performance of ChatGPT on the Peruvian national licensing medical examination: cross-sectional study","volume":"9","author":"Flores-Cohaila","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-08084-9","article-title":"Evaluating GPT-4o in high-stakes medical assessments: performance and error analysis on a Chilean anesthesiology exam","volume":"25","author":"Altermatt","journal-title":"BMC Med Educ"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-49483-6","article-title":"Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training","volume":"13","author":"Madrid-Garc\u00eda","journal-title":"Sci Rep"},{"key":"R17","unstructured":"Order SND\/888\/2024 of 14 august, approving the number of available positions and announcing the 2024 competitive selection examinations for access in 2025 to specialized health training positions for university degree holders in medicine, pharmacy, nursing, and in the fields of psychology, chemistry, biology, and physics [Article in Spanish]. Bolet\u00edn Oficial del Estado; 2024. URL: https:\/\/www.boe.es\/boe\/dias\/2024\/08\/23\/pdfs\/BOE-A-2024-17246.pdf [Accessed 07-01-2025]"},{"key":"R18","unstructured":"The Ministry of Health publishes the final list of results of the specialized health training examinations [Website in Spanish]. Ministry of Health, Spain. URL: https:\/\/www.sanidad.gob.es\/gabinete\/notasPrensa.do?id=6632 [Accessed 12-03-2025]"},{"key":"R19","unstructured":"Consulting previous examination booklets\u2014search by examination session [Website in Spanish]. Ministry of Health, Spain. URL: https:\/\/fse.sanidad.gob.es\/fseweb\/#\/principal\/datosAnteriores\/cuadernosExamen [Accessed 26-01-2025]"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/48002","article-title":"Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study","volume":"9","author":"Takagi","journal-title":"JMIR Med Educ"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/50965","article-title":"Comparison of the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination: observational study","volume":"10","author":"Meyer","journal-title":"JMIR Med Educ"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/65108","article-title":"ChatGPT\u2019s performance on Portuguese medical examination questions: comparative analysis of ChatGPT-3.5 Turbo and ChatGPT-4o Mini","volume":"11","author":"Prazeres","journal-title":"JMIR Med Educ"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000205","article-title":"ChatGPT passing USMLE shines a spotlight on the flaws of medical education","volume":"2","author":"Mbakwe","journal-title":"PLOS Digit Health"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105673","article-title":"Evaluating the effectiveness of advanced large language models in medical knowledge: a comparative study using the Japanese national medical examination","volume":"193","author":"Liu","journal-title":"Int J Med Inform"},{"issue":"8","key":"R25","doi-asserted-by":"publisher","first-page":"1304","DOI":"10.1016\/j.dld.2024.02.019","article-title":"Charting new AI education in gastroenterology: cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam","volume":"56","author":"Gravina","journal-title":"Dig Liver Dis"},{"key":"R26","unstructured":"OpenAI GPT-4.5 system card. OpenAI; 2025. URL: https:\/\/cdn.openai.com\/gpt-4-5-system-card-2272025.pdf [Accessed 06-01-2026]"},{"key":"R27","unstructured":"DeepSeek-AIGuo D Yang D Zhang H Song J Zhang R et al. DeepSeek-R1: incentivizing reasoning capability in llms via reinforcement learning. arXiv. Preprint posted online on Jan 22, 2025. [doi: 10.48550\/arXiv.2501.12948]"},{"key":"R28","unstructured":"QwenYang A Yang B Zhang B et al. Qwen2.5 technical report. arXiv. Preprint posted online on Dec 19, 2024. [doi: 10.48550\/arXiv.2412.15115]"},{"key":"R29","unstructured":"Grattafiori A Dubey A et al. The llama 3 herd of models. arXiv. Preprint posted online on Jul 31, 2024. [doi: 10.48550\/arXiv.2407.21783]"},{"key":"R30","unstructured":"Claude 3.7 sonnet system card. Anthropic; URL: https:\/\/assets.anthropic.com\/m\/785e231869ea8b3b\/original\/claude-3-7-sonnet-system-card.pdf [Accessed 07-01-2026]"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e75452","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:06:29Z","timestamp":1782842789000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e75452"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,12]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/75452","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,12]]},"article-number":"v12i1e75452"}}