{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T05:30:39Z","timestamp":1788931839436,"version":"build-2803163510"},"reference-count":56,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T00:00:00Z","timestamp":1784592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Generative Artificial Intelligence (GAI) models, such as GPT-4, have been extensively studied for their integration into medical practice and education. GPT-4 has demonstrated excellent performance on medical licensing examinations, including the United Kingdom Medical Licensing Assessment (UKMLA). However, the field lacks longitudinal data on whether such performance is stable or varies over time. Theoretically, iterative improvements updated by the vendor should enhance performance, but empirical evidence of such longitudinal trends remains limited. Given that GPT-4 undergoes periodic vendor-side updates, we aimed to analyse the categorical, time-spaced performance of GPT-4 on the UKMLA to characterise how its performance changes over time in a medical context.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Two publicly available UKMLA papers were fed into GPT-4 at two different time points, June 2023 and December 2024. 191 questions were provided with and without multiple-choice options to assess GPT-4\u2019s clinical competence. McNemar\u2019s test was performed to evaluate changes in GPT-4\u2019s performance over time, comparing domain-specific questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      GPT-4\u2019s accuracy improved noticeably between the two rounds (MCQ: 88.0 to 93.7%,\n                      <jats:italic>p<\/jats:italic>\n                      \u202f=\u202f0.027; non-MCQ: 68.1 to 81.7%,\n                      <jats:italic>p<\/jats:italic>\n                      \u202f=\u202f1.00). Single-step accuracy rose from 73.1 to 82.3%, and multi-step from 57.4 to 80.3% without MCQ. GPT-4 showed improved accuracy from Round 1 to Round 2 for both single-step and multi-step questions, with MCQ-prompted responses consistently outperforming non-MCQ responses (up to 95.1% accuracy for multi-step MCQ questions in Round 2). GPT-4\u2019s performance improved across all question categories from round one to round two, most notably in management questions without MCQ options (+23.30%), though these differences were not statistically significant.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion and conclusion<\/jats:title>\n                    <jats:p>GPT-4\u2019s performance on the UKMLA improved significantly over 18\u202fmonths, suggesting that iterative vendor-side model updates enhance clinical reasoning capabilities. These findings indicate that GPT-4 may serve as a supplementary educational tool for medical students and clinicians; however, the underlying drivers of performance changes remain opaque, and such tools should be deployed with structured oversight to prevent overreliance.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1888765","type":"journal-article","created":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T11:10:24Z","timestamp":1784632224000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["The temporal changes in GPT-4 performance on UKMLA practice questions: educational and clinical implications"],"prefix":"10.3389","volume":"13","author":[{"given":"Ravanth","family":"Baskaran","sequence":"first","affiliation":[{"name":"University Hospital Southampton NHS Trust","place":["Southampton, United Kingdom"]},{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sai","family":"Sirikonda","sequence":"additional","affiliation":[{"name":"University Hospitals of North Midlands NHS Trust","place":["Stoke-on-Trent, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditya","family":"Singh","sequence":"additional","affiliation":[{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sripradha","family":"Srinivasan","sequence":"additional","affiliation":[{"name":"University Hospital Southampton NHS Trust","place":["Southampton, United Kingdom"]},{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Becky","family":"Leveridge","sequence":"additional","affiliation":[{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]},{"name":"Division of Gastrointestinal and Hepatobiliary Surgery, Imperial College Healthcare NHS Trust","place":["London, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Octavi","family":"Casals-Farre","sequence":"additional","affiliation":[{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]},{"name":"Hawke\u2019s Bay Hospital","place":["Hastings, New Zealand"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harmeena","family":"Kaur","sequence":"additional","affiliation":[{"name":"Department of General Medicine, Doncaster Royal Infirmary","place":["Doncaster, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susruta","family":"Manivannan","sequence":"additional","affiliation":[{"name":"University Hospital Southampton NHS Trust","place":["Southampton, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kabilan","family":"Elangovan","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre","place":["Singapore, Singapore"]},{"name":"Artificial Intelligence Office, Singapore Health Services","place":["Singapore, Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chrystie Wan Ning","family":"Quek","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre","place":["Singapore, Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kanae","family":"Fukutsu","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre","place":["Singapore, Singapore"]},{"name":"Artificial Intelligence Office, Singapore Health Services","place":["Singapore, Singapore"]},{"name":"Department of Ophthalmology, Faculty of Medicine and Graduate School of Medicine, Hokkaido University","place":["Sapporo, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith","family":"Cave","sequence":"additional","affiliation":[{"name":"University Hospital Southampton NHS Trust","place":["Southampton, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bhaskar Kumar","family":"Somani","sequence":"additional","affiliation":[{"name":"University Hospital Southampton NHS Trust","place":["Southampton, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel Shu Wei","family":"Ting","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre","place":["Singapore, Singapore"]},{"name":"Artificial Intelligence Office, Singapore Health Services","place":["Singapore, Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Athanasios","family":"Hassoulas","sequence":"additional","affiliation":[{"name":"Centre for Medical Education (C4ME), School of Medicine, Cardiff University","place":["Cardiff, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,21]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1093\/mind\/LIX.236.433","article-title":"Computing machinery and intelligence","author":"Turing","year":"1950","journal-title":"Mind"},{"key":"ref2","author":"Vaswani","year":"2017"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"1099","DOI":"10.1038\/s41587-022-01618-2","article-title":"Large language models generate functional protein sequences across diverse families","volume":"41","author":"Madani","year":"2023","journal-title":"Nat Biotechnol"},{"key":"ref4","author":"Li","year":"2020"},{"key":"ref5","author":"Brown","year":"2020"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1007\/s12599-023-00795-x","article-title":"Welcome to the era of ChatGPT et al","volume":"65","author":"Teubner","year":"2023","journal-title":"Bus Inf Syst Eng"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"GPT-4 technical report","author":"Achiam","year":"2024","journal-title":"arXiv"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.15556","article-title":"Training compute-optimal large language models","author":"Hoffmann","year":"2022","journal-title":"arXiv"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s13000-021-01085-4","article-title":"Artificial intelligence (AI) in medicine, current applications and future role with special emphasis on its potential and promise in pathology: present and future impact, obstacles including costs and acceptance among pathologists, practical and philosophical considerations. A comprehensive review","volume":"16","author":"Ahmad","year":"2021","journal-title":"Diagn Pathol"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1177\/15533506241248239","article-title":"The role of artificial intelligence in medical education: a systematic review","volume":"31","author":"Tozsin","year":"2024","journal-title":"Surg Innov"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"e13930","DOI":"10.2196\/13930","article-title":"Applications and challenges of implementing artificial intelligence in medical education: integrative review","volume":"5","author":"Chan","year":"2019","journal-title":"JMIR Med Educ"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLoS Digit Health"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1080\/07380569.2023.2256710","article-title":"ChatGPT and generative AI technology: a mixed bag of concerns and new opportunities","volume":"41","author":"Lambert","year":"2024","journal-title":"Comput Sch"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1016\/j.jbi.2015.06.004","article-title":"Using natural language processing to provide personalized learning opportunities from trainee clinical notes","volume":"56","author":"Denny","year":"2015","journal-title":"J Biomed Inform"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"e2336100","DOI":"10.1001\/jamanetworkopen.2023.36100","article-title":"Generative artificial intelligence for chest radiograph interpretation in the emergency department","volume":"6","author":"Huang","year":"2023","journal-title":"JAMA Netw Open"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1007\/s10439-023-03329-4","article-title":"ChatGPT and clinical decision support: scope, application, and limitations","volume":"52","author":"Ferdush","year":"2024","journal-title":"Ann Biomed Eng"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"1653369","DOI":"10.3389\/fdgth.2025.1653369","article-title":"Generative AI in clinical (2020\u20132025): a mini-review of applications, emerging trends, and clinical challenges","volume":"7","author":"Fahad","year":"2025","journal-title":"Front Digit Health"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"e20701","DOI":"10.2196\/20701","article-title":"Artificial intelligence-based conversational agents for chronic conditions: systematic literature review","volume":"22","author":"Schachner","year":"2020","journal-title":"J Med Internet Res"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"1278","DOI":"10.1057\/s41599-024-03811-x","article-title":"AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content","volume":"11","author":"Sun","year":"2024","journal-title":"Humanit Soc Sci Commun"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"6","DOI":"10.3352\/jeehp.2024.21.6","article-title":"Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review","volume":"21","author":"Xu","year":"2024","journal-title":"J Educ Eval Health Prof"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"00031348251390958","DOI":"10.1177\/00031348251390958","article-title":"Advances in large language model performance: a comparative study of ChatGPT-4 and ChatGPT-5 on ABSITE questions","author":"Sanli","year":"2025","journal-title":"Am Surg"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"86","DOI":"10.3390\/informatics12030086","article-title":"The temporal evolution of large language model performance: a comparative analysis of past and current outputs in scientific and medical research","volume":"12","author":"Seth","year":"2025","journal-title":"Informatics (Basel)"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"e0312771","DOI":"10.1371\/journal.pone.0312771","article-title":"Performance of ChatGPT in medical licensing examinations in countries worldwide: a systematic review and meta-analysis protocol","volume":"19","author":"Liu","year":"2024","journal-title":"PLoS One"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"e58375","DOI":"10.2196\/58375","article-title":"Performance of plug-in augmented ChatGPT and its ability to quantify uncertainty: simulation study on the German medical board examination","volume":"11","author":"Madrid","year":"2025","journal-title":"JMIR Med Educ"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"13031","DOI":"10.1038\/s41598-025-97327-2","article-title":"Assessing ChatGPT 4.0\u2019s capabilities in the United Kingdom medical licensing examination (UKMLA): a robust categorical analysis","volume":"15","author":"Casals-Farre","year":"2025","journal-title":"Sci Rep"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"10","DOI":"10.3389\/fmed.2023.1240915","article-title":"Evaluating the performance of ChatGPT-4 on the United Kingdom medical licensing assessment","volume":"10","author":"Lai","year":"2023","journal-title":"Front Med"},{"key":"ref28","year":""},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang","year":"2022","journal-title":"arXiv"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.11903","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei","year":"2023","journal-title":"arXiv"},{"key":"ref31","year":""},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.09009","article-title":"How is ChatGPT\u2019s behavior changing over time?","author":"Chen","year":"2023","journal-title":"arXiv"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1111\/j.1365-2923.1979.tb01511.x","article-title":"A comparison of multiple-choice tests and free-response tests in examinations of clinical competence","volume":"13","author":"Newble","year":"1979","journal-title":"Med Educ"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1207\/s15326985ep3404_2","article-title":"Cognition and the question of test item format","volume":"34","author":"Martinez","year":"1999","journal-title":"Educ Psychol"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1002\/ase.1739","article-title":"Analysis of testing with multiple choice versus open-ended questions: outcome-based observations in an anatomy course","volume":"11","author":"Melovitz Vasan","year":"2018","journal-title":"Anat Sci Educ"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1037\/a0032918","article-title":"Comparing comprehension measured by multiple-choice and open-ended questions","volume":"67","author":"Ozuru","year":"2013","journal-title":"Canad J Exp Psychol"},{"key":"ref37","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1007\/978-3-030-49186-4_31","volume-title":"Artificial Intelligence Applications and Innovations","author":"Adamopoulou","year":"2020"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1097\/PAP.0000000000000406","article-title":"Application of ChatGPT in routine diagnostic pathology: promises, pitfalls, and potential future directions","volume":"31","author":"Schukow","year":"2024","journal-title":"Adv Anat Pathol"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1046\/j.1525-1497.1999.00323.x","article-title":"Evidence-based medicine and the practicing clinician","volume":"14","author":"McAlister","year":"1999","journal-title":"J Gen Intern Med"},{"key":"ref40","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1093\/oso\/9780197759066.001.0001","volume-title":"Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking","author":"Vallor","year":"2024"},{"key":"ref41","doi-asserted-by":"publisher","first-page":"47","DOI":"10.2139\/ssrn.3423704","article-title":"The mirror for (artificial) intelligence: in whose reflection?","volume":"41","author":"Moore","year":"2019","journal-title":"Comp Lab L Pol J"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.43313","article-title":"Artificial hallucinations by Google bard: think before you leap","volume":"15","author":"Kumar","year":"2023","journal-title":"Cureus"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"57","DOI":"10.47743\/saeb-2020-0025","article-title":"Automated vs manual content analysis \u2013 a retrospective look","volume":"67","author":"Anastasiei","year":"2020","journal-title":"Sci Ann Econ Bus"},{"key":"ref44","doi-asserted-by":"publisher","first-page":"403","DOI":"10.2217\/pme-2018-0037","article-title":"Enhancing diversity to reduce health information disparities and build an evidence base foR genomic medicine","volume":"15","author":"Hindorff","year":"2018","journal-title":"Pers Med"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1046\/j.1365-2753.1997.00092.x","article-title":"A Popperian perspective of the term \u2018evidence-based medicine\u2019","volume":"3","author":"Shahar","year":"1997","journal-title":"J Eval Clin Pract"},{"key":"ref46","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/978-0-387-85349-9_5","volume-title":"The Coming Robot Revolution: Expectations and Fears about Emerging Intelligent, Humanlike Machines","author":"Bar-Cohen","year":"2009"},{"key":"ref47","year":""},{"key":"ref48","doi-asserted-by":"publisher","first-page":"100900","DOI":"10.1016\/j.landig.2025.100900","article-title":"How can artificial intelligence transform the training of medical students and physicians?","volume":"7","author":"Ning","year":"2025","journal-title":"Lancet Digit Health"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"1901","DOI":"10.1039\/D5DD00073D","article-title":"PAL \u2013 parallel active learning for machine-learned potentials","volume":"4","author":"Zhou","year":"2025","journal-title":"Digital Disc"},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.64898\/2026.05.28.26353935","article-title":"Performance evaluation and benchmarking across 16 large language models on a comprehensive real-world emergency department triage data set","author":"Benning","year":"2026","journal-title":"medRxiv"},{"key":"ref51","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1016\/j.bjps.2025.03.053","article-title":"Evidence-based artificial intelligence: implementing retrieval-augmented generation models to enhance clinical decision support in plastic surgery","volume":"104","author":"Ozmen","year":"2025","journal-title":"J Plast Reconstr Aesthet Surg"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.10997","article-title":"Retrieval-augmented generation for large language models: a survey","author":"Gao","year":"2024","journal-title":"arXiv"},{"key":"ref53","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1093\/jamia\/ocaf008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","volume":"32","author":"Liu","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"ref54","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1038\/s41746-025-01519-z","article-title":"Retrieval augmented generation for large language models and its generalizability in assessing medical fitness","volume":"8","author":"Ke","year":"2025","journal-title":"NPJ Digit Med"},{"key":"ref55","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1053\/j.semdp.2023.02.006","article-title":"AI in pathology: what could possibly go wrong?","volume":"40","author":"Nakagawa","year":"2023","journal-title":"Semin Diagn Pathol"},{"key":"ref56","doi-asserted-by":"publisher","first-page":"2255","DOI":"10.3390\/electronics13122255","article-title":"Improving training dataset balance with ChatGPT prompt engineering","volume":"13","author":"Kochanek","year":"2024","journal-title":"Electronics (Basel)"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1888765\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T11:10:27Z","timestamp":1784632227000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1888765\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,21]]},"references-count":56,"alternative-id":["10.3389\/fmed.2026.1888765"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1888765","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,21]]},"article-number":"1888765"}}