{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T02:44:34Z","timestamp":1768877074389,"version":"3.49.0"},"reference-count":44,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec sec-type=\"background\">\n            <jats:title>Background<\/jats:title>\n            <jats:p>Advancements in ChatGPT are transforming medical education by providing new tools for assessment and learning, potentially enhancing evaluations for doctors and improving instructional effectiveness.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec sec-type=\"objective\">\n            <jats:title>Objective<\/jats:title>\n            <jats:p>This study evaluates the performance and consistency of ChatGPT-3.5 Turbo and ChatGPT-4o mini in solving European Portuguese medical examination questions (2023 National Examination for Access to Specialized Training; Prova Nacional de Acesso \u00e0 Forma\u00e7\u00e3o Especializada [PNA]) and compares their performance to human candidates.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec sec-type=\"methods\">\n            <jats:title>Methods<\/jats:title>\n            <jats:p>ChatGPT-3.5 Turbo was tested on the first part of the examination (74 questions) on July 18, 2024, and ChatGPT-4o mini on the second part (74 questions) on July 19, 2024. Each model generated an answer using its natural language processing capabilities. To test consistency, each model was asked, \u201cAre you sure?\u201d after providing an answer. Differences between the first and second responses of each model were analyzed using the McNemar test with continuity correction. A single-parameter <jats:italic>t<\/jats:italic> test compared the models\u2019 performance to human candidates. Frequencies and percentages were used for categorical variables, and means and CIs for numerical variables. Statistical significance was set at <jats:italic>P<\/jats:italic>&lt;.05.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec sec-type=\"results\">\n            <jats:title>Results<\/jats:title>\n            <jats:p>ChatGPT-4o mini achieved an accuracy rate of 65% (48\/74) on the 2023 PNA examination, surpassing ChatGPT-3.5 Turbo. ChatGPT-4o mini outperformed medical candidates, while ChatGPT-3.5 Turbo had a more moderate performance.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec sec-type=\"conclusions\">\n            <jats:title>Conclusions<\/jats:title>\n            <jats:p>This study highlights the advancements and potential of ChatGPT models in medical education, emphasizing the need for careful implementation with teacher oversight and further research.<\/jats:p>\n          <\/jats:sec>","DOI":"10.2196\/65108","type":"journal-article","created":{"date-parts":[[2025,3,5]],"date-time":"2025-03-05T20:47:26Z","timestamp":1741207646000},"page":"e65108-e65108","source":"Crossref","is-referenced-by-count":7,"title":["ChatGPT\u2019s Performance on Portuguese Medical Examination Questions: Comparative Analysis of ChatGPT-3.5 Turbo and ChatGPT-4o Mini"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2849-5194","authenticated-orcid":false,"given":"Filipe","family":"Prazeres","sequence":"first","affiliation":[]}],"member":"1010","published-online":{"date-parts":[[2025,3,5]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1007\/s12599-023-00834-7","article-title":"Generative AI","volume":"66","author":"Feuerriegel","journal-title":"Bus Inf Syst Eng"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.14445\/23488387\/IJCSE-V10I6P101","article-title":"The impact, advancements and applications of generative AI","volume":"10","author":"Ramdurai","journal-title":"Int J Comput Sci Eng"},{"key":"R3","unstructured":"Cao Y Li S Liu Y et al. A comprehensive survey of AI-generated content (AIGC): a history of generative AI from GAN to ChatGPT. arXiv. Preprint posted online onMar7, 2023. [doi: 10.48550\/arXiv.2303.04226]"},{"key":"R4","unstructured":"Introducing ChatGPT. OpenAI. Nov30, 2022. URL: https:\/\/openai.com\/index\/chatgpt [Accessed 30-07-2024]"},{"key":"R5","unstructured":"Conte N . Ranked: the most popular AI tools. Visual Capitalist. Jan24, 2024. URL: https:\/\/www.visualcapitalist.com\/ranked-the-most-popular-ai-tools [Accessed 19-02-2025]"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.lanwpc.2023.100905","article-title":"ChatGPT: promise and challenges for deployment in low- and middle-income countries","volume":"41","author":"Wang","journal-title":"Lancet Reg Health West Pac"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"118","DOI":"10.21541\/apjess.1293702","article-title":"Is ChatGPT leading generative AI? What is beyond expectations?","volume":"11","author":"Aydin","journal-title":"Acad Platform J Eng Smart Sys"},{"issue":"3","key":"R8","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1177\/1478077120934126","article-title":"Generative systems in the architecture, engineering and construction industry: a systematic review and analysis","volume":"19","author":"BuHamdan","journal-title":"Int J Archit Comput"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"100333","DOI":"10.1016\/j.jik.2023.100333","article-title":"A systematic literature review of artificial intelligence in the healthcare sector: benefits, challenges, methodologies, and functionalities","volume":"8","author":"Ali","journal-title":"J Innov Knowl"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics14010109","article-title":"A systematic review and meta-analysis of artificial intelligence tools in medicine and healthcare: applications, considerations, limitations, motivation and challenges","volume":"14","author":"Younis","journal-title":"Diagnostics (Basel)"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2147\/JMDH.S459946","article-title":"A systematic review of the application of artificial intelligence in nursing care: where are we, and what\u2019s next?","volume":"17","author":"Ruksakulpiwat","journal-title":"J Multidiscip Healthc"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231189331","article-title":"Application of artificial intelligence in medical technologies: a systematic review of main trends","volume":"9","author":"Bitkina","journal-title":"Digit Health"},{"issue":"3","key":"R13","doi-asserted-by":"publisher","first-page":"278","DOI":"10.7861\/clinmed.2023-0078","article-title":"Early applications of ChatGPT in medical practice, education and research","volume":"23","author":"Sedaghat","journal-title":"Clin Med (Lond)"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.34172\/jmdc.1232","article-title":"ChatGPT in medical education: how we can use in medical education: challenges and opportunities","volume":"12","author":"Gandomani","journal-title":"J Multidiscip Care"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.33844\/cjm.2024.6032","article-title":"Enhancing medical education with ChatGPT: a promising tool for the future","volume":"6","author":"Sani","journal-title":"Can J Med"},{"issue":"10","key":"R16","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0292216","article-title":"An exploratory survey about using ChatGPT in education, healthcare, and research","volume":"18","author":"Hosseini","journal-title":"PLoS One"},{"issue":"9","key":"R17","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2023.2198094","article-title":"Harnessing the power of ChatGPT in medical education","volume":"45","author":"Guo","journal-title":"Med Teach"},{"issue":"5","key":"R18","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","article-title":"ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports","volume":"34","author":"Jeblick","journal-title":"Eur Radiol"},{"issue":"8","key":"R19","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0290691","article-title":"ChatGPT versus human in generating medical graduate exam multiple choice questions\u2014a multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom)","volume":"18","author":"Cheung","journal-title":"PLoS One"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","journal-title":"JMIR Med Educ"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1111\/1471-0528.17641","article-title":"Performance of ChatGPT in medical examinations: a systematic review and a meta-analysis","volume":"131","author":"Levin","journal-title":"BJOG"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1177\/23821205241238641","article-title":"Can ChatGPT-3.5 pass a medical exam? A systematic review of ChatGPT\u2019s performance in academic testing","volume":"11","author":"Sumbal","journal-title":"J Med Educ Curric Dev"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/52818","article-title":"Appraisal of ChatGPT\u2019s aptitude for medical education: comparative analysis with third-year medical students in a pulmonology examination","volume":"10","author":"Cherif","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-023-01961-0","article-title":"ChatGPT performs on the Chinese National Medical Licensing Examination","volume":"47","author":"Wang","journal-title":"J Med Syst"},{"issue":"6","key":"R26","doi-asserted-by":"publisher","first-page":"1542","DOI":"10.1007\/s10439-023-03338-3","article-title":"Sailing the seven seas: a multinational comparison of ChatGPT\u2019s performance on medical licensing examinations","volume":"52","author":"Alfertshofer","journal-title":"Ann Biomed Eng"},{"key":"R27","unstructured":"Prova Nacional de Acesso \u00e0 Forma\u00e7\u00e3o Especializada 2023 Perguntas Frequentes. ACSS. 2023. URL: https:\/\/www.acss.min-saude.pt\/wp-content\/uploads\/2018\/09\/FAQ_PNA2023.pdf [Accessed 01-08-2024]"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"16492","DOI":"10.1038\/s41598-023-43436-9","article-title":"Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments","volume":"13","author":"Brin","journal-title":"Sci Rep"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNemar","journal-title":"Psychometrika"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/BF02289261","article-title":"Note on the correction for continuity in testing the significance of the difference between correlated proportions","volume":"13","author":"Edwards","journal-title":"Psychometrika"},{"issue":"32","key":"R31","doi-asserted-by":"publisher","DOI":"10.1097\/MD.0000000000034673","article-title":"ChatGPT performance in the medical specialty exam: an observational study","volume":"102","author":"Oztermeli","journal-title":"Medicine (Baltimore)"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2023-100815","article-title":"ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model","volume":"30","author":"Ebrahimian","journal-title":"BMJ Health Care Inform"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"442","DOI":"10.5603\/cj.97517","article-title":"Reshaping medical education: performance of ChatGPT on a PES medical examination","volume":"31","author":"W\u00f3jcik","journal-title":"Cardiol J"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","first-page":"20512","DOI":"10.1038\/s41598-023-46995-z","article-title":"Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination","volume":"13","author":"Roso\u0142","journal-title":"Sci Rep"},{"issue":"6","key":"R35","doi-asserted-by":"publisher","first-page":"1460","DOI":"10.3390\/clinpract13060130","article-title":"Evaluating the efficacy of ChatGPT in navigating the Spanish Medical Residency Entrance Examination (MIR): promising horizons for AI in clinical medicine","volume":"13","author":"Guillen-Grima","journal-title":"Clin Pract"},{"issue":"8","key":"R36","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.42924","article-title":"Assessing the performance of GPT-3.5 and GPT-4 on the 2023 Japanese nursing examination","volume":"15","author":"Kaneda","journal-title":"Cureus"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/48039","article-title":"Performance of ChatGPT on the Peruvian National Licensing Medical Examination: cross-sectional study","volume":"9","author":"Flores-Cohaila","journal-title":"JMIR Med Educ"},{"issue":"5","key":"R38","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1037\/0022-3514.88.5.725","article-title":"Counterfactual thinking and the first instinct fallacy","volume":"88","author":"Kruger","journal-title":"J Pers Soc Psychol"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2024.1252520","article-title":"Unskilled and unaware: second-order judgments increase with miscalibration for low performers","volume":"15","author":"Coutinho","journal-title":"Front Psychol"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.2196\/46482","article-title":"Artificial intelligence in medical education: comparative analysis of ChatGPT, Bing, and medical students in Germany","volume":"9","author":"Roos","journal-title":"JMIR Med Educ"},{"issue":"10","key":"R41","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1136\/bjo-2023-324091","article-title":"Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions","volume":"108","author":"Fowler","journal-title":"Br J Ophthalmol"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.2196\/50514","article-title":"Assessment of resident and AI chatbot performance on the University of Toronto Family Medicine Residency Progress Test: comparative study","volume":"9","author":"Huang","journal-title":"JMIR Med Educ"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.2196\/47274","article-title":"The intersection of ChatGPT, clinical medicine, and medical education","volume":"9","author":"Wong","journal-title":"JMIR Med Educ"},{"issue":"6","key":"R44","doi-asserted-by":"publisher","first-page":"6","DOI":"10.3352\/jeehp.2024.21.6","article-title":"Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review","volume":"21","author":"Xu","journal-title":"J Educ Eval Health Prof"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2025,3,5]],"date-time":"2025-03-05T20:47:28Z","timestamp":1741207648000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e65108"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,5]]},"references-count":44,"URL":"https:\/\/doi.org\/10.2196\/65108","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,5]]},"article-number":"v11i1e65108"}}