{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T01:36:54Z","timestamp":1768700214186,"version":"3.49.0"},"reference-count":13,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T00:00:00Z","timestamp":1715817600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T00:00:00Z","timestamp":1715817600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Discov Artif Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This study evaluates the proficiency of ChatGPT-4 across various medical specialties and assesses its potential as a study tool for medical students preparing for the United States Medical Licensing Examination (USMLE) Step 2 and related clinical subject exams. ChatGPT-4 answered board-level questions with 89% accuracy, but showcased significant discrepancies in performance across specialties. Although it excelled in psychiatry, neurology, and obstetrics and gynecology, it underperformed in pediatrics, emergency medicine, and family medicine. These variations may be potentially attributed to the depth and recency of training data as well as the scope of the specialties assessed. Specialties with significant interdisciplinary overlap had lower performance, suggesting complex clinical scenarios pose a challenge to the AI. In terms of the future, the overall efficacy of ChatGPT-4 indicates a promising supplemental role in medical education, but performance inconsistencies across specialties in the current version lead us to recommend that medical students use AI with caution.<\/jats:p>","DOI":"10.1007\/s44163-024-00135-2","type":"journal-article","created":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T10:01:42Z","timestamp":1715853702000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Evaluating ChatGPT-4 in medical education: an assessment of subject exam performance reveals limitations in clinical curriculum support for students"],"prefix":"10.1007","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7794-3421","authenticated-orcid":false,"given":"Brendan P.","family":"Mackey","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2095-1084","authenticated-orcid":false,"given":"Razmig","family":"Garabet","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6612-5143","authenticated-orcid":false,"given":"Laura","family":"Maule","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9699-7245","authenticated-orcid":false,"given":"Abay","family":"Tadesse","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1347-1740","authenticated-orcid":false,"given":"James","family":"Cross","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5864-535X","authenticated-orcid":false,"given":"Michael","family":"Weingarten","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,5,16]]},"reference":[{"key":"135_CR1","doi-asserted-by":"publisher","first-page":"605","DOI":"10.12669\/pjms.39.2.7653","volume":"39","author":"RA Khan","year":"2023","unstructured":"Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT\u2014reshaping medical education and clinical management. Pak J Med Sci. 2023;39:605\u20137. https:\/\/doi.org\/10.12669\/pjms.39.2.7653.","journal-title":"Pak J Med Sci."},{"key":"135_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iotcps.2023.04.003","author":"PP Ray","year":"2023","unstructured":"Ray PP. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Phys Syst. 2023. https:\/\/doi.org\/10.1016\/j.iotcps.2023.04.003.","journal-title":"Internet Things Phys Syst"},{"key":"135_CR3","doi-asserted-by":"publisher","first-page":"44316","DOI":"10.7759\/cureus.44316","volume":"29","author":"M Jeyaraman","year":"2023","unstructured":"Jeyaraman M, Jeyaraman N, Nallakumarasamy A, Yadav S, Bondili SK. ChatGPT in medical education and research: a boon or a bane? Cureus. 2023;29:44316\u201310. https:\/\/doi.org\/10.7759\/cureus.44316.","journal-title":"Cureus."},{"key":"135_CR4","doi-asserted-by":"publisher","DOI":"10.1007\/s40596-023-01791-9","author":"D Grabb","year":"2023","unstructured":"Grabb D. ChatGPT in medical education: a paradigm shift or a dangerous tool? Acad Psychiatry. 2023. https:\/\/doi.org\/10.1007\/s40596-023-01791-9.","journal-title":"Acad Psychiatry"},{"key":"135_CR5","doi-asserted-by":"publisher","DOI":"10.1002\/ase.2270","author":"H Lee","year":"2023","unstructured":"Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. 2023. https:\/\/doi.org\/10.1002\/ase.2270.","journal-title":"Anat Sci Educ"},{"issue":"8","key":"135_CR6","doi-asserted-by":"publisher","first-page":"867","DOI":"10.1097\/ACM.0000000000005242","volume":"98","author":"S Feng","year":"2023","unstructured":"Feng S, Shen Y. ChatGPT and the future of medical education. Acad Med. 2023;98(8):867\u20138. https:\/\/doi.org\/10.1097\/ACM.0000000000005242.","journal-title":"Acad Med"},{"key":"135_CR7","doi-asserted-by":"publisher","first-page":"0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepa\u00f1o C, Madriaga M, Aggabao R, DiazCandido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:0000198. https:\/\/doi.org\/10.1371\/journal.pdig.0000198.","journal-title":"PLOS Digit Health"},{"key":"135_CR8","doi-asserted-by":"publisher","first-page":"45312","DOI":"10.2196\/45312","volume":"9","author":"A Gilson","year":"2023","unstructured":"Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:45312. https:\/\/doi.org\/10.2196\/45312.","journal-title":"JMIR Med Educ"},{"key":"135_CR9","doi-asserted-by":"publisher","first-page":"0000205","DOI":"10.1371\/journal.pdig.0000205","volume":"9","author":"AB Mbakwe","year":"2023","unstructured":"Mbakwe AB, Lourentzou I, Celi LA, Mechanic OJ, Dagan A. ChatGPT passing USMLE shines a spotlight on the flaws of medical education. PLOS Dig Health. 2023;9:0000205. https:\/\/doi.org\/10.1371\/journal.pdig.0000205.","journal-title":"PLOS Dig Health"},{"key":"135_CR10","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","volume":"30","author":"S Liu","year":"2023","unstructured":"Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023;30:1237\u201345. https:\/\/doi.org\/10.1093\/jamia\/ocad072.","journal-title":"J Am Med Inform Assoc"},{"key":"135_CR11","doi-asserted-by":"publisher","first-page":"7933","DOI":"10.1002\/ccr3.7933","volume":"11","author":"G Pugliese","year":"2023","unstructured":"Pugliese G, Maccari A, Felisati E, Felisati G, Giudici L, Rapolla C, Pisani A, Saibene AM. Are artificial intelligence large language models a reliable tool for difficult differential diagnosis? An a posteriori analysis of a peculiar case of necrotizing otitis externa. Clin Case Rep. 2023;11:7933. https:\/\/doi.org\/10.1002\/ccr3.7933.","journal-title":"Clin Case Rep"},{"key":"135_CR12","doi-asserted-by":"publisher","first-page":"48568","DOI":"10.2196\/48568","volume":"28","author":"J Liu","year":"2023","unstructured":"Liu J, Wang C, Liu S. Utility of ChatGPT in clinical practice. J Med Internet Res. 2023;28:48568. https:\/\/doi.org\/10.2196\/48568.","journal-title":"J Med Internet Res"},{"key":"135_CR13","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230163","author":"Y Shen","year":"2023","unstructured":"Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Moy L. ChatGPT and other large language models are double-edged swords. Radiology. 2023. https:\/\/doi.org\/10.1148\/radiol.230163.","journal-title":"Radiology"}],"container-title":["Discover Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44163-024-00135-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44163-024-00135-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44163-024-00135-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T10:06:38Z","timestamp":1715853998000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44163-024-00135-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,16]]},"references-count":13,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["135"],"URL":"https:\/\/doi.org\/10.1007\/s44163-024-00135-2","relation":{},"ISSN":["2731-0809"],"issn-type":[{"value":"2731-0809","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,16]]},"assertion":[{"value":"22 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"38"}}