{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T23:12:41Z","timestamp":1777417961552,"version":"3.51.4"},"reference-count":17,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1 TR003163"],"award-info":[{"award-number":["UL1 TR003163"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We aimed to assess ChatGPT\u2019s performance on the Clinical Informatics Board Examination and to discuss the implications of large language models (LLMs) for board certification and maintenance. We tested ChatGPT using 260 multiple-choice questions from Mankowitz\u2019s Clinical Informatics Board Review book, omitting 6 image-dependent questions. ChatGPT answered 190 (74%) of 254 eligible questions correctly. While performance varied across the Clinical Informatics Core Content Areas, differences were not statistically significant. ChatGPT\u2019s performance raises concerns about the potential misuse in medical certification and the validity of knowledge assessment exams. Since ChatGPT is able to answer multiple-choice questions accurately, permitting candidates to use artificial intelligence (AI) systems for exams will compromise the credibility and validity of at-home assessments and undermine public trust. The advent of AI and LLMs threatens to upend existing processes of board certification and maintenance and necessitates new approaches to the evaluation of proficiency in medical education.<\/jats:p>","DOI":"10.1093\/jamia\/ocad104","type":"journal-article","created":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T18:53:03Z","timestamp":1687200783000},"page":"1558-1560","source":"Crossref","is-referenced-by-count":47,"title":["ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?"],"prefix":"10.1093","volume":"30","author":[{"given":"Yaa","family":"Kumah-Crystal","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Pediatric Endocrinology, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Scott","family":"Mankowitz","sequence":"additional","affiliation":[{"name":"Department of Clinical Informatics, Overlook Medical Center , Summit, New Jersey, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Embi","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9559-4646","authenticated-orcid":false,"given":"Christoph U","family":"Lehmann","sequence":"additional","affiliation":[{"name":"Clinical Informatics Center, UT Southwestern Medical Center , Dallas, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2023,6,19]]},"reference":[{"key":"2023081808480671500_ocad104-B1","author":"OpenAI","year":"2023"},{"issue":"2","key":"2023081808480671500_ocad104-B2","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2023081808480671500_ocad104-B3","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"2023081808480671500_ocad104-B4","author":"Nori","year":"2023"},{"key":"2023081808480671500_ocad104-B5","author":"United States Medical Licensing Examination","year":"2023"},{"issue":"1","key":"2023081808480671500_ocad104-B6","doi-asserted-by":"crossref","first-page":"e13930","DOI":"10.2196\/13930","article-title":"Applications and challenges of implementing artificial intelligence in medical education: integrative review","volume":"5","author":"Chan","year":"2019","journal-title":"JMIR Med Educ"},{"key":"2023081808480671500_ocad104-B7","author":"Clinical Informatics","year":"2023"},{"key":"2023081808480671500_ocad104-B8","author":"Mankowitz","year":"2018"},{"issue":"2","key":"2023081808480671500_ocad104-B9","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1197\/jamia.M3045","article-title":"Core content for the subspecialty of clinical informatics","volume":"16","author":"Gardner","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"2023081808480671500_ocad104-B10","doi-asserted-by":"crossref","first-page":"586","DOI":"10.1093\/jamia\/ocz051","article-title":"Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis","volume":"26","author":"Silverman","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2023081808480671500_ocad104-B11","volume-title":"Clinical Informatics Subspecialty Certification and Training. Informatics Education in Healthcare","author":"Lehmann","year":"2020"},{"key":"2023081808480671500_ocad104-B12","doi-asserted-by":"crossref","first-page":"S7","DOI":"10.1002\/chp.21201","article-title":"American Board of Medical Specialties Maintenance of Certification: theory and evidence regarding the current framework","volume":"33 Suppl 1","author":"Hawkins","year":"2013","journal-title":"J Contin Educ Health Prof"},{"issue":"8","key":"2023081808480671500_ocad104-B13","doi-asserted-by":"crossref","first-page":"972","DOI":"10.1080\/0142159X.2020.1811214","article-title":"Assessing open-book examination in medical education: the time is now","volume":"43","author":"Zagury-Orly","year":"2021","journal-title":"Med Teach"},{"key":"2023081808480671500_ocad104-B14","first-page":"45","volume-title":"Is Chess Finished? Commentary","author":"Gulko","year":"1997"},{"key":"2023081808480671500_ocad104-B15","author":"Beaton","year":"2022"},{"key":"2023081808480671500_ocad104-B16","author":"Li\u00e9vin","year":"2022"},{"issue":"1","key":"2023081808480671500_ocad104-B17","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/9\/1558\/51141429\/ocad104.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/9\/1558\/51141429\/ocad104.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T09:44:04Z","timestamp":1692351844000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/9\/1558\/7202064"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,19]]},"references-count":17,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,6,19]]},"published-print":{"date-parts":[[2023,8,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad104","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,9,1]]},"published":{"date-parts":[[2023,6,19]]}}}