{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T09:35:55Z","timestamp":1793352955219,"version":"4.5.0"},"reference-count":21,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"34","license":[{"start":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T00:00:00Z","timestamp":1787270400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Bio&Medical Technology Development Program of the National Research Foundation","award":["RS-2023-00220408"],"award-info":[{"award-number":["RS-2023-00220408"]}]},{"name":"2025 Yeungnam University Research Grant","award":["No. 225A580004"],"award-info":[{"award-number":["No. 225A580004"]}],"id":[{"id":"https:\/\/ror.org\/05yc6p159","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,8,21]]},"abstract":"<jats:sec>\n                    <jats:title\/>\n                    <jats:p>Attention toward Chat Generative Pre-trained Transformer (ChatGPT) has greatly increased, including in the medical field, but concerns about its problem-solving ability, accuracy, reliability, and limitations in medical problems always coexist. This study aimed to evaluate ChatGPT\u2019s performance in solving a medical school examination. We used 50-question examinations from the preventive medicine and psychiatry fields applied to 41 fourth-year medical students. Forty-nine questions were separately entered (one was excluded for being image-based) into the text box of ChatGPT models 3.5 and 4.0 with a request to select the single best answer. The initial answer was considered for primary evaluation, but incorrect answers led to model re-prompting to evaluate its answer regeneration function. We assessed the accuracy of both ChatGPT models and compared them with that of students. The student mean score was 29.5\u2005\u00b1\u20055.9. ChatGPT\u2019s performance was higher in model 4.0 and when re-prompted (models 3.5 and 4.0 initial scores, 28 and 43; re-prompt scores, 40 and 46). Model 4.0 outperformed all students at re-prompting. Both ChatGPT models and all prompts (except for the model 3.5 first prompt) had a higher probability than students of providing correct answers. For easy questions, the accuracy of both ChatGPT models differed significantly from that of students. For difficult questions, the difference between students and the first prompt of model 3.5 was nonsignificant. We observed a nonsignificant tendency in the point estimation when comparing the accuracy of ChatGPT models and prompts by question difficulty. Both models showed excellent performance and learning ability in the medical field, especially model 4.0, and can be considered reliable large language model tools. However, future studies should constantly verify the reliability of the information from these models. Also, educators must be cautious when interpreting\/applying ChatGPT output data for clinical and educational purposes.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1097\/md.0000000000050323","type":"journal-article","created":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:05:25Z","timestamp":1787331925000},"page":"e50323","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":1,"title":["Artificial intelligence in medical education"],"prefix":"10.1097","volume":"105","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3879-0768","authenticated-orcid":false,"given":"Kyung Wook","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Orthopaedic Surgery, Dankook University Hospital, Dankook University College of Medicine, Cheonan, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong Bae","family":"Jeon","sequence":"additional","affiliation":[{"name":"Department of Plastic and Reconstructive Surgery, Dankook University Hospital, Dankook University College of Medicine, Cheonan, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jong Hyuk","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine, College of Medicine, Dankook University, Cheonan, South Korea"},{"name":"Research Institute of Healthcare Bigdata, College of Medicine, Dankook University, Cheonan, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dong Uk","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Dankook University Hospital, Dankook University College of Medicine, Cheonan, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoon Sung","family":"Jeon","sequence":"additional","affiliation":[{"name":"Department of Orthopaedic Surgery, Dankook University Hospital, Dankook University College of Medicine, Cheonan, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3751-2304","authenticated-orcid":false,"given":"Bum-Jin","family":"Shim","sequence":"additional","affiliation":[{"name":"Department of Orthopedic Surgery, Yeungnam University Hospital, Yeungnam University College of Medicine, Daegu, South Korea."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","reference":[{"key":"R1-20260917","doi-asserted-by":"crossref","first-page":"e0000501","DOI":"10.1371\/journal.pdig.0000501","article-title":"PLOS-LLM: can and should AI enable a new paradigm of scientific knowledge sharing?","volume":"3","author":"Hughes","year":"2024","journal-title":"PLOS Digit Health"},{"key":"R2-20260917","doi-asserted-by":"crossref","first-page":"2248","DOI":"10.1007\/s00259-023-06227-y","article-title":"Large language models (LLM) and ChatGPT: a medical student perspective.","volume":"50","author":"Perera Molligoda Arachchige","year":"2023","journal-title":"Eur J Nucl Med Mol Imaging"},{"key":"R3-20260917","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1016\/j.jpurol.2023.05.018","article-title":"ChatGPT and large language model (LLM) chatbots: the current state of acceptability and a proposal for guidelines on utilization in academic medicine.","volume":"19","author":"Kim","year":"2023","journal-title":"J Pediatr Urol"},{"key":"R4-20260917","doi-asserted-by":"crossref","first-page":"e66114","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: platform development and comprehensive analysis.","volume":"26","author":"Zong","year":"2024","journal-title":"J Med Internet Res"},{"key":"R5-20260917","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.jor.2023.11.056","article-title":"The performance of ChatGPT on orthopaedic in-service training exams: a comparative study of the GPT-3.5 turbo and GPT-4 models in orthopaedic education.","volume":"50","author":"Rizzo","year":"2024","journal-title":"J Orthop"},{"key":"R6-20260917","doi-asserted-by":"crossref","first-page":"347","DOI":"10.4055\/cios23181","article-title":"Application of ChatGPT for orthopedic surgeries and patient care.","volume":"16","author":"Morya","year":"2024","journal-title":"Clin Orthop Surg"},{"key":"R7-20260917","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/s10916-024-02075-x","article-title":"ChatGPT: a conceptual review of applications and utility in the field of medicine.","volume":"48","author":"Rao","year":"2024","journal-title":"J Med Syst"},{"key":"R8-20260917","first-page":"e44541","article-title":"A radiation oncology board exam of ChatGPT.","volume":"15","author":"Barbour","year":"2023","journal-title":"Cureus"},{"key":"R9-20260917","doi-asserted-by":"crossref","first-page":"e230987","DOI":"10.1148\/radiol.230987","article-title":"GPT-4 in radiology: improvements in advanced reasoning.","volume":"307","author":"Bhayana","year":"2023","journal-title":"Radiology"},{"key":"R10-20260917","doi-asserted-by":"crossref","first-page":"e230582","DOI":"10.1148\/radiol.230582","article-title":"Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations.","volume":"307","author":"Bhayana","year":"2023","journal-title":"Radiology"},{"key":"R11-20260917","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models.","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"R12-20260917","doi-asserted-by":"crossref","first-page":"e37325","DOI":"10.1097\/MD.0000000000037325","article-title":"Comparison of the problem-solving performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard for the Korean emergency medicine board examination question bank.","volume":"103","author":"Lee","year":"2024","journal-title":"Medicine (Baltimore)"},{"key":"R13-20260917","doi-asserted-by":"crossref","first-page":"13553","DOI":"10.1038\/s41598-024-63997-7","article-title":"In-depth analysis of ChatGPT\u2019s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions.","volume":"14","author":"Knoedler","year":"2024","journal-title":"Sci Rep"},{"key":"R14-20260917","doi-asserted-by":"crossref","first-page":"669","DOI":"10.4055\/cios23179","article-title":"Performance of ChatGPT on solving orthopedic board-style questions: a comparative analysis of ChatGPT 3.5 and ChatGPT 4.","volume":"16","author":"Kim","year":"2024","journal-title":"Clin Orthop Surg"},{"key":"R15-20260917","doi-asserted-by":"crossref","first-page":"858","DOI":"10.1093\/postmj\/qgae065","article-title":"ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review.","volume":"100","author":"Kiyak","year":"2024","journal-title":"Postgrad Med J"},{"key":"R16-20260917","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1097\/JCMA.0000000000001064","article-title":"Performance of GPT-4 and GPT-3.5 in generating accurate and comprehensive diagnoses across medical subspecialties.","volume":"87","author":"Luk","year":"2024","journal-title":"J Chin Med Assoc"},{"key":"R17-20260917","doi-asserted-by":"crossref","first-page":"100032","DOI":"10.1016\/j.nlp.2023.100032","article-title":"GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts.","volume":"5","author":"L\u00f3pez Espejel","year":"2023","journal-title":"Natural Lang Processing J"},{"key":"R18-20260917","doi-asserted-by":"crossref","first-page":"e232715","DOI":"10.1148\/radiol.232715","article-title":"Evaluation of reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 on a radiology board-style examination.","volume":"311","author":"Krishna","year":"2024","journal-title":"Radiology"},{"key":"R19-20260917","doi-asserted-by":"crossref","first-page":"936","DOI":"10.1016\/j.surg.2023.12.014","article-title":"Evaluating capabilities of large language models: performance of GPT-4 on surgical knowledge assessments.","volume":"175","author":"Beaulieu-Jones","year":"2024","journal-title":"Surgery"},{"key":"R20-20260917","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1097\/JCMA.0000000000001032","article-title":"Limitations of GPT-4 as a geriatrician in geri-oncology case conference: a case series.","volume":"87","author":"Kuk","year":"2024","journal-title":"J Chin Med Assoc"},{"key":"R21-20260917","doi-asserted-by":"crossref","first-page":"e0000397","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language.","volume":"2","author":"Fang","year":"2023","journal-title":"PLOS Digit Health"}],"container-title":["Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.1097\/MD.0000000000050323","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T06:00:47Z","timestamp":1789624847000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.1097\/MD.0000000000050323"}},"subtitle":["Performance of ChatGPT versus medical students on medical examination"],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":21,"journal-issue":{"issue":"34","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.1097\/md.0000000000050323","relation":{},"ISSN":["0025-7974","1536-5964"],"issn-type":[{"value":"0025-7974","type":"print"},{"value":"1536-5964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2025-02-06","name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-05","name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}}]}}