{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T22:53:30Z","timestamp":1787698410482,"version":"build-2784847793"},"reference-count":27,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Artificial intelligence and large language models (LLMs)\u2014particularly GPT-4 and GPT-4o\u2014have demonstrated high correct-answer rates in medical examinations. GPT-4o has enhanced diagnostic capabilities, advanced image processing, and updated knowledge. Japanese surgeons face critical challenges, including a declining workforce, regional health care disparities, and work-hour-related challenges. Nonetheless, although LLMs could be beneficial in surgical education, no studies have yet assessed GPT-4o\u2019s surgical knowledge or its performance in the field of surgery.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the potential of GPT-4 and GPT-4o in surgical education by using them to take the Japan Surgical Board Examination (JSBE), which includes both textual questions and medical images\u2014such as surgical and computed tomography scans\u2014to comprehensively assess their surgical knowledge.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We used 297 multiple-choice questions from the 2021\u20102023 JSBEs. The questions were in Japanese, and 104 of them included images. First, the GPT-4 and GPT-4o responses to only the textual questions were collected via OpenAI\u2019s application programming interface to evaluate their correct-answer rate. Subsequently, the correct-answer rate of their responses to questions that included images was assessed by inputting both text and images.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The overall correct-answer rates of GPT-4o and GPT-4 for the text-only questions were 78% (231\/297) and 55% (163\/297), respectively, with GPT-4o outperforming GPT-4 by 23% (\n P<\/jats:italic>\n =<.01). By contrast, there was no significant improvement in the correct-answer rate for questions that included images compared with the results for the text-only questions.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n GPT-4o outperformed GPT-4 on the JSBE. However, the results of the LLMs were lower than those of the examinees. Despite the capabilities of LLMs, image recognition remains a challenge for them, and their clinical application requires caution owing to the potential inaccuracy of their results.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69313","type":"journal-article","created":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T20:28:16Z","timestamp":1753907296000},"page":"e69313-e69313","source":"Crossref","is-referenced-by-count":13,"title":["Role of Artificial Intelligence in Surgical Training by Assessing GPT-4 and GPT-4o on the Japan Surgical Board Examination With Text-Only and Image-Accompanied Questions: Performance Evaluation Study"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-7845-2281","authenticated-orcid":false,"given":"Hiroki","family":"Maruyama","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0027-9681","authenticated-orcid":false,"given":"Yoshitaka","family":"Toyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0098-7760","authenticated-orcid":false,"given":"Kentaro","family":"Takanami","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0931-9942","authenticated-orcid":false,"given":"Kei","family":"Takase","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1282-0463","authenticated-orcid":false,"given":"Takashi","family":"Kamei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,30]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1097\/01.sla.0000150066.83563.52","article-title":"American Surgical Association Blue Ribbon Committee Report on Surgical Education: 2004","volume":"241","author":"Debas","journal-title":"Ann Surg"},{"key":"R2","unstructured":"Overview of statistics on doctors, dentists [Article in Japanese]. Ministry of Health Labour and Welfare. 2024. URL: https:\/\/www.mhlw.go.jp\/toukei\/saikin\/hw\/ishi\/22\/index.html [Accessed 16-07-2025]"},{"key":"R3","unstructured":"Work style reform for doctors [Article in Japanese]. Ministry of Health, Labour and Welfare. 2024. URL: https:\/\/www.mhlw.go.jp\/content\/10800000\/001129457.pdf [Accessed 16-07-2025]"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1590\/0100-6991e-20233605-en","article-title":"Innovations in surgical training: exploring the role of artificial intelligence and large language models (LLM)","volume":"50","author":"Varas","journal-title":"Rev Col Bras Cir"},{"key":"R5","unstructured":"ChatGPT. Open AI. 2024. URL: https:\/\/openai.com\/chatgpt\/ [Accessed 16-07-2025]"},{"key":"R6","unstructured":"Open AIAchiam J Adler S et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 4, 2024. Preprint posted online on Mar 4, 2024. URL: https:\/\/arxiv.org\/pdf\/2303.08774 [Accessed 25-07-2025]"},{"issue":"8","key":"R7","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"2","key":"R8","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.04.060","article-title":"Performance of ChatGPT on American Board of Surgery In-Training Examination preparation questions","volume":"299","author":"Tran","journal-title":"J Surg Res"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","first-page":"269","DOI":"10.4174\/astr.2023.104.5.269","article-title":"ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models","volume":"104","author":"Oh","journal-title":"Ann Surg Treat Res"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Yang Z Yao Z Tasmin M et al. Performance of multimodal GPT-4V on USMLE with image: potential for imaging diagnostic support with explanations. Radiology and Imaging. Preprint posted online on Nov 5, 2023. [doi: 10.1101\/2023.10.26.23297629]","DOI":"10.1101\/2023.10.26.23297629"},{"key":"R12","unstructured":"Hello GPT-4o. OpenAI. 2024. URL: https:\/\/openai.com\/index\/hello-gpt-4o\/ [Accessed 16-07-2025]"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejca.2024.114132","article-title":"OpenAI\u2019s GPT-4o in surgical oncology: revolutionary advances in generative artificial intelligence","volume":"206","author":"Zhu","journal-title":"Eur J Cancer"},{"issue":"12","key":"R14","doi-asserted-by":"publisher","first-page":"1392","DOI":"10.1007\/s11604-024-01633-0","article-title":"Diagnostic accuracy of vision-language models on Japanese diagnostic radiology, nuclear medicine, and interventional radiology specialty board examinations","volume":"42","author":"Oura","journal-title":"Jpn J Radiol"},{"issue":"3","key":"R15","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240153","article-title":"Performance of GPT-4 with Vision on text- and image-based ACR diagnostic radiology in-training examination questions","volume":"312","author":"Hayden","journal-title":"Radiology"},{"issue":"17","key":"R16","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12171726","article-title":"Custom GPTs enhancing performance and evidence compared with GPT-3.5, GPT-4, and GPT-4o? A study on the Emergency Medicine Specialist Examination","volume":"12","author":"Liu","journal-title":"Healthcare (Basel)"},{"key":"R17","unstructured":"OpenAI. Models. 2024. URL: https:\/\/platform.openai.com\/docs\/models\/models [Accessed 16-07-2025]"},{"issue":"3","key":"R18","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.56402","article-title":"A comparison between GPT-3.5, GPT-4, and GPT-4V: can the large language model (ChatGPT) pass the Japanese Board of Orthopaedic Surgery Examination?","volume":"16","author":"Nakajima","journal-title":"Cureus"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/54283","article-title":"The performance of ChatGPT-4V in interpreting images and tables in the Japanese Medical Licensing Exam","volume":"10","author":"Takagi","journal-title":"JMIR Med Educ"},{"issue":"8","key":"R20","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1007\/s11604-024-01561-z","article-title":"GPT-4 Turbo with vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination","volume":"42","author":"Hirano","journal-title":"Jpn J Radiol"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-024-02103-w","article-title":"Analysis of responses of GPT-4 V to the Japanese National Clinical Engineer Licensing Examination","volume":"48","author":"Ishida","journal-title":"J Med Syst"},{"issue":"8","key":"R22","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.67347","article-title":"Performance of ChatGPT 4.0 on Japan\u2019s National Physical Therapist Examination: a comprehensive analysis of text and visual question handling","volume":"16","author":"Sawamura","journal-title":"Cureus"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1007\/s11604-024-01673-6","article-title":"Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan","volume":"43","author":"Harigai","journal-title":"Jpn J Radiol"},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Bender EM Gebru T McMillan-Major A Shmitchell S . On the dangers of stochastic parrots: can language models be too big. Presented at: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency; Mar 1, 2021:610-623; [doi: 10.1145\/3442188.3445922]","DOI":"10.1145\/3442188.3445922"},{"issue":"2","key":"R25","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232715","article-title":"Evaluation of reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 on a radiology board\u2013style examination","volume":"311","author":"Krishna","journal-title":"Radiology"},{"issue":"2","key":"R26","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","journal-title":"Cureus"},{"issue":"7","key":"R27","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1007\/s11604-024-01552-0","article-title":"The impact of large language models on radiology: a guide for radiologists on the latest innovations in AI","volume":"42","author":"Nakaura","journal-title":"Jpn J Radiol"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e69313","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:56:55Z","timestamp":1782842215000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e69313"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,30]]},"references-count":27,"URL":"https:\/\/doi.org\/10.2196\/69313","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,30]]},"article-number":"v11i1e69313"}}