{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T01:31:47Z","timestamp":1793323907748,"version":"4.5.0"},"reference-count":18,"publisher":"Open Medical Publishing","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"\n Background<\/jats:title>\n Previous studies evaluating large language models (LLMs) on the Orthopaedic In-Training Examination (OITE) have primarily focused on earlier-generation models and single-pass accuracy. These investigations did not assess newer multimodal systems such as GPT-5 and Gemini 2.5 Pro, nor did they examine the reasoning quality underlying model responses or the consistency of outputs across repeated trials. As LLMs are increasingly used as educational tools, a more comprehensive evaluation framework is needed to assess not only correctness but also reliability and explanatory validity on specialty-specific, image-rich examinations.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a controlled, parallel evaluation of GPT-5 and Gemini 2.5 Pro using 412 OITE-style questions from the 2023\u20132024 examination cycle obtained via an institutional AAOS ResStudy subscription. Primary outcomes included overall and subspecialty-specific accuracy. Secondary analyses evaluated explanatory quality, error-pattern classification, response consistency across repeated trials, and performance stratified by imaging burden. Paired accuracy was compared using McNemar\u2019s exact test.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Gemini 2.5 Pro demonstrated higher overall accuracy than GPT-5 on the 2023\u20132024 OITE question set (81.1% vs 76.0), with both models exceeding published PGY-5 resident benchmarks. Accuracy declined significantly with questions containing images (74.2% vs 71.6%). Subspecialty performance varied widely, with accuracy ranging from 42.9% to 94.1% for GPT-5 and from 57.1% to 95.8% for Gemini, and both models performing poorest in Hand and Wrist questions. Among incorrect responses, faulty reasoning accounted for 52.5% of GPT-5 errors, whereas stem misinterpretation was the predominant error for Gemini (43.6%). Incorrect or partially correct explanations accompanied 45.4% of GPT-5 and 41.7% of Gemini responses. Consistency testing showed high reproducibility (fully consistent responses: 88% for GPT-5 and 84% for Gemini), with all inconsistent outputs occurring in image-containing questions.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n GPT-5 and Gemini 2.5 Pro demonstrate strong performance on recent OITE content, exceeding prior LLM benchmarks; however, persistent limitations in multimodal reasoning, explanatory reliability, and response consistency indicate that high accuracy alone does not ensure dependable clinical reasoning, underscoring the need for cautious educational use.<\/jats:p>\n <\/jats:sec>","DOI":"10.52965\/001c.160184","type":"journal-article","created":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T00:27:23Z","timestamp":1776385643000},"source":"Crossref","is-referenced-by-count":3,"title":["Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination"],"prefix":"10.52965","volume":"18","author":[{"given":"Le Duc","family":"Huy","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luong Nhat","family":"Anh","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mai Huy","family":"Quang","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luu Huu","family":"Phuc","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nguyen Duc","family":"Trung","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vu Duc","family":"Thang","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Le Hoai","family":"Nam","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/04k25m262","id-type":"ROR","asserted-by":"publisher"}],"name":"108 Military Central Hospital","department":["Department of General Orthopedics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tran Trung","family":"Dung","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/052dmdr17","id-type":"ROR","asserted-by":"publisher"}],"name":"VinUniversity","department":["College of Health Sciences"]},{"id":[{"id":"https:\/\/ror.org\/01q2pxs68","id-type":"ROR","asserted-by":"publisher"}],"name":"Vinmec International Hospital","department":["Orthopaedics & Sports Medicine Center"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"30777","published-online":{"date-parts":[[2026,4,17]]},"reference":[{"issue":"11","key":"ref1","doi-asserted-by":"publisher","first-page":"1645","DOI":"10.1016\/j.jsurg.2024.08.002","article-title":"Artificial Intelligence in Orthopaedics: Performance of ChatGPT on Text and Image Questions on a Complete AAOS Orthopaedic In-Training Examination (OITE)","volume":"81","author":"D. S. Hayes","year":"2024","journal-title":"Journal of Surgical Education"},{"issue":"23","key":"ref2","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.5435\/JAAOS-D-23-00396","article-title":"Comparison of ChatGPT-3.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations","volume":"31","author":"P. A. Massey","year":"2023","journal-title":"J Am Acad Orthop Surg"},{"issue":"1","key":"ref3","doi-asserted-by":"publisher","DOI":"10.60118\/001c.124070","article-title":"Artificial Intelligence Large Language Models Are Nearly Equivalent to Fourth-Year Orthopaedic Residents on the Orthopaedic In-Training Examination: A Cause for Concern or Excitement?","volume":"6","author":"A. Nawari","year":"2025","journal-title":"J Orthopaedic Experience & Innovation"},{"issue":"10","key":"ref4","doi-asserted-by":"publisher","first-page":"900","DOI":"10.3348\/kjr.2025.0599","article-title":"Multimodal Large Language Models in Medical Imaging: Current State and Future Directions","volume":"26","author":"Y. Nam","year":"2025","journal-title":"Korean J Radiol"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"100728","DOI":"10.1016\/j.jorep.2025.100728","article-title":"Comparative evaluation of LLMs in orthopedic surgery","author":"G. Chundi","year":"2025","journal-title":"Journal of Orthopaedic Reports"},{"issue":"16","key":"ref6","doi-asserted-by":"publisher","first-page":"5876","DOI":"10.3390\/jcm14165876","article-title":"Systematic Review on Large Language Models in Orthopaedic Surgery","volume":"14","author":"K. Mo","year":"2025","journal-title":"Journal of Clinical Medicine"},{"issue":"1","key":"ref7","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1038\/s41746-025-02008-z","article-title":"When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior","volume":"8","author":"S. Chen","year":"2025","journal-title":"npj Digit Med"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"e65146","DOI":"10.2196\/65146","article-title":"Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study","volume":"27","author":"Z. Yang","year":"2025","journal-title":"J Med Internet Res"},{"issue":"4","key":"ref9","doi-asserted-by":"publisher","first-page":"100189","DOI":"10.1016\/j.ajoint.2025.100189","article-title":"Assessing consistency of AI chatbot responses in ophthalmology medical exams","volume":"2","author":"E. Shvartz","year":"2025","journal-title":"AJO International"},{"issue":"3","key":"ref10","doi-asserted-by":"publisher","first-page":"e23.00056","DOI":"10.2106\/JBJS.OA.23.00056","article-title":"Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination","volume":"8","author":"J. E. Kung","year":"2023","journal-title":"JBJS Open Access"},{"issue":"3","key":"ref11","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1016\/j.arthro.2024.08.023","article-title":"ChatGPT, Bard, and Bing Chat Are Large Language Processing Models That Answered Orthopaedic In-Training Examination Questions With Similar Accuracy to First-Year Orthopaedic Surgery Residents","volume":"41","author":"G. A. Guerra","year":"2025","journal-title":"Arthroscopy"},{"issue":"2","key":"ref12","doi-asserted-by":"publisher","first-page":"e85","DOI":"10.3928\/01477447-20230922-05","article-title":"The Rapid Development of Artificial Intelligence: GPT-4's Performance on Orthopedic Surgery Board Questions","volume":"47","author":"H. L. Hofmann","year":"2024","journal-title":"Orthopedics"},{"key":"ref13","article-title":"Orthopaedic In-Training Examination (OITE) Technical Report 2023"},{"key":"ref14","article-title":"Orthopaedic In-Training Examination (OITE) Technical Report 2024"},{"issue":"8","key":"ref15","doi-asserted-by":"publisher","first-page":"1623","DOI":"10.1097\/CORR.0000000000002704","article-title":"Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT","volume":"481","author":"Z. C. Lum","year":"2023","journal-title":"Clinical Orthopaedics and Related Research\u00ae"},{"issue":"7","key":"ref16","doi-asserted-by":"publisher","first-page":"2289","DOI":"10.3390\/jcm14072289","article-title":"An Assessment of the Performance of Different Chatbots on Shoulder and Elbow Questions","volume":"14","author":"M. Y. Fares","year":"2025","journal-title":"Journal of Clinical Medicine"},{"issue":"23","key":"ref17","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.5435\/JAAOS-D-23-00396","article-title":"Comparison of ChatGPT\u20133.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations","volume":"31","author":"P. A. Massey","year":"2023","journal-title":"JAAOS - Journal of the American Academy of Orthopaedic Surgeons"},{"issue":"2","key":"ref18","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1016\/j.jhsg.2023.10.013","article-title":"Comparison of Artificial Intelligence to Resident Performance on Upper-Extremity Orthopaedic In-Training Examination Questions","volume":"6","author":"Y. Ozdag","year":"2024","journal-title":"Journal of Hand Surgery Global Online"}],"container-title":["Orthopedic Reviews"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/orthopedicreviews.openmedicalpublishing.org\/article\/160184-performance-of-gpt-5-and-gemini-2-5-pro-on-the-orthopaedic-in-training-examination","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T00:45:31Z","timestamp":1793321131000},"score":1,"resource":{"primary":{"URL":"https:\/\/orthopedicreviews.openmedicalpublishing.org\/article\/160184-performance-of-gpt-5-and-gemini-2-5-pro-on-the-orthopaedic-in-training-examination"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,17]]},"references-count":18,"URL":"https:\/\/doi.org\/10.52965\/001c.160184","relation":{},"ISSN":["2035-8164"],"issn-type":[{"value":"2035-8164","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,17]]}}}