{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T23:52:21Z","timestamp":1793663541285,"version":"4.5.1"},"reference-count":28,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["J Orthop Surg (Hong Kong)"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) have many clinical applications. However, the comparative performance of different LLMs on orthopedic board style questions remains largely unknown.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Three LLMs, OpenAI\u2019s GPT-4 and GPT-3.5, and Google Bard, were tested on 189 official 2022 Orthopedic In-Training Examination (OITE) questions. Comparative analyses were conducted to assess their performance against orthopedic resident scores and on higher-order, image-associated, and subject category-specific questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>GPT-4 surpassed the passing threshold for the 2022 OITE, performing at the level of PGY-3 to PGY-5 ( p = .149, p = .502, and p = .818, respectively) and outperforming GPT-3.5 and Bard ( p &lt; .001 and p = .001, respectively). While GPT-3.5 and Bard did not meet the passing threshold for the exam, GPT-3.5 performed at the level of PGY-1 to PGY-2 ( p = .368 and p = .019, respectively) and Bard performed at the level of PGY-1 to PGY-3 ( p = .440, .498, and 0.036, respectively). GPT-4 outperformed both Bard and GPT-3.5 on image-associated ( p = .003 and p &lt; .001, respectively) and higher-order questions ( p &lt; .001). Among the 11 subject categories, all models performed similarly regardless of the subject matter. When individual LLM performance on higher-order questions was assessed, no significant differences were found compared to performance on first order questions (GPT-4 p = .139, GPT-3.5 p = .124, Bard p = .319). Finally, when individual model performance was assessed on image-associated questions, only GPT-3.5 performed significantly worse compared to performance on non-image-associated questions ( p = .045).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The AI-based LLM GPT-4, exhibits a robust ability to correctly answer a diverse range of OITE questions, exceeding the minimum score for the 2022 OITE, and outperforming predecessor GPT-3.5 and Google Bard.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/10225536241268789","type":"journal-article","created":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T13:53:00Z","timestamp":1741096380000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":12,"title":["Comparitive performance of artificial intelligence-based large language models on the orthopedic in-training examination"],"prefix":"10.1177","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9406-4675","authenticated-orcid":false,"given":"Andrew Y","family":"Xu","sequence":"first","affiliation":[{"name":"Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4524-6226","authenticated-orcid":false,"given":"Manjot","family":"Singh","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariah","family":"Balmaceno-Criss","sequence":"additional","affiliation":[{"name":"Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Allison","family":"Oh","sequence":"additional","affiliation":[{"name":"Harvard College, Harvard University, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4824-0994","authenticated-orcid":false,"given":"David","family":"Leigh","sequence":"additional","affiliation":[{"name":"Khoury College of Computer Sciences, Northeastern University, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9256-9952","authenticated-orcid":false,"given":"Mohammad","family":"Daher","sequence":"additional","affiliation":[{"name":"Department of Orthopedics, Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Alsoof","sequence":"additional","affiliation":[{"name":"Department of Orthopedics, Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher L","family":"McDonald","sequence":"additional","affiliation":[{"name":"Department of Orthopedics, Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bassel G","family":"Diebo","sequence":"additional","affiliation":[{"name":"Department of Orthopedics, Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9597-4139","authenticated-orcid":false,"given":"Alan H","family":"Daniels","sequence":"additional","affiliation":[{"name":"Department of Orthopedics, Warren Alpert Medical School, Brown University, Providence, RI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2025,3,3]]},"reference":[{"key":"bibr1-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-023-01925-4"},{"key":"bibr2-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.22.0095"},{"key":"bibr3-10225536241268789","unstructured":"ChatGPT. https:\/\/openai.com\/chatgpt (accessed 24 June 2023)."},{"key":"bibr4-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"bibr5-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1016\/j.xops.2023.100324"},{"key":"bibr6-10225536241268789","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2303.13375"},{"key":"bibr7-10225536241268789","doi-asserted-by":"publisher","DOI":"10.3171\/2023.2.JNS23419"},{"key":"bibr8-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1227\/neu.0000000000002551"},{"key":"bibr9-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582"},{"key":"bibr10-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1093\/asj\/sjad130"},{"key":"bibr11-10225536241268789","unstructured":"Try bard, an AI experiment by Google. https:\/\/bard.google.com (accessed 26 June 2023)."},{"key":"bibr12-10225536241268789","doi-asserted-by":"publisher","DOI":"10.5435\/JAAOS-D-20-01020"},{"key":"bibr13-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1097\/CORR.0000000000002704"},{"key":"bibr14-10225536241268789","unstructured":"Naveed H, Khan AU, Qiu S, et al. A comprehensive overview of Large Language Models. Published online September 13, 2023. https:\/\/arxiv.org\/abs\/2307.06435 (accessed 26 September 2023)."},{"key":"bibr15-10225536241268789","unstructured":"OpenAI. GPT-4 technical report. Published online March 27, 2023. https:\/\/arxiv.org\/abs\/2303.08774 (accessed 26 June 2023)."},{"key":"bibr16-10225536241268789","unstructured":"Thoppilan R, De Freitas D, Hall J, et al. LaMDA: language Models for dialog applications. Published online February 10, 2022. https:\/\/arxiv.org\/abs\/2201.08239 (accessed 21 June 2023)."},{"key":"bibr17-10225536241268789","doi-asserted-by":"publisher","DOI":"10.5435\/JAAOS-D-20-01019"},{"key":"bibr18-10225536241268789","unstructured":"American Academy of Orthopaedic Surgeons. Orthopaedic in-training examination (OITE). Technical Report. https:\/\/www.aaos.org\/globalassets\/education\/product-pages\/oite\/oite-2022-technical-report-20230125.pdf (2022, accessed 24 June 2023)."},{"key":"bibr19-10225536241268789","unstructured":"Liu H, Ning R, Teng Z, et al. Evaluating the logical reasoning ability of ChatGPT and GPT-4. Published online May 5, 2023 https:\/\/arxiv.org\/abs\/2304.03439 (accessed 27 June 2023)."},{"key":"bibr20-10225536241268789","unstructured":"Huang J, Gu SS, Hou L, et al. Large Language models can self-improve. Published online October 25, 2022 https:\/\/arxiv.org\/abs\/2210.11610 (accessed 27 June 2023)."},{"key":"bibr21-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1101\/2023.03.25.23287743"},{"key":"bibr22-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1186\/s42836-022-00112-z"},{"key":"bibr23-10225536241268789","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.RVW.22.00086"},{"key":"bibr24-10225536241268789","doi-asserted-by":"publisher","DOI":"10.5435\/JAAOS-D-20-00846"},{"key":"bibr25-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1302\/2058-5241.5.190092"},{"key":"bibr26-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1097\/CORR.0000000000002619"},{"key":"bibr27-10225536241268789","doi-asserted-by":"publisher","DOI":"10.1007\/s00063-023-01019-6"},{"key":"bibr28-10225536241268789","unstructured":"American Academy of Orthopaedic Surgeons. Artificial intelligence proves to be an effective tool for documenting orthopaedic encounters in hand surgery. https:\/\/www.prnewswire.com\/news-releases\/artificial-intelligence-proves-to-be-an-effective-tool-for-documenting-orthopaedic-encounters-in-hand-surgery-301764164.html (accessed 26 June 2023)."}],"container-title":["Journal of Orthopaedic Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10225536241268789","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10225536241268789","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10225536241268789","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T23:31:33Z","timestamp":1777678293000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10225536241268789"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1]]},"references-count":28,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10.1177\/10225536241268789"],"URL":"https:\/\/doi.org\/10.1177\/10225536241268789","relation":{},"ISSN":["1022-5536","2309-4990"],"issn-type":[{"value":"1022-5536","type":"print"},{"value":"2309-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1]]},"article-number":"10225536241268789"}}