{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T12:05:31Z","timestamp":1784549131197,"version":"3.55.0"},"reference-count":14,"publisher":"FapUNIFESP (SciELO)","issue":"4","license":[{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"am","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Acta ortop. bras."],"abstract":"<jats:p>ABSTRACT  Introduction: AI is increasingly used for medical education and assessment, yet effectiveness on specialized exams remains underexplored. We aim to compare multiple AI models with national human averages on SBCOC entrance exams (2021\u20132023), evaluate answer accuracy and citation reliability, and contrast models.  Methods: Five models\u2014ChatGPT-4 (standard and literature-trained), ChatGPT-o1-pro, Gemini, and Meta Llama 3.1\u2014answered official SBCOC exams (50 items\/year) individually using a unified prompt and identical images. Each response included one choice (A\u2013D) and a cited source. Outcomes were accuracy and source reliability (peer-reviewed articles\/textbooks versus websites). Statistics used chi-square tests and one-way ANOVA with Tukey post-hoc (p&lt;0.05).  Results: Across 150 questions, ChatGPT-o1-pro led (66%, 62%, 68%) and exceeded human national averages each year (p&lt;0.05). Gemini (40%, 28%, 38%) and Meta Llama 3.1 (32%, 44%, 50%) underperformed relative to humans, while both ChatGPT-4 versions hovered near the \u226550% pass threshold without significant differences. Regarding sources, ChatGPT-o1-pro and Llama predominantly cited articles or books, ChatGPT-4 alternated between literature and websites, and Gemini did not specify references.  Conclusions: ChatGPT-o1-pro outperformed the national human average and mostly used credible sources; ChatGPT-4 matched humans, while Gemini and Meta Llama 3.1 lagged. Level of evidence IV; Case series.<\/jats:p>","DOI":"10.1590\/1413-785220263404e300345","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T11:19:28Z","timestamp":1784546368000},"source":"Crossref","is-referenced-by-count":0,"title":["USE OF ARTIFICIAL INTELLIGENCE FOR THE EVALUATION OF THE ENTRANCE EXAMINATION OF THE BRAZILIAN SOCIETY OF SHOULDER AND ELBOW SURGERY"],"prefix":"10.1590","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5627-921X","authenticated-orcid":false,"given":"Caio Henrique","family":"Kenchian","sequence":"first","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0266-850X","authenticated-orcid":false,"given":"Karolina Stephany Pereira","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7116-7057","authenticated-orcid":false,"given":"Lucas Pereira","family":"Sarmento","sequence":"additional","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8887-9803","authenticated-orcid":false,"given":"Marcelo Silveira","family":"Teixeira Filho","sequence":"additional","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9289-7983","authenticated-orcid":false,"given":"Renato Aroca","family":"Zan","sequence":"additional","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9539-4545","authenticated-orcid":false,"given":"Marcel Jun Sugawara","family":"Tamaoki","sequence":"additional","affiliation":[{"name":"Universidade Federal de Sao Paulo,  Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"530","published-online":{"date-parts":[[2026]]},"reference":[{"issue":"3-4","key":"ref1","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1016\/0025-5564(75)90047-4","article-title":"A model of inexact reasoning in medicine","volume":"23","author":"Shortliffe EH","year":"1975","journal-title":"Math Biosci"},{"issue":"11","key":"ref2","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care\u2014addressing ethical challenges","volume":"378","author":"Char DS","year":"2018","journal-title":"N Engl J Med"},{"key":"ref3","article-title":"Uso da IA no Instituto de Radiologia da USP permitir\u00e1 laudos mais \u00e1geis","year":"2024","journal-title":"Jornal da USP [Internet]"},{"issue":"7788","key":"ref4","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","article-title":"International evaluation of an AI system for breast cancer screening","volume":"577","author":"McKinney SM","year":"2020","journal-title":"Nature"},{"issue":"25","key":"ref5","doi-asserted-by":"crossref","first-page":"2058","DOI":"10.1093\/eurheartj\/ehz056","article-title":"Deep learning for cardiovascular medicine: a practical primer","volume":"40","author":"Krittanawong C","year":"2019","journal-title":"Eur Heart J"},{"issue":"2","key":"ref6","doi-asserted-by":"crossref","first-page":"94","DOI":"10.7861\/futurehosp.6-2-94","article-title":"The potential for artificial intelligence in healthcare","volume":"6","author":"Davenport T","year":"2019","journal-title":"Future Healthc J"},{"issue":"2","key":"ref7","doi-asserted-by":"crossref","DOI":"10.2196\/16048","article-title":"Introducing artificial intelligence training in medical education","volume":"5","author":"Paranjape K","year":"2019","journal-title":"JMIR Med Educ"},{"issue":"3","key":"ref8","article-title":"Evaluating ChatGPT performance on the orthopaedic in-training examination","volume":"8","author":"Kung JE","year":"2023","journal-title":"JB JS Open Access"},{"key":"ref9","doi-asserted-by":"crossref","first-page":"e160","DOI":"10.1016\/j.wneu.2023.08.042","article-title":"GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions","volume":"179","author":"Guerra GA","year":"2023","journal-title":"World Neurosurg"},{"issue":"1","key":"ref10","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson A","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref11","article-title":"Capabilities of Gemini models in medicine","author":"Saab K","year":"2024","journal-title":"arXiv [preprint]"},{"issue":"4","key":"ref12","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: past, present and future","volume":"2","author":"Jiang F","year":"2017","journal-title":"Stroke Vasc Neurol"},{"issue":"101","key":"ref13","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1590\/s0103-4014.2021.35101.003","article-title":"Intelig\u00eancia artificial: riscos, benef\u00edcios e uso respons\u00e1vel","volume":"35","author":"Carvalho ACPLF","year":"2021","journal-title":"Estudos Avan\u00e7ados"},{"issue":"spe1","key":"ref14","article-title":"Evaluation of the performance of ChatGPT\/artificial intelligence in the multiple-choice test to obtain the title of specialist in orthopedics and traumatology","volume":"33","author":"Costa LPDB","year":"2025","journal-title":"Acta Ortop Bras"}],"container-title":["Acta Ortop\u00e9dica Brasileira"],"original-title":["USO DA INTELIG\u00caNCIA ARTIFICIAL PARA AVALIA\u00c7\u00c3O DO EXAME DE INGRESSO DA SOCIEDADE BRASILEIRA DE CIRURGIA DO OMBRO E COTOVELO"],"link":[{"URL":"http:\/\/www.scielo.br\/scielo.php?script=sci_pdf&pid=S1413-78522026000405401&tlng=en","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T11:19:31Z","timestamp":1784546371000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.scielo.br\/scielo.php?script=sci_arttext&pid=S1413-78522026000405401&tlng=en"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":14,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026]]}},"alternative-id":["S1413-78522026000405401"],"URL":"https:\/\/doi.org\/10.1590\/1413-785220263404e300345","relation":{},"ISSN":["1809-4406","1413-7852"],"issn-type":[{"value":"1809-4406","type":"electronic"},{"value":"1413-7852","type":"print"}],"subject":[],"published":{"date-parts":[[2026]]},"article-number":"e300345"}}