{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T17:54:33Z","timestamp":1792173273426,"version":"4.4.0"},"reference-count":21,"publisher":"American Veterinary Medical Association (AVMA)","issue":"11","license":[{"start":{"date-parts":[[2026,11,1]],"date-time":"2026-11-01T00:00:00Z","timestamp":1793491200000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["javma"],"published-print":{"date-parts":[[2026,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate the diagnostic performance of multiple large language models (LLMs) against expert consensus in determining surgical intervention needs for feline metacarpal and metatarsal fractures.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this retrospective study (December 2023 to February 2025), 73 clinical cases of feline metacarpal and metatarsal fractures were evaluated. Two board-certified veterinary orthopedic surgeons established a reference standard for surgical versus conservative management. Five LLMs (ChatGPT, version 5.2 [OpenAI Inc]; Gemini, version 3 Pro [Alphabet Inc]; Grok, version 4.1 [SpaceXAI]; Qwen, version 3.5 [Alibaba Cloud]; and Claude Sonnet, version 4.5 [Anthropic PBC]; and Claude Sonnet, version 4.5 [Anthropic PBC]) assessed anonymized case summaries using a standardized zero-shot prompt. Model recommendations were compared with the reference standard to calculate accuracy, sensitivity, specificity, and the Cohen \u03ba.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The reference standard classified 49 cases (67.1%) as surgical and 24 (32.9%) as conservative. ChatGPT achieved the highest performance (accuracy, 84.9%; sensitivity, 79.6%; specificity, 95.8%; \u03ba = 0.69). Other models showed lower performance; Qwen and Claude Sonnet failed to identify any surgical cases (0% sensitivity). A systematic bias toward conservative management was observed across all models, causing high false-negative rates (undertriage).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLMs demonstrate highly variable diagnostic performance and a systematic risk of undertriage in feline fracture assessment. While top-performing models approach expert-level agreement, others fail in critical clinical scenarios.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Clinical Relevance<\/jats:title>\n                    <jats:p>Independent LLM use for surgical decision-making in feline orthopedics is not recommended due to undertriage risks. These tools require strict clinician oversight for preliminary triage.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2460\/javma.26.03.0227","type":"journal-article","created":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T16:23:55Z","timestamp":1786724635000},"page":"1-6","source":"Crossref","is-referenced-by-count":0,"title":["Large language models demonstrate variable diagnostic performance and a systematic risk of undertriage in surgical triage of feline metacarpal and metatarsal fractures"],"prefix":"10.2460","volume":"264","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2620-897X","authenticated-orcid":true,"given":"S\u0131tk\u0131can","family":"Okur","sequence":"first","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9402-5679","authenticated-orcid":true,"given":"\u00c7a\u011flar","family":"\u00d6zkalip\u00e7i","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2787-1249","authenticated-orcid":true,"given":"B\u00fc\u015fra","family":"Baykal","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8394-003X","authenticated-orcid":true,"given":"Tu\u011f\u00e7e","family":"Kartal","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8855-8847","authenticated-orcid":true,"given":"M\u00fcmin G\u00f6khan","family":"\u015eenocak","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9798-8638","authenticated-orcid":true,"given":"Ay\u015fe G\u00f6lgeli","family":"Bedir","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Atat\u00fcrk University, Erzurum, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5890-8271","authenticated-orcid":true,"given":"Latif Emrah","family":"Yanmaz","sequence":"additional","affiliation":[{"name":"Department of Surgery, Faculty of Veterinary Medicine, Burdur Mehmet Akif Ersoy University, Burdur, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1272","reference":[{"issue":"4","key":"ref1","series-title":"Vet Surg","first-page":"382","article-title":"Intramedullary pinning of metacarpal and metatarsal fractures in cats using a simple distraction technique","volume":"36","author":"Degasperi B","year":"2007","unstructured":"Degasperi B, Gradner G, Dupr\u00e9 G. Intramedullary pinning of metacarpal and metatarsal fractures in cats using a simple distraction technique. Vet Surg. 2007;36(4):382-388. doi:10.1111\/j.1532-950X.2007.00279.x"},{"issue":"11","key":"ref2","series-title":"J Small Anim Pract","first-page":"661","article-title":"A survey of bone fractures in the dog and cat","volume":"20","author":"Phillips IR","year":"1979","unstructured":"Phillips IR. A survey of bone fractures in the dog and cat. J Small Anim Pract. 1979;20(11):661-674. doi:10.1111\/j.1748-5827.1979.tb06679.x"},{"issue":"8","key":"ref3","series-title":"Vet Surg","first-page":"1015","article-title":"Combined intramedullary and external skeletal fixation of metatarsal and metacarpal fractures in 12 dogs and 19 cats","volume":"40","author":"Fitzpatrick N","year":"2011","unstructured":"Fitzpatrick N, Riordan JO, Smith TJ, Modlinska JH, Tucker R, Yeadon R. Combined intramedullary and external skeletal fixation of metatarsal and metacarpal fractures in 12 dogs and 19 cats. Vet Surg. 2011;40(8):1015-1022. doi:10.1111\/j.1532-950X.2011.00885.x"},{"issue":"2","key":"ref4","series-title":"Eurasian J Vet Sci","first-page":"47","article-title":"Evaluation of conservative treatment or osteosynthesis in the treatment of cats with metacarpal and\/or metatarsal fractures: a retrospective study","volume":"40","author":"Calt\u0131ner H","year":"2024","unstructured":"Calt\u0131ner H, Satici I, Arican M. Evaluation of conservative treatment or osteosynthesis in the treatment of cats with metacarpal and\/or metatarsal fractures: a retrospective study. Eurasian J Vet Sci. 2024;40(2):47-57. doi:10.15312\/EurasianJVetSci.2024.425"},{"issue":"3","key":"ref5","series-title":"Vet Comp Orthop Traumatol","first-page":"123","article-title":"Conservative versus surgical treatment of metacarpal and metatarsal fractures in dogs","volume":"13","author":"Kapatkin A","year":"2000","unstructured":"Kapatkin A, Howe-Smith R, Shofer F. Conservative versus surgical treatment of metacarpal and metatarsal fractures in dogs. Vet Comp Orthop Traumatol. 2000;13(3):123-127."},{"issue":"1","key":"ref6","series-title":"Van Vet J","first-page":"1","article-title":"Evaluation of the treatment of feline metacarpal and metatarsal fractures using dowel pinning technique","volume":"33","author":"Karsl\u0131 B","year":"2022","unstructured":"Karsl\u0131 B. Evaluation of the treatment of feline metacarpal and metatarsal fractures using dowel pinning technique. Van Vet J. 2022;33(1):1-5. doi:10.36483\/vanvetj.986259"},{"key":"ref7","series-title":"Small Animal Surgery","author":"Fossum TW","year":"2018","unstructured":"Fossum TW. Small Animal Surgery. 5th ed. Elsevier; 2018."},{"issue":"1","key":"ref8","series-title":"J Med Syst","first-page":"33","article-title":"Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios","volume":"47","author":"Cascella M","year":"2023","unstructured":"Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33. doi:10.1007\/s10916-023-01925-4"},{"issue":"2","key":"ref9","series-title":"JAMA Surg","first-page":"148","article-title":"Artificial intelligence and surgical decision-making","volume":"155","author":"Loftus TJ","year":"2020","unstructured":"Loftus TJ, Tighe PJ, Filiberto AC,  Artificial intelligence and surgical decision-making. JAMA Surg. 2020;155(2):148-158. doi:10.1001\/jamasurg.2019.4917"},{"issue":"4","key":"ref10","series-title":"J Hand Surg Glob Online","first-page":"524","article-title":"Enhancing hand fracture care: a prospective study of artificial intelligence application with ChatGPT","volume":"6","author":"Atkinson CJ","year":"2024","unstructured":"Atkinson CJ, Seth I, Seifman MA, Rozen WM, Cuomo R. Enhancing hand fracture care: a prospective study of artificial intelligence application with ChatGPT. J Hand Surg Glob Online. 2024;6(4):524-528. doi:10.1016\/j.jhsg.2024.03.014"},{"issue":"7-8","key":"ref11","series-title":"ANZ J Surg","first-page":"1391","article-title":"Examining the role of ChatGPT in the management of distal radius fractures: insights into its accuracy and consistency","volume":"94","author":"Knee CJ","year":"2024","unstructured":"Knee CJ, Campbell RJ, Graham DJ, Handford C, Symes M, Sivakumar BS. Examining the role of ChatGPT in the management of distal radius fractures: insights into its accuracy and consistency. ANZ J Surg. 2024;94(7-8):1391-1396. doi:10.1111\/ans.19143"},{"key":"ref12","series-title":"Res Vet Sci","first-page":"105317","article-title":"Artificial intelligence in veterinary diagnostic imaging: perspectives and limitations","volume":"175","author":"Burti S","year":"2024","unstructured":"Burti S, Banzato T, Coghlan S, Wodzinski M, Bendazzoli M, Zotti A. Artificial intelligence in veterinary diagnostic imaging: perspectives and limitations. Res Vet Sci. 2024;175:105317. doi:10.1016\/j.rvsc.2024.105317"},{"issue":"suppl 1","key":"ref13","series-title":"Vet Radiol Ultrasound","first-page":"851","article-title":"Artificial intelligence in veterinary diagnostic imaging: a literature review","volume":"63","author":"Hennessey E","year":"2022","unstructured":"Hennessey E, DiFazio M, Hennessey R, Cassel N. Artificial intelligence in veterinary diagnostic imaging: a literature review. Vet Radiol Ultrasound. 2022;63(suppl 1):851-870. doi:10.1111\/vru.13163"},{"issue":"3","key":"ref14","series-title":"ANZ J Surg","first-page":"287","article-title":"Assessment of the bias of artificial intelligence generated images and large language models on their depiction of a surgeon","volume":"94","author":"Cevik J","year":"2024","unstructured":"Cevik J, Lim B, Seth I,  Assessment of the bias of artificial intelligence generated images and large language models on their depiction of a surgeon. ANZ J Surg. 2024;94(3):287-294. doi:10.1111\/ans.18792"},{"issue":"1","key":"ref15","series-title":"Diagn Pathol","first-page":"43","article-title":"Challenges and barriers of using large language models (LLMs) such as ChatGPT for diagnostic medicine with a focus on digital pathology - a recent scoping review","volume":"19","author":"Ullah E","year":"2024","unstructured":"Ullah E, Parwani A, Baig MM, Singh R. Challenges and barriers of using large language models (LLMs) such as ChatGPT for diagnostic medicine with a focus on digital pathology - a recent scoping review. Diagn Pathol. 2024;19(1):43. doi:10.1186\/s13000-024-01464-7"},{"issue":"9596","key":"ref16","series-title":"Lancet","first-page":"1453","article-title":"The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies","volume":"370","author":"von Elm E","year":"2007","unstructured":"von Elm E, Altman DG, Egger M,  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453-1457. doi:10.1016\/S0140-6736(07)61602-X"},{"key":"ref17","series-title":"Proceedings of the 12th ESVOT Congress","first-page":"24","article-title":"High-rise syndrome in cats","author":"Boudrieau RJ","year":"2004","unstructured":"Boudrieau RJ. High-rise syndrome in cats. In: Proceedings of the 12th ESVOT Congress. European Society of Veterinary Orthopaedics and Traumatology; 2004:24\u201325."},{"issue":"3","key":"ref18","series-title":"Lancet Digit Health","first-page":"e107","article-title":"ChatGPT: the future of discharge summaries?","volume":"5","author":"Patel SB","year":"2023","unstructured":"Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health. 2023;5(3):e107-e108. doi:10.1016\/S2589-7500(23)00021-3"},{"issue":"9","key":"ref19","series-title":"ANZ J Surg","first-page":"2270","article-title":"ChatGPT in surgical research and practice: a threat to academic integrity, authorship, and divergent thinking","volume":"93","author":"Bulloch G","year":"2023","unstructured":"Bulloch G, Seth I, Lee CHA. ChatGPT in surgical research and practice: a threat to academic integrity, authorship, and divergent thinking. ANZ J Surg. 2023;93(9):2270-2271. doi:10.1111\/ans.18544"},{"issue":"2","key":"ref20","series-title":"Vet Radiol Ultrasound","first-page":"e70012","article-title":"A pilot study on using an artificial intelligence algorithm to identify urolith composition through abdominal radiographs in the dog","volume":"66","author":"Canejo-Teixeira R","year":"2025","unstructured":"Canejo-Teixeira R, Carvalho M, Semi\u00e3o Teixeira G,  A pilot study on using an artificial intelligence algorithm to identify urolith composition through abdominal radiographs in the dog. Vet Radiol Ultrasound. 2025;66(2):e70012. doi:10.1111\/vru.70012"},{"key":"ref21","series-title":"Proceedings of the Annual Conference of KIPS","first-page":"922","article-title":"Development of an AI-based deep learning model for dog obesity assessment and customised feed recommendation system","author":"Kim YM","year":"2024","unstructured":"Kim YM, Chun SE, Kim CR. Development of an AI-based deep learning model for dog obesity assessment and customised feed recommendation system. In: Proceedings of the Annual Conference of KIPS. Korea Information Processing Society; 2024:922\u2013923."}],"container-title":["Journal of the American Veterinary Medical Association"],"original-title":[],"link":[{"URL":"https:\/\/avmajournals.avma.org\/view\/journals\/javma\/264\/11\/javma.26.03.0227.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/avmajournals.avma.org\/downloadpdf\/journals\/javma\/264\/11\/javma.26.03.0227.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/avmajournals.avma.org\/downloadpdf\/journals\/javma\/264\/11\/javma.26.03.0227.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T17:25:12Z","timestamp":1792171512000},"score":1,"resource":{"primary":{"URL":"https:\/\/avmajournals.avma.org\/view\/journals\/javma\/264\/11\/javma.26.03.0227.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,11,1]]},"references-count":21,"journal-issue":{"issue":"11"},"URL":"https:\/\/doi.org\/10.2460\/javma.26.03.0227","relation":{},"ISSN":["0003-1488"],"issn-type":[{"value":"0003-1488","type":"print"}],"subject":[],"published":{"date-parts":[[2026,11,1]]}}}