{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T18:27:06Z","timestamp":1795199226697,"version":"4.10.3"},"reference-count":64,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T00:00:00Z","timestamp":1756166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Vet. Sci."],"abstract":"<jats:p>The integration of artificial intelligence, particularly large language models (LLMs), into veterinary education and practice presents promising opportunities, yet their performance in veterinary-specific contexts remains understudied. This research comparatively evaluated the performance of nine advanced LLMs (ChatGPT o1Pro, ChatGPT 4o, ChatGPT 4.5, Grok 3, Gemini 2, Copilot, DeepSeek R1, Qwen 2.5 Max, and Kimi 1.5) on 250 multiple-choice questions (MCQs) sourced from a veterinary undergraduate final qualifying examination. Questions spanned various species, clinical topics and reasoning stages, and included both text-based and image-based formats. ChatGPT o1Pro and ChatGPT 4.5 achieved the highest overall performance, with correct response rates of 90.4 and 90.8% respectively, demonstrating strong agreement with the gold standard across most categories, while Kimi 1.5 showed the lowest performance at 64.8%. Performance consistently declined with increased question difficulty and was generally lower for image-based than text-based questions. OpenAI models excelled in visual interpretation compared to previous studies. Disparities in performance were observed across specific clinical reasoning stages and veterinary subdomains, highlighting areas for targeted improvement. This study underscores the promising role of LLMs as supportive tools for quality assurance in veterinary assessment design and indicates key factors influencing their performance, including question difficulty, format, and domain-specific training data.<\/jats:p>","DOI":"10.3389\/fvets.2025.1616566","type":"journal-article","created":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T13:04:39Z","timestamp":1756213479000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Performance of large language models on veterinary undergraduate multiple-choice examinations: a comparative evaluation"],"prefix":"10.3389","volume":"12","author":[{"given":"Santiago","family":"Alonso Sousa","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Syed Saad Ul Hassan","family":"Bukhari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paulo Vinicius","family":"Steagall","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pawe\u0142 M.","family":"B\u0119czkowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antonio","family":"Giuliano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kate J.","family":"Flay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,8,26]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"e105","DOI":"10.1016\/S2589-7500(23)00019-5","article-title":"Generating scholarly content with ChatGPT: ethical challenges for medical publishing","volume":"5","author":"Liebrenz","year":"2023","journal-title":"Lancet Digit Health"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"e0000205","DOI":"10.1371\/journal.pdig.0000205","article-title":"ChatGPT passing USMLE shines a spotlight on the flaws of medical education","volume":"2","author":"Mbakwe","year":"2023","journal-title":"PLOS Digit Health"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"278","DOI":"10.7861\/clinmed.2023-0078","article-title":"Early applications of ChatGPT in medical practice, education and research","volume":"23","author":"Sedaghat","year":"2023","journal-title":"Clin Med (Lond)"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"100943","DOI":"10.1016\/j.patter.2024.100943","article-title":"Can large language models reason about medical questions?","volume":"5","author":"Li\u00e9vin","year":"2024","journal-title":"Patterns"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"100017","DOI":"10.1016\/j.metrad.2023.100017","article-title":"Summary of ChatGPT-related research and perspective towards the future of large language models","volume":"1","author":"Liu","year":"2023","journal-title":"Meta-Radiol"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"926","DOI":"10.1002\/ase.2270","article-title":"The rise of ChatGPT: exploring its potential in medical education","volume":"17","author":"Lee","year":"2024","journal-title":"Anat Sci Educ"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","article-title":"ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports","volume":"34","author":"Jeblick","year":"2024","journal-title":"Eur Radiol"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1007\/s10916-024-02075-x","article-title":"ChatGPT: a conceptual review of applications and utility in the field of medicine","volume":"48","author":"Rao","year":"2024","journal-title":"J Med Syst"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"e13930","DOI":"10.2196\/13930","article-title":"Applications and challenges of implementing artificial intelligence in medical education: integrative review","volume":"5","author":"Chan","year":"2019","journal-title":"JMIR Med Educ"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"1169595","DOI":"10.3389\/frai.2023.1169595","article-title":"ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","volume":"6","author":"Dave","year":"2023","journal-title":"Front Artif Intell"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"237","DOI":"10.5694\/mja2.51932","article-title":"This too shall pass: the performance of ChatGPT-3.5, ChatGPT-4 and new Bing in an Australian medical licensing examination","volume":"219","author":"Kleinig","year":"2023","journal-title":"Med J Aust"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e48039","DOI":"10.2196\/48039","article-title":"Performance of ChatGPT on the Peruvian national licensing medical examination: cross-sectional study","volume":"9","author":"Flores-Cohaila","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"e100815","DOI":"10.1136\/bmjhci-2023-100815","article-title":"ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model","volume":"30","author":"Ebrahimian","year":"2023","journal-title":"BMJ Health Care Inform"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1093\/ced\/llad255","article-title":"An original study of ChatGPT-3.5 and ChatGPT-4 dermatological knowledge level based on the dermatology specialty certificate examinations","volume":"49","author":"Lewandowski","year":"2023","journal-title":"Clin Exp Dermatol"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"e230582","DOI":"10.1148\/radiol.230582","article-title":"Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations","volume":"307","author":"Bhayana","year":"2023","journal-title":"Radiology"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"1125","DOI":"10.1093\/postmj\/qgad058","article-title":"The impact of chat generative pre-trained transformer (ChatGPT) on medical education","volume":"99","author":"Heng","year":"2023","journal-title":"Postgrad Med J"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"2181052","DOI":"10.1080\/10872981.2023.2181052","article-title":"The future of medical education and research: is ChatGPT a blessing or blight in disguise?","volume":"28","author":"Arif","year":"2023","journal-title":"Med Educ Online"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"e48785","DOI":"10.2196\/48785","article-title":"Opportunities, challenges, and future directions of generative artificial intelligence in medical education: scoping review","volume":"9","author":"Preiksaitis","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"934","DOI":"10.1016\/j.scib.2025.01.034","article-title":"Large language models for diabetes training: a prospective study","volume":"70","author":"Li","year":"2025","journal-title":"Sci Bull"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"682","DOI":"10.1002\/JPER.23-0514","article-title":"Artificial intelligence in dental education: ChatGPT\u2019s performance on the periodontic in-service examination","volume":"95","author":"Danesh","year":"2024","journal-title":"J Periodontol"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"269","DOI":"10.4174\/astr.2023.104.5.269","article-title":"ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models","volume":"104","author":"Oh","year":"2023","journal-title":"Ann Surg Treat Res"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"1256459","DOI":"10.3389\/fonc.2023.1256459","article-title":"Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer?","volume":"13","author":"Ku\u015fcu","year":"2023","journal-title":"Front Oncol"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3350\/cmh.2023.0089","article-title":"Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma","volume":"29","author":"Yeo","year":"2023","journal-title":"Clin Mol Hepatol"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"e56532","DOI":"10.2196\/56532","article-title":"The accuracy and capability of artificial intelligence solutions in health care examinations and certificates: systematic review and meta-analysis","volume":"26","author":"Waldock","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"104620","DOI":"10.1016\/j.jbi.2024.104620","article-title":"Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis","volume":"151","author":"Wei","year":"2024","journal-title":"J Biomed Inform"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"e20240075","DOI":"10.3138\/jvme-2024-0075","article-title":"A survey of veterinary student perceptions on integrating ChatGPT in veterinary education through AI-driven exercises","author":"Alonso Sousa","year":"2024","journal-title":"J Vet Med Educ"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"1395934","DOI":"10.3389\/fvets.2024.1395934","article-title":"ChatGPT in veterinary medicine: a practical guidance of generative artificial intelligence in clinics, education, and research","volume":"11","author":"Chu","year":"2024","journal-title":"Front Vet Sci"},{"key":"ref30","doi-asserted-by":"publisher","first-page":"692","DOI":"10.2460\/javma.23.12.0666","article-title":"Two artificial intelligence models underperform on examinations in a veterinary curriculum","volume":"262","author":"Coleman","year":"2024","journal-title":"J Am Vet Med Assoc"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"e46885","DOI":"10.2196\/46885","article-title":"The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers","volume":"9","author":"Eysenbach","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.21203\/rs.3.rs-2566942\/v1","article-title":"Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the chat-GPT model","author":"Johnson","year":"2023","journal-title":"Res Sq"},{"key":"ref33","volume-title":"Day one competences","year":"2022"},{"key":"ref34","article-title":"Competency-based veterinary education: competency framework","year":"2024","journal-title":"AAVMC"},{"key":"ref35","volume-title":"Day One Competencies","year":"2024"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"497","DOI":"10.3138\/jvme.0918-116r","article-title":"Multiple-choice questions in small animal medicine: an analysis of cognitive level and structural reliability, and the impact of these characteristics on student performance","volume":"47","author":"Cook","year":"2020","journal-title":"J Vet Med Educ"},{"key":"ref37","volume-title":"A guide to assessment in veterinary medical education","author":"Baillie","year":"2022"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"762","DOI":"10.3138\/jvme-2023-0061","article-title":"Measures for quality assurance of electronic examinations in a veterinary medical curriculum","volume":"51","author":"Richter","year":"2024","journal-title":"J Vet Med Educ"},{"key":"ref39","year":""},{"key":"ref40","year":"2012"},{"key":"ref41","first-page":"89","article-title":"Setting performance standards on educational assessments and criteria for evaluating the process","volume-title":"Setting performance standards: Concepts, methods, and perspectives","author":"Hambleton","year":"2001"},{"key":"ref42","doi-asserted-by":"publisher","first-page":"799","DOI":"10.1186\/s12885-023-11325-z","article-title":"Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters","volume":"23","author":"Li","year":"2023","journal-title":"BMC Cancer"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Lessons in biostatistics: interrater reliability\u2014the kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem Med Zagreb"},{"key":"ref44","year":"2020"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.48550\/arXiv.2201.11903","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Proces Syst"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"7075798","DOI":"10.48550\/arXiv.2305.15408","article-title":"Towards revealing the mystery behind chain of thought: a theoretical perspective","volume":"36","author":"Feng","year":"2023","journal-title":"Adv Neural Inf Proces Syst"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.21706.89283","article-title":"Is deep seek-R1 a game changer in healthcare? A seed review","author":"Egger","year":"2025","journal-title":"Preprint"},{"key":"ref48","article-title":"KIMI K1.5: Scaling reinforcement learning with LLMs","year":"2025","journal-title":"arXiv"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"e1464","DOI":"10.1002\/vms3.1464","article-title":"AI chatbots in pet health care: opportunities and challenges for owners","volume":"10","author":"Jokar","year":"2024","journal-title":"Vet Med Sci"},{"key":"ref50","first-page":"100161","article-title":"Impact of the internet on veterinary surgery","volume-title":"Vet Anim Sci","author":"Souza","year":"2020"},{"key":"ref51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/15589447241279460","article-title":"The comparative performance of large language models on the hand surgery self-assessment examination","author":"Chen","year":"2024","journal-title":"Hand"},{"key":"ref52","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1007\/s00417-024-06625-4","article-title":"Gemini AI vs. ChatGPT: a comprehensive examination alongside ophthalmology residents in medical knowledge","volume":"263","author":"Bahir","year":"2025","journal-title":"Graefes Arch Clin Exp Ophthalmol"},{"key":"ref53","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/j.surge.2024.11.008","article-title":"Enhancements in artificial intelligence for medical examinations: a leap from ChatGPT 3.5 to ChatGPT 4.0 in the FRCS trauma & orthopaedics examination","volume":"23","author":"Khan","year":"2025","journal-title":"Surgeon"},{"key":"ref54","author":"He","year":"2016"},{"key":"ref55","first-page":"159","article-title":"Language models are few-shot learners","volume":"34","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Proces Syst"},{"key":"ref56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/2158244020947432","article-title":"Guidelines on writing multiple choice questions: a well-received and effective faculty development intervention","volume":"10","author":"Przymusza\u0142a","year":"2020","journal-title":"SAGE Open"},{"key":"ref57","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1097\/00001888-200202000-00016","article-title":"The quality of in-house medical school examinations","volume":"77","author":"Jozefowicz","year":"2002","journal-title":"Acad Med"},{"key":"ref58","doi-asserted-by":"publisher","first-page":"866","DOI":"10.12669\/pjms.334.12887","article-title":"Evaluation of cognitive levels and item writing flaws in medical pharmacology internal assessment examinations","volume":"33","author":"Tariq","year":"2017","journal-title":"Pak J Med Sci"},{"key":"ref59","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1111\/j.1365-2923.2007.02957.x","article-title":"Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments","volume":"42","author":"Tarrant","year":"2008","journal-title":"Med Educ"},{"key":"ref60","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1097\/NND.0b013e318286c2f1","article-title":"Assessment of item-writing flaws in multiple-choice questions","volume":"29","author":"Nedeau-Cayo","year":"2013","journal-title":"J Nurses Prof Dev"},{"key":"ref61","doi-asserted-by":"publisher","first-page":"100343","DOI":"10.1016\/j.caeai.2024.100343","article-title":"Assisting quality assurance of examination tasks: using a GPT model and Bayesian testing for formative assessment","volume":"8","author":"Willert","year":"2025","journal-title":"Comput Educ Artif Intell"},{"key":"ref62","doi-asserted-by":"publisher","first-page":"3","DOI":"10.24744\/jdea.40.13","article-title":"Comparison of the performance on the Japanese National Dental Examination using GPT-3.5 and GPT-4","volume":"40","author":"Morishita","year":"2024","journal-title":"Jpn J Dent Educ Assoc"},{"key":"ref63","doi-asserted-by":"publisher","first-page":"1490030","DOI":"10.3389\/fvets.2024.1490030","article-title":"Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records","volume":"11","author":"Wulcan","year":"2025","journal-title":"Front Vet Sci"},{"key":"ref64","doi-asserted-by":"publisher","DOI":"10.55041\/IJSREM17694","article-title":"Multilingual prompting in LLMs: investigating the accuracy and performance","volume":"7","author":"Vadlapati","year":"2023","journal-title":"Int J Scientific Res Eng Manag (IJSREM)"}],"container-title":["Frontiers in Veterinary Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fvets.2025.1616566\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T13:04:42Z","timestamp":1756213482000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fvets.2025.1616566\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,26]]},"references-count":64,"alternative-id":["10.3389\/fvets.2025.1616566"],"URL":"https:\/\/doi.org\/10.3389\/fvets.2025.1616566","relation":{},"ISSN":["2297-1769"],"issn-type":[{"value":"2297-1769","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,26]]},"article-number":"1616566"}}