{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T23:48:03Z","timestamp":1775519283588,"version":"3.50.1"},"reference-count":27,"publisher":"Georg Thieme Verlag KG","issue":"02","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2026,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This study evaluates the performance of multiple large language models (LLMs) on specialized otolaryngology knowledge, comparing OpenAI's GPT-4 Turbo with 10 commercially available models to assess their potential utility in otolaryngology medical education.<\/jats:p>\n                  <jats:p>A total of 1,075 questions from OTO QUEST, the official self-assessment resource of the American Academy of Otolaryngology\u2013Head and Neck Surgery, were administered to GPT-4 Turbo using a zero-shot approach. Accuracy was analyzed using logistic regression, adjusting for question difficulty, year, and subspecialty. Performance was subsequently compared with 10 other commercial models (including Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o) using the 1,075 question dataset, evaluated with Cochran's Q test (p\u2009&lt;\u20090.001) and McNemar's pairwise comparison.<\/jats:p>\n                  <jats:p>GPT-4 Turbo achieved an overall accuracy of 72.09% (95% confidence interval [CI]: 69.3\u201374.7%) across 1,075 questions. It performed best in Practice Management questions (odds ratio [OR]\u2009=\u20093.93, 95% CI: 1.12\u201313.73, p\u2009=\u20090.032) and declined in accuracy when faced with questions of moderate and hard difficulty (OR\u2009=\u20090.21, 95% CI: 0.16\u20130.29, p\u2009&lt;\u20090.001 and OR\u2009=\u20090.04, 95% CI: 0.01\u20130.10, p\u2009&lt;\u20090.001, respectively). In comparative analysis, Grok-3 ranked highest with 76.3% accuracy (95% CI: 73.6\u201378.7%), followed by Claude-3.5-sonnet (73.0%, 95% CI: 70.3\u201375.6%) and GPT-4o (69.9%, 95% CI: 67.1\u201372.5%), with GPT-4 Turbo application programming interface ranking fourth.<\/jats:p>\n                  <jats:p>This comprehensive model comparison reveals that while major commercial LLMs show promising capabilities in specialized medical knowledge assessment, they demonstrate an apparent accuracy plateau around 73 to 76%. These findings suggest current general-purpose LLMs may require specialized training approaches to advance beyond this performance threshold in medical domains.<\/jats:p>","DOI":"10.1055\/a-2835-4634","type":"journal-article","created":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T22:52:56Z","timestamp":1775515976000},"page":"194-203","source":"Crossref","is-referenced-by-count":0,"title":["Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions"],"prefix":"10.1055","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-4816-7545","authenticated-orcid":false,"given":"Ryan","family":"Cook","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abner","family":"Kahan","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas","family":"Scharfenberger","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Tasoulas","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology-Head and Neck Surgery, Thomas Jefferson University, Philadelphia, Pennsylvania, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noah","family":"Hawks-Ladds","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"Chouake","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology-Head and Neck Surgery, Montefiore Einstein, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6560-7418","authenticated-orcid":false,"given":"Sunit P.","family":"Jariwala","sequence":"additional","affiliation":[{"name":"Department of Medicine, Montefiore Einstein, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shitij","family":"Arora","sequence":"additional","affiliation":[{"name":"Department of Medicine, Montefiore Einstein, Bronx, New York, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"194","published-online":{"date-parts":[[2026,4,6]]},"reference":[{"issue":"03","key":"ref1","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1001\/jamainternmed.2018.7117","article-title":"Deep learning in medicine-promise, progress, and challenges","volume":"179","author":"F Wang","year":"2019","journal-title":"JAMA Intern Med"},{"issue":"01","key":"ref2","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1146\/annurev-bioeng-071516-044442","article-title":"Deep learning in medical image analysis","volume":"19","author":"D Shen","year":"2017","journal-title":"Annu Rev Biomed Eng"},{"issue":"01","key":"ref3","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1038\/s41746-020-00376-2","article-title":"Deep learning-enabled medical computer vision","volume":"4","author":"A Esteva","year":"2021","journal-title":"NPJ Digit Med"},{"key":"ref5","first-page":"1","volume-title":"Chat GPT & Google Bard AI: a review","author":"S K Singh","year":"2023"},{"issue":"02","key":"ref6","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"T H Kung","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"06","key":"ref7","doi-asserted-by":"crossref","first-page":"6112","DOI":"10.1007\/s12070-024-04935-x","article-title":"Performance of novel GPT-4 in otolaryngology knowledge assessment","volume":"76","author":"L Revercomb","year":"2024","journal-title":"Indian J Otolaryngol Head Neck Surg"},{"key":"ref8","first-page":"e68","article-title":"Performance of ChatGPT on the plastic surgery in-training examination","volume":"24","author":"B E Raine","year":"2024","journal-title":"Eplasty"},{"key":"ref9","first-page":"2215","article-title":"Performance of ChatGPT on nephrology test questions","volume":"10","author":"J Miao","year":"2023","journal-title":"Clin J Am Soc Nephrol"},{"key":"ref11","doi-asserted-by":"crossref","DOI":"10.1016\/j.xops.2023.100324","volume-title":"Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings","author":"F Antaki","year":"2023"},{"issue":"02","key":"ref12","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1002\/lary.31781","article-title":"Comparative assessment of otolaryngology knowledge among large language models","volume":"135","author":"D J Merlino","year":"2025","journal-title":"Laryngoscope"},{"issue":"05","key":"ref13","doi-asserted-by":"crossref","first-page":"1049","DOI":"10.1055\/a-2405-0138","article-title":"ChatGPT performs worse on USMLE-style ethics questions compared to medical knowledge questions","volume":"15","author":"T Danehy","year":"2024","journal-title":"Appl Clin Inform"},{"issue":"09","key":"ref14","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"P Hager","year":"2024","journal-title":"Nat Med"},{"issue":"05","key":"ref16","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1093\/jamia\/ocaf038","article-title":"Large language models are less effective at clinical prediction tasks than locally trained machine learning models","volume":"32","author":"K E Brown","year":"2025","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"ref18","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1038\/s44401-025-00018-3","article-title":"Enhancing medical coding efficiency through domain-specific fine-tuned large language models","volume":"2","author":"Z Hou","year":"2025","journal-title":"Npj Health Syst"},{"issue":"04","key":"ref19","doi-asserted-by":"crossref","first-page":"457","DOI":"10.20517\/ir.2024.27","article-title":"A survey of datasets in medicine for large language models","volume":"4","author":"D Zhang","year":"2024","journal-title":"Intell Robotics"},{"issue":"01","key":"ref20","doi-asserted-by":"crossref","first-page":"3280","DOI":"10.1038\/s41467-025-56989-2","article-title":"Benchmarking large language models for biomedical natural language processing applications and recommendations","volume":"16","author":"Q Chen","year":"2025","journal-title":"Nat Commun"},{"issue":"10","key":"ref22","doi-asserted-by":"crossref","first-page":"3386","DOI":"10.3390\/app10103386","article-title":"Comparing the quality and speed of sentence classification with modern language models","volume":"10","author":"K Fiok","year":"2020","journal-title":"Appl Sci (Basel)"},{"issue":"02","key":"ref24","doi-asserted-by":"crossref","first-page":"241","DOI":"10.4103\/idoj.idoj_221_24","article-title":"A comparative analysis of the performance of large language models and human respondents in dermatology","volume":"16","author":"A B Murthy","year":"2025","journal-title":"Indian Dermatol Online J"},{"issue":"S 01","key":"ref25","doi-asserted-by":"crossref","first-page":"i119","DOI":"10.1093\/bioinformatics\/btae238","article-title":"Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models","volume":"40","author":"M Jeong","year":"2024","journal-title":"Bioinformatics"},{"issue":"04","key":"ref26","doi-asserted-by":"crossref","first-page":"1042","DOI":"10.1002\/ohn.864","article-title":"ChatENT: augmented large language model for expert knowledge retrieval in otolaryngology\u2013head and neck surgery","volume":"171","author":"C Long","year":"2024","journal-title":"Otolaryngol Head Neck Surg"},{"issue":"01","key":"ref27","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"J Clusmann","year":"2023","journal-title":"Commun Med (Lond)"},{"issue":"06","key":"ref28","doi-asserted-by":"crossref","first-page":"631","DOI":"10.3390\/bioengineering12060631","article-title":"Large language models in healthcare and medical applications: a review","volume":"12","author":"S Maity","year":"2025","journal-title":"Bioengineering (Basel)"},{"key":"ref29","doi-asserted-by":"crossref","first-page":"e52935","DOI":"10.2196\/52935","article-title":"Evaluation of large language model performance and reliability for citations and references in scholarly writing: cross-disciplinary study","volume":"26","author":"J Mugaanyi","year":"2024","journal-title":"J Med Internet Res"},{"issue":"01","key":"ref30","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"E Asgari","year":"2025","journal-title":"NPJ Digit Med"},{"issue":"01","key":"ref31","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"S Shool","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"issue":"01","key":"ref32","doi-asserted-by":"crossref","first-page":"100184","DOI":"10.1016\/j.mcpdig.2024.11.005","article-title":"Fine-tuning large language models for specialized use cases","volume":"3","author":"D M Anisuzzaman","year":"2024","journal-title":"Mayo Clin Proc Digit Health"},{"issue":"01","key":"ref35","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1038\/s41746-025-01598-y","article-title":"Patient agency and large language models in worldwide encoding of equity","volume":"8","author":"A A Armoundas","year":"2025","journal-title":"NPJ Digit Med"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2835-4634.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T22:53:05Z","timestamp":1775515985000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2835-4634"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3]]},"references-count":27,"journal-issue":{"issue":"02","published-online":{"date-parts":[[2026,3,16]]},"published-print":{"date-parts":[[2026,3]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2835-4634","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3]]}}}