{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T05:34:31Z","timestamp":1789796071088,"version":"4.0.1"},"reference-count":12,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Clinical Otolaryngology"],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare the performance of Google Bard, Microsoft Copilot, GPT\u20104 with vision (GPT\u20104) and Gemini Ultra on the OTO Chautauqua, a student\u2010created, faculty\u2010reviewed otolaryngology question bank.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Study Design<\/jats:title>\n                    <jats:p>Comparative performance evaluation of different LLMs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Setting<\/jats:title>\n                    <jats:p>N\/A.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>N\/A.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Large language models (LLMs) are being extensively tested in medical education. However, their accuracy and effectiveness remain understudied, particularly in otolaryngology. This study involved inputting 350 single\u2010best\u2010answer multiple choice questions, including 18 image\u2010based questions, into four LLMS. Questions were sourced from six independent question banks related to (a) rhinology, (b) head and neck oncology, (c) endocrinology, (d) general otolaryngology, (e) paediatrics, (f) otology, (g) facial plastics, reconstruction and (h) trauma. LLMs were instructed to provide an output reasoning for their answers, the length of which was recorded.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Aggregate and subgroup analysis revealed that Gemini (79.8%) outperformed the other LLMs, followed by GPT\u20104 (71.1%), Copilot (68.0%), and Bard (65.1%) in accuracy.<\/jats:p>\n                    <jats:p>\n                      The LLMs had significantly different average response lengths, with Bard (x\u0304\u2009=\u20091685.24) being the longest and no difference between GPT\u20104 (x\u0304\u2009=\u2009827.34) and Copilot (x\u0304\u2009=\u2009904.12). Gemini's longer responses (x\u0304 =1291.68) included explanatory images and links. Gemini and GPT\u20104 correctly answered image\u2010based questions (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u200918), unlike Copilot and Bard, highlighting their adaptability and multimodal capabilities.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Gemini outperformed the other LLMs in terms of accuracy, followed by GPT\u20104, Copilot and Bard. GPT\u20104, although it has the second\u2010highest accuracy, provides concise and relevant explanations. Despite the promising performance of LLMs, medical learners should cautiously assess accuracy and decision\u2010making reliability.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/coa.14302","type":"journal-article","created":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T20:53:21Z","timestamp":1741899201000},"page":"704-711","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Comparison of\n                    <scp>ChatGPT<\/scp>\n                    \u20104, Copilot, Bard and Gemini Ultra on an Otolaryngology Question Bank"],"prefix":"10.1111","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6144-6423","authenticated-orcid":false,"given":"Rashi","family":"Ramchandani","sequence":"first","affiliation":[{"name":"Faculty of Medicine University of Ottawa  Ottawa Ontario Canada"},{"name":"Institute of Health Policy, Management and Evaluation, University of Toronto  Toronto Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eddie","family":"Guo","sequence":"additional","affiliation":[{"name":"Cumming School of Medicine, University of Calgary  Calgary Alberta Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Mostowy","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology Jacobs School of Medicine and Biomedical Sciences at the University of Buffalo  Buffalo New York USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Kreutz","sequence":"additional","affiliation":[{"name":"Cumming School of Medicine, University of Calgary  Calgary Alberta Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nick","family":"Sahlollbey","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2010Head and Neck Surgery University of Calgary  Calgary Alberta Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michele M.","family":"Carr","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology Jacobs School of Medicine and Biomedical Sciences at the University of Buffalo  Buffalo New York USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Janet","family":"Chung","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2010Head and Neck Surgery University of Toronto  Toronto Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Caulley","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2010Head and Neck Surgery University of Ottawa  Ottawa Ontario Canada"},{"name":"The Ottawa Hospital Research Institute  Ottawa Ontario Canada"},{"name":"Institut for Klinisk Medicin, Aarhus University  Aarhus Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"e_1_2_12_2_1","doi-asserted-by":"publisher","DOI":"10.1111\/medu.15402"},{"key":"e_1_2_12_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s43856\u2010023\u201000370\u20101"},{"key":"e_1_2_12_4_1","doi-asserted-by":"publisher","DOI":"10.1227\/neu.0000000000002634"},{"key":"e_1_2_12_5_1","unstructured":"\u201cOTO Chautauqua Question Bank \u201daccessed May 6 2024 https:\/\/otoqbank.com\/."},{"key":"e_1_2_12_6_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598\u2010023\u201043436"},{"key":"e_1_2_12_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.adaj.2023.07.016"},{"key":"e_1_2_12_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00405\u2010023\u201008381\u20103"},{"key":"e_1_2_12_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/lary.31434"},{"key":"e_1_2_12_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00405\u2010023\u201008051\u20104"},{"key":"e_1_2_12_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/01945998211068822"},{"key":"e_1_2_12_12_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12909\u2010023\u201004752\u2010w"},{"key":"e_1_2_12_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00405\u2010024\u201008643\u20108"}],"container-title":["Clinical Otolaryngology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/coa.14302","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,2]],"date-time":"2025-06-02T06:12:33Z","timestamp":1748844753000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/coa.14302"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":12,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1111\/coa.14302"],"URL":"https:\/\/doi.org\/10.1111\/coa.14302","archive":["Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.1111\/COA.14302\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1111\/COA.14302\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1111\/COA.14302\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1111\/COA.14302\/v2\/decision1","asserted-by":"object"}]},"ISSN":["1749-4478","1749-4486"],"issn-type":[{"value":"1749-4478","type":"print"},{"value":"1749-4486","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,13]]},"assertion":[{"value":"2024-08-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-28","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}