{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T05:34:28Z","timestamp":1789796068077,"version":"4.0.1"},"reference-count":33,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T00:00:00Z","timestamp":1764547200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T00:00:00Z","timestamp":1764547200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Otolaryngology - Head &amp; Neck Surgery"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:sec>\n                    <jats:title>Importance<\/jats:title>\n                    <jats:p>The performance of large language models has been compared to that of physicians.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate the performance of ChatGPT-4 in the field of otolaryngology and head and neck surgery (OTOHNS) residency training.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>Observational.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Setting<\/jats:title>\n                    <jats:p>Virtual.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>ChatGPT-4.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Interventions<\/jats:title>\n                    <jats:p>All questions from the OTOHNS National In-Training Exam (NITE) for 2022 and 2023 were submitted to ChatGPT-4. Answers were graded by 2 reviewers using the official grading rubric, and the average score was used. Mean exam results from residents who have taken this exam were obtained from the lead faculty.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Main Outcome Measures<\/jats:title>\n                    <jats:p>\n                      <jats:italic toggle=\"yes\">Z<\/jats:italic>\n                      -tests were used to compare ChatGPT-4\u2019s performance to that of residents. The questions were categorized by type (image or text), task, subspecialty, taxonomic level and prompt length.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      ChatGPT-4 scored 66% (350\/529) and 65% (243\/374) on the 2022 and 2023 exams, respectively. ChatGPT-4 outperformed the residents on both exams, among all training levels and within all sub-specialties except for the general\/pediatrics section of the 2023 exam (\n                      <jats:italic toggle=\"yes\">Z<\/jats:italic>\n                      -test \u22122.54). For the 2022 exam, ChatGPT-4 would rank in the 99th percentile among post-graduate year (PGY)-2 and 73rd percentile among PGY-4 classmates. For the 2023 exam, it would rank in the 99th percentile among PGY-2 and 71st percentile among PGY-4 classmates. ChatGPT-4 performed best on text-based questions (74%,\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2009&lt;\u2009.001) with an effect size of 1.27 (confidence interval (CI): 0.99-1.55), level 1 taxonomic questions (75%,\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2009&lt;\u2009.001) with an effect size of 0.084 (CI: 0.03-0.14) and guideline-based questions (70%,\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2009=\u2009.048) with an effect size of 0.11 (CI: 0-0.23). It had no significant difference in performance based on subspecialty (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2009=\u2009.36) or prompt length (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2009=\u2009.39).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>ChatGPT-4 not only achieved passing grades on 2 versions of the Canadian OTOHNS NITE, but it also significantly outperformed residents.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Relevance<\/jats:title>\n                    <jats:p>This study underscores a critical need to redesign residency assessment methods.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/19160216251387618","type":"journal-article","created":{"date-parts":[[2025,12,10]],"date-time":"2025-12-10T12:05:56Z","timestamp":1765368356000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Performance of Artificial Intelligence Chatbot on the Canadian Otolaryngology in-Training Exam: Unlocking Insights on the Intersection of Technology and Education"],"prefix":"10.1177","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-7362-2633","authenticated-orcid":false,"given":"Emily","family":"Ajit-Roger","sequence":"first","affiliation":[{"name":"Department of Otolaryngology\u2014Head &amp; Neck Surgery, McGill University, Montreal, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Koorosh Semsar","family":"Kazerooni","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2014Head &amp; Neck Surgery, McGill University, Montreal, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julian","family":"Savage","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2014Head &amp; Neck Surgery, Royal Berkshire Hospitals NHS Trust, Reading, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lily H.P.","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2014Head &amp; Neck Surgery, McGill University, Montreal, QC, Canada"},{"name":"Department of Pediatric Surgery, Montreal Children\u2019s Hospital, Montreal, QC, Canada"},{"name":"Institute of Health Sciences Education, McGill University, Montreal, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2025,12,10]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0300-7"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1080\/00016489.2023.2254809"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjoto.2024.104220"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1002\/lio2.1193"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.3390\/children10101634"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1089\/thy.2023.0491"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1177\/0003489418779414"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.2196\/11129"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.2196\/57594"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjno-2023-000530"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.5435\/JAAOS-D-23-00396"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.OA.23.00103"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1093\/asj\/sjad316"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/0194599812444386"},{"key":"e_1_3_3_16_2","unstructured":"GPT-4 is OpenAI\u2019s most advanced system producing safer and more useful responses. Accessed June 14 2023. https:\/\/openai.com\/product\/gpt-4"},{"key":"e_1_3_3_17_2","unstructured":"University of Waterloo Libraries. ChatGPT and generative artificial intelligence (AI): potential for bias based on prompt. Accessed June 14 2023. https:\/\/subjectguides.uwaterloo.ca\/chatgpt_generative_ai"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15430421tip4104_2"},{"key":"e_1_3_3_19_2","unstructured":"Canadian Resident Matching Service. CaRMS data & reports (2018 to 2023). Accessed June 20 2024. http:\/\/www.carms.ca\/en\/data-and-reports\/r-1\/"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00405-023-08051-4"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1147852"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.acra.2020.08.013"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.3163\/1536-5050.103.3.010"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1177\/08465371231220561"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.2196\/50638"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01029-4"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0070270"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0015719"},{"key":"e_1_3_3_29_2","unstructured":"Advancing AI in medical education through ethics evidence and equity. AMA. Accessed June 20 2024. https:\/\/www.ama-assn.org\/practice-management\/digital\/advancing-ai-medical-education-through-ethics-evidence-and-equity"},{"key":"e_1_3_3_30_2","unstructured":"Marr B. How generative AI will change the jobs of doctors and healthcare professionals. Forbes. Accessed June 20 2024. www.forbes.com\/sites\/bernardmarr\/2024\/03\/13\/how-generative-ai-will-change-the-jobs-of-doctors-and-healthcare-professionals"},{"key":"e_1_3_3_31_2","unstructured":"Gaerner L. Advancing cancer surgery through data science. MD Anderson. Accessed June 20 2024. https:\/\/www.mdanderson.org\/cancerwise\/advancing-cancer-surgery-through-data-science.h00-159694389.html"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.00000000000026"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582"},{"key":"e_1_3_3_34_2","unstructured":"Otolaryngology\u2014Head & Neck Surgery Competencies Royal College of Physicians and Surgeons of Canada. Accessed June 20 2024. https:\/\/www.royalcollege.ca\/content\/dam\/documents\/ibd\/otolaryngology-head-and-neck-surgery\/otolaryngology-competencies-e.html"}],"container-title":["Journal of Otolaryngology - Head &amp; Neck Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/19160216251387618","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/19160216251387618","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/19160216251387618","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T22:54:09Z","timestamp":1777589649000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/19160216251387618"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12]]},"references-count":33,"alternative-id":["10.1177\/19160216251387618"],"URL":"https:\/\/doi.org\/10.1177\/19160216251387618","relation":{},"ISSN":["1916-0216","1916-0216"],"issn-type":[{"value":"1916-0216","type":"print"},{"value":"1916-0216","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12]]},"article-number":"19160216251387618"}}