{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T08:26:51Z","timestamp":1786955211745,"version":"build-2736575974"},"reference-count":21,"publisher":"Wiley","license":[{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["World j. otorhinolaryngol.-head neck surg."],"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) such as ChatGPT are being explored for various medical applications, but their performance across languages is not established. Despite the promising results in English from previous studies, it is unclear how ChatGPT performs in other languages. We assessed the ability of ChatGPT\u20104 to answer common otolaryngology\u2010related frequently asked questions in Chinese, Korean, and Spanish in this study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Eighteen clinical questions were selected from the American Academy of Otolaryngology\u2010Head and Neck Surgery Foundation guidelines and ENT Health across six subspecialties. Bilingual clinicians translated the questions to Chinese, Korean, or Spanish and input them into ChatGPT\u20104 (OpenAI, San Francisco, USA). The responses were rated on a five\u2010point Likert scale for accuracy, completeness, and similarity to English reference answers. A linear mixed\u2010effects model with language as a fixed effect and random intercepts for question and rater was used to compare performance across languages. Inter\u2010rater reliability was assessed using intraclass correlation coefficients and Krippendorff's\n                      <jats:italic>\u03b1<\/jats:italic>\n                      . Subjective evaluations were also collected.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>ChatGPT\u20104's performance differed significantly by language in all assessment domains. Responses in Spanish scored highest throughout and most frequently were characterized as clear and organized. Korean answers were generally accurate but tended to be more conservative and less detailed. Chinese answers showed greater variability, with reviewers noting a mix of clear and less accurate outputs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The clinical performance of ChatGPT\u20104 differed across Chinese, Korean, and Spanish. These findings underscore the importance of ongoing evaluation and refinement of LLMs to ensure they provide reliable multilingual support in clinical practice.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/wjo2.70153","type":"journal-article","created":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T08:14:33Z","timestamp":1786954473000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Performance of Large Language Models in Answering Otolaryngology Clinical Questions in Non\u2010English Languages"],"prefix":"10.1002","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6350-1422","authenticated-orcid":false,"given":"Eugene","family":"Oh","sequence":"first","affiliation":[{"name":"David Geffen School of Medicine University of California, Los Angeles Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8993-6355","authenticated-orcid":false,"given":"Arthur W.","family":"Wu","sequence":"additional","affiliation":[{"name":"Division of Otolaryngology Cedars\u2010Sinai Medical Center Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Garcia\u2010Rodriguez","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology Henry Ford Health Detroit Michigan USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5185-3842","authenticated-orcid":false,"given":"Zhen\u2010Xiao","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2010Head and Neck Surgery Beijing Tongren Hospital Beijing China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei\u2010Chung","family":"Hsu","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology Taiwan University Hospital Taipei China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeongkyou","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology\u2010Head and Neck Surgery Seoul National University Hospital Seoul Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Young Chul","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology\u2010Head and Neck Surgery Seoul National University Hospital Seoul Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi\u2010Tsen","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology Taiwan University Hospital Taipei China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hae Chan","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology Seoul National University Bundang Hospital Seongnam Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hector Andres","family":"Perez","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology\u2010Head and Neck Surgery University of Washington Seattle Washington USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan San","family":"Juan","sequence":"additional","affiliation":[{"name":"Department of Head and Neck Surgery Kaiser Permanente West Los Angeles Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gene","family":"Liu","sequence":"additional","affiliation":[{"name":"Division of Otolaryngology Cedars\u2010Sinai Medical Center Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Lee","sequence":"additional","affiliation":[{"name":"Division of Otolaryngology Cedars\u2010Sinai Medical Center Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2315-9930","authenticated-orcid":false,"given":"Dennis M.","family":"Tang","sequence":"additional","affiliation":[{"name":"Division of Otolaryngology Cedars\u2010Sinai Medical Center Los Angeles California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"e_1_2_11_3_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsr2214184"},{"key":"e_1_2_11_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/oto2.67"},{"key":"e_1_2_11_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjoto.2024.104220"},{"key":"e_1_2_11_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00405-023-08282-5"},{"key":"e_1_2_11_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/ohn.465"},{"key":"e_1_2_11_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-00680-3"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajg.2023.08.001"},{"key":"e_1_2_11_10_1","volume-title":"American Academy of Otolaryngology\u2010Head and Neck Surgery","author":"Clinical Practice Guidelines","year":"2025"},{"key":"e_1_2_11_11_1","unstructured":"ENT Health. Accessed February 17 2025 https:\/\/www.enthealth.org\/."},{"key":"e_1_2_11_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eururo.2023.09.022"},{"key":"e_1_2_11_13_1","unstructured":"OpenAI J.Achiam S.Adler et al. \u201cGPT\u20104 Technical Report \u201d2024 https:\/\/doi.org\/10.48550\/arXiv.2303.08774."},{"key":"e_1_2_11_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejca.2025.115274"},{"key":"e_1_2_11_15_1","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.OA.25.00207"},{"key":"e_1_2_11_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.nbt.2023.08.004"},{"key":"e_1_2_11_17_1","doi-asserted-by":"publisher","DOI":"10.2196\/60939"},{"key":"e_1_2_11_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijoa.2024.104290"},{"key":"e_1_2_11_19_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2021.658220"},{"key":"e_1_2_11_20_1","doi-asserted-by":"publisher","DOI":"10.1097\/PHM.0b013e318186babf"},{"key":"e_1_2_11_21_1","doi-asserted-by":"publisher","DOI":"10.2196\/54706"},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tjnut.2024.12.018"}],"container-title":["World Journal of Otorhinolaryngology - Head and Neck Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/wjo2.70153","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/wjo2.70153","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/wjo2.70153","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T08:14:39Z","timestamp":1786954479000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/wjo2.70153"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,17]]},"references-count":21,"alternative-id":["10.1002\/wjo2.70153"],"URL":"https:\/\/doi.org\/10.1002\/wjo2.70153","archive":["Portico"],"relation":{},"ISSN":["2095-8811","2589-1081"],"issn-type":[{"value":"2095-8811","type":"print"},{"value":"2589-1081","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,17]]},"assertion":[{"value":"2025-07-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"wjo2.70153"}}