{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T21:11:47Z","timestamp":1791666707284,"version":"4.3.4"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,5,23]],"date-time":"2026-05-23T00:00:00Z","timestamp":1779494400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T00:00:00Z","timestamp":1785369600000},"content-version":"vor","delay-in-days":68,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    This study aimed to comparatively evaluate the medical information delivery capacity and content quality of current large language models (LLMs), specifically ChatGPT (GPT-5.2), Gemini (3.1), and DeepSeek (V4), regarding oral cavity cancer (OCC) based on expert opinions. 20 open-ended questions addressing the risk factors, diagnosis, and treatment of OCC were directed to the three models. The responses were evaluated using a blinded method by 31 expert physicians from Oral and Maxillofacial Surgery, Otorhinolaryngology (ENT), and Medical Oncology. The Modified Global Quality Scale (1\u20135 points) was utilised for evaluation. Statistical analyses were performed using Kruskal\u2013Wallis, ANOVA, and Bonferroni post-hoc tests, while Fleiss\u2019 Kappa coefficient determined inter-expert consistency. The general performance scores of the models were high (3.57\u20134.15). In the overall assessment, Gemini received statistically significantly higher scores than the DeepSeek model (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.036). Significant performance differences were identified across 15 of 20 questions (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.05); ChatGPT excelled on clinical and treatment-oriented questions, while Gemini stood out on comprehensive informational items. While no statistically significant difference was found among the specialist groups for the overall evaluation and 19 out of 20 questions (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&gt;\u20090.05), a significant difference was observed solely for Q6 (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.042). Although LLMs have the potential to generate high-quality information about OCC, their performance varies by content type and model architecture. While Gemini demonstrated more consistent performance overall, expert supervision remains essential before these tools can be used as reliable sources of clinical information. Clinicians must be aware of the specific strengths and limitations of different LLMs in OCC to better guide patients who increasingly use such tools for medical information.\n                  <\/jats:p>","DOI":"10.1038\/s41598-026-53630-0","type":"journal-article","created":{"date-parts":[[2026,5,23]],"date-time":"2026-05-23T09:33:59Z","timestamp":1779528839000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A comparative analysis of large language models for providing oral cavity cancer information"],"prefix":"10.1038","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2879-7798","authenticated-orcid":false,"given":"Eda","family":"\u0130zgi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1852-856X","authenticated-orcid":false,"given":"Turan Canmurat","family":"\u0130zgi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9140-1453","authenticated-orcid":false,"given":"Ceren","family":"Morda\u011f \u00c7i\u00e7ek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,23]]},"reference":[{"issue":"4","key":"53630_CR1","doi-asserted-by":"publisher","first-page":"5501","DOI":"10.22214\/ijraset.2024.61241","volume":"12","author":"DA Shrivastava","year":"2024","unstructured":"Shrivastava, D. A. Artificial intelligence (AI): Evolution, methodologies, and applications. Int. J. Res. Appl. Sci. Eng. Technol. 12(4), 5501\u20135505. https:\/\/doi.org\/10.22214\/ijraset.2024.61241 (2024).","journal-title":"Int. J. Res. Appl. Sci. Eng. Technol."},{"issue":"10","key":"53630_CR2","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1016\/j.joms.2024.05.042","volume":"82","author":"SO Mohseni","year":"2024","unstructured":"Mohseni, S. O. & Schlieve, T. Artificial intelligence in medicine: Is Oral and Maxillofacial Surgery following the trend?. J. Oral Maxillofac. Surg. 82(10), 1189\u20131190. https:\/\/doi.org\/10.1016\/j.joms.2024.05.042 (2024).","journal-title":"J. Oral Maxillofac. Surg."},{"issue":"1","key":"53630_CR3","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-025-05479-4","volume":"25","author":"M \u00c7itir","year":"2025","unstructured":"\u00c7itir, M. ChatGPT and oral cancer: A study on informational reliability. BMC Oral Health 25(1), 86. https:\/\/doi.org\/10.1186\/s12903-025-05479-4 (2025).","journal-title":"BMC Oral Health"},{"key":"53630_CR4","doi-asserted-by":"publisher","DOI":"10.1007\/s13187-025-02592-4","author":"A Lawson McLean","year":"2025","unstructured":"Lawson McLean, A. & Hristidis, V. Evidence-based analysis of AI chatbots in oncology patient education: Implications for trust, perceived realness, and misinformation management. J. Cancer Educ. https:\/\/doi.org\/10.1007\/s13187-025-02592-4 (2025).","journal-title":"J. Cancer Educ."},{"key":"53630_CR5","doi-asserted-by":"publisher","DOI":"10.3390\/cancers14194967","author":"M\u00c1 Gonz\u00e1lez-Moles","year":"2022","unstructured":"Gonz\u00e1lez-Moles, M. \u00c1. et al. Challenges in the early diagnosis of oral cancer, evidence gaps and strategies for improvement: A scoping review of systematic reviews. Cancers (Basel) https:\/\/doi.org\/10.3390\/cancers14194967 (2022).","journal-title":"Cancers (Basel)"},{"issue":"3","key":"53630_CR6","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1177\/0022034520902128","volume":"99","author":"B Ilhan","year":"2020","unstructured":"Ilhan, B. et al. Improving oral cancer outcomes with imaging and artificial intelligence. J. Dent. Res. 99(3), 241\u2013248. https:\/\/doi.org\/10.1177\/0022034520902128 (2020).","journal-title":"J. Dent. Res."},{"key":"53630_CR7","unstructured":"ChatGPT (GPT-5.2 version). OpenAI. Accessed 20 February 2026. https:\/\/chat.openai.com\/"},{"key":"53630_CR8","unstructured":"Gemini 3.1. Google DeepMind. Accessed 20 February 2026. https:\/\/gemini.google.com\/"},{"key":"53630_CR9","unstructured":"DeepSeek Chat (V4 Version). DeepSeek AI. Accessed 20 February 2026. https:\/\/chat.deepseek.com\/"},{"issue":"2","key":"53630_CR10","doi-asserted-by":"publisher","first-page":"2","DOI":"10.6004\/jnccn.2025.0007","volume":"23","author":"AD Colevas","year":"2025","unstructured":"Colevas, A. D. et al. Head and Neck Cancers, Version 2.2025. JNCCN J. Natl. Compr. Cancer Network 23(2), 2\u201311. https:\/\/doi.org\/10.6004\/jnccn.2025.0007 (2025).","journal-title":"JNCCN J. Natl. Compr. Cancer Network"},{"key":"53630_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.jormas.2023.101471","author":"Y Balel","year":"2023","unstructured":"Balel, Y. Can ChatGPT be used in oral and maxillofacial surgery?. J. Stomatol. Oral Maxillofac. Surg. https:\/\/doi.org\/10.1016\/j.jormas.2023.101471 (2023).","journal-title":"J. Stomatol. Oral Maxillofac. Surg."},{"issue":"1","key":"53630_CR12","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33(1), 159\u2013174 (1977).","journal-title":"Biometrics"},{"key":"53630_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjoto.2023.103980","author":"RT Wu","year":"2023","unstructured":"Wu, R. T. & Dang, R. R. ChatGPT in head and neck scientific writing: A precautionary anecdote. Am. J. Otolaryngol. https:\/\/doi.org\/10.1016\/j.amjoto.2023.103980 (2023).","journal-title":"Am. J. Otolaryngol."},{"issue":"2","key":"53630_CR14","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1111\/odi.70110","volume":"32","author":"B Yilmaz","year":"2026","unstructured":"Yilmaz, B. et al. Accuracy of ChatGPT\u20104 Plus in providing information on oral cancer management. Oral Dis. 32(2), 422\u2013430. https:\/\/doi.org\/10.1111\/odi.70110 (2026).","journal-title":"Oral Dis."},{"key":"53630_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jormas.2025.102644","author":"BE Yilmaz","year":"2026","unstructured":"Yilmaz, B. E. et al. Can large language models perform clinical anamnesis? Comparative evaluation of ChatGPT, Claude, and Gemini in diagnostic reasoning through case-based questioning in oral and maxillofacial disorders. J. Stomatol. Oral Maxillofac. Surg. https:\/\/doi.org\/10.1016\/j.jormas.2025.102644 (2026).","journal-title":"J. Stomatol. Oral Maxillofac. Surg."},{"issue":"6","key":"53630_CR16","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1016\/j.oooo.2024.12.028","volume":"139","author":"R Rokhshad","year":"2025","unstructured":"Rokhshad, R. et al. Efficacy and empathy of AI chatbots in answering frequently asked questions on oral oncology. Oral Surg. Oral Med. Oral Pathol. Oral Radiol. 139(6), 719\u2013728. https:\/\/doi.org\/10.1016\/j.oooo.2024.12.028 (2025).","journal-title":"Oral Surg. Oral Med. Oral Pathol. Oral Radiol."},{"issue":"2","key":"53630_CR17","doi-asserted-by":"publisher","first-page":"558","DOI":"10.1097\/SCS.0000000000010686","volume":"36","author":"G I\u015fik","year":"2025","unstructured":"I\u015fik, G. et al. Is artificial intelligence a useful tool for clinical practice of Oral and Maxillofacial Surgery?. J. Craniofac. Surg. 36(2), 558\u2013562. https:\/\/doi.org\/10.1097\/SCS.0000000000010686 (2025).","journal-title":"J. Craniofac. Surg."},{"key":"53630_CR18","doi-asserted-by":"publisher","DOI":"10.1111\/cts.70499","author":"NE Abosamak","year":"2026","unstructured":"Abosamak, N. E. et al. AI-Powered chatbot as a health literacy tool for enhancing oral cancer awareness: Expert feedback. Clin. Transl. Sci. https:\/\/doi.org\/10.1111\/cts.70499 (2026).","journal-title":"Clin. Transl. Sci."},{"key":"53630_CR19","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-025-06726-4","author":"A Bashah","year":"2025","unstructured":"Bashah, A. et al. Evaluation of Deepseek, Gemini, ChatGPT-4o, and Perplexity in responding to salivary gland cancer. BMC Oral Health https:\/\/doi.org\/10.1186\/s12903-025-06726-4 (2025).","journal-title":"BMC Oral Health"},{"key":"53630_CR20","doi-asserted-by":"publisher","DOI":"10.3390\/app15147778","author":"B Sezer","year":"2025","unstructured":"Sezer, B. & Aydo\u011fdu, T. Performance of advanced artificial intelligence models in traumatic dental injuries in primary dentition: A comparative evaluation of ChatGPT-4 Omni, DeepSeek, Gemini Advanced, and Claude 3.7 in terms of accuracy, completeness, response time, and readability. Appl. Sci. (Basel) https:\/\/doi.org\/10.3390\/app15147778 (2025).","journal-title":"Appl. Sci. (Basel)"},{"key":"53630_CR21","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13172114","author":"E Y\u00fcceer-\u00c7etiner","year":"2025","unstructured":"Y\u00fcceer-\u00c7etiner, E. et al. Evaluating the competence of AI chatbots in answering patient-oriented frequently asked questions on orthognathic surgery. Healthcare (Basel) https:\/\/doi.org\/10.3390\/healthcare13172114 (2025).","journal-title":"Healthcare (Basel)"},{"issue":"1","key":"53630_CR22","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-026-37800-8","volume":"16","author":"G Geduk","year":"2026","unstructured":"Geduk, G. et al. A comparative analysis of the performance of large language models in the dentistry specialty examination. Sci. Rep. 16(1), 6739. https:\/\/doi.org\/10.1038\/s41598-026-37800-8 (2026).","journal-title":"Sci. Rep."},{"key":"53630_CR23","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-026-07790-0","author":"E Haylaz","year":"2026","unstructured":"Haylaz, E. et al. Evaluation of the performance of four different large language models (ChatGPT, DeepSeek, Copilot, and Gemini) in answering oral, and maxillofacial radiology questions: pilot study. BMC Oral Health https:\/\/doi.org\/10.1186\/s12903-026-07790-0 (2026).","journal-title":"BMC Oral Health"},{"issue":"8","key":"53630_CR24","doi-asserted-by":"publisher","first-page":"5034","DOI":"10.1097\/JS9.0000000000001469","volume":"110","author":"JW Li","year":"2024","unstructured":"Li, J. W. et al. Diagnostic accuracy of artificial intelligence assisted clinical imaging in the detection of oral potentially malignant disorders and oral cancer: A systematic review and meta-analysis. Int. J. Surg. 110(8), 5034\u20135046. https:\/\/doi.org\/10.1097\/JS9.0000000000001469 (2024).","journal-title":"Int. J. Surg."},{"issue":"19-20","key":"53630_CR25","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1089\/ten.tea.2024.0096","volume":"30","author":"CT Viet","year":"2024","unstructured":"Viet, C. T. et al. Artificial intelligence applications in oral cancer and oral dysplasia. Tissue Eng. Part A 30(19\u201320), 640\u2013651. https:\/\/doi.org\/10.1089\/ten.tea.2024.0096 (2024).","journal-title":"Tissue Eng. Part A"},{"key":"53630_CR26","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics15030280","author":"V Vinay","year":"2025","unstructured":"Vinay, V. et al. Artificial intelligence in oral cancer: A comprehensive scoping review of diagnostic and prognostic applications. Diagnostics (Basel) https:\/\/doi.org\/10.3390\/diagnostics15030280 (2025).","journal-title":"Diagnostics (Basel)"},{"issue":"8","key":"53630_CR27","doi-asserted-by":"publisher","first-page":"2397","DOI":"10.1097\/SCS.0000000000010663","volume":"35","author":"VP Veeraraghavan","year":"2024","unstructured":"Veeraraghavan, V. P. et al. Assessing artificial intelligence in oral cancer diagnosis: A systematic review. J. Craniofac. Surg. 35(8), 2397\u20132403. https:\/\/doi.org\/10.1097\/SCS.0000000000010663 (2024).","journal-title":"J. Craniofac. Surg."},{"key":"53630_CR28","doi-asserted-by":"publisher","DOI":"10.1007\/s10266-025-01242-x","author":"A Abou-Bakr","year":"2025","unstructured":"Abou-Bakr, A. et al. ChatGPT-5 vs oral medicine experts for rank-based differential diagnosis of oral lesions: a prospective, biopsy-validated comparison. Odontology https:\/\/doi.org\/10.1007\/s10266-025-01242-x (2025).","journal-title":"Odontology"},{"key":"53630_CR29","doi-asserted-by":"publisher","DOI":"10.3389\/froh.2026.1748450","author":"FEA Hassanein","year":"2026","unstructured":"Hassanein, F. E. A. et al. Multimodal large language models for oral lesion diagnosis: A systematic review of diagnostic performance and clinical utility. Front. Oral Health https:\/\/doi.org\/10.3389\/froh.2026.1748450 (2026).","journal-title":"Front. Oral Health"},{"key":"53630_CR30","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-026-07665-4","author":"FEA Hassanein","year":"2026","unstructured":"Hassanein, F. E. A. et al. Calibration of AI large language models with human subject matter experts for grading of clinical short-answer responses in dental education. BMC Oral Health https:\/\/doi.org\/10.1186\/s12903-026-07665-4 (2026).","journal-title":"BMC Oral Health"},{"key":"53630_CR31","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07875-4","author":"YBN Gokkurt","year":"2025","unstructured":"Gokkurt, Y. B. N. et al. Effect of artificial intelligence-assisted personalized feedback on radiographic diagnostic performance of dental students: A controlled study. BMC Med. Educ. https:\/\/doi.org\/10.1186\/s12909-025-07875-4 (2025).","journal-title":"BMC Med. Educ."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53630-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53630-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53630-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T18:17:25Z","timestamp":1785521845000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-53630-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,23]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["53630"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-53630-0","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,23]]},"assertion":[{"value":"12 March 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study was conducted in accordance with the ethical standards of the institutional research committee and with the Helsinki Declaration and its later amendments or comparable ethical standards. Although the study analysed publicly available data generated by LLMs, the evaluation of these outputs involved expert physicians from various specialties. Therefore, ethical approval was obtained from the K\u00fctahya University of Health Sciences Ethics Committee for Non-Interventional Clinical Research (Date: February 9, 2026; Decision No: 2026\/02-46). Informed consent was obtained from all individual participants (physicians) included in the study.","order":2,"name":"Ethics","label":"Ethics approval","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"23601"}}