{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T11:19:04Z","timestamp":1783595944797,"version":"3.55.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T00:00:00Z","timestamp":1783555200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T00:00:00Z","timestamp":1783555200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009367","name":"Mansoura University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100009367","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Anaplastic thyroid cancer (ATC) is a rare, aggressive malignancy with poor prognosis. Adherence to guidelines from the National Comprehensive Cancer Network (NCCN), American Thyroid Association (ATA), and European Society for Medical Oncology (ESMO) is critical for optimal patient outcomes. As large language models (LLMs) increasingly enter clinical workflows, rigorous evaluation of their alignment with established guidelines is essential. We evaluated five leading LLMs for their ability to generate guideline-concordant responses to clinical questions about ATC. We conducted a comparative study in 2025 following TRIPOD-LLM (Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis, Large Language Models) guidelines. Seventy clinical questions of varying complexity were developed from ATA, NCCN, and ESMO guidelines. Three surgical oncology experts validated each question and subsequently evaluated responses from five LLMs: ChatGPT 4.1, ChatGPT 5, Gemini 2.5 Pro, Claude Sonnet 4, and DeepSeek R1. Each response was scored for relevancy, clarity, accuracy, and adequacy on a 5-point Likert scale. Inter-rater reliability was assessed using both intraclass correlation coefficients (ICC) and Gwet\u2019s AC2 with ordinal weights. Model comparisons used the Kruskal-Wallis test with Dunn\u2019s post-hoc analysis and Bonferroni correction. A pre-specified sensitivity analysis excluding the unblinded model (ChatGPT 5) was performed to confirm robustness. Significant performance differences emerged across all four metrics: accuracy (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.007), adequacy (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.003), clarity (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.014), and relevance (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.001). Gemini 2.5 Pro achieved the highest median accuracy (4.5), followed by DeepSeek R1 (4.4), while ChatGPT 4.1 scored lowest (4.0). ICC values ranged from 0.34 to 0.44 (poor to moderate), but Gwet\u2019s AC2 yielded substantially higher estimates of 0.61 to 0.73 (moderate to substantial agreement), reflecting the impact of restricted score range on conventional reliability metrics. The sensitivity analysis excluding ChatGPT 5 confirmed the performance hierarchy among blinded models, with significance preserved or strengthened across all four metrics. Leading LLMs show variable capacity to align with ATC clinical guidelines. While top-performing models hold promise as supportive tools, their inconsistencies across domains and complexity levels preclude autonomous clinical use. These models should serve strictly as decision aids under expert supervision.\n                  <\/jats:p>","DOI":"10.1038\/s41598-026-60786-2","type":"journal-article","created":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T10:04:06Z","timestamp":1783591446000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Performance of leading large language models in adhering to clinical guidelines for anaplastic thyroid cancer: a comparative study"],"prefix":"10.1038","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-6711-1159","authenticated-orcid":false,"given":"Mohamed","family":"Yasser","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghada","family":"Barakat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shadi","family":"Awny","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed","family":"Ezzat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed Sherif Ali","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sherif","family":"Wael","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariam","family":"Moustafa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Hamdy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,7,9]]},"reference":[{"key":"60786_CR1","doi-asserted-by":"publisher","first-page":"e13","DOI":"10.1016\/S2589-7500(19)30002-0","volume":"1","author":"T Panch","year":"2019","unstructured":"Panch, T., Pearson-Stuttard, J., Greaves, F. & Atun, R. Artificial intelligence: opportunities and risks for public health. Lancet Digit. Health. 1, e13\u2013e14 (2019).","journal-title":"Lancet Digit. Health"},{"key":"60786_CR2","first-page":"7","volume":"48","author":"A Abd-Alrazaq","year":"2024","unstructured":"Abd-Alrazaq, A. et al. The effectiveness of artificial intelligence driven tools in the healthcare sector. J. Med. Syst. 48, 7 (2024).","journal-title":"J. Med. Syst."},{"key":"60786_CR3","first-page":"1","volume":"21","author":"Z Ahmad","year":"2023","unstructured":"Ahmad, Z., Rahim, S., Zubair, M. & Abdul-Ghafar, J. Artificial intelligence (AI) in medicine. Int. J. Comput. Sci. Inf. Secur. 21, 1\u201315 (2023).","journal-title":"Int. J. Comput. Sci. Inf. Secur."},{"key":"60786_CR4","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930\u20131940 (2023).","journal-title":"Nat. Med."},{"key":"60786_CR5","unstructured":"Saab, K. et al. Capabilities of Gemini models in medicine. Preprint at (2024). https:\/\/arxiv.org\/abs\/2404.18416"},{"key":"60786_CR6","unstructured":"Yang, L. et al. Advancing multimodal medical capabilities of Gemini. Preprint at (2024). https:\/\/arxiv.org\/abs\/2405.03162"},{"key":"60786_CR7","doi-asserted-by":"crossref","first-page":"e4","DOI":"10.1016\/S2589-7500(24)00244-9","volume":"7","author":"S Seneviratne","year":"2025","unstructured":"Seneviratne, S., Shah, H., Memon, F. N. & Amin, A. The need for standardized evaluation of large language models in healthcare. Lancet Digit. Health. 7, e4\u2013e6 (2025).","journal-title":"Lancet Digit. Health"},{"key":"60786_CR8","doi-asserted-by":"publisher","first-page":"AIoa2300151","DOI":"10.1056\/AIoa2300151","volume":"1","author":"NR Rydzewski","year":"2024","unstructured":"Rydzewski, N. R. et al. Comparative evaluation of LLMs in clinical oncology. NEJM AI. 1, AIoa2300151 (2024).","journal-title":"NEJM AI"},{"key":"60786_CR9","doi-asserted-by":"publisher","first-page":"1459","DOI":"10.1001\/jamaoncol.2023.2954","volume":"9","author":"S Chen","year":"2023","unstructured":"Chen, S. et al. Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncol. 9, 1459\u20131462 (2023).","journal-title":"JAMA Oncol."},{"key":"60786_CR10","doi-asserted-by":"publisher","first-page":"e2417641","DOI":"10.1001\/jamanetworkopen.2024.17641","volume":"7","author":"JB Longwell","year":"2024","unstructured":"Longwell, J. B. et al. Performance of large language models on medical oncology examination questions. JAMA Netw. Open. 7, e2417641 (2024).","journal-title":"JAMA Netw. Open."},{"key":"60786_CR11","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1177\/00031348241269430","volume":"91","author":"S Pandya","year":"2025","unstructured":"Pandya, S., Bresler, T. E., Wilson, T., Htway, Z. & Fujita, M. Decoding the NCCN guidelines with AI: a comparative evaluation of ChatGPT-4.0 and Llama 2 in the management of thyroid carcinoma. Am. Surg. 91, 94\u201398 (2025).","journal-title":"Am. Surg."},{"key":"60786_CR12","doi-asserted-by":"publisher","first-page":"1590230","DOI":"10.3389\/fonc.2025.1590230","volume":"15","author":"C Li","year":"2025","unstructured":"Li, C., Cai, Y., Xuan, W. & Hu, S. ChatGPT 4o, Gemini Advanced and DeepSeek R1 in preoperative decision making for thyroid surgery. Front. Oncol. 15, 1590230 (2025).","journal-title":"Front. Oncol."},{"key":"60786_CR13","doi-asserted-by":"publisher","first-page":"111493","DOI":"10.1016\/j.isci.2024.111493","volume":"27","author":"CP Li","year":"2024","unstructured":"Li, C. P. et al. Comparing ChatGPT 3.5 and ChatGPT 4\u2019s alignments with the German S3 guideline for adult soft tissue sarcoma. iScience 27, 111493 (2024).","journal-title":"iScience"},{"key":"60786_CR14","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1002\/jso.70005","volume":"132","author":"TE Bresler","year":"2025","unstructured":"Bresler, T. E. et al. AI at the forefront: navigating oncologic care for six gastrointestinal cancers according to the NCCN guidelines utilizing Gemini-1.0 Ultra and ChatGPT-4. J. Surg. Oncol. 132, 317\u2013322 (2025).","journal-title":"J. Surg. Oncol."},{"key":"60786_CR15","doi-asserted-by":"publisher","first-page":"1941","DOI":"10.1097\/JS9.0000000000001066","volume":"110","author":"L Deng","year":"2024","unstructured":"Deng, L. et al. Evaluation of large language models in breast cancer clinical scenarios. Int. J. Surg. 110, 1941\u20131950 (2024).","journal-title":"Int. J. Surg."},{"key":"60786_CR16","doi-asserted-by":"crossref","first-page":"886","DOI":"10.3390\/cancers15030886","volume":"15","author":"M Ragazzi","year":"2023","unstructured":"Ragazzi, M. et al. Anaplastic thyroid carcinoma: current and emerging treatment options. Cancers (Basel). 15, 886 (2023).","journal-title":"Cancers (Basel)"},{"key":"60786_CR17","first-page":"227","volume":"85","author":"M Joner","year":"2024","unstructured":"Joner, M. et al. Anaplastic thyroid carcinoma: a comprehensive review. Endocrine 85, 227\u2013238 (2024).","journal-title":"Endocrine"},{"key":"60786_CR18","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1089\/thy.2020.0944","volume":"31","author":"KC Bible","year":"2021","unstructured":"Bible, K. C. et al. 2021 ATA guidelines for management of patients with anaplastic thyroid cancer. Thyroid 31, 337\u2013386 (2021).","journal-title":"Thyroid"},{"key":"60786_CR19","doi-asserted-by":"publisher","first-page":"e250033","DOI":"10.6004\/jnccn.2025.0033","volume":"23","author":"RI Haddad","year":"2025","unstructured":"Haddad, R. I. et al. NCCN Guidelines\u00ae Insights: Thyroid Carcinoma, Version 1.2025. J. Natl. Compr. Canc Netw. 23, e250033 (2025).","journal-title":"J. Natl. Compr. Canc Netw."},{"key":"60786_CR20","doi-asserted-by":"publisher","first-page":"1856","DOI":"10.1093\/annonc\/mdz400","volume":"30","author":"S Filetti","year":"2019","unstructured":"Filetti, S. et al. Thyroid cancer: ESMO clinical practice guidelines. Ann. Oncol. 30, 1856\u20131883 (2019).","journal-title":"Ann. Oncol."},{"key":"60786_CR21","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1016\/j.annonc.2022.04.009","volume":"33","author":"S Filetti","year":"2022","unstructured":"Filetti, S. et al. ESMO clinical practice guideline update on systemic therapy in advanced thyroid cancer. Ann. Oncol. 33, 674\u2013684 (2022).","journal-title":"Ann. Oncol."},{"key":"60786_CR22","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/s10916-023-01996-3","volume":"47","author":"M Sallam","year":"2023","unstructured":"Sallam, M. The utility of ChatGPT as a clinical decision support tool. J. Med. Syst. 47, 101 (2023).","journal-title":"J. Med. Syst."},{"key":"60786_CR23","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","volume":"30","author":"P Hager","year":"2024","unstructured":"Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613\u20132622 (2024).","journal-title":"Nat. Med."},{"key":"60786_CR24","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","volume":"31","author":"J Gallifant","year":"2025","unstructured":"Gallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat. Med. 31, 60\u201369 (2025).","journal-title":"Nat. Med."},{"key":"60786_CR25","unstructured":"Li, H. et al. The safety of large language models in medicine. Preprint at (2024). https:\/\/arxiv.org\/abs\/2405.13055"},{"key":"60786_CR26","doi-asserted-by":"publisher","first-page":"e2343689","DOI":"10.1001\/jamanetworkopen.2023.43689","volume":"6","author":"M Benary","year":"2023","unstructured":"Benary, M. et al. Leveraging large language models for decision support in personalized oncology. JAMA Netw. Open. 6, e2343689 (2023).","journal-title":"JAMA Netw. Open."},{"key":"60786_CR27","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","volume":"333","author":"S Bedi","year":"2025","unstructured":"Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Med. 3, 141 (2023).","journal-title":"JAMA"},{"key":"60786_CR28","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","volume":"3","author":"J Clusmann","year":"2023","unstructured":"Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333, 319\u2013328 (2025).","journal-title":"Commun. Med."},{"key":"60786_CR29","first-page":"e5795","volume":"16","author":"K Nguyen","year":"2024","unstructured":"Nguyen, K., Do, N., Nguyen, A. & Truong, H. A comprehensive evaluation of GPT-4 in otolaryngology specific tasks. Cureus 16, e5795 (2024).","journal-title":"Cureus"},{"key":"60786_CR30","unstructured":"Twaddle, A. Google\u2019s Gemini 1.5 Pro a \u2018step change\u2019 in AI capability. Genomics Engl. (2024). https:\/\/www.genomicsengland.co.uk\/news\/google-gemini-1-5-pro-step-change-ai-capability"},{"key":"60786_CR31","doi-asserted-by":"publisher","first-page":"e63010","DOI":"10.2196\/63010","volume":"12","author":"T Hirosawa","year":"2024","unstructured":"Hirosawa, T. et al. Comparative study to evaluate the accuracy of differential diagnosis lists generated by Gemini Advanced, Gemini, and Bard for a case report series analysis: cross-sectional study. JMIR Med. Inf. 12, e63010 (2024).","journal-title":"JMIR Med. Inf."},{"key":"60786_CR32","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10916-023-01985-6","volume":"47","author":"T Dave","year":"2023","unstructured":"Dave, T., Aufahy, F. & Al Ghanim, K. A. Are large language models a threat to evidence based medicine? J. Med. Syst. 47, 89 (2023).","journal-title":"J. Med. Syst."},{"key":"60786_CR33","doi-asserted-by":"crossref","first-page":"1531","DOI":"10.1038\/s41591-024-02980-1","volume":"30","author":"M Moor","year":"2024","unstructured":"Moor, M., Dal R\u00e9, R., Slattery, C. & Topol, E. J. The case for responsible innovation in medical AI. Nat. Med. 30, 1531\u20131533 (2024).","journal-title":"Nat. Med."},{"key":"60786_CR34","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1016\/S0140-6736(23)01367-3","volume":"402","author":"Z Kanjee","year":"2023","unstructured":"Kanjee, Z., Crowe, B. & Fire, A. Using GPT-4 to prescribe: a cautionary note. Lancet 402, 687 (2023).","journal-title":"Lancet"},{"key":"60786_CR35","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1126\/science.adk6139","volume":"381","author":"EJ Topol","year":"2023","unstructured":"Topol, E. J. The AI revolution in medicine. Science 381, 613 (2023).","journal-title":"Science"},{"key":"60786_CR36","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"60786_CR37","first-page":"9459","volume":"33","author":"M Lewis","year":"2020","unstructured":"Lewis, M. et al. Retrieval augmented generation for knowledge intensive NLP tasks. Adv. Neural Inf. Process. Syst. 33, 9459\u20139474 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"60786_CR38","first-page":"e50689","volume":"25","author":"B Mesk\u00f3","year":"2023","unstructured":"Mesk\u00f3, B. The practical guide to the real world use of AI in healthcare. J. Med. Internet Res. 25, e50689 (2023).","journal-title":"J. Med. Internet Res."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-60786-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-60786-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-60786-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T10:28:10Z","timestamp":1783592890000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-60786-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,9]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["60786"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-60786-2","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,9]]},"assertion":[{"value":"24 March 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"This study involved no human subjects, patient data, biological specimens, or clinical interventions. All data consisted exclusively of responses generated by publicly accessible artificial intelligence tools, evaluated by expert reviewers acting in their professional capacity. Accordingly, formal ethics committee approval was not required. This determination is consistent with Egyptian national research ethics regulations and with Mansoura University\u2019s institutional policy governing research exemptions, which exclude studies not involving human participants or identifiable data. This study was conducted in accordance with the Declaration of Helsinki insofar as applicable.\u00a0All experts were included as co-authors of the study. Informed consent was obtained from all experts prior to their participation, and they were fully informed of the study objectives, methodology, and their role in the evaluation process. Digital documentation of consent is available upon request.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}}],"article-number":"21415"}}