{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T16:09:14Z","timestamp":1792080554147,"version":"4.4.0"},"reference-count":55,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Although thyroid nodules are detected in up to 60% of adults on ultrasound, the vast majority are benign, creating a substantial decision-making burden compounded by heterogeneous practice guidelines. Large language models (LLMs) show promise in processing unstructured medical text and are emerging as tools for report interpretation among both clinicians and patients. However, their reliability across distinct clinical tasks in thyroid ultrasound interpretation remains poorly characterized.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study evaluates 2 LLMs, ChatGPT-4o and DeepSeek-R1, in interpreting thyroid nodule ultrasound text reports across three clinical tasks\u2014benign-malignant differentiation, Chinese Thyroid Imaging Reporting and Data System (C-TIRADS) classification, and management recommendation\u2014with concurrent assessment of output stability for each task.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We retrospectively analyzed 1063 ultrasound text reports from 3 medical centers, including 306 with histopathological confirmation. Each nodule report was submitted to both LLMs via their consumer web interfaces using task-specific prompts, with 5 repetitions per model; final outputs were determined by mode voting. Diagnostic performance was assessed by receiver operating characteristic analysis with DeLong testing; agreement was quantified using squared weighted \u03ba and Cohen \u03ba; and stability was measured using Krippendorff \u03b1 and Fleiss \u03ba.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      For benign-malignant differentiation, DeepSeek-R1 showed higher sensitivity (0.879 vs 0.692;\n                      <jats:italic>P<\/jats:italic>\n                      &lt;.001) and accuracy (0.729 vs 0.644;\n                      <jats:italic>P<\/jats:italic>\n                      =.008) than ChatGPT-4o. With access to images and clinical context unavailable to the LLMs, senior radiologists showed higher performance (area under the curve=0.865; accuracy=0.804). For C-TIRADS classification, DeepSeek-R1 showed substantial agreement with radiologists, exceeding ChatGPT-4o (\u03ba=0.770 vs 0.688; \u0394\u03ba=0.082, 95% CI 0.048-0.122). Both models yielded moderate, comparable agreement with clinicians on management recommendations (\u03ba=0.606 vs 0.608). Stability was near perfect for C-TIRADS classification (\u03b1=0.864 vs 0.866) and management recommendations (\u03ba=0.853 vs. 0.849) in both models; however, DeepSeek-R1 showed markedly greater stability than ChatGPT-4o in benign-malignant differentiation (\u03ba=0.869 vs 0.609; \u0394\u03ba=0.260, 95% CI 0.191-0.321).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Both LLMs demonstrate clinical potential for thyroid nodule ultrasound report interpretation, with DeepSeek-R1 showing advantages in diagnostic accuracy, classification consistency, and output stability. However, both LLMs remained inferior to senior radiologists, suggesting their role as decision-support tools rather than stand-alone diagnostic systems. These findings provide preliminary evidence to inform the responsible integration of LLMs into thyroid imaging workflows while highlighting the need for further evaluation before patient-facing deployment.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/93890","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:25:54Z","timestamp":1782843954000},"page":"e93890-e93890","source":"Crossref","is-referenced-by-count":1,"title":["The Performance of ChatGPT-4o and DeepSeek-R1 in Interpreting Thyroid Nodule Ultrasound Text Reports: Multicenter Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-4877-616X","authenticated-orcid":false,"given":"Yujie","family":"Xie","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0587-4632","authenticated-orcid":false,"given":"Jiarui","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2588-1114","authenticated-orcid":false,"given":"Bing","family":"Zhan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6862-0021","authenticated-orcid":false,"given":"Kangfan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2169-1182","authenticated-orcid":false,"given":"Yuchen","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1045-077X","authenticated-orcid":false,"given":"Chunping","family":"Ning","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,28]]},"reference":[{"issue":"12","key":"R1","doi-asserted-by":"publisher","first-page":"715","DOI":"10.1038\/s41574-024-01025-4","article-title":"Thyroid nodules: diagnosis and management","volume":"20","author":"Grani","journal-title":"Nat Rev Endocrinol"},{"issue":"9","key":"R2","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1001\/jama.2018.0898","article-title":"The diagnosis and management of thyroid nodules: a review","volume":"319","author":"Durante","journal-title":"JAMA"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","first-page":"622","DOI":"10.4158\/EP161208.GL","article-title":"American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules--2016 update","volume":"22","author":"Gharib","journal-title":"Endocr Pract"},{"issue":"7","key":"R4","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1016\/S2213-8587(22)00101-2","article-title":"Diagnosis of thyroid nodules","volume":"10","author":"Alexander","journal-title":"Lancet Diabetes Endocrinol"},{"issue":"Pt A","key":"R5","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.ijsu.2018.04.037","article-title":"A survey of current approaches to thyroid nodules and thyroid operations","volume":"54","author":"Isik","journal-title":"Int J Surg"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1148\/radiol.2019181723","article-title":"Diagnostic performance of practice guidelines for thyroid nodules: thyroid nodule size versus biopsy rates","volume":"291","author":"Ha","journal-title":"Radiology"},{"issue":"7","key":"R7","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1016\/S2213-8587(22)00139-5","article-title":"Management of thyroid nodules","volume":"10","author":"Alexander","journal-title":"Lancet Diabetes Endocrinol"},{"issue":"2","key":"R8","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1148\/radiol.2016160641","article-title":"Diagnosis and management of small thyroid nodules: a comparative study with six guidelines for thyroid nodules","volume":"283","author":"Yoon","journal-title":"Radiology"},{"issue":"8","key":"R9","doi-asserted-by":"publisher","first-page":"2550","DOI":"10.1038\/s41591-025-03726-3","article-title":"Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning","volume":"31","author":"Tordjman","journal-title":"Nat Med"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231147","article-title":"Feasibility of using the privacy-preserving large language model Vicuna for labeling radiology reports","volume":"309","author":"Mukherjee","journal-title":"Radiology"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230725","article-title":"Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: a multilingual feasibility study","volume":"307","author":"Adams","journal-title":"Radiology"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2024.111458","article-title":"Transforming free-text radiology reports into structured reports using ChatGPT: a study on thyroid ultrasonography","volume":"175","author":"Jiang","journal-title":"Eur J Radiol"},{"issue":"11","key":"R13","doi-asserted-by":"publisher","first-page":"1697","DOI":"10.1016\/j.ultrasmedbio.2024.07.007","article-title":"Harnessing large language models for structured reporting in breast ultrasound: a comparative study of Open AI (GPT-4.0) and Microsoft Bing (GPT-4)","volume":"50","author":"Liu","journal-title":"Ultrasound Med Biol"},{"issue":"3","key":"R14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer","volume":"308","author":"Fink","journal-title":"Radiology"},{"issue":"5","key":"R15","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230922","article-title":"How AI responds to common lung cancer questions: ChatGPT versus Google Bard","volume":"307","author":"Rahsepar","journal-title":"Radiology"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","first-page":"1041","DOI":"10.1007\/s12020-024-04066-x","article-title":"Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images","volume":"87","author":"Chen","journal-title":"Endocrine"},{"issue":"8","key":"R17","doi-asserted-by":"publisher","DOI":"10.1167\/tvst.14.8.19","article-title":"Enhancing the readability of online pediatric cataract education materials: a comparative study of large language models","volume":"14","author":"Qiu","journal-title":"Transl Vis Sci Technol"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1007\/s12020-024-03981-3","article-title":"Clinical application potential of large language model: a study based on thyroid nodules","volume":"87","author":"Xia","journal-title":"Endocrine"},{"issue":"3","key":"R19","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232255","article-title":"Collaborative enhancement of consistency and accuracy in US diagnosis of thyroid nodules using large language models","volume":"310","author":"Wu","journal-title":"Radiology"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1007\/s12020-020-02441-y","article-title":"2020 Chinese guidelines for ultrasound malignancy risk stratification of thyroid nodules: the C-TIRADS","volume":"70","author":"Zhou","journal-title":"Endocrine"},{"issue":"5","key":"R21","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1016\/j.jacr.2017.01.046","article-title":"ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee","volume":"14","author":"Tessler","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer","volume":"26","author":"Haugen","journal-title":"Thyroid"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Geng J Cai F Wang Y Koeppl H Nakov P Gurevych I . A survey of confidence estimation and calibration in large language models. In: Duh K Gomez H Bethard S , editors. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics; 6577-6595. [doi: 10.18653\/v1\/2024.naacl-long.366]","DOI":"10.18653\/v1\/2024.naacl-long.366"},{"key":"R24","unstructured":"Wang X Wei J Schuurmans D et al. Self-consistency improves chain of thought reasoning in language models. arXiv. Preprint posted online on Mar 21, 2022. [doi: 10.48550\/arXiv.2203.11171]"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Manakul P Liusie A Gales M . SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. Presented at: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing; Dec 6-10, 2023; Singapore. [doi: 10.18653\/v1\/2023.emnlp-main.557]","DOI":"10.18653\/v1\/2023.emnlp-main.557"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit","volume":"70","author":"Cohen","journal-title":"Psychol Bull"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","journal-title":"Biometrics"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","first-page":"1315","DOI":"10.1097\/JTO.0b013e3181ec173d","article-title":"Receiver operating characteristic curve in diagnostic test assessment","volume":"5","author":"Mandrekar","journal-title":"J Thorac Oncol"},{"issue":"3","key":"R29","doi-asserted-by":"publisher","first-page":"169","DOI":"10.4103\/2229-3485.159943","article-title":"Common pitfalls in statistical analysis: clinical versus statistical significance","volume":"6","author":"Ranganathan","journal-title":"Perspect Clin Res"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/s42492-023-00136-5","article-title":"Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential","volume":"6","author":"Lyu","journal-title":"Vis Comput Ind Biomed Art"},{"issue":"7956","key":"R31","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","journal-title":"Nature"},{"issue":"9","key":"R32","doi-asserted-by":"publisher","first-page":"2869","DOI":"10.1210\/clinem\/dgaa322","article-title":"Contemporary thyroid nodule evaluation and management","volume":"105","author":"Grani","journal-title":"J Clin Endocrinol Metab"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1148\/radiol.2018172572","article-title":"Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System","volume":"287","author":"Hoang","journal-title":"Radiology"},{"issue":"7","key":"R34","doi-asserted-by":"publisher","first-page":"598","DOI":"10.7326\/0003-4819-137-7-200210010-00011","article-title":"Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation","volume":"137","author":"Mulherin","journal-title":"Ann Intern Med"},{"key":"R35","unstructured":"Liu A Feng B Xue B et al. DeepSeek-v3 technical report. arXiv. Preprint posted online on Dec 27, 2024. [doi: 10.48550\/arXiv.2412.19437]"},{"key":"R36","unstructured":"Liu A Feng B Wang B et al. DeepSeek-v2: a strong, economical, and efficient mixture-of-experts language model. arXiv. Preprint posted online on May 7, 2024. [doi: 10.48550\/arXiv.2405.04434]"},{"key":"R37","unstructured":"Shazeer N Mirhoseini A Maziarz K et al. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv. Preprint posted online on Jan 23, 2017. [doi: 10.48550\/arXiv.1701.06538]"},{"issue":"8081","key":"R38","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1038\/s41586-025-09422-z","article-title":"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning","volume":"645","author":"Guo","journal-title":"Nature"},{"key":"R39","unstructured":"Achiam J Adler S Agarwal S . GPT-4 technical report. arXiv. Preprint posted online on Mar 15, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R40","unstructured":"Hurst A Lerer A Goucher AP et al. GPT-4o system card. arXiv. Preprint posted online on Oct 25, 2024. [doi: 10.48550\/arXiv.2410.21276]"},{"issue":"3","key":"R41","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1093\/ptj\/85.3.257","volume":"85","author":"Sim","journal-title":"Phys Ther"},{"issue":"6","key":"R42","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-l","article-title":"High agreement but low kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"R43","doi-asserted-by":"publisher","DOI":"10.1007\/s00432-024-05678-6","article-title":"Utilizing large language models in breast cancer management: systematic review","volume":"150","author":"Sorin","journal-title":"J Cancer Res Clin Oncol"},{"issue":"5","key":"R44","doi-asserted-by":"publisher","first-page":"524","DOI":"10.7499\/j.issn.1008-8830.2502042","article-title":"DeepSeek perspective on managing Kawasaki disease in Chinese children","volume":"27","author":"Pan","journal-title":"Zhongguo Dang Dai Er Ke Za Zhi"},{"issue":"1","key":"R45","doi-asserted-by":"publisher","DOI":"10.1186\/s12883-025-04280-8","article-title":"Evaluating ChatGPT and DeepSeek in postdural puncture headache management: a comparative study with international consensus guidelines","volume":"25","author":"Deng","journal-title":"BMC Neurol"},{"issue":"5","key":"R46","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics15050587","article-title":"Management of Dupuytren's disease: a multi-centric comparative analysis between experienced hand surgeons versus artificial intelligence","volume":"15","author":"Seth","journal-title":"Diagnostics (Basel)"},{"issue":"1","key":"R47","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.ijom.2023.09.005","article-title":"The impact and opportunities of large language models like ChatGPT in oral and maxillofacial surgery: a narrative review","volume":"53","author":"Puladi","journal-title":"Int J Oral Maxillofac Surg"},{"issue":"20","key":"R48","doi-asserted-by":"publisher","DOI":"10.3390\/jcm12206524","article-title":"Using generative artificial intelligence tools in cosmetic surgery: a study on rhinoplasty, facelifts, and blepharoplasty procedures","volume":"12","author":"Lim","journal-title":"J Clin Med"},{"issue":"6","key":"R49","doi-asserted-by":"publisher","DOI":"10.3390\/jpm14060612","article-title":"Comparative analysis of large language models in emergency plastic surgery decision-making: the role of physical exam data","volume":"14","author":"Borna","journal-title":"J Pers Med"},{"issue":"5","key":"R50","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","article-title":"ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports","volume":"34","author":"Jeblick","journal-title":"Eur Radiol"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2023.05.024","article-title":"Performance of generative large language models on ophthalmology board\u2013style questions","volume":"254","author":"Cai","journal-title":"Am J Ophthalmol"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1518049","article-title":"Benefits, limits, and risks of ChatGPT in medicine","volume":"8","author":"Tangsrivimol","journal-title":"Front Artif Intell"},{"issue":"5","key":"R53","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582","article-title":"Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations","volume":"307","author":"Bhayana","journal-title":"Radiology"},{"issue":"2","key":"R54","doi-asserted-by":"publisher","first-page":"176","DOI":"10.62347\/UIAP7979","article-title":"DeepSeek vs ChatGPT: a comparison study of their performance in answering prostate cancer radiotherapy questions in multiple languages","volume":"13","author":"Luo","journal-title":"Am J Clin Exp Urol"},{"key":"R55","doi-asserted-by":"publisher","DOI":"10.2196\/73469","article-title":"Evaluating the performance of DeepSeek-R1 and DeepSeek-V3 versus OpenAI models in the Chinese National Medical Licensing Examination: cross-sectional comparative study","volume":"11","author":"Wang","journal-title":"JMIR Med Educ"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93890","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:23:37Z","timestamp":1785536617000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93890"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,28]]},"references-count":55,"URL":"https:\/\/doi.org\/10.2196\/93890","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,28]]},"article-number":"v28i12e93890"}}