{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T06:16:48Z","timestamp":1785392208873,"version":"3.55.0"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Thyroid nodules are common, with ultrasound imaging as the primary modality for their assessment. Risk stratification systems like the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) have been developed but suffer from interobserver variability and low specificity. Artificial intelligence, particularly large language models (LLMs) with multimodal capabilities, presents opportunities for efficient end-to-end diagnostic processes. However, their clinical utility remains uncertain.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study evaluates the accuracy and consistency of multimodal LLMs for thyroid nodule risk stratification using the ACR TI-RADS system, examining the effects of model fine-tuning, image annotation, prompt engineering, and comparing open-source versus commercial models.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n In total, 3 multimodal vision-language models were evaluated: Microsoft\u2019s open-source Large Language and Visual Assistant (LLaVA) model, its medically fine-tuned variant (Large Language and Vision Assistant for bioMedicine [LLaVA-Med]), and OpenAI\u2019s commercial o3 model. A total of 192 thyroid nodules from publicly available ultrasound image datasets were assessed. Each model was evaluated using 2 prompts (basic and modified) and 2 image scenarios (unlabeled vs radiologist-annotated), yielding 6912 responses. Model outputs were compared with expert ratings for accuracy and consistency. Statistical comparisons included\n \n Chi-square tests, Mann-Whitney\n U<\/jats:italic>\n tests, and Fleiss\u2019 kappa for interrater reliability.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Overall, 88.4% (6110\/6912) of responses were valid, with the o3 model producing the highest validity rate (2273\/2304, 98.6%), followed by LLaVA (2108\/2304, 91.5%) and LLaVA-Med (1729\/2304, 75%;\n P<\/jats:italic>\n <.001). The o3 model demonstrated the highest accuracy overall, achieving up to 57.3% accuracy in Thyroid Imaging Reporting and Data System (TI-RADS) classification, although still remaining suboptimal. Labeled images improved accuracy marginally in nodule margin assessment only when evaluating LLaVA models (407\/768, 53% to 447\/768, 58.2%;\n P<\/jats:italic>\n =.04). Prompt engineering improved accuracy for composition (649\/1,152, 56.3% vs 483\/1152, 41.9%;\n P<\/jats:italic>\n <.001), but significantly reduced accuracy for shape, margins, and overall classification. Consistency was the highest with the o3 model (up to 85.4%), but was comparable for LLaVA and significantly improved with image labeling and modified prompts across multiple TI-RADS categories (\n P<\/jats:italic>\n <.001). Subgroup analysis for o3 alone showed prompt engineering did not affect accuracy significantly but markedly improved consistency across all TI-RADS categories (up to 97.1% for shape,\n P<\/jats:italic>\n <.001). Interrater reliability was consistently poor across all combinations (Fleiss\u2019 kappa<0.60).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The study demonstrates the comparative advantages and limitations of multimodal LLMs for thyroid nodule risk stratification. While the commercial model (o3) consistently outperformed open-source models in accuracy and consistency, even the best-performing model outputs remained suboptimal for direct clinical deployment. Prompt engineering significantly enhanced output consistency, particularly in the commercial model. These findings underline the importance of strategic model optimization techniques and highlight areas requiring further development before multimodal LLMs can be reliably used in clinical thyroid imaging workflows.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/70863","type":"journal-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T20:44:48Z","timestamp":1755636288000},"page":"e70863-e70863","source":"Crossref","is-referenced-by-count":5,"title":["A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8795-7491","authenticated-orcid":false,"given":"Gerald Gui Ren","family":"Sng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2602-5343","authenticated-orcid":false,"given":"Yi","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9715-6970","authenticated-orcid":false,"given":"Daniel Yan Zheng","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4991-9790","authenticated-orcid":false,"given":"Joshua Yi Min","family":"Tung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7785-2987","authenticated-orcid":false,"given":"Jen Hong","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2232-5298","authenticated-orcid":false,"given":"Chiaw Ling","family":"Chng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,8,19]]},"reference":[{"issue":"6","key":"R1","doi-asserted-by":"publisher","first-page":"901","DOI":"10.1016\/j.beem.2008.09.019","article-title":"Epidemiology of thyroid nodules","volume":"22","author":"Dean","journal-title":"Best Pract Res Clin Endocrinol Metab"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","first-page":"840","DOI":"10.3348\/kjr.2020.0578","article-title":"2020 imaging guidelines for thyroid nodules and differentiated thyroid cancer: Korean Society of Thyroid Radiology","volume":"22","author":"Lee","journal-title":"Korean J Radiol"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: The American Thyroid Association Guidelines Task Force on thyroid nodules and differentiated thyroid cancer","volume":"26","author":"Haugen","journal-title":"Thyroid"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1016\/j.jacr.2017.01.046","article-title":"ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee","volume":"14","author":"Tessler","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"162","DOI":"10.2214\/AJR.17.19192","article-title":"Interobserver variability of sonographic features used in the American College of Radiology Thyroid Imaging Reporting and Data System","volume":"211","author":"Hoang","journal-title":"AJR Am J Roentgenol"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1530\/EC-17-0336","article-title":"Interobserver agreement of various thyroid imaging reporting and data systems","volume":"7","author":"Grani","journal-title":"Endocr Connect"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"38","DOI":"10.2214\/AJR.19.22691","article-title":"Diagnostic performance of American College of Radiology TI-RADS: a systematic review and meta-analysis","volume":"216","author":"Li","journal-title":"AJR Am J Roentgenol"},{"issue":"8","key":"R8","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-080558","article-title":"Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework","volume":"14","author":"Maitland","journal-title":"BMJ Open"},{"key":"R10","unstructured":"Li Q Yang X Wang H et al. From beginner to expert: modeling medical knowledge into general llms. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2312.01040 [Accessed 11-08-2025]"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-46411-8","article-title":"Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks","volume":"15","author":"Sandmann","journal-title":"Nat Commun"},{"key":"R12","unstructured":"Liu H Li C Wu Q Lee YJ . Visual instruction tuning. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2304.08485 [Accessed 11-08-2025]"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Li C Wong C Zhang S et al. LLaVA-med: training a large language-and-vision assistant for biomedicine in one day. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2306.00890 [Accessed 11-08-2025]","DOI":"10.32388\/VLXB6M"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1089\/thy.2022.0560","article-title":"Artificial intelligence for evaluation of thyroid nodules: a primer","volume":"33","author":"Tessler","journal-title":"Thyroid"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2214\/AJR.22.27430","article-title":"Artificial intelligence (AI) tools for thyroid nodules on ultrasound, from the Special Series on AI applications","volume":"219","author":"Wildman-Tobriner","journal-title":"AJR Am J Roentgenol"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232255","article-title":"Collaborative enhancement of consistency and accuracy in US diagnosis of thyroid nodules using large language models","volume":"310","author":"Wu","journal-title":"Radiology"},{"issue":"6","key":"R17","doi-asserted-by":"publisher","first-page":"716","DOI":"10.1016\/j.eprac.2025.03.008","article-title":"ChatGPT-4\u2019s accuracy in estimating thyroid nodule features and cancer risk from ultrasound images","volume":"31","author":"Cabezas","journal-title":"Endocr Pract"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.lanwpc.2024.101048","article-title":"The application of multimodal large language models in medicine","volume":"45","author":"Qiu","journal-title":"Lancet Reg Health West Pac"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial","volume":"25","author":"Mesk\u00f3","journal-title":"J Med Internet Res"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Han T Adams LC Nebelung S Kather JN Bressem KK Truhn D . Multimodal large language models are generalist medical image interpreters. Health Informatics. [doi: 10.1101\/2023.12.21.23300146]","DOI":"10.1101\/2023.12.21.23300146"},{"key":"R21","unstructured":"Zhang S Dong L Li X et al. Instruction tuning for large language models: a survey. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2308.10792 [Accessed 11-08-2025]"},{"key":"R22","unstructured":"Introducing o3 and o4-mini. OpenAI. URL: https:\/\/openai.com\/index\/introducing-o3-and-o4-mini [Accessed 30-07-2025]"},{"issue":"3","key":"R23","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.210174","article-title":"Toward reduction in false-positive thyroid nodule biopsies with a deep learning-based risk stratification system using US cine-clip images","volume":"4","author":"Yamashita","journal-title":"Radiol Artif Intell"},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Imani S Du L Shrivastava H . MathPrompter: mathematical reasoning using large language models. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2303.05398 [Accessed 11-08-2025]","DOI":"10.18653\/v1\/2023.acl-industry.4"},{"issue":"6","key":"R25","doi-asserted-by":"publisher","first-page":"151","DOI":"10.3390\/bdcc9060151","article-title":"The use of large language models in ophthalmology: a scoping review on current use-cases and considerations for future works in this field","volume":"9","author":"See","journal-title":"BDCC"},{"issue":"5","key":"R26","doi-asserted-by":"publisher","DOI":"10.1056\/aioa2300151","article-title":"Comparative evaluation of LLMs in clinical oncology","volume":"1","author":"Rydzewski","journal-title":"NEJM AI"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2024.1513608","article-title":"Feasibility of large language models for CEUS LI-RADS categorization of small liver nodules in patients at risk for hepatocellular carcinoma","volume":"14","author":"Huang","journal-title":"Front Oncol"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Balduzzi A Pastena M Tondato S et al. Exploring chatbot applications in pancreatic disease treatment: potential and pitfalls. Scierxiv. Preprint posted online on 2025. [doi: 10.20517\/scierxiv202506.040.v1]","DOI":"10.20517\/scierxiv202506.040.v1"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1380148","article-title":"Evaluation of large language models as a diagnostic aid for complex medical cases","volume":"11","author":"R\u00edos-Hoyo","journal-title":"Front Med (Lausanne)"},{"issue":"6","key":"R30","doi-asserted-by":"publisher","first-page":"103510","DOI":"10.1016\/j.ipm.2023.103510","article-title":"What is the limitation of multimodal LLMs? A deeper look into multimodal LLMs through prompt probing","volume":"60","author":"Qi","journal-title":"Inf Process Manag"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2019190177","article-title":"The importance of image resolution in building deep learning models for medical imaging","volume":"2","author":"Lakhani","journal-title":"Radiol Artif Intell"},{"key":"R32","unstructured":"Thinking with images. OpenAI. URL: https:\/\/openai.com\/index\/thinking-with-images [Accessed 30-07-2025]"},{"key":"R33","unstructured":"Singhal K Tu T Gottweis J et al. Towards expert-level medical question answering with large language models. arXiv. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2305.09617 [Accessed 11-08-2025]"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-64827-6","article-title":"OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models","volume":"14","author":"Maharjan","journal-title":"Sci Rep"},{"issue":"19","key":"R35","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics14192138","article-title":"Interobserver agreement in ultrasound risk stratification systems for thyroid nodules on static images versus cine-loop video sequences","volume":"14","author":"Schenke","journal-title":"Diagnostics (Basel)"},{"issue":"4","key":"R36","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.26599\/BDMA.2024.9020058","article-title":"TV-SAM: increasing zero-shot segmentation performance on multimodal medical images using GPT-4 generated descriptive prompts without human annotation","volume":"7","author":"Jiang","journal-title":"Big Data Min Anal"},{"key":"R37","author":"Zhao","journal-title":"arXiv"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e70863","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:59:52Z","timestamp":1782842392000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e70863"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,19]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/70863","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,19]]},"article-number":"v9i6e70863"}}