{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T02:07:19Z","timestamp":1789092439667,"version":"build-2803163510"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Cancer"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Diagnosing and managing follicular thyroid neoplasms (FTNs) remains a significant challenge, as the malignancy risk cannot be determined until after diagnostic surgery.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to use interpretable machine learning to predict the malignancy risk of FTNs preoperatively in a real-world setting.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a retrospective cohort study at the Peking University Third Hospital in Beijing, China. Patients with postoperative pathological diagnoses of follicular thyroid adenoma (FTA) or follicular thyroid carcinoma (FTC) were included, excluding those without preoperative thyroid ultrasonography. We used 22 predictors involving demographic characteristics, thyroid sonography, and hormones to train 5 machine learning models: logistic regression, least absolute shrinkage and selection operator regression, random forest, extreme gradient boosting, and support vector machine. The optimal model was selected based on discrimination, calibration, interpretability, and parsimony. To address the highly imbalanced data (FTA:FTC ratio>5:1), model discrimination was assessed using both the area under the receiver operating characteristic curve and the area under the precision-recall curve (AUPRC). To interpret the model, we used Shapley Additive Explanations values and partial dependence and individual conditional expectation plots. Additionally, a systematic review was performed to synthesize existing evidence and validate the discrimination ability of the previously developed Thyroid Imaging Reporting and Data System for Follicular Neoplasm scoring criteria to differentiate between benign and malignant FTNs using our data.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The cohort included 1539 patients (mean age 47.98, SD 14.15 years; female: n=1126, 73.16%) with 1672 FTN tumors (FTA: n=1414; FTC: n=258; FTA:FTC ratio=5.5). The random forest model emerged as optimal, identifying mean thyroid-stimulating hormone (TSH) score, mean tumor diameter, mean TSH, TSH instability, and TSH measurement levels as the top 5 predictors in discriminating FTA from FTC, with the area under the receiver operating characteristic curve of 0.79 (95% CI 0.77\u20100.81) and AUPRC of 0.40 (95% CI 0.37-0.44). Malignancy risk increased nonlinearly with larger tumor diameters and higher TSH instability but decreased nonlinearly with higher mean TSH scores or mean TSH levels. FTCs with small sizes (mean diameter 2.88, SD 1.38 cm) were more likely to be misclassified as FTAs compared to larger ones (mean diameter 3.71, SD 1.36 cm). The systematic review of the 7 included studies revealed that (1) the FTA:FTC ratio varied from 0.6 to 4.0, lower than the natural distribution of 5.0; (2) no studies assessed prediction performance using AUPRC in unbalanced datasets; and (3) external validations of Thyroid Imaging Reporting and Data System for Follicular Neoplasm scoring criteria underperformed relative to the original study.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Tumor size and TSH measurements were important in screening FTN malignancy risk preoperatively, but accurately predicting the risk of small-sized FTNs remains challenging. Future research should address the limitations posed by the extreme imbalance in FTA and FTC distributions in real-world data.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/66269","type":"journal-article","created":{"date-parts":[[2024,12,4]],"date-time":"2024-12-04T10:13:38Z","timestamp":1733307218000},"page":"e66269-e66269","source":"Crossref","is-referenced-by-count":12,"title":["Interpretable Machine Learning to Predict the Malignancy Risk of Follicular Thyroid Neoplasms in Extremely Unbalanced Data: Retrospective Cohort Study and Literature Review"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-7603-337X","authenticated-orcid":false,"given":"Rui","family":"Shan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8445-5354","authenticated-orcid":false,"given":"Xin","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6640-7140","authenticated-orcid":false,"given":"Jing","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9251-5217","authenticated-orcid":false,"given":"Zheng","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5412-1367","authenticated-orcid":false,"given":"Yuan-Jia","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4966-1398","authenticated-orcid":false,"given":"Bo","family":"Han","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2968-555X","authenticated-orcid":false,"given":"Run-Ze","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-2655-3639","authenticated-orcid":false,"given":"Jiu-Ping","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0851-1644","authenticated-orcid":false,"given":"Gui-Lan","family":"Kong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9896-7871","authenticated-orcid":false,"given":"Hui","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5690-674X","authenticated-orcid":false,"given":"Fang","family":"Mei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6981-1142","authenticated-orcid":false,"given":"Shi-Bing","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6452-9082","authenticated-orcid":false,"given":"Bang-Kai","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6598-5400","authenticated-orcid":false,"given":"Hui","family":"Tian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7717-5031","authenticated-orcid":false,"given":"Yang","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7949-1297","authenticated-orcid":false,"given":"Wu-Cai","family":"Xiao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9073-0418","authenticated-orcid":false,"given":"Xiang-Yun","family":"Yao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5740-0602","authenticated-orcid":false,"given":"Jing-Ming","family":"Ye","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0019-3597","authenticated-orcid":false,"given":"Bo","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1427-1023","authenticated-orcid":false,"given":"Chun-Hui","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1165-7093","authenticated-orcid":false,"given":"Fan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0405-2348","authenticated-orcid":false,"given":"Zheng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,2,10]]},"reference":[{"issue":"5","key":"R1","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1001\/jama.2023.26348","article-title":"Thyroid cancer: a review","volume":"331","author":"Boucai","journal-title":"JAMA"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1016\/S2213-8587(17)30325-X","article-title":"Follicular thyroid cancer and H\u00fcrthle cell carcinoma: challenges in diagnosis, treatment, and clinical management","volume":"6","author":"Grani","journal-title":"Lancet Diabetes Endocrinol"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1634\/theoncologist.2010-0405","article-title":"Follicular adenoma and carcinoma of the thyroid gland","volume":"16","author":"McHenry","journal-title":"Oncologist"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on thyroid nodules and differentiated thyroid cancer","volume":"26","author":"Haugen","journal-title":"Thyroid"},{"issue":"5","key":"R5","doi-asserted-by":"publisher","first-page":"37358008","DOI":"10.1530\/ETJ-23-0067","article-title":"2023 European Thyroid Association Clinical Practice Guidelines for thyroid nodule management","volume":"12","author":"Durante","journal-title":"Eur Thyroid J"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.14514","article-title":"Prospective comparison of medical oncologists and a machine learning model to predict 3-month mortality in patients with metastatic solid tumors","volume":"5","author":"Zachariah","journal-title":"JAMA Netw Open"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.7226","article-title":"Development and validation of machine learning models for predicting occult nodal metastasis in early-stage oral cavity squamous cell carcinoma","volume":"5","author":"Farrokhian","journal-title":"JAMA Netw Open"},{"issue":"8","key":"R8","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.11768","article-title":"Development of a machine learning model for survival risk stratification of patients with advanced oral cancer","volume":"3","author":"Tseng","journal-title":"JAMA Netw Open"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Bennett M Hayes K Kleczyk EJ Mehta R . Similarities and differences between machine learning and traditional advanced statistical modeling in healthcare analytics. arXiv. Preprint posted online onJan7, 2022. [doi: 10.48550\/arXiv.2201.02469]","DOI":"10.5772\/intechopen.105116"},{"issue":"9","key":"R10","doi-asserted-by":"publisher","DOI":"10.3390\/medicina56090455","article-title":"Comparison of conventional statistical methods with machine learning in medicine: diagnosis, drug development, and treatment","volume":"56","author":"Rajula","journal-title":"Medicina (Kaunas)"},{"issue":"12","key":"R11","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1038\/nmeth.4526","article-title":"Points of significance: machine learning: a primer","volume":"14","author":"Bzdok","journal-title":"Nat Methods"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s42081-018-0009-3","article-title":"Data science vs. statistics: two cultures?","volume":"1","author":"Carmichael","journal-title":"Jpn J Stat Data Sci"},{"issue":"3","key":"R13","doi-asserted-by":"publisher","first-page":"257","DOI":"10.14366\/usg.19069","article-title":"Application of machine learning to ultrasound images to differentiate follicular neoplasms of the thyroid gland","volume":"39","author":"Shin","journal-title":"Ultrasonography"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/RCT.0000000000001078","article-title":"Development and validation of a diagnostic nomogram for the preoperative differentiation between follicular thyroid carcinoma and follicular thyroid adenomas","volume":"45","author":"Tang","journal-title":"J Comput Assist Tomogr"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/j.surg.2023.06.053","article-title":"Generating a multimodal artificial intelligence model to differentiate benign and malignant follicular neoplasms of the thyroid: a proof-of-concept study","volume":"175","author":"Lin","journal-title":"Surgery"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/3098293","article-title":"Differentiation of the follicular neoplasm on the gray-scale US by image selection subsampling along with the marginal outline using convolutional neural network","volume":"2017","author":"Seo","journal-title":"Biomed Res Int"},{"issue":"9","key":"R17","doi-asserted-by":"publisher","first-page":"2732","DOI":"10.1097\/JS9.0000000000000506","article-title":"Automated diagnosis and management of follicular thyroid nodules based on the devised small-dataset interpretable foreground optimization network deep learning: a multicenter diagnostic study","volume":"109","author":"Yang","journal-title":"Int J Surg"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230949","article-title":"US risk stratification system for follicular thyroid neoplasms","volume":"309","author":"Li","journal-title":"Radiology"},{"issue":"24","key":"R19","doi-asserted-by":"publisher","first-page":"33137800","DOI":"10.1088\/1361-6560\/abc6f2","article-title":"Segmentation and classification of thyroid follicular neoplasm using cascaded convolutional neural network","volume":"65","author":"Yang","journal-title":"Phys Med Biol"},{"issue":"5","key":"R20","doi-asserted-by":"publisher","first-page":"983","DOI":"10.1093\/jamia\/ocac002","article-title":"Trends in the conduct and reporting of clinical prediction model development and validation: a systematic review","volume":"29","author":"Yang","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"R21","doi-asserted-by":"publisher","first-page":"1107","DOI":"10.1089\/thy.2013.0588","article-title":"Hashimoto\u2019s thyroiditis pathology and risk for thyroid cancer","volume":"24","author":"Paparodis","journal-title":"Thyroid"},{"key":"R22","unstructured":"MDT team of thyroid and parathyroid tumor [Article in Chinese]. Peking University Third Hospital. URL: https:\/\/www.puh3.net.cn\/info\/2891\/118191.htm [Accessed 15-04-2024]"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.g7594","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement","volume":"350","author":"Collins","journal-title":"BMJ"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s12022-022-09707-3","article-title":"Overview of the 2022 WHO classification of thyroid neoplasms","volume":"33","author":"Baloch","journal-title":"Endocr Pathol"},{"issue":"11","key":"R25","doi-asserted-by":"publisher","first-page":"1174","DOI":"10.1089\/jwh.2023.0172","article-title":"The association of pregnancy with disease progression in patients previously treated for differentiated thyroid cancer: a propensity score-matched retrospective cohort study","volume":"32","author":"Li","journal-title":"J Womens Health (Larchmt)"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"837","DOI":"10.1210\/clinem\/dgad557","article-title":"Pregnancy and progression of differentiated thyroid cancer: a propensity score-matched retrospective cohort study","volume":"109","author":"Xiao","journal-title":"J Clin Endocrinol Metab"},{"issue":"5","key":"R27","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1097\/CM9.0000000000003008","article-title":"Pregnancy and the disease recurrence of patients previously treated for differentiated thyroid cancer: a systematic review and meta analysis","volume":"137","author":"Shan","journal-title":"Chin Med J (Engl)"},{"key":"R28","unstructured":"Mincer DL StatPearls J . Treasure Island (FL) Ineligible Companies Disclosure: Ishwarlal Jialal Declares No Relevant Financial Relationships with Ineligible Companies. StatPearls Publishing LLC; 2024."},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1007\/s00268-022-06770-z","article-title":"Thyroid-stimulating hormone, age, and tumor size are risk factors for progression during active surveillance of low-risk papillary thyroid microcarcinoma in adults","volume":"47","author":"Ito","journal-title":"World J Surg"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1016\/S2215-0366(23)00066-4","article-title":"Early trajectory of clinical global impression as a transdiagnostic predictor of psychiatric hospitalisation: a retrospective cohort study","volume":"10","author":"Taquet","journal-title":"Lancet Psychiatry"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.3389\/fendo.2024.1369344","article-title":"Associations of gestational thyrotropin levels with disease progression among pregnant women with differentiated thyroid cancer: a retrospective cohort study","volume":"15","author":"Li","journal-title":"Front Endocrinol (Lausanne)"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Kotthoff L Sonabend R Foss N Bischl B . Introduction and overview. In: Bischl B Sonabend R Kotthoff L Lang M , editors. Applied Machine Learning Using mlr3 in R. CRC Press; 2024. [doi: 10.1201\/9781003402848-1]","DOI":"10.1201\/9781003402848-1"},{"key":"R33","volume":"12","author":"Pedregosa","journal-title":"J Mach Learn Res"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","journal-title":"Nat Mach Intell"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1007\/s10994-021-06121-4","article-title":"InfoGram and admissible machine learning","volume":"111","author":"Mukhopadhyay","journal-title":"Mach Learn"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Mohammed R Rawashdeh J Abdullah M . Machine learning with oversampling and undersampling techniques: overview study and experimental results. Presented at: 2020 11th International Conference on Information and Communication Systems (ICICS); Apr 7-9, 2020; Irbid, Jordan. [doi: 10.1109\/ICICS49469.2020.239556]","DOI":"10.1109\/ICICS49469.2020.239556"},{"issue":"3","key":"R37","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","journal-title":"PLoS One"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2700","article-title":"The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration","volume":"339","author":"Liberati","journal-title":"BMJ"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s40537-023-00857-7","article-title":"Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data","volume":"11","author":"Yang","journal-title":"J Big Data"},{"issue":"7","key":"R40","doi-asserted-by":"publisher","first-page":"4903","DOI":"10.1007\/s10994-022-06296-4","article-title":"A theoretical distribution analysis of synthetic minority oversampling technique (SMOTE) for imbalanced learning","volume":"113","author":"Elreedy","journal-title":"Mach Learn"},{"issue":"2","key":"R41","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1109\/TSMCB.2008.2007853","article-title":"Exploratory undersampling for class-imbalance learning","volume":"39","author":"Liu","journal-title":"IEEE Trans Syst Man Cybern B Cybern"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-074820","article-title":"Evaluation of clinical prediction models (part 2): how to undertake an external validation study","volume":"384","author":"Riley","journal-title":"BMJ"},{"issue":"4","key":"R43","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1001\/jamasurg.2023.6687","article-title":"Practical guide to machine learning and artificial intelligence in surgical education research","volume":"159","author":"Hashimoto","journal-title":"JAMA Surg"}],"container-title":["JMIR Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e66269","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:28:57Z","timestamp":1785536937000},"score":1,"resource":{"primary":{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e66269"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,10]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/66269","relation":{},"ISSN":["2369-1999"],"issn-type":[{"value":"2369-1999","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,10]]},"article-number":"v11i5e66269"}}