{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T21:46:33Z","timestamp":1790804793324,"version":"4.1.0"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T00:00:00Z","timestamp":1676592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,3,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Indeterminate thyroid nodules (ITN) are common and often lead to (sometimes unnecessary) diagnostic surgery. We aimed to evaluate the performance of two machine learning methods (ML), based on routinely available features to predict the risk of malignancy (RM) of ITN.<\/jats:p><\/jats:sec><jats:sec><jats:title>Design<\/jats:title><jats:p>Multi-centric diagnostic retrospective cohort study conducted between 2010 and 2020.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Adult patients who underwent surgery for at least one Bethesda III-V thyroid nodule (TN) with fully available medical records were included. Of the 7917 records reviewed, eligibility criteria were met in 1288 patients with 1335 TN. Patients were divided into training (940 TN) and validation cohort (395 TN). The diagnostic performance of a multivariate logistic regression model (LR) and its nomogram, and a random forest model (RF) in predicting the nature and RM of a TN were evaluated. All available clinical, biological, ultrasound, and cytological data of the patients were collected and used to construct the two algorithms.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>There were 253 (19%), 693 (52%), and 389 (29%) TN classified as Bethesda III, IV, and V, respectively, with an overall RM of 35%. Both cohorts were well-balanced for baseline characteristics. Both models were validated on the validation cohort, with performances in terms of specificity, sensitivity, positive predictive value, negative predictive value, and area under the receiver operating characteristic curve of 90%, 57.3%, 73.4%, 81.4%, 84% (CI95%: 78.5%-89.5%) for the LR model, and 87.6%, 54.7%, 68.1%, 80%, 82.6% (CI95%: 77.4%-87.9%) for the RF model, respectively.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Our ML models performed well in predicting the nature of Bethesda III-V TN. In addition, our freely available online nomogram helped to refine the RM, identifying low-risk TN that may benefit from surveillance in up to a third of ITN, and thus may reduce the number of unnecessary surgeries.<\/jats:p><\/jats:sec>","DOI":"10.1093\/ejendo\/lvad017","type":"journal-article","created":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T15:39:24Z","timestamp":1676648364000},"page":"249-257","source":"Crossref","is-referenced-by-count":18,"title":["Application of machine learning methods to guide patient management by predicting the risk of malignancy of Bethesda III-V thyroid nodules"],"prefix":"10.1093","volume":"188","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3838-513X","authenticated-orcid":false,"given":"Gr\u00e9goire","family":"D\u2019Andr\u00e9a","sequence":"first","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jocelyn","family":"Gal","sequence":"additional","affiliation":[{"name":"Department of Statistics, Centre Antoine Lacassagne , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lo\u00efc","family":"Mandine","sequence":"additional","affiliation":[{"name":"Department of Statistics, Centre Antoine Lacassagne , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olivier","family":"Dassonville","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clair","family":"Vandersteen","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Guevara","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laurent","family":"Castillo","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gilles","family":"Poissonnet","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dorian","family":"Culi\u00e9","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roxane","family":"Elaldi","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J\u00e9r\u00f4me","family":"Sarini","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, University Cancer Institute of Toulouse-Oncopole , Toulouse 31400 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne","family":"Decotte","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Toulouse University Hospital, H\u00f4pital Larrey , Toulouse 31400 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claire","family":"Renaud","sequence":"additional","affiliation":[{"name":"Thoracic Surgery Department, Toulouse University Hospital, H\u00f4pital Larrey , Toulouse 31400 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"S\u00e9bastien","family":"Vergez","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, University Cancer Institute of Toulouse-Oncopole , Toulouse 31400 , France"},{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Toulouse University Hospital, H\u00f4pital Larrey , Toulouse 31400 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renaud","family":"Schiappa","sequence":"additional","affiliation":[{"name":"Department of Statistics, Centre Antoine Lacassagne , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emmanuel","family":"Chamorey","sequence":"additional","affiliation":[{"name":"Department of Statistics, Centre Antoine Lacassagne , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yann","family":"Ch\u00e2teau","sequence":"additional","affiliation":[{"name":"Department of Statistics, Centre Antoine Lacassagne , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandre","family":"Bozec","sequence":"additional","affiliation":[{"name":"Otorhinolaryngology and Head and Neck Surgery Department, Institut Universitaire de la Face et du Cou, GHS Nice University Hospital\u2014Antoine Lacassagne Centre, C\u00f4te d\u2019Azur University , Nice 06103 , France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,2,17]]},"reference":[{"issue":"Suppl 2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"S98","DOI":"10.21037\/gs.2018.12.06","article-title":"Approach to cytological indeterminate thyroid nodules","volume":"8","author":"Bongiovanni","year":"2019","journal-title":"Gland Surg"},{"issue":"1","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1089\/thy.2015.0020","article-title":"2015 American thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American thyroid association guidelines task force on thyroid nodules and differentiated thyroid cancer","volume":"26","author":"Haugen","year":"2016","journal-title":"Thyroid"},{"issue":"11","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1089\/thy.2017.0500","article-title":"The 2017 Bethesda system for reporting thyroid cytopathology","volume":"27","author":"Cibas","year":"2017","journal-title":"Thyroid"},{"issue":"8","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1056\/NEJM199302253280807","article-title":"Management of a solitary thyroid nodule","volume":"328","author":"Mazzaferri","year":"1993","journal-title":"N Engl J Med"},{"issue":"13","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1338","DOI":"10.1001\/jama.2017.2719","article-title":"Trends in thyroid cancer incidence and mortality in the United States, 1974-2013","volume":"317","author":"Lim","year":"2017","journal-title":"JAMA"},{"issue":"2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1089\/thy.2017.0419","article-title":"Cancer risk associated with nuclear atypia in cytologically indeterminate thyroid nodules: a systematic review and meta-analysis","volume":"28","author":"Valderrabano","year":"2018","journal-title":"Thyroid"},{"issue":"6","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1016\/S2213-8587(20)30115-7","article-title":"Global trends in thyroid cancer incidence and the impact of overdiagnosis","volume":"8","author":"Li","year":"2020","journal-title":"Lancet Diabetes Endocrinol"},{"issue":"9","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1089\/thy.2018.0779","article-title":"Molecular testing versus diagnostic lobectomy in Bethesda III\/IV thyroid nodules: a cost-effectiveness analysis","volume":"29","author":"Nicholson","year":"2019","journal-title":"Thyroid"},{"issue":"10","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"736","DOI":"10.3390\/genes10100736","article-title":"The role of molecular testing for the indeterminate thyroid FNA","volume":"10","author":"Rossi","year":"2019","journal-title":"Genes (Basel)"},{"issue":"2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1007\/s12020-015-0785-5","article-title":"A meta-analysis-derived proposal for a clinical, ultrasonographic, and cytological scoring system to evaluate thyroid nodules: the \u201cCUT\u201d score","volume":"52","author":"Ianni","year":"2016","journal-title":"Endocrine"},{"issue":"2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1159\/000494720","article-title":"The malignancy risk assessment of cytologically indeterminate thyroid nodules improves markedly by using a predictive model","volume":"8","author":"Ocal","year":"2019","journal-title":"Eur Thyroid J"},{"issue":"7","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.1002\/hed.23075","article-title":"Nomogram for selecting thyroid nodules for ultrasound-guided fine-needle aspiration biopsy based on a quantification of risk of malignancy","volume":"35","author":"Nixon","year":"2013","journal-title":"Head Neck"},{"issue":"Suppl 1","key":"2023032510163214500_","first-page":"S1","article-title":"Mcgill thyroid nodule score (MTNS): \u201crating the risk,\u201d a novel predictive scheme for cancer risk determination","volume":"40","author":"Sands","year":"2011","journal-title":"J Otolaryngol Head Neck Surg"},{"issue":"7","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1530\/EC-20-0648","article-title":"Indeterminate thyroid cytology: detecting malignancy using analysis of nuclear images","volume":"10","author":"Hayashi","year":"2021","journal-title":"Endocr Connect"},{"issue":"4","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"e250","DOI":"10.1016\/S2589-7500(21)00041-8","article-title":"Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study","volume":"3","author":"Peng","year":"2021","journal-title":"Lancet Digit Health"},{"issue":"5","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1159\/000478927","article-title":"European Thyroid association guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS","volume":"6","author":"Russ","year":"2017","journal-title":"Eur Thyroid J"},{"issue":"5","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1530\/EJE-12-0936","article-title":"Prospective evaluation of thyroid imaging reporting and data system on 4550 nodules with and without elastography","volume":"168","author":"Russ","year":"2013","journal-title":"Eur J Endocrinol"},{"issue":"5","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.1210\/jc.2008-1724","article-title":"An ultrasonogram reporting system for thyroid nodules stratifying cancer risk for clinical management","volume":"94","author":"Horvath","year":"2009","journal-title":"J Clin Endocrinol Metab"},{"issue":"11","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"e0225253","DOI":"10.1371\/journal.pone.0225253","article-title":"Visualising statistical models using dynamic nomograms","volume":"14","author":"Jalali","year":"2019","journal-title":"PLoS One"},{"issue":"1","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1001\/jamaoncol.2020.5935","article-title":"Effectiveness of molecular testing techniques for diagnosis of indeterminate thyroid nodules: a randomized clinical trial","volume":"7","author":"Livhits","year":"2021","journal-title":"JAMA Oncol"},{"issue":"2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1001\/jamaoncol.2018.4616","article-title":"Performance of a multigene genomic classifier in thyroid nodules with indeterminate cytology: a prospective blinded multicenter study","volume":"5","author":"Steward","year":"2019","journal-title":"JAMA Oncol"},{"issue":"9","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"783","DOI":"10.1001\/jamaoto.2019.1449","article-title":"Comparison of postmarketing findings vs the initial clinical validation findings of a thyroid nodule gene expression classifier: a systematic review and meta-analysis","volume":"145","author":"Valderrabano","year":"2019","journal-title":"JAMA Otolaryngol Head Neck Surg"},{"issue":"9","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1001\/jamaoto.2020.1851","article-title":"Molecular profiling of thyroid nodules-are these findings meaningful, or merely measurable? : A review","volume":"146","author":"Morris","year":"2020","journal-title":"JAMA Otolaryngol Head Neck Surg"},{"issue":"3","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1159\/000468519","article-title":"European Thyroid association guidelines regarding thyroid nodule molecular fine-needle aspiration cytology diagnostics","volume":"6","author":"Paschke","year":"2017","journal-title":"Eur Thyroid J"},{"issue":"3","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1310","DOI":"10.7150\/thno.50333","article-title":"From biomarkers to therapeutic targets: the promise of PD-L1 in thyroid autoimmunity and cancer","volume":"11","author":"D\u2019Andrea","year":"2021","journal-title":"Theranostics"},{"issue":"1","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"17901","DOI":"10.1038\/s41598-017-18207-y","article-title":"A proposal to stratify the intermediate-risk thyroid nodules according to the AACE\/ACE\/AME guidelines with ultrasound features","volume":"7","author":"Deng","year":"2017","journal-title":"Sci Rep"},{"issue":"1","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"6906","DOI":"10.1038\/s41598-017-07353-y","article-title":"Cytologically indeterminate thyroid nodules: increased diagnostic performance with combination of US TI-RADS and a new scoring system","volume":"7","author":"He","year":"2017","journal-title":"Sci Rep"},{"issue":"5","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"946","DOI":"10.1016\/j.amjsurg.2015.12.030","article-title":"Thyroglobulin antibodies as a potential predictive marker of papillary thyroid carcinoma in patients with indeterminate cytology","volume":"212","author":"Karatzas","year":"2016","journal-title":"Am J Surg"},{"issue":"2","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"170","DOI":"10.4103\/ijem.IJEM_620_19","article-title":"Approaching indeterminate thyroid nodules in the absence of molecular markers: \u201cthe BETH-TR score\u201d","volume":"24","author":"Mehta","year":"2020","journal-title":"Indian J Endocrinol Metab"},{"issue":"1","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s00405-021-06783-9","article-title":"Validating the \u2018CUT score\u2019 risk stratification tool for indeterminate thyroid nodules using the Bethesda system for reporting thyroid cytopathology","volume":"279","author":"Pinhas","year":"2021","journal-title":"Eur Arch Otorhinolaryngol"},{"issue":"6","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"1006","DOI":"10.1016\/j.surg.2013.12.035","article-title":"A nomogram for predicting malignancy in thyroid nodules diagnosed as atypia of undetermined significance\/follicular lesions of undetermined significance on fine needle aspiration","volume":"155","author":"Yoon","year":"2014","journal-title":"Surgery"},{"issue":"8","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"2723","DOI":"10.1002\/cam4.3866","article-title":"The gray zone of thyroid nodules: using a nomogram to provide malignancy risk assessment and guide patient management","volume":"10","author":"Yousefi","year":"2021","journal-title":"Cancer Med"},{"issue":"8","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"5689","DOI":"10.1007\/s00330-021-07703-5","article-title":"Choosing the best algorithm among five thyroid nodule ultrasound scores: from performance to cytology sparing-a single-center retrospective study in a large cohort","volume":"31","author":"Sparano","year":"2021","journal-title":"Eur Radiol"},{"issue":"5","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"e262","DOI":"10.1016\/S1470-2045(19)30149-4","article-title":"Big data and machine learning algorithms for health-care delivery","volume":"20","author":"Ngiam","year":"2019","journal-title":"Lancet Oncol"},{"issue":"7","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1016\/j.eprac.2021.04.007","article-title":"American Association of clinical endocrinology and associazione medici endocrinologi thyroid nodule algorithmic tool","volume":"27","author":"Garber","year":"2021","journal-title":"Endocr Pract"},{"issue":"6","key":"2023032510163214500_","doi-asserted-by":"crossref","first-page":"2923","DOI":"10.1109\/TPAMI.2020.3046439","article-title":"Discrete box-constrained minimax classifier for uncertain and imbalanced class proportions","volume":"44","author":"Gilet","year":"2020","journal-title":"IEEE Trans Pattern Anal Mach Intell"}],"container-title":["European Journal of Endocrinology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/ejendo\/advance-article-pdf\/doi\/10.1093\/ejendo\/lvad017\/49245921\/lvad017.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/ejendo\/article-pdf\/188\/3\/249\/49630928\/lvad017.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/ejendo\/article-pdf\/188\/3\/249\/49630928\/lvad017.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,25]],"date-time":"2023-03-25T11:14:06Z","timestamp":1679742846000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/ejendo\/article\/188\/3\/249\/7044677"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,17]]},"references-count":36,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,3,2]]}},"URL":"https:\/\/doi.org\/10.1093\/ejendo\/lvad017","relation":{},"ISSN":["0804-4643","1479-683X"],"issn-type":[{"value":"0804-4643","type":"print"},{"value":"1479-683X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,3,2]]},"published":{"date-parts":[[2023,2,17]]}}}