{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T01:20:53Z","timestamp":1789348853096,"version":"build-2803163510"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T00:00:00Z","timestamp":1711497600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T00:00:00Z","timestamp":1711497600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004622","name":"KWF Kankerbestrijding","doi-asserted-by":"publisher","award":["14113"],"award-info":[{"award-number":["14113"]}],"id":[{"id":"10.13039\/501100004622","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>To investigate the effect of uncertainty estimation on the performance of a Deep Learning (DL) algorithm for estimating malignancy risk of pulmonary nodules.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods and materials<\/jats:title>\n                <jats:p>In this retrospective study, we integrated an uncertainty estimation method into a previously developed DL algorithm for nodule malignancy risk estimation. Uncertainty thresholds were developed using CT data from the Danish Lung Cancer Screening Trial (DLCST), containing 883 nodules (65 malignant) collected between 2004 and 2010. We used thresholds on the 90th and 95th percentiles of the uncertainty score distribution to categorize nodules into certain and uncertain groups. External validation was performed on clinical CT data from a tertiary academic center containing 374 nodules (207 malignant) collected between 2004 and 2012. DL performance was measured using area under the ROC curve (AUC) for the full set of nodules, for the certain cases and for the uncertain cases. Additionally, nodule characteristics were compared to identify trends for inducing uncertainty.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The DL algorithm performed significantly worse in the uncertain group compared to the certain group of DLCST (AUC 0.62 (95% CI: 0.49, 0.76) vs 0.93 (95% CI: 0.88, 0.97); <jats:italic>p<\/jats:italic> &lt; .001) and the clinical dataset (AUC 0.62 (95% CI: 0.50, 0.73) vs 0.90 (95% CI: 0.86, 0.94); <jats:italic>p<\/jats:italic> &lt; .001). The uncertain group included larger benign nodules as well as more part-solid and non-solid nodules than the certain group.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>The integrated uncertainty estimation showed excellent performance for identifying uncertain cases in which the DL-based nodule malignancy risk estimation algorithm had significantly worse performance.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Clinical relevance statement<\/jats:title>\n                <jats:p>Deep Learning algorithms often lack the ability to gauge and communicate uncertainty. For safe clinical implementation, uncertainty estimation is of pivotal importance to identify cases where the deep learning algorithm harbors doubt in its prediction.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p>\u2022 <jats:italic>Deep learning (DL) algorithms often lack uncertainty estimation, which potentially reduce the risk of errors and improve safety during clinical adoption of the DL algorithm.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>Uncertainty estimation identifies pulmonary nodules in which the discriminative performance of the DL algorithm is significantly worse.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>Uncertainty estimation can further enhance the benefits of the DL algorithm and improve its safety and trustworthiness.<\/jats:italic><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-024-10714-7","type":"journal-article","created":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T07:03:46Z","timestamp":1711523026000},"page":"6639-6651","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Enhancing a deep learning model for pulmonary nodule malignancy risk estimation in chest CT with uncertainty estimation"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-1523-5549","authenticated-orcid":false,"given":"Dr\u00e9","family":"Peeters","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nat\u00e1lia","family":"Alves","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kiran V.","family":"Venkadesh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renate","family":"Dinnessen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zaigham","family":"Saghir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ernst T.","family":"Scholten","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cornelia","family":"Schaefer-Prokop","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rozemarijn","family":"Vliegenthart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mathias","family":"Prokop","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Colin","family":"Jacobs","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,3,27]]},"reference":[{"key":"10714_CR1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1102873","author":"DR Aberle","year":"2011","unstructured":"The National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med. https:\/\/doi.org\/10.1056\/NEJMoa1102873","journal-title":"N Engl J Med"},{"key":"10714_CR2","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1911793","author":"HJ de Koning","year":"2020","unstructured":"de Koning HJ, van der Aalst CM, de Jong PA et al (2020) Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med. https:\/\/doi.org\/10.1056\/NEJMoa1911793","journal-title":"N Engl J Med"},{"key":"10714_CR3","doi-asserted-by":"publisher","DOI":"10.3322\/caac.21708","author":"RL Siegel","year":"2022","unstructured":"Siegel RL, Miller KD, Fuchs HE, Jemal A (2022) Cancer statistics, 2022. CA Cancer J Clin. https:\/\/doi.org\/10.3322\/caac.21708","journal-title":"CA Cancer J Clin"},{"key":"10714_CR4","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.o1430","author":"M Limb","year":"2022","unstructured":"Limb M (2022) Shortages of radiology and oncology staff putting cancer patients at risk, college warns. BMJ. https:\/\/doi.org\/10.1136\/bmj.o1430","journal-title":"BMJ"},{"key":"10714_CR5","doi-asserted-by":"publisher","DOI":"10.1164\/rccm.201505-1040OC","author":"MM Wille","year":"2016","unstructured":"Wille MM, Dirksen A, Ashraf H et al (2016) Results of the randomized Danish lung cancer screening trial with focus on high-risk profiling. Am J Respir Crit Care Med. https:\/\/doi.org\/10.1164\/rccm.201505-1040OC","journal-title":"Am J Respir Crit Care Med"},{"key":"10714_CR6","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2021204433","author":"KV Venkadesh","year":"2021","unstructured":"Venkadesh KV, Setio AAA, Schreuder A et al (2021) Deep learning for malignancy risk estimation of pulmonary nodules detected at low-dose screening CT. Radiology. https:\/\/doi.org\/10.1148\/radiol.2021204433","journal-title":"Radiology"},{"key":"10714_CR7","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.222217","author":"S Faghani","year":"2023","unstructured":"Faghani S, Moassefi M, Rouzrokh P et al (2023) Quantifying uncertainty in deep learning of radiologic images. Radiology. https:\/\/doi.org\/10.1148\/radiol.222217","journal-title":"Radiology"},{"key":"10714_CR8","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-018-0004-1","author":"E Begoli","year":"2019","unstructured":"Begoli E, Bhattacharya T, Kusnezov D (2019) The need for uncertainty quantification in machine-assisted medical decision making. Nat Mach Intell. https:\/\/doi.org\/10.1038\/s42256-018-0004-1","journal-title":"Nat Mach Intell"},{"key":"10714_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00367-3","author":"B Kompa","year":"2021","unstructured":"Kompa B, Snoek J, Beam AL (2021) Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit Med. https:\/\/doi.org\/10.1038\/s41746-020-00367-3","journal-title":"NPJ Digit Med"},{"key":"10714_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101790","author":"L Herzog","year":"2020","unstructured":"Herzog L, Murina E, D\u00fcrr O, Wegener S, Sick B (2020) Integrating uncertainty in deep neural networks for MRI based stroke analysis. Med Image Anal. https:\/\/doi.org\/10.1016\/j.media.2020.101790","journal-title":"Med Image Anal"},{"key":"10714_CR11","doi-asserted-by":"publisher","unstructured":"Yang S, Fevens T. (2021) Uncertainty quantification and estimation in medical image classification. ICANN 2021. Springer, Cham.https:\/\/doi.org\/10.1007\/978-3-030-86365-4_54","DOI":"10.1007\/978-3-030-86365-4_54"},{"key":"10714_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102655","author":"J Linmans","year":"2023","unstructured":"Linmans J, Elfwing S, van der Laak J, Litjens G (2023) Predictive uncertainty estimation for out-of-distribution detection in digital pathology. Med Image Anal. https:\/\/doi.org\/10.1016\/j.media.2022.102655","journal-title":"Med Image Anal"},{"key":"10714_CR13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59710-8_80","author":"J Thagaard","year":"2020","unstructured":"Thagaard J, Hauberg S, van der Vegt B, Ebstrup T, Hansen JD, Dahl AB (2020) Can you trust predictive uncertainty under real dataset shifts in digital pathology? MICCAI 2020. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-030-59710-8_80","journal-title":"Springer, Cham."},{"key":"10714_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104418","author":"M Abdar","year":"2021","unstructured":"Abdar M, Samami M, Dehghani Mahmoodabad S et al (2021) Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning. Comput Biol Med. https:\/\/doi.org\/10.1016\/j.compbiomed.2021.104418","journal-title":"Comput Biol Med"},{"key":"10714_CR15","doi-asserted-by":"publisher","DOI":"10.2196\/36427","author":"A Kurz","year":"2022","unstructured":"Kurz A, Hauser K, Mehrtens HA et al (2022) Uncertainty estimation in medical image classification: systematic review. JMIR Med Inform. https:\/\/doi.org\/10.2196\/36427","journal-title":"JMIR Med Inform"},{"key":"10714_CR16","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230275","author":"N Alves","year":"2023","unstructured":"Alves N, Bosma JS, Venkadesh KV et al (2023) Prediction variability to identify reduced AI performance in cancer diagnosis at MRI and CT. Radiology. https:\/\/doi.org\/10.1148\/radiol.230275","journal-title":"Radiology"},{"key":"10714_CR17","doi-asserted-by":"publisher","DOI":"10.1136\/thoraxjnl-2017-211372","author":"K Chung","year":"2018","unstructured":"Chung K, Mets OM, Gerke PK et al (2018) Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population. Thorax. https:\/\/doi.org\/10.1136\/thoraxjnl-2017-211372","journal-title":"Thorax"},{"key":"10714_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2019.01.103","author":"G Wang","year":"2019","unstructured":"Wang G, Li W, Aertsen M, Deprest J, Ourselin S, Vercauteren T (2019) Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing. https:\/\/doi.org\/10.1016\/j.neucom.2019.01.103","journal-title":"Neurocomputing"},{"key":"10714_CR19","doi-asserted-by":"publisher","DOI":"10.3390\/e23050620","author":"Z Senousy","year":"2021","unstructured":"Senousy Z, Abdelsamea MM, Mohamed MM, Gaber MM (2021) 3E-Net: entropy-based elastic ensemble of deep convolutional neural networks for grading of invasive breast carcinoma histopathological microscopic images. Entropy. https:\/\/doi.org\/10.3390\/e23050620","journal-title":"Entropy"},{"key":"10714_CR20","unstructured":"American College of Radiology (2022) Lung-RADS\u00ae v2022. ACR. Available via https:\/\/www.acr.org\/-\/media\/ACR\/Files\/RADS\/Lung-RADS\/Lung-RADS-2022.pdf. Accessed 25 October 2023"},{"key":"10714_CR21","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-34025-x","author":"JM Dolezal","year":"2022","unstructured":"Dolezal JM, Srisuwananukorn A, Karpeyev D et al (2022) Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. Nat Commun. https:\/\/doi.org\/10.1038\/s41467-022-34025-x","journal-title":"Nat Commun"},{"key":"10714_CR22","doi-asserted-by":"publisher","unstructured":"Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv Neural Inf Process Syst. https:\/\/doi.org\/10.48550\/arXiv.1612.01474","DOI":"10.48550\/arXiv.1612.01474"},{"key":"10714_CR23","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-020-06946-y","author":"L Strohm","year":"2020","unstructured":"Strohm L, Hehakaya C, Ranschaert ER, Boon WPC, Moors EHM (2020) Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-020-06946-y","journal-title":"Eur Radiol"},{"key":"10714_CR24","doi-asserted-by":"publisher","DOI":"10.1097\/rli.0000000000000673","author":"JL Wichmann","year":"2020","unstructured":"Wichmann JL, Willemink MJ, De Cecco CN (2020) Artificial intelligence and machine learning in radiology: current state and considerations for routine clinical implementation. Invest Radiol. https:\/\/doi.org\/10.1097\/rli.0000000000000673","journal-title":"Invest Radiol"},{"key":"10714_CR25","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-023-09826-3","author":"W Hendrix","year":"2023","unstructured":"Hendrix W, Rutten M, Hendrix N et al (2023) Trends in the incidence of pulmonary nodules in chest computed tomography: 10-year results from two Dutch hospitals. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-023-09826-3","journal-title":"Eur Radiol"},{"key":"10714_CR26","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.10091808","author":"DR Aberle","year":"2011","unstructured":"Aberle DR, Berg CD, Black WC et al (2011) The National Lung Screening Trial: overview and study design. Radiology. https:\/\/doi.org\/10.1148\/radiol.10091808","journal-title":"Radiology"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-024-10714-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-024-10714-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-024-10714-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T17:12:48Z","timestamp":1726247568000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-024-10714-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,27]]},"references-count":26,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,10]]}},"alternative-id":["10714"],"URL":"https:\/\/doi.org\/10.1007\/s00330-024-10714-7","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,27]]},"assertion":[{"value":"9 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 February 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 March 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is Colin Jacobs.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors declare the following competing interests:MP receives grants from Canon Medical Systems, Siemens Healthineers; royalties from Mevis Medical Solutions; and payment for lectures from Canon Medical Systems and Siemens Healthineers. The host institution of MP is a minority shareholder in Thirona. He reports no other relationships that are related to the subject matter of the article.The host institution of CJ receives research grants and royalties from MeVis Medical Solutions, Bremen, Germany, and payment for lectures from Canon Medical Systems. CJ is a collaborator in a public-private research project where Radboudumc collaborates with Philips Medical Systems (Best, the Netherlands). CJ is a member of the Scientific Editorial Board for <i>European Radiology<\/i> (Imaging Informatics and Artificial Intelligence). He has not taken part in the selection or review processes for this article. He reports no other relationships that are related to the subject matter of the article.The host institution of HH receives grants from Siemens Healthineers. He reports no other relationships that are related to the subject matter of the article.RV is supported by an institutional research grant from Siemens Healthineers.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"No complex statistical methods were necessary for this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was obtained from all subjects (patients) in DLCST.Written informed consent was waived by the Institutional Review Board for the clinical dataset.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained for DLCST.Institutional Review Board approval was not required for the clinical dataset because of the retrospective design and data pseudonymization.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"DLCST nodule cohorts have been previously described in Venkadesh et al. <i>Radiology<\/i>, 2020 () and a subset of nodules from the clinical dataset are previously described in Chung et al. Thorax, 2018 ().","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Study subjects or cohorts overlap"}},{"order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}