{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T01:57:35Z","timestamp":1793239055923,"version":"4.5.0"},"reference-count":26,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T00:00:00Z","timestamp":1706054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/iop-standard\/v1"},{"start":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T00:00:00Z","timestamp":1706054400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2024,2,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    <jats:italic>Objective<\/jats:italic>\n                    . Prior to radiation therapy planning, accurate delineation of gross tumour volume (GTVs) and organs at risk (OARs) is crucial. In the current clinical practice, tumour delineation is performed manually by radiation oncologists, which is time-consuming and prone to large inter-observer variability. With the advent of deep learning (DL) models, automated contouring has become possible, speeding up procedures and assisting clinicians. However, these tools are currently used in the clinic mostly for contouring OARs, since these systems are not reliable yet for contouring GTVs. To improve the reliability of these systems, researchers have started exploring the topic of probabilistic neural networks. However, there is still limited knowledge of the practical implementation of such networks in real clinical settings.\n                    <jats:italic>Approach<\/jats:italic>\n                    . In this work, we developed a 3D probabilistic system that generates DL-based uncertainty maps for lung cancer CT segmentations. We employed the Monte Carlo (MC) dropout technique to generate probabilistic and uncertainty maps, while the model calibration was evaluated by using reliability diagrams. A clinical validation was conducted in collaboration with a radiation oncologist to qualitatively assess the value of the uncertainty estimates. We also proposed two novel metrics, namely mean uncertainty (MU) and relative uncertainty volume (RUV), as potential indicators for clinicians to assess the need for independent visual checks of the DL-based segmentation\n                    <jats:italic>. Main results<\/jats:italic>\n                    . Our study showed that uncertainty mapping effectively identified cases of under or over-contouring. Although the overconfidence of the model, a strong correlation was observed between the clinical opinion and MU metric. Moreover, both MU and RUV revealed high AUC values in discretising between low and high uncertainty cases.\n                    <jats:italic>Significance<\/jats:italic>\n                    . Our study is one of the first attempts to clinically validate uncertainty estimates in DL-based contouring. The two proposed metrics exhibited promising potential as indicators for clinicians to independently assess the quality of tumour delineation.\n                  <\/jats:p>","DOI":"10.1088\/1361-6560\/ad1a26","type":"journal-article","created":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T17:24:07Z","timestamp":1704302647000},"page":"035007","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Clinical assessment of deep learning-based uncertainty maps in lung cancer segmentation"],"prefix":"10.1088","volume":"69","author":[{"given":"Federica Carmen","family":"Maruccio","sequence":"first","affiliation":[{"name":"Philips Research, HTC 34, North Brabant, 5656 AE, NL,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wietse","family":"Eppinga","sequence":"additional","affiliation":[{"name":"University Medical Centre Utrecht"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Max-Heinrich","family":"Laves","sequence":"additional","affiliation":[{"name":"Philips Research, HTC 34, North Brabant, 5656 AE, NL,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roger Fonolla","family":"Navarro","sequence":"additional","affiliation":[{"name":"Philips Research, HTC 34, North Brabant, 5656 AE, NL,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Massimo","family":"Salvi","sequence":"additional","affiliation":[{"name":"Politecnico di Torino"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Filippo","family":"Molinari","sequence":"additional","affiliation":[{"name":"Politecnico di Torino"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pavlos","family":"Papaconstadopoulos","sequence":"additional","affiliation":[{"name":"Philips Research, HTC 34, North Brabant, 5656 AE, NL,"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,1,24]]},"reference":[{"key":"pmbad1a26bib1","doi-asserted-by":"publisher","first-page":"4006","DOI":"10.1038\/ncomms5006","type":"journal-article","article-title":"Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach","volume":"5","author":"Aerts","year":"2014","journal-title":"Nat. Commun."},{"key":"pmbad1a26bib2","doi-asserted-by":"publisher","first-page":"125011","DOI":"10.1088\/1361-6560\/acda0b","type":"journal-article","article-title":"Automated deep learning auto-segmentation of air volumes for MRI-guided online adaptive radiation therapy of abdominal tumors","volume":"68","author":"Ahunbay","year":"2023","journal-title":"Phys. Med. Biol."},{"key":"pmbad1a26bib3","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1016\/j.neucom.2022.07.070","type":"journal-article","article-title":"A review on the use of deep learning for medical images segmentation","volume":"506","author":"Aljabri","year":"2022","journal-title":"Neurocomputing"},{"key":"pmbad1a26bib4","doi-asserted-by":"crossref","DOI":"10.59275\/j.melba.2022-5937","type":"preprint","article-title":"Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection","author":"Bhat","year":"2022"},{"key":"pmbad1a26bib5","first-page":"424","type":"conference-proceedings","article-title":"3D U-Net: learning dense volumetric segmentation from sparse annotation","author":"\u00c7i\u00e7ek","year":"2016"},{"key":"pmbad1a26bib6","first-page":"1050","type":"conference-proceedings","article-title":"Dropout as a bayesian approximation: representing model uncertainty in deep learning","author":"Gal","year":"2016"},{"key":"pmbad1a26bib7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/15330338211016386","type":"journal-article","article-title":"The application and development of deep learning in radiotherapy: a systematic review","volume":"20","author":"Huang","year":"2021","journal-title":"Technol. Cancer Res. Treat."},{"key":"pmbad1a26bib8","doi-asserted-by":"publisher","first-page":"282","DOI":"10.3389\/fnins.2020.00282","type":"journal-article","article-title":"Analyzing the quality and challenges of uncertainty estimations for brain tumor segmentation","volume":"14","author":"Jungo","year":"2020","journal-title":"Front. Neurosci."},{"key":"pmbad1a26bib9","first-page":"682","type":"book","article-title":"On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation","author":"Jungo","year":"2018"},{"key":"pmbad1a26bib10","first-page":"1","type":"journal-article","article-title":"What uncertainties do we need in bayesian deep learning for computer vision?","volume":"30","author":"Kendall","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"pmbad1a26bib11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.promfg.2020.01.386","type":"journal-article","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"30","author":"Lakshminarayanan","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"pmbad1a26bib12","article-title":"Well-calibrated model uncertainty with temperature scaling for dropout variational inference","author":"Laves","year":"2019","type":"preprint"},{"key":"pmbad1a26bib13","author":"Li","year":"2018","type":"other"},{"key":"pmbad1a26bib14","article-title":"TIER-A: Denoising learning framework for information extraction,","author":"Li","year":"2022","type":"preprint"},{"key":"pmbad1a26bib15","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/j.ctro.2020.08.001","type":"journal-article","article-title":"A national perspective about the current work situation at modern radiotherapy departments","volume":"24","author":"Lindberg","year":"2020","journal-title":"Clin. Transl. Radiat. Oncol."},{"key":"pmbad1a26bib16","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1016\/j.promfg.2020.01.386","type":"book","article-title":"A 3D convolutional neural network for volumetric image semantic segmentation","volume":"vol 39","author":"Lu","year":"2019"},{"key":"pmbad1a26bib17","doi-asserted-by":"publisher","first-page":"210031","DOI":"10.1148\/ryai.2021210031","type":"journal-article","article-title":"A radiology-focused review of predictive uncertainty for ai interpretability in computer-assisted segmentation","volume":"3","author":"McCrindle","year":"2021","journal-title":"Radiol. Artif. Intell."},{"key":"pmbad1a26bib18","doi-asserted-by":"publisher","first-page":"12756","DOI":"10.5555\/3495724.3496794","type":"conference-proceedings","article-title":"Stochastic segmentation networks: modelling spatially correlated aleatoric uncertainty","volume":"vol 33","author":"Monteiro","year":"2020"},{"key":"pmbad1a26bib19","doi-asserted-by":"crossref","first-page":"1955","DOI":"10.1109\/TBME.2022.3232730","type":"journal-article","article-title":"Estimating uncertainty in neural networks for cardiac MRI segmentation: a benchmark study","volume":"70","author":"Ng","year":"2023","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"pmbad1a26bib20","doi-asserted-by":"publisher","first-page":"105017","DOI":"10.1088\/1361-6560\/abfbf5","type":"journal-article","article-title":"An anomaly detector as a clinical decision support system for parotid gland delineations","volume":"66","author":"Papaconstadopoulos","year":"2021","journal-title":"Phys. Med. Biol."},{"key":"pmbad1a26bib21","doi-asserted-by":"publisher","first-page":"3423","DOI":"10.1038\/s41467-022-30841-3","type":"journal-article","article-title":"Automated detection and segmentation of non-small cell lung cancer computed tomography images","volume":"13","author":"Primakov","year":"2022","journal-title":"Nat. Commun."},{"key":"pmbad1a26bib22","first-page":"234","type":"conference-proceedings","article-title":"U-net: Convolutional networks for biomedical image segmentation","author":"Ronneberger","year":"2015"},{"key":"pmbad1a26bib23","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1515\/raon-2016-0023","type":"journal-article","article-title":"Uncertainties in target volume delineation in radiotherapy - are they relevant and what can we do about them?","volume":"50","author":"Segedin","year":"2016","journal-title":"Radiol. Oncol."},{"key":"pmbad1a26bib24","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.107441","type":"journal-article","article-title":"Application of uncertainty quantification to artificial intelligence in healthcare: a review of last decade (2013\u20132023)","volume":"165","author":"Seoni","year":"2023","journal-title":"Comput. Biol. Med."},{"key":"pmbad1a26bib25","doi-asserted-by":"publisher","first-page":"56","DOI":"10.3389\/fncom.2019.00056","type":"journal-article","article-title":"Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation","volume":"13","author":"Wang","year":"2019","journal-title":"Frontiers in computational neuroscience"},{"key":"pmbad1a26bib26","article-title":"Mitigating Neural Network Overconfidence with Logit Normalization","author":"Wei","year":"2022","type":"conference-proceedings"}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T17:02:47Z","timestamp":1778778167000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad1a26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,24]]},"references-count":26,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,1,24]]},"published-print":{"date-parts":[[2024,2,7]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/ad1a26","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"value":"0031-9155","type":"print"},{"value":"1361-6560","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,24]]},"assertion":[{"value":"Clinical assessment of deep learning-based uncertainty maps in lung cancer segmentation","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 Institute of Physics and Engineering in Medicine. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2023-08-24","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-01-03","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-01-24","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}