{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T12:20:05Z","timestamp":1791980405399,"version":"4.4.0"},"reference-count":40,"publisher":"IOP Publishing","issue":"20","license":[{"start":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:00:00Z","timestamp":1760659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:00:00Z","timestamp":1760659200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"},{"start":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:00:00Z","timestamp":1760659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/questions\/alternative-author-rights-policies\/"},{"start":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:00:00Z","timestamp":1760659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/questions\/alternative-author-rights-policies\/"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2025,10,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    <jats:italic>Objective.<\/jats:italic>\n                    Deep learning auto-segmentation has greatly advanced contouring in radiotherapy. However, quality assurance remains necessary due to performance fluctuation among individual patients. This manual process reintroduces variability and partially reduces time-saving benefits. As a solution, uncertainty quantification (UQ) is increasingly explored for its ability to estimate output confidence. While numerous methods to quantify uncertainty exist, their comparative reliability remains underexplored. This study compares the reliability of commonly used UQ approaches for auto-segmentation in radiotherapy.\n                    <jats:italic>Approach.<\/jats:italic>\n                    We evaluated the reliability of three popular uncertainty methods (Monte Carlo dropout, deep ensemble modelling and test-time augmentation) and uncertainty metrics (predictive entropy, mutual information and variance). We trained a 3D U-Net within the nnU-Net framework to segment 19 organs at risk (OAR) for head and neck cancer patients. We evaluated the reliability of the UQ methods and metrics on a set of 10 patients using segmentation model accuracy (surface Dice similarity coefficient), confidence calibration (expected calibration error (label)), and error localisation ability (uncertainty-error (U-E) overlap). Both multi-class and class-specific uncertainty maps were assessed.\n                    <jats:italic>Main results.<\/jats:italic>\n                    Segmentation accuracy remained stable without significant deviations across all UQ methods with respect to the baseline model without UQ. The reliability of different UQ methods in terms of confidence calibration and error localisation was also comparable. In contrast, the choice of UQ metric significantly influenced reliability. Multi-class predictive entropy (ECE-label: 0.06\u20130.06, U-E overlap: 0.45\u20130.46) consistently outperformed variance (ECE-label: 0.13\u20130.14, U-E overlap: 0.32\u20130.40,\n                    <jats:italic>p<\/jats:italic>\n                    &lt; 0.001) and mutual information (ECE-label: 0.13\u20130.14, U-E overlap: 0.35\u20130.40,\n                    <jats:italic>p<\/jats:italic>\n                    &lt; 0.001). Predictive entropy also demonstrated superior reliability as a class-specific UQ metric, though variability was observed between OARs.\n                    <jats:italic>Significance.<\/jats:italic>\n                    This study demonstrates that the choice of UQ approach substantially impacts the reliability of uncertainty maps. While different UQ methods performed comparably, the specific UQ metric chosen significantly affected reliability. These findings underscore the importance of careful metric selection and evaluation prior to application.\n                  <\/jats:p>","DOI":"10.1088\/1361-6560\/ae110c","type":"journal-article","created":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T22:52:40Z","timestamp":1759963960000},"page":"205023","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Reliability of uncertainty quantification methods for deep learning auto-segmentation in head and neck organs at risk"],"prefix":"10.1088","volume":"70","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3090-9218","authenticated-orcid":true,"given":"Jo\u00eblle E","family":"van Aalst","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Federica C","family":"Maruccio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4976-3557","authenticated-orcid":true,"given":"Rita","family":"Simo\u1ebds","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2807-8670","authenticated-orcid":true,"given":"Tomas M","family":"Janssen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5505-475X","authenticated-orcid":false,"given":"Jelmer M","family":"Wolterink","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8995-1210","authenticated-orcid":false,"given":"Peter M A","family":"van Ooijen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2349-7369","authenticated-orcid":false,"given":"Charlotte L","family":"Brouwer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2025,10,17]]},"reference":[{"key":"pmbae110cbib1","article-title":"Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks","author":"Ayhan","year":"2018","type":"conference-proceedings"},{"key":"pmbae110cbib2","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s42256-018-0004-1","type":"journal-article","article-title":"The need for uncertainty quantification in machine-assisted medical decision making","volume":"1","author":"Begoli","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"pmbae110cbib3","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1016\/j.radonc.2015.07.041","type":"journal-article","article-title":"CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines","volume":"117","author":"Brouwer","year":"2015","journal-title":"Radiother. Oncol."},{"key":"pmbae110cbib4","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.phro.2020.10.001","type":"journal-article","article-title":"Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy","volume":"16","author":"Brouwer","year":"2020","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"pmbae110cbib5","first-page":"32","type":"conference-proceedings","article-title":"Quantitative comparison of monte-carlo dropout uncertainty measures for multi-class segmentation","author":"Camarasa","year":"2020"},{"key":"pmbae110cbib6","doi-asserted-by":"publisher","first-page":"524","DOI":"10.3389\/fonc.2020.00524","type":"journal-article","article-title":"CNN-based quality assurance for automatic segmentation of breast cancer in radiotherapy","volume":"10","author":"Chen","year":"2020","journal-title":"Front. Oncol."},{"key":"pmbae110cbib7","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.strusafe.2008.06.020","type":"journal-article","article-title":"Aleatory or epistemic? Does it matter?","volume":"31","author":"Der Kiureghian","year":"2009","journal-title":"Struct. Saf."},{"key":"pmbae110cbib8","article-title":"Dropout as a Bayesian approximation: representing model uncertainty in deep learning","author":"Gal","year":"2016","type":"conference-proceedings"},{"key":"pmbae110cbib9","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2023.100515","type":"journal-article","article-title":"Clinical implementation and evaluation of auto-segmentation tools for multi-site contouring in radiotherapy","volume":"28","author":"Heilemann","year":"2023","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"pmbae110cbib10","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2024.103223","type":"journal-article","article-title":"A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods","volume":"97","author":"Huang","year":"2024","journal-title":"Med. Image Anal."},{"key":"pmbae110cbib11","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","type":"journal-article","article-title":"Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods","volume":"110","author":"H\u00fcllermeier","year":"2021","journal-title":"Mach. Learn."},{"key":"pmbae110cbib12","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","type":"journal-article","article-title":"nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation","volume":"18","author":"Isensee","year":"2021","journal-title":"Nat. Methods"},{"key":"pmbae110cbib13","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2024.1375096","type":"journal-article","article-title":"Clinical validation of commercial deep-learning based auto-segmentation models for organs at risk in the head and neck region: a single institution study","volume":"14","author":"Johnson","year":"2024","journal-title":"Front. Oncol."},{"key":"pmbae110cbib14","doi-asserted-by":"publisher","first-page":"282","DOI":"10.3389\/fnins.2020.00282","type":"journal-article","article-title":"Analyzing the quality and challenges of uncertainty estimations for brain tumor segmentation","volume":"14","author":"Jungo","year":"2020","journal-title":"Front. Neurosci."},{"key":"pmbae110cbib15","article-title":"Values: a framework for systematic validation of uncertainty estimation in semantic segmentation","author":"Kahl","year":"2024","type":"preprint"},{"key":"pmbae110cbib16","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1038\/s41746-020-00367-3","type":"journal-article","article-title":"Second opinion needed: communicating uncertainty in medical machine learning","volume":"4","author":"Kompa","year":"2021","journal-title":"npj Digit. Med."},{"key":"pmbae110cbib17","type":"conference-proceedings","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"vol 30","author":"Lakshminarayanan","year":"2017"},{"key":"pmbae110cbib18","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2024.102830","type":"journal-article","article-title":"Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis","volume":"150","author":"Lambert","year":"2024","journal-title":"Artif. Intell. Med."},{"key":"pmbae110cbib19","doi-asserted-by":"publisher","DOI":"10.1259\/bjro.20230030","type":"journal-article","article-title":"The role of artificial intelligence in radiotherapy clinical practice","volume":"5","author":"Landry","year":"2023","journal-title":"BJR Open"},{"key":"pmbae110cbib20","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1186\/s12938-023-01159-y","type":"journal-article","article-title":"Deep learning algorithm performance in contouring head and neck organs at risk: a systematic review and single-arm meta-analysis","volume":"22","author":"Liu","year":"2023","journal-title":"Biomed. Eng. Online"},{"key":"pmbae110cbib21","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1137803","type":"journal-article","article-title":"Validation of clinical acceptability of deep-learning-based automated segmentation of organs-at-risk for head-and-neck radiotherapy treatment planning","volume":"13","author":"Lucido","year":"2023","journal-title":"Front. Oncol."},{"key":"pmbae110cbib22","doi-asserted-by":"publisher","first-page":"1048","DOI":"10.59275\/j.melba.2024-5gc8","type":"journal-article","article-title":"Improving uncertainty-error correspondence in deep Bayesian medical image segmentation","volume":"2","author":"Mody","year":"2024","journal-title":"Mach. Learn. Biomed. Imaging"},{"key":"pmbae110cbib23","article-title":"Evaluating Bayesian deep learning methods for semantic segmentation","author":"Mukhoti","year":"2018","type":"preprint"},{"key":"pmbae110cbib24","article-title":"Obtaining well calibrated probabilities using Bayesian binning","author":"Naeini","year":"2015","type":"conference-proceedings"},{"key":"pmbae110cbib25","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2019.101557","type":"journal-article","article-title":"Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation","volume":"59","author":"Nair","year":"2020","journal-title":"Med. Image Anal."},{"key":"pmbae110cbib26","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102161","type":"journal-article","article-title":"The predictive value of segmentation metrics on dosimetry in organs at risk of the brain","volume":"73","author":"Poel","year":"2021","journal-title":"Med. Image Anal."},{"key":"pmbae110cbib27","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/add640","type":"journal-article","article-title":"Patient-specific uncertainty calibration of deep learning-based autosegmentation networks for adaptive MRI-guided lung radiotherapy","volume":"70","author":"Rabe","year":"2025","journal-title":"Phys. Med. Biol."},{"key":"pmbae110cbib28","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ad682d","type":"journal-article","article-title":"Enhancing the reliability of deep learning-based head and neck tumour segmentation using uncertainty estimation with multi-modal images","volume":"69","author":"Ren","year":"2024","journal-title":"Phys. Med. Biol."},{"key":"pmbae110cbib29","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2023.100500","type":"journal-article","article-title":"A network score-based metric to optimize the quality assurance of automatic radiotherapy target segmentations","volume":"28","author":"Rodriguez Outeiral","year":"2023","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"pmbae110cbib30","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1038\/s43856-024-00528-5","type":"journal-article","article-title":"Application of simultaneous uncertainty quantification and segmentation for oropharyngeal cancer use-case with Bayesian deep learning","volume":"4","author":"Sahlsten","year":"2024","journal-title":"Commun. Med."},{"key":"pmbae110cbib31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","type":"journal-article","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"pmbae110cbib32","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/j.radonc.2021.01.024","type":"journal-article","article-title":"Comprehensive toxicity risk profiling in radiation therapy for head and neck cancer: a new concept for individually optimised treatment","volume":"157","author":"van den Bosch","year":"2021","journal-title":"Radiother. Oncol."},{"key":"pmbae110cbib33","doi-asserted-by":"publisher","DOI":"10.1016\/j.adro.2021.100658","type":"journal-article","article-title":"Using spatial probability maps to highlight potential inaccuracies in deep learning-based contours: facilitating online adaptive radiation therapy","volume":"6","author":"Van Rooij","year":"2021","journal-title":"Adv. Radiat. Oncol."},{"key":"pmbae110cbib34","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1016\/j.radonc.2016.09.009","type":"journal-article","article-title":"Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies","volume":"121","author":"Vinod","year":"2016","journal-title":"Radiother. Oncol."},{"key":"pmbae110cbib35","doi-asserted-by":"publisher","first-page":"e929","DOI":"10.1002\/mp.14320","type":"journal-article","article-title":"Auto-segmentation of organs at risk for head and neck radiotherapy planning: from atlas-based to deep learning methods","volume":"47","author":"Vrtovec","year":"2020","journal-title":"Med. Phys."},{"key":"pmbae110cbib36","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2024.110542","type":"journal-article","article-title":"Artificial intelligence uncertainty quantification in radiotherapy applications\u2014a scoping review","volume":"201","author":"Wahid","year":"2024a","journal-title":"Radiother. Oncol."},{"key":"pmbae110cbib37","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2023.100526","type":"journal-article","article-title":"Harnessing uncertainty in radiotherapy auto-segmentation quality assurance","volume":"29","author":"Wahid","year":"2024b","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"pmbae110cbib38","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.neucom.2019.01.103","type":"journal-article","article-title":"Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks","volume":"335","author":"Wang","year":"2019","journal-title":"Neurocomputing"},{"key":"pmbae110cbib39","first-page":"2282","type":"conference-proceedings","article-title":"Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures?","author":"Wimmer","year":"2023"},{"key":"pmbae110cbib40","article-title":"Confidence calibration for convolutional neural networks using structured dropout","author":"Zhang","year":"2019","type":"preprint"}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T10:08:27Z","timestamp":1760695707000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ae110c"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,17]]},"references-count":40,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2025,10,17]]},"published-print":{"date-parts":[[2025,10,19]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/ae110c","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"value":"0031-9155","type":"print"},{"value":"1361-6560","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,17]]},"assertion":[{"value":"Reliability of uncertainty quantification methods for deep learning auto-segmentation in head and neck organs at risk","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published on behalf of Institute of Physics and Engineering in Medicine by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-06-11","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-10-08","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-10-17","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}