{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T16:08:12Z","timestamp":1780589292543,"version":"3.54.1"},"reference-count":48,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,4,11]],"date-time":"2021-04-11T00:00:00Z","timestamp":1618099200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Mission-critical applications that rely on deep learning (DL) for automation suffer because DL models struggle to provide reliable indicators of failure. Reliable failure prediction can greatly improve the efficiency of a system, because it becomes easier to predict when human intervention is required. DL-based systems thus stand to benefit greatly from robust measures of uncertainty over model predictions. Monte Carlo dropout (MCD), a Bayesian method, and deep ensembles (DE) have emerged as two of the most popular and competitive ways to perform uncertainty estimation. Although literature exploring the usefulness of these approaches exists in medical imaging, robotics and autonomous driving domains, it is scarce to non-existent for remote sensing, and in particular, synthetic aperture radar (SAR) applications. To close this gap, we have created a deep learning model for road extraction (hereafter referred to as segmentation) in SAR and use it to compare standard model outputs against the aforementioned most popular methods for uncertainty estimation, MCD and DE. We demonstrate that these methods are not effective as an indicator of segmentation quality when measuring uncertainty (as indicated by model softmax outputs) across an entire image but are effective when uncertainty is measured from the set of road predictions only. Furthermore, we show a marked improvement in the correlation between prediction uncertainty and segmentation quality when we increase the set of road predictions by including predictions with lower softmax scores. We demonstrate the efficacy of our application of MCD and DE methods with an experimental design that measures performance in real-world quality assessment using in-distribution (ID) and out-of-distribution (OOD) data. These results inform the development of mission-critical deep learning systems in remote sensing. Tasks in medical image analysis that have a similar morphology to road structures, such as blood vessel segmentation, can also benefit from our findings.<\/jats:p>","DOI":"10.3390\/rs13081472","type":"journal-article","created":{"date-parts":[[2021,4,12]],"date-time":"2021-04-12T05:52:00Z","timestamp":1618206720000},"page":"1472","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Uncertainty Estimation for Deep Learning-Based Segmentation of Roads in Synthetic Aperture Radar Imagery"],"prefix":"10.3390","volume":"13","author":[{"given":"Jarrod","family":"Haas","sequence":"first","affiliation":[{"name":"Synthetic Aperture Radar Lab (SARlab), School of Engineering Science, Simon Fraser University, Burnaby, BC V5A 1S6, Canada"},{"name":"Digitalist Canada Ltd., Vancouver, BC V6B 4R3, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernhard","family":"Rabus","sequence":"additional","affiliation":[{"name":"Synthetic Aperture Radar Lab (SARlab), School of Engineering Science, Simon Fraser University, Burnaby, BC V5A 1S6, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,11]]},"reference":[{"key":"ref_1","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017). On Calibration of Modern Neural Networks. arXiv."},{"key":"ref_2","unstructured":"Gal, Y., and Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv."},{"key":"ref_3","unstructured":"Henne, M., Schwaiger, A., Roscher, K., and Weiss, G. (2020, January 9\u201312). Benchmarking Uncertainty Estimation Methods for Deep Learning with Safety-Related Metrics. Proceedings of the 34th AAAI Conference on Artificial Intelligence SafeAI, New York, NY, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3153","DOI":"10.1109\/LRA.2020.2974682","article-title":"A General Framework for Uncertainty Estimation in Deep Learning","volume":"5","author":"Loquercio","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"17816","DOI":"10.1038\/s41598-017-17876-z","article-title":"Leveraging Uncertainty Information from Deep Neural Networks for Disease Detection","volume":"7","author":"Leibig","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_6","unstructured":"Kendall, A., and Gal, Y. (2017). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. arXiv."},{"key":"ref_7","unstructured":"Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.V., Lakshminarayanan, B., and Snoek, J. (2019). Can You Trust Your Model\u2019s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv."},{"key":"ref_8","unstructured":"Nixon, J., Dusenberry, M., Jerfel, G., Nguyen, T., Liu, J., Zhang, L., and Tran, D. (2020). Measuring Calibration in Deep Learning. arXiv."},{"key":"ref_9","unstructured":"Nalisnick, E., Matsukawa, A., Teh, Y.W., Gorur, D., and Lakshminarayanan, B. (2019). Do Deep Generative Models Know What They Don\u2019t Know?. arXiv."},{"key":"ref_10","unstructured":"Wilson, A.G., and Izmailov, P. (2020). Bayesian Deep Learning and a Probabilistic Perspective of Generalization. arXiv."},{"key":"ref_11","unstructured":"Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., and Wilson, A.G. (2019). A Simple Baseline for Bayesian Uncertainty in Deep Learning. arXiv."},{"key":"ref_12","unstructured":"Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"106816","DOI":"10.1016\/j.csda.2019.106816","article-title":"Uncertainty Quantification Using Bayesian Neural Networks in Classification: Application to Biomedical Image Segmentation","volume":"142","author":"Kwon","year":"2020","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gruen, A., Kuebler, O., and Agouris, P. (1995). Model-Based Road Extraction from Images. Automatic Extraction of Man-Made Objects from Aerial and Space Images, Birkh\u00e4user.","DOI":"10.1007\/978-3-0348-9242-1"},{"key":"ref_16","unstructured":"Hedman, K., Wessel, B., Soergel, U., and Stilla, U. (2005, January 14\u201316). Automatic road extraction by fusion of multiple SAR views. Proceedings of the 3th International Symposium: Remote Sensing and Data Fusion on Urban Areas, Tempe, AZ, USA."},{"key":"ref_17","unstructured":"Hinz, S., Baumgartner, A., Steger, C., Mayer, H., Eckstein, W., Ebner, H., Radig, B., and Smati, S.M. (1999). Road Extraction in Rural and Urban Areas. Semantic Modeling for the Acquisition of Topographic Information from Images and Maps, Birkh\u00e4user."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Steger, C., Mayer, H., and Radig, B. (1997). The Role of Grouping for Road Extraction. Automatic Extraction of Man-Made Objects from Aerial and Space Images (II), Birkh\u00e4user.","DOI":"10.1007\/978-3-0348-8906-3_24"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Xu, R., He, C., Liu, X., Chen, D., and Qin, Q. (2017). Bayesian Fusion of Multi-Scale Detectors for Road Extraction from SAR Images. ISPRS Int. J. Geo. Inf., 6.","DOI":"10.3390\/ijgi6010026"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Elguebaly, T., and Bouguila, N. (2010, January 12\u201314). A Bayesian Approach for SAR Images Segmentation and Changes Detection. Proceedings of the 25th Biennial Symposium on Communications, Kingston, ON, Canada.","DOI":"10.1109\/BSC.2010.5473011"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"14484","DOI":"10.3390\/s131114484","article-title":"SAR Image Segmentation Using Voronoi Tessellation and Bayesian Inference Applied to Dark Spot Feature Extraction","volume":"13","author":"Zhao","year":"2013","journal-title":"Sensors"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1778","DOI":"10.1109\/TGRS.2004.831865","article-title":"Classification of Hyperspectral Remote Sensing Images with Support Vector Machines","volume":"42","author":"Melgani","year":"2004","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yager, N., and Sowmya, A. Support Vector Machines for Road Extraction from Remotely Sensed Images. Computer Analysis of Images and Patterns, Proceedings of the CAIP 2003, Groningen, The Netherlands, 25\u201327 August 2003, Springer.","DOI":"10.1007\/978-3-540-45179-2_36"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4288","DOI":"10.1109\/JSTARS.2014.2347171","article-title":"Improving SAR-Based Urban Change Detection by Combining MAP-MRF Classifier and Nonlocal Means Similarity Weights","volume":"7","author":"Yousif","year":"2014","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. arXiv.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet Classification with Deep Convolutional Neural Networks","volume":"60","author":"Krizhevsky","year":"2012","journal-title":"Commun. ACM"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.isprsjprs.2018.04.014","article-title":"Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery Using Deep Learning","volume":"145","author":"Kemker","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e148","DOI":"10.1002\/mp.13649","article-title":"Machine Learning Techniques for Biomedical Image Segmentation: An Overview of Technical Aspects and Introduction to State-of-Art Applications","volume":"47","author":"Seo","year":"2020","journal-title":"Med. Phys."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1080\/22797254.2019.1694447","article-title":"A New Road Extraction Method Using Sentinel-1 SAR Images Based on the Deep Fully Convolutional Neural Network","volume":"52","author":"Zhang","year":"2019","journal-title":"Eur. J. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1867","DOI":"10.1109\/LGRS.2018.2864342","article-title":"Road Segmentation in SAR Satellite Images with Deep Fully Convolutional Neural Networks","volume":"15","author":"Henry","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Xiong, D., He, C., Liu, X., and Liao, M. (2020). An End-To-End Bayesian Segmentation Network Based on a Generative Adversarial Network for Remote Sensing Images. Remote Sens., 12.","DOI":"10.3390\/rs12020216"},{"key":"ref_32","unstructured":"Wen, Y., Vicol, P., Ba, J., Tran, D., and Grosse, R.B. (2018). Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches. arXiv."},{"key":"ref_33","unstructured":"Hern\u00e1ndez-Lobato, J.M., and Adams, R. (2015, January 6\u201311). Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. Proceedings of the 32nd International Conference on Machine Learning, Lille, France."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Nair, T., Precup, D., Arnold, D., and Arbel, T. (2018). Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation. arXiv.","DOI":"10.1007\/978-3-030-00928-1_74"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"67","DOI":"10.3389\/fninf.2019.00067","article-title":"Knowing What You Know in Brain Segmentation Using Bayesian Deep Neural Networks","volume":"13","author":"McClure","year":"2019","journal-title":"Front. Neuroinformatics"},{"key":"ref_36","unstructured":"Filos, A., Farquhar, S., Gomez, A.N., Rudner, T.G.J., Kenton, Z., Smith, L., Alizadeh, M., Kroon, A.D., and Gal, Y. (2019). A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks. arXiv."},{"key":"ref_37","unstructured":"Hendrycks, D., and Dietterich, T.G. (2019). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2232","DOI":"10.1109\/TIP.2021.3051484","article-title":"Deep SAR Imaging and Motion Compensation","volume":"30","author":"Pu","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_40","unstructured":"Saxe, A.M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B.D., and Cox, D. (May, January 30). On the Information Bottleneck Theory of Deep Learning. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_41","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning (Information Science and Statistics), Springer."},{"key":"ref_42","unstructured":"Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012). Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell. Syst. Tech. J."},{"key":"ref_44","unstructured":"Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep Ensembles: A Loss Landscape Perspective. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1111\/j.1469-8137.1912.tb05611.x","article-title":"The Distribution of the Flora in the Alpine Zone","volume":"11","author":"Jaccard","year":"1912","journal-title":"New Phytol."},{"key":"ref_46","unstructured":"Rahman, M.A., and Wang, Y. (2016, January 12\u201314). Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation. Proceedings of the 12th International Symposium ISVC, Las Vegas, NV, USA."},{"key":"ref_47","unstructured":"Etten, A.V., Lindenbaum, D., and Bacastow, T.M. (2018). SpaceNet: A Remote Sensing Dataset and Challenge Series. arXiv."},{"key":"ref_48","unstructured":"D\u2019Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., and Hoffman, M.D. (2020). Underspecification Presents Challenges for Credibility in Modern Machine Learning. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/8\/1472\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:26:46Z","timestamp":1760362006000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/8\/1472"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,11]]},"references-count":48,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["rs13081472"],"URL":"https:\/\/doi.org\/10.3390\/rs13081472","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,11]]}}}