{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T06:33:15Z","timestamp":1772260395776,"version":"3.50.1"},"reference-count":33,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2024,12,14]],"date-time":"2024-12-14T00:00:00Z","timestamp":1734134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00174-22-1-0028"],"award-info":[{"award-number":["N00174-22-1-0028"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Using deep learning model predictions requires not only understanding the model\u2019s confidence but also its uncertainty, so we know when to trust the prediction or require support from a human. In this study, we used Monte Carlo dropout (MCDO) to characterize the uncertainty of deep learning image classification algorithms, including feature fusion models, on simulated synthetic aperture radar (SAR) images of persistent ship wakes. Comparing to a baseline, we used the distribution of predictions from dropout with simple mean value ensembling and the Kolmogorov\u2014Smirnov (KS) test to classify in-domain and out-of-domain (OOD) test samples, created by rotating images to angles not present in the training data. Our objective was to improve the classification robustness and identify OOD images during the test time. The mean value ensembling did not improve the performance over the baseline, in that there was a \u20131.05% difference in the Matthews correlation coefficient (MCC) from the baseline model averaged across all SAR bands. The KS test, by contrast, saw an improvement of +12.5% difference in MCC and was able to identify the majority of OOD samples. Leveraging the full distribution of predictions improved the classification robustness and allowed labeling test images as OOD. The feature fusion models, however, did not improve the performance over the single SAR-band models, demonstrating that it is best to rely on the highest quality data source available (in our case, C-band).<\/jats:p>","DOI":"10.3390\/rs16244669","type":"journal-article","created":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T10:08:53Z","timestamp":1734343733000},"page":"4669","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Uncertainty Quantification in Data Fusion Classifier for Ship-Wake Detection"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4616-9384","authenticated-orcid":false,"given":"Maice","family":"Costa","sequence":"first","affiliation":[{"name":"National Security Institute, Virginia Tech, Arlington, VA 22203, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3134-6461","authenticated-orcid":false,"given":"Daniel","family":"Sobien","sequence":"additional","affiliation":[{"name":"National Security Institute, Virginia Tech, Arlington, VA 22203, USA"}]},{"given":"Ria","family":"Garg","sequence":"additional","affiliation":[{"name":"Department of Aerospace and Ocean Engineering, Virginia Tech, Blacksburg, VA 24061, USA"}]},{"given":"Winnie","family":"Cheung","sequence":"additional","affiliation":[{"name":"Department of Aerospace and Ocean Engineering, Virginia Tech, Blacksburg, VA 24061, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2862-112X","authenticated-orcid":false,"given":"Justin","family":"Krometis","sequence":"additional","affiliation":[{"name":"National Security Institute, Virginia Tech, Blacksburg, VA 24060, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3410-1160","authenticated-orcid":false,"given":"Justin A.","family":"Kauffman","sequence":"additional","affiliation":[{"name":"National Security Institute, Virginia Tech, Arlington, VA 22203, USA"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/j.inffus.2021.05.008","article-title":"A review of uncertainty quantification in deep learning: Techniques, applications and challenges","volume":"76","author":"Abdar","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_2","unstructured":"Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv."},{"key":"ref_3","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1038\/nature14541","article-title":"Probabilistic machine learning and artificial intelligence","volume":"521","author":"Ghahramani","year":"2015","journal-title":"Nature"},{"key":"ref_5","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 20\u201322). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA. ISSN: 1938-7228."},{"key":"ref_6","unstructured":"Diederik, P.K., and Tim Salimans, M.W. (2015). Variational Dropout and the Local Reparameterization Trick. arXiv."},{"key":"ref_7","unstructured":"Gal, Y., Hron, J., and Kendall, A. (2017). Concrete Dropout. arXiv."},{"key":"ref_8","unstructured":"Gal, Y., and Ghahramani, Z. (2016). A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. arXiv."},{"key":"ref_9","unstructured":"Angela, F., and Edouard Grave, A.J. (2019). Reducing Transformer Depth on Demand with Structured Dropout. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wu, Z., Wu, L., Meng, Q., Xia, Y., Xie, S., Qin, T., Dai, X., and Liu, T.Y. (2021). UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost. arXiv.","DOI":"10.18653\/v1\/2021.naacl-main.302"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1016\/j.ins.2021.07.024","article-title":"BARF: A new direct and cross-based binary residual feature fusion with uncertainty-aware module for medical image classification","volume":"577","author":"Abdar","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Du, L., Wang, W., Pu, J., and Zhao, Z. (2023, January 22\u201324). Quantifying Uncertainty in Potato Leaf Disease Detection: A Comparative Study of Deep Learning Models Using Monte Carlo Dropout. Proceedings of the International Conference on Internet of Things, Communication and Intelligent Technology, Xuzhou, China.","DOI":"10.1007\/978-981-97-2757-5_55"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"015002","DOI":"10.1088\/2632-2153\/aba6f3","article-title":"Deeply uncertain: Comparing methods of uncertainty quantification in deep learning algorithms","volume":"2","author":"Caldeira","year":"2020","journal-title":"Mach. Learn. Sci. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"106248","DOI":"10.1016\/j.engappai.2023.106248","article-title":"Uncertainty-aware credit card fraud detection using deep learning","volume":"123","author":"Habibpour","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Abdar, M., Samami, M., Mahmoodabad, S.D., Doan, T., Mazoure, B., Hashemifesharaki, R., Liu, L., Khosravi, A., Acharya, U.R., and Makarenkov, V. (2021). Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning. Comput. Biol. Med., 135.","DOI":"10.1016\/j.compbiomed.2021.104418"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Salari, S., Rasoulian, A., Battie, M., Fortin, M., Rivaz, H., and Xiao, Y. (2023, January 19\u201324). Uncertainty-aware transformer model for anatomical landmark detection in paraspinal muscle MRIs. Proceedings of the Medical Imaging 2023: Image Processing, San Diego, CA, USA.","DOI":"10.1117\/12.2653155"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lewis, B., Liu, J., and Wong, A. (2018, January 15\u201319). Generative adversarial networks for SAR image realism. Proceedings of the Algorithms for Synthetic Aperture Radar Imagery XXV, Orlando, FL, USA.","DOI":"10.1117\/12.2301387"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lewis, B., DeGuchy, O., Sebastian, J., and Kaminski, J. (2019, January 14\u201318). Realistic SAR data augmentation using machine learning techniques. Proceedings of the Algorithms for Synthetic Aperture Radar Imagery XXVI, Baltimore, MD, USA.","DOI":"10.1117\/12.2518452"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Higgins, E., Sobien, D., Freeman, L., and Pitt, J.S. (2021, January 19\u201321). Data Fusion for Combining Information for Disparate Data Sources for Maritime Remote Sensing. Proceedings of the AIAA Scitech 2021 Forum, Online.","DOI":"10.2514\/6.2021-0915"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Neal, R.M. (1996). Bayesian Learning for Neural Networks, Springer.","DOI":"10.1007\/978-1-4612-0745-0"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/S0893-6080(00)00098-8","article-title":"Bayesian approach for neural networks\u2014Review and case studies","volume":"14","author":"Lampinen","year":"2001","journal-title":"Neural Netw."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mengersen, K.L., Pudlo, P., and Robert, C.P. (2020). Bayesian Neural Networks: An Introduction and Survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, Springer International Publishing. Lecture Notes in Mathematics.","DOI":"10.1007\/978-3-030-42553-1"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"12345","DOI":"10.1029\/JC093iC10p12345","article-title":"Full-Spectrum modeling of Synthetic Aperture Radar Internal Wave Signatures","volume":"93","author":"Lyzenga","year":"1988","journal-title":"J. Geophys. Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"665","DOI":"10.3390\/make4030031","article-title":"Improving Deep Learning for Maritime Remote Sensing through Data Augmentation and Latent Space","volume":"4","author":"Sobien","year":"2022","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Sobien, D., Kauffman, J.A., Higgins, E., Freeman, L., and Pitt, J.S. (2023, January 23\u201327). Evaluation of Machine-Learning Data Fusion Classifier Performance for Ship-Wake Detection with Modified Data Sets. Proceedings of the AIAA SCITECH 2023 Forum, National Harbor, MD, USA.","DOI":"10.2514\/6.2023-0195"},{"key":"ref_27","unstructured":"Miner, E.W., Ramberg, S.E., and Swean, T.F. (1988). A Method for Approximating the Initial Data Plane for Surface Ship Wake Simulations, Maritime Technical Information Facility. Technical Report."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"5305","DOI":"10.1029\/JC092iC05p05305","article-title":"Examples of calculation methods for flow and mixing in stratified fluids","volume":"92","author":"Rodi","year":"1987","journal-title":"J. Geophys. Res."},{"key":"ref_29","unstructured":"Higgins, E.T. (2023). Machine Learning and Data Fusion of Simulated Remote Sensing Data. [Ph.D. Thesis, Virginia Tech]."},{"key":"ref_30","unstructured":"Sussman, M., and Dommermuth, D.G. (2014). The numerical simulation of ship waves using Cartesian grid methods. arXiv."},{"key":"ref_31","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","article-title":"The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation","volume":"21","author":"Chicco","year":"2020","journal-title":"BMC Genom."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13040-021-00244-z","article-title":"The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation","volume":"14","author":"Chicco","year":"2021","journal-title":"BioData Min."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/24\/4669\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:55:07Z","timestamp":1760115307000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/24\/4669"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,14]]},"references-count":33,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["rs16244669"],"URL":"https:\/\/doi.org\/10.3390\/rs16244669","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,14]]}}}