{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T17:17:35Z","timestamp":1774027055045,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Despite achieving excellent performance on benchmarks, deep neural networks often underperform in real-world scenarios due to their sensitivity to minor shifts in input data, referred to as distributional shifts. These shifts are common in practical scenarios but are rarely accounted for during evaluations, leading to inflated performance metrics. To address this gap, we propose a novel methodology for the verification, evaluation, and risk assessment of deep learning systems. Our approach explicitly models the incidence of distributional shifts at run time by estimating their probability from the outputs of out-of-distribution detectors. We combine these estimates with conditional probabilities of network correctness, structuring them in a binary tree. By traversing this tree, we can compute reliable and precise estimates of network accuracy. We assess our approach on five datasets, simulating deployment conditions characterized by different frequencies of distributional shift. Our approach consistently outperforms conventional evaluations, with accuracy estimation errors typically ranging between 0.01 and 0.10. We further showcase the potential of our approach on a medical segmentation benchmark, wherein we apply our methods to risk assessment by associating costs with tree nodes, informing cost\u2013benefit analyses and decision-making. Overall, our approach offers a robust framework for improving the reliability and trustworthiness of deep learning systems, particularly in safety-critical applications, by providing more accurate evaluations and actionable risk assessments.<\/jats:p>","DOI":"10.1145\/3795523","type":"journal-article","created":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T14:11:49Z","timestamp":1770127909000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Probabilistic Runtime Verification, Evaluation, and Risk Assessment of Visual Deep Learning Systems"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9685-9906","authenticated-orcid":false,"given":"Birk","family":"Torpmann-Hagen","sequence":"first","affiliation":[{"name":"IFI, UiT: The Arctic University of Norway, Troms\u00f8, Norway and Simula Metropolitan Center for Digital  Engineering AS, Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2073-7029","authenticated-orcid":false,"given":"P\u00e5l","family":"Halvorsen","sequence":"additional","affiliation":[{"name":"Simula Metropolitan Center for Digital Engineering AS, Oslo, Norway and Oslo Metropolitan University, Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3153-2064","authenticated-orcid":false,"given":"Michael A.","family":"Riegler","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory AS, Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7067-6477","authenticated-orcid":false,"given":"Dag","family":"Johansen","sequence":"additional","affiliation":[{"name":"IFI, UiT: The Arctic University of Norway, Troms\u00f8, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,3,20]]},"reference":[{"key":"e_1_3_2_2_2","volume-title":"EndoCV2020: 2nd International Workshop and Challenge on Computer Vision in Endoscopy","volume":"2595","author":"Ali Sharib","year":"2020","unstructured":"Sharib Ali, Christian Daul, Jens Rittscher, Danail Stoyanov, and Enrico Grisan (Eds.). 2020. EndoCV2020: 2nd International Workshop and Challenge on Computer Vision in Endoscopy, Vol. 2595, CEUR Workshop Proceedings, Iowa, USA. Retrieved from http:\/\/ceur-ws.org\/Vol-2595\/"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","unstructured":"Sharib Ali Noha Ghatwary Debesh Jha Ece Isik-Polat Gorkem Polat Chen Yang Wuyang Li Adrian Galdran Miguel-Angel Gonzalez Ballester Vajira Thambawita et al. 2022. Assessing Generalisability of Deep Learning-Based Polyp Detection and Segmentation Methods through a Computer Vision Challenge. DOI: 10.48550\/ARXIV.2202.12031","DOI":"10.48550\/ARXIV.2202.12031"},{"key":"e_1_3_2_4_2","unstructured":"Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv:2308.14132. Retrieved from https:\/\/arxiv.org\/abs\/2308.14132"},{"key":"e_1_3_2_5_2","unstructured":"Dario Amodei Chris Olah Jacob Steinhardt Paul Christiano John Schulman and Dan Man\u00e9. 2016. Concrete problems in AI safety. arXiv:1606.06565. Retrieved from https:\/\/arxiv.org\/abs\/1606.06565"},{"key":"e_1_3_2_6_2","unstructured":"Lukas Berglund Meg Tong Max Kaufmann Mikita Balesni Asa Cooper Stickland Tomasz Korbak and Owain Evans. 2023. The reversal curse: LLMs trained on \u201cA is B\u201d fail to learn \u201cB is A\u201d. arXiv:2309.12288. Retrieved from https:\/\/arxiv.org\/abs\/2309.12288"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Liang-Chieh Chen George Papandreou Florian Schroff and Hartwig Adam. 2017. Rethinking Atrous Convolution for Semantic Image Segmentation. DOI: 10.48550\/ARXIV.1706.05587","DOI":"10.48550\/ARXIV.1706.05587"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","unstructured":"Corinna Cortes Mohri Mehryar and Rostamizadeh Afshin. 2012. L2 Regularization for Learning Kernels. DOI: 10.48550\/ARXIV.1205.2653","DOI":"10.48550\/ARXIV.1205.2653"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","unstructured":"Alexander D\u2019Amour Katherine Heller Dan Moldovan Ben Adlam Babak Alipanahi Alex Beutel Christina Chen Jonathan Deaton Jacob Eisenstein Matthew D. Hoffman et al. 2020. Underspecification Presents Challenges for Credibility in Modern Machine Learning. DOI: 10.48550\/ARXIV.2011.03395","DOI":"10.48550\/ARXIV.2011.03395"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1989.03430200123038"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3828935"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00257-z"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","unstructured":"Tejas Gokhale Swaroop Mishra Man Luo Bhavdeep Singh Sachdeva and Chitta Baral. 2022. Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness. DOI: 10.48550\/ARXIV.2203.07653","DOI":"10.48550\/ARXIV.2203.07653"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.5555\/3086952"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN52387.2021.9533442"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1205\/095758299529947"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2015. Deep Residual Learning for Image Recognition. DOI: 10.48550\/ARXIV.1512.03385","DOI":"10.48550\/ARXIV.1512.03385"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","unstructured":"Dan Hendrycks and Kevin Gimpel. 2016. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. DOI: 10.48550\/ARXIV.1610.02136","DOI":"10.48550\/ARXIV.1610.02136"},{"issue":"12","key":"e_1_3_2_19_2","doi-asserted-by":"crossref","first-page":"1381","DOI":"10.1002\/hec.1434","article-title":"The impact of detection and treatment on lifetime medical costs for patients with precancerous polyps and colorectal cancer","volume":"18","author":"Howard David H.","year":"2009","unstructured":"David H. Howard, Florence K. Tangka, Laura C. Seeff, Lisa C. Richardson, and Donatus U. Ekwueme. 2009. The impact of detection and treatment on lifetime medical costs for patients with precancerous polyps and colorectal cancer. Health Economics 18, 12 (Dec. 2009), 1381\u20131393.","journal-title":"Health Economics"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","unstructured":"Rui Huang Andrew Geng and Yixuan Li. 2021. On the Importance of Gradients for Detecting Distributional Shifts in the Wild. DOI: 10.48550\/ARXIV.2110.00218","DOI":"10.48550\/ARXIV.2110.00218"},{"key":"e_1_3_2_21_2","unstructured":"Conor Igoe Youngseog Chung Ian Char and Jeff Schneider. 2022. How Useful are Gradients for OOD Detection Really? arXiv:2205.10439. Retrieved from https:\/\/arxiv.org\/abs\/2205.10439"},{"key":"e_1_3_2_22_2","first-page":"448","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, Lille, France, 448\u2013456. Retrieved from https:\/\/proceedings.mlr.press\/v37\/ioffe15.html"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","unstructured":"Debesh Jha Pia H. Smedsrud Michael A. Riegler P\u00e5l Halvorsen Thomas de Lange Dag Johansen and H\u00e5vard D. Johansen. 2019. Kvasir-SEG: A Segmented Polyp Dataset. DOI: 10.48550\/ARXIV.1911.07069","DOI":"10.48550\/ARXIV.1911.07069"},{"key":"e_1_3_2_24_2","unstructured":"Jacob Kauffmann Lukas Ruff Gr\u00e9goire Montavon and Klaus-Robert M\u00fcller. 2020. The clever Hans effect in anomaly detection. arXiv:2006.10609. Retrieved from https:\/\/arxiv.org\/abs\/2006.10609"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP40778.2020.9190679"},{"key":"e_1_3_2_26_2","volume-title":"Advances in Neural Information Processing Systems","volume":"31","author":"Lee Kimin","year":"2018","unstructured":"Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/file\/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf"},{"key":"e_1_3_2_27_2","unstructured":"Johnny Li Saksham Consul Eda Zhou James Wong Naila Farooqui Yuxin Ye Nithyashree Manohar Zhuxiaona Wei Tian Wu Ben Echols et al. 2024. Banishing LLM hallucinations requires rethinking generalization. arXiv:2406.17642. Retrieved from https:\/\/arxiv.org\/abs\/2406.17642"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510457.3513036"},{"key":"e_1_3_2_29_2","first-page":"21464","volume-title":"Advances in Neural Information Processing Systems","volume":"33","author":"Liu Weitang","year":"2020","unstructured":"Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 21464\u201321475. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/f5496252609c43eb8a3d147ab9b9c006-Paper.pdf"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106368"},{"key":"e_1_3_2_31_2","unstructured":"Jan Hendrik Metzen Tim Genewein Volker Fischer and Bastian Bischoff. 2017. On detecting adversarial perturbations. arXiv:1702.04267. Retrieved from https:\/\/arxiv.org\/abs\/1702.04267"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2011.06.019"},{"key":"e_1_3_2_33_2","unstructured":"Eric Nalisnick Akihiro Matsukawa Yee Whye Teh and Balaji Lakshminarayanan. 2019. Detecting out-of-distribution inputs to deep generative models using typicality. arXiv:1906.02994. Retrieved from https:\/\/arxiv.org\/abs\/1906.02994"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368555.3384468"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533378"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","unstructured":"Stephan Rabanser Stephan G\u00fcnnemann and Zachary C. Lipton. 2018. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. DOI: 10.48550\/ARXIV.1810.11953","DOI":"10.48550\/ARXIV.1810.11953"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","unstructured":"Benjamin Recht Rebecca Roelofs Ludwig Schmidt and Vaishaal Shankar. 2018. Do CIFAR-10 Classifiers Generalize to CIFAR-10? DOI: 10.48550\/ARXIV.1806.00451","DOI":"10.48550\/ARXIV.1806.00451"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","unstructured":"Benjamin Recht Rebecca Roelofs Ludwig Schmidt and Vaishaal Shankar. 2019. Do ImageNet Classifiers Generalize to ImageNet? DOI: 10.48550\/ARXIV.1902.10811","DOI":"10.48550\/ARXIV.1902.10811"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"5193","DOI":"10.1007\/s10664-020-09881-0","article-title":"Testing machine learning based systems: A systematic mapping","volume":"25","author":"Riccio Vincenzo","year":"2020","unstructured":"Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: A systematic mapping. Empirical Software Engineering 25 (2020), 5193\u20135254.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","unstructured":"Bernhard Sch\u00f6lkopf. 2019. Causality for Machine Learning. DOI: 10.48550\/ARXIV.1911.10500","DOI":"10.48550\/ARXIV.1911.10500"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2007.06.032"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11548-013-0926-3"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403379"},{"key":"e_1_3_2_45_2","unstructured":"Yiyou Sun Yifei Ming Xiaojin Zhu and Yixuan Li. 2022. Out-of-distribution detection with deep nearest neighbors. arXiv:2204.06507. Retrieved from https:\/\/arxiv.org\/abs\/2204.06507"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2025.3552157"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.5220\/0013195500003928"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamadermatol.2019.1735"},{"key":"e_1_3_2_49_2","unstructured":"Jingkang Yang Kaiyang Zhou Yixuan Li and Ziwei Liu. 2022. Generalized out-of-distribution detection: A survey. arXiv:2110.11334. Retrieved from https:\/\/arxiv.org\/abs\/2110.11334"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683"},{"key":"e_1_3_2_51_2","volume-title":"Advances in Neural Information Processing Systems","volume":"31","author":"Zheng Zhihao","year":"2018","unstructured":"Zhihao Zheng and Pengyu Hong. 2018. Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks. In Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/file\/e7a425c6ece20cbc9056f98699b53c6f-Paper.pdf"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3795523","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T16:28:30Z","timestamp":1774024110000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3795523"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,20]]},"references-count":50,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3795523"],"URL":"https:\/\/doi.org\/10.1145\/3795523","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,20]]},"assertion":[{"value":"2025-03-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}