{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,24]],"date-time":"2025-10-24T16:48:30Z","timestamp":1761324510884,"version":"3.37.3"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:00:00Z","timestamp":1679529600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:00:00Z","timestamp":1679529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Umea University"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Sign Process Syst"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In an open world with a long-tail distribution of input samples, Deep Neural Networks (DNNs) may make unpredictable mistakes for Out-of-Distribution (OOD) inputs at test time, despite high levels of accuracy obtained during model training. OOD detection can be an effective runtime assurance mechanism for safe deployment of machine learning algorithms in safety\u2013critical applications such as medical imaging and autonomous driving. A large number of OOD detection algorithms have been proposed in recent years, with a wide range of performance metrics in terms of accuracy and execution time. For real-time safety\u2013critical applications, e.g., autonomous driving, timing performance is of great importance in addition to accuracy. We perform a comprehensive and systematic benchmark study of multiple OOD detection algorithms in terms of both accuracy and execution time on different hardware platforms, including a powerful workstation and a resource-constrained embedded device, equipped with both CPU and GPU. We also profile and analyze the internal details of each algorithm to identify the performance bottlenecks and potential for GPU acceleration. This paper aims to provide a useful reference for the practical deployment of OOD detection algorithms for real-time safety\u2013critical applications.<\/jats:p>","DOI":"10.1007\/s11265-023-01852-0","type":"journal-article","created":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T11:04:46Z","timestamp":1679569486000},"page":"1355-1370","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Timing Performance Benchmarking of Out-of-Distribution Detection Algorithms"],"prefix":"10.1007","volume":"95","author":[{"given":"Siyu","family":"Luan","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4228-2774","authenticated-orcid":false,"given":"Zonghua","family":"Gu","sequence":"additional","affiliation":[]},{"given":"Amin","family":"Saremi","sequence":"additional","affiliation":[]},{"given":"Leonid","family":"Freidovich","sequence":"additional","affiliation":[]},{"given":"Lili","family":"Jiang","sequence":"additional","affiliation":[]},{"given":"Shaohua","family":"Wan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,3,23]]},"reference":[{"key":"1852_CR1","unstructured":"Yang, J., Zhou, K., Li, Y., & Liu, Z. (2021). Generalized out-of-distribution detection: A survey. arXiv preprint:\u00a0arXiv:2110.11334"},{"key":"1852_CR2","unstructured":"Yang J.,\u00a0et al. (2022)\u00a0OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. arXiv preprint:\u00a0arXiv:2210.07242"},{"key":"1852_CR3","unstructured":"Shen Z., et al. (2021). Towards out-of-distribution generalization: a survey. arXiv preprint:\u00a0arXiv:2108.13624"},{"key":"1852_CR4","unstructured":"Gu, Z., Wang, S., Kodase, S., & Shin, K. G. (2003). An End-to-End Tool Chain for Multi-View Modeling and Analysis of Avionics Mission Computing Software. In RTSS 2003. 24th IEEE Real-Time Systems Symposium: IEEE Computer Society, pp. 78\u201378."},{"key":"1852_CR5","doi-asserted-by":"crossref","unstructured":"Al-bayati, Z., Zhao, Q., Youssef, A., Zeng, H.,\u00a0& Gu, Z. (2015) Enhanced partitioned scheduling of mixed-criticality systems on multicore platforms. In the 20th Asia and South Pacific Design Automation Conference: IEEE pp. 630\u2013635.","DOI":"10.1109\/ASPDAC.2015.7059079"},{"key":"1852_CR6","doi-asserted-by":"publisher","first-page":"497","DOI":"10.1016\/j.neucom.2021.12.093","volume":"493","author":"X Xia","year":"2022","unstructured":"Xia, X., et al. (2022). GAN-based anomaly detection: A review. Neurocomputing, 493, 497\u2013535.","journal-title":"Neurocomputing"},{"issue":"1","key":"1852_CR7","doi-asserted-by":"publisher","first-page":"2131056","DOI":"10.1080\/08839514.2022.2131056","volume":"36","author":"F Cai","year":"2022","unstructured":"Cai, F., Ozdagli, A. I., & Koutsoukos, X. (2022). Variational Autoencoder for Classification and Regression for Out-of-Distribution Detection in Learning-Enabled Cyber-Physical Systems. Applied Artificial Intelligence, 36(1), 2131056.","journal-title":"Applied Artificial Intelligence"},{"key":"1852_CR8","unstructured":"Henzinger, T. A., Lukina, A., & Schilling, C. (2019). Outside the box: Abstraction-based monitoring of neural networks. arXiv preprint:\u00a0arXiv:1911.09032"},{"key":"1852_CR9","doi-asserted-by":"publisher","first-page":"132980","DOI":"10.1109\/ACCESS.2021.3108451","volume":"9","author":"S Luan","year":"2021","unstructured":"Luan, S., Gu, Z., Freidovich, L. B., Jiang, L., & Zhao, Q. (2021). Out-of-distribution detection for deep neural networks with isolation forest and local outlier factor. IEEE Access, 9, 132980\u2013132989.","journal-title":"IEEE Access"},{"key":"1852_CR10","unstructured":"Hendrycks, D.,\u00a0& Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint:\u00a0arXiv:1610.02136"},{"issue":"4","key":"1852_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3540198","volume":"21","author":"Q Zhao","year":"2022","unstructured":"Zhao, Q., Chen, M., Gu, Z., Luan, S., Zeng, H., & Chakrabory, S. (2022). CAN bus intrusion detection based on auxiliary classifier GAN and out-of-distribution detection. ACM Transactions on Embedded Computing Systems (TECS), 21(4), 1\u201330.","journal-title":"ACM Transactions on Embedded Computing Systems (TECS)"},{"key":"1852_CR12","unstructured":"Liang, S., Li, Y., & Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint:\u00a0arXiv:1706.02690"},{"key":"1852_CR13","unstructured":"Hendrycks, D., et al. (2022). Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning,\u00a0PMLR, pp. 8759\u20138773."},{"key":"1852_CR14","doi-asserted-by":"crossref","unstructured":"Wang, H., Li, Z., Feng, L., & Zhang, W. (2022). ViM: Out-Of-Distribution with Virtual-logit Matching. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 4921\u20134930.","DOI":"10.1109\/CVPR52688.2022.00487"},{"key":"1852_CR15","first-page":"677","volume":"34","author":"R Huang","year":"2021","unstructured":"Huang, R., Geng, A., & Li, Y. (2021). On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34, 677\u2013689.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"1852_CR16","first-page":"691","volume-title":"European Conference on Computer Vision","author":"Y Sun","year":"2022","unstructured":"Sun, Y., & Li, Y. (2022). Dice: Leveraging sparsification for out-of-distribution detection. European Conference on Computer Vision (pp. 691\u2013708). Springer."},{"key":"1852_CR17","first-page":"144","volume":"34","author":"Y Sun","year":"2021","unstructured":"Sun, Y., Guo, C., & Li, Y. (2021). React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34, 144\u2013157.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"1852_CR18","first-page":"21464","volume":"33","author":"W Liu","year":"2020","unstructured":"Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. Advances in neural information processing systems, 33, 21464\u201321475.","journal-title":"Advances in neural information processing systems"},{"key":"1852_CR19","doi-asserted-by":"crossref","unstructured":"Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563\u20131572.","DOI":"10.1109\/CVPR.2016.173"},{"key":"1852_CR20","unstructured":"Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. arXiv preprint:\u00a0arXiv:1807.03888"},{"key":"1852_CR21","unstructured":"Sun, Y., Ming, Y., Zhu, X., & Li, Y. (2022). Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, PMLR, pp. 20827\u201320840."},{"key":"1852_CR22","unstructured":"Sastry,\u00a0C. S., & Oore, S (2020) Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, PMLR, pp. 8491\u20138501."},{"issue":"6","key":"1852_CR23","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","volume":"29","author":"L Deng","year":"2012","unstructured":"Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6), 141\u2013142.","journal-title":"IEEE signal processing magazine"},{"key":"1852_CR24","unstructured":"Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical Report, University of Toronto."},{"key":"1852_CR25","unstructured":"Bulatov, Y. NotMnist dataset. Retrieved March 1, 2023, from\u00a0http:\/\/yaroslavvb.com\/upload\/notMNIST"},{"key":"1852_CR26","unstructured":"Xiao, H., Rasul, K., & Vollgraf, R. (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint:\u00a0arXiv:1708.07747"},{"key":"1852_CR27","unstructured":"Kylberg, G. (2011). Kylberg texture dataset v. 1.0. Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University."},{"issue":"6","key":"1852_CR28","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1145\/3065386","volume":"60","author":"A Krizhevsky","year":"2017","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84\u201390.","journal-title":"Communications of the ACM"},{"issue":"6","key":"1852_CR29","doi-asserted-by":"publisher","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","volume":"40","author":"B Zhou","year":"2017","unstructured":"Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2017). Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6), 1452\u20131464.","journal-title":"IEEE transactions on pattern analysis and machine intelligence"},{"issue":"11","key":"1852_CR30","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278\u20132324.","journal-title":"Proceedings of the IEEE"},{"key":"1852_CR31","unstructured":"Krizhevsky, A., Nair, V., & Hinton, G. Cifar-10 and cifar-100 datasets. Retrieved March 1,\u00a02023, from\u00a0https:\/\/www.cs.toronto.edu\/kriz\/cifar.html"},{"key":"1852_CR32","unstructured":"Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning."},{"key":"1852_CR33","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"issue":"7825","key":"1852_CR34","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","volume":"585","author":"CR Harris","year":"2020","unstructured":"Harris, C. R., et al. (2020). Array programming with NumPy. Nature, 585(7825), 357\u2013362.","journal-title":"Nature"},{"key":"1852_CR35","unstructured":"Pedregosa F.,\u00a0et al. (2011)\u00a0Scikit-learn: Machine learning in Python. The Journal of Machine Learning Research,\u00a012, 2825\u20132830."},{"key":"1852_CR36","unstructured":"Tripathy, A. Timing your PyTorch Code Fragments. Retrieved March 1, 2023, from\u00a0https:\/\/auro-227.medium.com\/timing-your-pytorch-code-fragments-e1a556e81f2"},{"issue":"3","key":"1852_CR37","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1109\/TBDATA.2019.2921572","volume":"7","author":"J Johnson","year":"2019","unstructured":"Johnson, J., Douze, M., & J\u00e9gou, H. (2019). Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3), 535\u2013547.","journal-title":"IEEE Transactions on Big Data"},{"key":"1852_CR38","unstructured":"Tajwar, F., Kumar, A., Xie, S. M. & Liang, P. (2021). No true state-of-the-art? OOD detection methods are inconsistent across datasets. arXiv preprint:\u00a0arXiv:2109.05554"},{"key":"1852_CR39","doi-asserted-by":"crossref","unstructured":"Choudhary, T., Mishra, V., Goswami, A., & Sarangapani, J. (2020). A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, pp. 1\u201343.","DOI":"10.1007\/s10462-020-09816-7"},{"key":"1852_CR40","doi-asserted-by":"crossref","unstructured":"Luan, S., Gu, Z., Xu, R., Zhao, Q., & Chen, G. (2023) LRP\u2010based network pruning and policy distillation of robust and non\u2010robust DRL agents for embedded systems. Concurrency and Computation: Practice and Experience.","DOI":"10.1002\/cpe.7351"},{"key":"1852_CR41","unstructured":"Meng, W., Gu, Z., Zhang, M., & Wu, Z. (2017). Two-bit networks for deep learning on resource-constrained embedded devices. arXiv preprint:\u00a0arXiv:1701.00485"},{"issue":"3","key":"1852_CR42","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1109\/TPDS.2020.3030548","volume":"32","author":"M Li","year":"2020","unstructured":"Li, M., et al. (2020). The deep learning compiler: A comprehensive survey. IEEE Transactions on Parallel and Distributed Systems, 32(3), 708\u2013727.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"}],"container-title":["Journal of Signal Processing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-023-01852-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11265-023-01852-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-023-01852-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T10:09:41Z","timestamp":1705486181000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11265-023-01852-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,23]]},"references-count":42,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["1852"],"URL":"https:\/\/doi.org\/10.1007\/s11265-023-01852-0","relation":{},"ISSN":["1939-8018","1939-8115"],"issn-type":[{"type":"print","value":"1939-8018"},{"type":"electronic","value":"1939-8115"}],"subject":[],"published":{"date-parts":[[2023,3,23]]},"assertion":[{"value":"13 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 January 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 January 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 March 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This research involves no human participants and\/or animals.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}