{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T13:43:47Z","timestamp":1774446227791,"version":"3.50.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"23","license":[{"start":{"date-parts":[[2022,8,18]],"date-time":"2022-08-18T00:00:00Z","timestamp":1660780800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,18]],"date-time":"2022-08-18T00:00:00Z","timestamp":1660780800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004168","name":"Universit\u00e4t zu L\u00fcbeck","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004168","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Within the last decade Deep Learning has become a tool for solving challenging problems like image recognition. Still, Convolutional Neural Networks (CNNs) are considered black-boxes, which are difficult to understand by humans. Hence, there is an urge to visualize CNN architectures, their internal processes and what they actually learn. Previously, virtual realityhas been successfully applied to display small CNNs in immersive 3D environments. In this work, we address the problem how to feasibly render large-scale CNNs, thereby enabling the visualization of popular architectures with ten thousands of feature maps and branches in the computational graph in 3D. Our software \u201dDeepVisionVR\u201d enables the user to freely walk through the layered network, pick up and place images, move\/scale layers for better readability, perform feature visualization and export the results. We also provide a novel Pytorch module to dynamically link PyTorch with Unity, which gives developers and researchers a convenient interface to visualize their own architectures. The visualization is directly created from the PyTorch class that defines the Pytorch model used for training and testing. This approach allows full access to the network\u2019s internals and direct control over what exactly is visualized. In a use-case study, we apply the module to analyze models with different generalization abilities in order to understand how networks memorize images. We train two recent architectures, CovidResNet and CovidDenseNet on the Caltech101 and the SARS-CoV-2 datasets and find that bad generalization is driven by high-frequency features and the susceptibility to specific pixel arrangements, leading to implications for the practical application of CNNs. The code is available on Github <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/Criscraft\/DeepVisionVR\">https:\/\/github.com\/Criscraft\/DeepVisionVR<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s00521-022-07608-4","type":"journal-article","created":{"date-parts":[[2022,8,18]],"date-time":"2022-08-18T13:02:52Z","timestamp":1660827772000},"page":"21237-21252","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["A walk in the black-box: 3D visualization of large neural networks in virtual reality"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7039-5189","authenticated-orcid":false,"given":"Christoph","family":"Linse","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9942-8642","authenticated-orcid":false,"given":"Hammam","family":"Alshazly","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4539-4475","authenticated-orcid":false,"given":"Thomas","family":"Martinetz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,8,18]]},"reference":[{"key":"7608_CR1","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International conference on neural information processing systems, pp. 1097\u20131105"},{"issue":"3","key":"7608_CR2","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211\u2013252","journal-title":"Int J Comput Vis"},{"key":"7608_CR3","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garc\u00eda S, Gil-L\u00f3pez S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Inf Fusion 58:82\u2013115","journal-title":"Inf Fusion"},{"issue":"11","key":"7608_CR4","doi-asserted-by":"publisher","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","volume":"32","author":"E Tjoa","year":"2020","unstructured":"Tjoa E, Guan C (2020) A survey on explainable artificial intelligence (xai): toward medical xai. IEEE Trans Neural Netw Learn Syst 32(11):4793\u20134813","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"issue":"1","key":"7608_CR5","first-page":"1","volume":"1","author":"W Samek","year":"2017","unstructured":"Samek W, Wiegand T, M\u00fcller K-R (2017) Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. ITU J ICT Discov 1(1):1\u201310","journal-title":"ITU J ICT Discov"},{"key":"7608_CR6","doi-asserted-by":"publisher","unstructured":"Meissler N, Wohlan A, Hochgeschwender N, Schreiber A (2019) Using visualization of convolutional neural networks in virtual reality for machine learning newcomers. In: 2019 IEEE International Conference on artificial intelligence and virtual reality (AIVR), pp. 152\u20131526. IEEE, San Diego, CA. https:\/\/doi.org\/10.1109\/AIVR46125.2019.00031. https:\/\/ieeexplore.ieee.org\/document\/8942366\/ Accessed 05 apr 2022","DOI":"10.1109\/AIVR46125.2019.00031"},{"key":"7608_CR7","doi-asserted-by":"publisher","unstructured":"Bock M, Schreiber A (2018) Visualization of neural networks in virtual reality using Unreal Engine. In: Proceedings of the 24th ACM symposium on virtual reality software and technology, pp 1\u20132. ACM, Tokyo .https:\/\/doi.org\/10.1145\/3281505.3281605. https:\/\/dl.acm.org\/doi\/10.1145\/3281505.3281605 Accessed 05 apr 2022","DOI":"10.1145\/3281505.3281605"},{"key":"7608_CR8","doi-asserted-by":"publisher","unstructured":"Schreiber A, Bock M (2019) Visualization and exploration of deep learning networks in 3D and virtual reality. In: Stephanidis, C. (ed.) HCI International 2019 - Posters vol. 1033, pp. 206\u2013211. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-030-23528-4_29. Series Title: communications in computer and information science. http:\/\/link.springer.com\/10.1007\/978-3-030-23528-4_29 Accessed 2022-04-05","DOI":"10.1007\/978-3-030-23528-4_29"},{"key":"7608_CR9","unstructured":"VanHorn KC, Zinn M, Cobanoglu MC (2019) Deep learning development environment in virtual reality. arXiv:1906.05925 [cs, stat]. arXiv: 1906.05925. Accessed 5 Apr 2022"},{"key":"7608_CR10","doi-asserted-by":"publisher","unstructured":"Aamir A, Tamosiunaite M, W\u00f6rg\u00f6tter F (2021) Caffe2Unity: immersive visualization and interpretation of deep neural networks. Electronics 11(1):83. https:\/\/doi.org\/10.3390\/electronics11010083. Accessed 5 Apr 2022","DOI":"10.3390\/electronics11010083"},{"key":"7608_CR11","doi-asserted-by":"publisher","unstructured":"Wang H, Wu X, Huang Z, Xing EP (2020) High-frequency component helps explain the generalization of convolutional neural networks. In: 2020 IEEE\/CVF Conference on computer vision and pattern recognition (CVPR), pp. 8681\u20138691. IEEE, Seattle, WA, USA. https:\/\/doi.org\/10.1109\/CVPR42600.2020.00871. https:\/\/ieeexplore.ieee.org\/document\/9156428\/. Accessed 5 Mar 2022","DOI":"10.1109\/CVPR42600.2020.00871"},{"key":"7608_CR12","unstructured":"Gr\u00fcn F, Rupprecht C, Navab N, Tombari F (2016) A taxonomy and library for visualizing learned features in convolutional neural networks. arXiv preprint http:\/\/arxiv.org\/abs\/1606.07757"},{"issue":"4","key":"7608_CR13","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1109\/MCG.2018.042731661","volume":"38","author":"J Choo","year":"2018","unstructured":"Choo J, Liu S (2018) Visual analytics for explainable deep learning. IEEE Comput Gr Appl 38(4):84\u201392","journal-title":"IEEE Comput Gr Appl"},{"key":"7608_CR14","unstructured":"Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding neural networks through deep visualization. In: Proceedings of the 31st international conference on machine learning"},{"issue":"1","key":"7608_CR15","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1109\/TVCG.2016.2598831","volume":"23","author":"M Liu","year":"2016","unstructured":"Liu M, Shi J, Li Z, Li C, Zhu J, Liu S (2016) Towards better analysis of deep convolutional neural networks. IEEE Trans Vis Comput Gr 23(1):91\u2013100","journal-title":"IEEE Trans Vis Comput Gr"},{"key":"7608_CR16","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. In: Proceedings of the International Conference on learning representations"},{"key":"7608_CR17","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 618\u2013626","DOI":"10.1109\/ICCV.2017.74"},{"key":"7608_CR18","doi-asserted-by":"publisher","first-page":"122254","DOI":"10.1109\/ACCESS.2021.3109441","volume":"9","author":"H Alshazly","year":"2021","unstructured":"Alshazly H, Linse C, Barth E, Idris SA, Martinetz T (2021) Towards explainable ear recognition systems using deep residual networks. IEEE Access 9:122254\u2013122273","journal-title":"IEEE Access"},{"issue":"19","key":"7608_CR19","doi-asserted-by":"publisher","first-page":"4139","DOI":"10.3390\/s19194139","volume":"19","author":"H Alshazly","year":"2019","unstructured":"Alshazly H, Linse C, Barth E, Martinetz T (2019) Ensembles of deep learning models and transfer learning for ear recognition. Sensors 19(19):4139","journal-title":"Sensors"},{"issue":"2","key":"7608_CR20","doi-asserted-by":"publisher","first-page":"455","DOI":"10.3390\/s21020455","volume":"21","author":"H Alshazly","year":"2021","unstructured":"Alshazly H, Linse C, Barth E, Martinetz T (2021) Explainable COVID-19 detection using chest CT scans and deep learning. Sensors 21(2):455","journal-title":"Sensors"},{"key":"7608_CR21","doi-asserted-by":"crossref","unstructured":"Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Proceedings of the European conference on computer vision, pp. 818\u2013833, Springer","DOI":"10.1007\/978-3-319-10590-1_53"},{"issue":"3","key":"7608_CR22","first-page":"1","volume":"1341","author":"D Erhan","year":"2009","unstructured":"Erhan D, Bengio Y, Courville A, Vincent P (2009) Visualizing higher-layer features of a deep network. Univ Montreal 1341(3):1\u201313","journal-title":"Univ Montreal"},{"key":"7608_CR23","doi-asserted-by":"crossref","unstructured":"Mahendran A, Vedaldi A (2015) Understanding deep image representations by inverting them. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 5188\u20135196","DOI":"10.1109\/CVPR.2015.7299155"},{"issue":"3","key":"7608_CR24","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1007\/s11263-016-0911-8","volume":"120","author":"A Mahendran","year":"2016","unstructured":"Mahendran A, Vedaldi A (2016) Visualizing deep convolutional neural networks using natural pre-images. Int J Comput Vis 120(3):233\u2013255","journal-title":"Int J Comput Vis"},{"key":"7608_CR25","unstructured":"Mordvintsev A, Olah C, Tyka M Inceptionism: going deeper into neural networks (2015)"},{"key":"7608_CR26","doi-asserted-by":"crossref","unstructured":"Nguyen A, Yosinski J, Clune J Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 427\u2013436 (2015)","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"7608_CR27","unstructured":"Li H, Xu Z, Taylor G, Studer C, Goldstein T (2018) Visualizing the loss landscape of neural nets. In: Proceedings of the 32nd Conference on neural information processing system, pp 1\u201311 .https:\/\/proceedings.neurips.cc\/paper\/2018\/file\/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf"},{"key":"7608_CR28","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira, F, Burges CJ, Bottou, L, Weinberger KQ (eds.) Advances in neural information processing systems, vol 25. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2012\/file\/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf"},{"key":"7608_CR29","unstructured":"Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. In: Proceedings of the 31st Conference on neural information processing system, pp 1\u20134"},{"key":"7608_CR30","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) PyTorch: An imperative style, high-performance deep learning library. In: advances in neural information processing systems 32, pp 8024\u20138035"},{"key":"7608_CR31","unstructured":"Liu S, Papailiopoulos D, Achlioptas D (2020) Bad global minima exist and SGD can reach them. In: Proceedings of the 34th Conference on neural information processing system"},{"key":"7608_CR32","unstructured":"You Y, Li J, Reddi S, Hseu J, Kumar S, Bhojanapalli S, Song X, Demmel J, Hsieh C-J (2020) Large batch optimization for deep learning: training bert in 76 minutes. In: International Conference on learning representations"},{"key":"7608_CR33","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"7608_CR34","doi-asserted-by":"crossref","unstructured":"Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 178\u2013178","DOI":"10.1109\/CVPR.2004.383"},{"key":"7608_CR35","unstructured":"Soares E, Angelov P, Biaso S, Froes MH, Abe DK (2020) SARS-CoV-2 CT-scan dataset: a large dataset of real patients CT scans for SARS-CoV-2 identification. medRxiv"},{"key":"7608_CR36","doi-asserted-by":"publisher","first-page":"655","DOI":"10.7717\/peerj-cs.655","volume":"7","author":"H Alshazly","year":"2021","unstructured":"Alshazly H, Linse C, Abdalla M, Martinetz T (2021) COVID-Nets: deep CNN architectures for detecting COVID-19 using chest CT scans. Peer J Comput Sci 7:655","journal-title":"Peer J Comput Sci"},{"key":"7608_CR37","doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, Van Der\u00a0Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 4700\u20134708","DOI":"10.1109\/CVPR.2017.243"},{"issue":"1","key":"7608_CR38","doi-asserted-by":"publisher","first-page":"5494","DOI":"10.1038\/s41598-022-09293-8","volume":"12","author":"H Basak","year":"2022","unstructured":"Basak H, Kundu R, Singh PK, Ijaz MF, Wo\u017aniak M, Sarkar R (2022) A union of deep learning and swarm-based optimization for 3D human action recognition. Sci Rep 12(1):5494","journal-title":"Sci Rep"},{"key":"7608_CR39","doi-asserted-by":"crossref","unstructured":"Haker M, B\u00f6hme M, Martinetz T, Barth E (2009) Self-organizing maps for pose estimation with a time-of-flight camera. In: dynamic 3D imaging: workshop in conjunction with DAGM. lecture notes in computer science, vol. 5742, pp. 142\u2013153. http:\/\/www.springerlink.com\/content\/006305183070t383\/. https:\/\/webmail.inb.uni-luebeck.de\/inb-publications\/pdfs\/HaBoMaBa09a.pdf","DOI":"10.1007\/978-3-642-03778-8_11"},{"issue":"11","key":"7608_CR40","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1080\/14786440109462720","volume":"2","author":"KLIII Pearson","year":"1901","unstructured":"Pearson KLIII (1901) On lines and planes of closest fit to systems of points in space. Lond Edinb Dublin Philos Mag J Sci 2(11):559\u2013572","journal-title":"Lond Edinb Dublin Philos Mag J Sci"},{"key":"7608_CR41","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2022.3156658","author":"W Dong","year":"2022","unstructured":"Dong W, Wozniak M, Wu J, Li W (2022) Bai Z De-noising aggregation of graph neural networks by using principal component analysis. IEEE Trans Indus Inf. https:\/\/doi.org\/10.1109\/TII.2022.3156658","journal-title":"IEEE Trans Indus Inf"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07608-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-022-07608-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07608-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T23:48:37Z","timestamp":1667864917000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-022-07608-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,18]]},"references-count":41,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["7608"],"URL":"https:\/\/doi.org\/10.1007\/s00521-022-07608-4","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,18]]},"assertion":[{"value":"25 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}