{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:27:59Z","timestamp":1760171279719,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,12,8]],"date-time":"2023-12-08T00:00:00Z","timestamp":1701993600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Institute of Health","award":["1R01CA270454-01 and 1R01CA273058-01"],"award-info":[{"award-number":["1R01CA270454-01 and 1R01CA273058-01"]}]},{"DOI":"10.13039\/501100004063","name":"Knut and Alice Wallenberg Foundation","doi-asserted-by":"crossref","award":["2019.0024"],"award-info":[{"award-number":["2019.0024"]}],"id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2023,12,31]]},"abstract":"<jats:p>\n            Adversarial attacks on a convolutional neural network (CNN)\u2014injecting human-imperceptible perturbations into an input image\u2014could fool a high-performance CNN into making incorrect predictions. The success of adversarial attacks raises serious concerns about the robustness of CNNs, and prevents them from being used in safety-critical applications, such as medical diagnosis and autonomous driving. Our work introduces a visual analytics approach to understanding adversarial attacks by answering two questions: (1)\n            <jats:italic>Which neurons are more vulnerable to attacks?<\/jats:italic>\n            and (2)\n            <jats:italic>Which image features do these vulnerable neurons capture during the prediction?<\/jats:italic>\n            For the first question, we introduce multiple perturbation-based measures to break down the attacking magnitude into individual CNN neurons and rank the neurons by their vulnerability levels. For the second, we identify image features (e.g., cat ears) that highly stimulate a user-selected neuron to augment and validate the neuron\u2019s responsibility. Furthermore, we support an interactive exploration of a large number of neurons by aiding with hierarchical clustering based on the neurons\u2019 roles in the prediction. To this end, a visual analytics system is designed to incorporate visual reasoning for interpreting adversarial attacks. We validate the effectiveness of our system through multiple case studies as well as feedback from domain experts.\n          <\/jats:p>","DOI":"10.1145\/3587470","type":"journal-article","created":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T11:45:02Z","timestamp":1678880702000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Visual Analytics of Neuron Vulnerability to Adversarial Attacks on Convolutional Neural Networks"],"prefix":"10.1145","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5204-4935","authenticated-orcid":false,"given":"Yiran","family":"Li","sequence":"first","affiliation":[{"name":"University of California, Davis, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1130-9914","authenticated-orcid":false,"given":"Junpeng","family":"Wang","sequence":"additional","affiliation":[{"name":"Visa Research, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6382-2752","authenticated-orcid":false,"given":"Takanori","family":"Fujiwara","sequence":"additional","affiliation":[{"name":"Link\u00f6ping University, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8086-0366","authenticated-orcid":false,"given":"Kwan-Liu","family":"Ma","sequence":"additional","affiliation":[{"name":"University of California, Davis, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,12,8]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2807385"},{"issue":"1","key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"5393","DOI":"10.4249\/scholarpedia.5393","article-title":"Receptive field","volume":"4","author":"Alonso J.","year":"2009","unstructured":"J. Alonso and Y. Chen. 2009. Receptive field. Scholarpedia 4, 1 (2009), 5393.","journal-title":"Scholarpedia"},{"issue":"3","key":"e_1_3_2_4_2","article-title":"Deep learning and medical diagnosis: A review of literature","volume":"2","author":"Bakator Mihalj","year":"2018","unstructured":"Mihalj Bakator and Dragica Radosav. 2018. Deep learning and medical diagnosis: A review of literature. Multimodal Technol. Interact. 2, 3 (2018).","journal-title":"Multimodal Technol. Interact."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2018.07.023"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2020.2969185"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2018.042731661"},{"key":"e_1_3_2_8_2","first-page":"271","volume-title":"Proceedings of the VIS","author":"Das Nilaksh","year":"2020","unstructured":"Nilaksh Das, Haekyu Park, Zijie J. Wang, Fred Hohman, Robert Firstman, Emily Rogers, and Duen Horng Polo Chau. 2020. Bluff: Interactively deciphering adversarial attacks on deep neural networks. In Proceedings of the VIS. IEEE, 271\u2013275."},{"key":"e_1_3_2_9_2","volume-title":"Proceedings of the CVPR","author":"Eykholt Kevin","year":"2018","unstructured":"Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In Proceedings of the CVPR."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_2_11_2","volume-title":"Proceedings of the ICLR","author":"Goodfellow Ian","year":"2015","unstructured":"Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the ICLR."},{"key":"e_1_3_2_12_2","first-page":"1792","volume-title":"Proceedings of the ICML","volume":"80","author":"Greydanus Samuel","year":"2018","unstructured":"Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding Atari agents. In Proceedings of the ICML, Vol. 80. 1792\u20131801."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21918"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"issue":"1","key":"e_1_3_2_15_2","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1109\/TVCG.2019.2934659","article-title":"Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations","volume":"26","author":"Hohman Fred","year":"2019","unstructured":"Fred Hohman, Haekyu Park, Caleb Robinson, and Duen Horng Polo Chau. 2019. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Trans. Visual. Comput. Graph. 26, 1 (2019), 1096\u20131106.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"e_1_3_2_16_2","volume-title":"Proceedings of the NeurIPS","volume":"32","author":"Ilyas Andrew","year":"2019","unstructured":"Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. In Proceedings of the NeurIPS, Vol. 32."},{"key":"e_1_3_2_17_2","article-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky A.","year":"2009","unstructured":"A. Krizhevsky. 2009. Learning multiple layers of features from tiny images. Master\u2019s Thesis, University of Toronto.","journal-title":"Master\u2019s Thesis, University of Toronto"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865230"},{"issue":"1","key":"e_1_3_2_19_2","first-page":"163","article-title":"Visual diagnosis of tree boosting methods","volume":"24","author":"Liu Shixia","year":"2017","unstructured":"Shixia Liu, Jiannan Xiao, Junlin Liu, Xiting Wang, Jing Wu, and Jun Zhu. 2017. Visual diagnosis of tree boosting methods. IEEE Trans. Visual. Comput. Graph. 24, 1 (2017), 163\u2013173.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"e_1_3_2_20_2","volume-title":"Proceedings of the NIPS","volume":"27","author":"Long Jonathan L.","year":"2014","unstructured":"Jonathan L. Long, Ning Zhang, and Trevor Darrell. 2014. Do convnets learn correspondence? In Proceedings of the NIPS, Vol. 27."},{"issue":"1","key":"e_1_3_2_21_2","first-page":"241","article-title":"Visual analysis of class separations with locally linear segments","volume":"27","author":"Ma Yuxin","year":"2020","unstructured":"Yuxin Ma and Ross Maciejewski. 2020. Visual analysis of class separations with locally linear segments. IEEE Trans. Visual. Comput. Graph. 27, 1 (2020), 241\u2013253.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"issue":"1","key":"e_1_3_2_22_2","first-page":"1075","article-title":"Explaining vulnerabilities to adversarial machine learning through visual analytics","volume":"26","author":"Ma Yuxin","year":"2019","unstructured":"Yuxin Ma, Tiankai Xie, Jundong Li, and Ross Maciejewski. 2019. Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Trans. Visual. Comput. Graph. 26, 1 (2019), 1075\u20131085.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"e_1_3_2_23_2","volume-title":"Proceedings of the ICLR","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the ICLR."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0931-3"},{"issue":"5","key":"e_1_3_2_25_2","article-title":"DeepDream\u2014A code example for visualizing neural networks","volume":"2","author":"Mordvintsev Alexander","year":"2015","unstructured":"Alexander Mordvintsev, Christopher Olah, and Mike Tyka. 2015. DeepDream\u2014A code example for visualizing neural networks. Google Res. 2, 5 (2015). Retrieved from http:\/\/googleresearch.blogspot.co.uk\/2015\/07\/deepdream-code-example-for-visualizing.html.","journal-title":"Google Res."},{"key":"e_1_3_2_26_2","unstructured":"Daniel M\u00fcllner. 2011. Modern hierarchical agglomerative clustering algorithms. Retrieved from https:\/\/arXiv:1109.2378."},{"key":"e_1_3_2_27_2","first-page":"3387","volume-title":"Proceedings of the NIPS","volume":"29","author":"Nguyen Anh","year":"2016","unstructured":"Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. 2016. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Proceedings of the NIPS, Vol. 29. 3387\u20133395."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00007"},{"key":"e_1_3_2_29_2","first-page":"372","volume-title":"Proceedings of the EuroS&P","author":"Papernot Nicolas","year":"2016","unstructured":"Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. 2016. The limitations of deep learning in adversarial settings. In Proceedings of the EuroS&P. 372\u2013387."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.41"},{"key":"e_1_3_2_31_2","first-page":"91","volume-title":"Proceedings of the NIPS","volume":"28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the NIPS, Vol. 28. 91\u201399."},{"key":"e_1_3_2_32_2","first-page":"3296874","article-title":"A review of deep learning methods and applications for unmanned aerial vehicles","volume":"2017","author":"Tyrsa Vera","year":"2017","unstructured":"Vera Tyrsa, Adrian Carrio, Carlos Sampedro, Alejandro Rodriguez-Ramos, and Pascual Campoy. 2017. A review of deep learning methods and applications for unmanned aerial vehicles. J. Sensors 2017 (2017), 3296874.","journal-title":"J. Sensors"},{"issue":"86","key":"e_1_3_2_33_2","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten Laurens van der","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 86 (2008), 2579\u20132605.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864504"},{"key":"e_1_3_2_35_2","first-page":"186","volume-title":"Proceedings of the PacificVis","author":"Wang Junpeng","year":"2021","unstructured":"Junpeng Wang, Wei Zhang, Liang Wang, and Hao Yang. 2021. Investigating the evolution of tree boosting models with visual analytics. In Proceedings of the PacificVis. 186\u2013195."},{"key":"e_1_3_2_36_2","volume-title":"Proceedings of the NDSS","author":"Xu Weilin","year":"2018","unstructured":"Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature squeezing: Detecting adversarial examples in deep neural networks. In Proceedings of the NDSS."},{"key":"e_1_3_2_37_2","volume-title":"Proceedings of the BMVC","author":"Zagoruyko Sergey","year":"2016","unstructured":"Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. In Proceedings of the BMVC."},{"key":"e_1_3_2_38_2","first-page":"818","volume-title":"Proceedings of the ECCV","author":"Zeiler Matthew D.","year":"2014","unstructured":"Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the ECCV. Springer, 818\u2013833."},{"issue":"1","key":"e_1_3_2_39_2","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1109\/TVCG.2018.2864475","article-title":"iForest: Interpreting random forests via visual analytics","volume":"25","author":"Zhao Xun","year":"2018","unstructured":"Xun Zhao, Yanhong Wu, Dik Lun Lee, and Weiwei Cui. 2018. iForest: Interpreting random forests via visual analytics. IEEE Trans. Visual. Comput. Graph. 25, 1 (2018), 407\u2013416.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the ICLR","author":"Zhou Bolei","year":"2015","unstructured":"Bolei Zhou, Aditya Khosla, \u00c0gata Lapedriza, Aude Oliva, and Antonio Torralba. 2015. Object detectors emerge in deep scene CNNs. In Proceedings of the ICLR."},{"key":"e_1_3_2_41_2","article-title":"Places: A 10 million image database for scene recognition","author":"Zhou Bolei","year":"2018","unstructured":"Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40, 6 (2018), 1452\u20131464.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587470","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3587470","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:07:58Z","timestamp":1750183678000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587470"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,8]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12,31]]}},"alternative-id":["10.1145\/3587470"],"URL":"https:\/\/doi.org\/10.1145\/3587470","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2023,12,8]]},"assertion":[{"value":"2022-02-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-14","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}