{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T11:39:04Z","timestamp":1761824344721,"version":"3.41.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2022,9,30]],"date-time":"2022-09-30T00:00:00Z","timestamp":1664496000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>We present a dynamic network rewiring (DNR) method to generate pruned deep neural network (DNN) models that both are robust against adversarially generated images and maintain high accuracy on clean images. In particular, the disclosed DNR training method is based on a unified constrained optimization formulation using a novel hybrid loss function that merges sparse learning with robust adversarial training. This training strategy dynamically adjusts inter-layer connectivity based on per-layer normalized momentum computed from the hybrid loss function. To further improve the robustness of the pruned models, we propose DNR++, an extension of the DNR method where we introduce the idea of sparse parametric Gaussian noise tensor that is added to the weight tensors to yield robust regularization. In contrast to existing robust pruning frameworks that require multiple training iterations, the proposed DNR and DNR++ achieve an overall target pruning ratio with only a single training iteration and can be tuned to support both irregular and structured channel pruning. To demonstrate the efficacy of the proposed method under the no-increased-training-time \u201cfree\u201d adversarial training scenario, we finally present FDNR++, a simple yet effective training modification that can yield robust yet compressed models requiring training time comparable to that of an unpruned non-adversarial training. To evaluate the merits of our disclosed training methods, experiments were performed with two widely accepted models, namely VGG16 and ResNet18, on CIFAR-10 and CIFAR-100 as well as with VGG16 on Tiny-ImageNet. Compared to the baseline uncompressed models, our methods provide over 20\u00d7 compression on all the datasets without any significant drop of either clean or adversarial classification performance. Moreover, extensive experiments show that our methods consistently find compressed models with better clean and adversarial image classification performance than what is achievable through state-of-the-art alternatives. We provide insightful observations to help make various model, parameter density, and prune-type selection choices and have open-sourced our saved models and test codes to ensure reproducibility of our results.<\/jats:p>","DOI":"10.1145\/3510833","type":"journal-article","created":{"date-parts":[[2022,1,26]],"date-time":"2022-01-26T18:13:20Z","timestamp":1643220800000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Toward Adversary-aware Non-iterative Model Pruning through\n            <u>D<\/u>\n            ynamic\n            <u>N<\/u>\n            etwork\n            <u>R<\/u>\n            ewiring of DNNs"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3533-9405","authenticated-orcid":false,"given":"Souvik","family":"Kundu","sequence":"first","affiliation":[{"name":"University of Southern California, Los Angeles, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yao","family":"Fu","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bill","family":"Ye","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter A.","family":"Beerel","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Massoud","family":"Pedram","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,12,13]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2020.3012865"},{"key":"e_1_3_3_3_2","article-title":"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples","author":"Athalye Anish","year":"2018","unstructured":"Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 (2018).","journal-title":"arXiv preprint arXiv:1802.00420"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1561\/2200000016"},{"key":"e_1_3_3_5_2","first-page":"39","volume-title":"IEEE Symposium on Security and Privacy","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy. IEEE, 39\u201357."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_3_3_7_2","article-title":"Sparse networks from scratch: Faster training without losing performance","author":"Dettmers Tim","year":"2019","unstructured":"Tim Dettmers and Luke Zettlemoyer. 2019. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 (2019).","journal-title":"arXiv preprint arXiv:1907.04840"},{"key":"e_1_3_3_8_2","first-page":"6379","volume-title":"Advances in Neural Information Processing Systems","author":"Ding Xiaohan","year":"2019","unstructured":"Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et\u00a0al. 2019. Global sparse momentum SGD for pruning very deep neural networks. In Advances in Neural Information Processing Systems. 6379\u20136391."},{"key":"e_1_3_3_9_2","article-title":"Convergence of a relaxed variable splitting method for learning sparse neural networks via  \\(l_1,l_0\\) , and transformed- \\(l_1\\)  penalties","author":"Dinh Thu","year":"2018","unstructured":"Thu Dinh and Jack Xin. 2018. Convergence of a relaxed variable splitting method for learning sparse neural networks via \\(l_1,l_0\\) , and transformed- \\(l_1\\) penalties. arXiv preprint arXiv:1812.05719.","journal-title":"arXiv preprint arXiv:1812.05719"},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1109\/ISVLSI.2019.00090","volume-title":"2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI\u201919)","author":"Fayyazi Arash","year":"2019","unstructured":"Arash Fayyazi, Souvik Kundu, Shahin Nazarian, Peter A. Beerel, and Massoud Pedram. 2019. CSrram: Area-efficient low-power ex-situ training framework for memristive neuromorphic circuits based on clustered sparsity. In 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI\u201919). IEEE, 465\u2013470."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.81"},{"key":"e_1_3_3_12_2","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.","journal-title":"arXiv preprint arXiv:1412.6572"},{"key":"e_1_3_3_13_2","article-title":"Uncovering the limits of adversarial training against norm-bounded adversarial examples","author":"Gowal Sven","year":"2020","unstructured":"Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. 2020. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593.","journal-title":"arXiv preprint arXiv:2010.03593"},{"key":"e_1_3_3_14_2","article-title":"An alternative surrogate loss for PGD-based adversarial testing","author":"Gowal Sven","year":"2019","unstructured":"Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli. 2019. An alternative surrogate loss for PGD-based adversarial testing. arXiv preprint arXiv:1910.09338.","journal-title":"arXiv preprint arXiv:1910.09338"},{"key":"e_1_3_3_15_2","first-page":"1283","volume-title":"Advances in Neural Information Processing Systems","author":"Gui Shupeng","year":"2019","unstructured":"Shupeng Gui, Haotao N. Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. 2019. Model compression with adversarial robustness: A unified optimization framework. In Advances in Neural Information Processing Systems. 1283\u20131294."},{"key":"e_1_3_3_16_2","article-title":"Tiny ImageNet challenge submission","author":"Hansen Lucas","year":"2015","unstructured":"Lucas Hansen. 2015. Tiny ImageNet challenge submission. CS 231N.","journal-title":"CS 231N"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_48"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.155"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00068"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01501"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_3_3_23_2","article-title":"Self-adaptive training: Beyond empirical risk minimization","volume":"33","author":"Huang Lang","year":"2020","unstructured":"Lang Huang, Chao Zhang, and Hongyang Zhang. 2020. Self-adaptive training: Beyond empirical risk minimization. Advances in Neural Information Processing Systems 33 (2020).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_24_2","article-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky, Geoffrey Hinton, et\u00a0al. 2009. Learning multiple layers of features from tiny images. Technical Report, Citeseer (2009).","journal-title":"Technical Report, Citeseer"},{"key":"e_1_3_3_25_2","first-page":"1097","volume-title":"Advances in Neural Information Processing Systems","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097\u20131105."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394885.3431542"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2020.2972520"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00516"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ALLERTON.2019.8919683"},{"key":"e_1_3_3_30_2","article-title":"Analyzing the confidentiality of undistillable teachers in knowledge distillation","volume":"34","author":"Kundu Souvik","year":"2021","unstructured":"Souvik Kundu, Qirui Sun, Yao Fu, Massoud Pedram, and Peter A. Beerel. 2021. Analyzing the confidentiality of undistillable teachers in knowledge distillation. In Advances in Neural Information Processing Systems (NeurIPS\u201921), 34.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS\u201921),"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9415117"},{"key":"e_1_3_3_32_2","article-title":"BMPQ: Bit-gradient sensitivity-driven mixed-precision quantization of DNNs from scratch","author":"Kundu Souvik","year":"2022","unstructured":"Souvik Kundu, Shikai Wang, Qirui Sun, Peter A. Beerel, and Massoud Pedram. 2022. BMPQ: Bit-gradient sensitivity-driven mixed-precision quantization of DNNs from scratch. In DATE.","journal-title":"DATE"},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1109\/SP.2019.00044","volume-title":"2019 IEEE Symposium on Security and Privacy (SP\u201919)","author":"Lecuyer Mathias","year":"2019","unstructured":"Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. 2019. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP\u201919). IEEE, 656\u2013672."},{"key":"e_1_3_3_34_2","article-title":"SNIP: Single-shot network pruning based on connection sensitivity","author":"Lee Namhoon","year":"2018","unstructured":"Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2018. SNIP: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340.","journal-title":"arXiv preprint arXiv:1810.02340"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00573"},{"key":"e_1_3_3_36_2","article-title":"Pruning filters for efficient convnets","author":"Li Hao","year":"2016","unstructured":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710.","journal-title":"arXiv preprint arXiv:1608.08710"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_23"},{"key":"e_1_3_3_39_2","article-title":"Rethinking the value of network pruning","author":"Liu Zhuang","year":"2018","unstructured":"Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2018. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270.","journal-title":"arXiv preprint arXiv:1810.05270"},{"key":"e_1_3_3_40_2","article-title":"Non-structured DNN weight pruning\u2013is it beneficial in any platform?","author":"Ma Xiaolong","year":"2021","unstructured":"Xiaolong Ma, Sheng Lin, Shaokai Ye, Zhezhi He, Linfeng Zhang, Geng Yuan, Sia Huat Tan, Zhengang Li, Deliang Fan, Xuehai Qian, et\u00a0al. 2021. Non-structured DNN weight pruning\u2013is it beneficial in any platform? IEEE Transactions on Neural Networks and Learning Systems.","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"e_1_3_3_41_2","article-title":"Towards deep learning models resistant to adversarial attacks","author":"Madry Aleksander","year":"2017","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.","journal-title":"arXiv preprint arXiv:1706.06083"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134057"},{"key":"e_1_3_3_43_2","unstructured":"Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch."},{"key":"e_1_3_3_44_2","article-title":"Robust sparse regularization: Simultaneously optimizing neural network robustness and compactness","author":"Rakin Adnan Siraj","year":"2019","unstructured":"Adnan Siraj Rakin, Zhezhi He, Li Yang, Yanzhi Wang, Liqiang Wang, and Deliang Fan. 2019. Robust sparse regularization: Simultaneously optimizing neural network robustness and compactness. arXiv preprint arXiv:1905.13074.","journal-title":"arXiv preprint arXiv:1905.13074"},{"key":"e_1_3_3_45_2","article-title":"Fixing data augmentation to improve adversarial robustness","author":"Rebuffi Sylvestre-Alvise","year":"2021","unstructured":"Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. 2021. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946.","journal-title":"arXiv preprint arXiv:2103.01946"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.690"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304076"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01443"},{"key":"e_1_3_3_49_2","article-title":"Hydra: Pruning adversarially robust neural networks","author":"Sehwag Vikash","year":"2020","unstructured":"Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. 2020. Hydra: Pruning adversarially robust neural networks. arXiv preprint arXiv:2002.10509.","journal-title":"arXiv preprint arXiv:2002.10509"},{"key":"e_1_3_3_50_2","article-title":"Adversarial training for free!","author":"Shafahi Ali","year":"2019","unstructured":"Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. 2019. Adversarial training for free! arXiv preprint arXiv:1904.12843.","journal-title":"arXiv preprint arXiv:1904.12843"},{"key":"e_1_3_3_51_2","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.","journal-title":"arXiv preprint arXiv:1409.1556"},{"key":"e_1_3_3_52_2","article-title":"Ensemble adversarial training: Attacks and defenses","author":"Tram\u00e8r Florian","year":"2017","unstructured":"Florian Tram\u00e8r, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2017. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204.","journal-title":"arXiv preprint arXiv:1705.07204"},{"key":"e_1_3_3_53_2","article-title":"Fast is better than free: Revisiting adversarial training","author":"Wong Eric","year":"2020","unstructured":"Eric Wong, Leslie Rice, and J. Zico Kolter. 2020. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994.","journal-title":"arXiv preprint arXiv:2001.03994"},{"key":"e_1_3_3_54_2","article-title":"Autoprune: Automatic network pruning by regularizing auxiliary parameters","volume":"32","author":"Xiao Xia","year":"2019","unstructured":"Xia Xiao and Zigeng Wang. 2019. Autoprune: Automatic network pruning by regularizing auxiliary parameters. In Advances in Neural Information Processing Systems (NeurIPS\u201919), 32.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS\u201919),"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00020"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00958"},{"key":"e_1_3_3_57_2","first-page":"7472","volume-title":"International Conference on Machine Learning","author":"Zhang Hongyang","year":"2019","unstructured":"Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning. PMLR, 7472\u20137482."}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510833","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3510833","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:11Z","timestamp":1750186931000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510833"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,30]]},"references-count":56,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3510833"],"URL":"https:\/\/doi.org\/10.1145\/3510833","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"type":"print","value":"1539-9087"},{"type":"electronic","value":"1558-3465"}],"subject":[],"published":{"date-parts":[[2022,9,30]]},"assertion":[{"value":"2021-08-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}