{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T16:09:17Z","timestamp":1780675757865,"version":"3.54.1"},"reference-count":27,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,1,21]],"date-time":"2024-01-21T00:00:00Z","timestamp":1705795200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>We present a novel technique for pruning called activation-based pruning to effectively prune fully connected feedforward neural networks for multi-object classification. Our technique is based on the number of times each neuron is activated during model training. We compare the performance of activation-based pruning with a popular pruning method: magnitude-based pruning. Further analysis demonstrated that activation-based pruning can be considered a dimensionality reduction technique, as it leads to a sparse low-rank matrix approximation for each hidden layer of the neural network. We also demonstrate that the rank-reduced neural network generated using activation-based pruning has better accuracy than a rank-reduced network using principal component analysis. We provide empirical results to show that, after each successive pruning, the amount of reduction in the magnitude of singular values of each matrix representing the hidden layers of the network is equivalent to introducing the sum of singular values of the hidden layers as a regularization parameter to the objective function.<\/jats:p>","DOI":"10.3390\/a17010048","type":"journal-article","created":{"date-parts":[[2024,1,22]],"date-time":"2024-01-22T06:49:31Z","timestamp":1705906171000},"page":"48","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Activation-Based Pruning of Neural Networks"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-7575-1706","authenticated-orcid":false,"given":"Tushar","family":"Ganguli","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7622-4815","authenticated-orcid":false,"given":"Edwin K. P.","family":"Chong","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Colorado State University, Fort Collins, CO 80523, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,21]]},"reference":[{"key":"ref_1","unstructured":"Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. (2017). Pruning Convolutional Neural Networks for Resource Efficient Inference. arXiv."},{"key":"ref_2","unstructured":"Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_3","unstructured":"Hu, H., Peng, R., Tai, Y.W., and Tang, C.K. (2016). Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures. arXiv."},{"key":"ref_4","unstructured":"Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2017). Pruning Filters for Efficient ConvNets. arXiv."},{"key":"ref_5","unstructured":"Frankle, J., and Carbin, M. (2019, January 6\u20139). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_6","unstructured":"Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (2016). Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_7","unstructured":"Han, S., Mao, H., and Dally, W.J. (2016). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv."},{"key":"ref_8","unstructured":"Zhu, M., and Gupta, S. (2017). To prune, or not to prune: Exploring the efficacy of pruning for model compression. arXiv."},{"key":"ref_9","unstructured":"Shlens, J. (2014). A Tutorial on Principal Component Analysis. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.neucom.2020.02.035","article-title":"Sparse low rank factorization for deep neural network compression","volume":"398","author":"Swaminathan","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yang, H., Tang, M., Wen, W., Yan, F., Hu, D., Li, A., Li, H., and Chen, Y. (2020, January 14\u201319). Learning Low-Rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Virtually.","DOI":"10.1109\/CVPRW50498.2020.00347"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"He, Y., Zhang, X., and Sun, J. (2017, January 22\u201329). Channel Pruning for Accelerating Very Deep Neural Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.155"},{"key":"ref_13","unstructured":"Gray, S., Radford, A., and Kingma, D.P. (2017). GPU Kernels for Block-Sparse Weights. arXiv."},{"key":"ref_14","unstructured":"Dy, J., and Krause, A. (2018, January 10\u201315). Efficient Neural Audio Synthesis. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden. Proceedings of Machine Learning Research."},{"key":"ref_15","unstructured":"Frankle, J., Dziugaite, G.K., Roy, D.M., and Carbin, M. (2020). Stabilizing the Lottery Ticket Hypothesis. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yu, R., Li, A., Chen, C.F., Lai, J.H., Morariu, V.I., Han, X., Gao, M., Lin, C.Y., and Davis, L.S. (2018, January 18\u201323). NISP: Pruning Networks Using Neuron Importance Score Propagation. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00958"},{"key":"ref_17","unstructured":"Precup, D., and Teh, Y.W. (2017, January 6\u201311). Variational Dropout Sparsifies Deep Neural Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia. Proceedings of Machine Learning Research."},{"key":"ref_18","unstructured":"Mariet, Z., and Sra, S. (2017). Diversity Networks: Neural Network Compression Using Determinantal Point Processes. arXiv."},{"key":"ref_19","unstructured":"Lee, N., Ajanthan, T., and Torr, P. (2019, January 6\u20139). SNIP: Single-Shot Network Pruning Based on Connection Sensitivity. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_20","unstructured":"Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. (2019, January 6\u20139). Rethinking the Value of Network Pruning. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_21","unstructured":"Yuan, X., Savarese, P., and Maire, M. (2020). Growing Efficient Deep Networks by Structured Continuous Sparsification. arXiv."},{"key":"ref_22","unstructured":"Wang, C., Zhang, G., and Grosse, R. (2020). Picking Winning Tickets Before Training by Preserving Gradient Flow. arXiv."},{"key":"ref_23","first-page":"6377","article-title":"Pruning neural networks without any data by iteratively conserving synaptic flow","volume":"Volume 33","author":"Larochelle","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_24","unstructured":"Frantar, E., and Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. arXiv."},{"key":"ref_25","unstructured":"Sun, M., Liu, Z., Bair, A., and Kolter, J.Z. (2023). A Simple and Effective Pruning Approach for Large Language Models. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Gu, S., Zhang, L., Zuo, W., and Feng, X. (2014, January 23\u201328). Weighted Nuclear Norm Minimization with Application to Image Denoising. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.366"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zha, Z., Wen, B., Zhang, J., Zhou, J., and Zhu, C. (2019, January 22\u201325). A Comparative Study for the Nuclear Norms Minimization Methods. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803145"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/48\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:46:45Z","timestamp":1760104005000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/48"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,21]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["a17010048"],"URL":"https:\/\/doi.org\/10.3390\/a17010048","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,21]]}}}