{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:01:01Z","timestamp":1760234461071,"version":"build-2065373602"},"reference-count":54,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,5,16]],"date-time":"2021-05-16T00:00:00Z","timestamp":1621123200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key R&amp;D Program of China","award":["No.2018YFD0700300"],"award-info":[{"award-number":["No.2018YFD0700300"]}]},{"name":"Chinese Academy of Sciences Engineering Laboratory for Intelligent Logistics Equipment System","award":["No.KFJ-PTXM-025"],"award-info":[{"award-number":["No.KFJ-PTXM-025"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Convolutional neural networks (CNNs) have achieved significant breakthroughs in various domains, such as natural language processing (NLP), and computer vision. However, performance improvement is often accompanied by large model size and computation costs, which make it not suitable for resource-constrained devices. Consequently, there is an urgent need to compress CNNs, so as to reduce model size and computation costs. This paper proposes a layer-wise differentiable compression (LWDC) algorithm for compressing CNNs structurally. A differentiable selection operator OS is embedded in the model to compress and train the model simultaneously by gradient descent in one go. Instead of pruning parameters from redundant operators by contrast to most of the existing methods, our method replaces the original bulky operators with more lightweight ones directly, which only needs to specify the set of lightweight operators and the regularization factor in advance, rather than the compression rate for each layer. The compressed model produced by our method is generic and does not need any special hardware\/software support. Experimental results on CIFAR-10, CIFAR-100 and ImageNet have demonstrated the effectiveness of our method. LWDC obtains more significant compression than state-of-the-art methods in most cases, while having lower performance degradation. The impact of lightweight operators and regularization factor on the compression rate and accuracy also is evaluated.<\/jats:p>","DOI":"10.3390\/s21103464","type":"journal-article","created":{"date-parts":[[2021,5,17]],"date-time":"2021-05-17T02:31:34Z","timestamp":1621218694000},"page":"3464","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4998-5803","authenticated-orcid":false,"given":"Huabin","family":"Diao","sequence":"first","affiliation":[{"name":"Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuexing","family":"Hao","sequence":"additional","affiliation":[{"name":"Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaoyun","family":"Xu","sequence":"additional","affiliation":[{"name":"Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gongyan","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,16]]},"reference":[{"key":"ref_1","unstructured":"Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5 mb model size. arXiv."},{"key":"ref_2","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (2019, January 27\u201328). Searching for mobilenetv3. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00140"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8\u201314). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01264-9_8"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18\u201323). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00716"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1016\/j.neucom.2020.05.056","article-title":"Mobilecount: An efficient encoder-decoder framework for real-time crowd counting","volume":"407","author":"Wang","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ntakolia, C., Diamantis, D.E., Papandrianos, N., Moustakidis, S., and Papageorgiou, E.I. (2020). A lightweight convolutional neural network architecture applied for bone metastasis classification in nuclear medicine: A case study on prostate cancer patients. Healthcare, 8.","DOI":"10.3390\/healthcare8040493"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Khaki, S., Safaei, N., Pham, H., and Wang, L. (2021). Wheatnet: A lightweight convolutional neural network for high-throughput image-based wheat head detection and counting. arXiv.","DOI":"10.1016\/j.neucom.2022.03.017"},{"key":"ref_10","unstructured":"Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv."},{"key":"ref_11","unstructured":"Xie, S., Zheng, H., Liu, C., and Lin, L. (2018). Snas: Stochastic neural architecture search. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18\u201323). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00907"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1487","DOI":"10.1109\/TC.2019.2914438","article-title":"Nest: A neural network synthesis tool based on a grow-and-prune paradigm","volume":"68","author":"Dai","year":"2019","journal-title":"IEEE Trans. Comput."},{"key":"ref_14","first-page":"4780","article-title":"Regularized evolution for image classifier architecture search","volume":"33","author":"Real","year":"2019","journal-title":"Proc. AAAI Conf. Artif."},{"key":"ref_15","first-page":"7","article-title":"Auto-balanced filter pruning for efficient convolutional neural networks","volume":"3","author":"Ding","year":"2018","journal-title":"AAAI"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"He, Y., Kang, G., Dong, X., Fu, Y., and Yang, Y. (2018). Soft filter pruning for accelerating deep convolutional neural networks. arXiv.","DOI":"10.24963\/ijcai.2018\/309"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, Y., Zhang, X., and Sun, J. (2017, January 22\u201329). Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.155"},{"key":"ref_18","unstructured":"Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2016). Pruning filters for efficient convnets. arXiv."},{"key":"ref_19","unstructured":"Han, S., Mao, H., and Dally, W.J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv."},{"key":"ref_20","unstructured":"Guo, Y., Yao, A., and Chen, Y. (2016). Dynamic network surgery for efficient dnns. arXiv."},{"key":"ref_21","unstructured":"Hu, H., Peng, R., Tai, Y.W., and Tang, C.K. (2016). Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Luo, J.H., Wu, J., and Lin, W. (2017, January 22\u201329). Thinet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.541"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Singh, P., Kadi, V.S., Verma, N., and Namboodiri, V.P. (2019, January 7\u201311). Stability based filter pruning for accelerating deep cnns. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.","DOI":"10.1109\/WACV.2019.00129"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"He, Y., Liu, P., Wang, Z., Hu, Z., and Yang, Y. (2019, January 16\u201320). Filter pruning via geometric median for deep convolutional neural networks acceleration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00447"},{"key":"ref_25","unstructured":"Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. arXiv."},{"key":"ref_26","unstructured":"Yoon, J., and Hwang, S.J. (2017, January 6\u201311). Combined group and exclusive sparsity for deep neural networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., and Wang, Y. (2018, January 8\u201314). A systematic dnn weight pruning framework using alternating direction method of multipliers. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01237-3_12"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ma, Y., Chen, R., Li, W., Shang, F., Yu, W., Cho, M., and Yu, B. (2019, January 4\u20136). A unified approximation framework for compressing and accelerating deep neural networks. Proceedings of the IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), Portland, OR, USA.","DOI":"10.1109\/ICTAI.2019.00060"},{"key":"ref_29","unstructured":"Ye, S., Feng, X., Zhang, T., Ma, X., Lin, S., Li, Z., Xu, K., Wen, W., Liu, S., and Tang, J. (2019). Progressive dnn compression: A key to achieve ultra-high weight pruning and quantization rates using admm. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gusak, J., Kholiavchenko, M., Ponomarev, E., Markeeva, L., Blagoveschensky, P., Cichocki, A., and Oseledets, I. (2019, January 27\u201328). Automated multi-stage compression of neural networks. Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00306"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.neucom.2020.02.035","article-title":"Sparse low rank factorization for deep neural network compression","volume":"398","author":"Swaminathan","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ruan, X., Liu, Y., Yuan, C., Li, B., Hu, W., Li, Y., and Maybank, S. (2020). Edp: An efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Trans. Neural Netw. Learn. Syst.","DOI":"10.1109\/TNNLS.2020.3018177"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., and Han, S. (2018, January 8\u201314). Amc: Automl for model compression and acceleration on mobile devices. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_48"},{"key":"ref_34","unstructured":"Zhu, M., and Gupta, S. (2017). To prune, or not to prune: Exploring the efficacy of pruning for model compression. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"4876","DOI":"10.1609\/aaai.v34i04.5924","article-title":"Autocompress: An automatic dnn structured pruning framework for ultra-high compression rates","volume":"34","author":"Liu","year":"2020","journal-title":"AAAI"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Boyd, S., Parikh, N., and Chu, E. (2011). Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, Now Publishers Inc.","DOI":"10.1561\/9781601984616"},{"key":"ref_37","unstructured":"Shang, W., Sohn, K., Almeida, D., and Lee, H. (2016, January 19\u201324). Understanding and improving convolutional neural networks via concatenated rectified linear units. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_38","unstructured":"Bhardwaj, K., Suda, N., and Marculescu, R. (2019). Dream distillation: A data-independent model compression framework. arXiv."},{"key":"ref_39","unstructured":"Koratana, A., Kang, D., Bailis, P., and Zaharia, M. (2019, January 10\u201315). Lit: Learned intermediate representation training for model compression. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_40","first-page":"1190","article-title":"Private model compression via knowledge distillation","volume":"33","author":"Wang","year":"2019","journal-title":"Proc. AAAI Conf. Artif."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"175703","DOI":"10.1109\/ACCESS.2019.2957203","article-title":"Pruning blocks for cnn compression and acceleration via online ensemble distillation","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"101695","DOI":"10.1016\/j.sysarc.2019.101695","article-title":"Multi-teacher knowledge distillation for compressed video action recognition based on deep learning","volume":"103","author":"Wu","year":"2020","journal-title":"J. Syst. Archit."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Prakosa, S.W., Leu, J.S., and Chen, Z.H. (2020). Improving the accuracy of pruned network using knowledge distillation. Pattern Anal. Appl., 1\u201312.","DOI":"10.1007\/s10044-020-00940-2"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Ahmed, W., Zunino, A., Morerio, P., and Murino, V. (2021, January 10\u201315). Compact cnn structure learning by knowledge distillation. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413006"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"4452","DOI":"10.1609\/aaai.v34i04.5872","article-title":"Plug-in, trainable gate for streamlining arbitrary neural networks","volume":"34","author":"Kim","year":"2020","journal-title":"AAAI"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., and Song, J. (2020, January 13\u201319). Forward and backward information retention for accurate binary neural networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00232"},{"key":"ref_48","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto."},{"key":"ref_49","unstructured":"Chrabaszcz, P., Loshchilov, I., and Hutter, F. (2017). A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv."},{"key":"ref_50","unstructured":"Dong, X., and Yang, Y. (2020). Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv."},{"key":"ref_51","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_52","unstructured":"Duggal, R., Xiao, C., Vuduc, R., and Sun, J. (2019). Cup: Cluster pruning for compressing deep neural networks. arXiv."},{"key":"ref_53","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3464\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:02:13Z","timestamp":1760162533000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/10\/3464"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,16]]},"references-count":54,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21103464"],"URL":"https:\/\/doi.org\/10.3390\/s21103464","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,5,16]]}}}