{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T10:49:12Z","timestamp":1768819752014,"version":"3.49.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,4,7]],"date-time":"2023-04-07T00:00:00Z","timestamp":1680825600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,7]],"date-time":"2023-04-07T00:00:00Z","timestamp":1680825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100017668","name":"Key Research and Development Project of Anhui Province","doi-asserted-by":"crossref","award":["1804a07020108"],"award-info":[{"award-number":["1804a07020108"]}],"id":[{"id":"10.13039\/501100017668","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100017668","name":"Key Research and Development Project of Anhui Province","doi-asserted-by":"crossref","award":["202104a06020012"],"award-info":[{"award-number":["202104a06020012"]}],"id":[{"id":"10.13039\/501100017668","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100017668","name":"Key Research and Development Project of Anhui Province","doi-asserted-by":"crossref","award":["201904a06020056"],"award-info":[{"award-number":["201904a06020056"]}],"id":[{"id":"10.13039\/501100017668","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Independent Project of Anhui Key Laboratory of Smart Agricultural Technology and Equipment","award":["APKLSATE2019X001"],"award-info":[{"award-number":["APKLSATE2019X001"]}]},{"name":"Independent Project of Anhui Key Laboratory of Smart Agricultural Technology and Equipment","award":["APKLSATE2021X001"],"award-info":[{"award-number":["APKLSATE2021X001"]}]},{"name":"the Ministry of Agriculture Agricultural Internet of Things Technology Integration and Application Key Laboratory Open Fund in 2016","award":["2016KL05"],"award-info":[{"award-number":["2016KL05"]}]},{"name":"Major Project of Natural Science Research in Universities of Anhui Province","award":["KJ2019ZD20"],"award-info":[{"award-number":["KJ2019ZD20"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Although the classification method based on the deep neural network has achieved excellent results in classification tasks, it is difficult to apply to real-time scenarios because of high memory footprints and prohibitive inference times. Compared to unstructured pruning, structured pruning techniques can reduce the computation cost of the model runtime more effectively, but inevitably reduces the precision of the model. Traditional methods use fine tuning to restore model damage performance. However, there is still a large gap between the pruned model and the original one. In this paper, we use progressive multi-level distillation learning to compensate for the loss caused by pruning. Pre-pruning and post-pruning networks serve as the teacher and student networks. The proposed approach utilizes the complementary properties of structured pruning and knowledge distillation, which allows the pruned network to learn the intermediate and output representations of the teacher network, thus reducing the influence of the model subject to pruning. Experiments demonstrate that our approach performs better on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets with different pruning rates. For instance, GoogLeNet can achieve near lossless pruning on the CIFAR-10 dataset with 60% pruning. Moreover, this paper also proves that using the proposed distillation learning method during the pruning process achieves more significant performance gains than after completing the pruning.<\/jats:p>","DOI":"10.1007\/s40747-023-01036-0","type":"journal-article","created":{"date-parts":[[2023,4,7]],"date-time":"2023-04-07T02:02:16Z","timestamp":1680832936000},"page":"5779-5791","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Progressive multi-level distillation learning for pruning network"],"prefix":"10.1007","volume":"9","author":[{"given":"Ruiqing","family":"Wang","sequence":"first","affiliation":[]},{"given":"Shengmin","family":"Wan","sequence":"additional","affiliation":[]},{"given":"Wu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Chenlu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Yu","family":"Li","sequence":"additional","affiliation":[]},{"given":"Shaoxiang","family":"Xu","sequence":"additional","affiliation":[]},{"given":"Lifu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Xiu","family":"Jin","sequence":"additional","affiliation":[]},{"given":"Zhaohui","family":"Jiang","sequence":"additional","affiliation":[]},{"given":"Yuan","family":"Rao","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,4,7]]},"reference":[{"issue":"11","key":"1036_CR1","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324","journal-title":"Proc IEEE"},{"key":"1036_CR2","unstructured":"Krizhevsky A, Sutskever I, Hinton G E (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst"},{"key":"1036_CR3","first-page":"4700","volume":"2017","author":"G Huang","year":"2017","unstructured":"Huang G, Liu Z, Van Der Maaten L et al (2017) Densely connected convolutional networks. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2017:4700\u20134708","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR4","first-page":"580","volume":"2014","author":"R Girshick","year":"2014","unstructured":"Girshick R, Donahue J, Darrell T et al (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2014:580\u2013587","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR5","unstructured":"Ren S, He K, Girshick R, et al (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst"},{"key":"1036_CR6","unstructured":"Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv:1804.02767"},{"issue":"3","key":"1036_CR7","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1109\/MCI.2018.2840738","volume":"13","author":"T Young","year":"2018","unstructured":"Young T, Hazarika D, Poria S et al (2018) Recent trends in deep learning based natural language processing. IEEE Comput Intell Mag 13(3):55\u201375","journal-title":"IEEE Comput Intell Mag"},{"key":"1036_CR8","doi-asserted-by":"crossref","unstructured":"Zagoruyko S, Komodakis N (2016) Wide residual networks. arXiv:1605.07146","DOI":"10.5244\/C.30.87"},{"key":"1036_CR9","first-page":"1492","volume":"2017","author":"S Xie","year":"2017","unstructured":"Xie S, Girshick R, Doll\u00e1r P et al (2017) Aggregated residual transformations for deep neural networks. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2017:1492\u20131500","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"issue":"7","key":"1036_CR10","doi-asserted-by":"publisher","first-page":"5113","DOI":"10.1007\/s10462-020-09816-7","volume":"53","author":"T Choudhary","year":"2020","unstructured":"Choudhary T, Mishra V, Goswami A et al (2020) A comprehensive survey on model compression and acceleration. Artif Intell Rev 53(7):5113\u20135155","journal-title":"Artif Intell Rev"},{"key":"1036_CR11","unstructured":"Vadera S, Ameen S (2020) Methods for pruning deep neural networks. arXiv:2011.00241"},{"issue":"6","key":"1036_CR12","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","volume":"129","author":"J Gou","year":"2021","unstructured":"Gou J, Yu B, Maybank SJ et al (2021) Knowledge distillation: a survey. Int J Comput Vision 129(6):1789\u20131819","journal-title":"Int J Comput Vision"},{"key":"1036_CR13","first-page":"1","volume":"2021","author":"L Chen","year":"2021","unstructured":"Chen L, Chen Y, Xi J et al (2021) Knowledge from the original network: restore a better pruned network with knowledge distillation. Complex Intell Syst 2021:1\u201310","journal-title":"Complex Intell Syst"},{"key":"1036_CR14","doi-asserted-by":"crossref","unstructured":"Kim J, Chang S, Kwak N (2021) PQK: model compression via pruning, quantization, and knowledge distillation. arXiv:2106.14681","DOI":"10.21437\/Interspeech.2021-248"},{"key":"1036_CR15","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.neucom.2021.05.084","volume":"458","author":"B Cui","year":"2021","unstructured":"Cui B, Li Y, Zhang Z (2021) Joint structured pruning and dense knowledge distillation for efficient transformer model compression. Neurocomputing 458:56\u201369","journal-title":"Neurocomputing"},{"issue":"10","key":"1036_CR16","doi-asserted-by":"publisher","first-page":"1769","DOI":"10.3390\/sym13101769","volume":"13","author":"R Wang","year":"2021","unstructured":"Wang R, Zhang W, Ding J et al (2021) Deep neural network compression for plant disease recognition. Symmetry 13(10):1769","journal-title":"Symmetry"},{"issue":"3","key":"1036_CR17","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1145\/3007787.3001163","volume":"44","author":"S Han","year":"2016","unstructured":"Han S, Liu X, Mao H et al (2016) EIE: efficient inference engine on compressed deep neural network. ACM SIGARCH Comput Architecture News 44(3):243\u2013254","journal-title":"ACM SIGARCH Comput Architecture News"},{"key":"1036_CR18","unstructured":"LeCun Y, Denker J, Solla S (1989) Optimal brain damage. Adv Neural Inf Process Syst"},{"key":"1036_CR19","unstructured":"Hassibi B, Stork D (1992) Second order derivatives for network pruning: Optimal brain surgeon. Adv Neural Inf Process Syst"},{"key":"1036_CR20","unstructured":"Han S, Pool J, Tran J, et al (2015) Learning both weights and connections for efficient neural network. Adv Neural Inf Process Syst"},{"key":"1036_CR21","unstructured":"Li H, Kadav A, Durdanovic I, et al (2016) Pruning filters for efficient convnets. arXiv:1608.08710"},{"key":"1036_CR22","first-page":"2736","volume":"2017","author":"Z Liu","year":"2017","unstructured":"Liu Z, Li J, Shen Z et al (2017) Learning efficient convolutional networks through network slimming. Proc IEEE Int Conf Computer Vision (ICCV) 2017:2736\u20132744","journal-title":"Proc IEEE Int Conf Computer Vision (ICCV)"},{"key":"1036_CR23","unstructured":"Molchanov P, Tyree S, Karras T, et al (2016) Pruning convolutional neural networks for resource efficient inference. arXiv:1611.06440"},{"key":"1036_CR24","first-page":"1529","volume":"2020","author":"M Lin","year":"2020","unstructured":"Lin M, Ji R, Wang Y et al (2020) Hrank: filter pruning using high-rank feature map. Proc IEEE Confer Comput Vision Pattern Recogn (CVPR) 2020:1529\u20131538","journal-title":"Proc IEEE Confer Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR25","unstructured":"Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv:1503.02531"},{"key":"1036_CR26","unstructured":"Romero A, Ballas N, Kahou S E, et al (2014) Fitnets: hints for thin deep nets. arXiv:1412.6550"},{"key":"1036_CR27","unstructured":"Komodakis N, Zagoruyko S (2017) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: International conference on learning representations (ICLR)"},{"key":"1036_CR28","first-page":"4133","volume":"2017","author":"J Yim","year":"2017","unstructured":"Yim J, Joo D, Bae J et al (2017) A gift from knowledge distillation: fast optimization, network minimization and transfer learning. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2017:4133\u20134141","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR29","first-page":"1365","volume":"2019","author":"F Tung","year":"2019","unstructured":"Tung F, Mori G (2019) Similarity-preserving knowledge distillation. Proc IEEE Int Conf Comput Vision (ICCV) 2019:1365\u20131374","journal-title":"Proc IEEE Int Conf Comput Vision (ICCV)"},{"key":"1036_CR30","first-page":"6356","volume":"2017","author":"Q Li","year":"2017","unstructured":"Li Q, Jin S, Yan J (2017) Mimicking very efficient network for object detection. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2017:6356\u20136364","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR31","unstructured":"Zhang L, Ma K (2020) Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In: International conference on learning representations (ICLR)"},{"issue":"2","key":"1036_CR32","doi-asserted-by":"publisher","first-page":"2149","DOI":"10.1007\/s12652-020-02312-4","volume":"12","author":"H Xie","year":"2021","unstructured":"Xie H, Jiang W, Luo H et al (2021) Model compression via pruning and knowledge distillation for person re-identification. J Ambient Intell Humaniz Comput 12(2):2149\u20132161","journal-title":"J Ambient Intell Humaniz Comput"},{"key":"1036_CR33","unstructured":"Gong Y, Liu L, Yang M, et al (2014) Compressing deep convolutional networks using vector quantization. arXiv:1412.6115"},{"key":"1036_CR34","unstructured":"Hubara I, Courbariaux M, Soudry D, et al (2016) Binarized neural networks: Training neural networks with weights and activations constrained to +1 or -1. arXiv:1602.02830"},{"key":"1036_CR35","unstructured":"Han S, Mao H, Dally W J (2015) Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149"},{"key":"1036_CR36","first-page":"18","volume":"2020","author":"D Walawalkar","year":"2020","unstructured":"Walawalkar D, Shen Z, Savvides M (2020) Online ensemble model compression using knowledge distillation. Eur Conf Comput Vision (ECCV) 2020:18\u201335","journal-title":"Eur Conf Comput Vision (ECCV)"},{"key":"1036_CR37","unstructured":"Chen G, Choi W, Yu X, et al (2017) Learning efficient object detection models with knowledge distillation. Adv Neural Inf Process Syst"},{"key":"1036_CR38","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556"},{"key":"1036_CR39","first-page":"770","volume":"2016","author":"K He","year":"2016","unstructured":"He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. Proce IEEE Conf Comput Vision Pattern Recogn (CVPR) 2016:770\u2013778","journal-title":"Proce IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR40","first-page":"1","volume":"2015","author":"C Szegedy","year":"2015","unstructured":"Szegedy C, Liu W, Jia Y et al (2015) Going deeper with convolutions. Proc IEEE Conf Comput Vision Pattern Recogn (CVPR) 2015:1\u20139","journal-title":"Proc IEEE Conf Comput Vision Pattern Recogn (CVPR)"},{"key":"1036_CR41","unstructured":"Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images"},{"issue":"04","key":"1036_CR42","first-page":"5191","volume":"34","author":"SI Mirzadeh","year":"2020","unstructured":"Mirzadeh SI, Farajtabar M, Li A et al (2020) Improved knowledge distillation via teacher assistant. Proc Conf AAAI Artif Intell 34(04):5191\u20135198","journal-title":"Proc Conf AAAI Artif Intell"},{"key":"1036_CR43","unstructured":"Le Y, Yang X (2015) Tiny imagenet visual recognition challenge. CS 231N 7(7):3"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01036-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-01036-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01036-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T17:25:41Z","timestamp":1695403541000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-01036-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,7]]},"references-count":43,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["1036"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-01036-0","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,7]]},"assertion":[{"value":"27 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 March 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}