{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:09:20Z","timestamp":1760148560699,"version":"build-2065373602"},"reference-count":61,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,5,17]],"date-time":"2023-05-17T00:00:00Z","timestamp":1684281600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62271153","22ZR1406700"],"award-info":[{"award-number":["62271153","22ZR1406700"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007219","name":"Natural Science Foundation of Shanghai","doi-asserted-by":"publisher","award":["62271153","22ZR1406700"],"award-info":[{"award-number":["62271153","22ZR1406700"]}],"id":[{"id":"10.13039\/100007219","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The rapid development of neural networks has come at the cost of increased computational complexity. Neural networks are both computationally intensive and memory intensive; as such, the minimal energy and computing power of satellites pose a challenge for automatic target recognition (ATR). Knowledge distillation (KD) can distill knowledge from a cumbersome teacher network to a lightweight student network, transferring the essential information learned by the teacher network. Thus, the concept of KD can be used to improve the accuracy of student networks. Even when learning from a teacher network, there is still redundancy in the student network. Traditional networks fix the structure before training, such that training does not improve the situation. This paper proposes a distillation sparsity training (DST) algorithm based on KD and network pruning to address the above limitations. We first improve the accuracy of the student network through KD, and then through network pruning, allowing the student network to learn which connections are essential. DST allows the teacher network to teach the pruned student network directly. The proposed algorithm was tested on the CIFAR-100, MSTAR, and FUSAR-Ship data sets, with a 50% sparsity setting. First, a new loss function for the teacher-pruned student was proposed, and the pruned student network showed a performance close to that of the teacher network. Second, a new sparsity model (uniformity half-pruning UHP) was designed to solve the problem that unstructured pruning does not facilitate the implementation of general-purpose hardware acceleration and storage. Compared with traditional unstructured pruning, UHP can double the speed of neural networks.<\/jats:p>","DOI":"10.3390\/rs15102609","type":"journal-article","created":{"date-parts":[[2023,5,18]],"date-time":"2023-05-18T06:32:58Z","timestamp":1684391578000},"page":"2609","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-9365-1361","authenticated-orcid":false,"given":"Penghao","family":"Xiao","sequence":"first","affiliation":[{"name":"Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Teng","family":"Xu","sequence":"additional","affiliation":[{"name":"Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiayang","family":"Xiao","sequence":"additional","affiliation":[{"name":"Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1486-4363","authenticated-orcid":false,"given":"Weisong","family":"Li","sequence":"additional","affiliation":[{"name":"Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1912-7143","authenticated-orcid":false,"given":"Haipeng","family":"Wang","sequence":"additional","affiliation":[{"name":"Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_2","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18\u201322). Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00214"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Godard, C., Aodha, O.M., and Brostow, G.J. (2017, January 21\u201326). Unsupervised Monocular Depth Estimation with Left-Right Consistency. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.699"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017). Focal Loss for Dense Object Detection. arXiv.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_7","unstructured":"Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.M.A., Yang, Y., and Zhou, Y. (2017). Deep Learning Scaling Is Predictable, Empirically. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hestness, J., Ardalani, N., and Diamos, G. (2019). Beyond Human-Level Accuracy: Computational Challenges in Deep Learning. arXiv.","DOI":"10.1145\/3293883.3295710"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Patwary, M., Chabbi, M., Jun, H., Huang, J., Diamos, G., and Church, K. (2018). Language Modeling at Scale. arXiv.","DOI":"10.1109\/IPDPS.2019.00068"},{"key":"ref_10","first-page":"1","article-title":"Mesh-TensorFlow: Deep Learning for Supercomputers","volume":"31","author":"Shazeer","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"5113","DOI":"10.1007\/s10462-020-09816-7","article-title":"A Comprehensive Survey on Model Compression and Acceleration","volume":"53","author":"Choudhary","year":"2020","journal-title":"Artif. Intell. Rev."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1109\/MSP.2017.2765695","article-title":"Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges","volume":"35","author":"Cheng","year":"2018","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","article-title":"Knowledge Distillation: A Survey","volume":"129","author":"Gou","year":"2021","journal-title":"Int. J. Comput. Vis."},{"key":"ref_14","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"5191","DOI":"10.1609\/aaai.v34i04.5963","article-title":"Improved Knowledge Distillation via Teacher Assistant","volume":"34","author":"Mirzadeh","year":"2020","journal-title":"AAAI"},{"key":"ref_16","first-page":"13292","article-title":"Learning Student-Friendly Teacher Networks for Knowledge Distillation","volume":"34","author":"Park","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_17","first-page":"598","article-title":"Optimal Brain Damage","volume":"2","author":"LeCun","year":"1990","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/502172a","article-title":"Peter huttenlocher (1931\u20132013)","volume":"502","author":"Walsh","year":"2013","journal-title":"Nature"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Str\u00f6m, N. (1997, January 22\u201325). Sparse Connection and Pruning in Large Dynamic Artificial Neural Networks. Proceedings of the 5th European Conference on Speech Communication and Technology (Eurospeech 1997), Rhodes, Greece.","DOI":"10.21437\/Eurospeech.1997-708"},{"key":"ref_20","unstructured":"Frankle, J., and Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv."},{"key":"ref_21","unstructured":"NVIDIA (2018). NVIDIA Turing GPU Architecture Whitepaper, NVIDIA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T.J., and Choi, E. (2018, January 18\u201322). MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00171"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1007\/978-3-030-01252-6_17","article-title":"Learning Deep Representations with Probabilistic Knowledge Transfer","volume":"Volume 11215","author":"Ferrari","year":"2018","journal-title":"Computer Vision\u2013ECCV 2018"},{"key":"ref_25","unstructured":"Zagoruyko, S., and Komodakis, N. (2017). Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. arXiv."},{"key":"ref_26","unstructured":"Tian, Y., Krishnan, D., and Isola, P. (2022). Contrastive Representation Distillation. arXiv."},{"key":"ref_27","first-page":"1","article-title":"Paraphrasing Complex Network: Network Compression via Factor Transfer","volume":"31","author":"Kim","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_28","first-page":"1","article-title":"Predicting Parameters in Deep Learning","volume":"26","author":"Denil","year":"2013","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","unstructured":"Lin, M., Chen, Q., and Yan, S. (2014). Network In Network. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going Deeper with Convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_31","first-page":"177","article-title":"Comparing Biases for Minimal Network Construction with Back-Propagation","volume":"1","author":"Hanson","year":"1988","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","first-page":"164","article-title":"Second Order Derivatives for Network Pruning: Optimal Brain Surgeon","volume":"5","author":"Hassibi","year":"1992","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","unstructured":"Han, S., Mao, H., and Dally, W.J. (2016). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv."},{"key":"ref_34","unstructured":"Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. (2019). Rethinking the Value of Network Pruning. arXiv."},{"key":"ref_35","unstructured":"Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M.A., and Dally, W.J. (2016). EIE: Efficient Inference Engine on Compressed Deep Neural Network. arXiv.","DOI":"10.1109\/ISCA.2016.30"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3005348","article-title":"Structured Pruning of Deep Convolutional Neural Networks","volume":"13","author":"Anwar","year":"2017","journal-title":"J. Emerg. Technol. Comput. Syst."},{"key":"ref_38","unstructured":"Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2017). Pruning Filters for Efficient ConvNets. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.T., and Sun, J. (November, January 27). MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00339"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Su, X., You, S., Wang, F., Qian, C., Zhang, C., and Xu, C. (2021, January 19\u201325). BCNet: Searching for Network Width with Bilaterally Coupled Network. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00221"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"He, Y., Kang, G., Dong, X., Fu, Y., and Yang, Y. (2018). Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. arXiv.","DOI":"10.24963\/ijcai.2018\/309"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"He, Y., Zhang, X., and Sun, J. (2017, January 22\u201329). Channel Pruning for Accelerating Very Deep Neural Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.155"},{"key":"ref_43","first-page":"1","article-title":"Learning Structured Sparsity in Deep Neural Networks","volume":"29","author":"Wen","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1007\/978-3-319-46493-0_40","article-title":"Less Is More: Towards Compact CNNs","volume":"Volume 9908","author":"Leibe","year":"2016","journal-title":"Computer Vision\u2014ECCV 2016"},{"key":"ref_45","unstructured":"Ye, J., Lu, X., Lin, Z., and Wang, J.Z. (2018). Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers. arXiv."},{"key":"ref_46","first-page":"9865","article-title":"Neuron-Level Structured Pruning Using Polarization Regularizer","volume":"33","author":"Zhuang","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Aghli, N., and Ribeiro, E. (2021, January 20\u201325). Combining Weight Pruning and Knowledge Distillation For CNN Compression. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00356"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"3048","DOI":"10.1109\/TPAMI.2018.2874634","article-title":"Shallowing Deep Networks: Layer-Wise Pruning Based on Feature Representations","volume":"41","author":"Chen","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_49","unstructured":"Sarridis, I., Koutlis, C., Kordopatis-Zilos, G., Kompatsiaris, I., and Papadopoulos, S. (2022). InDistill: Information Flow-Preserving Knowledge Distillation for Model Compression. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Park, J., and No, A. (2022). Prune Your Model Before Distill It. arXiv.","DOI":"10.1007\/978-3-031-20083-0_8"},{"key":"ref_51","unstructured":"Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Yang, Y., Qiu, J., Song, M., Tao, D., and Wang, X. (2020, January 13\u201319). Distilling Knowledge From Graph Convolutional Networks. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00710"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Yuan, L., Tay, F.E., Li, G., Wang, T., and Feng, J. (2020, January 13\u201319). Revisiting Knowledge Distillation via Label Smoothing Regularization. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00396"},{"key":"ref_54","unstructured":"NVIDIA (2022, September 01). Sparse Matrix Storage Formats. Available online: https:\/\/www.gnu.org\/software\/gsl\/doc\/html\/spmatrix.html."},{"key":"ref_55","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report."},{"key":"ref_56","unstructured":"Zelnio, E.G., and Douglass, R.J. (1996). MSTAR Extended Operating Conditions: A Tutorial, Aerospace\/Defense Sensing and Controls."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"140303","DOI":"10.1007\/s11432-019-2772-5","article-title":"FUSAR-Ship: Building a High-Resolution SAR-AIS Matchup Dataset of Gaofen-3 for Ship Detection and Recognition","volume":"63","author":"Hou","year":"2020","journal-title":"Sci. China Inf. Sci."},{"key":"ref_58","first-page":"1","article-title":"Learning Both Weights and Connections for Efficient Neural Network","volume":"28","author":"Han","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_59","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., and Pang, R. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00140"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/10\/2609\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:36:58Z","timestamp":1760125018000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/10\/2609"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,17]]},"references-count":61,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["rs15102609"],"URL":"https:\/\/doi.org\/10.3390\/rs15102609","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2023,5,17]]}}}