{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T07:48:10Z","timestamp":1777967290295,"version":"3.51.4"},"reference-count":58,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T00:00:00Z","timestamp":1715299200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T00:00:00Z","timestamp":1715299200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LDT23F01013F01"],"award-info":[{"award-number":["LDT23F01013F01"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LDT23F01013F01"],"award-info":[{"award-number":["LDT23F01013F01"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LDT23F01013F01"],"award-info":[{"award-number":["LDT23F01013F01"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LDT23F01013F01"],"award-info":[{"award-number":["LDT23F01013F01"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LDT23F01013F01"],"award-info":[{"award-number":["LDT23F01013F01"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Process Lett"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Knowledge distillation is a model compression technique that transfers knowledge learned by teacher networks to student networks. Existing knowledge distillation methods greatly expand the forms of knowledge, but also make the distillation models complex and symmetric. However, few studies have explored the commonalities among these methods. In this study, we propose a concise distillation framework to unify these methods and a method to construct asymmetric knowledge distillation under the framework. Asymmetric distillation aims to enable differentiated knowledge transfers for different distillation objects. We designed a multi-stage shallow-wide branch bifurcation method to distill different knowledge representations and a grouping ensemble strategy to supervise the network to teach and learn selectively. Consequently, we conducted experiments using image classification benchmarks to verify the proposed method. Experimental results show that our implementation can achieve considerable improvements over existing methods, demonstrating the effectiveness of the method and the potential of the framework.<\/jats:p>","DOI":"10.1007\/s11063-024-11606-z","type":"journal-article","created":{"date-parts":[[2024,5,11]],"date-time":"2024-05-11T01:38:56Z","timestamp":1715391536000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A Unified Asymmetric Knowledge Distillation Framework for Image Classification"],"prefix":"10.1007","volume":"56","author":[{"given":"Xin","family":"Ye","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiang","family":"Tian","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bolun","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yaowu","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,5,10]]},"reference":[{"key":"11606_CR1","unstructured":"Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531"},{"key":"11606_CR2","doi-asserted-by":"crossref","unstructured":"Yuan L, Tay FE, Li G, Wang T, Feng J (2020) Revisiting knowledge distillation via label smoothing regularization. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 3903\u20133911","DOI":"10.1109\/CVPR42600.2020.00396"},{"issue":"6","key":"11606_CR3","doi-asserted-by":"publisher","first-page":"4735","DOI":"10.1007\/s11063-022-10830-9","volume":"54","author":"X Li","year":"2022","unstructured":"Li X, Xiong H, An H, Xu C, Dou D (2022) Colam: co-learning of deep neural networks and soft labels via alternating minimization. Neural Process Lett 54(6):4735\u20134749","journal-title":"Neural Process Lett"},{"key":"11606_CR4","doi-asserted-by":"crossref","unstructured":"Li Y, Yang J, Song Y, Cao L, Luo J, Li L-J (2017) Learning from noisy labels with distillation. In: Proceedings of the IEEE international conference on computer vision, pp 1910\u20131918","DOI":"10.1109\/ICCV.2017.211"},{"key":"11606_CR5","unstructured":"Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550"},{"key":"11606_CR6","doi-asserted-by":"crossref","unstructured":"Yim J, Joo D, Bae J, Kim J (2017) A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4133\u20134141","DOI":"10.1109\/CVPR.2017.754"},{"key":"11606_CR7","doi-asserted-by":"crossref","unstructured":"Heo B, Kim J, Yun S, Park H, Kwak N, Choi JY (2019) A comprehensive overhaul of feature distillation. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 1921\u20131930","DOI":"10.1109\/ICCV.2019.00201"},{"issue":"5","key":"11606_CR8","doi-asserted-by":"publisher","first-page":"6165","DOI":"10.1007\/s11063-022-11132-w","volume":"55","author":"L Li","year":"2023","unstructured":"Li L, Su W, Liu F, He M, Liang X (2023) Knowledge fusion distillation: improving distillation with multi-scale attention mechanisms. Neural Process Lett 55(5):6165\u20136180","journal-title":"Neural Process Lett"},{"key":"11606_CR9","doi-asserted-by":"crossref","unstructured":"Zhu J, Tang S, Chen D, Yu S, Liu Y, Rong M, Yang A, Wang X (2021) Complementary relation contrastive distillation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 9260\u20139269","DOI":"10.1109\/CVPR46437.2021.00914"},{"key":"11606_CR10","doi-asserted-by":"crossref","unstructured":"Ji M, Heo B, Park S (2021) Show, attend and distill: Knowledge distillation via attention-based feature matching. In: Proceedings of the AAAI conference on artificial intelligence, vol. 35, pp 7945\u20137952","DOI":"10.1609\/aaai.v35i9.16969"},{"key":"11606_CR11","doi-asserted-by":"crossref","unstructured":"Liu W, Nie S, Yin J, Wang R, Gao D, Jin L (2021) Sskd: Self-supervised knowledge distillation for cross domain adaptive person re-identification. In: 2021 7th IEEE international conference on network intelligence and digital content, pp 81\u201385 . IEEE","DOI":"10.1109\/IC-NIDC54101.2021.9660538"},{"key":"11606_CR12","doi-asserted-by":"crossref","unstructured":"Yang Z, Shou L, Gong M, Lin W, Jiang D (2020) Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In: Proceedings of the 13th international conference on web search and data mining, pp 690\u2013698","DOI":"10.1145\/3336191.3371792"},{"key":"11606_CR13","doi-asserted-by":"crossref","unstructured":"Zhang Y, Xiang T, Hospedales TM, Lu H (2018) Deep mutual learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4320\u20134328","DOI":"10.1109\/CVPR.2018.00454"},{"key":"11606_CR14","doi-asserted-by":"crossref","unstructured":"Chen P, Liu S, Zhao H, Jia J (2021) Distilling knowledge via knowledge review. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, pp 5008\u20135017","DOI":"10.1109\/CVPR46437.2021.00497"},{"issue":"11","key":"11606_CR15","first-page":"8183","volume":"44","author":"G-H Wang","year":"2021","unstructured":"Wang G-H, Ge Y, Wu J (2021) Distilling knowledge by mimicking features. IEEE Trans Pattern Anal Mach Intell 44(11):8183\u20138195","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"11606_CR16","unstructured":"Wang Y, Cheng L, Duan M, Wang Y, Feng Z, Kong S (2023) Improving knowledge distillation via regularizing feature norm and direction. arXiv preprint arXiv:2305.17007"},{"key":"11606_CR17","unstructured":"Kim J, Park S, Kwak N (2018) Paraphrasing complex network: network compression via factor transfer. Adv Neural Inform Process Syst, 31"},{"key":"11606_CR18","doi-asserted-by":"crossref","unstructured":"Lee SH, Kim DH, Song BC (2018) Self-supervised knowledge distillation using singular value decomposition. In: Proceedings of the European conference on computer vision, pp 335\u2013350","DOI":"10.1007\/978-3-030-01231-1_21"},{"key":"11606_CR19","unstructured":"Srinivas S, Fleuret F (2018) Knowledge transfer with jacobian matching. In: International conference on machine learning, pp 4723\u20134731. PMLR"},{"key":"11606_CR20","doi-asserted-by":"crossref","unstructured":"Park W, Kim D, Lu Y, Cho M (2019) Relational knowledge distillation. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, pp 3967\u20133976","DOI":"10.1109\/CVPR.2019.00409"},{"key":"11606_CR21","doi-asserted-by":"crossref","unstructured":"Liu Y, Cao J, Li B, Yuan C, Hu W, Li Y, Duan Y (2019) Knowledge distillation via instance relationship graph. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, pp 7096\u20137104","DOI":"10.1109\/CVPR.2019.00726"},{"key":"11606_CR22","doi-asserted-by":"crossref","unstructured":"Chen Z, Zheng X, Shen H, Zeng Z, Zhou Y, Zhao R (2020) Improving knowledge distillation via category structure. In: Proceedings of the European conference on computer vision, pp 205\u2013219. Springer","DOI":"10.1007\/978-3-030-58604-1_13"},{"key":"11606_CR23","doi-asserted-by":"crossref","unstructured":"Tung F, Mori G (2019) Similarity-preserving knowledge distillation. In: Proceedings of the IEEE\/CVF International conference on computer vision, pp 1365\u20131374","DOI":"10.1109\/ICCV.2019.00145"},{"key":"11606_CR24","unstructured":"Komodakis N, Zagoruyko S (2017) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: International conference on learning representations"},{"key":"11606_CR25","doi-asserted-by":"crossref","unstructured":"Chariton A, Passalis N, Pleros N, Tefas A (2023) Mutual information-based neural network distillation for improving photonic neural network training. Neural Process Lett, 1\u201316","DOI":"10.1007\/s11063-023-11170-y"},{"key":"11606_CR26","unstructured":"Tian Y, Krishnan D, Isola P (2019) Contrastive representation distillation. In: international conference on learning representations"},{"key":"11606_CR27","unstructured":"Lee H, Hwang SJ, Shin J (2020) Self-supervised label augmentation via input transformations. In: International conference on machine learning, pp 5714\u20135724. PMLR"},{"key":"11606_CR28","first-page":"7517","volume":"31","author":"X Zhu","year":"2018","unstructured":"Zhu X, Gong S et al (2018) Knowledge distillation by on-the-fly native ensemble. Adv Neural Inf Process Syst 31:7517\u20137527","journal-title":"Adv Neural Inf Process Syst"},{"key":"11606_CR29","doi-asserted-by":"crossref","unstructured":"Chen D, Mei J-P, Wang C, Feng Y, Chen C (2020) Online knowledge distillation with diverse peers. In: Proceedings of the AAAI conference on artificial intelligence, 34, pp 3430\u20133437","DOI":"10.1609\/aaai.v34i04.5746"},{"key":"11606_CR30","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.neucom.2020.07.048","volume":"415","author":"Y Liu","year":"2020","unstructured":"Liu Y, Zhang W, Wang J (2020) Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing 415:106\u2013113","journal-title":"Neurocomputing"},{"key":"11606_CR31","doi-asserted-by":"crossref","unstructured":"Guo Q, Wang X, Wu Y, Yu Z, Liang D, Hu X, Luo P (2020) Online knowledge distillation via collaborative learning. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, pp 11020\u201311029 . IEEE","DOI":"10.1109\/CVPR42600.2020.01103"},{"key":"11606_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.121884","volume":"238","author":"J Gou","year":"2024","unstructured":"Gou J, Hu Y, Sun L, Wang Z, Ma H (2024) Collaborative knowledge distillation via filter knowledge transfer. Expert Syst Appl 238:121884. https:\/\/doi.org\/10.1016\/j.eswa.2023.121884","journal-title":"Expert Syst Appl"},{"key":"11606_CR33","doi-asserted-by":"crossref","unstructured":"Zhang L, Song J, Gao A, Chen J, Bao C, Ma K (2019) Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 3713\u20133722","DOI":"10.1109\/ICCV.2019.00381"},{"key":"11606_CR34","unstructured":"Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp 562\u2013570. PMLR"},{"key":"11606_CR35","unstructured":"Zhang Z, Sabuncu M (2020) Self-distillation as instance-specific label smoothing. Adv Neural Inform Process Syst, vol. 33"},{"key":"11606_CR36","doi-asserted-by":"crossref","unstructured":"Ji M, Shin S, Hwang S, Park G, Moon I-C (2021) Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp. 10664\u201310673","DOI":"10.1109\/CVPR46437.2021.01052"},{"key":"11606_CR37","doi-asserted-by":"crossref","unstructured":"Xu G, Liu Z, Li X, Loy CC (2020) Knowledge distillation meets self-supervision. In: Proceedings of the European conference on computer vision, Springer. pp 588\u2013604","DOI":"10.1007\/978-3-030-58545-7_34"},{"key":"11606_CR38","doi-asserted-by":"crossref","unstructured":"Yang C, An Z, Cai L, Xu Y (2021) Hierarchical self-supervised augmented knowledge distillation. In: Proceedings of the international joint conference on artificial intelligence, pp 1217\u20131223","DOI":"10.24963\/ijcai.2021\/168"},{"key":"11606_CR39","doi-asserted-by":"crossref","unstructured":"Robbins H, Monro S (1951) A stochastic approximation method. The Annals Math Statist, 400\u2013407","DOI":"10.1214\/aoms\/1177729586"},{"key":"11606_CR40","doi-asserted-by":"crossref","unstructured":"Li H, Zhang H, Qi X, Yang R, Huang G (2019) Improved techniques for training adaptive deep networks. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 1891\u20131900. IEEE","DOI":"10.1109\/ICCV.2019.00198"},{"key":"11606_CR41","doi-asserted-by":"crossref","unstructured":"Li Z, Huang Y, Chen D, Luo T, Cai N, Pan Z (2020) Online knowledge distillation via multi-branch diversity enhancement. In: Proceedings of the Asian conference on computer vision","DOI":"10.1007\/978-3-030-69538-5_20"},{"key":"11606_CR42","doi-asserted-by":"crossref","unstructured":"Walawalkar D, Shen Z, Savvides M (2020) Online ensemble model compression using knowledge distillation. In: Proceedings of the European conference on computer vision, Springer, pp 18\u201335","DOI":"10.1007\/978-3-030-58529-7_2"},{"key":"11606_CR43","doi-asserted-by":"crossref","unstructured":"Sun D, Yao A, Zhou A, Zhao H (2019) Deeply-supervised knowledge synergy. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, IEEE, pp 6997\u20137006","DOI":"10.1109\/CVPR.2019.00716"},{"key":"11606_CR44","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, IEEE, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"11606_CR45","doi-asserted-by":"crossref","unstructured":"Zagoruyko S, Komodakis N (2016) Wide residual networks. In: Proceedings of the British machine vision conference . British Machine Vision Association","DOI":"10.5244\/C.30.87"},{"key":"11606_CR46","unstructured":"Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861"},{"key":"11606_CR47","doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, IEEE, pp 4700\u20134708","DOI":"10.1109\/CVPR.2017.243"},{"key":"11606_CR48","doi-asserted-by":"crossref","unstructured":"Lin T-Y, Doll\u00e1r P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, IEEE, pp 2117\u20132125","DOI":"10.1109\/CVPR.2017.106"},{"key":"11606_CR49","first-page":"1097","volume":"25","author":"A Krizhevsky","year":"2012","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst 25:1097\u20131105","journal-title":"Adv Neural Inform Process Syst"},{"key":"11606_CR50","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556"},{"key":"11606_CR51","doi-asserted-by":"crossref","unstructured":"Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, IEEE, pp 1\u20139","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"11606_CR52","volume-title":"Advances in neural information processing systems","author":"R Muller","year":"2019","unstructured":"Muller R, Kornblith S, Hinton GE (2019) When does label smoothing help? In: Wallach H, Larochelle H, Beygelzimer A, Alch\u00e9-Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems, vol 32. Curran Associates Inc, New York"},{"key":"11606_CR53","doi-asserted-by":"crossref","unstructured":"Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510\u20134520","DOI":"10.1109\/CVPR.2018.00474"},{"key":"11606_CR54","doi-asserted-by":"crossref","unstructured":"Lou Z, Ye X, Zhang L, Wu W, He Y, Zhou H (2023) Rethinking the value of local feature fusion in convolutional neural networks. Neural Process Lett, 1\u201316","DOI":"10.21203\/rs.3.rs-2229460\/v1"},{"key":"11606_CR55","unstructured":"Krizhevsky A, Hinton G, et al (2009) Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases"},{"key":"11606_CR56","first-page":"33716","volume":"35","author":"T Huang","year":"2022","unstructured":"Huang T, You S, Wang F, Qian C, Xu C (2022) Knowledge distillation from a stronger teacher. Adv Neural Inf Process Syst 35:33716\u201333727","journal-title":"Adv Neural Inf Process Syst"},{"key":"11606_CR57","doi-asserted-by":"crossref","unstructured":"Zhao B, Cui Q, Song R, Qiu Y, Liang J (2022) Decoupled knowledge distillation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 11953\u201311962","DOI":"10.1109\/CVPR52688.2022.01165"},{"key":"11606_CR58","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, IEEE, pp 248\u2013255","DOI":"10.1109\/CVPR.2009.5206848"}],"container-title":["Neural Processing Letters"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11606-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11063-024-11606-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11606-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,18]],"date-time":"2024-11-18T16:57:16Z","timestamp":1731949036000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11063-024-11606-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,10]]},"references-count":58,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["11606"],"URL":"https:\/\/doi.org\/10.1007\/s11063-024-11606-z","relation":{},"ISSN":["1573-773X"],"issn-type":[{"value":"1573-773X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,10]]},"assertion":[{"value":"21 March 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 May 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"172"}}