{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T03:47:44Z","timestamp":1747194464857,"version":"3.40.5"},"reference-count":29,"publisher":"SAGE Publications","issue":"11","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61976107, 61502208"],"award-info":[{"award-number":["61976107, 61502208"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["International Journal of Distributed Sensor Networks"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:p> Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher\u2019s knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature\/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods. <\/jats:p>","DOI":"10.1177\/15501477211057037","type":"journal-article","created":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T21:53:32Z","timestamp":1638309212000},"page":"155014772110570","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Feature fusion-based collaborative learning for knowledge distillation"],"prefix":"10.1177","volume":"17","author":[{"given":"Yiting","family":"Li","sequence":"first","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, China"}]},{"given":"Liyuan","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8438-7286","authenticated-orcid":false,"given":"Jianping","family":"Gou","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, China"}]},{"given":"Lan","family":"Du","sequence":"additional","affiliation":[{"name":"Department of Data Science and Artificial Intelligence, Faculty of Information Technology, Monash University, Clayton, VIC, Australia"}]},{"given":"Weihua","family":"Ou","sequence":"additional","affiliation":[{"name":"School of Big Data and Computer Science, Guizhou Normal University, Guiyang, China"}]}],"member":"179","published-online":{"date-parts":[[2021,11,30]]},"reference":[{"first-page":"12115","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","author":"Jung S","key":"bibr1-15501477211057037"},{"issue":"4","key":"bibr2-15501477211057037","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1049\/trit.2020.0031","volume":"5","author":"Aljunid MF","year":"2020","journal-title":"CAAI Trans Intell Technol"},{"first-page":"3971","volume-title":"Proceedings of the 30th international joint conference on artificial intelligence (IJCAI)","author":"Wu X","key":"bibr3-15501477211057037"},{"first-page":"1135","volume-title":"Proceedings of the 28th international conference on neural information processing systems (NIPS)","author":"Han S","key":"bibr4-15501477211057037"},{"volume-title":"Proceedings of the international conference on learning representations (ICLR)","author":"Li H","key":"bibr5-15501477211057037"},{"first-page":"658","volume-title":"Proceedings of the IEEE international conference on computer vision (ICCV)","author":"Wen W","key":"bibr6-15501477211057037"},{"first-page":"1135","volume-title":"Proceedings of the international conference on pattern recognition and machine intelligence (PReMI)","author":"Grachev AM","key":"bibr7-15501477211057037"},{"key":"bibr8-15501477211057037","unstructured":"Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network, https:\/\/arxiv.org\/abs\/1503.02531"},{"volume-title":"Proceedings of the international conference on learning representations (ICLR)","author":"Romero A","key":"bibr9-15501477211057037"},{"first-page":"7028","volume-title":"Proceedings of the 35th AAAI conference on artificial intelligence: a virtual conference","author":"Chen D","key":"bibr10-15501477211057037"},{"first-page":"2541","volume-title":"Proceedings of the 35th AAAI conference on artificial intelligence: a virtual conference","author":"Shen C","key":"bibr11-15501477211057037"},{"volume-title":"Proceedings of the European conference on artificial life (ECAL)","author":"Jafari A","key":"bibr12-15501477211057037"},{"first-page":"4320","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (CVPR)","author":"Zhang Y","key":"bibr13-15501477211057037"},{"first-page":"11017","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","author":"Guo Q","key":"bibr14-15501477211057037"},{"first-page":"502","volume-title":"Proceedings of the IEEE international conference on computer vision (ICCV)","author":"Hou S","key":"bibr15-15501477211057037"},{"first-page":"4619","volume-title":"Proceedings of the international conference on pattern recognition (ICPR)","author":"Kim J","key":"bibr16-15501477211057037"},{"first-page":"122","volume-title":"Proceedings of the European conference on computer vision (ECCV)","author":"Ma N","key":"bibr17-15501477211057037"},{"issue":"6","key":"bibr18-15501477211057037","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","volume":"129","author":"Gou J","year":"2021","journal-title":"Int J Comput Vision"},{"first-page":"3909","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","author":"Yuan L","key":"bibr19-15501477211057037"},{"first-page":"3712","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision (ICCV)","author":"Zhang L","key":"bibr20-15501477211057037"},{"first-page":"1355","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision (ICCV)","author":"Phuong M","key":"bibr21-15501477211057037"},{"first-page":"13873","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (CVPR)","author":"Yun S","key":"bibr22-15501477211057037"},{"first-page":"6794","volume-title":"Proceedings of the 35th AAAI conference on artificial intelligence: a virtual conference","author":"Boo Y","key":"bibr23-15501477211057037"},{"first-page":"1","volume-title":"Proceedings of the 35th AAAI conference on artificial intelligence: a virtual conference","author":"Wu G","key":"bibr24-15501477211057037"},{"first-page":"7517","volume-title":"Proceedings of the 32nd international conference on neural information processing systems (NIPS)","author":"Lan X","key":"bibr25-15501477211057037"},{"first-page":"7028","volume-title":"Proceedings of the 35th AAAI conference on artificial intelligence: a virtual conference","author":"Chen D","key":"bibr26-15501477211057037"},{"key":"bibr27-15501477211057037","unstructured":"Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. Master\u2019s thesis, Department of Computer Science, University of Toronto, 2009, Citeseer."},{"first-page":"770","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","author":"He K","key":"bibr28-15501477211057037"},{"volume-title":"Proceedings of the British machine vision conference (BMVC)","author":"Zagoruyko S","key":"bibr29-15501477211057037"}],"container-title":["International Journal of Distributed Sensor Networks"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/15501477211057037","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/15501477211057037","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/15501477211057037","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T21:54:00Z","timestamp":1638309240000},"score":1,"resource":{"primary":{"URL":"http:\/\/journals.sagepub.com\/doi\/10.1177\/15501477211057037"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":29,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["10.1177\/15501477211057037"],"URL":"https:\/\/doi.org\/10.1177\/15501477211057037","relation":{},"ISSN":["1550-1477","1550-1477"],"issn-type":[{"type":"print","value":"1550-1477"},{"type":"electronic","value":"1550-1477"}],"subject":[],"published":{"date-parts":[[2021,11]]}}}