{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T00:07:18Z","timestamp":1769818038558,"version":"3.49.0"},"reference-count":11,"publisher":"SAGE Publications","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IFS"],"published-print":{"date-parts":[[2022,2,2]]},"abstract":"<jats:p>The prime focus of knowledge distillation (KD) seeks a light proxy termed student to mimic the outputs of its heavy neural networks termed teacher, and makes the student run real-time on the resource-limited devices. This paradigm requires aligning the soft logits of both teacher and student. However, few doubts whether the process of softening the logits truly give full play to the teacher-student paradigm. In this paper, we launch several analyses to delve into this issue from scratch. Subsequently, several simple yet effective functions are devised to replace the vanilla KD. The ultimate function can be an effective alternative to its original counterparts and work well with other skills like FitNets. To claim this point, we conduct several visual tasks on individual benchmarks, and experimental results verify the potential of our proposed function in terms of performance gains. For example, when the teacher and student networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, our proposed method achieves 40.88%top-1 error rate on Tiny ImageNet.<\/jats:p>","DOI":"10.3233\/jifs-211549","type":"journal-article","created":{"date-parts":[[2021,9,28]],"date-time":"2021-09-28T13:19:48Z","timestamp":1632835188000},"page":"2247-2259","source":"Crossref","is-referenced-by-count":0,"title":["Modifying the softening process for\u00a0knowledge distillation"],"prefix":"10.1177","volume":"42","author":[{"given":"Chao","family":"Tan","sequence":"first","affiliation":[{"name":"Science and Parallel and Distributed Processing Laboratory, National University of Defense Technology, Changsha, China"},{"name":"Laboratory of Software Engineering for Complex Systems, National University of Defense Technology, Changsha, China"}]},{"given":"Jie","family":"Liu","sequence":"additional","affiliation":[{"name":"Science and Parallel and Distributed Processing Laboratory, National University of Defense Technology, Changsha, China"},{"name":"Laboratory of Software Engineering for Complex Systems, National University of Defense Technology, Changsha, China"}]}],"member":"179","reference":[{"key":"10.3233\/JIFS-211549_ref2","doi-asserted-by":"crossref","unstructured":"Bertinetto L. , Valmadre J. , Henriques J.F. , Vedaldi A. and Torr P.H.S. , Fully-convolutional siamese networks for object tracking, In European conference on computer vision, 850\u2013865. Springer, (2016).","DOI":"10.1007\/978-3-319-48881-3_56"},{"issue":"1","key":"10.3233\/JIFS-211549_ref8","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1007\/s11263-014-0733-5","article-title":"The pascal visual object classes challenge: A retrospective","volume":"111","author":"Everingham","year":"2015","journal-title":"International Journal of Computer Vision"},{"key":"10.3233\/JIFS-211549_ref13","doi-asserted-by":"crossref","first-page":"3771","DOI":"10.1609\/aaai.v33i01.33013771","article-title":"Knowledge distillation with adversarial samples supporting decision boundary, In","volume":"33","author":"Heo","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"10.3233\/JIFS-211549_ref17","unstructured":"Le Y. and Yang X. , Tiny imagenet visual recognition challenge, CS 231N 7 (2015)."},{"key":"10.3233\/JIFS-211549_ref18","doi-asserted-by":"crossref","unstructured":"Lee S.H. , Kim D.H. and Song B.C. , Selfsupervised knowledge distillation using singular value decomposition, In European Conference on Computer Vision, pages 339\u2013354. Springer, (2018).","DOI":"10.1007\/978-3-030-01231-1_21"},{"issue":"Nov","key":"10.3233\/JIFS-211549_ref24","first-page":"2579","article-title":"Visualizing data using t-sne","volume":"9","author":"van der Maaten","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"10.3233\/JIFS-211549_ref33","doi-asserted-by":"crossref","unstructured":"Ristani E. , Solera F. , Zou R. , Cucchiara R. and Tomasi C. , Performance measures and a data set for multi-target, multicamera tracking, In European Conference on Computer Vision pages 17\u201335. Springer, (2016).","DOI":"10.1007\/978-3-319-48881-3_2"},{"key":"10.3233\/JIFS-211549_ref36","doi-asserted-by":"crossref","first-page":"106837","DOI":"10.1016\/j.knosys.2021.106837","article-title":"Improving knowledge distillation via an expressive teacher","volume":"218","author":"Tan","year":"2021","journal-title":"Knowledge-Based Systems"},{"key":"10.3233\/JIFS-211549_ref39","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.neucom.2021.04.102","article-title":"Preparing lessons: Improve knowledge distillation with better supervision","volume":"454","author":"Wen","year":"2021","journal-title":"Neurocomputing"},{"key":"10.3233\/JIFS-211549_ref40","doi-asserted-by":"crossref","unstructured":"Xu G. , Liu Z. , Li X. and Loy C.C. , Knowledge distillation meets self-supervision, In European Conference on Computer Vision, pages 588\u2013604. Springer, (2020).","DOI":"10.1007\/978-3-030-58545-7_34"},{"key":"10.3233\/JIFS-211549_ref41","doi-asserted-by":"crossref","first-page":"5628","DOI":"10.1609\/aaai.v33i01.33015628","article-title":"Training deep neural networks in generations: A more tolerant teacher educates better students","volume":"33","author":"Yang","year":"2019","journal-title":"In, Proceedings of the AAAI Conference on Artificial Intelligence"}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/JIFS-211549","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T12:54:43Z","timestamp":1769777683000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/JIFS-211549"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,2]]},"references-count":11,"journal-issue":{"issue":"3"},"URL":"https:\/\/doi.org\/10.3233\/jifs-211549","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,2]]}}}