{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,11]],"date-time":"2026-01-11T12:04:08Z","timestamp":1768133048003,"version":"3.49.0"},"reference-count":46,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62376282"],"award-info":[{"award-number":["62376282"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Most logit-based knowledge distillation methods transfer soft labels from the teacher model to the student model via Kullback\u2013Leibler divergence based on softmax, an exponential normalization function. However, this exponential nature of softmax tends to prioritize the largest class (target class) while neglecting smaller ones (non-target classes), leading to an oversight of the non-target classes\u2019s significance. To address this issue, we propose Non-Target-Class-Enhanced Knowledge Distillation (NTCE-KD) to amplify the role of non-target classes both in terms of magnitude and diversity. Specifically, we present a magnitude-enhanced Kullback\u2013Leibler (MKL) divergence multi-shrinking the target class to enhance the impact of non-target classes in terms of magnitude. Additionally, to enrich the diversity of non-target classes, we introduce a diversity-based data augmentation strategy (DDA), further enhancing overall performance. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate that non-target classes are of great significance and that our method achieves state-of-the-art performance across a wide range of teacher\u2013student pairs.<\/jats:p>","DOI":"10.3390\/s24113617","type":"journal-article","created":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T12:19:46Z","timestamp":1717417186000},"page":"3617","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["NTCE-KD: Non-Target-Class-Enhanced Knowledge Distillation"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-8041-2538","authenticated-orcid":false,"given":"Chuan","family":"Li","sequence":"first","affiliation":[{"name":"College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8012-2088","authenticated-orcid":false,"given":"Xiao","family":"Teng","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4741-517X","authenticated-orcid":false,"given":"Yan","family":"Ding","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4238-8985","authenticated-orcid":false,"given":"Long","family":"Lan","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8\u201314). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01264-9_8"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201323). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_4","first-page":"7132","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"4585","DOI":"10.1109\/TIP.2018.2843129","article-title":"Interacting Tracklets for Multi-Object Tracking","volume":"27","author":"Lan","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3911","DOI":"10.1007\/s10115-020-01488-7","article-title":"Learning Sequence-to-Sequence Affinity Metric for Near-Online Multi-Object Tracking","volume":"62","author":"Feng","year":"2021","journal-title":"Knowl. Inf. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21\u201326). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.660"},{"key":"ref_10","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv."},{"key":"ref_11","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning multiple layers of features from tiny images. Handb. Syst. Autoimmun. Dis., 1."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_13","first-page":"1","article-title":"Fitnets: Hints for thin deep nets","volume":"2","author":"Adriana","year":"2015","journal-title":"Proc. ICLR"},{"key":"ref_14","unstructured":"Komodakis, N., and Zagoruyko, S. (2017, January 24\u201326). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. Proceedings of the ICLR, Toulon, France."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, P., Liu, S., Zhao, H., and Jia, J. (2021, January 20\u201325). Distilling knowledge via knowledge review. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00497"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yim, J., Joo, D., Bae, J., and Kim, J. (2017, January 21\u201326). A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.754"},{"key":"ref_17","first-page":"3830","article-title":"Asymmetric temperature scaling makes larger networks teach well again","volume":"35","author":"Li","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jafari, A., Rezagholizadeh, M., Sharma, P., and Ghodsi, A. (2021, January 19\u201323). Annealing Knowledge Distillation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online.","DOI":"10.18653\/v1\/2021.eacl-main.212"},{"key":"ref_19","unstructured":"Li, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo, L., Li, J., and Yang, J. (2023, January 20\u201327). Curriculum temperature for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jin, Y., Wang, J., and Lin, D. (2023, January 17\u201324). Multi-Level Logit Distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.02325"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, J. (2022, January 18\u201324). Decoupled knowledge distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01165"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q.V. (2018). Autoaugment: Learning augmentation policies from data. arXiv.","DOI":"10.1109\/CVPR.2019.00020"},{"key":"ref_23","first-page":"6665","article-title":"Fast autoaugment","volume":"32","author":"Lim","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_24","unstructured":"Ho, D., Liang, E., Chen, X., Stoica, I., and Abbeel, P. (2019, January 9\u201315). Population based augmentation: Efficient learning of augmentation policy schedules. Proceedings of the International Conference on Machine Learning. PMLR, Long Beach, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cubuk, E.D., Zoph, B., Shlens, J., and Le, Q.V. (2020, January 14\u201319). Randaugment: Practical automated data augmentation with a reduced search space. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00359"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zagoruyko, S., and Komodakis, N. (2016). Wide residual networks. arXiv.","DOI":"10.5244\/C.30.87"},{"key":"ref_27","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18\u201322). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00716"},{"key":"ref_29","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201323). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhao, B., Cui, Q., Song, R., and Liang, J. (2023). DOT: A Distillation-Oriented Trainer. arXiv.","DOI":"10.1109\/ICCV51070.2023.00569"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Sun, S., Ren, W., Li, J., Wang, R., and Cao, X. (2024). Logit Standardization in Knowledge Distillation. arXiv.","DOI":"10.1109\/CVPR52733.2024.01489"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Park, W., Kim, D., Lu, Y., and Cho, M. (2019, January 15\u201320). Relational knowledge distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00409"},{"key":"ref_34","unstructured":"Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., and Choi, J.Y. (November, January 27). A comprehensive overhaul of feature distillation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_35","unstructured":"Tian, Y., Krishnan, D., and Isola, P. (2020, January 26\u201330). Contrastive Representation Distillation. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Guo, Z., Yan, H., Li, H., and Lin, X. (2023, January 17\u201324). Class Attention Transfer Based Knowledge Distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01142"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1016\/j.neunet.2023.11.003","article-title":"Enhancement, integration, expansion: Activating representation of detailed features for occluded person re-identification","volume":"169","author":"Ning","year":"2024","journal-title":"Neural Netw."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Somers, V., De Vleeschouwer, C., and Alahi, A. (2023, January 2\u20137). Body part-based representation learning for occluded person re-identification. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV56688.2023.00166"},{"key":"ref_39","unstructured":"Han, K., Gong, S., Huang, Y., Wang, L., and Tan, T. (2023, January 17\u201324). Clothing-change feature augmentation for person re-identification. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3338","DOI":"10.1109\/TIP.2023.3278860","article-title":"Learning to Purification for Unsupervised Person Re-Identification","volume":"32","author":"Lan","year":"2023","journal-title":"IEEE Trans. Image Process."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Teng, X., Lan, L., Zhao, J., Li, X., and Tang, Y. (2023). Highly Efficient Active Learning with Tracklet-Aware Co-Cooperative Annotators for Person Re-Identification. IEEE Trans. Neural Networks Learn. Syst., 1\u201314.","DOI":"10.1109\/TNNLS.2023.3289178"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lin, H., Zheng, X., Li, L., Chao, F., Wang, S., Wang, Y., Tian, Y., and Ji, R. (2023, January 17\u201324). Meta architecture for point cloud analysis. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01696"},{"key":"ref_43","first-page":"5703215","article-title":"Learning discriminative features by covering local geometric space for point cloud analysis","volume":"60","author":"Wang","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_44","first-page":"1962","article-title":"3d point cloud classification method based on dynamic coverage of local area","volume":"34","author":"Wang","year":"2022","journal-title":"J. Softw."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Luo, H., Gu, Y., Liao, X., Lai, S., and Jiang, W. (2019, January 15\u201320). Bag of Tricks and A Strong Baseline for Deep Person Re-identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00190"},{"key":"ref_46","unstructured":"Ma, X., Qin, C., You, H., Ran, H., and Fu, Y. (2022). Rethinking network design and local geometry in point cloud: A simple residual MLP framework. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/11\/3617\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:53:11Z","timestamp":1760107991000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/11\/3617"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":46,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["s24113617"],"URL":"https:\/\/doi.org\/10.3390\/s24113617","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,3]]}}}