{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T03:28:10Z","timestamp":1784086090284,"version":"3.55.0"},"reference-count":77,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T00:00:00Z","timestamp":1682380800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T00:00:00Z","timestamp":1682380800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61976107"],"award-info":[{"award-number":["61976107"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61502208"],"award-info":[{"award-number":["61502208"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Vis"],"published-print":{"date-parts":[[2023,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher\u2019s ability in unfolding the knowledge to be distilled, but also improve the student\u2019s capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.\n<\/jats:p>","DOI":"10.1007\/s11263-023-01792-z","type":"journal-article","created":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T17:07:25Z","timestamp":1682442445000},"page":"1857-1874","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":48,"title":["Multi-target Knowledge Distillation via Student Self-reflection"],"prefix":"10.1007","volume":"131","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1413-0693","authenticated-orcid":false,"given":"Jianping","family":"Gou","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiangshuo","family":"Xiong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baosheng","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lan","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yibing","family":"Zhan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dacheng","family":"Tao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,4,25]]},"reference":[{"key":"1792_CR1","doi-asserted-by":"crossref","unstructured":"Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019). Variational information distillation for knowledge transfer. In CVPR (pp. 9163\u20139171).","DOI":"10.1109\/CVPR.2019.00938"},{"key":"1792_CR2","doi-asserted-by":"crossref","unstructured":"Chen, S., Hong, Z., Xie, G. S., Yang, W., Peng, Q., Wang, K., Zhao, J., & You, X. (2022). MSDN: Mutually semantic distillation network for zero-shot learning. In: CVPR (pp. 7612\u20137621).","DOI":"10.1109\/CVPR52688.2022.00746"},{"key":"1792_CR3","doi-asserted-by":"crossref","unstructured":"Chen, W., Li, S., Huang, C., Yu, Y., Jiang, Y., & Dong, J. (2022). Mutual Distillation Learning Network for Trajectory-User Linking. In: IJCAI.","DOI":"10.24963\/ijcai.2022\/274"},{"key":"1792_CR4","doi-asserted-by":"crossref","unstructured":"Chen, P., Liu, S., Zhao, H., & Jia, J.(2021). Distilling knowledge via knowledge review. In: CVPR (pp. 5008\u20135017).","DOI":"10.1109\/CVPR46437.2021.00497"},{"key":"1792_CR5","doi-asserted-by":"crossref","unstructured":"Chen, D., Mei, J. P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022). Knowledge distillation with the reused teacher classifier. In: CVPR (pp. 11933-11942).","DOI":"10.1109\/CVPR52688.2022.01163"},{"key":"1792_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106550","volume":"211","author":"J Chen","year":"2021","unstructured":"Chen, J., Chen, Y., Li, W., Ning, G., Tong, M., & Hilton, A. (2021). Channel and spatial attention based deep object co-segmentation. Knowledge-Based Systems, 211, 106550.","journal-title":"Knowledge-Based Systems"},{"key":"1792_CR7","unstructured":"Chennupati, S., Kamani, M. M., Cheng, Z., & Chen, L. (2021). Adaptive distillation: Aggregating knowledge from multiple paths for efficient distillation. arXiv preprint arXiv:2110.09674."},{"key":"1792_CR8","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hier-archical image database. In: CVPR (pp. 248\u2013255).","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"2","key":"1792_CR9","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","volume":"88","author":"M Everingham","year":"2010","unstructured":"Everingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2), 303\u2013338.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR10","doi-asserted-by":"crossref","unstructured":"Fan, S., Cheng, X., Wang, X., Yang, C., Deng, P., Liu, M., Deng J., & Liu, M. (2022). Channel Self-Supervision for Online Knowledge Distillation. arXiv preprint arXiv: 2203.11660.","DOI":"10.1109\/ICCC56324.2022.10065706"},{"key":"1792_CR11","doi-asserted-by":"crossref","unstructured":"Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021). Compressing visual-linguistic model via knowledge distillation. In: ICCV (pp. 1428\u20131438).","DOI":"10.1109\/ICCV48922.2021.00146"},{"key":"1792_CR12","doi-asserted-by":"publisher","first-page":"2610","DOI":"10.1109\/TIP.2019.2950508","volume":"29","author":"S Ge","year":"2019","unstructured":"Ge, S., Luo, Z., Zhang, C., Hua, Y., & Tao, D. (2019). Distilling channels for efficient deep tracking. IEEE Transactions on Image Processing, 29, 2610\u20132621.","journal-title":"IEEE Transactions on Image Processing"},{"key":"1792_CR13","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2022.3212733","author":"J Gou","year":"2022","unstructured":"Gou, J., Sun, L., Yu, B., Du, L., Ramamohanarao, K., & Tao, D. (2022). Collaborative knowledge distillation via multiknowledge transfer. IEEE Transactions on Neural Networks and Learning Systems. https:\/\/doi.org\/10.1109\/TNNLS.2022.3212733","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"issue":"6","key":"1792_CR14","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","volume":"129","author":"J Gou","year":"2021","unstructured":"Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129(6), 1789\u20131819.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR15","unstructured":"Guo, S., Alvarez, J. M., & Salzmann, M. (2021). Distilling image classifiers in object detectors. In NeurIPS (vol. 34, pp. 1036\u20131047)."},{"key":"1792_CR16","unstructured":"Hagstr\u00f6m, L., & Johansson, R. (2021). Knowledge distillation for swedish ner models: A search for performance and efficiency. Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa) (pp. 124\u2013134)."},{"key":"1792_CR17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR (pp. 770\u2013778).","DOI":"10.1109\/CVPR.2016.90"},{"key":"1792_CR18","doi-asserted-by":"crossref","unstructured":"Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019). A comprehensive overhaul of feature distillation. In ICCV (pp. 1921-1930).","DOI":"10.1109\/ICCV.2019.00201"},{"key":"1792_CR19","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-022-01728-z","author":"Z He","year":"2022","unstructured":"He, Z., Zhang, L., Gao, X., & Zhang, D. (2022). Multi-adversarial faster-RCNN with paradigm teacher for unrestricted object detection. International Journal of Computer Vision. https:\/\/doi.org\/10.1007\/s11263-022-01728-z","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR20","unstructured":"Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531."},{"key":"1792_CR21","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In CVPR (pp. 7132\u20137141).","DOI":"10.1109\/CVPR.2018.00745"},{"key":"1792_CR22","doi-asserted-by":"crossref","unstructured":"Huang, Y., Wu, J., Xu, X., & Ding, S. (2022). Evaluation-oriented knowledge distillation for deep face recognition. In CVPR (pp. 18740-18749).","DOI":"10.1109\/CVPR52688.2022.01818"},{"key":"1792_CR23","doi-asserted-by":"publisher","first-page":"1364","DOI":"10.1109\/TIP.2022.3141255","volume":"31","author":"Z Huang","year":"2022","unstructured":"Huang, Z., Yang, S., Zhou, M., Li, Z., Gong, Z., & Chen, Y. (2022). Feature map distillation of thin nets for low-resolution object recognition. IEEE Transactions on Image Processing, 31, 1364\u20131379.","journal-title":"IEEE Transactions on Image Processing"},{"key":"1792_CR24","doi-asserted-by":"crossref","unstructured":"Jafari, A., Rezagholizadeh, M., Sharma, P., & Ghodsi, A. (2021). Annealing knowledge distillation. arXiv preprint arXiv: 2104.07163.","DOI":"10.18653\/v1\/2021.eacl-main.212"},{"key":"1792_CR25","doi-asserted-by":"crossref","unstructured":"Ji, M., Shin, S., Hwang, S., Park, G., & Moon, I. C. (2021). Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In CVPR (pp. 10664\u201310673).","DOI":"10.1109\/CVPR46437.2021.01052"},{"issue":"10","key":"1792_CR26","doi-asserted-by":"publisher","first-page":"2385","DOI":"10.1007\/s11263-022-01651-3","volume":"130","author":"G Kordopatis-Zilos","year":"2022","unstructured":"Kordopatis-Zilos, G., Tzelepis, C., Papadopoulos, S., Kompatsiaris, I., & Patras, I. (2022). DnS: Distill-and-select for efficient and accurate video indexing and retrieval. International Journal of Computer Vision, 130(10), 2385\u20132407.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR27","unstructured":"Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical Report."},{"key":"1792_CR28","doi-asserted-by":"crossref","unstructured":"Li, B., Chen, B., Wang, Y., Dai, T., Hu, M., Jiang, Y., & Xia, S. (2021). Knowledge distillation via channel correlation structure. In: International conference on knowledge science, engineering and management (pp. 357\u2013368).","DOI":"10.1007\/978-3-030-82136-4_29"},{"key":"1792_CR29","doi-asserted-by":"crossref","unstructured":"Li, J., Liu, X., Zhang, S., Yang, M., Xu, R., & Qin, F. (2021). Accelerating neural architecture search for natural language processing with knowledge distillation and earth mover\u2019s distance. In ACM SIGIR (pp. 2091\u20132095).","DOI":"10.1145\/3404835.3463017"},{"key":"1792_CR30","doi-asserted-by":"crossref","unstructured":"Li, C., Peng, J., Yuan, L., Wang, G., Liang, X., Lin, L., & Chang, X. (2020). Block-wisely supervised neural architecture search with knowledge distillation. In CVPR (pp. 1989\u20131998).","DOI":"10.1109\/CVPR42600.2020.00206"},{"key":"1792_CR31","unstructured":"Li, Z., Ye, J., Song, M., Huang, Y., & Pan, Z. (2018). Online knowledge distillation for efficient pose estimation. In ICCV (pp. 11740\u201311750)."},{"key":"1792_CR32","doi-asserted-by":"crossref","unstructured":"Liu, L., Huang, Q., Lin, S., Xie, H., Wang, B., Chang, X., & Liang, X. (2021). Exploring inter-channel correlation for diversity-preserved knowledge distillation. In ICCV (pp. 8271\u20138280).","DOI":"10.1109\/ICCV48922.2021.00816"},{"key":"1792_CR33","doi-asserted-by":"crossref","unstructured":"Liu, W., Zhou, P., Wang, Z., Zhao, Z., Deng, H., & Ju, Q. (2020). Fastbert: A self-distilling bert with adaptive inference time. In ACL (pp. 6035\u20136044).","DOI":"10.18653\/v1\/2020.acl-main.537"},{"key":"1792_CR34","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3001940","author":"Y Liu","year":"2020","unstructured":"Liu, Y., Shu, C., Wang, J., & Shen, C. (2020). Structured knowledge distillation for dense prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence. https:\/\/doi.org\/10.1109\/TPAMI.2020.3001940","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"1792_CR35","doi-asserted-by":"crossref","unstructured":"Lou, A., & Loew, M.(2021). Cfpnet: Channel-wise feature pyramid for real-time semantic segmentation. In ICIP (pp. 1894\u20131898).","DOI":"10.1109\/ICIP42928.2021.9506485"},{"key":"1792_CR36","unstructured":"Ma, Z., Luo, G., Gao, J., Li, L., Chen, Y., Wang, S., Zhang, C., & Hu, W. (2022). Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In CVPR (pp. 14074\u201314083)."},{"key":"1792_CR37","doi-asserted-by":"crossref","unstructured":"Mirzadeh, S. I., Farajtabar, M., Li, A., & Ghasemzadeh, H. (2020). Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. In AAAI (pp. 5191\u20135198).","DOI":"10.1609\/aaai.v34i04.5963"},{"key":"1792_CR38","unstructured":"Mobahi, H., Farajtabar, M., & Bartlett, P. (2020). Self-distillation amplifies regularization in hilbert space. In NeurIPS (pp. 3351\u20133361)."},{"key":"1792_CR39","unstructured":"Muhammad, A., Zhou, F., Xie, C., Li, J., Bae, S. H., & Li, Z. (2021). MixACM: Mixup-based robustness transfer via distillation of activated channel maps. In NeurIPS (pp. 4555\u20134569)."},{"key":"1792_CR40","unstructured":"Park, D. Y., Cha, M. H., Kim, D., & Han, B. (2021). Learning student-friendly teacher networks for knowledge distillation. arXiv preprint arXiv: 2102.07650."},{"issue":"4","key":"1792_CR41","doi-asserted-by":"publisher","first-page":"921","DOI":"10.1007\/s11263-020-01392-1","volume":"129","author":"Y Peng","year":"2021","unstructured":"Peng, Y., Qi, J., Ye, Z., & Zhuo, Y. (2021). Hierarchical visual-textual knowledge distillation for life-long correlation learning. International Journal of Computer Vision, 129(4), 921\u2013941.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR42","doi-asserted-by":"crossref","unstructured":"Phan, M. H., Phung, S. L., Tran-Thanh, L., & Bouzerdoum, A. (2022). Class similarity weighted knowledge distillation for continual semantic segmentation. In CVPR (pp. 16866\u201316875).","DOI":"10.1109\/CVPR52688.2022.01636"},{"key":"1792_CR43","doi-asserted-by":"crossref","unstructured":"Phuong, M., & Lampert C. H. (2019). Distillation-based training for multi-exit architectures. In ICCV (pp. 1355\u20131364).","DOI":"10.1109\/ICCV.2019.00144"},{"key":"1792_CR44","doi-asserted-by":"crossref","unstructured":"Qu, Y., Deng, W., & Hu, J.: H-at. (2020). Hybrid attention transfer for knowledge distillation. In PRCV (pp. 249\u2013260).","DOI":"10.1007\/978-3-030-60636-7_21"},{"key":"1792_CR45","doi-asserted-by":"crossref","unstructured":"Shen, Y., Xu, L., Yang, Y., Li, Y., & Guo, Y. (2022). Self-Distillation from the last mini-batch for consistency regularization. In CVPR (pp. 11943\u201311952).","DOI":"10.1109\/CVPR52688.2022.01164"},{"key":"1792_CR46","doi-asserted-by":"crossref","unstructured":"Shu, C., Liu, Y., Gao, J., Yan, Z., & Shen, C. (2021). Channel-wise knowledge distillation for dense prediction. In ICCV (pp. 5311\u20135320).","DOI":"10.1109\/ICCV48922.2021.00526"},{"key":"1792_CR47","doi-asserted-by":"crossref","unstructured":"Sun, D., Yao, A., Zhou, A., & Zhao, H. (2019). Deeply-supervised knowledge synergy. In CVPR (pp. 6997\u20137006).","DOI":"10.1109\/CVPR.2019.00716"},{"key":"1792_CR48","doi-asserted-by":"crossref","unstructured":"Tung, F., & Mori, G. (2019). Similarity-preserving knowledge distillation. In ICCV (pp. 1365\u20131374).","DOI":"10.1109\/ICCV.2019.00145"},{"issue":"11","key":"1792_CR49","first-page":"2579","volume":"9","author":"L Van der Maaten","year":"2008","unstructured":"Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2579\u20132605.","journal-title":"Journal of Machine Learning Research"},{"key":"1792_CR50","unstructured":"Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The caltech-ucsd birds-200-2011 dataset."},{"key":"1792_CR51","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhou, W., Jiang, T., Bai, X., & Xu, Y. (2020). Intra-class feature variation distillation for semantic segmentation. In ICCV (pp. 346\u2013362).","DOI":"10.1007\/978-3-030-58571-6_21"},{"issue":"10","key":"1792_CR52","doi-asserted-by":"publisher","first-page":"3349","DOI":"10.1109\/TPAMI.2020.2983686","volume":"43","author":"J Wang","year":"2020","unstructured":"Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., & Xiao, B. (2020). Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10), 3349\u20133364.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"6","key":"1792_CR53","doi-asserted-by":"publisher","first-page":"3048","DOI":"10.1109\/TPAMI.2021.3055564","volume":"44","author":"L Wang","year":"2022","unstructured":"Wang, L., & Yoon, K. J. (2022). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6), 3048\u20133068.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"1792_CR54","doi-asserted-by":"crossref","unstructured":"Wu, G., & Gong, S. (2021). Peer collaborative learning for online knowledge distillation. In AAAI (pp. 10302\u201310310).","DOI":"10.1609\/aaai.v35i12.17234"},{"issue":"8","key":"1792_CR55","doi-asserted-by":"publisher","first-page":"2089","DOI":"10.1007\/s11263-019-01286-x","volume":"128","author":"X Wu","year":"2020","unstructured":"Wu, X., He, R., Hu, Y., & Sun, Z. (2020). Learning an evolutionary embedding via massive knowledge distillation. International Journal of Computer Vision, 128(8), 2089\u20132106.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR56","doi-asserted-by":"crossref","unstructured":"Xu, J., Huang, S., Zhou, F., Huangfu, L., Zeng, D., & Liu, B. (2022). Boosting multi-label image classification with complementary parallel self-distillation. In IJCAI.","DOI":"10.24963\/ijcai.2022\/208"},{"key":"1792_CR57","unstructured":"Yan, H., Zhang, J., Niu, G., Feng, J., Tan, V., & Sugiyama, M. (2021). Improving adversarial robustness of cnns via channel-wise importance-based feature selection. In ICML (pp. 11693\u201311703)."},{"key":"1792_CR58","doi-asserted-by":"crossref","unstructured":"Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., & Yuan, C. (2022). Focal and global knowledge distillation for detectors. In CVPR (pp. 4643\u20134652).","DOI":"10.1109\/CVPR52688.2022.00460"},{"key":"1792_CR59","doi-asserted-by":"crossref","unstructured":"Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., & Zhang, Q. (2022). Cross-image relational knowledge distillation for semantic segmentation. In CVPR (pp. 12319\u201312328).","DOI":"10.1109\/CVPR52688.2022.01200"},{"key":"1792_CR60","doi-asserted-by":"crossref","unstructured":"Yim, J., Joo, D., Bae, J., & Kim, J. (2017). A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR (pp. 4133\u20134141).","DOI":"10.1109\/CVPR.2017.754"},{"key":"1792_CR61","doi-asserted-by":"crossref","unstructured":"You, C., Chen, N., & Zou, Y.(2021). Knowledge distillation for improved accuracy in spoken question answering. In ICASSP (pp. 7793\u20137797).","DOI":"10.1109\/ICASSP39728.2021.9414999"},{"key":"1792_CR62","doi-asserted-by":"crossref","unstructured":"Yuan, F., Shou, L., Pei, J., Lin, W., Gong, M., Fu, Y., & Jiang, D. (2021). Reinforced multi-teacher selection for knowledge distillation. In AAAI.","DOI":"10.1609\/aaai.v35i16.17680"},{"key":"1792_CR63","doi-asserted-by":"crossref","unstructured":"Yuan, L., Tay, F. E., Li, G., Wang, T., & Feng, J. (2020). Revisiting knowledge distillation via label smoothing regularization. In CVPR (pp. 3903\u20133911).","DOI":"10.1109\/CVPR42600.2020.00396"},{"issue":"11","key":"1792_CR64","doi-asserted-by":"publisher","first-page":"8276","DOI":"10.1109\/TPAMI.2021.3103980","volume":"44","author":"B Yu","year":"2021","unstructured":"Yu, B., & Tao, D. (2021). Heatmap regression via randomized rounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8276\u20138289.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"1792_CR65","unstructured":"Zagoruyko, S., & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR (pp. 1\u201313)."},{"key":"1792_CR66","unstructured":"Zhang, S., Liu, H., Hopcroft, J. E., & He, K. (2022). Class-aware Information for Logit-based Knowledge Distillation. arXiv preprint arXiv:2211.14773."},{"key":"1792_CR67","unstructured":"Zhang, L., Shi, Y., Shi, Z., Ma, K., & Bao, C. (2020). Task-oriented feature distillation. In NeurIPS (pp. 14759\u201314771)."},{"key":"1792_CR68","doi-asserted-by":"crossref","unstructured":"Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019). Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV (pp. 3713\u20133722).","DOI":"10.1109\/ICCV.2019.00381"},{"key":"1792_CR69","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018). Deep mutual learning. In CVPR (pp. 4320\u20134328).","DOI":"10.1109\/CVPR.2018.00454"},{"issue":"8","key":"1792_CR70","first-page":"4388","volume":"44","author":"L Zhang","year":"2022","unstructured":"Zhang, L., Bao, C., & Ma, K. (2022). Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8), 4388\u20134403.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"1792_CR71","doi-asserted-by":"crossref","unstructured":"Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022). Decoupled knowledge distillation. In CVPR (pp. 11953\u201311962).","DOI":"10.1109\/CVPR52688.2022.01165"},{"key":"1792_CR72","doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. In CVPR (pp. 2881\u20132890).","DOI":"10.1109\/CVPR.2017.660"},{"issue":"8","key":"1792_CR73","doi-asserted-by":"publisher","first-page":"2474","DOI":"10.1007\/s11263-021-01473-9","volume":"129","author":"T Zhao","year":"2021","unstructured":"Zhao, T., Han, J., Yang, L., Wang, B., & Zhang, D. (2021). SODA: Weakly supervised temporal action localization based on astute background response and self-distillation learning. International Journal of Computer Vision, 129(8), 2474\u20132498.","journal-title":"International Journal of Computer Vision"},{"key":"1792_CR74","doi-asserted-by":"crossref","unstructured":"Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., & Tian, Q. (2015). Scalable person reidentification: A benchmark. In ICCV (pp. 1116\u20131124).","DOI":"10.1109\/ICCV.2015.133"},{"key":"1792_CR75","unstructured":"Zhou, Z., Zhuge, C., Guan, X., & Liu, W. (2006). Channel distillation: Channel-wise attention for knowledge distillation. arXiv preprint arXiv: 2006.01683"},{"key":"1792_CR76","unstructured":"Zhu, X., & Gong, S. (2018). Knowledge distillation by on-the-fly native ensemble. In NeurIPS (pp. 7517\u20137527)."},{"key":"1792_CR77","doi-asserted-by":"crossref","unstructured":"Zhu, Y., & Wang, Y.(2021). Student customized knowledge distillation: Bridging the gap between student and teacher. In ICCV (pp. 5057\u20135066).","DOI":"10.1109\/ICCV48922.2021.00501"}],"container-title":["International Journal of Computer Vision"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-023-01792-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11263-023-01792-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-023-01792-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,17]],"date-time":"2023-06-17T09:10:57Z","timestamp":1686993057000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11263-023-01792-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,25]]},"references-count":77,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2023,7]]}},"alternative-id":["1792"],"URL":"https:\/\/doi.org\/10.1007\/s11263-023-01792-z","relation":{},"ISSN":["0920-5691","1573-1405"],"issn-type":[{"value":"0920-5691","type":"print"},{"value":"1573-1405","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,25]]},"assertion":[{"value":"11 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 March 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}