{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T12:28:07Z","timestamp":1768220887619,"version":"3.49.0"},"reference-count":47,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T00:00:00Z","timestamp":1768176000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62202435"],"award-info":[{"award-number":["62202435"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The success of large-scale deep learning models in remote sensing tasks has been transformative, enabling significant advances in image classification, object detection, and image\u2013text retrieval. However, their computational and memory demands pose challenges for deployment in resource-constrained environments. Knowledge distillation (KD) alleviates these issues by transferring knowledge from a strong teacher to a student model, which can be compact for efficient deployment or architecturally matched to improve accuracy under the same inference budget. In this paper, we introduce Hierarchical Multi-Segment Knowledge Distillation (HIMS_KD), a multi-stage framework that sequentially distills knowledge from a teacher into multiple assistant models specialized in low-, mid-, and high-level representations, and then aggregates their knowledge into the final student. We integrate feature-level alignment, auxiliary similarity-logit alignment, and supervised loss during distillation. Experiments on benchmark remote sensing datasets (RSITMD and RSICD) show that HIMS_KD improves retrieval performance and enhances zero-shot classification; and when a compact student is used, it reduces deployment cost while retaining strong accuracy.<\/jats:p>","DOI":"10.3390\/info17010070","type":"journal-article","created":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T09:13:01Z","timestamp":1768209181000},"page":"70","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Hierarchical Knowledge Distillation for Efficient Model Compression and Transfer: A Multi-Level Aggregation Approach"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7254-1787","authenticated-orcid":false,"given":"Titinunt","family":"Kitrungrotsakul","sequence":"first","affiliation":[{"name":"Research Center for Space Computing System, Zhejiang Lab, Hangzhou 311121, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8012-0405","authenticated-orcid":false,"given":"Preeyanuch","family":"Srichola","sequence":"additional","affiliation":[{"name":"Kasetsart Agricultural and Agro-Industrial Product Improvement Institute, Kasetsart University, Bangkok 10900, Thailand"},{"name":"Cellulose for Future Materials and Technologies Special Research Unit, Department of Biotechnology, Faculty of Agro-Industry, Kasetsart University, Bangkok 10900, Thailand"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/MGRS.2024.3383473","article-title":"Vision-Language Models in Remote Sensing: Current progress and future trends","volume":"12","author":"Li","year":"2024","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., and Chanussot, J. (2022). Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review. arXiv.","DOI":"10.1016\/j.jag.2022.102926"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","article-title":"Knowledge Distillation: A Survey","volume":"129","author":"Gou","year":"2021","journal-title":"Int. J. Comput. Vision"},{"key":"ref_4","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_5","unstructured":"Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. arXiv."},{"key":"ref_6","first-page":"5191","article-title":"Improved Knowledge Distillation via Teacher Assistant","volume":"34","author":"Mirzadeh","year":"2020","journal-title":"Proc. Aaai Conf. Artif. Intell."},{"key":"ref_7","unstructured":"Gu, Y., Dong, L., Wei, F., and Huang, M. (2024, January 7\u201311). MiniLLM: Knowledge Distillation of Large Language Models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna Austria."},{"key":"ref_8","first-page":"37193","article-title":"Segment Any Point Cloud Sequences by Distilling Vision Foundation Models","volume":"Volume 36","author":"Oh","year":"2023","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_9","first-page":"32938","article-title":"Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models","volume":"Volume 36","author":"Oh","year":"2023","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sun, X., Zhang, P., Zhang, P., Shah, H., Saenko, K., and Xia, X. (2023, January 2\u20136). DIME-FM: DIstilling Multimodal and Efficient Foundation Models. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Paris, France.","DOI":"10.1109\/ICCV51070.2023.01423"},{"key":"ref_11","first-page":"69719","article-title":"Module-wise Adaptive Distillation for Multimodality Foundation Models","volume":"Volume 36","author":"Oh","year":"2023","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., and Liu, Z. (2021, January 11\u201317). Compressing Visual-Linguistic Model via Knowledge Distillation. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00146"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wu, K., Peng, H., Zhou, Z., Xiao, B., Liu, M., Yuan, L., Xuan, H., Valenzuela, M., Chen, X.S., and Wang, X. (2023, January 2\u20136). Tinyclip: Clip distillation via affinity mimicking and weight inheritance. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.02008"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yang, C., An, Z., Huang, L., Bi, J., Yu, X., Yang, H., Diao, B., and Xu, Y. (2024, January 16\u201322). CLIP-KD: An Empirical Study of CLIP Model Distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01510"},{"key":"ref_15","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18\u201324). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhao, T., Guo, Y., and Yin, J. (2024). RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing. arXiv.","DOI":"10.1109\/TGRS.2024.3449154"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, Z., Prabha, R., Huang, T., Wu, J., and Rajagopal, R. (2023). SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing. arXiv.","DOI":"10.1609\/aaai.v38i6.28393"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Muhtar, D., Li, Z., Gu, F., Zhang, X., and Xiao, P. (2024). LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model. arXiv.","DOI":"10.1007\/978-3-031-72904-1_26"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhan, Y., Xiong, Z., and Yuan, Y. (2024). SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model. arXiv.","DOI":"10.1016\/j.isprsjprs.2025.01.020"},{"key":"ref_20","unstructured":"Hu, Y., Yuan, J., Wen, C., Lu, X., and Li, X. (2023). RSGPT: A Remote Sensing Vision Language Model and Benchmark. arXiv."},{"key":"ref_21","first-page":"5622216","article-title":"RemoteCLIP: A Vision Language Foundation Model for Remote Sensing","volume":"62","author":"Liu","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","first-page":"5917820","article-title":"EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain","volume":"62","author":"Zhang","year":"2024","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kuckreja, K., Danish, M.S., Naseer, M., Das, A., Khan, S., and Khan, F.S. (2024, January 16\u201322). GeoChat: Grounded Large Vision-Language Model for Remote Sensing. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02629"},{"key":"ref_24","first-page":"28541","article-title":"LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day","volume":"Volume 36","author":"Oh","year":"2023","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Son, W., Na, J., Choi, J., and Hwang, W. (2021, January 11\u201317). Densely Guided Knowledge Distillation Using Multiple Teacher Assistants. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00926"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Liang, H., Liu, Y., Wang, H., and Jia, Z. (2023, January 19\u201325). Teacher Assistant-Based Knowledge Distillation Extracting Multi-level Features on Single Channel Sleep EEG. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI, Macao, SAR, China.","DOI":"10.24963\/ijcai.2023\/439"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3965","DOI":"10.1109\/TGRS.2017.2685945","article-title":"AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification","volume":"55","author":"Xia","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2217","DOI":"10.1109\/JSTARS.2019.2918242","article-title":"EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification","volume":"12","author":"Helber","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Christie, G., Fendley, N., Wilson, J., and Mukherjee, R. (2018, January 18\u201323). Functional Map of the World. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00646"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4205","DOI":"10.1109\/JSTARS.2021.3070368","article-title":"On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances, and Million-AID","volume":"14","author":"Long","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.isprsjprs.2018.01.004","article-title":"PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval","volume":"145","author":"Zhou","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1865","DOI":"10.1109\/JPROC.2017.2675998","article-title":"Remote Sensing Image Scene Classification: Benchmark and State of the Art","volume":"105","author":"Cheng","year":"2017","journal-title":"Proc. IEEE"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, H., Dou, X., Tao, C., Wu, Z., Chen, J., Peng, J., Deng, M., and Zhao, L. (2020). RSI-CB: A Large-Scale Remote Sensing Image Classification Benchmark Using Crowdsourced Data. Sensors, 20.","DOI":"10.3390\/s20061594"},{"key":"ref_34","unstructured":"Faghri, F., Fleet, D.J., Kiros, J.R., and Fidler, S. (2018). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). Stacked Cross Attention for Image-Text Matching. Proceedings of the Computer Vision\u2014ECCV 2018, Munich, Germany, 8\u201314 September 2018, Springer.","DOI":"10.1007\/978-3-030-01249-6"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, T., Xu, X., Yang, Y., Hanjalic, A., Shen, H.T., and Song, J. (2019, January 21\u201325). Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking. Proceedings of the 27th ACM International Conference on Multimedia (MM \u201919), Nice, France.","DOI":"10.1145\/3343031.3350875"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"4462","DOI":"10.1109\/JSTARS.2020.3013818","article-title":"Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective","volume":"13","author":"Hoxha","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Rahhal, M.M.A., Bazi, Y., Abdullah, T., Mekhalfi, M.L., and Zuair, M. (2020). Deep Unsupervised Embedding for Remote Sensing Image Retrieval Using Textual Cues. Appl. Sci., 10.","DOI":"10.3390\/app10248931"},{"key":"ref_39","first-page":"5620616","article-title":"Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information","volume":"60","author":"Yuan","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"812","DOI":"10.1109\/JSTARS.2022.3231851","article-title":"Text-Image Matching for Cross-Modal Remote Sensing Image Retrieval via Graph Neural Network","volume":"16","author":"Yu","year":"2023","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1109\/JSTARS.2022.3226325","article-title":"Hypergraph-Enhanced Textual-Visual Matching Network for Cross-Modal Remote Sensing Image Retrieval via Dynamic Hypergraph Learning","volume":"16","author":"Yao","year":"2023","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"5621815","DOI":"10.1109\/TGRS.2023.3333375","article-title":"Hypersphere-Based Remote Sensing Cross-Modal Text\u2013Image Retrieval via Curriculum Learning","volume":"61","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"3359","DOI":"10.1080\/01431161.2022.2091964","article-title":"A fusion-based contrastive learning model for cross-modal remote sensing retrieval","volume":"43","author":"Li","year":"2022","journal-title":"Int. J. Remote Sens."},{"key":"ref_44","first-page":"4704014","article-title":"Direction-Oriented Visual\u2013Semantic Embedding Model for Remote Sensing Image\u2013Text Retrieval","volume":"62","author":"Ma","year":"2024","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_45","unstructured":"Pan, J., Ma, Q., and Bai, C. (November, January 29). A Prior Instruction Representation Framework for Remote Sensing Image-text Retrieval. Proceedings of the 31st ACM International Conference on Multimedia (MM \u201923), Ottawa, ON, Canada."},{"key":"ref_46","first-page":"1","article-title":"Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval","volume":"60","author":"Yuan","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2183","DOI":"10.1109\/TGRS.2017.2776321","article-title":"Exploring models and data for remote sensing image caption generation","volume":"56","author":"Lu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/1\/70\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T09:30:03Z","timestamp":1768210203000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/1\/70"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,12]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["info17010070"],"URL":"https:\/\/doi.org\/10.3390\/info17010070","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,12]]}}}