{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:14:05Z","timestamp":1760238845386,"version":"build-2065373602"},"reference-count":48,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2020,8,26]],"date-time":"2020-08-26T00:00:00Z","timestamp":1598400000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["41701508","61725105"],"award-info":[{"award-number":["41701508","61725105"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The deep convolutional neural network has made significant progress in cloud detection. However, the compromise between having a compact model and high accuracy has always been a challenging task in cloud detection for large-scale remote sensing imagery. A promising method to tackle this problem is knowledge distillation, which usually lets the compact model mimic the cumbersome model\u2019s output to get better generalization. However, vanilla knowledge distillation methods cannot properly distill the characteristics of clouds in remote sensing images. In this paper, we propose a novel self-attention knowledge distillation approach for compact and accurate cloud detection, named Bidirectional Self-Attention Distillation (Bi-SAD). Bi-SAD lets a model learn from itself without adding additional parameters or supervision. With bidirectional layer-wise features learning, the model can get a better representation of the cloud\u2019s textural information and semantic information, so that the cloud\u2019s boundaries become more detailed and the predictions become more reliable. Experiments on a dataset acquired by GaoFen-1 satellite show that our Bi-SAD has a great balance between compactness and accuracy, and outperforms vanilla distillation methods. Compared with state-of-the-art cloud detection models, the parameter size and FLOPs are reduced by 100 times and 400 times, respectively, with a small drop in accuracy.<\/jats:p>","DOI":"10.3390\/rs12172770","type":"journal-article","created":{"date-parts":[[2020,8,26]],"date-time":"2020-08-26T09:05:37Z","timestamp":1598432737000},"page":"2770","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Compact Cloud Detection with Bidirectional Self-Attention Knowledge Distillation"],"prefix":"10.3390","volume":"12","author":[{"given":"Yajie","family":"Chai","sequence":"first","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kun","family":"Fu","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xian","family":"Sun","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenhui","family":"Diao","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiyuan","family":"Yan","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yingchao","family":"Feng","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Wang","sequence":"additional","affiliation":[{"name":"Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"Key Laboratory of Network Information System Technology (NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1016\/j.isprsjprs.2020.01.025","article-title":"Rotation-aware and multi-scale convolutional neural network for object detection in remote sensing images","volume":"161","author":"Fu","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3377","DOI":"10.1109\/TGRS.2019.2954328","article-title":"FMSSD: Feature-Merged Single-Shot Detection for Multiscale Objects in Large-Scale Remote Sensing Imagery","volume":"58","author":"Wang","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"doi-asserted-by":"crossref","unstructured":"Zhang, Y., Rossow, W.B., Lacis, A.A., Oinas, V., and Mishchenko, M.I. (2004). Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and other global data sets: Refinements of the radiative transfer model and the input data. J. Geophys. Res. Atmos., 109.","key":"ref_3","DOI":"10.1029\/2003JD004457"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2845","DOI":"10.1080\/01431169608949110","article-title":"Cloud detection from thermal infrared images using a segmentation technique","volume":"17","author":"Shin","year":"1996","journal-title":"Int. J. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1080\/01431168908903929","article-title":"An algorithm for snow and ice detection using AVHRR data An extension to the APOLLO software package","volume":"10","author":"Gesell","year":"1989","journal-title":"Int. J. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1705","DOI":"10.1109\/TGRS.2008.916208","article-title":"Spatial and temporal varying thresholds for cloud detection in GOES imagery","volume":"46","author":"Jedlovec","year":"2008","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.rse.2011.10.028","article-title":"Object-based cloud and cloud shadow detection in Landsat imagery","volume":"118","author":"Zhu","year":"2012","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1016\/j.rse.2017.01.026","article-title":"Multi-feature combined cloud and cloud shadow detection in GaoFen-1 wide field of view imagery","volume":"191","author":"Li","year":"2017","journal-title":"Remote Sens. Environ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1117\/12.410358","article-title":"Landsat 7 automatic cloud cover assessment","volume":"Volume 4049","author":"Irish","year":"2000","journal-title":"Algorithms for Multispectral, Hyperspectral, and Ultraspectral Imagery VI"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"776","DOI":"10.3390\/rs6010776","article-title":"Cloud and cloud-shadow detection in SPOT5 HRG imagery with automated morphological feature extraction","volume":"6","author":"Fisher","year":"2014","journal-title":"Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2662","DOI":"10.1109\/TGRS.2003.815404","article-title":"Multifeature texture analysis for the classification of clouds in satellite imagery","volume":"41","author":"Christodoulou","year":"2003","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5100","DOI":"10.1109\/TGRS.2013.2286834","article-title":"A class of cloud detection algorithms based on a MAP-MRF approach in space and time","volume":"52","author":"Vivone","year":"2013","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4913","DOI":"10.1109\/TGRS.2017.2692264","article-title":"A Novel Bayesian Spatial\u2013Temporal Random Field Model Applied to Cloud Detection From Remotely Sensed Imagery","volume":"55","author":"Xu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3273","DOI":"10.1109\/TGRS.2012.2223217","article-title":"A hybrid cloud detection algorithm to improve MODIS sea surface temperature data quality and coverage over the Eastern Gulf of Mexico","volume":"51","author":"Barnes","year":"2012","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.neucom.2014.09.102","article-title":"A cloud image detection method based on SVM vector machine","volume":"169","author":"Li","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4907","DOI":"10.3390\/rs6064907","article-title":"Automated detection of cloud and cloud shadow in single-date Landsat imagery using neural networks and spatial post-processing","volume":"6","author":"Hughes","year":"2014","journal-title":"Remote Sens."},{"doi-asserted-by":"crossref","unstructured":"Bai, T., Li, D., Sun, K., Chen, Y., and Li, W. (2016). Cloud detection for high-resolution satellite imagery using machine learning and multi-feature fusion. Remote Sens., 8.","key":"ref_17","DOI":"10.3390\/rs8090715"},{"doi-asserted-by":"crossref","unstructured":"Tan, K., Zhang, Y., and Tong, X. (2016). Cloud extraction from chinese high resolution satellite imagery by probabilistic latent semantic analysis and object-based machine learning. Remote Sens., 8.","key":"ref_18","DOI":"10.3390\/rs8110963"},{"unstructured":"Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv.","key":"ref_19"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1109\/TPAMI.2017.2699184","article-title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","volume":"40","author":"Chen","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"unstructured":"Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv.","key":"ref_21"},{"doi-asserted-by":"crossref","unstructured":"Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8\u201314). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","key":"ref_22","DOI":"10.1007\/978-3-030-01234-2_49"},{"doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21\u201326). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","key":"ref_23","DOI":"10.1109\/CVPR.2017.660"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1600","DOI":"10.1109\/LGRS.2018.2846802","article-title":"Cloud and cloud shadow detection using multilevel feature fused segmentation network","volume":"15","author":"Yan","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"4062","DOI":"10.1109\/TGRS.2018.2889677","article-title":"Cloud Detection in Remote Sensing Images Based on Multiscale Features-Convolutional Neural Network","volume":"57","author":"Shao","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3631","DOI":"10.1109\/JSTARS.2017.2686488","article-title":"Multilevel cloud detection in remote sensing images based on deep learning","volume":"10","author":"Xie","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"6195","DOI":"10.1109\/TGRS.2019.2904868","article-title":"CDnet: CNN-Based Cloud Detection for Remote Sensing Imagery","volume":"57","author":"Yang","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/j.rse.2019.03.007","article-title":"Cloud and cloud shadow detection in Landsat imagery based on deep convolutional neural networks","volume":"225","author":"Chai","year":"2019","journal-title":"Remote Sens. Environ."},{"doi-asserted-by":"crossref","unstructured":"Shao, Z., Deng, J., Wang, L., Fan, Y., Sumari, N.S., and Cheng, Q. (2017). Fuzzy autoencode based cloud detection for remote sensing imagery. Remote Sens., 9.","key":"ref_29","DOI":"10.3390\/rs9040311"},{"doi-asserted-by":"crossref","unstructured":"Jiang, H., and Lu, N. (2018). Multi-scale residual convolutional neural network for haze removal of remote sensing images. Remote Sens., 10.","key":"ref_30","DOI":"10.3390\/rs10060945"},{"doi-asserted-by":"crossref","unstructured":"Zhou, K., Ming, D., Lv, X., Fang, J., and Wang, M. (2019). CNN-Based Land Cover Classification Combining Stratified Segmentation and Fusion of Point Cloud and Very High-Spatial Resolution Remote Sensing Image Data. Remote Sens., 11.","key":"ref_31","DOI":"10.3390\/rs11172065"},{"doi-asserted-by":"crossref","unstructured":"Francis, A., Sidiropoulos, P., and Muller, J.P. (2019). CloudFCN: Accurate and robust cloud detection for satellite imagery with deep learning. Remote Sens., 11.","key":"ref_32","DOI":"10.3390\/rs11192312"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"111446","DOI":"10.1016\/j.rse.2019.111446","article-title":"Cloud detection algorithm for multi-modal satellite imagery using convolutional neural-networks (CNN)","volume":"237","author":"Li","year":"2020","journal-title":"Remote Sens. Environ."},{"doi-asserted-by":"crossref","unstructured":"Ghorbanzadeh, O., and Blaschke, T. (2019, January 3\u20135). Optimizing Sample Patches Selection of CNN to Improve the mIOU on Landslide Detection. Proceedings of the International Conference on Geographical Information Systems Theory, Applications and Management, Heraklion, Crete, Greece.","key":"ref_34","DOI":"10.5220\/0007675300330040"},{"doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","key":"ref_35","DOI":"10.1109\/CVPR.2016.90"},{"unstructured":"Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E. (2016). Enet: A deep neural network architecture for real-time semantic segmentation. arXiv.","key":"ref_36"},{"unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.","key":"ref_37"},{"unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.","key":"ref_38"},{"unstructured":"Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv.","key":"ref_39"},{"unstructured":"Xie, J., Shuai, B., Hu, J.F., Lin, J., and Zheng, W.S. (2018). Improving fast segmentation with teacher-student learning. arXiv.","key":"ref_40"},{"doi-asserted-by":"crossref","unstructured":"Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., and Wang, J. (2019, January 16\u201320). Structured Knowledge Distillation for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","key":"ref_41","DOI":"10.1109\/CVPR.2019.00271"},{"unstructured":"Zagoruyko, S., and Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv.","key":"ref_42"},{"doi-asserted-by":"crossref","unstructured":"Hou, Y., Ma, Z., Liu, C., and Loy, C.C. (2019, January 15\u201318). Learning lightweight lane detection cnns by self attention distillation. Proceedings of the IEEE International Conference on Computer Vision, Jeju Island, Korea.","key":"ref_43","DOI":"10.1109\/ICCV.2019.00110"},{"unstructured":"Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., and Anandkumar, A. (2018). Born again neural networks. arXiv.","key":"ref_44"},{"doi-asserted-by":"crossref","unstructured":"Yim, J., Joo, D., Bae, J., and Kim, J. (2017, January 21\u201326). A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","key":"ref_45","DOI":"10.1109\/CVPR.2017.754"},{"unstructured":"Bagherinezhad, H., Horton, M., Rastegari, M., and Farhadi, A. (2018). Label refinery: Improving imagenet classification through label progression. arXiv.","key":"ref_46"},{"doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","key":"ref_47","DOI":"10.1109\/CVPR.2015.7298965"},{"doi-asserted-by":"crossref","unstructured":"Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., and Garcia-Rodriguez, J. (2017). A review on deep learning techniques applied to semantic segmentation. arXiv.","key":"ref_48","DOI":"10.1016\/j.asoc.2018.05.018"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/17\/2770\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:07:02Z","timestamp":1760177222000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/17\/2770"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,26]]},"references-count":48,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["rs12172770"],"URL":"https:\/\/doi.org\/10.3390\/rs12172770","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2020,8,26]]}}}