{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T00:36:07Z","timestamp":1760229367189,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,6,9]],"date-time":"2022-06-09T00:00:00Z","timestamp":1654732800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["62071136","62002083","61971153","61801142","LBH-Q20085","LBH-Z20051"],"award-info":[{"award-number":["62071136","62002083","61971153","61801142","LBH-Q20085","LBH-Z20051"]}]},{"name":"Heilongjiang Postdoctoral Foundation","award":["62071136","62002083","61971153","61801142","LBH-Q20085","LBH-Z20051"],"award-info":[{"award-number":["62071136","62002083","61971153","61801142","LBH-Q20085","LBH-Z20051"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Object tracking based on RGB images may fail when the color of the tracked object is similar to that of the background. Hyperspectral images with rich spectral features can provide more information for RGB-based trackers. However, there is no fusion tracking algorithm based on hyperspectral and RGB images. In this paper, we propose a reliability-guided aggregation network (RANet) for hyperspectral and RGB tracking, which guides the combination of hyperspectral information and RGB information through modality reliability to improve tracking performance. Specifically, a dual branch based on the Transformer Tracking (TransT) structure is constructed to obtain the information of hyperspectral and RGB modalities. Then, a classification response aggregation module is designed to combine the different modality information by fusing the response predicted through the classification head. Finally, the reliability of different modalities is also considered in the aggregation module to guide the aggregation of the different modality information. Massive experimental results on the public dataset composed of hyperspectral and RGB image sequences show that the performance of the tracker based on our fusion method is better than that of the corresponding single-modality tracker, which fully proves the effectiveness of the fusion method. Among them, the RANet tracker based on the TransT tracker achieves the best performance accuracy of 0.709, indicating the effectiveness and superiority of the RANet tracker.<\/jats:p>","DOI":"10.3390\/rs14122765","type":"journal-article","created":{"date-parts":[[2022,6,12]],"date-time":"2022-06-12T23:55:24Z","timestamp":1655078124000},"page":"2765","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking"],"prefix":"10.3390","volume":"14","author":[{"given":"Chunhui","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China"},{"name":"Key Laboratory of Advanced Marine Communication and Information Technology, Ministry of Industry and Information Technology, Harbin Engineering University, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongjiao","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China"},{"name":"Key Laboratory of Advanced Marine Communication and Information Technology, Ministry of Industry and Information Technology, Harbin Engineering University, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9601-536X","authenticated-orcid":false,"given":"Nan","family":"Su","sequence":"additional","affiliation":[{"name":"College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China"},{"name":"Key Laboratory of Advanced Marine Communication and Information Technology, Ministry of Industry and Information Technology, Harbin Engineering University, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lu","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China"},{"name":"Key Laboratory of Advanced Marine Communication and Information Technology, Ministry of Industry and Information Technology, Harbin Engineering University, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0751-7726","authenticated-orcid":false,"given":"Yiming","family":"Yan","sequence":"additional","affiliation":[{"name":"College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China"},{"name":"Key Laboratory of Advanced Marine Communication and Information Technology, Ministry of Industry and Information Technology, Harbin Engineering University, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1182","DOI":"10.1109\/TGRS.2020.2998782","article-title":"Simultaneous Detection and Tracking of Moving-Target Shadows in ViSAR Imagery","volume":"59","author":"Tian","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.1109\/TGRS.2014.2369060","article-title":"Moving Target Tracking in Single- and Multichannel SAR","volume":"53","author":"Henke","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","first-page":"1","article-title":"An Enhanced SiamMask Network for Coastal Ship Tracking","volume":"60","author":"Yang","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5064","DOI":"10.1109\/TGRS.2011.2158005","article-title":"Motion Tracking of Discontinuous Sea Ice","volume":"49","author":"Thomas","year":"2011","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1074","DOI":"10.1109\/TGRS.2019.2943366","article-title":"Object Tracking in Satellite Videos by Improved Correlation Filters with Motion Estimations","volume":"58","author":"Xuan","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"7010","DOI":"10.1109\/TGRS.2020.2978512","article-title":"Small Target Tracking in Satellite Videos Using Background Compensation","volume":"58","author":"Wang","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1812","DOI":"10.1109\/TIP.2020.3045630","article-title":"Exploring the Effects of Blur and Deblurring to Visual Object Tracking","volume":"30","author":"Guo","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1109\/TIP.2020.3036723","article-title":"Siamese Regression Tracking with Reinforced Template Updating","volume":"30","author":"Zhao","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1109\/TPAMI.2014.2345390","article-title":"High-Speed Tracking with Kernelized Correlation Filters","volume":"37","author":"Henriques","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"8719","DOI":"10.1109\/TGRS.2019.2922648","article-title":"Can We Track Targets From Space? A Hybrid Kernel Correlation Filter Tracker for Satellite Video","volume":"57","author":"Shao","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_11","first-page":"1","article-title":"Onboard Real-Time Aerial Tracking with Efficient Siamese Anchor Proposal Network","volume":"60","author":"Fu","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dasari, M.M., and Gorthi, R.K.S.S. (2020, January 25\u201328). IOU\u2014Siamtrack: IOU Guided Siamese Network For Visual Object Tracking. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/ICIP40778.2020.9191188"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sodhro, A.H., Sennersten, C., and Ahmad, A. (2022). Towards Cognitive Authentication for Smart Healthcare Applications. Sensors, 22.","DOI":"10.3390\/s22062101"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Danelljan, M., Khan, F.S., Felsberg, M., and Van De Weijer, J. (2014, January 23\u201328). Adaptive Color Attributes for Real-Time Visual Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.143"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7860","DOI":"10.1109\/TGRS.2019.2916953","article-title":"Tracking Objects From Satellite Videos: A Velocity Feature Based Correlation Filter","volume":"57","author":"Shao","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cen, M., and Jung, C. (2018, January 7\u201310). Fully Convolutional Siamese Fusion Networks for Object Tracking. Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451102"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3056","DOI":"10.1109\/TIP.2020.3045634","article-title":"HRSiam: High-Resolution Siamese Network, Towards Space-Borne Satellite Video Tracking","volume":"30","author":"Shao","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.1109\/TIP.2019.2942506","article-title":"DP-Siam: Dynamic Policy Siamese Network for Robust Object Tracking","volume":"29","author":"Abdelpakey","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Li, B., Yan, J., Wu, W., Zhu, Z., and Hu, X. (2018, January 18\u201322). High Performance Visual Tracking with Siamese Region Proposal Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00935"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Guo, D., Wang, J., Cui, Y., Wang, Z., and Chen, S. (2020, January 14\u201319). SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.","DOI":"10.1109\/CVPR42600.2020.00630"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Xu, Y., Wang, Z., Li, Z., Yuan, Y., and Yu, G. (2020, January 7\u201312). SiamFC++: Towards robust and accurate visual tracking with target estimation guidelines. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6944"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., and Lu, H. (2021, January 19\u201325). Transformer tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.","DOI":"10.1109\/CVPR46437.2021.00803"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lan, X., Zhang, W., Zhang, S., Jain, D.K., and Zhou, H. (2019). Robust Multi-modality Anchor Graph-based Label Prediction for RGB-Infrared Tracking. IEEE Trans. Ind. Inf., 1.","DOI":"10.1109\/TII.2019.2947293"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"6044","DOI":"10.1109\/TGRS.2020.3010826","article-title":"Target-Constrained Interference-Minimized Band Selection for Hyperspectral Target Detection","volume":"59","author":"Shang","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2020.3040273","article-title":"Feedback Attention-Based Dense CNN for Hyperspectral Image Classification","volume":"60","author":"Yu","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1205","DOI":"10.1109\/TGRS.2018.2865197","article-title":"Hyperspectral Image Denoising Employing a Spatial\u2013Spectral Deep Residual Convolutional Neural Network","volume":"57","author":"Yuan","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"6690","DOI":"10.1109\/TGRS.2019.2907932","article-title":"Deep Learning for Hyperspectral Image Classification: An Overview","volume":"57","author":"Li","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3719","DOI":"10.1109\/TIP.2020.2965302","article-title":"Material Based Object Tracking in Hyperspectral Videos","volume":"29","author":"Xiong","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Qian, K., Du, J., and Zhou, H. (2021, January 24\u201326). Multi-Features Integration Based Hyperspectral Videos Tracker. Proceedings of the Workshop Hyperspectral Image Signal Procesing: Evolution in Remote Sensing (WHISPERS), Amsterdam, The Netherlands.","DOI":"10.1109\/WHISPERS52202.2021.9484029"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, Z., Xiong, F., Zhou, J., Wang, J., Lu, J., and Qian, Y. (2020, January 25\u201328). BAE-Net: A Band Attention Aware Ensemble Network for Hyperspectral Object Tracking. Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/ICIP40778.2020.9191105"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Li, Z., Ye, X., Xiong, F., Lu, J., Zhou, J., and Qian, Y. (2021, January 24\u201326). Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking. Proceedings of the Workshop Hyperspectral Image Signal Processing: Evolution in Remote Sensing (WHISPERS), Amsterdam, The Netherlands.","DOI":"10.1109\/WHISPERS52202.2021.9484032"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2672","DOI":"10.1109\/TNNLS.2018.2885616","article-title":"Learning a Low Tensor-Train Rank Representation for Hyperspectral Image Super-Resolution","volume":"30","author":"Dian","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"9887","DOI":"10.1109\/TIE.2019.2898618","article-title":"Learning Modality-Consistency Feature Templates: A Robust RGB-Infrared Tracking System","volume":"66","author":"Lan","year":"2019","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"115756","DOI":"10.1016\/j.image.2019.115756","article-title":"DSiamMFT: An RGB-T fusion tracking method via dynamic Siamese networks using multi-layer feature fusion","volume":"84","author":"Zhang","year":"2020","journal-title":"Signal Process. Image Commun."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"122122","DOI":"10.1109\/ACCESS.2019.2936914","article-title":"SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_36","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_37","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, I., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Guo, D., Shao, Y., Cui, Y., Wang, Z., Zhang, L., and Shen, C. (2021, January 19\u201325). Graph Attention Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.","DOI":"10.1109\/CVPR46437.2021.00942"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Chen, Z., Zhong, B., Li, G., Zhang, S., and Ji, R. (2020, January 14\u201319). Siamese Box Adaptive Network for Visual Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.","DOI":"10.1109\/CVPR42600.2020.00670"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Danelljan, M., Bhat, G., Khan, F.S., and Felsberg, M. (2017, January 21\u201326). ECO: Efficient Convolution Operators for Tracking. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.733"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zhang, Z., and Peng, H. (2019, January 16\u201320). Deeper and Wider Siamese Networks for Real-Time Visual Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00472"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1561","DOI":"10.1109\/TPAMI.2016.2609928","article-title":"Discriminative Scale Space Tracking","volume":"39","author":"Danelljan","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the 13th European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"M\u00fcller, M., Bibi, A., Giancola, S., Al-Subaihi, S., and Ghanem, B. (2018, January 8\u201314). TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild. Proceedings of the 15th European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01246-5_19"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., and Ling, H. (2019, January 16\u201320). LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00552"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1562","DOI":"10.1109\/TPAMI.2019.2957464","article-title":"GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild","volume":"43","author":"Huang","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wu, Y., Lim, J., and Yang, M.H. (2013, January 23\u201328). Online Object Tracking: A Benchmark. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.312"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/12\/2765\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:26:49Z","timestamp":1760138809000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/12\/2765"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,9]]},"references-count":47,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["rs14122765"],"URL":"https:\/\/doi.org\/10.3390\/rs14122765","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2022,6,9]]}}}