{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T02:54:25Z","timestamp":1794538465563,"version":"4.9.1"},"reference-count":45,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,10]],"date-time":"2020-01-10T00:00:00Z","timestamp":1578614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No.61602018"],"award-info":[{"award-number":["No.61602018"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No.61531006"],"award-info":[{"award-number":["No.61531006"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No.61971016"],"award-info":[{"award-number":["No.61971016"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No.61701011"],"award-info":[{"award-number":["No.61701011"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Beijing Municipal Natural Science Foundation Cooperation Beijing Education Committee","award":["No. KZ 201810005002"],"award-info":[{"award-number":["No. KZ 201810005002"]}]},{"name":"Beijing Municipal Natural Science Foundation Cooperation Beijing Education Committee","award":["No. KZ 201910005007"],"award-info":[{"award-number":["No. KZ 201910005007"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Object tracking in RGB-thermal (RGB-T) videos is increasingly used in many fields due to the all-weather and all-day working capability of the dual-modality imaging system, as well as the rapid development of low-cost and miniaturized infrared camera technology. However, it is still very challenging to effectively fuse dual-modality information to build a robust RGB-T tracker. In this paper, an RGB-T object tracking algorithm based on a modal-aware attention network and competitive learning (MaCNet) is proposed, which includes a feature extraction network, modal-aware attention network, and classification network. The feature extraction network adopts the form of a two-stream network to extract features from each modality image. The modal-aware attention network integrates the original data, establishes an attention model that characterizes the importance of different feature layers, and then guides the feature fusion to enhance the information interaction between modalities. The classification network constructs a modality-egoistic loss function through three parallel binary classifiers acting on the RGB branch, the thermal infrared branch, and the fusion branch, respectively. Guided by the training strategy of competitive learning, the entire network is fine-tuned in the direction of the optimal fusion of the dual modalities. Extensive experiments on several publicly available RGB-T datasets show that our tracker has superior performance compared to other latest RGB-T and RGB tracking approaches.<\/jats:p>","DOI":"10.3390\/s20020393","type":"journal-article","created":{"date-parts":[[2020,1,10]],"date-time":"2020-01-10T04:06:51Z","timestamp":1578629211000},"page":"393","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":138,"title":["Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8012-4684","authenticated-orcid":false,"given":"Hui","family":"Zhang","sequence":"first","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"},{"name":"Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5732-2973","authenticated-orcid":false,"given":"Lei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9937-2669","authenticated-orcid":false,"given":"Li","family":"Zhuo","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"},{"name":"Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1290-0738","authenticated-orcid":false,"given":"Jing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"},{"name":"Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing 100124, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2096","DOI":"10.1109\/TPAMI.2015.2509974","article-title":"Struck: Structured Output Tracking with Kernels","volume":"38","author":"Hare","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, C., Zhu, C., Huang, Y., Tang, J., and Wang, L. (2018, January 8\u201314). Cross-Modal Ranking with Soft Consistency and Noisy Labels for Robust RGB-T Tracking. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_49"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, C., Zhao, N., Lu, Y., Zhu, C., and Tang, J. (2017, January 23\u201327). Weighted Sparse Representation Regularized Graph Learning for RGB-T Object Tracking. Proceedings of the 2017 ACM on Multimedia Conference (MM 2017), Mountain View, CA, USA.","DOI":"10.1145\/3123266.3123289"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1007\/s11432-011-4536-9","article-title":"Fusion tracking in color and infrared images using joint sparse representation","volume":"55","author":"Liu","year":"2012","journal-title":"Sci. China Inf. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Li, C., Hu, S., Gao, S., and Tang, J. (2016, January 4\u20136). Real-Time Grayscale-Thermal Tracking via Laplacian Sparse Representation. Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USA.","DOI":"10.1007\/978-3-319-27674-8_6"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, Y., Li, C., and Tang, J. (2018, January 23\u201326). Learning Soft-Consistent Correlation Filters for RGB-T Object Tracking. Proceedings of the First Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2018), Guangzhou, China.","DOI":"10.1007\/978-3-030-03341-5_25"},{"key":"ref_7","unstructured":"Wu, Y., Blasch, E., Chen, G., Bai, L., and Ling, H. (2011, January 5\u20138). Multiple source data fusion via sparse representation for robust visual tracking. Proceedings of the 14th International Conference on Information Fusion (FUSION 2011), Chicago, IL, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.neucom.2017.11.068","article-title":"Fusing two-stream convolutional neural networks for RGB-T object tracking","volume":"281","author":"Li","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_9","unstructured":"Zhu, Y., Li, C., Lu, Y., Lin, L., Luo, B., and Tang, J. (2018). FANet: Quality-Aware Feature Aggregation Network for RGB-T Tracking. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Long Li, C., Lu, A., Hua Zheng, A., Tu, Z., and Tang, J. (2019). Multi-Adapter RGBT Tracking. arXiv.","DOI":"10.1109\/ICCVW.2019.00279"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Li, C., Luo, B., Tang, J., and Wang, X. (2019, January 21\u201325). Dense Feature Aggregation and Pruning for RGBT Tracking. Proceedings of the 27th ACM International Conference on Multimedia (MM 2019), Nice, France.","DOI":"10.1145\/3343031.3350928"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5743","DOI":"10.1109\/TIP.2016.2614135","article-title":"Learning Collaborative Sparse Representation for Grayscale-Thermal Tracking","volume":"25","author":"Li","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"106977","DOI":"10.1016\/j.patcog.2019.106977","article-title":"RGB-T object tracking: Benchmark and baseline","volume":"96","author":"Li","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1145\/1177352.1177355","article-title":"Object tracking: A survey","volume":"38","author":"Yilmaz","year":"2006","journal-title":"ACM Comput. Surv."},{"key":"ref_15","first-page":"242","article-title":"Systematic survey on object tracking methods in video","volume":"1","author":"Athanesious","year":"2012","journal-title":"Int. J. Adv. Res. Comput. Eng. Technol."},{"key":"ref_16","first-page":"58","article-title":"A survey of appearance models in visual object tracking","volume":"4","author":"Li","year":"2013","journal-title":"ACM TIST"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Valmadre, J., Bertinetto, L., Henriques, J.F., Vedaldi, A., and Torr, P.H.S. (2017, January 21\u201326). End-to-End Representation Learning for Correlation Filter Based Tracking. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.531"},{"key":"ref_18","unstructured":"Li, Y., and Zhu, J. (2014, January 6\u201312). A Scale Adaptive Kernel Correlation Filter Tracker with Feature Integration. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1109\/TPAMI.2014.2345390","article-title":"High-Speed Tracking with Kernelized Correlation Filters","volume":"37","author":"Henriques","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Danelljan, M., H\u00e4ger, G., Khan, F.S., and Felsberg, M. (2015, January 7\u201313). Learning Spatially Regularized Correlation Filters for Visual Tracking. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.490"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Danelljan, M., Robinson, A., Khan, F.S., and Felsberg, M. (2016, January 11\u201314). Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46454-1_29"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lukezic, A., Vojir, T., Zajc, L.C., Matas, J., and Kristan, M. (2017, January 21\u201326). Discriminative Correlation Filter with Channel and Spatial Reliability. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.515"},{"key":"ref_23","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Nam, H., and Han, B. (2016, January 27\u201330). Learning Multi-domain Convolutional Neural Networks for Visual Tracking. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.465"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Danelljan, M., Bhat, G., Khan, F.S., and Felsberg, M. (2017, January 21\u201326). ECO: Efficient Convolution Operators for Tracking. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.733"},{"key":"ref_26","unstructured":"Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., and Torr, P.H.S. (15\u201316, January 8\u201310). Fully-Convolutional Siamese Networks for Object Tracking. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands."},{"key":"ref_27","unstructured":"Pu, S., Song, Y., Ma, C., Zhang, H., and Yang, M. (2018, January 3\u20138). Deep Attentive Tracking via Reciprocative Learning. Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), Montr\u00e9al, QC, Canada."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21\u201326). Residual Attention Network for Image Classification. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.683"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8\u201314). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_18"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhu, Z., Wu, W., Zou, W., and Yan, J. (2018, January 18\u201322). End-to-End Flow Correlation Tracking With Spatial-Temporal Attention. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00064"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Suganuma, M., Liu, X., and Okatani, T. (2019, January 16\u201320). Attention-Based Adaptive Selection of Operations for Image Restoration in the Presence of Unknown Combined Distortions. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00925"},{"key":"ref_32","first-page":"75","article-title":"Feature discovery by competitive learning","volume":"9","author":"Rumelhart","year":"1985","journal-title":"Cogn. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/0893-6080(90)90071-R","article-title":"Competitive learning algorithms for vector quantization","volume":"3","author":"Ahalt","year":"1990","journal-title":"Neural Netw."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1109\/72.238318","article-title":"Rival penalized competitive learning for clustering analysis, RBF net, and curve detection","volume":"4","author":"Xu","year":"1993","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Song, Y., Ma, C., Wu, X., Gong, L., Bao, L., Zuo, W., Shen, C., Lau, R.W.H., and Yang, M. (2018, January 18\u201322). VITAL: Visual Tracking via Adversarial Learning. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00937"},{"key":"ref_36","unstructured":"Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., and Bengio, Y. (2014, January 8\u201313). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, QC, Canada."},{"key":"ref_37","unstructured":"Zhao, J., Mathieu, M., and LeCun, Y. (2016). Energy-based generative adversarial network. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Zhong, Z., Zheng, L., Zheng, Z., Li, S., and Yang, Y. (2018, January 18\u201322). Camera Style Adaptation for Person Re-Identification. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00541"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhang, J., Ma, S., and Sclaroff, S. (2014, January 6\u201312). MEEM: Robust Tracking via Multiple Experts Using Entropy Minimization. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10599-4_13"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Yun, S., Choi, J., Yoo, Y., Yun, K., and Choi, J.Y. (2017, January 21\u201326). Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.148"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Jung, I., Son, J., Baek, M., and Han, B. (2018, January 8\u201314). Real-Time MDNet. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01225-0_6"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Zhang, Z., and Peng, H. (2019, January 16\u201320). Deeper and Wider Siamese Networks for Real-Time Visual Tracking. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR2019), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00472"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Kim, H., Lee, D., Sim, J., and Kim, C. (2015, January 7\u201313). SOWP: Spatially Ordered and Weighted Patch Descriptor for Visual Tracking. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.345"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Danelljan, M., H\u00e4ger, G., Khan, F.S., and Felsberg, M. (2014, January 1\u20135). Accurate Scale Estimation for Robust Visual Tracking. Proceedings of the British Machine Vision Conference, BMVC 2014, Nottingham, UK.","DOI":"10.5244\/C.28.65"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/393\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:19:24Z","timestamp":1760361564000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/393"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,10]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["s20020393"],"URL":"https:\/\/doi.org\/10.3390\/s20020393","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,10]]}}}