{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:11:25Z","timestamp":1760148685396,"version":"build-2065373602"},"reference-count":30,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T00:00:00Z","timestamp":1684713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practical task is widely used in night security, as well as for understanding night scenes and other scenarios. However, due to the differences in image features and the complexity of semantic information, generating captions for infrared images remains a challenging task. From the perspective of deployment and application, to improve the correlation between descriptions and objects, we introduced the YOLOv6 and LSTM as encoder-decoder structure and proposed infrared image caption based on object-oriented attention. Firstly, to improve the domain adaptability of the detector, we optimized the pseudo-label learning process. Secondly, we proposed the object-oriented attention method to address the alignment problem between complex semantic information and embedded words. This method helps select the most crucial features of the object region and guides the caption model in generating words that are more relevant to the object. Our methods have shown good performance on the infrared image and can produce words explicitly associated with the object regions located by the detector. The robustness and effectiveness of the proposed methods were demonstrated through evaluation on various datasets, along with other state-of-the-art methods. Our approach achieved BLUE-4 scores of 31.6 and 41.2 on KAIST and Infrared City and Town datasets, respectively. Our approach provides a feasible solution for the deployment of embedded devices in industrial applications.<\/jats:p>","DOI":"10.3390\/e25050826","type":"journal-article","created":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T05:04:44Z","timestamp":1684731884000},"page":"826","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Infrared Image Caption Based on Object-Oriented Attention"],"prefix":"10.3390","volume":"25","author":[{"given":"Junfeng","family":"Lv","sequence":"first","affiliation":[{"name":"Institute of Unmanned System Research, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tian","family":"Hui","sequence":"additional","affiliation":[{"name":"Institute of Unmanned System Research, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongfeng","family":"Zhi","sequence":"additional","affiliation":[{"name":"Institute of Unmanned System Research, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuelei","family":"Xu","sequence":"additional","affiliation":[{"name":"Institute of Unmanned System Research, Northwestern Polytechnical University, Xi\u2019an 710072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/TPAMI.2016.2587640","article-title":"Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge","volume":"39","author":"Vinyals","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lu, J., Yang, J., Batra, D., and Parikh, D. (2018, January 18\u201322). Neural Baby Talk. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00754"},{"key":"ref_3","first-page":"1","article-title":"Infrared Small Target Detection Using Nonoverlapping Patch Spatial\u2013Temporal Tensor Factorization with Capped Nuclear Norm Regularization","volume":"60","author":"Wang","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Li, B., Zhou, Y., and Ren, H. (2020, January 11\u201314). Image Emotion Caption Based on Visual Attention Mechanisms. Proceedings of the 2020 IEEE 6th International Conference on Computer and Communications (ICCC), Chengdu, China.","DOI":"10.1109\/ICCC51575.2020.9344900"},{"key":"ref_5","unstructured":"Everingham, M., Van Gool, L., Williams, C.K., Winn, J., and Zisserman, A. (2022, May 10). The PASCAL Visual Object Classes Challenge 2012 Results. Available online: http:\/\/host.robots.ox.ac.uk\/pascal\/VOC\/voc2012\/."},{"key":"ref_6","unstructured":"Li, C., Li, L., Jiang, H., Weng, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., and Nie, W. (2022). YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hsu, H.-K., Hung, W.-C., Tseng, H.-Y., Yao, C.-H., Tsai, Y.-H., Singh, M., and Yang, M.-H. (2020, January 1\u20135). Progressive domain adaptation for object detection. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093358"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Venugopalan, S., Hendricks, L.A., Rohrbach, M., Mooney, R., Darrell, T., and Saenko, K. (2017, January 21\u201326). Captioning Images with Diverse Objects. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.130"},{"key":"ref_9","unstructured":"Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., and Chua, T.S. (2017, January 21\u201326). SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.667"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2743","DOI":"10.1109\/TIP.2018.2889922","article-title":"Topic-Oriented Image Captioning Based on Order-Embedding","volume":"28","author":"Yu","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhu, Z., Xue, Z., and Yuan, Z. (2018, January 7\u201310). Topic-Guided Attention for Image Captioning. Proceedings of the 25th IEEE International Conference on Image Processing, Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451083"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chen, F., Xie, S., Li, X., Li, S., Tang, J., and Wang, T. (2019, January 8\u201312). What Topics Do Images Say: A Neural Image Captioning Model with Topic Representation. Proceedings of the 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shanghai, China.","DOI":"10.1109\/ICMEW.2019.00083"},{"key":"ref_14","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017). Attention Is All You Need. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. ECCV 2020, Springer.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_16","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, J., Chen, Z., Ma, A., and Zhong, Y. (2022, January 17\u201322). Capformer: Pure Transformer for Remote Sensing Image Caption. Proceedings of the 2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia.","DOI":"10.1109\/IGARSS46834.2022.9883199"},{"key":"ref_18","first-page":"1","article-title":"Remote Sensing Image Change Captioning with Dual-Branch Transformers: A New Method and a Large Scale Dataset","volume":"60","author":"Liu","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_19","first-page":"1","article-title":"Remote-Sensing Image Captioning Based on Multilayer Aggregated Transformer","volume":"19","author":"Liu","year":"2022","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020, January 14\u201319). Meshed-Memory Transformer for Image Captioning. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01059"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., and Gao, J. (2019). Unified Vision-Language Pre-Training for Image Captioning and VQA. arXiv.","DOI":"10.1609\/aaai.v34i07.7005"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., and Cao, Z. (2022). mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.488"},{"key":"ref_23","unstructured":"Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022). OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., and Ji, R. (2021, January 20\u201325). RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01521"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yao, T., Pan, Y., Li, Y., and Mei, T. (2017, January 21\u201326). Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.559"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhu, J.-Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hwang, S., Park, J., Kim, N., Choi, Y., and So Kweon, I. (2015, January 7\u201312). Multispectral pedestrian detection: Benchmark dataset and baseline. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298706"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016). SSD: Single shot multibox detector, In ECCV, Springer.","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"ref_30","unstructured":"Bochkovskiy, A., Wang, C.Y., and Liao, H.Y. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/5\/826\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:39:45Z","timestamp":1760125185000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/5\/826"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,22]]},"references-count":30,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["e25050826"],"URL":"https:\/\/doi.org\/10.3390\/e25050826","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2023,5,22]]}}}