{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T15:19:59Z","timestamp":1783091999865,"version":"3.54.6"},"reference-count":44,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T00:00:00Z","timestamp":1783036800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Object detection in unmanned aerial vehicle (UAV) imagery suffers from severe scale variation, dense object packing, and prohibitive computational cost when conventional detectors exhaustively evaluate high-resolution frames. Reinforcement learning (RL)-based sequential detectors offer a promising alternative by formulating localization as an active search process, yet existing methods are limited by discrete-time state transitions, sparse reward signals, and premature policy collapse. In this paper, we propose HRL-Det, a hierarchical reinforcement learning framework that addresses these challenges through two tightly coupled innovations. First, a Neural ODE-driven Continuous-Time Bellman State Evolution module models the agent\u2019s state dynamics as a stochastic differential equation governed by the Hamilton\u2013Jacobi\u2013Bellman equation, enabling fine-grained temporal reasoning with memory-efficient adjoint-based backpropagation. Second, a Lyapunov-Guided Entropy-Regularized Reward Shaping mechanism constructs convergence-promoting dense rewards informed by Lyapunov stability analysis while maintaining exploration diversity through maximum entropy optimization. Extensive experiments on VisDrone2019, DroneVehicle, and MS COCO 2017 show that HRL-Det achieves mAP@0.5 of 0.412, 0.812, and 0.735, respectively, outperforming existing RL-based detectors and achieving competitive accuracy relative to representative non-RL detectors under the same COCO metric, while requiring only 17.3 M parameters and an average of 6.3 search steps per object.<\/jats:p>","DOI":"10.3390\/s26134232","type":"journal-article","created":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T14:33:06Z","timestamp":1783089186000},"page":"4232","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["HRL-Det: Hierarchical Reinforcement Learning for Sequential Object Detection in Aerial Imagery"],"prefix":"10.3390","volume":"26","author":[{"given":"Meng","family":"Li","sequence":"first","affiliation":[{"name":"College of Geography and Environment, Xianyang Normal University, Xianyang 712000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2081-9358","authenticated-orcid":false,"given":"Yaowen","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Computer Science, National University of Defense Technology (NUDT), Changsha 410073, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"6047","DOI":"10.1109\/TNNLS.2021.3080276","article-title":"Vehicle Detection from UAV Imagery with Deep Learning: A Review","volume":"33","author":"Bouguettaya","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lygouras, E., Santavas, N., Taitzoglou, A., Tarchanidis, K., Mitropoulos, A., and Gasteratos, A. (2019). Unsupervised Human Detection with an Embedded Vision System on a Fully Autonomous UAV for Search and Rescue Operations. Sensors, 19.","DOI":"10.3390\/s19163542"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.ijepes.2017.12.016","article-title":"Automatic Autonomous Vision-Based Power Line Inspection: A Review of Current Status and the Potential Role of Deep Learning","volume":"99","author":"Nguyen","year":"2018","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.engappai.2019.01.008","article-title":"Railway Track Fastener Defect Detection Based on Image Processing and Deep Learning Techniques: A Comparative Study","volume":"80","author":"Wei","year":"2019","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_5","unstructured":"Du, D., Zhu, P., Wen, L., Bian, X., Lin, H., Hu, Q., Peng, T., Zheng, J., Wang, X., and Zhang, Y. (2019, January 27\u201328). VisDrone-DET2019: The Vision Meets Drone Object Detection in Image Challenge Results. Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops (ICCVW), Seoul, Republic of Korea."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"7380","DOI":"10.1109\/TPAMI.2021.3119563","article-title":"Detection and Tracking Meet Drones Challenge","volume":"44","author":"Zhu","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Cai, Z., and Vasconcelos, N. (2018, January 18\u201323). Cascade R-CNN: Delving into High Quality Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00644"},{"key":"ref_9","unstructured":"Jocher, G., Chaurasia, A., Stoken, A., Borovec, J., NanoCode012, Kwon, Y., Michael, K., Xie, T., Fang, J., and imyhxy (2022). Ultralytics\/YOLOv5: V7.0\u2014YOLOv5 SOTA Realtime Instance Segmentation. Zenodo."},{"key":"ref_10","unstructured":"Jocher, G., Chaurasia, A., and Qiu, J. (2025, January 15). Ultralytics YOLOv8. Available online: https:\/\/github.com\/ultralytics\/ultralytics."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016, January 11\u201314). SSD: Single Shot MultiBox Detector. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11\u201317). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_13","unstructured":"Zhou, X., Wang, D., and Krahenbuhl, P. (2019). Objects as Points. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, C., Yang, T., Zhu, S., Chen, C., and Guan, S. (2020, January 14\u201319). Density Map Guided Object Detection in Aerial Images. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00103"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"104046","DOI":"10.1016\/j.imavis.2020.104046","article-title":"Deep Learning-Based Object Detection in Low-Altitude UAV Datasets: A Survey","volume":"104","author":"Mittal","year":"2020","journal-title":"Image Vis. Comput."},{"key":"ref_16","first-page":"13467","article-title":"Towards Large-Scale Small Object Detection: Survey and Benchmarks","volume":"45","author":"Cheng","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Caicedo, J.C., and Lazebnik, S. (2015, January 7\u201313). Active Object Localization with Deep Reinforcement Learning. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.286"},{"key":"ref_18","unstructured":"Bellver, M., Giro-i-Nieto, X., Marques, F., and Torres, J. (2016, January 5\u201310). Hierarchical Object Detection with Deep Reinforcement Learning. Proceedings of the NIPS 2016 Deep Reinforcement Learning Workshop, Barcelona, Spain."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2733","DOI":"10.1007\/s10462-021-10061-9","article-title":"Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey","volume":"55","author":"Le","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Uzkent, B., and Yoon, S. (2020, January 13\u201319). Learning When and Where to Zoom with Deep Reinforcement Learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01236"},{"key":"ref_21","first-page":"2544","article-title":"Pay Attention to Them: Deep Reinforcement Learning-Based Cascade Object Detection","volume":"31","author":"Liu","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1109\/TIP.2025.3541564","article-title":"Hybrid DQN-Based Low-Computational Reinforcement Learning Object Detection with Adaptive Dynamic Reward Function and ROI Align-Based Bounding Box Regression","volume":"34","author":"Zhou","year":"2025","journal-title":"IEEE Trans. Image Process."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kong, X., Xin, B., Wang, Y., and Hua, G. (2017, January 21\u201326). Collaborative Deep Reinforcement Learning for Joint Object Search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.748"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017, January 22\u201329). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wang, A., Chen, H., Liu, L., Chen, K., Lin, Z., Han, J., and Ding, G. (2024, January 9\u201315). YOLOv10: Real-Time End-to-End Object Detection. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.","DOI":"10.52202\/079017-3429"},{"key":"ref_27","unstructured":"Wang, C.-Y., Yeh, I.-H., and Liao, H.-Y.M. (October, January 29). YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy."},{"key":"ref_28","unstructured":"Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2021, January 3\u20137). Deformable DETR: Deformable Transformers for End-to-End Object Detection. Proceedings of the International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_29","unstructured":"Lv, W., Xu, S., Zhao, Y., Wang, G., Wei, J., Cui, C., Du, Y., Dang, Q., and Liu, Y. (2024, January 17\u201321). DETRs Beat YOLOs on Real-Time Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA."},{"key":"ref_30","unstructured":"Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., and Shum, H.-Y. (2022). DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mathe, S., Pirinen, A., and Sminchisescu, C. (2016, January 27\u201330). Reinforcement Learning for Visual Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.316"},{"key":"ref_32","unstructured":"Jie, Z., Liang, X., Feng, J., Jin, X., Lu, W., and Yan, S. (2016, January 5\u201310). Tree-Structured Reinforcement Learning for Sequential Object Localization. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Pirinen, A., and Sminchisescu, C. (2018, January 18\u201323). Deep Reinforcement Learning of Region Proposal Networks for Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00726"},{"key":"ref_34","unstructured":"Ding, W., Majcherczyk, N., Deshpande, M., Qi, X., Zhao, D., Madhivanan, R., and Sen, A. (June, January 29). Learning to View: Decision Transformers for Active Object Detection. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, J., Yang, X., He, W., Ren, J., Zhang, Q., Zhao, Y., Bai, R., He, X., and Liu, J. (2024, January 20\u201327). Scale Optimization Using Evolutionary Reinforcement Learning for Object Detection on Drone Imagery. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i1.27795"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Akhloufi, M.A., Arola, S., and Bonnet, A. (2019). Drones Chasing Drones: Reinforcement Learning and Deep Search Area Proposal. Drones, 3.","DOI":"10.3390\/drones3030058"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"8199","DOI":"10.1007\/s00521-024-09466-8","article-title":"A Reinforcement Learning (RL)-Based Hybrid Method for Ground Penetrating Radar (GPR)-Driven Buried Object Detection","volume":"36","author":"Alpdemir","year":"2024","journal-title":"Neural Comput. Appl."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/0771-050X(80)90013-3","article-title":"A Family of Embedded Runge-Kutta Formulae","volume":"6","author":"Dormand","year":"1980","journal-title":"J. Comput. Appl. Math."},{"key":"ref_40","unstructured":"Ng, A.Y., Harada, D., and Russell, S. (1999, January 27\u201330). Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. Proceedings of the International Conference on Machine Learning (ICML), Bled, Slovenia."},{"key":"ref_41","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"6734","DOI":"10.1109\/TCSVT.2022.3168279","article-title":"DroneVehicle: Drone-Based RGB-Infrared Vehicle Detection Benchmark","volume":"32","author":"Sun","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zong, Z., Song, G., and Liu, Y. (2023, January 1\u20136). DETRs with Collaborative Hybrid Assignments Training. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Paris, France.","DOI":"10.1109\/ICCV51070.2023.00621"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/13\/4232\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T14:38:25Z","timestamp":1783089505000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/13\/4232"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,3]]},"references-count":44,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["s26134232"],"URL":"https:\/\/doi.org\/10.3390\/s26134232","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,3]]}}}