{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T15:21:58Z","timestamp":1777130518859,"version":"3.51.4"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,9,1]],"date-time":"2023-09-01T00:00:00Z","timestamp":1693526400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T00:00:00Z","timestamp":1693958400000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61703138"],"award-info":[{"award-number":["61703138"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Intell Robot Syst"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mapless navigation for mobile Unmanned Ground Vehicles (UGVs) using Deep Reinforcement Learning (DRL) has attracted significantly rising attention in robotic and related research communities. Collision avoidance from dynamic obstacles in unstructured environments, such as pedestrians and other vehicles, is one of the key challenges for mapless navigation. This paper proposes a DRL algorithm based on heuristic correction learning for autonomous navigation of a UGV in mapless configuration. We use a 24-dimensional lidar sensor, and merge the target position information and the speed information of the UGV as the input of the reinforcement learning agent. The actions of the UGV are produced as the output of the agent. Our proposed algorithm has been trained and evaluated in both static and dynamic environments. The experimental result shows that our proposed algorithm can reach the target in less time with shorter distances under the premise of ensuring safety than other algorithms. Especially, the success rate of our proposed algorithm is 2.05 times higher than the second effective algorithm and the trajectory efficiency is improved by <jats:inline-formula><jats:alternatives><jats:tex-math>$$24\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>24<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> in the dynamic environment. Finally, our proposed algorithm is deployed on a real robot in the real-world environment to validate and evaluate the algorithm performance. Experimental results show that our proposed algorithm can be directly applied to real robots robustly.<\/jats:p>","DOI":"10.1007\/s10846-023-01950-y","type":"journal-article","created":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T03:29:56Z","timestamp":1693970996000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Deep Reinforcement Learning with Heuristic Corrections for UGV Navigation"],"prefix":"10.1007","volume":"109","author":[{"given":"Changyun","family":"Wei","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yajun","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongping","family":"Ouyang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8968-9902","authenticated-orcid":false,"given":"Ze","family":"Ji","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,9,6]]},"reference":[{"key":"1950_CR1","unstructured":"Likhachev, M., Ferguson, D.I., Gordon, G.J., Stentz, A., Thrun, S.: Anytime dynamic a*: An anytime, replanning algorithm. In: International Conference on Automated Planning and Scheduling (ICAPS), vol. 5, pp. 262\u2013271 (2005)"},{"issue":"7","key":"1950_CR2","doi-asserted-by":"publisher","first-page":"1651","DOI":"10.5772\/56718","volume":"10","author":"J Nasir","year":"2013","unstructured":"Nasir, J., Islam, F., Malik, U., Ayaz, Y., Hasan, O., Khan, M., Muhammad, M.S.: Rrt*-smart: A rapid convergence implementation of rrt. Int J Adv Robot Syst 10(7), 1651\u20131656 (2013)","journal-title":"Int J Adv Robot Syst"},{"issue":"2","key":"1950_CR3","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1109\/MRA.2006.1638022","volume":"13","author":"H Durrant-Whyte","year":"2006","unstructured":"Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part i. IEEE Robot Autom Mag 13(2), 99\u2013110 (2006)","journal-title":"IEEE Robot Autom Mag"},{"key":"1950_CR4","doi-asserted-by":"crossref","unstructured":"Chen, Y.F., Everett, M., Liu, M., How, J.P.: Socially aware motion planning with deep reinforcement learning. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1343\u20131350 (2017). IEEE","DOI":"10.1109\/IROS.2017.8202312"},{"issue":"7587","key":"1950_CR5","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"529","author":"D Silver","year":"2016","unstructured":"Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484\u2013489 (2016)","journal-title":"Nature"},{"key":"1950_CR6","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013). arXiv:1312.5602"},{"key":"1950_CR7","doi-asserted-by":"crossref","unstructured":"Ye, D., Liu, Z., Sun, M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., Guo, Q., et al.: Mastering complex control in moba games with deep reinforcement learning. In: AAAI Conference on Artificial Intelligence, vol. 34, pp. 6672\u20136679 (2020)","DOI":"10.1609\/aaai.v34i04.6144"},{"key":"1950_CR8","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning (ICML), pp. 1928\u20131937 (2016). PMLR"},{"key":"1950_CR9","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning (2015). arXiv:1509.02971"},{"key":"1950_CR10","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning (ICML), pp. 1889\u20131897 (2015). PMLR"},{"key":"1950_CR11","unstructured":"Wang, Y., He, H., Tan, X.: Truly proximal policy optimization. In: Uncertainty in Artificial Intelligence, pp. 113\u2013122 (2020). PMLR"},{"key":"1950_CR12","doi-asserted-by":"crossref","unstructured":"Tai, L., Paolo, G., Liu, M.: Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 31\u201336 (2017). IEEE","DOI":"10.1109\/IROS.2017.8202134"},{"key":"1950_CR13","doi-asserted-by":"crossref","unstructured":"Zhang, P., Wei, C., Cai, B., Ouyang, Y.: Mapless navigation for autonomous robots: A deep reinforcement learning approach. In: Chinese Automation Congress (CAC), pp. 3141\u20133146 (2019). IEEE","DOI":"10.1109\/CAC48633.2019.8997292"},{"key":"1950_CR14","doi-asserted-by":"crossref","unstructured":"Gu, S., Holly, E., Lillicrap, T., Levine, S.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3389\u20133396 (2017). IEEE","DOI":"10.1109\/ICRA.2017.7989385"},{"key":"1950_CR15","doi-asserted-by":"crossref","unstructured":"Chaffre, T., Moras, J., Chan-Hon-Tong, A., Marzat, J.: Sim-to-real transfer with incremental environment complexity for reinforcement learning of depth-based robot navigation (2020). arXiv:2004.14684","DOI":"10.5220\/0009821603140323"},{"key":"1950_CR16","doi-asserted-by":"publisher","first-page":"1541","DOI":"10.1007\/s11277-021-08294-6","volume":"119","author":"R Pakrooh","year":"2021","unstructured":"Pakrooh, R., Bohlooli, A.: A survey on unmanned aerial vehicles-assisted internet of things: A service-oriented classification. Wirel Pers Commun 119, 1541\u20131575 (2021)","journal-title":"Wirel Pers Commun"},{"issue":"2","key":"1950_CR17","doi-asserted-by":"publisher","first-page":"995","DOI":"10.1007\/s11277-022-09780-1","volume":"126","author":"T Alam","year":"2022","unstructured":"Alam, T.: Blockchain-enabled deep reinforcement learning approach for performance optimization on the internet of things. Wirel Pers Commun 126(2), 995\u20131011 (2022)","journal-title":"Wirel Pers Commun"},{"issue":"4","key":"1950_CR18","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1080\/03772063.1989.11436815","volume":"35","author":"A Swarup","year":"1989","unstructured":"Swarup, A., Gopal, M.: Control strategies for robot manipulators-a review. IETE J Res 35(4), 198\u2013207 (1989)","journal-title":"IETE J Res"},{"key":"1950_CR19","doi-asserted-by":"crossref","unstructured":"An, X., Wang, Y.: Smart wearable medical devices for isometric contraction of muscles and joint tracking with gyro sensors for elderly people. J Ambient Intell Hum Comput, 1\u201312 (2021)","DOI":"10.1007\/s12652-021-02993-5"},{"key":"1950_CR20","doi-asserted-by":"publisher","first-page":"6261","DOI":"10.1007\/s12652-020-01877-4","volume":"11","author":"H Ding","year":"2020","unstructured":"Ding, H.: Motion path planning of soccer training auxiliary robot based on genetic algorithm in fixed-point rotation environment. J Ambient Intell Hum Comput 11, 6261\u20136270 (2020)","journal-title":"J Ambient Intell Hum Comput"},{"key":"1950_CR21","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.1007\/s11277-019-06138-y","volume":"105","author":"P Pawar","year":"2019","unstructured":"Pawar, P., Yadav, S.M., Trivedi, A.: Performance study of dual unmanned aerial vehicles with underlaid device-to-device communications. Wirel Pers Commun 105, 1111\u20131132 (2019)","journal-title":"Wirel Pers Commun"},{"issue":"1","key":"1950_CR22","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1007\/s11277-019-06271-8","volume":"106","author":"IA Alimi","year":"2019","unstructured":"Alimi, I.A., Teixeira, A.L., Monteiro, P.P.: Effects of correlated multivariate fso channel on outage performance of space-air-ground integrated network (sagin). Wirel Pers Commun 106(1), 7\u201325 (2019)","journal-title":"Wirel Pers Commun"},{"key":"1950_CR23","doi-asserted-by":"publisher","first-page":"2107","DOI":"10.1007\/s11277-019-06974-y","volume":"111","author":"H Li","year":"2020","unstructured":"Li, H., Luo, J., Li, J.: Reinforcement learning based full-duplex cognitive anti-jamming using improved energy detector. Wirel Pers Commun 111, 2107\u20132127 (2020)","journal-title":"Wirel Pers Commun"},{"key":"1950_CR24","doi-asserted-by":"publisher","first-page":"3793","DOI":"10.1007\/s11277-017-4699-9","volume":"97","author":"L Li","year":"2017","unstructured":"Li, L., Mao, Y.: Autonomously coordinating multiple unmanned vehicles for data communication between two stations. Wirel Pers Commun 97, 3793\u20133810 (2017)","journal-title":"Wirel Pers Commun"},{"key":"1950_CR25","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1007\/s11277-020-07608-4","volume":"115","author":"JJ Praise","year":"2020","unstructured":"Praise, J.J., Raj, R.J.S., Benifa, J.B.: Development of reinforcement learning and pattern matching (rlpm) based firewall for secured cloud infrastructure. Wirel Pers Commun 115, 993\u20131018 (2020)","journal-title":"Wirel Pers Commun"},{"key":"1950_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11220-020-00295-2","volume":"21","author":"PP Tasgaonkar","year":"2020","unstructured":"Tasgaonkar, P.P., Garg, R.D., Garg, P.K.: Vehicle detection and traffic estimation with sensors technologies for intelligent transportation systems. Sens & Imaging 21, 1\u201328 (2020)","journal-title":"Sens & Imaging"},{"key":"1950_CR27","doi-asserted-by":"publisher","first-page":"2663","DOI":"10.1007\/s11277-020-07496-8","volume":"114","author":"V Annepu","year":"2020","unstructured":"Annepu, V., Rajesh, A.: Implementation of an efficient artificial bee colony algorithm for node localization in unmanned aerial vehicle assisted wireless sensor networks. Wirel Pers Commun 114, 2663\u20132680 (2020)","journal-title":"Wirel Pers Commun"},{"key":"1950_CR28","doi-asserted-by":"crossref","unstructured":"Kumar, A.: Real-time performance comparison of vision-based autonomous landing of quadcopter on a ground moving target. IETE J Res, 1\u201318 (2021)","DOI":"10.1080\/03772063.2021.1963332"},{"key":"1950_CR29","doi-asserted-by":"publisher","first-page":"6271","DOI":"10.1007\/s12652-020-01878-3","volume":"11","author":"X Li","year":"2020","unstructured":"Li, X.: Robot target localization and interactive multi-mode motion trajectory tracking based on adaptive iterative learning. J Ambient Intell Hum Comput 11, 6271\u20136282 (2020)","journal-title":"J Ambient Intell Hum Comput"},{"key":"1950_CR30","first-page":"739","volume":"18","author":"U Muller","year":"2005","unstructured":"Muller, U., Ben, J., Cosatto, E., Flepp, B., Cun, Y.: Off-road obstacle avoidance through end-to-end learning. Adv Neural Inf Process Syst 18, 739\u2013746 (2005). (MIT Press)","journal-title":"Adv Neural Inf Process Syst"},{"key":"1950_CR31","doi-asserted-by":"crossref","unstructured":"Tai, L., Li, S., Liu, M.: A deep-network solution towards model-less obstacle avoidance. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2759\u20132764 (2016). IEEE","DOI":"10.1109\/IROS.2016.7759428"},{"key":"1950_CR32","doi-asserted-by":"crossref","unstructured":"Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: Learning affordance for direct perception in autonomous driving. In: IEEE International Conference on Computer Vision, pp. 2722\u20132730 (2015). IEEE","DOI":"10.1109\/ICCV.2015.312"},{"issue":"11","key":"1950_CR33","doi-asserted-by":"publisher","first-page":"1289","DOI":"10.1177\/0278364915619772","volume":"35","author":"H Kretzschmar","year":"2016","unstructured":"Kretzschmar, H., Spies, M., Sprunk, C., Burgard, W.: Socially compliant mobile robot navigation via inverse reinforcement learning. Int J Robot Res 35(11), 1289\u20131307 (2016)","journal-title":"Int J Robot Res"},{"key":"1950_CR34","doi-asserted-by":"crossref","unstructured":"Pfeiffer, M., Schaeuble, M., Nieto, J., Siegwart, R., Cadena, C.: From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1527\u20131533 (2017). IEEE","DOI":"10.1109\/ICRA.2017.7989182"},{"key":"1950_CR35","doi-asserted-by":"crossref","unstructured":"Zhu, D., Li, T., Ho, D., Wang, C., Meng, M.Q.-H.: Deep reinforcement learning supervised autonomous exploration in office environments. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 7548\u20137555 (2018). IEEE","DOI":"10.1109\/ICRA.2018.8463213"},{"key":"1950_CR36","doi-asserted-by":"crossref","unstructured":"Zhang, J., Springenberg, J.T., Boedecker, J., Burgard, W.: Deep reinforcement learning with successor features for navigation across similar environments. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371\u20132378 (2017). IEEE","DOI":"10.1109\/IROS.2017.8206049"},{"key":"1950_CR37","doi-asserted-by":"crossref","unstructured":"Everett, M., Chen, Y.F., How, J.P.: Motion planning among dynamic, decision-making agents with deep reinforcement learning. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3052\u20133059 (2018). IEEE","DOI":"10.1109\/IROS.2018.8593871"},{"key":"1950_CR38","unstructured":"Gu, S., Lillicrap, T., Sutskever, I., Levine, S.: Continuous deep q-learning with model-based acceleration. In: International Conference on Machine Learning (ICML), pp. 2829\u20132838 (2016). PMLR"},{"key":"1950_CR39","doi-asserted-by":"crossref","unstructured":"Chen, C., Liu, Y., Kreiss, S., Alahi, A.: Crowd-robot interaction: Crowdaware robot navigation with attention-based deep reinforcement learning. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 6015\u20136022 (2019). IEEE","DOI":"10.1109\/ICRA.2019.8794134"},{"key":"1950_CR40","doi-asserted-by":"crossref","unstructured":"Ciou, P.-H., Hsiao, Y.-T., Wu, Z.-Z., Tseng, S.-H., Fu, L.-C.: Composite reinforcement learning for social robot navigation. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2553\u20132558 (2018). IEEE","DOI":"10.1109\/IROS.2018.8593410"},{"key":"1950_CR41","doi-asserted-by":"crossref","unstructured":"Chen, Y.F., Liu, M., Everett, M., How, J.P.: Decentralized noncommunicating multiagent collision avoidance with deep reinforcement learning. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 285\u2013292 (2017). IEEE","DOI":"10.1109\/ICRA.2017.7989037"},{"key":"1950_CR42","unstructured":"Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning (ICML), pp. 1587\u20131596 (2018). PMLR"},{"key":"1950_CR43","first-page":"2613","volume":"23","author":"H Hasselt","year":"2010","unstructured":"Hasselt, H.: Double q-learning. Adv Neural Inf Process Syst 23, 2613\u20132621 (2010)","journal-title":"Adv Neural Inf Process Syst"},{"key":"1950_CR44","unstructured":"Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay (2015). arXiv:1511.05952"},{"key":"1950_CR45","unstructured":"Koenig, N., Howard, A.: Design and use paradigms for gazebo, an opensource multi-robot simulator. In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), vol. 3, pp. 2149\u20132154 (2004). IEEE"}],"container-title":["Journal of Intelligent &amp; Robotic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-023-01950-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10846-023-01950-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10846-023-01950-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T14:41:55Z","timestamp":1695998515000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10846-023-01950-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9]]},"references-count":45,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["1950"],"URL":"https:\/\/doi.org\/10.1007\/s10846-023-01950-y","relation":{},"ISSN":["0921-0296","1573-0409"],"issn-type":[{"value":"0921-0296","type":"print"},{"value":"1573-0409","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9]]},"assertion":[{"value":"13 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable. Our manuscript does not report results of studies involving humans or animals.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable. Our manuscript does not report results of studies involving humans or animals.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"All authors have approved and consented to publish the manuscript.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}},{"value":"The authors have no relevant financial or nonfinancial interests to disclose.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"18"}}