{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T15:43:16Z","timestamp":1778600596152,"version":"3.51.4"},"reference-count":17,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,5,19]],"date-time":"2022-05-19T00:00:00Z","timestamp":1652918400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Spanish projects","award":["PID2019-105390RB-I00"],"award-info":[{"award-number":["PID2019-105390RB-I00"]}]},{"name":"Spanish projects","award":["DGA_FSE-T45_20R"],"award-info":[{"award-number":["DGA_FSE-T45_20R"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Autonomous navigation in dynamic environments where people move unpredictably is an essential task for service robots in real-world populated scenarios. Recent works in reinforcement learning (RL) have been applied to autonomous vehicle driving and to navigation around pedestrians. In this paper, we present a novel planner (reinforcement learning dynamic object velocity space, RL-DOVS) based on an RL technique for dynamic environments. The method explicitly considers the robot kinodynamic constraints for selecting the actions in every control period. The main contribution of our work is to use an environment model where the dynamism is represented in the robocentric velocity space as input to the learning system. The use of this dynamic information speeds the training process with respect to other techniques that learn directly either from raw sensors (vision, lidar) or from basic information about obstacle location and kinematics. We propose two approaches using RL and dynamic obstacle velocity (DOVS), RL-DOVS-A, which automatically learns the actions having the maximum utility, and RL-DOVS-D, in which the actions are selected by a human driver. Simulation results and evaluation are presented using different numbers of active agents and static and moving passive agents with random motion directions and velocities in many different scenarios. The performance of the technique is compared with other state-of-the-art techniques for solving navigation problems in environments such as ours.<\/jats:p>","DOI":"10.3390\/s22103847","type":"journal-article","created":{"date-parts":[[2022,5,20]],"date-time":"2022-05-20T00:18:11Z","timestamp":1653005891000},"page":"3847","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["RL-DOVS: Reinforcement Learning for Autonomous Robot Navigation in Dynamic Environments"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2032-948X","authenticated-orcid":false,"given":"Andrew K.","family":"Mackay","sequence":"first","affiliation":[{"name":"Arag\u00f3n Institute for Engineering Research (I3A), University of Zaragoza, 50009 Zaragoza, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6722-5541","authenticated-orcid":false,"given":"Luis","family":"Riazuelo","sequence":"additional","affiliation":[{"name":"Arag\u00f3n Institute for Engineering Research (I3A), University of Zaragoza, 50009 Zaragoza, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0449-2300","authenticated-orcid":false,"given":"Luis","family":"Montano","sequence":"additional","affiliation":[{"name":"Arag\u00f3n Institute for Engineering Research (I3A), University of Zaragoza, 50009 Zaragoza, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1177\/0278364918775520","article-title":"Model-based robocentric planning and navigation for dynamic environments","volume":"37","author":"Lorente","year":"2018","journal-title":"Int. J. Robot. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wolf, P., Hubschneider, C., Weber, M., Bauer, A., H\u00e4rtl, J., D\u00fcrr, F., and Z\u00f6llner, J.M. (2017, January 11\u201314). Learning how to drive in a real world simulation with deep Q-Networks. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995727"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, P., Liu, D., Che, J., Li, H., and Chan, C.Y. (June, January 30). Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning. Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9560907"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wolf, P., Kurzer, K., Wingert, T., Kuhnt, F., and Zollner, J.M. (2018, January 26\u201330). Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Suzhou, China.","DOI":"10.1109\/IVS.2018.8500427"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Song, Y., Lin, H., Kaufmann, E., D\u00fcrr, P., and Scaramuzza, D. (June, January 30). Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning. Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9561049"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Folkers, A., Rick, M., and B\u00fcskens, C. (2019). Controlling an Autonomous Vehicle with Deep Reinforcement Learning. arXiv.","DOI":"10.1109\/IVS.2019.8814124"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, L., Dugas, D., Cesari, G., Siegwart, R., and Dub\u00e9, R. (2020\u201324, January 24). Robot Navigation in Crowded Environments Using Deep Reinforcement Learning. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341540"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"K\u00e4stner, L., Li, J., Shen, Z., and Lambrecht, J. (2021). Enhancing Navigational Safety in Crowded Environments using Semantic-Deep-Reinforcement-Learning-based Navigation. arXiv.","DOI":"10.1109\/SSRR56537.2022.10018699"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Everett, M., Chen, Y.F., and How, J.P. (2018, January 1\u20135). Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593871"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, C., Liu, Y., Kreiss, S., and Alahi, A. (2019, January 20\u201324). Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794134"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"10357","DOI":"10.1109\/ACCESS.2021.3050338","article-title":"Collision Avoidance in Pedestrian-Rich Environments With Deep Reinforcement Learning","volume":"9","author":"Everett","year":"2021","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yao1, S., Chen, G., Qiu, Q., Ma, J., Chen, X., and Ji, J. (2021). Crowd-Aware Robot Navigation for Pedestrians with Multiple Collision Avoidance Strategies via Map-based Deep Reinforcement Learning. arXiv.","DOI":"10.1109\/IROS51168.2021.9636579"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-D\u2019Arpino, C., Liu, C., Goebel, P., Mart\u00edn-Mart\u00edn, R., and Savarese, S. (June, January 30). Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning. Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9560893"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Patel, U., Kumar, N., Sathyamoorthy, A.J., and Manocha, D. (June, January 30). DWA-RL: Dynamically Feasible Deep Reinforcement Learning Policy for Robot Navigation among Mobile Obstacles. Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9561462"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Liu, S., Chang, P., Liangy, W., Chakrabortyy, N., and Driggs-Campbell, K. (June, January 30). Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning. Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9561595"},{"key":"ref_17","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3847\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:14:44Z","timestamp":1760138084000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3847"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,19]]},"references-count":17,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["s22103847"],"URL":"https:\/\/doi.org\/10.3390\/s22103847","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,19]]}}}