{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:51:15Z","timestamp":1776783075861,"version":"3.51.2"},"reference-count":39,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,3,18]],"date-time":"2023-03-18T00:00:00Z","timestamp":1679097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003339","name":"Spanish Project","doi-asserted-by":"publisher","award":["PID2021-122944OB-I00"],"award-info":[{"award-number":["PID2021-122944OB-I00"]}],"id":[{"id":"10.13039\/501100003339","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents the use of deep Reinforcement Learning (RL) for autonomous navigation of an Unmanned Ground Vehicle (UGV) with an onboard three-dimensional (3D) Light Detection and Ranging (LiDAR) sensor in off-road environments. For training, both the robotic simulator Gazebo and the Curriculum Learning paradigm are applied. Furthermore, an Actor\u2013Critic Neural Network (NN) scheme is chosen with a suitable state and a custom reward function. To employ the 3D LiDAR data as part of the input state of the NNs, a virtual two-dimensional (2D) traversability scanner is developed. The resulting Actor NN has been successfully tested in both real and simulated experiments and favorably compared with a previous reactive navigation approach on the same UGV.<\/jats:p>","DOI":"10.3390\/s23063239","type":"journal-article","created":{"date-parts":[[2023,3,20]],"date-time":"2023-03-20T03:30:31Z","timestamp":1679283031000},"page":"3239","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Reinforcement and Curriculum Learning for Off-Road Navigation of an UGV with a 3D LiDAR"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2303-1742","authenticated-orcid":false,"given":"Manuel","family":"S\u00e1nchez","sequence":"first","affiliation":[{"name":"Institute for Mechatronics Engineering and Cyber-Physical Systems, Universidad de M\u00e1laga, 29071 M\u00e1laga, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1095-4775","authenticated-orcid":false,"given":"Jes\u00fas","family":"Morales","sequence":"additional","affiliation":[{"name":"Institute for Mechatronics Engineering and Cyber-Physical Systems, Universidad de M\u00e1laga, 29071 M\u00e1laga, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8940-2465","authenticated-orcid":false,"given":"Jorge L.","family":"Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Institute for Mechatronics Engineering and Cyber-Physical Systems, Universidad de M\u00e1laga, 29071 M\u00e1laga, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Islam, F., Nabi, M.M., and Ball, J.E. (2022). Off-Road Detection Analysis for Autonomous Ground Vehicles: A Review. Sensors, 22.","DOI":"10.3390\/s22218463"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1017\/S0263574706003171","article-title":"High-speed navigation of unmanned ground vehicles on uneven terrain using potential fields","volume":"25","author":"Shimoda","year":"2007","journal-title":"Robotica"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1109\/MRA.2010.936946","article-title":"Learning for autonomous navigation","volume":"17","author":"Bagnell","year":"2010","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_4","unstructured":"Sutton, R.S., and Barto, A.G. (2020). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2733","DOI":"10.1007\/s10462-021-10061-9","article-title":"Deep reinforcement learning in computer vision: A comprehensive survey","volume":"55","author":"Le","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1177\/0278364913495721","article-title":"Reinforcement Learning in Robotics: A Survey","volume":"32","author":"Kober","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"7326","DOI":"10.1109\/TSMC.2020.2975232","article-title":"Reinforcement Learning Control of a Flexible Two-Link Manipulator: An Experimental Investigation","volume":"51","author":"He","year":"2021","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Bakale, V.A., Kumar V S, Y., Roodagi, V.C., Kulkarni, Y.N., Patil, M.S., and Chickerur, S. (2020, January 26\u201328). Indoor Navigation with Deep Reinforcement Learning. Proceedings of the International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India.","DOI":"10.1109\/ICICT48043.2020.9112385"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4345","DOI":"10.1109\/LRA.2021.3068106","article-title":"Visual Navigation in Real-World Indoor Environments Using End-to-End Deep Reinforcement Learning","volume":"6","author":"Derner","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_10","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., L\u00f3pez, A., and Koltun, V. (2017, January 13\u201315). CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Conference on Robot Learning, Mountain View, CA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"6991","DOI":"10.1109\/TITS.2021.3066366","article-title":"Reinforcement-Tracking: An Effective Trajectory Tracking and Navigation Method for Autonomous Urban Driving","volume":"23","author":"Liu","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/978-3-030-75178-4_7","article-title":"Synthetic Simulated Environments","volume":"Volume 174","author":"Nikolenko","year":"2021","journal-title":"Synthetic Data for Deep Learning"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1526","DOI":"10.1007\/s11263-022-01611-x","article-title":"Curriculum Learning: A Survey","volume":"130","author":"Soviany","year":"2022","journal-title":"Int. J. Comput. Vis."},{"key":"ref_14","unstructured":"Koenig, K., and Howard, A. (October, January 28). Design and Use Paradigms for Gazebo, an Open-Source Multi-Robot Simulator. Proceedings of the IEEE-RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan."},{"key":"ref_15","unstructured":"Konda, V., and Tsitsiklis, J. (December, January 29). Actor\u2013Critic Algorithms. Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"96331","DOI":"10.1109\/ACCESS.2022.3202545","article-title":"A Survey of Traversability Estimation for Mobile Robots","volume":"10","author":"Sevastopoulos","year":"2022","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Colas, F., Mahesh, S., Pomerleau, F., Liu, M., and Siegwart, R. (2013, January 3\u20137). 3D Path Planning and Execution for Search and Rescue Ground Robots. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696431"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"940","DOI":"10.1002\/rob.21700","article-title":"Driving on Point Clouds: Motion Planning, Trajectory Optimization, and Terrain Assessment in Generic Nonplanar Environments","volume":"34","author":"Furgale","year":"2017","journal-title":"J. Field Robot."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez, J.L., Mor\u00e1n, M., Morales, J., Reina, A.J., and Zafra, M. (2018). Field Navigation Using Fuzzy Elevation Maps Built with Local 3D Laser Scans. Appl. Sci., 8.","DOI":"10.3390\/app8030397"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Guastella, D.C., and Muscato, G. (2021). Learning-Based Methods of Perception and Navigation for Ground Vehicles in Unstructured Environments: A Review. Sensors, 21.","DOI":"10.3390\/s21010073"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.1177\/0278364910369715","article-title":"Learning from Demonstration for Autonomous Navigation in Complex Unstructured Terrain","volume":"29","author":"Silver","year":"2010","journal-title":"Int. J. Robot. Res."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1695","DOI":"10.1109\/LRA.2018.2801794","article-title":"Learning ground traversability from simulations","volume":"3","author":"Guzzi","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Schilling, F., Chen, X., Folkesson, J., and Jensfelt, P. (2017, January 24\u201328). Geometric and visual terrain classification for autonomous mobile navigation. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8206092"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez, J.L., Mor\u00e1n, M., Morales, J., Robles, A., and S\u00e1nchez, M. (2020). Supervised Learning of Natural-Terrain Traversability with Synthetic 3D Laser Scans. Appl. Sci., 10.","DOI":"10.3390\/app10031140"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez, J.L., Morales, J., S\u00e1nchez, M., Mor\u00e1n, M., Reina, A.J., and Fern\u00e1ndez-Lozano, J.J. (2020). Reactive Navigation on Natural Environments by Continuous Classification of Ground Traversability. Sensors, 20.","DOI":"10.3390\/s20226423"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"S\u00e1nchez, M., Morales, J., Mart\u00ednez, J.L., Fern\u00e1ndez-Lozano, J.J., and Garc\u00eda-Cerezo, A. (2022). Automatically Annotated Dataset of a Ground Mobile Robot in Natural Environments via Gazebo Simulations. Sensors, 22.","DOI":"10.3390\/s22155599"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Cheng, X., Zhang, S., Cheng, S., Xia, Q., and Zhang, J. (2022). Path-Following and Obstacle Avoidance Control of Nonholonomic Wheeled Mobile Robot Based on Deep Reinforcement Learning. Appl. Sci., 12.","DOI":"10.3390\/app12146874"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Doukhi, O., and Lee, D.J. (2021). Deep Reinforcement Learning for End-to-End Local Motion Planning of Autonomous Aerial Robots in Unknown Outdoor Environments: Real-Time Flight Experiments. Sensors, 21.","DOI":"10.3390\/s21072534"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zeng, J., Ju, R., Qin, L., Hu, Y., Yin, Q., and Hu, C. (2019). Navigation in Unknown Dynamic Environments Based on Deep Reinforcement Learning. Sensors, 19.","DOI":"10.3390\/s19183837"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gao, J., Ye, W., Guo, J., and Li, Z. (2020). Deep Reinforcement Learning for Indoor Mobile Robot Path Planning. Sensors, 20.","DOI":"10.3390\/s20195493"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s10846-021-01367-5","article-title":"Soft Actor\u2013Critic for Navigation of Mobile Robots","volume":"102","author":"Kich","year":"2021","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_32","unstructured":"Chen, Y., Chen, G., Pan, L., Ma, J., Zhang, Y., Zhang, Y., and Ji, J. (October, January 27). DRQN-based 3D Obstacle Avoidance with a Limited Field of View. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic."},{"key":"ref_33","unstructured":"Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., and Farhadi, A. (June, January 29). Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yokoyama, K., and Morioka, K. (2020, January 12\u201315). Autonomous Mobile Robot with Simple Navigation System Based on Deep Reinforcement Learning and a Monocular Camera. Proceedings of the IEEE\/SICE International Symposium on System Integration (SII), Honolulu, HI, USA.","DOI":"10.1109\/SII46433.2020.9025987"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6569","DOI":"10.1109\/LRA.2021.3093551","article-title":"A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain","volume":"6","author":"Hu","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Luo, S., Kasaei, H., and Schomaker, L. (2020, January 19\u201324). Accelerating Reinforcement Learning for Reaching Using Continuous Curriculum Learning. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207427"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Xue, H., Hein, B., Bakr, M., Schildbach, G., Abel, B., and Rueckert, E. (2022). Using Deep Reinforcement Learning with Automatic Curriculum Learning for Mapless Navigation in Intralogistics. Appl. Sci., 12.","DOI":"10.3390\/app12063153"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Anzalone, L., Barra, S., and Nappi, M. (2021, January 19\u201322). Reinforced Curriculum Learning For Autonomous Driving in Carla. Proceedings of the IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506673"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez, J.L., Morales, J., Reina, A., Mandow, A., Peque\u00f1o Boter, A., and Garcia-Cerezo, A. (2015, January 17\u201319). Construction and calibration of a low-cost 3D laser scanner with 360\u00b0 field of view for mobile robots. Proceedings of the IEEE International Conference on Industrial Technology (ICIT), Seville, Spain.","DOI":"10.1109\/ICIT.2015.7125091"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/6\/3239\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:58:18Z","timestamp":1760122698000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/6\/3239"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,18]]},"references-count":39,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["s23063239"],"URL":"https:\/\/doi.org\/10.3390\/s23063239","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,18]]}}}