{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T09:02:15Z","timestamp":1790672535809,"version":"4.1.0"},"reference-count":48,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T00:00:00Z","timestamp":1790294400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100011785","name":"Guangxi Natural Science Foundation","doi-asserted-by":"publisher","award":["2025GXNSFHA069207"],"award-info":[{"award-number":["2025GXNSFHA069207"]}],"id":[{"id":"10.13039\/501100011785","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Innovation Project of Guangxi Graduate Education","award":["YCSW2026634"],"award-info":[{"award-number":["YCSW2026634"]}]},{"award":["YCSW2026634"],"award-info":[{"award-number":["YCSW2026634"]}],"id":[{"id":"https:\/\/ror.org\/01fjs1443","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mapless autonomous navigation in unknown multi-obstacle environments remains challenging for deep reinforcement learning due to limited perception, sparse rewards, and distribution shifts between training and deployment scenarios. Existing methods mainly rely on instantaneous observations or individual enhancement strategies, which often struggle to balance exploration robustness and navigation efficiency. This paper proposes a Curriculum Learning and Intrinsic Curiosity-enhanced Temporal Proximal Policy Optimization framework, referred to as CLI-PPO, for end-to-end autonomous navigation. The framework integrates temporal state representation, intrinsic exploration, and progressive task scheduling into PPO, where an LSTM network improves sequential decision-making under partial observability, an Intrinsic Curiosity Module enhances exploration in unfamiliar regions, and Curriculum Learning stabilizes policy optimization by gradually increasing task difficulty. A hybrid reward function combining target guidance, obstacle avoidance, and exploration incentives is designed for policy training. Experiments were conducted using ablation configurations, three independent training seeds, and an evaluation of one SAC policy under the corresponding task protocol, with 5000 randomized episodes considered at each obstacle-density level. CLI-PPO achieved a success rate of 66.24%, a collision rate of 33.58%, and an average return of 3433.14, compared with 46.06%, 53.94%, and 2083.21 for PPO, respectively. In the 10-obstacle setting, the three training seeds produced a CLI-PPO success rate of 62.72\u00b13.59% and an average return of 3337.12\u00b148.93, while PPO+ICM+Curriculum achieved 63.83\u00b16.61% success and a slightly higher Euclidean-reference SPL. As obstacle density increased, PPO+ICM+Curriculum maintained higher success and SPL, whereas CLI-PPO yielded higher returns and more consistent performance across training runs.<\/jats:p>","DOI":"10.3390\/s26196098","type":"journal-article","created":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T14:36:34Z","timestamp":1790346994000},"page":"6098","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["CLI-PPO: A Memory-Augmented PPO Framework with Intrinsic Curiosity and Curriculum Learning for Mapless Multi-Obstacle Navigation"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-8564-5856","authenticated-orcid":false,"given":"Qifan","family":"Kang","sequence":"first","affiliation":[{"name":"School of Automation, Guangxi University of Science and Technology, Liuzhou 545006, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6868-6542","authenticated-orcid":false,"given":"Shigang","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Automation, Guangxi University of Science and Technology, Liuzhou 545006, China"},{"name":"Guangxi Low-Altitude Unmanned Aircraft Key Technologies Engineering Research Center, Liuzhou 545616, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5182-4656","authenticated-orcid":false,"given":"Debing","family":"Xie","sequence":"additional","affiliation":[{"name":"School of Automation, Guangxi University of Science and Technology, Liuzhou 545006, China"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"methodology","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"100722","DOI":"10.1016\/j.dajour.2026.100722","article-title":"A critical review of analytical approaches to path planning for autonomous mobile robots","volume":"20","author":"Shareef","year":"2026","journal-title":"Decis. Anal. J."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zhang, D., Chen, C., and Zhang, G. (2024, January 15\u201317). AGV Path Planning Based on Improved A-star Algorithm. Proceedings of the 2024 IEEE 7th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China.","DOI":"10.1109\/IAEAC59436.2024.10503919"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"161920","DOI":"10.1109\/ACCESS.2020.3021073","article-title":"Research and Optimization of D-Start Lite Algorithm in Track Planning","volume":"8","author":"Xie","year":"2020","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, S., Ran, Y., and Chen, Z. (2025). An Improved Bi-RRT Algorithm for Optimal Puncture Path Planning. Algorithms, 18.","DOI":"10.3390\/a18110702"},{"key":"ref_5","unstructured":"Wang, H., Yu, Y., and Yuan, Q. (2011, January 15\u201317). Application of Dijkstra algorithm in robot path-planning. Proceedings of the 2011 Second International Conference on Mechanic Automation and Control Engineering, Hohhot, China."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/j.protcy.2016.03.010","article-title":"Time-efficient A* Algorithm for Robot Path Planning","volume":"23","author":"Guruji","year":"2016","journal-title":"Procedia Technol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1109\/LRA.2019.2891991","article-title":"Deep Reinforcement Learning Robot for Search and Rescue Applications: Exploration in Unknown Cluttered Environments","volume":"4","author":"Niroui","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pan, X., Huang, L., Li, L., and Wang, S. (2026). Deep Reinforcement Learning-Based Path-Following Control for Underactuated Autonomous Underwater Vehicles. Sensors, 26.","DOI":"10.3390\/s26144548"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhang, T., Zhou, F., Liu, F., Qian, B., and Liu, J. (2026). Enhanced Soft Actor\u2013Critic with Dual-Path Channel Attention for UAV Autonomous Navigation in Complex Environments. Appl. Sci., 16.","DOI":"10.3390\/app16147159"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1540","DOI":"10.1016\/j.ifacol.2025.09.259","article-title":"Comparative Analysis of PPO and DQN for UAV Obstacle Avoidance in Simulated Environments","volume":"59","author":"Junior","year":"2025","journal-title":"IFAC-PapersOnLine"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cui, S., Li, H., Su, Y., Huang, J., Cheng, K., and Li, H. (2026). A Rule-Guided Distributional Soft Actor\u2013Critic Algorithm for Safe Lane-Changing in Complex Driving Scenarios. Vehicles, 8.","DOI":"10.3390\/vehicles8030058"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhou, X., Li, S., Xia, Q., and Li, S. (2026). Executable Reference Trajectory Construction and Conflict-Aware Residual Reinforcement Learning for Urban Multi-UAV Navigation. Aerospace, 13.","DOI":"10.3390\/aerospace13070636"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Liao, P., Meng, L., Guo, P., and Li, B. (2026). An Interval Incentive and Predictive Interpolation-Based PPO Method for AGV Path Planning. Actuators, 15.","DOI":"10.3390\/act15040198"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1016\/j.neunet.2008.02.003","article-title":"Reinforcement learning of motor skills with policy gradients","volume":"21","author":"Peters","year":"2008","journal-title":"Neural Netw."},{"key":"ref_15","unstructured":"Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. arXiv."},{"key":"ref_16","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2019). Deep Reinforcement Learning that Matters. arXiv.","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1007\/s40998-024-00782-2","article-title":"Robust H\u221e Output Consensus in Heterogeneous Multi-agent Discrete-Time Systems Using Q-Learning Algorithm","volume":"49","author":"Valadbeigi","year":"2025","journal-title":"Iran. J. Sci. Technol. Trans. Electr. Eng."},{"key":"ref_19","first-page":"3003514","article-title":"Real-Time UUV Obstacle Avoidance Through Flexible Steering Technology Based on Improved Soft Actor-Critic Framework","volume":"74","author":"Wang","year":"2025","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"16132","DOI":"10.1109\/TIE.2024.3393153","article-title":"Finite-Time Control Design for a Coaxial Tilt-Rotor UAV","volume":"71","author":"Lv","year":"2024","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"113096","DOI":"10.1016\/j.patcog.2026.113096","article-title":"Adaptive saliency based contextual metric learning for few-shot open-set recognition","volume":"175","author":"Li","year":"2026","journal-title":"Pattern Recognit."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/S0004-3702(98)00023-X","article-title":"Planning and acting in partially observable stochastic domains","volume":"101","author":"Kaelbling","year":"1998","journal-title":"Artif. Intell."},{"key":"ref_23","unstructured":"Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2018). Hindsight Experience Replay. arXiv."},{"key":"ref_24","unstructured":"Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2018). Diversity is All You Need: Learning Skills without a Reward Function. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"131036","DOI":"10.1016\/j.neucom.2025.131036","article-title":"Advancing mobile robot navigation with DRL and heuristic rewards: A comprehensive review","volume":"652","author":"Khan","year":"2025","journal-title":"Neurocomputing"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"121340","DOI":"10.1109\/ACCESS.2022.3223382","article-title":"AUV Dynamic Obstacle Avoidance Method Based on Improved PPO Algorithm","volume":"10","author":"Zhu","year":"2022","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"6544029","DOI":"10.1155\/2023\/6544029","article-title":"Intelligent Navigation of Indoor Robot Based on Improved DDPG Algorithm","volume":"2023","author":"He","year":"2023","journal-title":"Math. Probl. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Yu, Z., Guo, H., Adiwahono, A.H., Chan, J., Tynn, B.S.W., Chew, C.M., and Yau, W.Y. (2024). Multi-Robot Reliable Navigation in Uncertain Topological Environments with Graph Attention Networks. arXiv.","DOI":"10.1109\/LRA.2025.3557751"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Xu, Y., Zhang, B., Yang, F., Chai, J., Lu, Q., and Choi, Y. (2023, January 16\u201319). UAV Agile Navigation Method for Unknown Environment via Deep Reinforcement Learning. Proceedings of the IECON 2023\u201449th Annual Conference of the IEEE Industrial Electronics Society, Singapore.","DOI":"10.1109\/IECON51785.2023.10312262"},{"key":"ref_30","unstructured":"Pleines, M., Pallasch, M., Zimmer, F., and Preuss, M. (2022). Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.ifacol.2025.11.620","article-title":"Automatic Ship Collision Avoidance Decision-making Method via Improved PPO Algorithm","volume":"59","author":"Hao","year":"2025","journal-title":"IFAC-PapersOnLine"},{"key":"ref_32","unstructured":"Zhou, S.K., Rueckert, D., and Fichtinger, G. (2020). Chapter 21\u2014Deep learning: RNNs and LSTM. Handbook of Medical Image Computing and Computer Assisted Intervention, Academic Press."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1162\/neco_a_01199","article-title":"A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures","volume":"31","author":"Yu","year":"2019","journal-title":"Neural Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning to Forget: Continual Prediction with LSTM","volume":"12","author":"Gers","year":"2000","journal-title":"Neural Comput."},{"key":"ref_36","unstructured":"Staudemeyer, R.C., and Morris, E.R. (2019). Understanding LSTM\u2014A tutorial into Long Short-Term Memory Recurrent Neural Networks. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kim, J., El-Khamy, M., and Lee, J. (2017). Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition. arXiv.","DOI":"10.21437\/Interspeech.2017-477"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Tao, F., and Liu, G. (2017). Advanced LSTM: A Study about Better Time Dependency Modeling in Emotion Recognition. arXiv.","DOI":"10.1109\/ICASSP.2018.8461750"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"110633","DOI":"10.1016\/j.cie.2024.110633","article-title":"Probing an LSTM-PPO-Based reinforcement learning algorithm to solve dynamic job shop scheduling problem","volume":"197","author":"Chen","year":"2024","journal-title":"Comput. Ind. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Pathak, D., Agrawal, P., Efros, A.A., and Darrell, T. (2017). Curiosity-driven Exploration by Self-supervised Prediction. arXiv.","DOI":"10.1109\/CVPRW.2017.70"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Shi, M., Zhang, T., and Zeng, Y. (2020). Corrigendum: A Curiosity-Based Learning Method for Spiking Neural Networks. Front. Comput. Neurosci., 14.","DOI":"10.3389\/fncom.2020.00028"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1007\/s10994-019-05845-8","article-title":"Skill-based curiosity for intrinsically motivated reinforcement learning","volume":"109","author":"Bougie","year":"2020","journal-title":"Mach. Learn."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1222","DOI":"10.1007\/s40815-020-01035-0","article-title":"A Fuzzy Curiosity-Driven Mechanism for Multi-Agent Reinforcement Learning","volume":"23","author":"Chen","year":"2021","journal-title":"Int. J. Fuzzy Syst."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Lefort, M., and Gepperth, A. (2015, January 13\u201316). Active learning of local predictable representations with artificial curiosity. Proceedings of the 2015 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), Providence, RI, USA.","DOI":"10.1109\/DEVLRN.2015.7346145"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s44443-025-00215-y","article-title":"Efficient evolutionary curriculum learning for scalable multi-agent reinforcement learning","volume":"37","author":"Li","year":"2025","journal-title":"J. King Saud Univ. Comput. Inf. Sci."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Yang, H., Pasupa, K., Leung, A.C.S., Kwok, J.T., Chan, J.H., and King, I. (2020). Automatic Curriculum Generation by Hierarchical Reinforcement Learning. Proceedings of the Neural Information Processing, Springer.","DOI":"10.1007\/978-3-030-63820-7"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Foglino, F., Christakou, C.C., and Leonetti, M. (2019). An Optimization Framework for Task Sequencing in Curriculum Learning. arXiv.","DOI":"10.1109\/DEVLRN.2019.8850690"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Kong, Y., Liu, L., Wang, J., and Tao, D. (2021, January 10\u201317). Adaptive Curriculum Learning. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00502"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6098\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T08:16:44Z","timestamp":1790669804000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/19\/6098"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,25]]},"references-count":48,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["s26196098"],"URL":"https:\/\/doi.org\/10.3390\/s26196098","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,25]]}}}