{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T16:16:29Z","timestamp":1784736989571,"version":"3.55.0"},"reference-count":29,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T00:00:00Z","timestamp":1768953600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Deep reinforcement learning (DRL) has shown great potential in path planning tasks. However, in sparse reward environments, DRL still faces significant challenges such as low training efficiency and a tendency to converge to suboptimal policies. Traditional reward shaping methods can partially alleviate these issues, but they typically rely on hand-crafted designs, which often introduce complex reward coupling, make hyperparameter tuning difficult, and limit generalization capability. To address these challenges, this paper proposes Curriculum-guided Learning with Adaptive Reward Shaping for Deep Q-Network (CLARS-DQN), a path planning algorithm that integrates Adaptive Reward Shaping (ARS) and Curriculum Learning (CL). The algorithm consists of two key components: (1) ARS-DQN, which augments the DQN framework with a learnable intrinsic reward function to reduce reward sparsity and dependence on expert knowledge; and (2) a curriculum strategy that guides policy optimization through a staged training process, progressing from simple to complex tasks to enhance generalization. Training also incorporates Prioritized Experience Replay (PER) to improve sample efficiency and training stability. CLARS-DQN outperforms baseline methods in task success rate, path quality, training efficiency, and hyperparameter robustness. In unseen environments, the method improves task success rate and average path length by 12% and 26%, respectively, demonstrating strong generalization. Ablation studies confirm the critical contribution of each module.<\/jats:p>","DOI":"10.3390\/a19010089","type":"journal-article","created":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T08:48:08Z","timestamp":1768985288000},"page":"89","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Path Planning in Sparse Reward Environments: A DQN Approach with Adaptive Reward Shaping and Curriculum Learning"],"prefix":"10.3390","volume":"19","author":[{"given":"Hongyi","family":"Yang","sequence":"first","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5261-0191","authenticated-orcid":false,"given":"Bo","family":"Cai","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunlong","family":"Li","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"6932","DOI":"10.1109\/LRA.2020.3026638","article-title":"Mobile robot path planning in dynamic environments through globally guided reinforcement learning","volume":"5","author":"Wang","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"127958","DOI":"10.1016\/j.neucom.2024.127958","article-title":"EPPE: An Efficient Progressive Policy Enhancement framework of deep reinforcement learning in path planning","volume":"596","author":"Zhao","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"110543","DOI":"10.1016\/j.asoc.2023.110543","article-title":"Deep reinforcement learning for optimal rescue path planning in uncertain and complex urban pluvial flood scenarios","volume":"144","author":"Li","year":"2023","journal-title":"Appl. Soft Comput."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lin, X., McConnell, J., and Englot, B. (2023, January 1\u20135). Robust unmanned surface vehicle navigation with distributional reinforcement learning. Proceedings of the 2023 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA.","DOI":"10.1109\/IROS55552.2023.10342389"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/LRA.2022.3233261","article-title":"Policy optimization to learn adaptive motion primitives in path planning with dynamic obstacles","volume":"8","author":"Angulo","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"147755","DOI":"10.1109\/ACCESS.2019.2946659","article-title":"Improved multi-agent deep deterministic policy gradient for path planning-based crowd simulation","volume":"7","author":"Zheng","year":"2019","journal-title":"IEEE Access"},{"key":"ref_7","first-page":"182","article-title":"Survey on sparse reward in deep reinforcement learning","volume":"47","author":"Ang","year":"2020","journal-title":"Comput. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"175473","DOI":"10.1109\/ACCESS.2024.3504735","article-title":"Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications","volume":"12","author":"Ibrahim","year":"2024","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2393","DOI":"10.1109\/TII.2019.2936167","article-title":"End-to-end navigation strategy with deep reinforcement learning for mobile robots","volume":"16","author":"Shi","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_10","unstructured":"Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. (2017, January 4\u20139). #Exploration: A study of count-based exploration for deep reinforcement learning. Proceedings of the Advances in Neural Information Processing Systems 30 (NeurIPS), Long Beach, CA, USA."},{"key":"ref_11","unstructured":"Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"110601","DOI":"10.1016\/j.asoc.2023.110601","article-title":"Immune deep reinforcement learning-based path planning for mobile robot in unknown environment","volume":"145","author":"Yan","year":"2023","journal-title":"Appl. Soft Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"22153","DOI":"10.1109\/JIOT.2024.3379361","article-title":"Path planning of autonomous mobile robot in comprehensive unknown environment using deep reinforcement learning","volume":"11","author":"Bai","year":"2024","journal-title":"IEEE Internet Things J."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"115208","DOI":"10.1016\/j.oceaneng.2023.115208","article-title":"Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships","volume":"284","author":"Wu","year":"2023","journal-title":"Ocean Eng."},{"key":"ref_15","unstructured":"Zhang, C., Vinyals, O., Munos, R., and Bengio, S. (2018). A study on overfitting in deep reinforcement learning. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hafiz, A.M. (2022). A survey of deep q-networks used for reinforcement learning: State of the art. Intelligent Communication Technologies and Virtual Mobile Networks: Proceedings of ICICV 2022, Tirunelveli, India, 10\u201311 February 2022, Springer.","DOI":"10.1007\/978-981-19-1844-5_30"},{"key":"ref_17","unstructured":"Cederle, M., Fabris, M., and Susto, G.A. (2024, January 19\u201324). A Distributed Approach to Autonomous Intersection Management via Multi-Agent Reinforcement Learning. Proceedings of the 27th European Conference on Artificial Intelligence, Santiago de Compostela, Spain."},{"key":"ref_18","first-page":"5829","article-title":"Exploration-guided reward shaping for reinforcement learning under sparse rewards","volume":"35","author":"Devidze","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_19","unstructured":"Zheng, Z., Oh, J., and Singh, S. (2018, January 3\u20138). On learning intrinsic rewards for policy gradient methods. Proceedings of the Advances in Neural Information Processing Systems 31, Montr\u00e9al, QC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"123202","DOI":"10.1016\/j.eswa.2024.123202","article-title":"Autonomous navigation of mobile robots in unknown environments using off-policy reinforcement learning with curriculum learning","volume":"247","author":"Yin","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_21","unstructured":"Sutton, R.S., McAllester, D., Singh, S., and Mansour, Y. (December, January 29). Policy gradient methods for reinforcement learning with function approximation. Proceedings of the Advances in Neural Information Processing Systems 12, Denver, CO, USA."},{"key":"ref_22","unstructured":"Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M.W., Pfau, D., Schaul, T., Shillingford, B., and De Freitas, N. (2016, January 5\u201310). Learning to Learn by Gradient Descent by Gradient Descent. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_23","unstructured":"Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016, January 20\u201322). Meta-Learning with Memory-Augmented Neural Networks. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_24","unstructured":"Nichol, A., Achiam, J., and Schulman, J. (2018). On First-Order Meta-Learning Algorithms. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"109112","DOI":"10.1016\/j.cie.2023.109112","article-title":"A* guiding DQN algorithm for automated guided vehicle pathfinding problem of robotic mobile fulfillment systems","volume":"178","author":"Luo","year":"2023","journal-title":"Comput. Ind. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"108069","DOI":"10.1016\/j.ijpe.2021.108069","article-title":"A method for efficiently routing order pickers in the leaf warehouse","volume":"234","author":"Masae","year":"2021","journal-title":"Int. J. Prod. Econ."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1109\/TAMD.2010.2051031","article-title":"Intrinsically motivated reinforcement learning: An evolutionary perspective","volume":"2","author":"Singh","year":"2010","journal-title":"IEEE Trans. Auton. Ment. Dev."},{"key":"ref_28","unstructured":"Ng, A.Y., Harada, D., and Russell, S. (1999, January 27\u201330). Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of the International Conference on Machine Learning, Bled, Slovenia."},{"key":"ref_29","unstructured":"Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., and Pontil, M. (2018, January 10\u201315). Bilevel programming for hyperparameter optimization and meta-learning. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/89\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T08:52:58Z","timestamp":1768985578000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/89"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["a19010089"],"URL":"https:\/\/doi.org\/10.3390\/a19010089","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]}}}