{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,5]],"date-time":"2026-04-05T08:15:09Z","timestamp":1775376909721,"version":"3.50.1"},"reference-count":28,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T00:00:00Z","timestamp":1686787200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62272426"],"award-info":[{"award-number":["62272426"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62106238"],"award-info":[{"award-number":["62106238"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["202201150401021"],"award-info":[{"award-number":["202201150401021"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["202104021301055"],"award-info":[{"award-number":["202104021301055"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["447-110103"],"award-info":[{"award-number":["447-110103"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Shanxi Province Science and Technology Major Special Project","award":["62272426"],"award-info":[{"award-number":["62272426"]}]},{"name":"Shanxi Province Science and Technology Major Special Project","award":["62106238"],"award-info":[{"award-number":["62106238"]}]},{"name":"Shanxi Province Science and Technology Major Special Project","award":["202201150401021"],"award-info":[{"award-number":["202201150401021"]}]},{"name":"Shanxi Province Science and Technology Major Special Project","award":["202104021301055"],"award-info":[{"award-number":["202104021301055"]}]},{"name":"Shanxi Province Science and Technology Major Special Project","award":["447-110103"],"award-info":[{"award-number":["447-110103"]}]},{"name":"Shanxi Province Science and Technology Achievements Transformation Guidance Special Project","award":["62272426"],"award-info":[{"award-number":["62272426"]}]},{"name":"Shanxi Province Science and Technology Achievements Transformation Guidance Special Project","award":["62106238"],"award-info":[{"award-number":["62106238"]}]},{"name":"Shanxi Province Science and Technology Achievements Transformation Guidance Special Project","award":["202201150401021"],"award-info":[{"award-number":["202201150401021"]}]},{"name":"Shanxi Province Science and Technology Achievements Transformation Guidance Special Project","award":["202104021301055"],"award-info":[{"award-number":["202104021301055"]}]},{"name":"Shanxi Province Science and Technology Achievements Transformation Guidance Special Project","award":["447-110103"],"award-info":[{"award-number":["447-110103"]}]},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality","award":["62272426"],"award-info":[{"award-number":["62272426"]}]},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality","award":["62106238"],"award-info":[{"award-number":["62106238"]}]},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality","award":["202201150401021"],"award-info":[{"award-number":["202201150401021"]}]},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality","award":["202104021301055"],"award-info":[{"award-number":["202104021301055"]}]},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality","award":["447-110103"],"award-info":[{"award-number":["447-110103"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the advancement of robotics, the field of path planning is currently experiencing a period of prosperity. Researchers strive to address this nonlinear problem and have achieved remarkable results through the implementation of the Deep Reinforcement Learning (DRL) algorithm DQN (Deep Q-Network). However, persistent challenges remain, including the curse of dimensionality, difficulties of model convergence and sparsity in rewards. To tackle these problems, this paper proposes an enhanced DDQN (Double DQN) path planning approach, in which the information after dimensionality reduction is fed into a two-branch network that incorporates expert knowledge and an optimized reward function to guide the training process. The data generated during the training phase are initially discretized into corresponding low-dimensional spaces. An \u201cexpert experience\u201d module is introduced to facilitate the model\u2019s early-stage training acceleration in the Epsilon\u2013Greedy algorithm. To tackle navigation and obstacle avoidance separately, a dual-branch network structure is presented. We further optimize the reward function enabling intelligent agents to receive prompt feedback from the environment after performing each action. Experiments conducted in both virtual and real-world environments have demonstrated that the enhanced algorithm can accelerate model convergence, improve training stability and generate a smooth, shorter and collision-free path.<\/jats:p>","DOI":"10.3390\/s23125622","type":"journal-article","created":{"date-parts":[[2023,6,16]],"date-time":"2023-06-16T02:54:33Z","timestamp":1686884073000},"page":"5622","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Improved Robot Path Planning Method Based on Deep Reinforcement Learning"],"prefix":"10.3390","volume":"23","author":[{"given":"Huiyan","family":"Han","sequence":"first","affiliation":[{"name":"School of Computer Science and Technology, North University of China, Taiyuan 030051, China"},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality, Taiyuan 030051, China"},{"name":"Shanxi Vision Information Processing and Intelligent Robot Engineering Research Center, Taiyuan 030051, China"}]},{"given":"Jiaqi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, North University of China, Taiyuan 030051, China"},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality, Taiyuan 030051, China"},{"name":"Shanxi Vision Information Processing and Intelligent Robot Engineering Research Center, Taiyuan 030051, China"}]},{"given":"Liqun","family":"Kuang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, North University of China, Taiyuan 030051, China"},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality, Taiyuan 030051, China"},{"name":"Shanxi Vision Information Processing and Intelligent Robot Engineering Research Center, Taiyuan 030051, China"}]},{"given":"Xie","family":"Han","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, North University of China, Taiyuan 030051, China"},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality, Taiyuan 030051, China"},{"name":"Shanxi Vision Information Processing and Intelligent Robot Engineering Research Center, Taiyuan 030051, China"}]},{"given":"Hongxin","family":"Xue","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, North University of China, Taiyuan 030051, China"},{"name":"Shanxi Key Laboratory of Machine Vision and Virtual Reality, Taiyuan 030051, China"},{"name":"Shanxi Vision Information Processing and Intelligent Robot Engineering Research Center, Taiyuan 030051, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.robot.2016.08.001","article-title":"Heuristic Approaches in Robot Path Planning","volume":"86","author":"Mac","year":"2016","journal-title":"Robot. Auton. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yin, Y., Chen, Z., Liu, G., and Guo, J. (2023). A Mapless Local Path Planning Approach Using Deep Reinforcement Learning Framework. Sensors, 23.","DOI":"10.3390\/s23042036"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Prianto, E., Kim, M., Park, J.-H., Bae, J.-H., and Kim, J.-S. (2020). Path Planning for Multi-Arm Manipulators Using Deep Reinforcement Learning: Soft Actor\u2013Critic with Hindsight Experience Replay. Sensors, 20.","DOI":"10.3390\/s20205911"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1007\/s10845-021-01867-z","article-title":"A Review of Motion Planning Algorithms for Intelligent Robots","volume":"33","author":"Zhou","year":"2022","journal-title":"J. Intell. Manuf."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"63","DOI":"10.3389\/fnbot.2020.00063","article-title":"The Path Planning of Mobile Robot by Neural Networks and Hierarchical Reinforcement Learning","volume":"14","author":"Yu","year":"2020","journal-title":"Front. Neurorobot."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1405","DOI":"10.1109\/TII.2021.3088407","article-title":"Adaptive Digital Twin and Multiagent Deep Reinforcement Learning for Vehicular Edge Computing and Networks","volume":"18","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"147755","DOI":"10.1109\/ACCESS.2019.2946659","article-title":"Improved Multi-Agent Deep Deterministic Policy Gradient for Path Planning-Based Crowd Simulation","volume":"7","author":"Zheng","year":"2019","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"(2023). Study of Convolutional Neural Network-Based Semantic Segmentation Methods on Edge Intelligence Devices for Field Agricultural Robot Navigation Line Extraction. Comput. Electron. Agric., 209, 107811.","DOI":"10.1016\/j.compag.2023.107811"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.neucom.2019.05.001","article-title":"TDPP-Net: Achieving Three-Dimensional Path Planning via a Deep Neural Network Architecture","volume":"357","author":"Wu","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.eswa.2016.06.021","article-title":"Neural Networks Based Reinforcement Learning for Mobile Robots Obstacle Avoidance","volume":"62","author":"Duguleana","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zeng, J., Qin, L., Hu, Y., Yin, Q., and Hu, C. (2019). Integrating a Path Planner and an Adaptive Motion Controller for Navigation in Dynamic Environments. Appl. Sci., 9.","DOI":"10.3390\/app9071384"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Chen, Y., and Liang, L. (2023). SLP-Improved DDPG Path-Planning Algorithm for Mobile Robot in Large-Scale Dynamic Environment. Sensors, 23.","DOI":"10.3390\/s23073521"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3826","DOI":"10.1109\/TCYB.2020.2977374","article-title":"Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications","volume":"50","author":"Nguyen","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1587\/transinf.2022DLP0057","article-title":"SPSD: Semantics and Deep Reinforcement Learning Based Motion Planning for Supermarket Robot","volume":"E106.D","author":"Cai","year":"2023","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"19111","DOI":"10.1109\/ACCESS.2023.3247730","article-title":"Path Planning for Mobile Robot Considering Turnabouts on Narrow Road by Deep Q-Network","volume":"11","author":"Nakamura","year":"2023","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2158","DOI":"10.1109\/LRA.2023.3246844","article-title":"Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments From Temporal Logic Specifications","volume":"8","author":"Cai","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-Level Control through Deep Reinforcement Learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_19","unstructured":"Puterman, M.L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Son."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"674","DOI":"10.26599\/TST.2021.9010012","article-title":"Deep Reinforcement Learning Based Mobile Robot Navigation: A Review","volume":"26","author":"Zhu","year":"2021","journal-title":"Tsinghua Sci. Technol."},{"key":"ref_21","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv, 05952."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2655","DOI":"10.1007\/s12555-018-0551-6","article-title":"Approximate Optimal Stabilization Control of Servo Mechanisms Based on Reinforcement Learning Scheme","volume":"17","author":"Lv","year":"2019","journal-title":"Int. J. Control Autom. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kubov\u010d\u00edk, M., Dirgov\u00e1 Lupt\u00e1kov\u00e1, I., and Posp\u00edchal, J. (2023). Signal Novelty Detection as an Intrinsic Reward for Robotics. Sensors, 23.","DOI":"10.3390\/s23083985"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gao, J., Ye, W., Guo, J., and Li, Z. (2020). Deep Reinforcement Learning for Indoor Mobile Robot Path Planning. Sensors, 20.","DOI":"10.3390\/s20195493"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1038\/s41586-020-03157-9","article-title":"First Return, Then Explore","volume":"590","author":"Ecoffet","year":"2021","journal-title":"Nature"},{"key":"ref_26","unstructured":"Liu, Y., Luo, Y., Zhong, Y., Chen, X., Liu, Q., and Peng, J. (2019). Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Pathak, D., Agrawal, P., Efros, A.A., and Darrell, T. (2017, January 17). Curiosity-Driven Exploration by Self-Supervised Prediction. Proceedings of the 34th International Conference on Machine Learning, PMLR, Sydney, Australia.","DOI":"10.1109\/CVPRW.2017.70"},{"key":"ref_28","unstructured":"Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. (2017, January 18). Reverse Curriculum Generation for Reinforcement Learning. Proceedings of the 1st Annual Conference on Robot Learning, PMLR, California, CA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5622\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:55:54Z","timestamp":1760126154000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5622"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,15]]},"references-count":28,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23125622"],"URL":"https:\/\/doi.org\/10.3390\/s23125622","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,15]]}}}