{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T15:12:42Z","timestamp":1780585962893,"version":"3.54.1"},"reference-count":27,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Foundation of National Natural Science Foundation of China under Grant","award":["61973065"],"award-info":[{"award-number":["61973065"]}]},{"name":"the Foundation of National Natural Science Foundation of China under Grant","award":["52075531"],"award-info":[{"award-number":["52075531"]}]},{"name":"the Fundamental Research Funds for the Central Universities of China under Grant","award":["N182612002"],"award-info":[{"award-number":["N182612002"]}]},{"name":"the Fundamental Research Funds for the Central Universities of China under Grant","award":["N2026002"],"award-info":[{"award-number":["N2026002"]}]},{"name":"the Central Government Guides The Local Science And Technology Development Special Fund","award":["2021JH6\/10500129"],"award-info":[{"award-number":["2021JH6\/10500129"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, a novel path planning algorithm with Reinforcement Learning is proposed based on the topological map. The proposed algorithm has a two-level structure. At the first level, the proposed method generates the topological area using the region dynamic growth algorithm based on the grid map. In the next level, the Multi-SARSA method divided into two layers is applied to find a near-optimal global planning path, in which the artificial potential field method, first of all, is used to initialize the first Q table for faster learning speed, and then the second Q table is initialized with the connected domain obtained by topological map, which provides the prior information. A combination of the two algorithms makes the algorithm easier to converge. Simulation experiments for path planning have been executed. The results indicate that the method proposed in this paper can find the optimal path with a shorter path length, which demonstrates the effectiveness of the presented method.<\/jats:p>","DOI":"10.3390\/s22062367","type":"journal-article","created":{"date-parts":[[2022,3,20]],"date-time":"2022-03-20T21:37:17Z","timestamp":1647812237000},"page":"2367","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["A Hierarchical Path Planning Approach with Multi-SARSA Based on Topological Map"],"prefix":"10.3390","volume":"22","author":[{"given":"Shiguang","family":"Wen","sequence":"first","affiliation":[{"name":"Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110169, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9694-7001","authenticated-orcid":false,"given":"Yufan","family":"Jiang","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Cui","sequence":"additional","affiliation":[{"name":"Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110169, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ke","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8296-8039","authenticated-orcid":false,"given":"Fei","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110169, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/j.neucom.2014.09.092","article-title":"A hierarchical path planning approach based on A* and least-squares policy iteration for mobile robots","volume":"170","author":"Zuo","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"106796","DOI":"10.1016\/j.asoc.2020.106796","article-title":"Optimal path planning approach based on Q-learning algorithm for mobile robots","volume":"97","author":"Maoudj","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1007\/s10845-021-01867-z","article-title":"A review of motion planning algorithms for intelligent robots","volume":"33","author":"Zhou","year":"2022","journal-title":"J. Intell. Manuf."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3081","DOI":"10.1109\/LRA.2018.2849610","article-title":"Efficient Object Search With Belief Road Map Using Mobile Robot","volume":"3","author":"Wang","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_5","unstructured":"Zou, Q., Zhang, Y., and Liu, S. (2020, January 22\u201324). A path planning algorithm based on RRT and SARSA (\u03bb) in unknown and complex conditions. Proceedings of the 2020 Chinese Control And Decision Conference (CCDC), Hefei, China."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liao, X., Wang, Y., Xuan, Y., and Wu, D. (2020, January 6\u20138). AGV Path Planning Model based on Reinforcement Learning. Proceedings of the 2020 Chinese Automation Congress (CAC), Shanghai, China.","DOI":"10.1109\/CAC51589.2020.9326742"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Gu, S., and Mao, G. (2020, January 3\u20134). An Improved Q-Learning Algorithm for Path Planning in Maze Environments. Proceedings of the SAI Intelligent Systems Conference, London, UK.","DOI":"10.1007\/978-3-030-55187-2_40"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.robot.2019.02.013","article-title":"Solving the optimal path planning of a mobile robot using improved Q-learning","volume":"115","author":"Low","year":"2019","journal-title":"Robot. Auton. Syst."},{"key":"ref_9","unstructured":"Wang, J., Hirota, K., Wu, X., Dai, Y., and Jia, Z. (November, January 31). An improved Q-learning algorithm for mobile robot path planning. Proceedings of the 9th International Symposium on Computational Intelligence and Industrial Applications (ISCIIA2020), Beijing, China."},{"key":"ref_10","unstructured":"Khatib, O. (1985, January 25\u201328). Real-time obstacle avoidance system for manipulators and mobile robots. Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, MO, USA."},{"key":"ref_11","unstructured":"LaValle, S.M. (2022, February 16). Rapidly-Exploring Random Trees: A New Tool for Path Planning. Available online: https:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.35.1853."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Geng, Y., Liu, E., Wang, R., Liu, Y., Rao, W., Feng, S., Dong, Z., Fu, Z., and Chen, Y. (2021, January 14\u201323). Deep Reinforcement Learning Based Dynamic Route Planning for Minimizing Travel Time. Proceedings of the 2021 IEEE International Conference on Communications Workshops (ICC Workshops), Montreal, QC, Canada.","DOI":"10.1109\/ICCWorkshops50388.2021.9473555"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1109\/70.508439","article-title":"Probabilistic roadmaps for path planning in high-dimensional configuration spaces","volume":"12","author":"Kavraki","year":"1996","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mohan, P., Sharma, L., and Narayan, P. (2021, January 6\u20138). Optimal Path Finding using Iterative SARSA. Proceedings of the 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.","DOI":"10.1109\/ICICCS51141.2021.9432202"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"67319","DOI":"10.1109\/ACCESS.2019.2918703","article-title":"Path planning via an improved DQN-based learning policy","volume":"7","author":"Lv","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","unstructured":"Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. [Ph.D. Thesis, Cambridge University]."},{"key":"ref_17","unstructured":"Sutton, R.S. (1995, January 27\u201330). Generalization in reinforcement learning: Successful examples using sparse coarse coding. Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.robot.2016.11.007","article-title":"Integrated online trajectory planning and optimization in distinctive topologies","volume":"88","author":"Hoffmann","year":"2017","journal-title":"Robot. Auton. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1898","DOI":"10.1016\/j.sbspro.2013.08.215","article-title":"A hierarchical path planning method using the experience of taxi drivers","volume":"96","author":"Hu","year":"2013","journal-title":"Procedia-Soc. Behav. Sci."},{"key":"ref_20","unstructured":"Wang, C., Soh, Y., Wang, H., and Wang, H. (2002, January 12\u201315). A hierarchical genetic algorithm for path planning in a static environment with obstacles. Proceedings of the IEEE CCECE2002. Canadian Conference on Electrical and Computer Engineering. Conference Proceedings (Cat. No. 02CH37373), Winnipeg, MB, Canada."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1109\/70.68066","article-title":"New heuristic algorithms for efficient hierarchical path planning","volume":"7","author":"Zhu","year":"1991","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1214","DOI":"10.1109\/TSMCB.2005.850177","article-title":"A layered goal-oriented fuzzy motion planning strategy for mobile robot navigation","volume":"35","author":"Yang","year":"2005","journal-title":"IEEE Trans. Syst. Man Cybern. Part B (Cybern.)"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1109\/JRA.1986.1087051","article-title":"Multiresolution path planning for mobile robots","volume":"2","author":"Kambhampati","year":"1986","journal-title":"IEEE J. Robot. Autom."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, F., Liu, Y., Xiao, L., Wu, C., and Chu, H. (2019). Topological map construction based on region dynamic growing and map representation method. Appl. Sci., 9.","DOI":"10.3390\/app9050816"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/100.580977","article-title":"The dynamic window approach to collision avoidance","volume":"4","author":"Fox","year":"1997","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1109\/TSSC.1968.300136","article-title":"A formal basis for the heuristic determination of minimum cost paths","volume":"4","author":"Hart","year":"1968","journal-title":"IEEE Trans. Syst. Sci. Cybern."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/0004-3702(85)90084-0","article-title":"Depth-first iterative-deepening: An optimal admissible tree search","volume":"27","author":"Korf","year":"1985","journal-title":"Artif. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2367\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:39:07Z","timestamp":1760135947000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2367"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,18]]},"references-count":27,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22062367"],"URL":"https:\/\/doi.org\/10.3390\/s22062367","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,18]]}}}