{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:19:20Z","timestamp":1777407560826,"version":"3.51.4"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,4,10]],"date-time":"2021-04-10T00:00:00Z","timestamp":1618012800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,10]],"date-time":"2021-04-10T00:00:00Z","timestamp":1618012800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Major projects of science and technology plan of Guangdong Province","award":["2015B090922013"],"award-info":[{"award-number":["2015B090922013"]}]},{"name":"Major projects of science and technology plan of Guangdong Province","award":["2017B090901019"],"award-info":[{"award-number":["2017B090901019"]}]},{"name":"Key Program of NSFC-Guangdong Joint Funds","award":["U1801263"],"award-info":[{"award-number":["U1801263"]}]},{"name":"Key Program of NSFC-Guangdong Joint Funds","award":["U1701262"],"award-info":[{"award-number":["U1701262"]}]},{"name":"Key Program of NSFC-Guangdong Joint Funds","award":["U2001201"],"award-info":[{"award-number":["U2001201"]}]},{"DOI":"10.13039\/501100013076","name":"National Major Science and Technology Projects of China","doi-asserted-by":"crossref","award":["1920001001367"],"award-info":[{"award-number":["1920001001367"]}],"id":[{"id":"10.13039\/501100013076","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Guangdong Provincial Key Laboratory of Cyber-Physical System","award":["2016B030301008"],"award-info":[{"award-number":["2016B030301008"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In actual welding scenarios, an effective path planner is needed to find a collision-free path in the configuration space for the welding manipulator with obstacles around. However, as a state-of-the-art method, the sampling-based planner only satisfies the probability completeness and its computational complexity is sensitive with state dimension. In this paper, we propose a path planner for welding manipulators based on deep reinforcement learning for solving path planning problems in high-dimensional continuous state and action spaces. Compared with the sampling-based method, it is more robust and is less sensitive with state dimension. In detail, to improve the learning efficiency, we introduce the inverse kinematics module to provide prior knowledge while a gain module is also designed to avoid the local optimal policy, we integrate them into the training algorithm. To evaluate our proposed planning algorithm in multiple dimensions, we conducted multiple sets of path planning experiments for welding manipulators. The results show that our method not only improves the convergence performance but also is superior in terms of optimality and robustness of planning compared with most other planning algorithms.<\/jats:p>","DOI":"10.1007\/s40747-021-00366-1","type":"journal-article","created":{"date-parts":[[2021,4,10]],"date-time":"2021-04-10T17:02:35Z","timestamp":1618074155000},"page":"1899-1912","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Collision-free path planning for welding manipulator via hybrid algorithm of deep reinforcement learning and inverse kinematics"],"prefix":"10.1007","volume":"8","author":[{"given":"Jie","family":"Zhong","sequence":"first","affiliation":[]},{"given":"Tao","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Lianglun","family":"Cheng","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,4,10]]},"reference":[{"issue":"2","key":"366_CR1","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.rcim.2010.08.006","volume":"27","author":"D Lee","year":"2011","unstructured":"Lee D, Ku N, Kim TW, Kim J, Lee KY, Son YS (2011) Development and application of an intelligent welding robot system for shipbuilding. Robot Comput-Integr Manufact 27(2):377\u2013388","journal-title":"Robot Comput-Integr Manufact"},{"issue":"5","key":"366_CR2","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1016\/j.rcim.2014.02.002","volume":"30","author":"D Lee","year":"2014","unstructured":"Lee D (2014) Robots in the shipbuilding industry. Robot Comput-Integr Manufact 30(5):442\u2013450","journal-title":"Robot Comput-Integr Manufact"},{"issue":"3","key":"366_CR3","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s40747-020-00167-y","volume":"6","author":"G Budak","year":"2020","unstructured":"Budak G, Chen X (2020) Evaluation of the size of time windows for the travelling salesman problem in delivery operations. Compl Intell Syst 6(3):681\u2013695","journal-title":"Compl Intell Syst"},{"key":"366_CR4","doi-asserted-by":"crossref","unstructured":"Hsu D, Latombe J.C, Motwani R (1997) Path planning in expansive configuration spaces. In: Proceedings of International Conference on Robotics and Automation, IEEE, vol 3, pp 2719\u20132726","DOI":"10.1109\/ROBOT.1997.619371"},{"key":"366_CR5","unstructured":"Amato N.M, Wu Y (1996) A randomized roadmap method for path and manipulation planning. In: Proceedings of IEEE international conference on robotics and automation, IEEE, vol 1, pp 113\u2013120"},{"issue":"4","key":"366_CR6","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1109\/70.508439","volume":"12","author":"LE Kavraki","year":"1996","unstructured":"Kavraki LE, Svestka P, Latombe JC, Overmars MH (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans Robot Autom 12(4):566\u2013580","journal-title":"IEEE Trans Robot Autom"},{"key":"366_CR7","unstructured":"LaValle SM, Kuffner JJ, Donald BR (2001) Rapidly-exploring random trees: progress and prospects[J]. In: Algorithmic and computational robotics: new directions, vol 5. pp 293\u2013308"},{"key":"366_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.robot.2015.02.007","volume":"68","author":"AH Qureshi","year":"2015","unstructured":"Qureshi AH, Ayaz Y (2015) Intelligent bidirectional rapidly-exploring random trees for optimal motion planning in complex cluttered environments. Robot Autonom Syst 68:1\u201311","journal-title":"Robot Autonom Syst"},{"issue":"1","key":"366_CR9","doi-asserted-by":"publisher","first-page":"220","DOI":"10.3390\/s20010220","volume":"20","author":"N P\u00e9rez-Higueras","year":"2020","unstructured":"P\u00e9rez-Higueras N, Jard\u00f3n A, Rodr\u00edguez \u00c1, Balaguer C (2020) 3d exploration and navigation with optimal-rrt planners for ground robots in indoor incidents. Sensors 20(1):220","journal-title":"Sensors"},{"key":"366_CR10","doi-asserted-by":"crossref","unstructured":"Connell D, La HM (2017) Dynamic path planning and replanning for mobile robots using rrt. In: 2017 IEEE international conference on systems, man, and cybernetics (SMC), IEEE, pp 1429\u20131434","DOI":"10.1109\/SMC.2017.8122814"},{"issue":"6","key":"366_CR11","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1080\/01691864.2013.756386","volume":"27","author":"K Yang","year":"2013","unstructured":"Yang K, Keat Gan S, Sukkarieh S (2013) A gaussian process-based rrt planner for the exploration of an unknown and cluttered environment with a uav. Adv Robot 27(6):431\u2013443","journal-title":"Adv Robot"},{"issue":"5\u20138","key":"366_CR12","first-page":"1401","volume":"89","author":"H Akbaripour","year":"2016","unstructured":"Akbaripour H, Masehian E (2016) Semi-lazy probabilistic roadmap: a parameter-tuned, resilient and robust path planning method for manipulator robots. Int J Adv Manuf Technol 89(5\u20138):1401\u20131430","journal-title":"Int J Adv Manuf Technol"},{"key":"366_CR13","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.compag.2018.10.031","volume":"156","author":"X Cao","year":"2019","unstructured":"Cao X, Zou X, Jia C, Chen M, Zeng Z (2019) Rrt-based path planning for an intelligent litchi-picking manipulator. Comput Electron Agric 156:105\u2013118","journal-title":"Comput Electron Agric"},{"key":"366_CR14","unstructured":"Jenamani R.K, Kumar R, Mall P, Kedia K (2020) Robotic motion planning using learned critical sources and local sampling. arXiv:2006.04194"},{"key":"366_CR15","doi-asserted-by":"crossref","unstructured":"Khatib O (1986) Real-time obstacle avoidance for manipulators and mobile robots. In: Autonomous robot vehicles, pp 396\u2013404. Springer, Berlin","DOI":"10.1007\/978-1-4613-8997-2_29"},{"issue":"9","key":"366_CR16","first-page":"27","volume":"35","author":"L Xie","year":"2018","unstructured":"Xie L, Liu S (2018) Dynamic obstacle-avoiding motion planning for manipulator based on improved artificial potential filed. Control Theory Appl 35(9):27\u201337","journal-title":"Control Theory Appl"},{"key":"366_CR17","doi-asserted-by":"crossref","unstructured":"Li H, Wang Z, Ou Y (2019) Obstacle avoidance of manipulators based on improved artificial potential field method. In: 2019 IEEE international conference on robotics and biomimetics (ROBIO), IEEE, pp 564\u2013569","DOI":"10.1109\/ROBIO49542.2019.8961506"},{"issue":"1","key":"366_CR18","first-page":"31","volume":"95","author":"S Liu","year":"2014","unstructured":"Liu S, Zhang Q, Zhou D (2014) Obstacle avoidance path planning of space manipulator based on improved artificial potential field method. J Inst Engineers (India): Ser C 95(1):31\u201339","journal-title":"J Inst Engineers (India): Ser C"},{"key":"366_CR19","doi-asserted-by":"crossref","unstructured":"Sathya A.S, Gillis J, Pipeleers G, Swevers J (2020) Real-time robot arm motion planning and control with nonlinear model predictive control using augmented lagrangian on a first-order solver. In: 2020 European Control Conference (ECC), IEEE, pp 507\u2013512","DOI":"10.23919\/ECC51009.2020.9143732"},{"key":"366_CR20","unstructured":"Chen B, Dai B, Lin Q, Ye G, Liu H, Song L (2020) Learning to plan in high dimensions via neural exploration-exploitation trees. arXiv:1903.00070"},{"key":"366_CR21","doi-asserted-by":"crossref","unstructured":"Qureshi A.H, Simeonov A, Bency M.J, Yip M.C (2019) Motion planning networks. In: 2019 international conference on robotics and automation (ICRA), IEEE, pp 2118\u20132124","DOI":"10.1109\/ICRA.2019.8793889"},{"key":"366_CR22","doi-asserted-by":"crossref","unstructured":"Nguyen H, La H (2019) Review of deep reinforcement learning for robot manipulation. In: 2019 Third IEEE international conference on robotic computing (IRC), IEEE, pp. 590\u2013595","DOI":"10.1109\/IRC.2019.00120"},{"issue":"11","key":"366_CR23","doi-asserted-by":"publisher","first-page":"1238","DOI":"10.1177\/0278364913495721","volume":"32","author":"J Kober","year":"2013","unstructured":"Kober J, Bagnell JA, Peters J (2013) Reinforcement learning in robotics: a survey. Int J Robot Res 32(11):1238\u20131274","journal-title":"Int J Robot Res"},{"issue":"4","key":"366_CR24","first-page":"1740","volume":"13","author":"Y Li","year":"2018","unstructured":"Li Y, Xia H, Zhao B (2018) Policy iteration algorithm based fault tolerant tracking control: an implementation on reconfigurable manipulators. J Electr Eng Technol 13(4):1740\u20131751","journal-title":"J Electr Eng Technol"},{"key":"366_CR25","first-page":"5","volume":"2020","author":"Y Zhang","year":"2020","unstructured":"Zhang Y, Zhao B, Liu D (2020) Deterministic policy gradient adaptive dynamic programming for model-free optimal control. Neurocomputing 2020:5","journal-title":"Neurocomputing"},{"issue":"12","key":"366_CR26","doi-asserted-by":"publisher","first-page":"3735","DOI":"10.1109\/TNNLS.2019.2897814","volume":"30","author":"Y Yang","year":"2019","unstructured":"Yang Y, Guo Z, Xiong H, Ding DW, Yin Y, Wunsch DC (2019) Data-driven robust control of discrete-time uncertain linear systems via off-policy reinforcement learning. IEEE Trans Neural Netw Learn Syst 30(12):3735\u20133747","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"366_CR27","first-page":"1","volume":"2020","author":"Y Yang","year":"2020","unstructured":"Yang Y, Vamvoudakis KG, Modares H, Yin Y, Wunsch DC (2020) Hamiltonian-driven hybrid adaptive dynamic programming. IEEE Trans Syst Man Cybernet Syst 2020:1\u201312","journal-title":"IEEE Trans Syst Man Cybernet Syst"},{"issue":"6","key":"366_CR28","doi-asserted-by":"publisher","first-page":"3316","DOI":"10.1016\/j.jfranklin.2019.12.017","volume":"357","author":"Y Yang","year":"2019","unstructured":"Yang Y, Ding DW, Xiong H, Yin Y, Wunsch DC (2019) Online barrier-actor-critic learning for h $$ \\infty $$ control with full-state constraints and input saturation. J Franklin Inst 357(6):3316\u20133344","journal-title":"J Franklin Inst"},{"issue":"99","key":"366_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2020.3010524","volume":"PP","author":"Y Yang","year":"2020","unstructured":"Yang Y, Vamvoudakis KG, Modares H, Yin Y, Wunsch DC (2020) Safe intermittent reinforcement learning with static and dynamic event generators. IEEE Trans Neural Netw Learn Syst PP(99):1\u201315","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"366_CR30","doi-asserted-by":"crossref","unstructured":"Xue X, Li Z, Zhang D, Yan Y (2019) A deep reinforcement learning method for mobile robot collision avoidance based on double dqn. In: 2019 IEEE 28th international symposium on industrial electronics (ISIE), IEEE, pp 2131\u20132136","DOI":"10.1109\/ISIE.2019.8781522"},{"key":"366_CR31","doi-asserted-by":"crossref","unstructured":"Jesus J.C, Bottega J.A, Cuadros M.A, Gamarra D.F (2019) Deep deterministic policy gradient for navigation of mobile robots in simulated environments. In: 2019 19th international conference on advanced robotics (ICAR), IEEE, pp 362\u2013367","DOI":"10.1109\/ICAR46387.2019.8981638"},{"key":"366_CR32","doi-asserted-by":"crossref","unstructured":"Sangiovanni B, Rendiniello A, Incremona G.P, Ferrara A, Piastra M (2018) Deep reinforcement learning for collision avoidance of robotic manipulators. In: 2018 European Control Conference (ECC), IEEE, pp 2063\u20132068","DOI":"10.23919\/ECC.2018.8550363"},{"key":"366_CR33","unstructured":"Gu S, Lillicrap T, Sutskever I, Levine S (2016) Continuous deep q-learning with model-based acceleration. In: International conference on machine learning, pp 2829\u20132838"},{"issue":"2","key":"366_CR34","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1109\/LCSYS.2020.3002852","volume":"5","author":"B Sangiovanni","year":"2020","unstructured":"Sangiovanni B, Incremona GP, Piastra M, Ferrara A (2020) Self-configuring robot path planning with obstacle avoidance via deep reinforcement learning. IEEE Control Syst Lett 5(2):397\u2013402","journal-title":"IEEE Control Syst Lett"},{"key":"366_CR35","first-page":"1","volume":"2019","author":"X Hua","year":"2019","unstructured":"Hua X, Wang G, Xu J, Chen K et al (2019) Reinforcement learning-based collision-free path planner for redundant robot in narrow duct. J Intell Manufact 2019:1\u201312","journal-title":"J Intell Manufact"},{"key":"366_CR36","unstructured":"Lillicrap T.P, Hunt J.J, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv:1509.02971"},{"issue":"3","key":"366_CR37","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1177\/027836499301200306","volume":"12","author":"CJ Paredis","year":"1993","unstructured":"Paredis CJ, Khosla PK (1993) Kinematic design of serial link manipulators from task specifications. Int J Robot Res 12(3):274\u2013287","journal-title":"Int J Robot Res"},{"key":"366_CR38","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-32552-1","volume-title":"Springer handbook of robotics","author":"B Siciliano","year":"2016","unstructured":"Siciliano B, Khatib O (2016) Springer handbook of robotics. Springer, Berlin"},{"issue":"4","key":"366_CR39","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1016\/0094-114X(91)90008-R","volume":"26","author":"SC Thomopoulos","year":"1991","unstructured":"Thomopoulos SC, Tam RY (1991) An iterative solution to the inverse kinematics of robotic manipulators. Mech Mach Theory 26(4):359\u2013373","journal-title":"Mech Mach Theory"},{"issue":"1","key":"366_CR40","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1109\/TSMC.1986.289285","volume":"16","author":"CW Wampler","year":"1986","unstructured":"Wampler CW (1986) Manipulator inverse kinematic solutions based on vector formulations and damped least-squares methods. IEEE Trans Syst Man Cybern 16(1):93\u2013101","journal-title":"IEEE Trans Syst Man Cybern"},{"key":"366_CR41","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-54413-7","volume-title":"Robotics, vision and control: fundamental algorithms in MATLAB\u00ae second, completely revised","author":"P Corke","year":"2017","unstructured":"Corke P (2017) Robotics, vision and control: fundamental algorithms in MATLAB\u00ae second, completely revised, vol 118. Springer, Berlin"},{"key":"366_CR42","volume-title":"Reinforcement learning: an introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press, Hoboken"},{"key":"366_CR43","unstructured":"Henderson P, Islam R, Bachman P, Pineau J, Precup D, Meger D (2017) Deep reinforcement learning that matters. arXiv:1709.06560"},{"issue":"7540","key":"366_CR44","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529\u2013533","journal-title":"Nature"},{"key":"366_CR45","unstructured":"Silver D, Lever G, Heess N, Degris T, Wierstra D, Riedmiller M (2014) Deterministic policy gradient algorithms[C]. In: International conference on machine learning. PMLR, pp 387\u2013395"},{"key":"366_CR46","unstructured":"Kuffner J.J, LaValle S.M (2000) Rrt-connect: an efficient approach to single-query path planning. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), IEEE, vol 2, pp 995\u20131001"},{"issue":"7","key":"366_CR47","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1177\/0278364911406761","volume":"30","author":"S Karaman","year":"2011","unstructured":"Karaman S, Frazzoli E (2011) Sampling-based algorithms for optimal motion planning. Int J Robot Res 30(7):846\u2013894","journal-title":"Int J Robot Res"},{"issue":"5","key":"366_CR48","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1177\/0278364919890396","volume":"39","author":"JD Gammell","year":"2020","unstructured":"Gammell JD, Barfoot TD, Srinivasa SS (2020) Batch informed trees (bit*): informed asymptotically optimal anytime search. Int J Robot Res 39(5):543\u2013567","journal-title":"Int J Robot Res"},{"issue":"4","key":"366_CR49","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1109\/MRA.2012.2205651","volume":"19","author":"IA Sucan","year":"2012","unstructured":"Sucan IA, Moll M, Kavraki LE (2012) The open motion planning library. IEEE Robot Autom Mag 19(4):72\u201382","journal-title":"IEEE Robot Autom Mag"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-021-00366-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-021-00366-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-021-00366-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,30]],"date-time":"2022-05-30T01:08:57Z","timestamp":1653872937000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-021-00366-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,10]]},"references-count":49,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["366"],"URL":"https:\/\/doi.org\/10.1007\/s40747-021-00366-1","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,10]]},"assertion":[{"value":"28 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 April 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}