{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T06:59:14Z","timestamp":1764053954511,"version":"3.41.0"},"reference-count":68,"publisher":"Springer Science and Business Media LLC","issue":"18","license":[{"start":{"date-parts":[[2025,4,11]],"date-time":"2025-04-11T00:00:00Z","timestamp":1744329600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,11]],"date-time":"2025-04-11T00:00:00Z","timestamp":1744329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Funda\u00e7\u00e3o para Ci\u00eancia e a Tecnologia"},{"DOI":"10.13039\/501100006752","name":"Universidade do Porto","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006752","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>The RoboCup 3D soccer simulation league serves as a competitive platform for showcasing innovation in autonomous humanoid robot agents through simulated soccer matches. Our team, FC Portugal, developed a new codebase from scratch in Python after RoboCup 2021. The team\u2019s performance relies on a set of skills centered around novel unifying primitives and a custom, symmetry-extended version of the proximal policy optimization algorithm. Our methods have been thoroughly tested in official RoboCup matches, where FC Portugal has won the last two main competitions, in 2022 and 2023. This paper presents our training framework, as well as a timeline of skills developed using our skill-set-primitives, which considerably improve the sample efficiency and stability of skills, and motivate seamless transitions. We start with a significantly fast Sprint-Kick developed in 2021 and progress to the most recent skill set, including a multi-purpose omnidirectional walk, a dribble with unprecedented ball control, a solid kick, and a push skill. The push addresses low-level collision scenarios and high-level strategies to increase ball possession. We address the resource-intensive nature of this task through an innovative multi-agent learning approach. Finally, we release the team\u2019s codebase to the RoboCup community, providing other teams with a robust and modern foundation upon which they can build new features.<\/jats:p>","DOI":"10.1007\/s00521-025-11151-3","type":"journal-article","created":{"date-parts":[[2025,4,11]],"date-time":"2025-04-11T05:15:21Z","timestamp":1744348521000},"page":"12699-12734","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Designing a skilled soccer team for RoboCup: exploring skill-set-primitives through reinforcement learning"],"prefix":"10.1007","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6342-2054","authenticated-orcid":false,"given":"Miguel","family":"Abreu","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4709-1718","authenticated-orcid":false,"given":"Lu\u00eds Paulo","family":"Reis","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0513-158X","authenticated-orcid":false,"given":"Nuno","family":"Lau","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,4,11]]},"reference":[{"key":"11151_CR1","doi-asserted-by":"crossref","unstructured":"Kitano H, Asada M (1998) RoboCup humanoid challenge: that\u2019s one small step for a robot, one giant leap for mankind. In: Proceedings. 1998 IEEE\/RSJ international conference on intelligent robots and systems. Innovations in theory, practice and applications (Cat. No. 98CH36190), vol 1, IEEE, pp 419\u2013424","DOI":"10.1109\/IROS.1998.724655"},{"key":"11151_CR2","first-page":"181","volume":"174","author":"J Boedecker","year":"2008","unstructured":"Boedecker J, Asada M (2008) Simspark-concepts and application in the RoboCup 3D soccer simulation league. Autonomous Robots 174:181","journal-title":"Autonomous Robots"},{"issue":"1","key":"11151_CR3","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/s10458-024-09642-z","volume":"38","author":"Z Gao","year":"2024","unstructured":"Gao Z, Yi M, Jin Y, Zhang H, Hao Y, Yin M, Cai Z, Shen F (2024) A survey of research on several problems in the RoboCup3D simulation environment. Autonomous Agents and Multi-Agent Systems 38(1):13","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"key":"11151_CR4","unstructured":"Ravindran B, Barto AG (2001) Symmetries and model minimization in Markov decision processes. Technical report, University of Massachusetts, USA"},{"key":"11151_CR5","doi-asserted-by":"publisher","first-page":"128771","DOI":"10.1016\/j.neucom.2024.128771","volume":"614","author":"M Abreu","year":"2025","unstructured":"Abreu M, Reis LP, Lau N (2025) Addressing imperfect symmetry: a novel symmetry-learning actor-critic extension. Neurocomputing 614:128771","journal-title":"Neurocomputing"},{"key":"11151_CR6","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347"},{"key":"11151_CR7","first-page":"24611","volume":"35","author":"C Yu","year":"2022","unstructured":"Yu C, Velu A, Vinitsky E, Gao J, Wang Y, Bayen A, Wu Y (2022) The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems 35:24611\u201324624","journal-title":"Advances in Neural Information Processing Systems"},{"key":"11151_CR8","doi-asserted-by":"crossref","unstructured":"Xu Y, Vatankhah H (2014) SimSpark: an open source robot simulator developed by the RoboCup community. In: RoboCup 2013: Robot World Cup XVII. Springer, pp 632\u2013639","DOI":"10.1007\/978-3-662-44468-9_59"},{"issue":"1","key":"11151_CR9","first-page":"73","volume":"18","author":"H Kitano","year":"1997","unstructured":"Kitano H, Asada M, Kuniyoshi Y, Noda I, Osawa E, Matsubara H (1997) RoboCup: a challenge problem for AI. AI Magazine 18(1):73\u201373","journal-title":"AI Magazine"},{"key":"11151_CR10","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/4151.001.0001","volume-title":"Layered learning in multiagent systems: a winning approach to robotic soccer","author":"P Stone","year":"2000","unstructured":"Stone P (2000) Layered learning in multiagent systems: a winning approach to robotic soccer. The MIT Press, Cambridge"},{"key":"11151_CR11","doi-asserted-by":"crossref","unstructured":"Whiteson S, Stone P (2003) Concurrent layered learning. In: Proceedings of the second international joint conference on autonomous agents and multiagent systems, pp 193\u2013200","DOI":"10.1145\/860575.860607"},{"key":"11151_CR12","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.artint.2017.09.001","volume":"254","author":"P MacAlpine","year":"2018","unstructured":"MacAlpine P, Stone P (2018) Overlapping layered learning. Artificial Intelligence 254:21\u201343","journal-title":"Artificial Intelligence"},{"key":"11151_CR13","doi-asserted-by":"crossref","unstructured":"MacAlpine P, Depinet M, Stone P (2015) UT Austin villa 2014: RoboCup 3D simulation league champion via overlapping layered learning. In: Proceedings of the AAAI conference on artificial intelligence, p 29","DOI":"10.1609\/aaai.v29i1.9540"},{"key":"11151_CR14","doi-asserted-by":"crossref","unstructured":"Johannink T, Bahl S, Nair A, Luo J, Kumar A, Loskyll M, Ojea JA, Solowjow E, Levine S (2019) Residual reinforcement learning for robot control. In: 2019 international conference on robotics and automation (ICRA). IEEE, pp 6023\u20136029","DOI":"10.1109\/ICRA.2019.8794127"},{"key":"11151_CR15","unstructured":"Silver T, Allen K, Tenenbaum J, Kaelbling L (2018) Residual policy learning. arXiv preprint arXiv:1812.06298"},{"key":"11151_CR16","unstructured":"Kasaei M, Abreu M, Lau N, Pereira A, Reis LP (2021) A CPG-based agile and versatile locomotion framework using proximal symmetry loss. arXiv preprint arXiv:2103.00928"},{"key":"11151_CR17","doi-asserted-by":"crossref","unstructured":"Kumar A, Li Z, Zeng J, Pathak D, Sreenath K, Malik J (2022) Adapting rapid motor adaptation for bipedal robots. In: 2022 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1161\u20131168","DOI":"10.1109\/IROS47612.2022.9981091"},{"issue":"11","key":"11151_CR18","doi-asserted-by":"publisher","first-page":"7799","DOI":"10.1109\/LRA.2023.3320827","volume":"8","author":"D Youm","year":"2023","unstructured":"Youm D, Jung H, Kim H, Hwangbo J, Park H-W, Ha S (2023) Imitating and finetuning model predictive control for robust and symmetric quadrupedal locomotion. IEEE Robotics and Automation Letters 8(11):7799\u20137806","journal-title":"IEEE Robotics and Automation Letters"},{"key":"11151_CR19","doi-asserted-by":"crossref","unstructured":"Garcia-Hernando G, Johns E, Kim T-K (2020) Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning. In: 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 9561\u20139568","DOI":"10.1109\/IROS45743.2020.9340947"},{"issue":"4","key":"11151_CR20","doi-asserted-by":"publisher","first-page":"1307","DOI":"10.1109\/TRO.2020.2988642","volume":"36","author":"A Zeng","year":"2020","unstructured":"Zeng A, Song S, Lee J, Rodriguez A, Funkhouser T (2020) TossingBot: learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics 36(4):1307\u20131319","journal-title":"IEEE Transactions on Robotics"},{"key":"11151_CR21","doi-asserted-by":"crossref","unstructured":"Zhang K, Yang Z, Ba\u015far T (2021) Multi-agent reinforcement learning: a selective overview of theories and algorithms. In: Handbook of reinforcement learning and control, pp 321\u2013384","DOI":"10.1007\/978-3-030-60990-0_12"},{"issue":"4","key":"11151_CR22","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1287\/moor.27.4.819.297","volume":"27","author":"DS Bernstein","year":"2002","unstructured":"Bernstein DS, Givan R, Immerman N, Zilberstein S (2002) The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research 27(4):819\u2013840","journal-title":"Mathematics of Operations Research"},{"issue":"3","key":"11151_CR23","doi-asserted-by":"publisher","first-page":"193","DOI":"10.3390\/drones7030193","volume":"7","author":"R Azzam","year":"2023","unstructured":"Azzam R, Boiko I, Zweiri Y (2023) Swarm cooperative navigation using centralized training and decentralized execution. Drones 7(3):193","journal-title":"Drones"},{"key":"11151_CR24","doi-asserted-by":"crossref","unstructured":"Gupta JK, Egorov M, Kochenderfer M (2017) Cooperative multi-agent control using deep reinforcement learning. In: Autonomous agents and multiagent systems: AAMAS 2017 workshops, best papers, S\u00e3o Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16. Springer, pp 66\u201383","DOI":"10.1007\/978-3-319-71682-4_5"},{"key":"11151_CR25","first-page":"637","volume":"30","author":"R Lowe","year":"2017","unstructured":"Lowe R, Wu YI, Tamar A, Harb J, Pieter Abbeel O, Mordatch I (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems 30:637","journal-title":"Advances in Neural Information Processing Systems"},{"key":"11151_CR26","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1016\/j.neucom.2020.01.079","volume":"390","author":"D Sim\u00f5es","year":"2020","unstructured":"Sim\u00f5es D, Lau N, Reis LP (2020) Multi-agent actor centralized-critic with communication. Neurocomputing 390:40\u201356","journal-title":"Neurocomputing"},{"issue":"1","key":"11151_CR27","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1007\/s10458-023-09603-y","volume":"37","author":"A Smit","year":"2023","unstructured":"Smit A, Engelbrecht HA, Brink W, Pretorius A (2023) Scaling multi-agent reinforcement learning to full 11 versus 11 simulated robotic football. Autonomous Agents and Multi-Agent Systems 37(1):20","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"key":"11151_CR28","doi-asserted-by":"crossref","unstructured":"Hirai K, Hirose M, Haikawa Y, Takenaka T (1998) The development of Honda humanoid robot. In: Proceedings. 1998 IEEE international conference on robotics and automation (Cat. No. 98CH36146), vol 2. IEEE, pp 1321\u20131326","DOI":"10.1109\/ROBOT.1998.677288"},{"key":"11151_CR29","doi-asserted-by":"crossref","unstructured":"Kajita S, Kanehiro F, Kaneko K, Fujiwara K, Harada K, Yokoi K, Hirukawa H (2003) Biped walking pattern generation by using preview control of zero-moment point. In: 2003 IEEE international conference on robotics and automation (Cat. No. 03CH37422), vol 2. IEEE, pp 1620\u20131626","DOI":"10.1109\/ROBOT.2003.1241826"},{"key":"11151_CR30","doi-asserted-by":"crossref","unstructured":"Kasaei M, Lau N, Pereira A (2019) A fast and stable omnidirectional walking engine for the NAO humanoid robot. In: RoboCup 2019: Robot world cup XXIII 23. Springer, pp 99\u2013111","DOI":"10.1007\/978-3-030-35699-6_8"},{"key":"11151_CR31","doi-asserted-by":"crossref","unstructured":"Liang Z, Zhao H, Hao Y (2014) An omnidirectional walk for a biped robot based on gyroscope-accelerometer measurement. In: 2014 IEEE international conference on mechatronics and automation. IEEE, pp 1052\u20131057","DOI":"10.1109\/ICMA.2014.6885843"},{"issue":"3\u20134","key":"11151_CR32","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1007\/s13218-016-0437-9","volume":"30","author":"A Seekircher","year":"2016","unstructured":"Seekircher A, Visser U (2016) An adaptive LIPM-based dynamic walk using model parameter optimization on humanoid robots. KI-K\u00fcnstliche Intelligenz 30(3\u20134):233\u2013244","journal-title":"KI-K\u00fcnstliche Intelligenz"},{"key":"11151_CR33","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s10846-015-0191-5","volume":"80","author":"N Shafii","year":"2015","unstructured":"Shafii N, Lau N, Reis LP (2015) Learning to walk fast: optimized hip height movement for simulated and real humanoid robots. Journal of Intelligent & Robotic Systems 80:555\u2013571","journal-title":"Journal of Intelligent & Robotic Systems"},{"key":"11151_CR34","doi-asserted-by":"crossref","unstructured":"Muniz F, Maximo MR, Ribeiro CH (2019) Keyframe movement optimization for simulated humanoid robot using a parallel optimization framework. In: 2016 XIII Latin American robotics symposium and IV Brazilian robotics symposium (LARS\/SBR). IEEE, pp 79\u201384","DOI":"10.1109\/LARS-SBR.2016.20"},{"issue":"3","key":"11151_CR35","doi-asserted-by":"publisher","first-page":"145","DOI":"10.2991\/ijndc.2016.4.3.2","volume":"4","author":"H Shi","year":"2016","unstructured":"Shi H, Li X, Chen H, Wang S (2016) Adaptive omni-directional walking method with fuzzy interpolation for biped robots. International Journal of Networked and Distributed Computing 4(3):145\u2013158","journal-title":"International Journal of Networked and Distributed Computing"},{"key":"11151_CR36","doi-asserted-by":"crossref","unstructured":"Shi H, Li X, Liang W, Chen H, Wang S et\u00a0al. (2016) A novel fuzzy omni-directional gait planning algorithm for biped robot. In: 2016 17th IEEE\/ACIS international conference on software engineering, artificial intelligence, networking and parallel\/distributed computing (SNPD). IEEE, pp 71\u201376","DOI":"10.1109\/SNPD.2016.7515880"},{"key":"11151_CR37","doi-asserted-by":"crossref","unstructured":"Wang S, Hu M, Shi H, Zhang S, Li X, Li W (2015) Humanoid robot\u2019s omnidirectional walking. In: 2015 IEEE international conference on information and automation, pp 381\u2013385","DOI":"10.1109\/ICInfA.2015.7279317"},{"key":"11151_CR38","doi-asserted-by":"publisher","first-page":"103900","DOI":"10.1016\/j.robot.2021.103900","volume":"146","author":"M Kasaei","year":"2021","unstructured":"Kasaei M, Abreu M, Lau N, Pereira A, Reis LP (2021) Robust biped locomotion using deep reinforcement learning on top of an analytical control approach. Robotics and Autonomous Systems 146:103900","journal-title":"Robotics and Autonomous Systems"},{"key":"11151_CR39","doi-asserted-by":"publisher","first-page":"1004490","DOI":"10.3389\/frobt.2023.1004490","volume":"10","author":"M Kasaei","year":"2023","unstructured":"Kasaei M, Abreu M, Lau N, Pereira A, Reis LP, Li Z (2023) Learning hybrid locomotion skills\u2014learn to exploit residual actions and modulate model-based gait control. Frontiers in Robotics and AI 10:1004490","journal-title":"Frontiers in Robotics and AI"},{"issue":"1","key":"11151_CR40","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1007\/s10846-022-01656-7","volume":"106","author":"DC Melo","year":"2022","unstructured":"Melo DC, Maximo MR, Cunha AM (2022) Learning push recovery behaviors for humanoid walking using deep reinforcement learning. Journal of Intelligent & Robotic Systems 106(1):8","journal-title":"Journal of Intelligent & Robotic Systems"},{"key":"11151_CR41","doi-asserted-by":"publisher","first-page":"600885","DOI":"10.3389\/fnbot.2020.600885","volume":"14","author":"C Tao","year":"2021","unstructured":"Tao C, Xue J, Zhang Z, Cao F, Li C, Gao H (2021) Gait optimization method for humanoid robots based on parallel comprehensive learning particle swarm optimizer algorithm. Frontiers in Neurorobotics 14:600885","journal-title":"Frontiers in Neurorobotics"},{"key":"11151_CR42","unstructured":"Fischer J, Dorer K (2020) Learning to walk with toes. In: Artificial intelligence: research impact on key industries. In: Proceedings of the upper-Rhine artificial intelligence symposium, pp 3\u201311"},{"issue":"8","key":"11151_CR43","doi-asserted-by":"publisher","first-page":"860","DOI":"10.1016\/j.robot.2009.03.005","volume":"57","author":"CAA Calderon","year":"2009","unstructured":"Calderon CAA, Mohan RE, Hu L, Zhou C, Hu H (2009) Generating human-like soccer primitives from human data. Robotics and Autonomous Systems 57(8):860\u2013869","journal-title":"Robotics and Autonomous Systems"},{"issue":"69","key":"11151_CR44","doi-asserted-by":"publisher","first-page":"0235","DOI":"10.1126\/scirobotics.abo0235","volume":"7","author":"S Liu","year":"2022","unstructured":"Liu S, Lever G, Wang Z, Merel J, Eslami SA, Hennes D, Czarnecki WM, Tassa Y, Omidshafiei S, Abdolmaleki A et al (2022) From motor control to team play in simulated humanoid football. Science Robotics 7(69):0235","journal-title":"Science Robotics"},{"issue":"6","key":"11151_CR45","first-page":"2802","volume":"69","author":"C Tao","year":"2022","unstructured":"Tao C, Xue J, Zhang Z, Gao Z (2022) Parallel deep reinforcement learning method for gait control of biped robot. IEEE Transactions on Circuits and Systems II: Express Briefs 69(6):2802\u20132806","journal-title":"IEEE Transactions on Circuits and Systems II: Express Briefs"},{"key":"11151_CR46","doi-asserted-by":"crossref","unstructured":"Abreu M, Lau N, Sousa A, Reis LP (2019) Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning. In: 2019 IEEE international conference on autonomous robot systems and competitions (ICARSC). IEEE, pp 256\u2013263","DOI":"10.1109\/ICARSC.2019.8733632"},{"key":"11151_CR47","doi-asserted-by":"crossref","unstructured":"Abreu M, Reis LP, Lau N (2019) Learning to run faster in a humanoid robot soccer environment through reinforcement learning. In: RoboCup 2019: robot world cup XXIII. Springer, pp 3\u201315","DOI":"10.1007\/978-3-030-35699-6_1"},{"issue":"3","key":"11151_CR48","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1007\/s10846-021-01355-9","volume":"102","author":"LC Melo","year":"2021","unstructured":"Melo LC, Melo DC, Maximo MR (2021) Learning humanoid robot running motions with symmetry incentive through proximal policy optimization. Journal of Intelligent & Robotic Systems 102(3):54","journal-title":"Journal of Intelligent & Robotic Systems"},{"key":"11151_CR49","doi-asserted-by":"crossref","unstructured":"Melo LC, M\u00e1ximo MROA (2019) Learning humanoid robot running skills through proximal policy optimization. In: 2019 Latin American robotics symposium (LARS), 2019 Brazilian symposium on robotics (SBR) and 2019 workshop on robotics in education (WRE). IEEE, pp 37\u201342","DOI":"10.1109\/LARS-SBR-WRE48964.2019.00015"},{"key":"11151_CR50","doi-asserted-by":"crossref","unstructured":"Leottau DL, Ruiz-del-Solar J, MacAlpine P, Stone P (2015) A study of layered learning strategies applied to individual behaviors in robot soccer. In: RoboCup 2015: robot world cup XIX 19. Springer, pp 290\u2013302","DOI":"10.1007\/978-3-319-29339-4_24"},{"key":"11151_CR51","doi-asserted-by":"crossref","unstructured":"Leottau L, Celemin C, Ruiz-del-Solar J (2015) Ball dribbling for humanoid biped robots: a reinforcement learning and fuzzy control approach. In: RoboCup 2014: robot world cup XVIII 18. Springer, pp 549\u2013561","DOI":"10.1007\/978-3-319-18615-3_45"},{"issue":"1","key":"11151_CR52","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1007\/s10846-022-01619-y","volume":"105","author":"AF Muzio","year":"2022","unstructured":"Muzio AF, Maximo MR, Yoneyama T (2022) Deep reinforcement learning for humanoid robot behaviors. Journal of Intelligent & Robotic Systems 105(1):12","journal-title":"Journal of Intelligent & Robotic Systems"},{"issue":"2","key":"11151_CR53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10846-021-01385-3","volume":"102","author":"M Abreu","year":"2021","unstructured":"Abreu M, Silva T, Teixeira H, Reis LP, Lau N (2021) 6D localization and kicking for humanoid robotic soccer. Journal of Intelligent & Robotic Systems 102(2):1\u201325","journal-title":"Journal of Intelligent & Robotic Systems"},{"key":"11151_CR54","doi-asserted-by":"publisher","first-page":"1235","DOI":"10.1007\/s12369-020-00712-2","volume":"13","author":"A Rezaeipanah","year":"2021","unstructured":"Rezaeipanah A, Amiri P, Jafari S (2021) Performing the kick during walking for RoboCup 3d soccer simulation league using reinforcement learning algorithm. International Journal of Social Robotics 13:1235\u20131252","journal-title":"International Journal of Social Robotics"},{"key":"11151_CR55","doi-asserted-by":"crossref","unstructured":"Depinet M, MacAlpine P, Stone P (2015) Keyframe sampling, optimization, and behavior integration: towards long-distance kicking in the RoboCup 3D simulation league. In: RoboCup 2014: robot world cup XVIII 18. Springer, pp 571\u2013582","DOI":"10.1007\/978-3-319-18615-3_47"},{"key":"11151_CR56","doi-asserted-by":"crossref","unstructured":"Hu X, Li Z, Sun G, Fang B (2020) Apply acceleration sampling to learn kick motion for NAO humanoid robot. In: 2020 international conference on computer engineering and intelligent control (ICCEIC). IEEE, pp 318\u2013323","DOI":"10.1109\/ICCEIC51584.2020.00068"},{"key":"11151_CR57","doi-asserted-by":"crossref","unstructured":"Jouandeau N, Hugel V (2014) Optimization of parametrised kicking motion for humanoid soccer player. In: 2014 IEEE international conference on autonomous robot systems and competitions (ICARSC). IEEE, pp 241\u2013246","DOI":"10.1109\/ICARSC.2014.6849793"},{"key":"11151_CR58","doi-asserted-by":"crossref","unstructured":"Dorer K (2018) Learning to use toes in a humanoid robot. In: RoboCup 2017: robot world cup XXI 11. Springer, pp 168\u2013179","DOI":"10.1007\/978-3-030-00308-1_14"},{"key":"11151_CR59","doi-asserted-by":"crossref","unstructured":"Abdolmaleki A, Sim\u00f5es D, Lau N, Reis LP, Neumann G (2017) Learning a humanoid kick with controlled distance. In: RoboCup 2016: robot world cup XX 20. Springer, pp 45\u201357","DOI":"10.1007\/978-3-319-68792-6_4"},{"key":"11151_CR60","doi-asserted-by":"crossref","unstructured":"Kawazoe A, Ito N, Iwata K (2019) A design method for kicks of the desired distance by combinations of based kicks in rss3d. In: 2019 6th international conference on computational science\/intelligence and applied informatics (CSII). IEEE, pp 66\u201371","DOI":"10.1109\/CSII.2019.00019"},{"key":"11151_CR61","doi-asserted-by":"crossref","unstructured":"Spitznagel M, Weiler D, Dorer K (2021) Deep reinforcement multi-directional kick-learning of a simulated robot with toes. In: 2021 IEEE international conference on autonomous robot systems and competitions (ICARSC). IEEE, pp 104\u2013110","DOI":"10.1109\/ICARSC52212.2021.9429811"},{"key":"11151_CR62","doi-asserted-by":"crossref","unstructured":"Denk J, Schmidt G (2003) Synthesis of walking primitive databases for biped robots in 3d-environments. In: 2003 IEEE international conference on robotics and automation (Cat. No. 03CH37422), vol 1. IEEE, pp 1343\u20131349","DOI":"10.1109\/ROBOT.2003.1241778"},{"issue":"2\u20133","key":"11151_CR63","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.robot.2004.03.003","volume":"47","author":"J Nakanishi","year":"2004","unstructured":"Nakanishi J, Morimoto J, Endo G, Cheng G, Schaal S, Kawato M (2004) Learning from demonstration and adaptation of biped locomotion. Robotics and Autonomous Systems 47(2\u20133):79\u201391","journal-title":"Robotics and Autonomous Systems"},{"issue":"11\u201312","key":"11151_CR64","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.1080\/02640414.2019.1677391","volume":"38","author":"A F\u00edlter","year":"2020","unstructured":"F\u00edlter A, Olivares J, Santalla A, Nakamura FY, Loturco I, Requena B (2020) New curve sprint test for soccer players: reliability and relationship with linear sprint. Journal of Sports Sciences 38(11\u201312):1320\u20131325","journal-title":"Journal of Sports Sciences"},{"key":"11151_CR65","doi-asserted-by":"crossref","unstructured":"Kasaei SM, Lau N, Pereira A, Shahri E (2017) A reliable model-based walking engine with push recovery capability. In: 2017 IEEE international conference on autonomous robot systems and competitions (ICARSC). IEEE, pp 122\u2013127","DOI":"10.1109\/ICARSC.2017.7964063"},{"key":"11151_CR66","doi-asserted-by":"crossref","unstructured":"Muzio AF, Maximo MR, Yoneyama T (2020) Deep reinforcement learning for humanoid robot dribbling. In: 2020 Latin American robotics symposium (LARS), 2020 Brazilian symposium on robotics (SBR) and 2020 workshop on robotics in education (WRE). IEEE, pp 1\u20136","DOI":"10.1109\/LARS\/SBR\/WRE51543.2020.9307084"},{"key":"11151_CR67","unstructured":"Glaser S robocup.info archive: RoboCup soccer\u20143D simulation league. https:\/\/archive.robocup.info\/Soccer\/Simulation\/3D\/"},{"key":"11151_CR68","unstructured":"Abreu M, Sim\u00f5es D, Lau N, Reis LP (2019) Fast, human-like running and sprinting. https:\/\/archive.robocup.info\/Soccer\/Simulation\/3D\/FCPs\/RoboCup\/2019\/FCPortugal_SS3D_RC2019_FCP.pdf"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-025-11151-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-025-11151-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-025-11151-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T14:44:51Z","timestamp":1750171491000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-025-11151-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,11]]},"references-count":68,"journal-issue":{"issue":"18","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["11151"],"URL":"https:\/\/doi.org\/10.1007\/s00521-025-11151-3","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"type":"print","value":"0941-0643"},{"type":"electronic","value":"1433-3058"}],"subject":[],"published":{"date-parts":[[2025,4,11]]},"assertion":[{"value":"14 June 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 April 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}