{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T04:08:54Z","timestamp":1778472534989,"version":"3.51.4"},"reference-count":49,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T00:00:00Z","timestamp":1774569600000},"content-version":"unspecified","delay-in-days":54,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This paper addresses the challenge of balance control for the underactuated triple pendulum robot (UTPR) using a model-free reinforcement learning (RL) strategy. A curriculum-based Soft Actor-Critic strategy, with a quadratic form and an integral term in the reward function (CSAC-QI), is proposed. By incorporating the integral of cumulative joint angle errors into the reward function, the CSAC-QI method significantly reduces steady-state errors and enhances control precision. CSAC-QI improves convergence efficiency through an adaptive curriculum learning (CL) framework that enables a structured transition from simpler to more complex tasks. To enhance control robustness, motor friction identification and domain randomization are implemented during training, thereby equipping the UTPR to cope with real-world uncertainties. Simulation experiments demonstrate superior performance of the CSAC-QI method in handling larger initial joint deviations, achieving accurate end-effector positioning, and maintaining balance under dynamic randomization, sensor noise, and external disturbances. Notably, the trained policy is directly deployed on the UTPR prototype, where it successfully maintains balance in real-world conditions.<\/jats:p>","DOI":"10.1017\/s0263574726103282","type":"journal-article","created":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T06:17:59Z","timestamp":1774592279000},"page":"698-722","source":"Crossref","is-referenced-by-count":0,"title":["Adaptive curriculum reinforcement learning with sim-to-real strategy in balance control of underactuated triple pendulum robots"],"prefix":"10.1017","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3508-7815","authenticated-orcid":false,"given":"Yunfan","family":"Fu","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9589-7866","authenticated-orcid":false,"given":"Jing","family":"Guo","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7653-3821","authenticated-orcid":false,"given":"Donghao","family":"Li","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junpeng","family":"Chen","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liangliang","family":"Han","sequence":"additional","affiliation":[{"name":"National Key Laboratory of Aerospace Mechanism"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuanju","family":"Qu","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Pan","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/049tv2d57","id-type":"ROR","asserted-by":"publisher"}],"name":"Southern University of Science and Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2026,3,27]]},"reference":[{"key":"S0263574726103282_ref12","unstructured":"[12] Haarnoja, T. , Zhou, A. , Abbeel, P. and Levine, S. , \u201cSoft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,\u201d In: Proceedings of the 35th International Conference on Machine Learning 2018 (PMLR, 2018) pp. 1861\u20131870."},{"key":"S0263574726103282_ref38","doi-asserted-by":"crossref","unstructured":"[38] Tan, J. , Zhang, T. , Coumans, E. , Iscen, A. , Bai, Y. , Hafner, D. , Bohez, S. and Vanhoucke, V. , Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. In: Robotics: Science and Systems XIV (2018).10.15607\/RSS.2018.XIV.010","DOI":"10.15607\/RSS.2018.XIV.010"},{"key":"S0263574726103282_ref37","volume-title":"Reinforcement Learning, Second Edition: An Introduction","author":"Sutton","year":"2018"},{"key":"S0263574726103282_ref16","doi-asserted-by":"publisher","DOI":"10.1109\/TMECH.2016.2519529"},{"key":"S0263574726103282_ref47","doi-asserted-by":"publisher","DOI":"10.1115\/1.4001003"},{"key":"S0263574726103282_ref32","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6907267"},{"key":"S0263574726103282_ref46","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561961"},{"key":"S0263574726103282_ref24","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460528"},{"key":"S0263574726103282_ref25","unstructured":"[25] py0330. py0330\/aris. py0330-https:\/\/github.com\/py0330\/aris. Accessed: March 11, 2026."},{"key":"S0263574726103282_ref10","doi-asserted-by":"publisher","DOI":"10.1109\/WCICA.2014.7052991"},{"key":"S0263574726103282_ref14","doi-asserted-by":"publisher","DOI":"10.1016\/j.promfg.2018.06.004"},{"key":"S0263574726103282_ref31","doi-asserted-by":"publisher","DOI":"10.1007\/s11071-020-05719-y"},{"key":"S0263574726103282_ref2","doi-asserted-by":"crossref","unstructured":"[2] Allione, F. , Gkikakis, A. E. and Featherstone, R. , \u201cExperimental Demonstration of a General Balancing Controller on an Untethered Planar Inverted Double Pendulum,\u201d In: 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022 (IEEE, 2022) pp. 8292\u20138297.10.1109\/IROS47612.2022.9981380","DOI":"10.1109\/IROS47612.2022.9981380"},{"key":"S0263574726103282_ref8","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-4141-9"},{"key":"S0263574726103282_ref33","unstructured":"[33] Schulman, J. , Wolski, F. , Dhariwal, P. , Radford, A. and Klimov, O. , \u201cProximal policy optimization algorithms (2017). arXiv preprint arXiv: 1707.06347."},{"key":"S0263574726103282_ref4","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553380"},{"key":"S0263574726103282_ref39","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"S0263574726103282_ref11","unstructured":"[11] Fujimoto, S. , Hoof, H. and Meger, D. , \u201c Addressing Function Approximation Error in Actor-Critic Methods,\u201d In: Proceedings of the 35th International Conference on Machine Learning 2018 (PMLR, 2018) pp. 1587\u20131596."},{"key":"S0263574726103282_ref17","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abc5986"},{"key":"S0263574726103282_ref36","doi-asserted-by":"publisher","DOI":"10.1017\/S026357472500044X"},{"key":"S0263574726103282_ref41","doi-asserted-by":"publisher","DOI":"10.3901\/JME.2007.08.032"},{"key":"S0263574726103282_ref40","unstructured":"[40] Towers, M. , Kwiatkowski, A. , Terry, J. , Balis, J. U. , De Cola, G. , Deleu, T. , Goulao, M. , Kallinteris, A. , Krimmel, M. , KG, A. , \u201cGymnasium: A standard interface for reinforcement learning environments (2024), arXiv preprint arXiv: 2407.17032."},{"key":"S0263574726103282_ref22","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"S0263574726103282_ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2019.12.367"},{"key":"S0263574726103282_ref43","doi-asserted-by":"publisher","DOI":"10.1016\/j.mechmachtheory.2021.104245"},{"key":"S0263574726103282_ref19","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2024.3469822"},{"key":"S0263574726103282_ref44","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3297274"},{"key":"S0263574726103282_ref7","doi-asserted-by":"publisher","DOI":"10.1002\/rob.10031"},{"key":"S0263574726103282_ref30","doi-asserted-by":"publisher","DOI":"10.3934\/math.20241264"},{"key":"S0263574726103282_ref20","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN48605.2020.9207427"},{"key":"S0263574726103282_ref26","doi-asserted-by":"publisher","DOI":"10.1016\/j.mechmachtheory.2014.06.006"},{"key":"S0263574726103282_ref48","doi-asserted-by":"publisher","DOI":"10.1002\/rob.10075"},{"key":"S0263574726103282_ref45","unstructured":"[45] Xie, Z. , Clary, P. , Dao, J. , Morais, P. , Hurst, J. and Panne, M. , \u201cLearning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real,\u201d In: Proceedings of the Conference on Robot Learning 2020 (PMLR, 2020) pp. 317\u2013329."},{"key":"S0263574726103282_ref1","unstructured":"[1] Installing C++ Distributions of PyTorch \u2014 PyTorch main documentation. https:\/\/pytorch.org\/cppdocs\/installing.html."},{"key":"S0263574726103282_ref34","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574725000426"},{"key":"S0263574726103282_ref42","unstructured":"[42] Wang, L. , Zheng, Z. and Lin, Y. , \u201cSteady-State Error Compensation for Reinforcement Learning with Quadratic Rewards,\u201d In: 2024 14th Asian Control Conference (ASCC) (IEEE, 2024) pp. 1608\u20131613."},{"key":"S0263574726103282_ref5","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574700019792"},{"key":"S0263574726103282_ref18","unstructured":"[18] Leitianjian. Leitianjian\/sire. https:\/\/github.com\/leitianjian\/sire. Accessed: March 11, 2026."},{"key":"S0263574726103282_ref29","doi-asserted-by":"publisher","DOI":"10.3390\/fractalfract8090548"},{"key":"S0263574726103282_ref9","doi-asserted-by":"publisher","DOI":"10.1177\/0278364917691114"},{"key":"S0263574726103282_ref21","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574724001061"},{"key":"S0263574726103282_ref49","doi-asserted-by":"crossref","first-page":"3416","DOI":"10.1017\/S0263574724001437","article-title":"Online robust self-learning terminal sliding mode control for balancing control of reaction wheel bicycle robots","volume":"42","author":"Zhu","year":"2024","journal-title":"Robotica"},{"key":"S0263574726103282_ref15","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981516"},{"key":"S0263574726103282_ref13","doi-asserted-by":"publisher","DOI":"10.1177\/1729881419862164"},{"key":"S0263574726103282_ref35","doi-asserted-by":"publisher","DOI":"10.1109\/TIE.2020.2994886"},{"key":"S0263574726103282_ref23","first-page":"1","article-title":"Curriculum learning for reinforcement learning domains: A framework and survey","volume":"21","author":"Narvekar","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"S0263574726103282_ref3","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.107518"},{"key":"S0263574726103282_ref27","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.adi9579"},{"key":"S0263574726103282_ref28","doi-asserted-by":"publisher","DOI":"10.3390\/math11214463"}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574726103282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T03:31:51Z","timestamp":1778470311000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574726103282\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["S0263574726103282"],"URL":"https:\/\/doi.org\/10.1017\/s0263574726103282","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"value":"0263-5747","type":"print"},{"value":"1469-8668","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2]]}}}