{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:19:20Z","timestamp":1777407560904,"version":"3.51.4"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,5,1]],"date-time":"2024-05-01T00:00:00Z","timestamp":1714521600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>In uncertain environments with robot input saturation, both model-based reinforcement learning (MBRL) and traditional controllers struggle to perform control tasks optimally. In this study, an algorithmic framework of Curiosity Model Policy Optimization (CMPO) is proposed by combining curiosity and model-based approach, where tracking errors are reduced via training agents on control gains for traditional model-free controllers. To begin with, a metric for judging positive and negative curiosity is proposed. Constrained optimization is employed to update the curiosity ratio, which improves the efficiency of agent training. Next, the novelty distance buffer ratio is defined to reduce bias between the environment and the model. Finally, CMPO is simulated with traditional controllers and baseline MBRL algorithms in the robotic environment designed with non-linear rewards. The experimental results illustrate that the algorithm achieves superior tracking performance and generalization capabilities.<\/jats:p>","DOI":"10.3389\/fnbot.2024.1376215","type":"journal-article","created":{"date-parts":[[2024,5,1]],"date-time":"2024-05-01T05:07:47Z","timestamp":1714540067000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Curiosity model policy optimization for robotic manipulator tracking control with input saturation in uncertain environment"],"prefix":"10.3389","volume":"18","author":[{"given":"Tu","family":"Wang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fujie","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhongye","family":"Xie","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Feiyan","family":"Qin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,5,1]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511804441","volume-title":"Convex Optimization","author":"Boyd","year":"2004"},{"key":"B2","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1146\/annurev-control-042920-020211","article-title":"Safe learning in robotics: from learning-based control to safe reinforcement learning","volume":"5","author":"Brunke","year":"2022","journal-title":"Ann. Rev. Control, Robot. Auton. Syst"},{"key":"B3","article-title":"Large-scale study of curiosity-driven learning","author":"Burda","year":"","journal-title":"arXiv preprint arXiv:1808.04355"},{"key":"B4","article-title":"Exploration by random network distillation","author":"Burda","year":"","journal-title":"arXiv preprint arXiv:1810.12894"},{"key":"B5","doi-asserted-by":"publisher","first-page":"4584","DOI":"10.1109\/TNNLS.2021.3116713","article-title":"Reinforcement learning-based fixed-time trajectory tracking control for uncertain robotic manipulators with input saturation","volume":"34","author":"Cao","year":"2021","journal-title":"IEEE Trans. Neural Netw. Lear. Syst"},{"key":"B6","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1016\/j.isatra.2022.07.024","article-title":"Averaged sub-gradient integral sliding mode control design for cueing end-effector acceleration of a two-link robotic arm","volume":"133","author":"Chertopolokhov","year":"2023","journal-title":"ISA Trans"},{"key":"B7","article-title":"\u201cDeep reinforcement learning in a handful of trials using probabilistic dynamics models,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Chua","year":"2018"},{"key":"B8","doi-asserted-by":"publisher","first-page":"102517","DOI":"10.1016\/j.rcim.2022.102517","article-title":"A review on reinforcement learning for contact-rich robotic manipulation tasks","volume":"81","author":"Elguea-Aguinaco","year":"2023","journal-title":"Robot. Comput. Integr. Manuf"},{"key":"B9","first-page":"1","author":"Gao","year":"2023"},{"key":"B10","doi-asserted-by":"publisher","first-page":"21271","DOI":"10.48550\/arXiv.2006.07733","article-title":"Bootstrap your own latent: a new approach to self-supervised learning","volume":"33","author":"Grill","year":"2020","journal-title":"Adv. Neural Inf. Proc. Syst"},{"key":"B11","doi-asserted-by":"publisher","first-page":"2048","DOI":"10.1109\/TMECH.2020.3030032","article-title":"Quasi-synchronization control of multiple electrohydraulic actuators with load disturbance and uncertain parameters","volume":"26","author":"Guo","year":"2021","journal-title":"IEEE\/ASME Trans. Mechatr"},{"key":"B12","article-title":"Soft actor-critic algorithms and applications","author":"Haarnoja","year":"2019","journal-title":"arXiv preprint arXiv:1812.05905"},{"key":"B13","doi-asserted-by":"publisher","first-page":"3570","DOI":"10.1109\/TNNLS.2019.2945019","article-title":"Reinforcement learning tracking control for robotic manipulator with kernel-based dynamic model","volume":"31","author":"Hu","year":"2020","journal-title":"IEEE Trans. Neural Netw. Lear. Syst"},{"key":"B14","doi-asserted-by":"publisher","first-page":"108352","DOI":"10.1016\/j.patcog.2021.108352","article-title":"Unified curiosity-driven learning with smoothed intrinsic reward estimation","volume":"123","author":"Huang","year":"2022","journal-title":"Patt. Recogn"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2444","DOI":"10.1109\/TIE.2010.2062472","article-title":"Robust sliding mode control for robot manipulators","volume":"58","author":"Islam","year":"2011","journal-title":"IEEE Trans. Ind. Electr"},{"key":"B16","article-title":"\u201cWhen to trust your model: model-based policy optimization,\u201d","author":"Janner","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B17","article-title":"Human-level Atari 200x faster","author":"Kapturowski","year":"2022","journal-title":"arXiv preprint arXiv:2209.07550"},{"key":"B18","article-title":"Model-ensemble trust-region policy optimization","author":"Kurutach","year":"2018","journal-title":"arXiv preprint arXiv:1802.10592"},{"key":"B19","first-page":"3694","article-title":"\u201cOn effective scheduling of model-based reinforcement learning,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Lai","year":"2021"},{"key":"B20","first-page":"5618","article-title":"\u201cBidirectional model-based policy optimization,\u201d","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Lai","year":"2020"},{"key":"B21","first-page":"5757","article-title":"\u201cContext-aware dynamics model for generalization in model-based reinforcement learning,\u201d","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Lee","year":"2020"},{"key":"B22","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/j.neucom.2020.08.024","article-title":"Random curiosity-driven exploration in deep reinforcement learning","volume":"418","author":"Li","year":"2020","journal-title":"Neurocomputing"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2055","DOI":"10.3390\/math9172055","article-title":"Adaptive proportional integral robust control of an uncertain robotic manipulator based on deep deterministic policy gradient","volume":"9","author":"Lu","year":"2021","journal-title":"Mathematics"},{"key":"B24","doi-asserted-by":"publisher","first-page":"121101","DOI":"10.1007\/s11432-022-3696-5","article-title":"A survey on model-based reinforcement learning","volume":"67","author":"Luo","year":"2022","journal-title":"Sci. China Inf. Sci"},{"key":"B25","article-title":"Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees","author":"Luo","year":"2021","journal-title":"arXiv preprint arXiv:1807.03858"},{"key":"B26","article-title":"Playing atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"arXiv preprint arXiv:1312.5602"},{"key":"B27","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1016\/j.engappai.2018.11.006","article-title":"Reinforcement learning based compensation methods for robot manipulators","volume":"78","author":"Pane","year":"2019","journal-title":"Eng. Applic. Artif. Intell"},{"key":"B28","first-page":"2778","article-title":"\u201cCuriosity-driven exploration by self-supervised prediction,\u201d","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Pathak","year":"2017"},{"key":"B29","article-title":"Deep dyna-q: integrating planning for task-completion dialogue policy learning","author":"Peng","year":"2018","journal-title":"arXiv preprint arXiv:1801.06176"},{"key":"B30","doi-asserted-by":"publisher","first-page":"2823","DOI":"10.48550\/arXiv.2010.09546","article-title":"Model-based policy optimization with unsupervised model adaptation","volume":"33","author":"Shen","year":"2020","journal-title":"Adv. Neural Inf. Proc. Syst"},{"key":"B31","article-title":"Incentivizing exploration in reinforcement learning with deep predictive models","author":"Stadie","year":"2015","journal-title":"arXiv preprint arXiv:1507.00814"},{"key":"B32","article-title":"CCLF: a contrastive-curiosity-driven learning framework for sample-efficient reinforcement learning","author":"Sun","year":"","journal-title":"arXiv preprint arXiv:2205.00943"},{"key":"B33","article-title":"From psychological curiosity to artificial curiosity: curiosity-driven learning in artificial intelligence tasks","author":"Sun","year":"","journal-title":"arXiv preprint arXiv:2201.08300"},{"key":"B34","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1109\/TRO.2018.2878318","article-title":"Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators","volume":"35","author":"Thuruthel","year":"2019","journal-title":"IEEE Trans. Robot"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1007\/s40998-019-00286-4","article-title":"Trajectory tracking control for mobile robots using reinforcement learning and PID","volume":"44","author":"Wang","year":"2020","journal-title":"Iranian J. Sci. Technol. Trans. Electr. Eng"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2023.3280750","article-title":"\u201cContact force estimation of robot manipulators with imperfect dynamic model: on gaussian process adaptive disturbance kalman filter,\u201d","author":"Wei","year":"2023","journal-title":"IEEE Transactions on Automation Science and Engineering"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i8.20847","article-title":"\u201cGeneralizing reinforcement learning through fusing self-supervised learning into intrinsic motivation,\u201d","author":"Wu","year":"2022","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1658","DOI":"10.1109\/TII.2018.2868859","article-title":"Feedback deep deterministic policy gradient with fuzzy reward for robotic multiple peg-in-hole assembly tasks","volume":"15","author":"Xu","year":"2019","journal-title":"IEEE Trans. Industr. Inf"},{"key":"B39","article-title":"Never forget: balancing exploration and exploitation via learning optical flow","author":"Yang","year":"2019","journal-title":"arXiv preprint arXiv:1901.08486"},{"key":"B40","doi-asserted-by":"publisher","first-page":"718","DOI":"10.1109\/TIE.2023.3250753","article-title":"Parameter identification of hydraulic manipulators considering physical feasibility and control stability","volume":"71","author":"Zhang","year":"2024","journal-title":"IEEE Trans. Industr. Electr"},{"key":"B41","article-title":"Curiosity-driven exploration for mapless navigation with deep reinforcement learning","author":"Zhelo","year":"2018","journal-title":"arXiv preprint arXiv:1804.00456"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1376215\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T10:21:25Z","timestamp":1719397285000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1376215\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,1]]},"references-count":41,"alternative-id":["10.3389\/fnbot.2024.1376215"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2024.1376215","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,1]]},"article-number":"1376215"}}