{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:05:49Z","timestamp":1753891549665,"version":"3.41.2"},"reference-count":48,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,3,7]],"date-time":"2023-03-07T00:00:00Z","timestamp":1678147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Reinforcement learning (RL) empowers the agent to learn robotic manipulation skills autonomously. Compared with traditional single-goal RL, semantic-goal-conditioned RL expands the agent capacity to accomplish multiple semantic manipulation instructions. However, due to sparsely distributed semantic goals and sparse-reward agent-environment interactions, the hard exploration problem arises and impedes the agent training process. In traditional RL, curiosity-motivated exploration shows effectiveness in solving the hard exploration problem. However, in semantic-goal-conditioned RL, the performance of previous curiosity-motivated methods deteriorates, which we propose is because of their two defects: uncontrollability and distraction. To solve these defects, we propose a conservative curiosity-motivated method named mutual information motivation with hybrid policy mechanism (MIHM). MIHM mainly contributes two innovations: the decoupled-mutual-information-based intrinsic motivation, which prevents the agent from being motivated to explore dangerous states by uncontrollable curiosity; the precisely trained and automatically switched hybrid policy mechanism, which eliminates the distraction from the curiosity-motivated policy and achieves the optimal utilization of exploration and exploitation. Compared with four state-of-the-art curiosity-motivated methods in the sparse-reward robotic manipulation task with 35 valid semantic goals, including stacks of 2 or 3 objects and pyramids, our MIHM shows the fastest learning speed. Moreover, MIHM achieves the highest 0.9 total success rate, which is up to 0.6 in other methods. Throughout all the baseline methods, our MIHM is the only one that achieves to stack three objects.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1089270","type":"journal-article","created":{"date-parts":[[2023,3,7]],"date-time":"2023-03-07T05:41:34Z","timestamp":1678167694000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Learning robotic manipulation skills with multiple semantic goals by conservative curiosity-motivated exploration"],"prefix":"10.3389","volume":"17","author":[{"given":"Changlin","family":"Han","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiyong","family":"Peng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yadong","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingsheng","family":"Tang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Yu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zongtan","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,3,7]]},"reference":[{"key":"B1","unstructured":"AkakziaA.\n            ColasC.\n            OudeyerP-. Y.\n            ChetouaniM.\n            SigaudO.\n          Grounding Language to Autonomously-Acquired Skills via Goal Generation2021"},{"key":"B2","article-title":"Hindsight experience replay","author":"Andrychowicz","year":"2017","journal-title":"arXiv preprint arXiv:"},{"journal-title":"A new Potential-Based Reward Shaping for Reinforcement Learning Agent","year":"2019","author":"Badnava","key":"B3"},{"key":"B4","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1016\/j.neucom.2019.06.022","article-title":"Guided goal generation for hindsight multi-goal reinforcement learning","volume":"359","author":"Bai","year":"2019","journal-title":"Neurocomputing"},{"key":"B5","unstructured":"BelghaziM. I.\n            BaratinA.\n            RajeswarS.\n            OzairS.\n            BengioY.\n            CourvilleA.\n          MINE: Mutual Information Neural Estimation. arXiv:1801.04062 [cs, stat]2021"},{"key":"B6","unstructured":"BrockmanG.\n            CheungV.\n            PetterssonL.\n            SchneiderJ.\n            SchulmanJ.\n            TangJ.\n          OpenAI Gym. arXiv:1606.01540 [cs]2016"},{"journal-title":"Large-Scale Study of Curiosity-Driven Learning","year":"","author":"Burda","key":"B7"},{"key":"B8","unstructured":"BurdaY.\n            EdwardsH.\n            StorkeyA.\n            KlimovO.\n          Exploration by Random Network Distillation."},{"key":"B9","unstructured":"CamposV.\n            TrottA.\n            XiongC.\n            SocherR. Giro-i-Nieto, X.\n            TorresJ.\n          Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills. arXiv:2002.03647 [cs, stat]2020"},{"journal-title":"InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets.","year":"2016","author":"Chen","key":"B10"},{"journal-title":"Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration","year":"2020","author":"Colas","key":"B11"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.13554","author":"Colas","year":"2022","journal-title":"Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey"},{"journal-title":"Go-Explore: A New Approach for Hard-Exploration Problems. arXiv preprint arXiv:","year":"2019","author":"Ecoffet","key":"B13"},{"key":"B14","unstructured":"EysenbachB.\n            GuptaA.\n            IbarzJ.\n            LevineS.\n          Diversity is All You Need: Learning Skills without a Reward Function2018"},{"journal-title":"Addressing Function Approximation Error in Actor-Critic Methods","year":"2018","author":"Fujimoto","key":"B15"},{"key":"B16","unstructured":"HaarnojaT.\n            ZhouA.\n            AbbeelP.\n            LevineS.\n          Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs, stat]2018"},{"key":"B17","unstructured":"HanS.\n            SungY.\n          Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration2021"},{"key":"B18","unstructured":"HesselM.\n            SoyerH.\n            EspeholtL.\n            CzarneckiW.\n            SchmittS.\n            van HasseltH.\n          Multi-task Deep Reinforcement Learning with PopArt2018"},{"key":"B19","unstructured":"HouthooftR.\n            ChenX.\n            DuanY.\n            SchulmanJ.\n            TurckD. e.\n            AbbeelF.\n          VIME: Variational Information Maximizing Exploration2017"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aau5872","article-title":"Learning agile and dynamic motor skills for legged robots","author":"Hwangbo","year":"2019","journal-title":"Sci. Robot."},{"journal-title":"The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal.","year":"2018","author":"Jiao","key":"B21"},{"key":"B22","unstructured":"KimH.\n            KimJ.\n            JeongY.\n            LevineS.\n            SongH. O.\n          EMI: Exploration with Mutual Information. 2019"},{"key":"B23","unstructured":"LanierJ. B.\n            McAleerS.\n            BaldiP.\n          Curiosity-Driven Multi-Criteria Hindsight Experience Replay2019"},{"key":"B24","unstructured":"LeeL.\n            EysenbachB.\n            ParisottoE.\n            XingE.\n            LevineS.\n            SalakhutdinovR.\n          32792510Efficient Exploration via State Marginal Matching2020"},{"key":"B25","unstructured":"LiY.\n            SongJ.\n            ErmonS.\n          InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations2017"},{"key":"B26","unstructured":"LiuH.\n            AbbeelP.\n          Behavior From the Void: Unsupervised Active Pre-Training2021"},{"journal-title":"Playing Atari with Deep Reinforcement Learning","year":"2013","author":"Mnih","key":"B27"},{"key":"B28","unstructured":"MohamedS.\n            RezendeD. J.\n          Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. arXiv:1509.08731 [cs, stat]2015"},{"key":"B29","unstructured":"OstrovskiG.\n            BellemareM. G.\n            OordA.\n            van den\n            MunosR.\n          Count-Based Exploration with Neural Density Models2017"},{"key":"B30","article-title":"How can we define intrinsic motivation?,","author":"Oudeyer","year":"2008","journal-title":"8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems"},{"key":"B31","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1109\/CVPRW.2017.70","article-title":"Curiosity-driven exploration by self-supervised prediction,","author":"Pathak","year":"2017"},{"journal-title":"Episodic Curiosity Through Reachability","year":"2018","author":"Savinov","key":"B32"},{"key":"B33","first-page":"1312","article-title":"Universal value function approximators,","author":"Schaul","year":"2015","journal-title":"International Conference on Machine Learning (PMLR)."},{"journal-title":"Prioritized Experience Replay","year":"2016","author":"Schaul","key":"B34"},{"key":"B35","unstructured":"SchulmanJ.\n            MoritzP.\n            LevineS.\n            JordanM.\n            AbbeelP.\n          High-Dimensional Continuous Control Using Generalized Advantage Estimation2018"},{"key":"B36","unstructured":"SeoY.\n            ChenL.\n            ShinJ.\n            LeeH.\n            AbbeelP.\n            LeeK.\n          State Entropy Maximization with Random Encoders for Efficient Exploration2021"},{"key":"B37","unstructured":"SharmaA.\n            GuS.\n            LevineS.\n            KumarV.\n            HausmanK.\n          Dynamics-Aware Unsupervised Discovery of Skills2020"},{"year":"","author":"Silver","key":"B38"},{"key":"B39","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1080\/01966324.2003.10737616","article-title":"Nearest Neighbor Estimates of Entropy","volume":"23","author":"Singh","year":"2003","journal-title":"Am. J. Math. Manage. Sci."},{"key":"B40","unstructured":"StadieB. C.\n            LevineS.\n            AbbeelP.\n          Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models2015"},{"journal-title":"Reinforcement Learning: An Introduction","year":"2018","author":"Sutton","key":"B41"},{"key":"B42","unstructured":"TangH.\n            HouthooftR.\n            FooteD.\n            StookeA.\n            ChenX.\n            DuanY.\n          2017"},{"key":"B43","doi-asserted-by":"publisher","first-page":"5026","DOI":"10.1109\/IROS.2012.6386109","article-title":"MuJoCo: A physics engine for model-based control,","author":"Todorov","year":"2012","journal-title":"2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems"},{"key":"B44","unstructured":"Learning values across many orders of magnitude,\n            van HasseltH. P.\n            GuezA.\n            GuezA.\n            HesselM.\n            MnihV.\n            SilverD.\n          Advances in Neural Information Processing Systems2016"},{"journal-title":"Hybrid Reward Architecture for Reinforcement Learning","year":"2017","author":"van Seijen","key":"B45"},{"journal-title":"Many-goals Reinforcement Learning","year":"2018","author":"Veeriah","key":"B46"},{"key":"B47","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","article-title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","volume":"575","author":"Vinyals","year":"2019","journal-title":"Nature"},{"journal-title":"Deep Sets","year":"2018","author":"Zaheer","key":"B48"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1089270\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,7]],"date-time":"2023-03-07T05:41:56Z","timestamp":1678167716000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1089270\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,7]]},"references-count":48,"alternative-id":["10.3389\/fnbot.2023.1089270"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1089270","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2023,3,7]]},"article-number":"1089270"}}