{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T09:18:04Z","timestamp":1785403084453,"version":"3.56.0"},"reference-count":30,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2026,6,7]],"date-time":"2026-06-07T00:00:00Z","timestamp":1780790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["1652454"],"award-info":[{"award-number":["1652454"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["2114464"],"award-info":[{"award-number":["2114464"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]},{"award":["1652454"],"award-info":[{"award-number":["1652454"]}],"id":[{"id":"https:\/\/ror.org\/021nxhr62","id-type":"ROR","asserted-by":"publisher"}]},{"award":["2114464"],"award-info":[{"award-number":["2114464"]}],"id":[{"id":"https:\/\/ror.org\/021nxhr62","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Deep reinforcement learning (DRL) has shown strong capability for multi-finger dexterous in-hand manipulation, where high-dimensional control and complex object interactions make policy learning challenging. However, many existing DRL approaches emphasize task completion and learning efficiency without explicitly accounting for manipulation risk, which can lead to overly aggressive behaviors and unstable object handling. This study proposes Risk-Prioritized Experience Replay (Risk-PER), a replay-sampling strategy that incorporates task-specific risk scores derived from prior transitions. The proposed method assigns each transition a risk score based on three binary indicators related to manipulation instability and then biases replay toward lower-risk experiences while still allowing the agent to learn from risk-related events. Risk-PER is integrated with Deep Deterministic Policy Gradient (DDPG) and evaluated in MuJoCo simulation on two Allegro Hand in-hand manipulation tasks involving a block and an egg. Across the evaluated settings, Risk-PER achieves higher success rates, lower manipulation risk, and more stable learning behavior than HER and reward\u2013penalty-based risk-averse baselines. These results suggest that incorporating task-specific risk awareness into replay prioritization can improve both learning efficiency and manipulation stability in dexterous in-hand manipulation.<\/jats:p>","DOI":"10.3390\/s26123633","type":"journal-article","created":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T05:43:46Z","timestamp":1780897426000},"page":"3633","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Risk-Prioritized Experience Replay for Stable In-Hand Manipulation"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-3005-659X","authenticated-orcid":false,"given":"Yunsik","family":"Jung","sequence":"first","affiliation":[{"name":"Intelligent Robotics and Systems Lab, Colorado School of Mines, Golden, CO 80401, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5989-0587","authenticated-orcid":false,"given":"Lingfeng","family":"Tao","sequence":"additional","affiliation":[{"name":"Department of Robotics and Mechatronics, Kennesaw State University, Kennesaw, GA 30144, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6092-4668","authenticated-orcid":false,"given":"Michael","family":"Bowman","sequence":"additional","affiliation":[{"name":"The Bowman Lab, Cancer Biology Department, University of Pennsylvania, Philadelphia, PA 19104, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiucai","family":"Zhang","sequence":"additional","affiliation":[{"name":"GAC R&D Center Silicon Valley, Sunnyvale, CA 94085, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoli","family":"Zhang","sequence":"additional","affiliation":[{"name":"Intelligent Robotics and Systems Lab, Colorado School of Mines, Golden, CO 80401, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Solowjow, E., Ugalde, I., Shahapurkar, Y., Aparicio, J., Mahler, J., Satish, V., Goldberg, K., and Claussen, H. (2020). Industrial Robot Grasping with Deep Learning Using a Programmable Logic Controller (PLC). 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), IEEE.","DOI":"10.1109\/CASE48305.2020.9216902"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Pusalkar, N.G., Giolando, M., and Adams, J.A. (2023). Autonomous Underwater Robot Grasping Decision Support System. HRI \u201923: Companion of the 2023 ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Association for Computing Machinery.","DOI":"10.1145\/3568294.3580210"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Arunachalam, S.P., Silwal, S., Evans, B., and Pinto, L. (2023). Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation. 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160275"},{"key":"ref_4","unstructured":"Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., and Handa, A. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning. arXiv."},{"key":"ref_5","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Li, T., Srinivasan, K., Meng, M.Q.-H., Yuan, W., and Bohg, J. (2020). Learning Hierarchical Control for Robust In-Hand Manipulation. 2020 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA40945.2020.9197343"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Tao, L., Zhang, J., Bowman, M., and Zhang, X. (2023). A Multi-Agent Approach for Adaptive Finger Cooperation in Learning-Based In-Hand Manipulation. 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160909"},{"key":"ref_8","unstructured":"Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., and Ribas, R. (2019). Solving Rubik\u2019s Cube with a Robot Hand. arXiv."},{"key":"ref_9","first-page":"5048","article-title":"Hindsight Experience Replay","volume":"30","author":"Andrychowicz","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_10","unstructured":"Tao, L., Zhang, J., and Zhang, X. (2023). Stable In-Hand Manipulation with Finger Specific Multi-Agent Shadow Reward. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1049\/trit.2019.0094","article-title":"Learning Synergies-Based In-Hand Manipulation with Reward Shaping","volume":"5","author":"Deng","year":"2020","journal-title":"CAAI Trans. Intell. Technol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Tobin, J., Biewald, L., Duan, R., Andrychowicz, M., Handa, A., Kumar, V., McGrew, B., Ray, A., Schneider, J., and Welinder, P. (2018). Domain Randomization and Generative Models for Robotic Grasping. 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS.2018.8593933"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1109\/TAC.2014.2299335","article-title":"Rapidly Exponentially Stabilizing Control Lyapunov Functions and Hybrid Zero Dynamics","volume":"59","author":"Ames","year":"2014","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Scholz, J., and Stilman, M. (2010). Combining Motion Planning and Optimization for Flexible Robot Manipulation. 2010 10th IEEE-RAS International Conference on Humanoid Robots, IEEE.","DOI":"10.1109\/ICHR.2010.5686849"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fan, Y., Tang, T., Lin, H.-C., Zhao, Y., and Tomizuka, M. (2017). Real-Time Robust Finger Gaits Planning under Object Shape and Dynamics Uncertainties. 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS.2017.8202302"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2017). Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. arXiv.","DOI":"10.15607\/RSS.2018.XIV.049"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/0278364919887447","article-title":"Learning Dexterous In-Hand Manipulation","volume":"39","author":"Andrychowicz","year":"2019","journal-title":"Int. J. Robot. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1298","DOI":"10.1162\/NECO_a_00600","article-title":"Risk-Sensitive Reinforcement Learning","volume":"26","author":"Shen","year":"2014","journal-title":"Neural Comput."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1023\/A:1017940631555","article-title":"Risk-Sensitive Reinforcement Learning","volume":"49","author":"Mihatsch","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_20","unstructured":"Del\u00e9tang, G., Grau-Moya, J., Kunesch, M., Genewein, T., Brekelmans, R., Legg, S., and Ortega, P.A. (2021). Model-Free Risk-Sensitive Reinforcement Learning. arXiv."},{"key":"ref_21","unstructured":"Du, Y., Wang, S., and Huang, L. (2023, January 1\u20135). Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path. Proceedings of the Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda. Available online: https:\/\/openreview.net\/forum?id=Yn0xg-kHNW-."},{"key":"ref_22","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv."},{"key":"ref_23","unstructured":"Hou, Y., and Zhang, Y. (2026, May 27). Improving DDPG via Prioritized Experience Replay. Technical Report, 2019. Available online: https:\/\/cardwing.github.io\/files\/RL_course_report.pdf."},{"key":"ref_24","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016). Continuous Control with Deep Reinforcement Learning. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gao, J., Li, X., Liu, W., and Zhao, J. (2021). Prioritized Experience Replay Method Based on Experience Reward. 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), IEEE.","DOI":"10.1109\/MLISE54096.2021.00045"},{"key":"ref_26","first-page":"8","article-title":"Accelerating Reinforcement Learning with Prioritized Experience Replay for Maze Game","volume":"3","author":"Hu","year":"2020","journal-title":"SMU Data Sci. Rev."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A Physics Engine for Model-Based Control. 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, IEEE.","DOI":"10.1109\/IROS.2012.6386109"},{"key":"ref_28","unstructured":"Barker, B. (2015). Message Passing Interface (MPI). Workshop: High Performance Computing on Stampede, Cornell University Publisher."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3661","DOI":"10.1002\/sim.2832","article-title":"Chi-Squared and Fisher-Irwin Tests of Two-by-Two Tables with Small Sample Recommendations","volume":"26","author":"Campbell","year":"2007","journal-title":"Stat. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/s00362-009-0224-x","article-title":"The Two-Sample t Test: Pre-Testing Its Assumptions Does Not Pay Off","volume":"52","author":"Rasch","year":"2011","journal-title":"Stat. Pap."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3633\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:18:10Z","timestamp":1781324290000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/12\/3633"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,7]]},"references-count":30,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["s26123633"],"URL":"https:\/\/doi.org\/10.3390\/s26123633","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,7]]}}}