{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,1]],"date-time":"2026-11-01T10:39:55Z","timestamp":1793529595562,"version":"4.5.0"},"reference-count":84,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,11,9]],"date-time":"2023-11-09T00:00:00Z","timestamp":1699488000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Deployment of Reinforcement Learning (RL) algorithms for robotics applications in the real world requires ensuring the safety of the robot and its environment. Safe Robot RL (SRRL) is a crucial step toward achieving human-robot coexistence. In this paper, we envision a human-centered SRRL framework consisting of three stages: safe exploration, safety value alignment, and safe collaboration. We examine the research gaps in these areas and propose to leverage interactive behaviors for SRRL. Interactive behaviors enable bi-directional information transfer between humans and robots, such as conversational robot ChatGPT. We argue that interactive behaviors need further attention from the SRRL community. We discuss four open challenges related to the robustness, efficiency, transparency, and adaptability of SRRL with interactive behaviors.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1280341","type":"journal-article","created":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T02:57:15Z","timestamp":1699844235000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["A human-centered safe robot reinforcement learning framework with interactive behaviors"],"prefix":"10.3389","volume":"17","author":[{"given":"Shangding","family":"Gu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alap","family":"Kshirsagar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yali","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guang","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"Peters","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alois","family":"Knoll","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,11,9]]},"reference":[{"key":"B1","first-page":"22","article-title":"\u201cConstrained policy optimization,\u201d","author":"Achiam","year":"2017","journal-title":"International Conference on Machine Learning"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2014.7039601","article-title":"\u201cReachability-based safe learning with Gaussian processes,\u201d","author":"Akametalu","year":"2014","journal-title":"53rd IEEE Conference on Decision and Control"},{"key":"B3","first-page":"25","article-title":"\u201cBehavioral cloning of student pilots with modular neural networks,\u201d","author":"Anderson","year":"2000","journal-title":"ICML"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1145\/3357236.3395525","article-title":"\u201cA survey on interactive reinforcement learning: design principles and open challenges,\u201d","author":"Arzate Cruz","year":"2020","journal-title":"Proceedings of the 2020 ACM Designing Interactive Systems Conference"},{"key":"B5","article-title":"Explainable artificial intelligence for autonomous driving: a comprehensive overview and field guide for future research directions","author":"Atakishiyev","year":"","journal-title":"arXiv preprint arXiv:2112.11561"},{"key":"B6","article-title":"Towards safe, explainable, and regulated autonomous driving","author":"Atakishiyev","year":"","journal-title":"arXiv preprint arXiv:2111.10518"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9560738","article-title":"\u201cGosafe: globally optimal safe robot learning,\u201d","author":"Baumann","year":"2021","journal-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2016.7798979","article-title":"\u201cSafe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes,\u201d","author":"Berkenkamp","year":"2016","journal-title":"2016 IEEE 55th Conference on Decision and Control (CDC)"},{"key":"B9","article-title":"\u201cConservative safety critics for exploration,\u201d","author":"Bharadhwaj","year":"2021","journal-title":"International Conference on Learning Representations (ICLR)"},{"key":"B10","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1146\/annurev-control-042920-020211","article-title":"Safe learning in robotics: from learning-based control to safe reinforcement learning","volume":"5","author":"Brunke","year":"2021","journal-title":"Annu. Rev. Control Robot. Auton. Syst"},{"key":"B11","article-title":"\u201cOn the utility of learning about humans for human-ai coordination,\u201d","author":"Carroll","year":"2019","journal-title":"Advances in Neural Information Processing Systems, Vol"},{"key":"B12","article-title":"\u201cLearning end-to-end 6dof grasp choice of human-to-robot handover using affordance prediction and deep reinforcement learning,\u201d","author":"Chang","year":"2022","journal-title":"2022 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B13","article-title":"\u201cA Lyapunov-based approach to safe reinforcement learning,\u201d","author":"Chow","year":"2018","journal-title":"Advances in Neural Information Processing Systems, Vol. 31"},{"key":"B14","article-title":"Lyapunov-based safe policy optimization for continuous control","author":"Chow","year":"2019","journal-title":"arXiv preprint arXiv:1901.10031"},{"key":"B15","article-title":"Supervising strong learners by amplifying weak experts","author":"Christiano","year":"2018","journal-title":"arXiv preprint arXiv:1810.08575"},{"key":"B16","first-page":"4403","article-title":"\u201cLIIR: learning individual intrinsic reward in multi-agent reinforcement learning,\u201d","author":"Du","year":"2019","journal-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS)"},{"key":"B17","article-title":"Parenting: safe reinforcement learning from human input","author":"Frye","year":"2019","journal-title":"arXiv preprint arXiv:1902.06766"},{"key":"B18","author":"Fudenberg","year":"1991","journal-title":"Game Theory"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12107","article-title":"\u201cSafe reinforcement learning via formal methods: toward safe control through proof and learning,\u201d","author":"Fulton","year":"2018","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32"},{"key":"B20","article-title":"Scaling laws for reward model overoptimization","author":"Gao","year":"2022","journal-title":"arXiv preprint arXiv:2210.10760"},{"key":"B21","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1613\/jair.3761","article-title":"Safe exploration of state and action spaces in reinforcement learning","volume":"45","author":"Garcia","year":"2012","journal-title":"J. Artif. Intell. Res"},{"key":"B22","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1109\/LRA.2020.3047730","article-title":"Human-centered collaborative robots with deep reinforcement learning","volume":"6","author":"Ghadirzadeh","year":"2020","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B23","doi-asserted-by":"publisher","first-page":"81","DOI":"10.3390\/robotics11040081","article-title":"Constrained reinforcement learning for vehicle motion planning with topological reachability analysis","volume":"11","author":"Gu","year":"","journal-title":"Robotics"},{"key":"B24","doi-asserted-by":"publisher","first-page":"103905","DOI":"10.1016\/j.artint.2023.103905","article-title":"Safe multi-agent reinforcement learning for multi-robot control","volume":"319","author":"Gu","year":"2023","journal-title":"Artif. Intell"},{"key":"B25","article-title":"A review of safe reinforcement learning: methods, theory and applications","author":"Gu","year":"","journal-title":"arXiv preprint arXiv:2205.10330"},{"key":"B26","first-page":"2576","article-title":"\u201cGrid-wise control for multi-agent reinforcement learning in video game AI,\u201d","author":"Han","year":"2019","journal-title":"International Conference on Machine Learning (ICML)"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020233","article-title":"\u201cImproving robot controller transparency through autonomous policy explanation,\u201d","author":"Hayes","year":"2017","journal-title":"2017 12th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1145\/3447928.3456653","article-title":"\u201cVerifiably safe exploration for end-to-end reinforcement learning,\u201d","author":"Hunt","year":"2021","journal-title":"Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control"},{"key":"B29","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1177\/0278364920987859","article-title":"How to train your robot with deep reinforcement learning: lessons we have learned","volume":"40","author":"Ibarz","year":"2021","journal-title":"Int. J. Robot. Res"},{"key":"B30","doi-asserted-by":"publisher","first-page":"6838","DOI":"10.1109\/LRA.2022.3177294","article-title":"Safeapt: safe simulation-to-real robot learning using diverse policies learned in simulation","volume":"7","author":"Kaushik","year":"2022","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B31","first-page":"1654","article-title":"\u201cHow to train your agent: active learning from human preferences and justifications in safety-critical environments,\u201d","author":"Kazantzidis","year":"2022","journal-title":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1145\/1597735.1597738","article-title":"\u201cInteractively shaping agents via human reinforcement: the tamer framework,\u201d","author":"Knox","year":"2009","journal-title":"Proceedings of the Fifth International Conference on KNOWLEDGE CAPTURE"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1109\/OJCSYS.2023.3256305","article-title":"Provably safe reinforcement learning via action projection using reachability analysis and polynomial zonotopes","author":"Kochdumper","year":"2022","journal-title":"arXiv preprint arXiv:2210.10691"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2018.8619572","article-title":"\u201cLearning-based model predictive control for safe exploration,\u201d","author":"Koller","year":"2018","journal-title":"2018 IEEE Conference on Decision and Control (CDC)"},{"key":"B35","doi-asserted-by":"publisher","first-page":"4611","DOI":"10.1109\/ACCESS.2018.2887341","article-title":"Human interactive behavior: a bibliographic review","volume":"7","author":"Kong","year":"2018","journal-title":"IEEE Access"},{"key":"B36","article-title":"Provably safe reinforcement learning: a theoretical and experimental comparison","author":"Krasowski","year":"2022","journal-title":"arXiv preprint arXiv:2205.06750"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC45102.2020.9294259","article-title":"\u201cSafe reinforcement learning for autonomous lane changing using set-based prediction,\u201d","author":"Krasowski","year":"2020","journal-title":"2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)"},{"key":"B38","first-page":"1395","article-title":"A review of robot learning for manipulation: challenges, representations, and algorithms","volume":"22","author":"Kroemer","year":"2021","journal-title":"J. Mach. Learn. Res"},{"key":"B39","doi-asserted-by":"publisher","first-page":"3933","DOI":"10.1109\/LRA.2021.3067299","article-title":"Evaluating guided policy search for human-robot handovers","volume":"6","author":"Kshirsagar","year":"2021","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B40","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-51532-8_10","article-title":"\u201cLearning dynamic robot-to-human object handover from human feedback,\u201d","author":"Kupcsik","year":"2018","journal-title":"Robotics Research. Springer Proceedings in Advanced Robotics, Vol. 2"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561036","article-title":"\u201cAdversarial training is not ready for robot learning,\u201d","author":"Lechner","year":"2021","journal-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B42","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA46639.2022.9811766","article-title":"\u201cTowards safe, realistic testbed for robotic systems with human interaction,\u201d","author":"Lee","year":"2022","journal-title":"2022 International Conference on Robotics and Automation (ICRA)"},{"key":"B44","article-title":"Scalable agent alignment via reward modeling: a research direction","author":"Leike","year":"2018","journal-title":"arXiv preprint arXiv:1811.07871"},{"key":"B45","article-title":"Temporal logic guided safe reinforcement learning using control barrier functions","author":"Li","year":"2019","journal-title":"arXiv preprint arXiv:1903.09885"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2020.103568","article-title":"Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving","author":"Likmeta","year":"2020","journal-title":"Robot. Auton. Syst"},{"key":"B47","doi-asserted-by":"publisher","first-page":"120757","DOI":"10.1109\/ACCESS.2020.3006254","article-title":"A review on interactive reinforcement learning from human social feedback","volume":"8","author":"Lin","year":"2020","journal-title":"IEEE Access"},{"key":"B48","first-page":"608","article-title":"\u201cRobust regression for safe exploration in control,\u201d","author":"Liu","year":"2020","journal-title":"Learning for Dynamics and Control"},{"key":"B49","first-page":"1357","article-title":"\u201cRobot reinforcement learning on the constraint manifold,\u201d","author":"Liu","year":"2022","journal-title":"Conference on Robot Learning"},{"key":"B50","article-title":"\u201cMeta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,\u201d","author":"Liu","year":"2022","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"B51","article-title":"On the robustness of safe reinforcement learning under observational perturbations","author":"Liu","year":"2022","journal-title":"arXiv preprint arXiv:2205.14691"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2023.110040","article-title":"EGNN: graph structure learning based on evolutionary computation helps more in graph neural networks","author":"Liu","year":"2023","journal-title":"Appl. Soft Comput"},{"key":"B53","first-page":"1654","article-title":"\u201cPecan: leveraging policy ensemble for context-aware zero-shot human-AI coordination,\u201d","author":"Lou","year":"2023","journal-title":"Proceedings of the 22st International Conference on Autonomous Agents and Multiagent Systems (AAMAS)"},{"key":"B54","first-page":"2285","article-title":"\u201cInteractive learning from policy-dependent human feedback,\u201d","author":"MacGlashan","year":"2017","journal-title":"International Conference on Machine Learning"},{"key":"B55","doi-asserted-by":"publisher","first-page":"1439","DOI":"10.1109\/LRA.2021.3057055","article-title":"Robot learning with crash constraints","volume":"6","author":"Marco","year":"2021","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B56","doi-asserted-by":"publisher","first-page":"1923","DOI":"10.1002\/rnc.5132","article-title":"Safe reinforcement learning: a control barrier function optimization approach","volume":"31","author":"Marvi","year":"2021","journal-title":"Int. J. Robust Nonlin. Control"},{"key":"B57","doi-asserted-by":"publisher","first-page":"578","DOI":"10.1109\/THMS.2021.3116119","article-title":"Toward robots' behavioral transparency of temporal difference reinforcement learning with a human teacher","volume":"51","author":"Matarese","year":"2021","journal-title":"IEEE Trans. Hum. Mach. Syst"},{"key":"B58","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1016\/j.ins.2022.11.051","article-title":"Integrating safety constraints into adversarial training for robust deep reinforcement learning","volume":"619","author":"Meng","year":"2022","journal-title":"Inform. Sci"},{"key":"B59","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s10703-016-0241-z","article-title":"Modelplex: verified runtime validation of verified cyber-physical system models","volume":"49","author":"Mitsch","year":"2016","journal-title":"Formal Methods Syst. Des"},{"key":"B60","doi-asserted-by":"publisher","first-page":"655","DOI":"10.1109\/TCYB.2015.2412554","article-title":"Optimized assistive human-robot interaction using reinforcement learning","volume":"46","author":"Modares","year":"2015","journal-title":"IEEE Trans. Cybern"},{"key":"B61","unstructured":"Chatgpt2023"},{"key":"B62","first-page":"8020","article-title":"\u201cLearning human objectives by evaluating hypothetical behavior,\u201d","author":"Reddy","year":"2020","journal-title":"International Conference on Machine Learning"},{"key":"B63","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1007\/s10846-020-01183-3","article-title":"Model-based reinforcement learning variable impedance control for human-robot collaboration","volume":"100","author":"Roveda","year":"2020","journal-title":"J. Intell. Robot. Syst"},{"key":"B64","first-page":"2067","article-title":"\u201cTrial without error: towards safe reinforcement learning via human intervention,\u201d","author":"Saunders","year":"2018","journal-title":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems"},{"key":"B65","doi-asserted-by":"publisher","first-page":"102432","DOI":"10.1016\/j.rcim.2022.102432","article-title":"Human-robot collaboration and machine learning: a systematic review of recent research","volume":"79","author":"Semeraro","year":"2023","journal-title":"Robot. Comput. Integr. Manufact"},{"key":"B66","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"B67","first-page":"3008","article-title":"\u201cLearning to summarize with human feedback,\u201d","volume":"33","author":"Stiennon","year":"2020","journal-title":"Advances in Neural Information Processing Systems, Vol"},{"key":"B68","first-page":"14502","article-title":"\u201cCollaborating with humans without human data,\u201d","author":"Strouse","year":"2021","journal-title":"Advances in Neural Information Processing Systems, Vol. 34"},{"key":"B69","first-page":"997","article-title":"\u201cSafe exploration for optimization with Gaussian processes,\u201d","author":"Sui","year":"2015","journal-title":"International Conference on Machine Learning"},{"key":"B70","first-page":"4781","article-title":"\u201cStagewise safe Bayesian optimization with Gaussian processes,\u201d","author":"Sui","year":"2018","journal-title":"International Conference on Machine Learning"},{"key":"B71","author":"Sutton","year":"2018","journal-title":"Reinforcement Learning: An Introduction"},{"key":"B72","first-page":"1651","article-title":"\u201cPolicy gradients with variance related risk criteria,\u201d","author":"Tamar","year":"2012","journal-title":"Proceedings of the 29th International Coference on International Conference on Machine Learning"},{"key":"B73","first-page":"1000","article-title":"\u201cReinforcement learning with human teachers: evidence of feedback and guidance with implications for learning performance,\u201d","author":"Thomaz","year":"2006","journal-title":"Proceedings of the 21st National Conference on Artificial Intelligence, Vol. 1"},{"key":"B74","unstructured":"TurchettaM.\n          Ph.D. thesisSafety and robustness in reinforcement learning2021"},{"key":"B75","article-title":"\u201cSafe exploration in finite Markov decision processes with Gaussian processes,\u201d","author":"Turchetta","year":"2016","journal-title":"Advances in Neural Information Processing Systems, Vol. 29"},{"key":"B76","article-title":"\u201cSafe exploration for interactive machine learning,\u201d","author":"Turchetta","year":"2019","journal-title":"Advances in Neural Information Processing Systems, Vol. 32"},{"key":"B77","article-title":"\u201cContrastive explanations for reinforcement learning in terms of expected consequences,\u201d","author":"van der Waa","year":"2018","journal-title":"Proceedings of the Workshop on Explainable AI on the IJCAI Conference, Vol"},{"key":"B78","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12103","article-title":"\u201cSafe exploration and optimization of constrained MDPS using Gaussian processes,\u201d","author":"Wachi","year":"2018","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32"},{"key":"B79","doi-asserted-by":"publisher","first-page":"890","DOI":"10.1109\/TCSS.2022.3164719","article-title":"Heterogeneous network representation learning approach for ethereum identity identification","volume":"10","author":"Wang","year":"2022","journal-title":"IEEE Trans. Comput. Soc. Syst"},{"key":"B80","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.neucom.2020.09.055","article-title":"Safety robustness of reinforcement learning policies: a view from robust control","volume":"422","author":"Xiong","year":"2021","journal-title":"Neurocomputing"},{"key":"B81","article-title":"Safe model-based reinforcement learning with an uncertainty-aware reachability certificate","author":"Yu","year":"2022","journal-title":"arXiv preprint arXiv:2210.07553"},{"key":"B82","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abm4183","article-title":"In situ bidirectional human-robot value alignment","author":"Yuan","year":"2022","journal-title":"Sci. Robot"},{"key":"B83","doi-asserted-by":"publisher","first-page":"1108","DOI":"10.1007\/s12555-021-0882-6","article-title":"Output-feedback robust tracking control of uncertain systems via adaptive learning","volume":"21","author":"Zhao","year":"2023","journal-title":"Int. J. Control Autom. Syst"},{"key":"B84","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i1.16156","article-title":"\u201cDear: deep reinforcement learning for online advertising impression in recommender systems,\u201d","author":"Zhao","year":"2021","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280341\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T02:57:31Z","timestamp":1699844251000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280341\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,9]]},"references-count":84,"alternative-id":["10.3389\/fnbot.2023.1280341"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1280341","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,9]]},"article-number":"1280341"}}