{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T16:47:23Z","timestamp":1754153243259,"version":"3.41.2"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>Interactive Reinforcement Learning (IntRL) with human advice has shown great potential for human-guided self-improvement of robots and can accelerate learning compared to traditional Reinforcement Learning. However, most existing approaches assume perfectly correct advice or partially incorrect advice is only accounted for by assessing trustworthiness of human advice equally across all states. This can lead to problems in practical scenarios, where human advice might be inaccurate in some states but still useful in others. We propose a novel IntRL algorithm that handles state-dependent unreliable action advice by computing a trust estimate for both human advice and the robot\u2019s own policy. We use three indicators to assess trustworthiness of human advice, namely consistency of advice, retrospective optimality, and a multi-modal human uncertainty classifier based on behavioral cues. For estimating the state-dependent trust in the robot\u2019s policy, we compare five different methods. Evaluations in gridworlds with simulated advice show that our approach significantly outperforms a state-independent baseline. Robotic experiments with perceptual uncertainty and advice from 26 participants confirm the usefulness of the included human uncertainty classification as an indicator for unreliable advice. Additionally, we show that our approach is more robust to incorrect advice compared to a state-independent computation of trust in the policy.<\/jats:p>","DOI":"10.1145\/3736424","type":"journal-article","created":{"date-parts":[[2025,5,21]],"date-time":"2025-05-21T12:01:31Z","timestamp":1747828891000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Can I Trust You?\u2014Handling Unreliable Human Action Advice in Interactive Reinforcement Learning"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-0950-5184","authenticated-orcid":false,"given":"Lisa","family":"Kempf","sequence":"first","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8104-9049","authenticated-orcid":false,"given":"Christian","family":"Maurer","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4836-6023","authenticated-orcid":false,"given":"Cigdem","family":"Turan-Schwiewager","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3571-6848","authenticated-orcid":false,"given":"Dorothea","family":"Koert","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]}],"member":"320","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"unstructured":"Riku Arakawa Sosuke Kobayashi Yuya Unno Yuta Tsuboi and Shin-Ichi Maeda. 2018. DQN-TAMER: Human-in-the-loop reinforcement learning with intractable feedback. arXiv:1810.11748. Retrieved from https:\/\/arxiv.org\/abs\/1810.11748","key":"e_1_3_2_2_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_3_2","DOI":"10.1145\/3357236.3395525"},{"doi-asserted-by":"publisher","key":"e_1_3_2_4_2","DOI":"10.1007\/978-1-349-20181-5_26"},{"doi-asserted-by":"publisher","key":"e_1_3_2_5_2","DOI":"10.3389\/fnins.2012.00085"},{"doi-asserted-by":"publisher","key":"e_1_3_2_6_2","DOI":"10.5555\/923832"},{"doi-asserted-by":"publisher","key":"e_1_3_2_7_2","DOI":"10.1109\/IJCNN.2015.7280477"},{"key":"e_1_3_2_8_2","volume-title":"Proceedings of the Conference on Robot Learning","author":"Cui Yuchen","year":"2020","unstructured":"Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Niekum, and W. Bradley Knox. 2020. The empathic framework for task learning from implicit human feedback. In Proceedings of the Conference on Robot Learning."},{"key":"e_1_3_2_9_2","first-page":"761","article-title":"Bayesian Q-learning","author":"Dearden Richard","year":"1998","unstructured":"Richard Dearden, Nir Friedman, and Stuart Russell. 1998. Bayesian Q-learning. In Proceedings of the 15th National\/10th Conference on Artificial Intelligence\/Innovative Applications of Artificial Intelligence (AAAI\/IAAI), 761\u2013768.","journal-title":"Proceedings of the 15th National\/10th Conference on Artificial Intelligence\/Innovative Applications of Artificial Intelligence (AAAI\/IAAI)"},{"doi-asserted-by":"publisher","key":"e_1_3_2_10_2","DOI":"10.3389\/frobt.2019.00064"},{"doi-asserted-by":"publisher","key":"e_1_3_2_11_2","DOI":"10.1145\/3152832.3152859"},{"key":"e_1_3_2_12_2","article-title":"Policy shaping: Integrating human feedback with reinforcement learning","volume":"26","author":"Griffith Shane","year":"2013","unstructured":"Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell, and Andrea L. Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 26.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_2","first-page":"131","volume-title":"Proceedings of the Conference on Uncertainty in Artificial Intelligence","author":"He Xu","year":"2020","unstructured":"Xu He, Haipeng Chen, and Bo An. 2020. Learning behaviors with uncertain human feedback. In Proceedings of the Conference on Uncertainty in Artificial Intelligence. PMLR, 131\u2013140."},{"doi-asserted-by":"publisher","key":"e_1_3_2_14_2","DOI":"10.1063\/1.5002892"},{"doi-asserted-by":"publisher","key":"e_1_3_2_15_2","DOI":"10.1098\/rstb.2012.0037"},{"doi-asserted-by":"publisher","key":"e_1_3_2_16_2","DOI":"10.1109\/ICRA40945.2020.9197219"},{"doi-asserted-by":"publisher","key":"e_1_3_2_17_2","DOI":"10.1145\/3434074.3446361"},{"key":"e_1_3_2_18_2","first-page":"292","volume-title":"Proceedings of the 2008 7th IEEE International Conference on Development and Learning","author":"Knox W. Bradley","year":"2008","unstructured":"W. Bradley Knox and Peter Stone. 2008. TAMER: Training an agent manually via evaluative reinforcement. In Proceedings of the 2008 7th IEEE International Conference on Development and Learning. IEEE, 292\u2013297."},{"doi-asserted-by":"publisher","key":"e_1_3_2_19_2","DOI":"10.5555\/1838206.1838208"},{"doi-asserted-by":"publisher","key":"e_1_3_2_20_2","DOI":"10.5555\/2343576.2343644"},{"doi-asserted-by":"publisher","key":"e_1_3_2_21_2","DOI":"10.3389\/frobt.2020.00097"},{"doi-asserted-by":"publisher","key":"e_1_3_2_22_2","DOI":"10.1177\/00238309050480010201"},{"unstructured":"Andrey Kurenkov Ajay Mandlekar Roberto Martin-Martin Silvio Savarese and Animesh Garg. 2019. AC-Teach: A Bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. arXiv:1909.04121. Retrieved from https:\/\/arxiv.org\/abs\/1909.04121","key":"e_1_3_2_23_2"},{"key":"e_1_3_2_24_2","first-page":"2529","volume-title":"Proceedings of the IJCAI International Joint Conference on Artificial Intelligence","author":"Lall\u00e9 S\u00e9bastien","year":"2016","unstructured":"S\u00e9bastien Lall\u00e9, Cristina Conati, and Giuseppe Carenini. 2016. Predicting confusion in information visualization from eye tracking and interaction data. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, 2529\u20132535."},{"doi-asserted-by":"publisher","key":"e_1_3_2_25_2","DOI":"10.1109\/THMS.2019.2912447"},{"unstructured":"Zhiyu Lin Brent Harrison Aaron Keech and Mark O. Riedl. 2017. Explore exploit or listen: Combining human feedback and policy model to speed up deep reinforcement learning in 3D worlds. arXiv:1709.03969. Retrieved from https:\/\/arxiv.org\/abs\/1709.03969","key":"e_1_3_2_26_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_27_2","DOI":"10.1007\/s10458-015-9283-7"},{"key":"e_1_3_2_28_2","volume-title":"Workshop on Human-aligned Reinforcement Learning for Autonomous Agents and Robots.","author":"Love Tamlin","year":"2021","unstructured":"Tamlin Love, Ritesh Ajoodha, and Benjamin Rosman. 2021. Should I trust you? Incorporating unreliable expert advice in human-agent interaction. In Workshop on Human-aligned Reinforcement Learning for Autonomous Agents and Robots."},{"doi-asserted-by":"publisher","key":"e_1_3_2_29_2","DOI":"10.3389\/frobt.2021.584075"},{"doi-asserted-by":"publisher","key":"e_1_3_2_30_2","DOI":"10.1109\/ISIT.2007.4557293"},{"doi-asserted-by":"publisher","key":"e_1_3_2_31_2","DOI":"10.1016\/j.biopsych.2017.12.017"},{"doi-asserted-by":"publisher","key":"e_1_3_2_32_2","DOI":"10.1016\/j.neuron.2016.03.025"},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the 19th Annual ACM\/IEEE International Conference on Human Robot Interaction (HRI)","author":"Scherf Lisa","year":"2024","unstructured":"Lisa Scherf, Lisa Alina Gasche, Eya Chemangui, and Dorothea Koert. 2024. Are you sure?\u2014Multi-modal human decision uncertainty detection in human-robot-interaction. In Proceedings of the 19th Annual ACM\/IEEE International Conference on Human Robot Interaction (HRI)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_34_2","DOI":"10.1109\/Humanoids53995.2022.10000078"},{"doi-asserted-by":"publisher","key":"e_1_3_2_35_2","DOI":"10.1109\/ICMLA.2011.37"},{"doi-asserted-by":"publisher","key":"e_1_3_2_36_2","DOI":"10.1109\/ROMAN.2011.6005223"},{"key":"e_1_3_2_37_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press."},{"key":"e_1_3_2_38_2","first-page":"5724","article-title":"Improving reinforcement learning with human input","volume":"328","author":"Taylor Matthew E.","year":"2018","unstructured":"Matthew E. Taylor and A. I. Borealis. 2018. Improving reinforcement learning with human input. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), Vol. 328, 5724\u20135728.","journal-title":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)"},{"key":"e_1_3_2_39_2","first-page":"1000","article-title":"Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance","volume":"6","author":"Thomaz Andrea Lockerd","year":"2006","unstructured":"Andrea Lockerd Thomaz and Cynthia Breazeal. 2006. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI \u201906), Vol. 6, 1000\u20131005.","journal-title":"Proceedings of the 21st National Conference on Artificial Intelligence (AAAI \u201906)"},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the AAAI 2005 Workshop on Human Comprehensible Machine Learning","author":"Thomaz Andrea Lockerd","year":"2005","unstructured":"Andrea Lockerd Thomaz, Guy Hoffman, and Cynthia Breazeal. 2005. Real-time interactive reinforcement learning for robots. In Proceedings of the AAAI 2005 Workshop on Human Comprehensible Machine Learning."},{"key":"e_1_3_2_41_2","first-page":"41","volume-title":"Proceedings of the Adaptive and Learning Agents Workshop (AAMAS \u201912)","author":"Torrey Lisa","year":"2012","unstructured":"Lisa Torrey and Matthew E. Taylor. 2012. Help an agent out: Student\/teacher learning in sequential decision tasks. In Proceedings of the Adaptive and Learning Agents Workshop (AAMAS \u201912), 41\u201348."},{"doi-asserted-by":"publisher","key":"e_1_3_2_42_2","DOI":"10.1109\/LRA.2022.3182100"},{"doi-asserted-by":"publisher","key":"e_1_3_2_43_2","DOI":"10.1038\/ncomms14637"},{"doi-asserted-by":"publisher","key":"e_1_3_2_44_2","DOI":"10.1007\/BF00992698"},{"doi-asserted-by":"publisher","key":"e_1_3_2_45_2","DOI":"10.1109\/WI-IAT.2015.237"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3736424","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T19:26:38Z","timestamp":1753212398000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3736424"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":44,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3736424"],"URL":"https:\/\/doi.org\/10.1145\/3736424","relation":{},"ISSN":["2573-9522"],"issn-type":[{"type":"electronic","value":"2573-9522"}],"subject":[],"published":{"date-parts":[[2025,7,22]]},"assertion":[{"value":"2024-07-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}