{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T17:06:56Z","timestamp":1770916016627,"version":"3.50.1"},"reference-count":50,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,7,17]],"date-time":"2025-07-17T00:00:00Z","timestamp":1752710400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>With the continuous advancement of Artificial intelligence (AI), robots as embodied intelligent systems are increasingly becoming more present in daily life like households or in elderly care. As a result, lay users are required to interact with these systems more frequently and teach them to meet individual needs. Human-in-the-loop reinforcement learning (HIL-RL) offers an effective way to realize this teaching. Studies show that various feedback modalities, such as preference, guidance, or demonstration can significantly enhance learning success, though their suitability varies among users expertise in robotics. Research also indicates that users apply different scaffolding strategies when teaching a robot, such as motivating it to explore actions that promise success. Thus, providing a collection of different feedback modalities allows users to choose the method that best suits their teaching strategy, and allows the system to individually support the user based on their interaction behavior. However, most state-of-the-art approaches provide users with only one feedback modality at a time. Investigating combined feedback modalities in interactive robot learning remains an open challenge. To address this, we conducted a study that combined common feedback modalities. Our research questions focused on whether these combinations improve learning outcomes, reveal user preferences, show differences in perceived effectiveness, and identify which modalities influence learning the most. The results show that combining the feedback modalities improves learning, with users perceiving the effectiveness of the modalities vary ways, and certain modalities directly impacting learning success. The study demonstrates that combining feedback modalities can support learning even in a simplified setting and suggests the potential for broader applicability, especially in robot learning scenarios with a focus on user interaction. Thus, this paper aims to motivate the use of combined feedback modalities in interactive imitation learning.<\/jats:p>","DOI":"10.3389\/frobt.2025.1598968","type":"journal-article","created":{"date-parts":[[2025,7,17]],"date-time":"2025-07-17T15:26:43Z","timestamp":1752766003000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["The power of combined modalities in interactive robot learning"],"prefix":"10.3389","volume":"12","author":[{"given":"Helen","family":"Beierling","sequence":"first","affiliation":[]},{"given":"Robin","family":"Beierling","sequence":"additional","affiliation":[]},{"given":"Anna-Lisa","family":"Vollmer","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2025,7,17]]},"reference":[{"key":"B1","unstructured":"Movement primitives\n          \n          \n            \n              Alexander Fabisch\n              J. K.\n            \n          \n          \n          2020"},{"key":"B2","doi-asserted-by":"publisher","first-page":"105954","DOI":"10.1016\/j.chb.2019.03.018","article-title":"Developing young children\u2019s computational thinking with educational robotics: an interaction effect between gender and scaffolding strategy","volume":"105","author":"Angeli","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"B3","doi-asserted-by":"publisher","first-page":"11748","DOI":"10.48550\/arXiv.1810.11748","article-title":"Dqn-tamer: human-in-the-loop reinforcement learning with intractable feedback","author":"Arakawa","year":"2018","journal-title":"arXiv Prepr. arXiv:1810"},{"key":"B4","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/j.robot.2010.11.004","article-title":"Teacher feedback to scaffold and refine demonstrated motion primitives on a mobile robot","volume":"59","author":"Argall","year":"2011","journal-title":"Robotics Aut. Syst."},{"key":"B5","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1145\/3357236.3395525","article-title":"A survey on interactive reinforcement learning: design principles and open challenges","volume-title":"Proceedings of the 2020 ACM designing interactive syst","author":"Arzate Cruz","year":"2020"},{"key":"B6","first-page":"141","article-title":"Learning from physical human corrections, one feature at a time","volume-title":"Proc. Of 2018 ACM\/IEEE int. Conf. On human-robot interaction","author":"Bajcsy","year":"2018"},{"key":"B7","doi-asserted-by":"publisher","first-page":"18215","DOI":"10.1007\/s00521-021-06850-6","article-title":"Human engagement providing evaluative and informative advice for interactive reinforcement learning","volume":"35","author":"Bignold","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"B8","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1177\/02783649211041652","article-title":"Learning reward functions from diverse sources of human feedback: optimally integrating demonstrations and preferences","volume":"41","author":"B\u0131y\u0131k","year":"2022","journal-title":"Int. J. Robotics Res."},{"key":"B9","first-page":"1","volume-title":"Learning by scaffolding.","author":"Breazeal","year":"1998"},{"key":"B10","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1016\/j.robot.2006.02.004","article-title":"Using perspective taking to learn from ambiguous demonstrations","volume":"54","author":"Breazeal","year":"2006","journal-title":"Robotics Aut. Syst."},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1201\/9781498710411-35","article-title":"Sus: a quick and dirty usability scale","author":"Brooke","year":"1995","journal-title":"Usability Eval. Ind."},{"key":"B12","first-page":"783","article-title":"Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations","volume-title":"International conference on machine learning","author":"Brown","year":"2019"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.15217","article-title":"Open problems and fundamental limitations of reinforcement learning from human feedback","author":"Casper","year":"2023","journal-title":"arXiv Prepr. arXiv:2307.15217"},{"key":"B14","doi-asserted-by":"publisher","first-page":"16821","DOI":"10.1007\/s00521-022-08118-z","article-title":"Knowledge-and ambiguity-aware robot learning from corrective and evaluative feedback","volume":"35","author":"Celemin","year":"2023","journal-title":"Neural comput. Appl."},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2300000072","article-title":"Interactive imitation learning in robotics: a survey","volume":"10","author":"Celemin","year":"2022","journal-title":"Found. Trends Robotics"},{"key":"B16","doi-asserted-by":"publisher","first-page":"101007","DOI":"10.1115\/1.4054297","article-title":"Real-time multi-modal human\u2013robot collaboration using gestures and speech","volume":"144","author":"Chen","year":"2022","journal-title":"J. Manuf. Sci. Eng."},{"key":"B17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.2584","article-title":"Interactive policy learning through confidence-based autonomy","volume":"34","author":"Chernova","year":"2009","journal-title":"J. Artif. Intell. Res."},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.04602","article-title":"Learning a universal human prior for dexterous manipulation from human preference","author":"Ding","year":"2023","journal-title":"arXiv Prepr. arXiv:2304.04602"},{"key":"B20","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1038\/s41586-020-03157-9","article-title":"First return, then explore","volume":"590","author":"Ecoffet","year":"2021","journal-title":"Nature"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2022.3186270","article-title":"Interactive robot task learning: human teaching proficiency with different feedback approaches","author":"Hindemith","year":"2022","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"B18","unstructured":"Kinova assistiv\n          \n          \n          2023"},{"key":"B22","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1145\/375735.376334","article-title":"A social reinforcement learning agent","volume-title":"Proceedings of the fifth international conference on Autonomous agents","author":"Isbell","year":"2001"},{"key":"B23","first-page":"4415","article-title":"Reward-rational (implicit) choice: a unifying formalism for reward learning","volume":"33","author":"Jeon","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B24","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1109\/LaTiCE.2014.22","article-title":"Instructional scaffolding in online learning environment: a meta-analysis","volume-title":"2014 international conference on teaching and learning in computing and engineering","author":"Jumaat","year":"2014"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.14925","article-title":"A survey of reinforcement learning from human feedback","author":"Kaufmann","year":"2023","journal-title":"arXiv Prepr. arXiv:2312.14925"},{"key":"B26","doi-asserted-by":"publisher","first-page":"103829","DOI":"10.1016\/j.artint.2022.103829","article-title":"Reward (mis) design for autonomous driving","volume":"316","author":"Knox","year":"2023","journal-title":"Artif. Intell."},{"key":"B27","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1109\/DEVLRN.2008.4640845","article-title":"Tamer: training an agent manually via evaluative reinforcement","volume-title":"2008 7th IEEE international conference on development and learning","author":"Knox","year":"2008"},{"key":"B28","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1109\/thms.2019.2912447","article-title":"Human-centered reinforcement learning: a survey","volume":"49","author":"Li","year":"2019","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"B29","doi-asserted-by":"crossref","first-page":"2877","DOI":"10.1109\/ICRA48506.2021.9560829","article-title":"Learning human objectives from sequences of physical corrections","volume-title":"2021 IEEE international conference on robotics and automation (ICRA)","author":"Li","year":"2021"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.17555","article-title":"Interactive robot learning from verbal correction","author":"Liu","year":"2023","journal-title":"arXiv Prepr. arXiv:2310.17555"},{"key":"B31","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1109\/ICHR.2010.5686326","article-title":"Complementary humanoid behavior shaping using corrective demonstration","volume-title":"2010 10th IEEE-RAS international Conference on humanoid robots","author":"Meri\u00e7li","year":"2010"},{"key":"B32","doi-asserted-by":"publisher","first-page":"16","DOI":"10.5772\/10575","article-title":"Task refinement for autonomous robots using complementary corrective human feedback","volume":"8","author":"Meri\u00e7li","year":"2011","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B33","article-title":"Probabilistic movement primitives","author":"Paraschos","year":"2013","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B34","first-page":"1634","article-title":"Smooth exploration for robotic reinforcement learning","volume-title":"Conference on robot learning","author":"Raffin","year":"2022"},{"key":"B35","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1146\/annurev-control-100819-063206","article-title":"Recent advances in robot learning from demonstration","volume":"3","author":"Ravichandar","year":"2020","journal-title":"Annu. Rev. Control Rob. Auton. Syst."},{"key":"B36","unstructured":"Ros noetic ninjemys\n          \n          \n            \n              Robotics\n              O.\n            \n          \n          \n          2020"},{"key":"B37","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1145\/1121241.1121263","article-title":"Teaching robots by moulding behavior and scaffolding the environment","volume-title":"Proceedings of the 1st ACM SIGCHI\/SIGART conference on Human-robot interaction","author":"Saunders","year":"2006"},{"key":"B38","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1177\/002221949803100404","article-title":"The metaphor of scaffolding: its utility for the field of learning disabilities","volume":"31","author":"Stone","year":"","journal-title":"J. Learn. Disabil."},{"key":"B39","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1177\/002221949803100411","article-title":"Should we salvage the scaffolding metaphor?","volume":"31","author":"Stone","year":"","journal-title":"J. Learn. Disabil."},{"key":"B40","article-title":"Policy improvement methods: between black-box optimization and episodic reinforcement learning","author":"Stulp","year":"2012"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1084000","DOI":"10.3389\/fnbot.2023.1084000","article-title":"Recent advancements in multimodal human\u2013robot interaction","volume":"17","author":"Su","year":"2023","journal-title":"Front. Neurorob."},{"key":"B42","first-page":"1000","article-title":"Reinforcement learning with human teachers: evidence of feedback and guidance with implications for learning performance","volume-title":"Proceedings of the 21st national conference on artificial intelligence","author":"Thomaz","year":"2006"},{"key":"B43","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/devlrn.2007.4354078","article-title":"Robot learning via socially guided exploration","author":"Thomaz","year":"2007","journal-title":"Dev. Learn."},{"key":"B44","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1145\/1514095.1514101","article-title":"Learning about objects with human teachers","volume-title":"Proceedings of the 4th ACM\/IEEE international conference on Human robot interaction","author":"Thomaz","year":"2009"},{"key":"B45","article-title":"Optimizing robot behavior via comparative language feedback","author":"Tien","year":"2024"},{"key":"B46","first-page":"63222","article-title":"Breadcrumbs to the goal: goal-conditioned exploration from human-in-the-loop feedback","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"Torne","year":"2023"},{"key":"B47","doi-asserted-by":"publisher","first-page":"77","DOI":"10.3389\/frobt.2018.00077","article-title":"A user study on robot skill learning without a cost function: optimization of dynamic movement primitives via naive user feedback","volume":"5","author":"Vollmer","year":"2018","journal-title":"Front. Rob. AI"},{"key":"B48","doi-asserted-by":"publisher","first-page":"10","DOI":"10.3389\/fnbot.2016.00010","article-title":"Pragmatic frames for teaching and learning in human\u2013robot interaction: review and challenges","volume":"10","author":"Vollmer","year":"2016","journal-title":"Front. neurorobotics"},{"key":"B49","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1111\/j.1469-7610.1976.tb00381.x","article-title":"The role of tutoring in problem solving","volume":"17","author":"Wood","year":"1976","journal-title":"J. child Psychol. psychiatry"},{"key":"B50","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1007\/978-3-030-35888-4_59","article-title":"Interactive robot learning for multimodal emotion recognition","volume-title":"Social robotics: 11th international conference, ICSR 2019, Madrid, Spain, november 26\u201329, 2019, proceedings 11","author":"Yu","year":"2019"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1598968\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,17]],"date-time":"2025-07-17T15:26:49Z","timestamp":1752766009000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1598968\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,17]]},"references-count":50,"alternative-id":["10.3389\/frobt.2025.1598968"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2025.1598968","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,17]]},"article-number":"1598968"}}