{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T16:26:48Z","timestamp":1775665608680,"version":"3.50.1"},"reference-count":71,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["TRR169"],"award-info":[{"award-number":["TRR169"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["LeCAREbot"],"award-info":[{"award-number":["LeCAREbot"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["IDEAS"],"award-info":[{"award-number":["IDEAS"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"name":"BMWK","award":["VeriKAS"],"award-info":[{"award-number":["VeriKAS"]}]},{"DOI":"10.13039\/501100005711","name":"Universit\u00e4t Hamburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Explainable AI has become an important field of research on neural machine learning models. However, most existing methods are designed as tools that provide expert users with additional insights into their models. In contrast, in human-robot interaction scenarios, non-expert users are frequently confronted with complex, embodied AI systems whose inner workings are unknown. Therefore, eXplainable Human-Robot Interaction (XHRI) should leverage the user\u2019s intuitive ability to collaborate and to use efficient communication. Using NICO, the Neuro-Inspired COmpanion, as a use-case study, we propose an XHRI framework and show how different types of explanations enhance the interaction experience. These explanations range from (a) non-verbal cues for simple and intuitive feedback of inner states via (b) comprehensive verbal explanations of the robot\u2019s intentions, knowledge and reasoning to (c) multimodal explanations using visualizations, speech and text. We revisit past HRI-related studies conducted with NICO and analyze them with the proposed framework. Furthermore, we present two novel XHRI approaches to extract suitable verbal and multimodal explanations from neural network modules in an HRI scenario.<\/jats:p>","DOI":"10.1007\/s13218-022-00772-8","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T14:05:12Z","timestamp":1660745112000},"page":"237-254","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["What\u2019s on Your Mind, NICO?"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1378-0435","authenticated-orcid":false,"given":"Matthias","family":"Kerzel","sequence":"first","affiliation":[]},{"given":"Jakob","family":"Ambsdorf","sequence":"additional","affiliation":[]},{"given":"Dennis","family":"Becker","sequence":"additional","affiliation":[]},{"given":"Wenhao","family":"Lu","sequence":"additional","affiliation":[]},{"given":"Erik","family":"Strahl","sequence":"additional","affiliation":[]},{"given":"Josua","family":"Spisak","sequence":"additional","affiliation":[]},{"given":"Connor","family":"G\u00e4de","sequence":"additional","affiliation":[]},{"given":"Tom","family":"Weber","sequence":"additional","affiliation":[]},{"given":"Stefan","family":"Wermter","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"772_CR1","doi-asserted-by":"crossref","unstructured":"Ambsdorf J, Munir A, Wei Y, Degkwitz K, Harms HM, Stannek S, Ahrens K, Becker D, Strahl E, Weber T, et\u00a0al (2022) Explain yourself! effects of explanations in human\u2013robot interaction. arXiv preprint arXiv:2204.04501","DOI":"10.1109\/RO-MAN53752.2022.9900558"},{"key":"772_CR2","unstructured":"Anjomshoae S, Najjar A, Calvaresi D, Fr\u00e4mling K (2019) Explainable agents and robots: Results from a systematic literature review. In: 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13\u201317, 2019, pp 1078\u20131088. International Foundation for Autonomous Agents and Multiagent Systems"},{"key":"772_CR3","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garc\u00eda S, Gil-L\u00f3pez S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Inf Fusion 58:82\u2013115","journal-title":"Inf Fusion"},{"issue":"2","key":"772_CR4","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/s12369-019-00588-x","volume":"12","author":"J Banks","year":"2020","unstructured":"Banks J (2020) Theory of mind in social robots: replication of five established human tests. Int J Soc Robot 12(2):403\u2013414","journal-title":"Int J Soc Robot"},{"issue":"1","key":"772_CR5","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/s12369-008-0001-3","volume":"1","author":"C Bartneck","year":"2009","unstructured":"Bartneck C, Kuli\u0107 D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71\u201381","journal-title":"Int J Soc Robot"},{"key":"772_CR6","doi-asserted-by":"crossref","unstructured":"Beik-Mohammadi H, Xirakia N, Abawi F, Barykina I, Chandran K, Nair G, Nguyen C, Speck, D, Alpay T, Griffiths S, Heinrich S, Strahl E, Weber C, Wermter S (2019) Designing a personality-driven robot for a human-robot interaction scenario. In: 2019 IEEE International Conference on Robotics and Automation (ICRA), pp. 4317\u20134324. Montreal, Canada. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2019\/BXABCNNSAGHSWW19\/","DOI":"10.1109\/ICRA.2019.8793770"},{"issue":"3","key":"772_CR7","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1109\/MRA.2015.2448951","volume":"22","author":"B Calli","year":"2015","unstructured":"Calli B, Walsman A, Singh A, Srinivasa S, Abbeel P, Dollar AM (2015) Benchmarking in manipulation research: Using the Yale-CMU-Berkeley object and model set. IEEE Robot Autom Magz 22(3):36\u201352","journal-title":"IEEE Robot Autom Magz"},{"key":"772_CR8","doi-asserted-by":"publisher","unstructured":"Churamani N, Anton P, Br\u00fcgger M, Flie\u00dfwasser E, Hummel T, Mayer J, Mustafa W, Ng, HG, Nguyen TLC, Nguyen Q, Soll M, Springenberg S, Griffiths S, Heinrich S, Navarro-Guerrero N, Strahl E, Twiefel J, Weber C, Wermter S (2017) The impact of personalisation on human-robot interaction in learning scenarios. In: Proceedings of the Fifth International Conference on Human Agent Interaction, HAI \u201917, pp 171\u2013180. ACM, ACM, Bielefeld, Germany. https:\/\/doi.org\/10.1145\/3125739.3125756. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2017\/CABFHMMNNNSSGHNSTWW17\/","DOI":"10.1145\/3125739.3125756"},{"key":"772_CR9","doi-asserted-by":"publisher","unstructured":"Churamani N, Kerzel M, Strahl E, Barros P, Wermter S (2017) Teaching emotion expressions to a human companion robot using deep neural architectures. In: International Joint Conference on Neural Networks (IJCNN), pp 627\u2013634. IEEE, Anchorage, Alaska. https:\/\/doi.org\/10.1109\/IJCNN.2017.7965911. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2017\/CKSBW17\/","DOI":"10.1109\/IJCNN.2017.7965911"},{"key":"772_CR10","doi-asserted-by":"crossref","unstructured":"Dautenhahn K, Woods S, Kaouri C, Walters ML, Koay KL, Werry I (2005) What is a robot companion-friend, assistant or butler? In: 2005 IEEE\/RSJ international conference on intelligent robots and systems, pp 1192\u20131197. IEEE","DOI":"10.1109\/IROS.2005.1545189"},{"key":"772_CR11","doi-asserted-by":"crossref","unstructured":"Devin S, Alami R (2016) An implemented theory of mind to improve human\u2013robot shared plans execution. In: 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), pp 319\u2013326. IEEE","DOI":"10.1109\/HRI.2016.7451768"},{"key":"772_CR12","unstructured":"Dogan FI, Melsi\u00f3n GI, Leite I (2021) Leveraging explainability for comprehending referring expressions in the real world. arXiv:2107.05593"},{"key":"772_CR13","doi-asserted-by":"crossref","unstructured":"Eppe M, Kerzel M, Griffiths S, Ng HG, Wermter S (2017) Combining deep learning for visuomotor coordination with object identification to realize a high-level interface for robot object-picking. In: IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp 612\u2013617. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2017\/EKGNW17\/","DOI":"10.1109\/HUMANOIDS.2017.8246935"},{"key":"772_CR14","doi-asserted-by":"publisher","DOI":"10.3386\/w19614","volume-title":"Why ask why? Forward causal inference and reverse causal questions","author":"A Gelman","year":"2013","unstructured":"Gelman A, Imbens G (2013) Why ask why? Forward causal inference and reverse causal questions. Tech. rep, National Bureau of Economic Research"},{"key":"772_CR15","doi-asserted-by":"crossref","unstructured":"Goebel R, Chander A, Holzinger K, Lecue F, Akata Z, Stumpf S, Kieseberg P, Holzinger A (2018) Explainable ai: the new 42? In: International cross-domain conference for machine learning and knowledge extraction, pp 295\u2013303. Springer","DOI":"10.1007\/978-3-319-99740-7_21"},{"key":"772_CR16","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1093\/acprof:osobl\/9780199927418.003.0004","volume":"2","author":"HM Gray","year":"2014","unstructured":"Gray HM, Gray K, Wegner DM (2014) Dimensions of mind perception. Exp Philos 2:88. https:\/\/doi.org\/10.1093\/acprof:osobl\/9780199927418.003.0004","journal-title":"Exp Philos"},{"key":"772_CR17","unstructured":"Greydanus S, Koul A, Dodge J, Fern A (2018) Visualizing and understanding atari agents. In: International conference on machine learning, pp 1792\u20131801. PMLR"},{"key":"772_CR18","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"772_CR19","doi-asserted-by":"crossref","unstructured":"Hegel F, Krach S, Kircher T, Wrede B, Sagerer G (2008) Theory of mind (tom) on robots: a functional neuroimaging study. In: 2008 3rd ACM\/IEEE International Conference on Human-Robot Interaction (HRI), pp 335\u2013342. IEEE","DOI":"10.1145\/1349822.1349866"},{"issue":"2","key":"772_CR20","doi-asserted-by":"publisher","first-page":"243","DOI":"10.2307\/1416950","volume":"57","author":"F Heider","year":"1944","unstructured":"Heider F, Simmel M (1944) An experimental study of apparent behavior. Am J Psychol 57(2):243\u2013259","journal-title":"Am J Psychol"},{"issue":"1","key":"772_CR21","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1515\/pjbr-2018-0009","volume":"9","author":"T Hellstr\u00f6m","year":"2018","unstructured":"Hellstr\u00f6m T, Bensch S (2018) Understandable robots-what, why, and how. Paladyn J Behav Robot 9(1):110\u2013123","journal-title":"Paladyn J Behav Robot"},{"key":"772_CR22","doi-asserted-by":"crossref","unstructured":"Iyer R, Li Y, Li H, Lewis M, Sundar R, Sycara K (2018) Transparency and explanation in deep reinforcement learning neural networks. In: Proceedings of the 2018 AAAI\/ACM conference on AI, ethics, and society, pp 144\u2013150","DOI":"10.1145\/3278721.3278776"},{"key":"772_CR23","unstructured":"James S, Davison AJ, Johns E (2017) Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In: Conference on robot learning, pp 334\u2013343. PMLR"},{"key":"772_CR24","doi-asserted-by":"crossref","unstructured":"Jung B, Kopp S (2003) Flurmax: an interactive virtual agent for entertaining visitors in a hallway. In: International workshop on intelligent virtual agents, pp 23\u201326, Springer","DOI":"10.1007\/978-3-540-39396-2_5"},{"key":"772_CR25","unstructured":"Juozapaitis Z, Koul A, Fern A, Erwig M, Doshi-Velez F (2019) Explainable reinforcement learning via reward decomposition. In: IJCAI\/ECAI Workshop on explainable artificial intelligence"},{"key":"772_CR26","unstructured":"Kansky K, Silver T, M\u00e9ly DA, Eldawy M, L\u00e1zaro-Gredilla M, Lou X, Dorfman N, Sidor S, Phoenix S, George D (2017) Schema networks: zero-shot transfer with a generative causal model of intuitive physics. In: International conference on machine learning, pp 1809\u20131818. PMLR"},{"key":"772_CR27","doi-asserted-by":"publisher","unstructured":"Kerzel M, Abawi F, Eppe M, Wermter S (2020) Enhancing a neurocognitive shared visuomotor model for object identification, localization, and grasping with learning from auxiliary tasks. IEEE transactions on cognitive and developmental systems pp 1\u201313. https:\/\/doi.org\/10.1109\/TCDS.2020.3028460. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2020\/KAEW20\/09211758.pdf","DOI":"10.1109\/TCDS.2020.3028460"},{"key":"772_CR28","unstructured":"Kerzel M, Ng HG, Griffiths S, Wermter S (2017) Effect of a humanoid\u2019s active role during learning with embodied dialogue system. In: V.N.T.T.S.W.A.T. Amir Aly Sascha Griffiths (ed) Proceedings of the Workshop: Towards Intelligent Social Robots: Social Cognitive Systems in Smart Environments at the 26th IEEE International Symposium on Robot and Human Interactive Communication. RO-MAN, Lisbon, Portugal. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2017\/KNGW17\/"},{"key":"772_CR29","doi-asserted-by":"publisher","unstructured":"Kerzel M, Pekarek-Rosin T, Strahl E, Heinrich S, Wermter S (2020) Teaching nico how to grasp: An empirical study on crossmodal social interaction as a key factor for robots learning from humans. Front Neurorobot 12. https:\/\/doi.org\/10.3389\/fnbot.2020.00028. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2020\/KPSHW20\/KPSHW20.pdf","DOI":"10.3389\/fnbot.2020.00028"},{"key":"772_CR30","doi-asserted-by":"publisher","unstructured":"Kerzel M, Spisak J, Strahl E, Wermter S (2020) Neuro-genetic visuomotor architecture for robotic grasping. In: M.P.W.S. Farka\u0161 Igor (ed.) Artificial Neural Networks and Machine Learning - ICANN 2020, LNCS, pp. 533\u2013545. Springer. https:\/\/doi.org\/10.1007\/978-3-030-61616-8_43. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2020\/KSSW20\/ICANN_2020__Neuro_Genetic_Visuomotor_Framework_Preprint.pdf","DOI":"10.1007\/978-3-030-61616-8_43"},{"key":"772_CR31","doi-asserted-by":"crossref","unstructured":"Kerzel M, Strahl E, Magg S, Navarro-Guerrero, N, Heinrich S, Wermter S (2017) Nico - neuro-inspired companion: A developmental humanoid robot platform for multimodal interaction. In: Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 113\u2013120. https:\/\/www2.informatik.uni-hamburg.de\/wtm\/publications\/2017\/KSMNHW17\/","DOI":"10.1109\/ROMAN.2017.8172289"},{"key":"772_CR32","doi-asserted-by":"publisher","unstructured":"Kerzel M, Wermter S (2017) Neural end-to-end self-learning of visuomotor skills by environment interaction. In: V.P.V.A. Lintas\u00a0A. Rovetta\u00a0S. (ed.) Artificial Neural Networks and Machine Learning - ICANN 2017, Lecture Notes in Computer Science, vol 10613, pp 27\u201334. Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-319-68600-4_4","DOI":"10.1007\/978-3-319-68600-4_4"},{"key":"772_CR33","unstructured":"Kim B, Wattenberg M, Gilmer J, Cai C, Wexler J, Viegas F, et\u00a0al (2018) Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: International conference on machine learning, pp. 2668\u20132677. PMLR"},{"key":"772_CR34","doi-asserted-by":"crossref","unstructured":"Kopp S, Gesellensetter L, Kr\u00e4mer NC, Wachsmuth I (2005) A conversational agent as museum guide\u2013design and evaluation of a real-world application. In: International workshop on intelligent virtual agents, pp 329\u2013343. Springer","DOI":"10.1007\/11550617_28"},{"key":"772_CR35","doi-asserted-by":"crossref","unstructured":"Kr\u00e4mer NC, P\u00fctten Avd, Eimler S (2012) Human-agent and human-robot interaction theory: similarities to and differences from human\u2013human interaction. In: Human\u2013computer interaction: The agency perspective, pp 215\u2013240. Springer","DOI":"10.1007\/978-3-642-25691-2_9"},{"issue":"260","key":"772_CR36","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","volume":"47","author":"WH Kruskal","year":"1952","unstructured":"Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583\u2013621","journal-title":"J Am Stat Assoc"},{"issue":"7553","key":"772_CR37","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444","journal-title":"Nature"},{"key":"772_CR38","unstructured":"Lee Sl, Lau IYm, Kiesler S, Chiu CY (2005) Human mental models of humanoid robots. In: Proceedings of the 2005 IEEE international conference on robotics and automation, pp 2767\u20132772, IEEE"},{"key":"772_CR39","doi-asserted-by":"publisher","unstructured":"Lescevica M, Ginters E, Mazza R (2013) Unified theory of acceptance and use of technology (UTAUT) for market analysis of FP7 CHOReOS products. Procedia Comput Sci 26(December):51\u201368. https:\/\/doi.org\/10.1016\/j.procs.2013.12.007","DOI":"10.1016\/j.procs.2013.12.007"},{"key":"772_CR40","doi-asserted-by":"crossref","unstructured":"Lichtenth\u00e4ler C, Lorenzy T, Kirsch A (2012) Influence of legibility on perceived safety in a virtual human\u2013robot path crossing task. In: 2012 IEEE RO-MAN: The 21st IEEE international symposium on robot and human interactive communication, pp 676\u2013681, IEEE","DOI":"10.1109\/ROMAN.2012.6343829"},{"key":"772_CR41","doi-asserted-by":"crossref","unstructured":"Lin TY, Goyal P, Girshick R, He K, Doll\u00e1r P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980\u20132988","DOI":"10.1109\/ICCV.2017.324"},{"key":"772_CR42","doi-asserted-by":"crossref","unstructured":"Madumal P, Miller T, Sonenberg L, Vetere F (2020) Explainable reinforcement learning through a causal lens. In: Proceedings of the AAAI conference on artificial intelligence, vol\u00a034, pp 2493\u20132500","DOI":"10.1609\/aaai.v34i03.5631"},{"key":"772_CR43","unstructured":"Malle BF (2006) How the mind explains behavior: Folk explanations, meaning, and social interaction. MIT Press, New York"},{"key":"772_CR44","doi-asserted-by":"publisher","unstructured":"Meertens RM, Lion R (2008) Measuring an individual\u2019s tendency to take risks: the risk propensity scale. J Appl Soc Psychol 38(6):1506\u20131520. https:\/\/doi.org\/10.1111\/j.1559-1816.2008.00357.x","DOI":"10.1111\/j.1559-1816.2008.00357.x"},{"key":"772_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"key":"772_CR46","unstructured":"Miller T, Howe P, Sonenberg L (2017) Explainable ai: beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv:1712.00547"},{"key":"772_CR47","doi-asserted-by":"crossref","unstructured":"Mittelstadt B, Russell C, Wachter S (2019) Explaining explanations in ai. In: Proceedings of the conference on fairness, accountability, and transparency, pp 279\u2013288","DOI":"10.1145\/3287560.3287574"},{"key":"772_CR48","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv:1312.5602"},{"key":"772_CR49","unstructured":"Nakata T, Sato T, Mori T, Mizoguchi H (1998) Expression of emotion and intention by robot body movement. In: International conference on intelligent autonomous systems 5 (IAS-5), pp 352\u2013359"},{"key":"772_CR50","doi-asserted-by":"publisher","unstructured":"Ng HG, Anton P, Br\u00fcgger M, Churamani N, Flie\u00dfwasser E, Hummel T, Mayer J, Mustafa W, Nguyen TLC, Nguyen Q, Soll M, Springenberg S, Griffiths S, Heinrich S, Navarro-Guerrero N, Strahl E, Twiefel J, Weber C, Wermter S (2017) Hey robot, why don\u2019t you talk to me? In: Proceedings of the IEEE international symposium on robot and human interactive communication (RO-MAN), pp 728\u2013731. https:\/\/doi.org\/10.1109\/ROMAN.2017.8172383","DOI":"10.1109\/ROMAN.2017.8172383"},{"key":"772_CR51","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","volume":"113","author":"GI Parisi","year":"2019","unstructured":"Parisi GI, Kemker R, Part JL, Kanan C, Wermter S (2019) Continual lifelong learning with neural networks: a review. Neural Netw 113:54\u201371","journal-title":"Neural Netw"},{"key":"772_CR52","unstructured":"Puri N, Verma S, Gupta P, Kayastha D, Deshmukh S, Krishnamurthy B, Singh S (2019) Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv:1912.12191"},{"key":"772_CR53","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201cwhy should I trust you\u201d: explaining the predictions of any classifier. arxiv:1602.04938","DOI":"10.1145\/2939672.2939778"},{"key":"772_CR54","unstructured":"Rist T, Baldes S, Gebhard P, Kipp M, Klesen M, Rist P, Schmitt M (2002) Crosstalk: An interactive installation with animated presentation agents. In: Proceedings of COSIGN, vol.\u00a02. Citeseer"},{"key":"772_CR55","unstructured":"Sado F, Loo CK, Kerzel M, Wermter S (2020) Explainable goal-driven agents and robots\u2013a comprehensive review and new framework. arXiv:2004.09705"},{"issue":"1","key":"772_CR56","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1023\/A:1013298507114","volume":"12","author":"B Scassellati","year":"2002","unstructured":"Scassellati B (2002) Theory of mind for a humanoid robot. Auton Robot 12(1):13\u201324","journal-title":"Auton Robot"},{"issue":"7839","key":"772_CR57","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1038\/s41586-020-03051-4","volume":"588","author":"J Schrittwieser","year":"2020","unstructured":"Schrittwieser J, Antonoglou I, Hubert T, Simonyan K, Sifre L, Schmitt S, Guez A, Lockhart E, Hassabis D, Graepel T et al (2020) Mastering atari, go, chess and shogi by planning with a learned model. Nature 588(7839):604\u2013609","journal-title":"Nature"},{"key":"772_CR58","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Das A, Vedantam R, Cogswell M, Parikh D, Batra D (2016) Grad-cam: why did you say that? Visual explanations from deep networks via gradient-based localization. arXiv:1610.02391","DOI":"10.1109\/ICCV.2017.74"},{"key":"772_CR59","unstructured":"Sheh RK (2017) Different xai for different hri. In: 2017 AAAI Fall Symposium Series"},{"issue":"4","key":"772_CR60","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1177\/0018720816644364","volume":"58","author":"TB Sheridan","year":"2016","unstructured":"Sheridan TB (2016) Human-robot interaction: status and challenges. Hum Fact 58(4):525\u2013532","journal-title":"Hum Fact"},{"key":"772_CR61","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv:1312.6034"},{"key":"772_CR62","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: Visualising image classification models and saliency maps. In: In Workshop at International Conference on Learning Representations. Citeseer"},{"key":"772_CR63","doi-asserted-by":"crossref","unstructured":"Takayama L, Dooley D, Ju W (2011) Expressing thought: improving robot readability with animation principles. In: Proceedings of the 6th international conference on Human\u2013robot interaction, pp 69\u201376","DOI":"10.1145\/1957656.1957674"},{"key":"772_CR64","doi-asserted-by":"crossref","unstructured":"Van\u00a0Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, vol.\u00a030","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"772_CR65","doi-asserted-by":"crossref","unstructured":"Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Computational intelligence and neuroscience 2018","DOI":"10.1155\/2018\/7068349"},{"issue":"4","key":"772_CR66","doi-asserted-by":"publisher","first-page":"767","DOI":"10.1037\/0022-3514.70.4.767","volume":"70","author":"GC Williams","year":"1996","unstructured":"Williams GC, Deci EL (1996) Internalization of biopsychosocial values by medical students: a test of self-determination theory. J Pers Soc Psychol 70(4):767","journal-title":"J Pers Soc Psychol"},{"issue":"3","key":"772_CR67","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1080\/09540091.2017.1313816","volume":"29","author":"RH Wortham","year":"2017","unstructured":"Wortham RH, Theodorou A (2017) Robot transparency, trust and utility. Connect Sci 29(3):242\u2013248","journal-title":"Connect Sci"},{"key":"772_CR68","doi-asserted-by":"crossref","unstructured":"Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing. IEEE Comput Intell Magz 13(3):55\u201375","DOI":"10.1109\/MCI.2018.2840738"},{"key":"772_CR69","unstructured":"Zahavy T, Ben-Zrihem N, Mannor S (2016) Graying the black box: understanding dqns. In: International conference on machine learning, pp 1899\u20131908"},{"key":"772_CR70","doi-asserted-by":"crossref","unstructured":"Zhou B, Khosla A, Lapedriza \u00c0, Oliva A, Torralba A (2015) Learning deep features for discriminative localization. arXiv:1512.04150","DOI":"10.1109\/CVPR.2016.319"},{"issue":"June","key":"772_CR71","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/frobt.2021.644529","volume":"8","author":"S Z\u00f6rner","year":"2021","unstructured":"Z\u00f6rner S, Arts E, Vasiljevic B, Srivastava A, Schmalzl F, Mir G, Bhatia K, Strahl E, Peters A, Alpay T, Wermter S (2021) An immersive investment game to study human-robot trust. Front Robot AI 8(June):1\u201316. https:\/\/doi.org\/10.3389\/frobt.2021.644529","journal-title":"Front Robot AI"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00772-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-022-00772-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00772-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T17:58:54Z","timestamp":1672163934000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-022-00772-8"}},"subtitle":["XHRI: A Framework for eXplainable Human-Robot Interaction"],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":71,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["772"],"URL":"https:\/\/doi.org\/10.1007\/s13218-022-00772-8","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]},"assertion":[{"value":"11 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}