{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T03:31:33Z","timestamp":1776828693281,"version":"3.51.2"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,7,11]],"date-time":"2021-07-11T00:00:00Z","timestamp":1625961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Union's Horizon 2020 research and innovation programme","award":["765955"],"award-info":[{"award-number":["765955"]}]},{"name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","award":["UIDB\/50021\/2020"],"award-info":[{"award-number":["UIDB\/50021\/2020"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2021,9,30]]},"abstract":"<jats:p>The issue of how to make embodied agents explainable has experienced a surge of interest over the past 3 years, and there are many terms that refer to this concept, such as transparency and legibility. One reason for this high variance in terminology is the unique array of social cues that embodied agents can access in contrast to that accessed by non-embodied agents. Another reason is that different authors use these terms in different ways. Hence, we review the existing literature on explainability and organize it by (1) providing an overview of existing definitions, (2) showing how explainability is implemented and how it exploits different social cues, and (3) showing how the impact of explainability is measured. Additionally, we present a list of open questions and challenges that highlight areas that require further investigation by the community. This provides the interested reader with an overview of the current state of the art.<\/jats:p>","DOI":"10.1145\/3457188","type":"journal-article","created":{"date-parts":[[2021,7,12]],"date-time":"2021-07-12T00:21:25Z","timestamp":1626049285000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":71,"title":["Explainable Embodied Agents Through Social Cues"],"prefix":"10.1145","volume":"10","author":[{"given":"Sebastian","family":"Wallk\u00f6tter","sequence":"first","affiliation":[{"name":"Uppsala University and INESC-ID-Instituto Superior T\u00e9cnico, Uppsala, Sweden"}]},{"given":"Silvia","family":"Tulli","sequence":"additional","affiliation":[{"name":"Uppsala University and INESC-ID-Instituto Superior T\u00e9cnico, Lisbon, Portugal"}]},{"given":"Ginevra","family":"Castellano","sequence":"additional","affiliation":[{"name":"Uppsala University, Uppsala, Sweden"}]},{"given":"Ana","family":"Paiva","sequence":"additional","affiliation":[{"name":"INESC-ID and Instituto Superior T\u00e9cnico"}]},{"given":"Mohamed","family":"Chetouani","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Systems and Robotics, CNRS UMR 7222, Sorbonne Universit\u00e9, Paris, France"}]}],"member":"320","published-online":{"date-parts":[[2021,7,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2019.01.028"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2019.01.026"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2018.00083"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS\u201919)","author":"Anjomshoae Sule","year":"2019","unstructured":"Sule Anjomshoae , Amro Najjar , Davide Calvaresi , and Kary Fr\u00e4mling . 2019 . Explainable agents and robots: Results from a systematic literature review . In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS\u201919) . ACM, New York, NY, 1078\u20131088. Sule Anjomshoae, Amro Najjar, Davide Calvaresi, and Kary Fr\u00e4mling. 2019. Explainable agents and robots: Results from a systematic literature review. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS\u201919). ACM, New York, NY, 1078\u20131088."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2942288"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201916)","author":"Baraka Kim","year":"2016","unstructured":"Kim Baraka , Stephanie Rosenthal , and Manuela M. Veloso . 2016. Enhancing human understanding of a mobile robot\u2019s state and actions using expressive lights . In Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201916) . IEEE, Los Alamitos, CA, 652\u2013657. DOI:https:\/\/doi.org\/10.1109\/ROMAN. 2016 .7745187 10.1109\/ROMAN.2016.7745187 Kim Baraka, Stephanie Rosenthal, and Manuela M. Veloso. 2016. Enhancing human understanding of a mobile robot\u2019s state and actions using expressive lights. In Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201916). IEEE, Los Alamitos, CA, 652\u2013657. DOI:https:\/\/doi.org\/10.1109\/ROMAN.2016.7745187"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-017-0431-x"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 10th Annual ACM\/IEEE International Conference on Human-Robot Interaction Extended Abstracts (HRI\u201915 Extended Abstracts)","volume":"180","author":"Boyce Michael W.","year":"1973","unstructured":"Michael W. Boyce , Jessie Y. C. C. Chen , Anthony R. Selkowitz , and Shan G. Lakhmani . 2015. Effects of agent transparency on operator trust . In Proceedings of the 10th Annual ACM\/IEEE International Conference on Human-Robot Interaction Extended Abstracts (HRI\u201915 Extended Abstracts) , Vol. 02-05-Marc. IEEE, Los Alamitos, CA, 179\u2013 180 . DOI:https:\/\/doi.org\/10.1145\/270 1973 .2702059 10.1145\/2701973.2702059 Michael W. Boyce, Jessie Y. C. C. Chen, Anthony R. Selkowitz, and Shan G. Lakhmani. 2015. Effects of agent transparency on operator trust. In Proceedings of the 10th Annual ACM\/IEEE International Conference on Human-Robot Interaction Extended Abstracts (HRI\u201915 Extended Abstracts), Vol. 02-05-Marc. IEEE, Los Alamitos, CA, 179\u2013180. DOI:https:\/\/doi.org\/10.1145\/2701973.2702059"},{"key":"e_1_2_1_11_1","first-page":"2019","article-title":"Towards transparent robot learning through TDRL-based emotional expressions","volume":"16","author":"Broekens Joost","year":"2019","unstructured":"Joost Broekens and Mohamed Chetouani . 2019 . Towards transparent robot learning through TDRL-based emotional expressions . IEEE Transactions on Affective Computing. Early Access , January 16 , 2019 . DOI:https:\/\/doi.org\/10.1109\/taffc.2019.2893348 10.1109\/taffc.2019.2893348 Joost Broekens and Mohamed Chetouani. 2019. Towards transparent robot learning through TDRL-based emotional expressions. IEEE Transactions on Affective Computing. Early Access, January 16, 2019. DOI:https:\/\/doi.org\/10.1109\/taffc.2019.2893348","journal-title":"IEEE Transactions on Affective Computing. Early Access"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025462"},{"key":"e_1_2_1_13_1","unstructured":"Tathagata Chakraborti Anagha Kulkarni Sarath Sreedharan David E. Smith and Subbarao Kambhampati. 2018. Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. arxiv:1811.09722  Tathagata Chakraborti Anagha Kulkarni Sarath Sreedharan David E. Smith and Subbarao Kambhampati. 2018. Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. arxiv:1811.09722"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/185"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2018.8593830"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/23"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 5th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201910)","author":"Chao Crystal","year":"2010","unstructured":"Crystal Chao , Maya Cakmak , and Andrea L. Thomaz . 2010. Transparent active learning for robots . In Proceedings of the 5th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201910) . IEEE, Los Alamitos, CA, 317\u2013324. DOI:https:\/\/doi.org\/10.1109\/HRI. 2010 .5453178 10.1109\/HRI.2010.5453178 Crystal Chao, Maya Cakmak, and Andrea L. Thomaz. 2010. Transparent active learning for robots. In Proceedings of the 5th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201910). IEEE, Los Alamitos, CA, 317\u2013324. DOI:https:\/\/doi.org\/10.1109\/HRI.2010.5453178"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201917)","author":"Chen Jessie Y. C.","year":"2017","unstructured":"Jessie Y. C. Chen , Michael J. Barnes , Anthony R. Selkowitz , and Kimberly Stowers . 2017 . Effects of agent transparency on human-autonomy teaming effectiveness . In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201917) . IEEE, Los Alamitos, CA , 1838\u20131843. DOI:https:\/\/doi.org\/10.1109\/SMC.2016.7844505 10.1109\/SMC.2016.7844505 Jessie Y. C. Chen, Michael J. Barnes, Anthony R. Selkowitz, and Kimberly Stowers. 2017. Effects of agent transparency on human-autonomy teaming effectiveness. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC\u201917). IEEE, Los Alamitos, CA, 1838\u20131843. DOI:https:\/\/doi.org\/10.1109\/SMC.2016.7844505"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1080\/1463922X.2017.1315750"},{"key":"e_1_2_1_20_1","volume-title":"Barnes","author":"Chen Jessie Y. C.","year":"2014","unstructured":"Jessie Y. C. Chen , Katelyn Procci , Michael Boyce , Julia Wright , Andre Garcia , and Michael J . Barnes . 2014 . Situation Awareness\u2013Based Agent Transparency. Technical Report. Army Research Laboratory, Washington, DC. https:\/\/www.researchgate.net\/publication\/264963346_Situation_Awareness-Based_Agent_Transparency. Jessie Y. C. Chen, Katelyn Procci, Michael Boyce, Julia Wright, Andre Garcia, and Michael J. Barnes. 2014. Situation Awareness\u2013Based Agent Transparency. Technical Report. Army Research Laboratory, Washington, DC. https:\/\/www.researchgate.net\/publication\/264963346_Situation_Awareness-Based_Agent_Transparency."},{"key":"e_1_2_1_21_1","volume-title":"Towards a rigorous science of interpretable machine learning. arxiv:1702.086ACM08","author":"Doshi-Velez Finale","unstructured":"Finale Doshi-Velez and Been Kim . 2017. Towards a rigorous science of interpretable machine learning. arxiv:1702.086ACM08 . Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arxiv:1702.086ACM08."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 7th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913)","author":"Dragan Anca D.","year":"2013","unstructured":"Anca D. Dragan , Kenton C. T. Lee , and Siddhartha S. Srinivasa . 2013. Legibility and predictability of robot motion . In Proceedings of the 7th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913) . IEEE, Los Alamitos, CA, 301\u2013308. DOI:https:\/\/doi.org\/10.1109\/HRI. 2013 .6483603 10.1109\/HRI.2013.6483603 Anca D. Dragan, Kenton C. T. Lee, and Siddhartha S. Srinivasa. 2013. Legibility and predictability of robot motion. In Proceedings of the 7th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201913). IEEE, Los Alamitos, CA, 301\u2013308. DOI:https:\/\/doi.org\/10.1109\/HRI.2013.6483603"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781315092898-13"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/MRA.2019.2904644"},{"key":"e_1_2_1_25_1","volume-title":"Manstead","author":"Fischer Agneta H.","year":"2008","unstructured":"Agneta H. Fischer and Antony S. R . Manstead . 2008 . Social functions of emotion and emotion regulation. In Handbook of Emotions. Guilford Press , 456\u2013468. Agneta H. Fischer and Antony S. R. Manstead. 2008. Social functions of emotion and emotion regulation. In Handbook of Emotions. Guilford Press, 456\u2013468."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1515\/pjbr-2018-0007"},{"key":"e_1_2_1_27_1","volume-title":"Aha","author":"Floyd Michael W.","year":"2016","unstructured":"Michael W. Floyd and David W . Aha . 2016 . Incorporating transparency during trust-guided behavior adaptation. In Case-Based Reasoning Research and Development. Lecture Notes in Computer Science , Vol. Springer, 124\u2013 138 . DOI:https:\/\/doi.org\/10.1007\/978-3-319-47096-2_9 10.1007\/978-3-319-47096-2_9 Michael W. Floyd and David W. Aha. 2016. Incorporating transparency during trust-guided behavior adaptation. In Case-Based Reasoning Research and Development. Lecture Notes in Computer Science, Vol. Springer, 124\u2013138. DOI:https:\/\/doi.org\/10.1007\/978-3-319-47096-2_9"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3233\/AIC-170733"},{"key":"e_1_2_1_29_1","series-title":"Lecture Notes in Computer Science","volume-title":"Intelligent Agents V: Agents Theories, Architectures, and Languages","author":"Georgeff Michael","unstructured":"Michael Georgeff , Barney Pell , Martha Pollack , Milind Tambe , and Michael Wooldridge . 1999. The belief-desire-intention model of agency . In Intelligent Agents V: Agents Theories, Architectures, and Languages . Lecture Notes in Computer Science , Vol. 1555 . Springer , 1\u201310. DOI:https:\/\/doi.org\/10.1007\/3-540-49057-4_1 10.1007\/3-540-49057-4_1 Michael Georgeff, Barney Pell, Martha Pollack, Milind Tambe, and Michael Wooldridge. 1999. The belief-desire-intention model of agency. In Intelligent Agents V: Agents Theories, Architectures, and Languages. Lecture Notes in Computer Science, Vol. 1555. Springer, 1\u201310. DOI:https:\/\/doi.org\/10.1007\/3-540-49057-4_1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2018.8525675"},{"key":"e_1_2_1_31_1","unstructured":"Shane Griffith Kaushik Subramanian Jonathan Scholz Charles L. Isbell and Andrea Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems. 2625\u20132633. http:\/\/papers.nips.cc\/paper\/5187-policy-shaping-integrating-human-feedback-with-reinforcement-learning.pdf  Shane Griffith Kaushik Subramanian Jonathan Scholz Charles L. Isbell and Andrea Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems. 2625\u20132633. http:\/\/papers.nips.cc\/paper\/5187-policy-shaping-integrating-human-feedback-with-reinforcement-learning.pdf"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2018.8593716"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/DevLrn.2013.6652523"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 2017 ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201917)","author":"Hayes Bradley","year":"2023","unstructured":"Bradley Hayes and Julie A. Shah . 2017. Improving robot controller transparency through autonomous policy explanation . In Proceedings of the 2017 ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201917) . IEEE, Los Alamitos, CA, 303\u2013312. DOI:https:\/\/doi.org\/10.1145\/2909824.30 2023 3 10.1145\/2909824.3020233 Bradley Hayes and Julie A. Shah. 2017. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 2017 ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201917). IEEE, Los Alamitos, CA, 303\u2013312. DOI:https:\/\/doi.org\/10.1145\/2909824.3020233"},{"key":"e_1_2_1_35_1","volume-title":"Ethics Guidelines for Trustworthy AI. Retrieved","author":"High-Level Expert Group on Artificial Intelligence. 2019.","year":"2021","unstructured":"High-Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy AI. Retrieved April 2, 2021 from https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai. High-Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy AI. Retrieved April 2, 2021 from https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-018-9771-0"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 2012 IEEE 2nd International Conference on Cloud Computing and Intelligence Systems (CCIS\u201912)","volume":"3","author":"Huang Xiangyang","year":"2013","unstructured":"Xiangyang Huang , Cuihuan Du , Yan Peng , Xuren Wang , and Jie Liu . 2013 . Goal-oriented action planning in partially observable stochastic domains . In Proceedings of the 2012 IEEE 2nd International Conference on Cloud Computing and Intelligence Systems (CCIS\u201912) , Vol. 3 . IEEE, Los Alamitos, CA, 1381\u20131385. DOI:https:\/\/doi.org\/10.1109\/CCIS. 2012.6664612 10.1109\/CCIS.2012.6664612 Xiangyang Huang, Cuihuan Du, Yan Peng, Xuren Wang, and Jie Liu. 2013. Goal-oriented action planning in partially observable stochastic domains. In Proceedings of the 2012 IEEE 2nd International Conference on Cloud Computing and Intelligence Systems (CCIS\u201912), Vol. 3. IEEE, Los Alamitos, CA, 1381\u20131385. DOI:https:\/\/doi.org\/10.1109\/CCIS.2012.6664612"},{"key":"e_1_2_1_38_1","series-title":"Lecture Notes in Computer Science","volume-title":"Symbiotic Interaction","author":"Jacucci Giulio","unstructured":"Giulio Jacucci , Anna Spagnolli , Jonathan Freeman , and Luciano Gamberini . 2014. Symbiotic interaction: A critical definition and comparison to other human-computer paradigms . In Symbiotic Interaction . Lecture Notes in Computer Science , Vol. 8820 . Springer , 3\u201320. DOI:https:\/\/doi.org\/10.1007\/978-3-319-13500-7_1 10.1007\/978-3-319-13500-7_1 Giulio Jacucci, Anna Spagnolli, Jonathan Freeman, and Luciano Gamberini. 2014. Symbiotic interaction: A critical definition and comparison to other human-computer paradigms. In Symbiotic Interaction. Lecture Notes in Computer Science, Vol. 8820. Springer, 3\u201320. DOI:https:\/\/doi.org\/10.1007\/978-3-319-13500-7_1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-018-9764-z"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1597735.1597738"},{"key":"e_1_2_1_41_1","series-title":"Lecture Notes in Computer Science","volume-title":"Social Robotics","author":"Knox W. Bradley","unstructured":"W. Bradley Knox , Peter Stone , and Cynthia Breazeal . 2013. Training a robot via human feedback: A case study . In Social Robotics . Lecture Notes in Computer Science , Vol. 8239 . Springer , 460\u2013470. DOI:https:\/\/doi.org\/10.1007\/978-3-319-02675-6_46 10.1007\/978-3-319-02675-6_46 W. Bradley Knox, Peter Stone, and Cynthia Breazeal. 2013. Training a robot via human feedback: A case study. In Social Robotics. Lecture Notes in Computer Science, Vol. 8239. Springer, 460\u2013470. DOI:https:\/\/doi.org\/10.1007\/978-3-319-02675-6_46"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.2307\/412428"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201919)","author":"Kulkarni Anagha","year":"2019","unstructured":"Anagha Kulkarni , Satya Gautam Vadlamudi , Yantian Zha , Yu Zhang , Tathagata Chakraborti , and Subbarao Kambhampati . 2019 . Explicable planning as minimizing distance from expected behavior . In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201919) .2075\u20132077. https:\/\/dl.acm.org\/doi\/10.5555\/3306127.3332015 Anagha Kulkarni, Satya Gautam Vadlamudi, Yantian Zha, Yu Zhang, Tathagata Chakraborti, and Subbarao Kambhampati. 2019. Explicable planning as minimizing distance from expected behavior. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201919).2075\u20132077. https:\/\/dl.acm.org\/doi\/10.5555\/3306127.3332015"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE","author":"Kwon Minae","unstructured":"Minae Kwon , Sandy H. Huang , and Anca D. Dragan . 2018. Expressing robot incapability . In Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE , Los Alamitos, CA, 87\u201395. DOI:https:\/\/doi.org\/10.1145\/3171221.3171276 10.1145\/3171221.3171276 Minae Kwon, Sandy H. Huang, and Anca D. Dragan. 2018. Expressing robot incapability. In Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE, Los Alamitos, CA, 87\u201395. DOI:https:\/\/doi.org\/10.1145\/3171221.3171276"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE","author":"Lamb Maurice","unstructured":"Maurice Lamb , Riley Mayr , Tamara Lorenz , Ali A. Minai , and Michael J. Richardson . 2018. The paths we pick together: A behavioral dynamics algorithm for an HRI pick-and-place task . In Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE , Los Alamitos, CA, 165\u2013166. DOI:https:\/\/doi.org\/10.1145\/3173386.3177022 10.1145\/3173386.3177022 Maurice Lamb, Riley Mayr, Tamara Lorenz, Ali A. Minai, and Michael J. Richardson. 2018. The paths we pick together: A behavioral dynamics algorithm for an HRI pick-and-place task. In Proceedings of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE, Los Alamitos, CA, 165\u2013166. DOI:https:\/\/doi.org\/10.1145\/3173386.3177022"},{"key":"e_1_2_1_46_1","volume-title":"Computationally modeling interpersonal trust. Frontiers in Psychology 4 (Dec","author":"Lee Jin Joo","year":"2013","unstructured":"Jin Joo Lee , W. Bradley Knox , Jolie B. Wormwood , Cynthia Breazeal , and David DeSteno . 2013. Computationally modeling interpersonal trust. Frontiers in Psychology 4 (Dec . 2013 ), 893. DOI:https:\/\/doi.org\/10.3389\/fpsyg.2013.00893 10.3389\/fpsyg.2013.00893 Jin Joo Lee, W. Bradley Knox, Jolie B. Wormwood, Cynthia Breazeal, and David DeSteno. 2013. Computationally modeling interpersonal trust. Frontiers in Psychology 4 (Dec. 2013), 893. DOI:https:\/\/doi.org\/10.3389\/fpsyg.2013.00893"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/1734454.1734544"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673083"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13673-019-0167-8"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2015.01.001"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/1703775.1703800"},{"key":"e_1_2_1_53_1","volume-title":"AAAI Spring Symposium Series. AAAI, Stanford, 48\u201353","author":"Lyons Joseph B.","year":"2013","unstructured":"Joseph B. Lyons . 2013 . Being transparent about transparency: A model for human-robot interaction . In AAAI Spring Symposium Series. AAAI, Stanford, 48\u201353 . DOI:https:\/\/doi.org\/10.1021\/ja00880a025 10.1021\/ja00880a025 Joseph B. Lyons. 2013. Being transparent about transparency: A model for human-robot interaction. In AAAI Spring Symposium Series. AAAI, Stanford, 48\u201353. DOI:https:\/\/doi.org\/10.1021\/ja00880a025"},{"key":"e_1_2_1_54_1","first-page":"42","article-title":"How the mind explains behavior: Folk explanations, meaning, and social interaction","volume":"42","author":"Malle B. F.","year":"2005","unstructured":"B. F. Malle . 2005 . How the mind explains behavior: Folk explanations, meaning, and social interaction . Choice Reviews Online 42 , 10 (2005), 42 - 6170 . DOI:https:\/\/doi.org\/10.5860\/choice.42-6170 10.5860\/choice.42-6170 B. F. Malle. 2005. How the mind explains behavior: Folk explanations, meaning, and social interaction. Choice Reviews Online 42, 10 (2005), 42-6170. DOI:https:\/\/doi.org\/10.5860\/choice.42-6170","journal-title":"Choice Reviews Online"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2016.7745140"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-020-09459-6"},{"key":"e_1_2_1_58_1","volume-title":"Chen","author":"Ososky Scott","year":"2014","unstructured":"Scott Ososky , Tracy Sanders , Florian Jentsch , Peter Hancock , and Jessie Y. C . Chen . 2014 . Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems. In Unmanned Systems Technology XVI, Robert E. Karlsen, Douglas W. Gage, Charles M. Shoemaker, and Grant R. Gerhart (Eds.), Vol. 9084 . SPIE , 90840E. DOI:https:\/\/doi.org\/10.1117\/12.2050622 10.1117\/12.2050622 Scott Ososky, Tracy Sanders, Florian Jentsch, Peter Hancock, and Jessie Y. C. Chen. 2014. Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems. In Unmanned Systems Technology XVI, Robert E. Karlsen, Douglas W. Gage, Charles M. Shoemaker, and Grant R. Gerhart (Eds.), Vol. 9084. SPIE, 90840E. DOI:https:\/\/doi.org\/10.1117\/12.2050622"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2017.8172279"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2016.XII.040"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/TENCONSpring.2018.8692047"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989122"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-019-09408-y"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201916)","author":"Schaefer Kristin E.","year":"2016","unstructured":"Kristin E. Schaefer , Ralph W. Brewer , Joe Putney , Edward Mottern , Jeffrey Barghout , and Edward R. Straub . 2016. Relinquishing manual control collaboration requires the capability to understand robot intention . In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201916) . IEEE, Los Alamitos, CA, 359\u2013366. DOI:https:\/\/doi.org\/10.1109\/CTS. 2016 .69 10.1109\/CTS.2016.69 Kristin E. Schaefer, Ralph W. Brewer, Joe Putney, Edward Mottern, Jeffrey Barghout, and Edward R. Straub. 2016. Relinquishing manual control collaboration requires the capability to understand robot intention. In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201916). IEEE, Los Alamitos, CA, 359\u2013366. DOI:https:\/\/doi.org\/10.1109\/CTS.2016.69"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2014.2312399"},{"key":"e_1_2_1_66_1","first-page":"J77","article-title":"Explanation capabilities of production-based consultation systems","volume":"14","author":"Scott A Carlisle","year":"1977","unstructured":"A Carlisle Scott , William J. Clancey , Randall Davis , and Edward H. Shortliffe . 1977 . Explanation capabilities of production-based consultation systems . American Journal of Computational Linguistics 14 , J77 - J71 (1977), 1\u201350. https:\/\/www.aclweb.org\/anthology\/J77-1006 A Carlisle Scott, William J. Clancey, Randall Davis, and Edward H. Shortliffe. 1977. Explanation capabilities of production-based consultation systems. American Journal of Computational Linguistics 14, J77-1 (1977), 1\u201350. https:\/\/www.aclweb.org\/anthology\/J77-1006","journal-title":"American Journal of Computational Linguistics"},{"key":"e_1_2_1_67_1","volume-title":"Companion of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE","author":"Sheikholeslami Sara","unstructured":"Sara Sheikholeslami , Justin W. Hart , Wesley P. Chan , Camilo P. Quintero , and Elizabeth A. Croft . 2018. Prediction and production of human reaching trajectories for human-robot interaction . In Companion of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE , Los Alamitos, CA, 321\u2013322. DOI:https:\/\/doi.org\/10.1145\/3173386.3176924 10.1145\/3173386.3176924 Sara Sheikholeslami, Justin W. Hart, Wesley P. Chan, Camilo P. Quintero, and Elizabeth A. Croft. 2018. Prediction and production of human reaching trajectories for human-robot interaction. In Companion of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction. IEEE, Los Alamitos, CA, 321\u2013322. DOI:https:\/\/doi.org\/10.1145\/3173386.3176924"},{"key":"e_1_2_1_68_1","series-title":"Lecture Notes in Computer Science","volume-title":"Symbiotic Interaction","author":"Spagnolli Anna","unstructured":"Anna Spagnolli , Lily E. Frank , Pim Haselager , and David Kirsh . 2018. Transparency as an ethical safeguard . In Symbiotic Interaction . Lecture Notes in Computer Science , Vol. 10727 . Springer , 1\u20136. DOI:https:\/\/doi.org\/10.1007\/978-3-319-91593-7_1 10.1007\/978-3-319-91593-7_1 Anna Spagnolli, Lily E. Frank, Pim Haselager, and David Kirsh. 2018. Transparency as an ethical safeguard. In Symbiotic Interaction. Lecture Notes in Computer Science, Vol. 10727. Springer, 1\u20136. DOI:https:\/\/doi.org\/10.1007\/978-3-319-91593-7_1"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the 20th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201911)","author":"Halit","year":"2011","unstructured":"Halit B. Suay and Sonia Chernova. 2011. Effect of human guidance and state space size on interactive reinforcement learning . In Proceedings of the 20th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201911) . IEEE, Los Alamitos, CA, 1\u20136. DOI:https:\/\/doi.org\/10.1109\/ROMAN. 2011 .6005223 10.1109\/ROMAN.2011.6005223 Halit B. Suay and Sonia Chernova. 2011. Effect of human guidance and state space size on interactive reinforcement learning. In Proceedings of the 20th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201911). IEEE, Los Alamitos, CA, 1\u20136. DOI:https:\/\/doi.org\/10.1109\/ROMAN.2011.6005223"},{"key":"e_1_2_1_70_1","volume-title":"Barto","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G . Barto . 2018 . Reinforcement Learning : An Introduction. MIT Press , Cambridge, MA. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the AISB Workshop on Principles of Robotics. 4. http:\/\/opus.bath.ac.uk\/49713\/.","author":"Theodorou Andreas","unstructured":"Andreas Theodorou , Robert H. Wortham , and Joanna J. Bryson . 2016. Why is my robot behaving like that? Designing transparency for real time inspection of autonomous robots . In Proceedings of the AISB Workshop on Principles of Robotics. 4. http:\/\/opus.bath.ac.uk\/49713\/. Andreas Theodorou, Robert H. Wortham, and Joanna J. Bryson. 2016. Why is my robot behaving like that? Designing transparency for real time inspection of autonomous robots. In Proceedings of the AISB Workshop on Principles of Robotics. 4. http:\/\/opus.bath.ac.uk\/49713\/."},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the 5th International Conference on Development and Learning (ICDL\u201906)","author":"Andrea","unstructured":"Andrea L. Thomaz and Cynthia Breazeal. 2006. Transparency and socially guided machine learning . In Proceedings of the 5th International Conference on Development and Learning (ICDL\u201906) . IEEE, Los Alamitos, CA, 1\u2013146. https:\/\/dspace.mit.edu\/handle\/1721.1\/36160. Andrea L. Thomaz and Cynthia Breazeal. 2006. Transparency and socially guided machine learning. In Proceedings of the 5th International Conference on Development and Learning (ICDL\u201906). IEEE, Los Alamitos, CA, 1\u2013146. https:\/\/dspace.mit.edu\/handle\/1721.1\/36160."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2007.09.009"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(78)80049-2"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201916)","author":"Wang Ning","unstructured":"Ning Wang , David V. Pynadath , and Susan G. Hill . 2016. The impact of POMDP-generated explanations on trust and performance in human-robot teams . In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201916) . ACM, New York, NY, 997\u20131005. https:\/\/dl.acm.org\/citation.cfm?id=2937071. Ning Wang, David V. Pynadath, and Susan G. Hill. 2016. The impact of POMDP-generated explanations on trust and performance in human-robot teams. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201916). ACM, New York, NY, 997\u20131005. https:\/\/dl.acm.org\/citation.cfm?id=2937071."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916)","volume":"116","author":"Wang Ning","year":"2016","unstructured":"Ning Wang , David V. Pynadath , and Susan G. Hill . 2016. Trust calibration within a human-robot team: Comparing automatically generated explanations . In Proceedings of the 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916) , Vol. 2016-April. IEEE, Los Alamitos, CA, 109\u2013 116 . DOI:https:\/\/doi.org\/10.1109\/HRI. 2016 .7451741 10.1109\/HRI.2016.7451741 Ning Wang, David V. Pynadath, and Susan G. Hill. 2016. Trust calibration within a human-robot team: Comparing automatically generated explanations. In Proceedings of the 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI\u201916), Vol. 2016-April. IEEE, Los Alamitos, CA, 109\u2013116. DOI:https:\/\/doi.org\/10.1109\/HRI.2016.7451741"},{"key":"e_1_2_1_77_1","volume-title":"Hill","author":"Wang Ning","year":"2018","unstructured":"Ning Wang , David V. Pynadath , Ericka Rovira , Michael J. Barnes , and Susan G . Hill . 2018 . Is it my looks? Or something I said? The impact of explanations, embodiment, and expectations on trust and performance in human-robot teams. In Persuasive Technology. Lecture Notes in Computer Science, Vol. 10809 . Springer , 56\u201369. DOI:https:\/\/doi.org\/10.1007\/978-3-319-78978-1_5 10.1007\/978-3-319-78978-1_5 Ning Wang, David V. Pynadath, Ericka Rovira, Michael J. Barnes, and Susan G. Hill. 2018. Is it my looks? Or something I said? The impact of explanations, embodiment, and expectations on trust and performance in human-robot teams. In Persuasive Technology. Lecture Notes in Computer Science, Vol. 10809. Springer, 56\u201369. DOI:https:\/\/doi.org\/10.1007\/978-3-319-78978-1_5"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(92)90087-E"},{"key":"e_1_2_1_80_1","volume-title":"Advances in Human Factors in Robots and Unmanned Systems. Advances in Intelligent Systems and Computing","author":"Evans A. William","unstructured":"A. William Evans , Matthew Marge , Ethan Stump , Garrett Warnell , Joseph Conroy , Douglas Summers-Stay , and David Baran . 2017. The future of human robot teams in the army: Factors affecting a model of human-system dialogue towards greater team collaboration . In Advances in Human Factors in Robots and Unmanned Systems. Advances in Intelligent Systems and Computing , Vol. 499 . Springer , 197\u2013210. DOI:https:\/\/doi.org\/10.1007\/978-3-319-41959-6_17 10.1007\/978-3-319-41959-6_17 A. William Evans, Matthew Marge, Ethan Stump, Garrett Warnell, Joseph Conroy, Douglas Summers-Stay, and David Baran. 2017. The future of human robot teams in the army: Factors affecting a model of human-system dialogue towards greater team collaboration. In Advances in Human Factors in Robots and Unmanned Systems. Advances in Intelligent Systems and Computing, Vol. 499. Springer, 197\u2013210. DOI:https:\/\/doi.org\/10.1007\/978-3-319-41959-6_17"},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the Ethics for Artificial Intelligence Workshop (IJCAI\u201916)","author":"Wortham Robert H.","unstructured":"Robert H. Wortham , Andreas Theodorou , and Joanna J. Bryson . 2016. What does the robot think? Transparency as a fundamental design requirement for intelligent systems . In Proceedings of the Ethics for Artificial Intelligence Workshop (IJCAI\u201916) . http:\/\/www.robwortham.com\/instinct-planner\/http:\/\/opus.bath.ac.uk\/50294\/1\/WorthamTheodorouBryson_EFAI16.pdf. Robert H. Wortham, Andreas Theodorou, and Joanna J. Bryson. 2016. What does the robot think? Transparency as a fundamental design requirement for intelligent systems. In Proceedings of the Ethics for Artificial Intelligence Workshop (IJCAI\u201916). http:\/\/www.robwortham.com\/instinct-planner\/http:\/\/opus.bath.ac.uk\/50294\/1\/WorthamTheodorouBryson_EFAI16.pdf."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.5555\/1819998.1820502"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction","volume":"1271","author":"Zhou Allan","year":"2022","unstructured":"Allan Zhou , Dylan Hadfield-Menell , Anusha Nagabandi , and Anca D. Dragan . 2017. Expressive robot motion timing . In Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction , Vol. Part F 1271 . IEEE, Los Alamitos, CA, 22\u201331. DOI:https:\/\/doi.org\/10.1145\/2909824.30 2022 1 arxiv:1802.01536 10.1145\/2909824.3020221 Allan Zhou, Dylan Hadfield-Menell, Anusha Nagabandi, and Anca D. Dragan. 2017. Expressive robot motion timing. In Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction, Vol. Part F1271. IEEE, Los Alamitos, CA, 22\u201331. DOI:https:\/\/doi.org\/10.1145\/2909824.3020221 arxiv:1802.01536"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3457188","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3457188","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:19Z","timestamp":1750191439000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3457188"}},"subtitle":["A Review"],"short-title":[],"issued":{"date-parts":[[2021,7,11]]},"references-count":82,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,9,30]]}},"alternative-id":["10.1145\/3457188"],"URL":"https:\/\/doi.org\/10.1145\/3457188","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,11]]},"assertion":[{"value":"2019-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}