{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:35:57Z","timestamp":1776108957245,"version":"3.50.1"},"reference-count":54,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T00:00:00Z","timestamp":1713225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002428","name":"Austrian Science Fund","doi-asserted-by":"publisher","award":["CM 100-N"],"award-info":[{"award-number":["CM 100-N"]}],"id":[{"id":"10.13039\/501100002428","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>In human-robot collaboration, failures are bound to occur. A thorough understanding of potential errors is necessary so that robotic system designers can develop systems that remedy failure cases. In this work, we study failures that occur when participants interact with a working system and focus especially on errors in a robotic system\u2019s knowledge base of which the system is not aware. A human interaction partner can be part of the error detection process if they are given insight into the robot\u2019s knowledge and decision-making process. We investigate different communication modalities and the design of shared task representations in a joint human-robot object organization task. We conducted a user study (<jats:italic>N<\/jats:italic> = 31) in which the participants showed a Pepper robot how to organize objects, and the robot communicated the learned object configuration to the participants by means of speech, visualization, or a combination of speech and visualization. The multimodal, combined condition was preferred by 23 participants, followed by seven participants preferring the visualization. Based on the interviews, the errors that occurred, and the object configurations generated by the participants, we conclude that participants tend to test the system\u2019s limitations by making the task more complex, which provokes errors. This trial-and-error behavior has a productive purpose and demonstrates that failures occur that arise from the combination of robot capabilities, the user\u2019s understanding and actions, and interaction in the environment. Moreover, it demonstrates that failure can have a productive purpose in establishing better user mental models of the technology.<\/jats:p>","DOI":"10.3389\/frobt.2024.1356827","type":"journal-article","created":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T04:49:47Z","timestamp":1713242987000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Human-in-the-loop error detection in an object organization task with a social robot"],"prefix":"10.3389","volume":"11","author":[{"given":"Helena Anna","family":"Frijns","sequence":"first","affiliation":[]},{"given":"Matthias","family":"Hirschmanner","sequence":"additional","affiliation":[]},{"given":"Barbara","family":"Sienkiewicz","sequence":"additional","affiliation":[]},{"given":"Peter","family":"H\u00f6nig","sequence":"additional","affiliation":[]},{"given":"Bipin","family":"Indurkhya","sequence":"additional","affiliation":[]},{"given":"Markus","family":"Vincze","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,4,16]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.05814","article-title":"Deep ViT features as dense visual descriptors","author":"Amir","year":"2022"},{"key":"B2","article-title":"A human-centric approach to autonomous robot failures","author":"Brooks","year":"2017"},{"key":"B3","first-page":"431","article-title":"Teaching people how to teach robots: the effect of instructional materials and dialog design","author":"Cakmak","year":"2014"},{"key":"B4","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TRO.2004.838027","article-title":"How UGVs physically fail in the field","volume":"21","author":"Carlson","year":"2005","journal-title":"IEEE Trans. Robotics"},{"key":"B5","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511620539.005","article-title":"Common ground","volume-title":"Using Language","author":"Clark","year":"1996"},{"key":"B6","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/978-3-319-53133-5_8","article-title":"Key elements for human-robot joint action","volume-title":"Sociality and normativity for robots","author":"Clodic","year":"2017"},{"key":"B7","first-page":"351","article-title":"Explainable AI for robot failures: generating explanations that improve user assistance in fault recovery","author":"Das","year":"2021"},{"key":"B8","first-page":"3034","article-title":"Semantic-based explainable AI: leveraging semantic scene graphs and pairwise ranking to explain robot failures","author":"Das","year":"2021"},{"key":"B9","doi-asserted-by":"publisher","first-page":"937772","DOI":"10.3389\/frobt.2022.937772","article-title":"Leveraging explainability for understanding object descriptions in ambiguous 3D environments","volume":"9","author":"Do\u011fan","year":"2023","journal-title":"Front. Robotics AI"},{"key":"B10","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-642-00437-7_1","article-title":"Multimodal interfaces: a survey of principles, models and frameworks","volume-title":"Human machine interaction","author":"Dumas","year":"2009"},{"key":"B11","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.zefq.2019.10.003","article-title":"The \u2019National Aeronautics and Space Administration-Task Load Index\u2019 (NASA-TLX) - an instrument for measuring consultation workload within general practice: evaluation of psychometric properties","author":"Fl\u00e4gel","year":"2019","journal-title":"Z. f\u00fcr Evidenz, Fortbild. Qual. im Gesundheitswes."},{"key":"B12","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1007\/s12369-021-00785-7","article-title":"Communication models in human\u2013robot interaction: an asymmetric MODel of ALterity in human\u2013robot interaction (AMODAL-HRI)","volume":"15","author":"Frijns","year":"2023","journal-title":"Int. J. Soc. Robotics"},{"key":"B13","first-page":"2918","article-title":"Simple copy-paste is a strong data augmentation method for instance segmentation","author":"Ghiasi","year":"2021"},{"key":"B14","doi-asserted-by":"publisher","first-page":"931","DOI":"10.3389\/fpsyg.2015.00931","article-title":"Systematic analysis of video data from different human\u2013robot interaction studies: a categorization of social signals during error situations","volume":"6","author":"Giuliani","year":"2015","journal-title":"Front. Psychol."},{"key":"B15","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1561\/1100000005","article-title":"Human-robot interaction: a survey","volume":"1","author":"Goodrich","year":"2007","journal-title":"Found. Trends\u00ae Human-Computer Interact."},{"key":"B16","first-page":"1640","article-title":"Grounding spatial relations for human-robot interaction","author":"Guadarrama","year":"2013"},{"key":"B17","first-page":"218","article-title":"\u201cAn error occurred!\u201d - trust repair with virtual robot using levels of mistake explanation","author":"Hald","year":"2021"},{"key":"B18","first-page":"493","article-title":"Believing in BERT: using expressive communication to enhance trust and counteract operational error in physical human-robot interaction","author":"Hamacher","year":"2016"},{"key":"B19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3568024","article-title":"Communicating missing causal information to explain a robot\u2019s past behavior","volume":"12","author":"Han","year":"2023","journal-title":"ACM Trans. Human-Robot Interact."},{"key":"B20","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"NASA-task load index (NASA-TLX); 20 years later","volume":"50","author":"Hart","year":"2006","journal-title":"Proc. Hum. factors ergonomics Soc. Annu. Meet."},{"key":"B21","first-page":"175","article-title":"Investigating transparency methods in a robot word-learning system and their effects on human teaching behaviors","author":"Hirschmanner","year":"2021"},{"key":"B22","doi-asserted-by":"publisher","first-page":"861","DOI":"10.3389\/fpsyg.2018.00861","article-title":"Understanding and resolving failures in human-robot interaction: literature review and model development","volume":"9","author":"Honig","year":"2018","journal-title":"Front. Psychol."},{"key":"B23","volume-title":"ultralytics\/yolov5: v6.1 - TensorRT, TensorFlow edge TPU and OpenVINO export and inference","author":"Jocher","year":"2022"},{"key":"B24","first-page":"53","article-title":"Behavioural responses to robot conversational failures","author":"Kontogiorgos","year":""},{"key":"B25","first-page":"1","article-title":"Embodiment effects in interactions with failing robots","author":"Kontogiorgos","year":""},{"key":"B26","unstructured":"Paper.js\n            LehniJ.\n            PuckeyJ.\n          2011"},{"key":"B27","first-page":"740","article-title":"Microsoft COCO: common objects in context","author":"Lin","year":"2014"},{"key":"B28","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3389\/frobt.2017.00021","article-title":"To err is robot: how humans assess and act toward an erroneous social robot","volume":"4","author":"Mirnig","year":"2017","journal-title":"Front. Robotics AI"},{"key":"B29","first-page":"224","article-title":"Screen feedback in human-robot interaction: how to enhance robot expressiveness","author":"Mirnig","year":"2014"},{"key":"B30","first-page":"313","article-title":"Transparency in HRI: trust and decision making in the face of robot errors","author":"Nesset","year":"2021"},{"key":"B31","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1515\/pjbr-2020-0007","article-title":"User-centered feedback design in person-following robots for older adults","volume":"11","author":"Olatunji","year":"2020","journal-title":"Paladyn, J. Behav. Robotics"},{"key":"B32","unstructured":"OpenCV: detection of ArUco markers2023"},{"key":"B33","first-page":"8779","article-title":"Multimodal explanations: justifying decisions and pointing to the evidence","author":"Park","year":"2018"},{"key":"B34","volume-title":"statsmodels\/statsmodels: release 0.14.1","author":"Perktold","year":"2023"},{"key":"B35","doi-asserted-by":"crossref","DOI":"10.15607\/RSS.2016.XII.040","article-title":"Situated language understanding with human-like and visualization-based transparency","author":"Perlmutter","year":"2016"},{"key":"B36","article-title":"A view-based Approach to three-dimensional object perception (universit\u00e4t bielefeld)","author":"Peters","year":"2001"},{"key":"B37","doi-asserted-by":"publisher","first-page":"533","DOI":"10.2190\/EC.43.3.f","article-title":"Fail better: toward a taxonomy of e-learning error","volume":"43","author":"Priem","year":"2010","journal-title":"J. Educ. Comput. Res."},{"key":"B38","first-page":"20","volume-title":"Human error","author":"Reason","year":"2009"},{"key":"B39","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-26633-6","volume-title":"Modern statistical Methods for HCI","author":"Robertson","year":"2016"},{"key":"B40","first-page":"43","article-title":"A comparison of visualisation methods for disambiguating verbal requests in human-robot interaction","author":"Sibirtseva","year":"2018"},{"key":"B41","unstructured":"Python SDK - overview \u2014 aldebaran 2.5.11.14a documentation Note: website2023"},{"key":"B42","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1007\/978-3-642-39250-4_31","article-title":"A survey about faults of robots used in RoboCup","volume-title":"RoboCup 2012: robot soccer world cup XVI","author":"Steinbauer","year":"2013"},{"key":"B43","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1080\/13875868.2016.1143474","article-title":"Time will not help unskilled observers to understand a cluttered spatial scene","volume":"16","author":"Tenbrink","year":"2016","journal-title":"Spatial Cognition Comput."},{"key":"B44","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1080\/0163853X.2010.549452","article-title":"Spatial strategies in the description of complex configurations","volume":"48","author":"Tenbrink","year":"2011","journal-title":"Discourse Process."},{"key":"B45","doi-asserted-by":"publisher","first-page":"101625","DOI":"10.1016\/j.isci.2020.101625","article-title":"Do you see what I see? tracking the perceptual beliefs of robots","volume":"23","author":"Thellman","year":"2020","journal-title":"iScience"},{"key":"B46","volume-title":"scipy.stats.friedmanchisquare \u2014 SciPy v1.12.0 manual","year":"2008"},{"key":"B47","first-page":"1","article-title":"Redesigning Human-Robot Interaction in response to robot failures: a participatory design methodology","author":"Tian","year":"2021"},{"key":"B48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3439720","article-title":"A taxonomy of social errors in human-robot interaction","volume":"10","author":"Tian","year":"2021","journal-title":"ACM Trans. Human-Robot Interact."},{"key":"B49","first-page":"3","article-title":"Taxonomy of trust-relevant failures and mitigation strategies","author":"Tolmeijer","year":"2020"},{"key":"B50","first-page":"213","article-title":"Designing emotionally expressive robots: a comparative study on the perception of communication modalities","author":"Tsiourti","year":"2017"},{"key":"B51","first-page":"493","article-title":"Correct me if I\u2019m wrong: using non-experts to repair reinforcement learning policies","author":"Van Waveren","year":"2022"},{"key":"B52","first-page":"1","article-title":"A user interface for sense-making of the reasoning process while interacting with robots","author":"Wang","year":"2023"},{"key":"B53","first-page":"7","article-title":"What does the robot think? transparency as a fundamental design requirement for intelligent systems","author":"Wortham","year":"2016"},{"key":"B54","doi-asserted-by":"publisher","first-page":"1619","DOI":"10.1007\/s12369-023-01059-0","article-title":"Effects of failure types on trust repairs in human\u2013robot interactions","volume":"15","author":"Zhang","year":"2023","journal-title":"Int. J. Soc. Robotics"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1356827\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,16]],"date-time":"2024-04-16T04:49:57Z","timestamp":1713242997000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1356827\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,16]]},"references-count":54,"alternative-id":["10.3389\/frobt.2024.1356827"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1356827","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,16]]},"article-number":"1356827"}}