{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:16:38Z","timestamp":1750220198520,"version":"3.41.0"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Research Council under the European Unions Horizon 2020 Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>As social robots become more prominent in our lives, their interaction with humans takes an increasing role, and new collaborative scenarios emerge. This development brings the need to realize robust test methods enabling the design and evaluation of Human-Robot Interaction (HRI) teaming tasks to prove functionality and promote adoption. In this article, we present a general-purpose and repeatable methodology for conducting studies in collaborative HRI in the range of robotic competitions. The methodology includes a step-by-step approach to design HRI teaming tasks tailored to be enacted in a robotic competition and to evaluate the performance of social robots to execute the designed tasks, exploring the relationship between robots\u2019 performance and user perceptions based on the feedback of the users participating to such tasks. We assess the feasibility of the methodology to design and evaluate an HRI teaming task in the context of \u201cSmart CIties RObotics Challenges\u201d (SciRoc) competition, which targets at investigating the impact of social of robots in smart cities.<\/jats:p>","DOI":"10.1145\/3528415","type":"journal-article","created":{"date-parts":[[2022,4,21]],"date-time":"2022-04-21T12:48:58Z","timestamp":1650545338000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A Methodology to Design and Evaluate HRI Teaming Tasks in Robotic Competitions"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1031-0374","authenticated-orcid":false,"given":"Andrea","family":"Marrella","sequence":"first","affiliation":[{"name":"Sapienza University of Rome, Rome, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7509-9250","authenticated-orcid":false,"given":"Lun","family":"Wang","sequence":"additional","affiliation":[{"name":"Sapienza University of Rome, Rome, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9057-8946","authenticated-orcid":false,"given":"Luca","family":"Iocchi","sequence":"additional","affiliation":[{"name":"Sapienza University of Rome, Rome, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6606-200X","authenticated-orcid":false,"given":"Daniele","family":"Nardi","sequence":"additional","affiliation":[{"name":"Sapienza University of Rome, Rome, Italy"}]}],"member":"320","published-online":{"date-parts":[[2022,9,2]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/DEST.2008.4635149"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Francesco Amigoni Emanuele Bastianelli Jakob Berghofer Andrea Bonarini Giulio Fontana Nico Hochgeschwender Luca Iocchi Gerhard Kraetzschmar Pedro Lima Matteo Matteucci Pedro Miraldo Daniele Nardi and Viola Schiaffonati. 2015. Competitions for benchmarking: Task and functionality scoring complete performance assessment. IEEE Robotics & Automation Magazine 22 3 (2015) 53\u201361.","DOI":"10.1109\/MRA.2015.2448871"},{"key":"e_1_3_2_4_2","volume-title":"Proceedings of the Dagstuhl Reports","author":"Andr\u00e9 Elisabeth","year":"2020","unstructured":"Elisabeth Andr\u00e9, Ana Paiva, Julie Shah, and Selma \u0160abanovic. 2020. Social agents for teamwork and group interactions (dagstuhl seminar 19411). In Proceedings of the Dagstuhl Reports. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1754046"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781108676649"},{"key":"e_1_3_2_7_2","unstructured":"Christoph Bartneck Tatsuya Nomura Takayuki Kanda Tomohiro Suzuki and Kennsuke Kato. 2005. Cultural differences in attitudes towards robots. In Proceedings of the SSAISB 2005 Convention on Social Intelligence and Interaction in Animals Robots and Agents . 1\u20134."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2006.314414"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-006-0052-7"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0064-9"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2007.4415182"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00018-1"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673222"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121272"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020236"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2012.6343858"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Terrence Fong Illah Nourbakhsh and Kerstin Dautenhahn. 2003. A survey of socially interactive robots. Robotics and Autonomous Systems 42 3\u20134 (2003) 143\u2013166.","DOI":"10.1016\/S0921-8890(02)00372-X"},{"issue":"16","key":"e_1_3_2_18_2","first-page":"3","article-title":"Alfa de cronbach y consistencia interna de los \u00edtems de un instrumento de medida","volume":"3","author":"George S.","year":"2003","unstructured":"S. George and L. Mallery. 2003. Alfa de cronbach y consistencia interna de los \u00edtems de un instrumento de medida. Revista de Estudios Interdisciplinarios en Ciencias Sociales 3, 16 (2003), 3\u20139.","journal-title":"Revista de Estudios Interdisciplinarios en Ciencias Sociales"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.5555\/1571813"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1957656.1957704"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tins.2020.03.013"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12130-009-9070-3"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02713-0_36"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.7.3.15hut"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2015.08.002"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2147783.2147787"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2009.5326292"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2559636.2559676"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0056-9"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8535.2011.01258.x"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.12.315"},{"key":"e_1_3_2_32_2","unstructured":"Andrea Marrella Massimo Mecella and Alessandro Russo. 2011. Collaboration on-the-field: Suggestions and beyond. In Proceedings of the 8th International Conference on Information Systems for Crisis Response and Management ."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3371382.3374845"},{"key":"e_1_3_2_34_2","first-page":"1","article-title":"Towards robots able to measure in real-time the quality of interaction in HRI contexts","author":"Mayima Amandine","year":"2021","unstructured":"Amandine Mayima, Aur\u00e9lie Clodic, and Rachid Alami. 2021. Towards robots able to measure in real-time the quality of interaction in HRI contexts. International Journal of Social Robotics 14, 3 (2021), 1\u201319.","journal-title":"International Journal of Social Robotics"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1080\/01691864.2019.1698462"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2006.314397"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0050-2"},{"key":"e_1_3_2_38_2","first-page":"29","volume-title":"Proceedings of the AAAI-06 Workshop on Human Implications of Human-robot Interaction","author":"Nomura Tatsuya","year":"2006","unstructured":"Tatsuya Nomura, Tomohiro Suzuki, Takayuki Kanda, and Kensuke Kato. 2006. Altered attitudes of people toward robots: Investigation through the negative attitudes toward robots scale. In Proceedings of the AAAI-06 Workshop on Human Implications of Human-robot Interaction. 29\u201335."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364916688254"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967974"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316436"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3185371"},{"key":"e_1_3_2_43_2","article-title":"A code of ethics for the human-robot interaction profession","author":"Riek Laurel","year":"2014","unstructured":"Laurel Riek and Don Howard. 2014. A code of ethics for the human-robot interaction profession. Proceedings of We Robot (2014). https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2757805.","journal-title":"Proceedings of We Robot"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.1.1.Riek"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2017.06.002"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/1349822.1349857"},{"key":"e_1_3_2_47_2","volume-title":"Research Methods in Psychology: A Handbook","author":"Schweigert Wendy A.","year":"2021","unstructured":"Wendy A. Schweigert. 2021. Research Methods in Psychology: A Handbook. Waveland Press."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/1952222.1952294"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2005.1570533"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020210"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121249"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203809716"},{"key":"e_1_3_2_53_2","first-page":"21","volume-title":"Proceedings of the 4th international Symposium on New Frontiers in Human Robot Interaction","author":"Strait Megan","year":"2015","unstructured":"Megan Strait, Priscilla Briggs, and Matthias Scheutz. 2015. Gender, more so than age, modulates positive perceptions of language-based human-robot interactions. In Proceedings of the 4th international Symposium on New Frontiers in Human Robot Interaction. 21\u201322."},{"key":"e_1_3_2_54_2","unstructured":"Dag Sverre Syrdal Kerstin Dautenhahn Kheng Lee Koay and Michael L. Walters. 2009. The negative attitudes to wards robots scale and reactions to robot behaviour in a live human-robot interaction study. In Proceedings of AISB09-23rd Convention of the Society for the Adaptive and Emergent Behaviour and Complex Systems ."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21619-0_76"},{"key":"e_1_3_2_56_2","first-page":"123","volume-title":"Proceedings of the 6th International Conference on Advances in Computer\u2013Human Interactions","author":"Walters M.","year":"2013","unstructured":"M. Walters, Samuel Marcos, Dag Sverre Syrdal, and Kerstin Dautenhahn. 2013. An interactive game with a robot: People\u2019s perceptions of robot faces and a gesture-based user interface. In Proceedings of the 6th International Conference on Advances in Computer\u2013Human Interactions. 123\u2013128."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/RO-MAN46459.2019.8956394"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29390-1_34"},{"key":"e_1_3_2_59_2","first-page":"150","volume-title":"Proceedings of the AISB2009: Proceedings of the Symposium on New Frontiers in Human-Robot Interaction.","author":"Weiss Astrid","year":"2009","unstructured":"Astrid Weiss, Regina Bernhaupt, Manfred Tscheligi, and Eiichi Yoshida. 2009. Addressing user experience and societal impact in a user study with a humanoid robot. In Proceedings of the AISB2009: Proceedings of the Symposium on New Frontiers in Human-Robot Interaction. Citeseer, 150\u2013157."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1514095.1514166"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/142167.142170"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-014-0248-9"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci1901&2_6"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21568"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0081-8"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528415","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528415","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:42Z","timestamp":1750186962000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528415"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,2]]},"references-count":64,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3528415"],"URL":"https:\/\/doi.org\/10.1145\/3528415","relation":{},"ISSN":["2573-9522","2573-9522"],"issn-type":[{"type":"print","value":"2573-9522"},{"type":"electronic","value":"2573-9522"}],"subject":[],"published":{"date-parts":[[2022,9,2]]},"assertion":[{"value":"2020-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}