{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T15:35:37Z","timestamp":1778168137695,"version":"3.51.4"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000104","name":"National Aeronautics and Space Administration","doi-asserted-by":"crossref","award":["NNX17AB08G"],"award-info":[{"award-number":["NNX17AB08G"]}],"id":[{"id":"10.13039\/100000104","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Jessica Marquez serving as Program Manager"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>Metrics for human-robot teaming should extend to teams consisting of multiple human and robotic agents, and to teams working in complex, dynamic work domains. This work proposes that to comprehensively analyze and evaluate multi-human, multi-robot teams, traditional HRI metrics of performance, and efficiency must be expanded upon to incorporate metrics of teamwork. We develop five distinct metrics to capture both ecological and cognitive aspects of teamwork found to be important in human-automation interaction, inspired by research in the cognitive systems engineering (CSE) community. We demonstrate the application of these metrics in a spacecraft maintenance case study comparing multiple human-robot team architectures. The case study demonstrates that the teamwork metrics capture aspects of human-robot interaction (HRI) not apparent when using only traditional performance and efficiency metrics. The article concludes that the proposed teamwork metrics are complementary to existing metrics in HRI and should be included in the evaluation of human-robot teams.<\/jats:p>","DOI":"10.1145\/3522581","type":"journal-article","created":{"date-parts":[[2022,4,14]],"date-time":"2022-04-14T17:50:26Z","timestamp":1649958626000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Metrics for Human-Robot Team Design: A Teamwork Perspective on Evaluation of Human-Robot Teams"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9067-0769","authenticated-orcid":false,"given":"Lanssie Mingyue","family":"Ma","sequence":"first","affiliation":[{"name":"NASA Ames, Mountain View, CA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3992-2788","authenticated-orcid":false,"given":"Martijn","family":"Ijtsma","sequence":"additional","affiliation":[{"name":"The Ohio State University,  Neil AveColumbus, OH"}]},{"given":"Karen M.","family":"Feigh","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, GA"}]},{"given":"Amy R.","family":"Pritchett","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, PA"}]}],"member":"320","published-online":{"date-parts":[[2022,9,2]]},"reference":[{"key":"e_1_3_1_2_2","volume-title":"Proceedings of the 11th Conference on Computer-Generated Forces and Behavior Representation","author":"Acquisti Alessandro","year":"2002","unstructured":"Alessandro Acquisti, Maarten Sierhuis, William J. Clancev, and Jeffrey M. Bradshaw. 2002. Agent based modeling of collaboration and work practices onboard the international space station. In Proceedings of the 11th Conference on Computer-Generated Forces and Behavior Representation. Orlando, FL. Retrieved from https:\/\/ntrs.nasa.gov\/archive\/nasa\/casi.ntrs.nasa.gov\/20020063473.pdf."},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343412445577"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/SIS.2008.4668287"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1518\/155723408X299825"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781420063059.ch1"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.2514\/6.2015-4643"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470519851"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2016.7451876"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.2307\/2216075"},{"key":"e_1_3_1_11_2","unstructured":"Richard I. Cook and David D. Woods. 1994. Operating at the Sharp End: The Complexity of Human Error. In Human error in medicine . M. S. Bogner (Ed.). Lawrence Erlbaum Associates Inc. 225\u2013310."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872000779656561"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5979830"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1201\/b12461"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1080\/001401399185595"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343413490945"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010147"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2006.384609"},{"key":"e_1_3_1_19_2","volume-title":"Proceedings of the 10th International Symposium of Robotics Research","author":"Fong T.","year":"2001","unstructured":"T. Fong, C. Thorpe, and C. Baur. 2001. Collaboration, dialogue and human-robot interaction. In Proceedings of the 10th International Symposium of Robotics Research."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364916688255"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228721"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24004-1_2"},{"key":"e_1_3_1_23_2","volume-title":"Proceedings of the International Symposium on Artificial Intelligence","author":"Hiltz Michael","year":"2001","unstructured":"Michael Hiltz, Craig Rice, Keith Boyle, and Ronald Allison. 2001. Canadarm: 20 years of mission success through adaptation. In Proceedings of the International Symposium on Artificial Intelligence. Montreal, Canada."},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.2514\/6.2004-6434"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2005.1545010"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343419869484"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.3.1.Johnson"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1177\/001872089303500203"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1002\/0471739448.ch6"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67361-5_42"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1037\/10690-004"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/99332.99367"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/174666.174668"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/AERO.2016.7500580"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.85.2.273"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872007779598037"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274397"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISR.2013.6695625"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.5555\/2447556.2447563"},{"key":"e_1_3_1_41_2","first-page":"61","article-title":"Cognitive engineering","volume":"31","author":"Norman Donald A.","year":"1986","unstructured":"Donald A. Norman. 1986. Cognitive engineering. User Centered System Design 31 (1986), 61. Lawrence Erlbaum Associates, Hillsdale, NJ.","journal-title":"User Centered System Design"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.hrmr.2017.09.001"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720820960865"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/3468.844354"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1080\/00140130050084879"},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","first-page":"21","DOI":"10.21236\/ADA476904","volume-title":"Human Robotics Interaction Army Technology Objective Raven Small Unmanned Aerial Vehicle Task Analysis and Modeling","author":"Pomranky Regina A.","year":"2006","unstructured":"Regina A. Pomranky. 2006. Human Robotics Interaction Army Technology Objective Raven Small Unmanned Aerial Vehicle Task Analysis and Modeling. Technical Report. Human Research and Engineering Directorate, Army Research Lab. 21 pages."},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","unstructured":"Amy Pritchett. 2017. Work and the Expertise of Workers. In Cognitive Systems Engineering J. Smith Philip and R. Hoffman Robert (Eds.). CRC Press 229\u2013246.","DOI":"10.1201\/9781315572529-13"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.2514\/1.D0024"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010146"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343413490166"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343413490944"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022279618991"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1999.0293"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872008X288457"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-33261-1_2"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1177\/1046496405277134"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2003.1174284"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2003.1244385"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2004.1307409"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3415247"},{"key":"e_1_3_1_62_2","first-page":"1","article-title":"Analyzing air traffic management systems using agent-based modeling and simulation","author":"Shah J.","year":"2005","unstructured":"J. Shah, A. R. Pritchett, K. M. Feigh, S. A. Kalaver, A. Jadhav, D. M. Holl, R. C. Bea, and A. Z. Gilgur. 2005. Analyzing air traffic management systems using agent-based modeling and simulation. In Proceedings of the 6th USA\/Europe Air Traffic Management Research and Development. 1\u201312.","journal-title":"Proceedings of the 6th USA\/Europe Air Traffic Management Research and Development"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2007.900644"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.4271\/2009-01-2529"},{"key":"e_1_3_1_65_2","volume-title":"Human Factors Methods: A Practical Guide for Engineering and Design","author":"Stanton Neville A.","year":"2017","unstructured":"Neville A. Stanton, Paul M. Salmon, Laura A. Rafferty, Guy H. Walker, Chris Baber, and Daniel P. Jenkins. 2017. Human Factors Methods: A Practical Guide for Engineering and Design. CRC Press."},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121249"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2012.07.002"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1518\/155534309X12599553478791"},{"key":"e_1_3_1_69_2","unstructured":"Katia Sycara and Gita Sukthankar. 2006. Literature review of teamwork models. Robotics Institute Carnegie Mellon University. https:\/\/www.cs.cmu.edu\/gitars\/Papers\/CMU-RI-TR-06-50.pdf."},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1201\/b12457"},{"issue":"4","key":"e_1_3_1_71_2","first-page":"86","article-title":"Cognitive technologies: The design of joint human-machine cognitive systems","volume":"6","author":"Woods David D.","year":"1985","unstructured":"David D. Woods. 1985. Cognitive technologies: The design of joint human-machine cognitive systems. AI Magazine 6, 4 (1985), 86\u201386.","journal-title":"AI Magazine"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781420005684"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2004.826272"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3522581","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3522581","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:33Z","timestamp":1750183773000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3522581"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,2]]},"references-count":72,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3522581"],"URL":"https:\/\/doi.org\/10.1145\/3522581","relation":{},"ISSN":["2573-9522","2573-9522"],"issn-type":[{"value":"2573-9522","type":"print"},{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,2]]},"assertion":[{"value":"2021-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}