{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T04:10:53Z","timestamp":1775880653343,"version":"3.50.1"},"reference-count":165,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,5,31]],"date-time":"2020-05-31T00:00:00Z","timestamp":1590883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2020,12,31]]},"abstract":"<jats:p>We present a comprehensive framework and test methodology for the evaluation of human-machine interfaces (HMI) and human-robot interactions (HRI) in collaborative manufacturing applications. An overview of the challenges that face current- and next-generation collaborative robot systems is presented, specifically focused on the interactions between man and machine, and a series of objectively quantitative and subjectively qualitative metrics are given to guide the development and assessment of interfaces and interactions. A generalized set of guidelines for the design of HMI is also proposed to address these challenges and thereby enable effective and intuitive diagnostics and error corrections when process failures occur. These guidelines are aimed at aiding researchers in developing effective interface and interaction technologies, maximizing operator situation awareness in human-robot collaborative manufacturing teams, promoting effective process and system diagnostics reporting, and enabling faster responses to equipment or application errors.<\/jats:p>","DOI":"10.1145\/3385009","type":"journal-article","created":{"date-parts":[[2020,6,1]],"date-time":"2020-06-01T04:28:38Z","timestamp":1590985718000},"page":"1-55","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":67,"title":["Towards Effective Interface Designs for Collaborative HRI in Manufacturing"],"prefix":"10.1145","volume":"9","author":[{"given":"Jeremy A.","family":"Marvel","sequence":"first","affiliation":[{"name":"U.S. National Institute of Standards and Technology, USA"}]},{"given":"Shelly","family":"Bagchi","sequence":"additional","affiliation":[{"name":"U.S. National Institute of Standards and Technology, USA"}]},{"given":"Megan","family":"Zimmerman","sequence":"additional","affiliation":[{"name":"U.S. National Institute of Standards and Technology, USA"}]},{"given":"Brian","family":"Antonishek","sequence":"additional","affiliation":[{"name":"U.S. National Institute of Standards and Technology, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,5,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmathb.2008.08.001"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3229088"},{"key":"e_1_2_1_3_1","unstructured":"ASTM. 2014. ASTM E2919-14 - Standard Test Method for Evaluating the Performance of Systems that Measure Static Six Degrees of Freedom (6DOF) Pose. Standard. ASTM International.  ASTM. 2014. ASTM E2919-14 - Standard Test Method for Evaluating the Performance of Systems that Measure Static Six Degrees of Freedom (6DOF) Pose. Standard. ASTM International."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2010.05.010"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.44.4.462"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-008-0001-3"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541883"},{"key":"e_1_2_1_8_1","unstructured":"C. Batini and M. Scannapieco. 2006. Data Quality Concepts Methodologies and Techniques. Springer.  C. Batini and M. Scannapieco. 2006. Data Quality Concepts Methodologies and Techniques. Springer."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.leaqua.2015.07.007"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/0005-7916(94)90063-9"},{"key":"e_1_2_1_11_1","first-page":"4","article-title":"SUS--A \u201cquick and dirty\u201d usability scale","volume":"189","author":"Brooke John","year":"1996","journal-title":"Usab. Eval. Ind."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ETFA.2001.997725"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008986004181"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE International Conference on Robotics Automation","volume":"4","author":"Caccavale Fabrizio"},{"key":"e_1_2_1_16_1","unstructured":"Stuart K. Card Allen Newell and Thomas P. Moran. 1983. The Psychology of Human-computer Interaction. L. Erlbaum Associates Inc.  Stuart K. Card Allen Newell and Thomas P. Moran. 1983. The Psychology of Human-computer Interaction. L. Erlbaum Associates Inc."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interactions. ACM, 254--262","author":"Carpinella Colleen M."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2003.811794"},{"key":"e_1_2_1_19_1","unstructured":"Samuel G. Charlton. 2002. Measurement of cognitive states in test and evaluation. Handbook Human Factors Testing Eval. (2002) 115--122.  Samuel G. Charlton. 2002. Measurement of cognitive states in test and evaluation. Handbook Human Factors Testing Eval. (2002) 115--122."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815595"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the Comp. Rob. Vision","author":"St-Martin Cormier Olivier","year":"2015"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2005.850587"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Michael Davis. 1984. The mammalian startle. Neural Mechanisms Startle Behav. (1984) 287.  Michael Davis. 1984. The mammalian startle. Neural Mechanisms Startle Behav. (1984) 287.","DOI":"10.1007\/978-1-4899-2286-1_10"},{"key":"e_1_2_1_24_1","first-page":"115","article-title":"A methodology for data quality assessment on financial data","volume":"4","author":"Amicis Fabricio De","year":"2004","journal-title":"Studies Comm. Sci."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/apl0000110"},{"key":"e_1_2_1_26_1","unstructured":"B. Doyle. 2015. North American Robotics Market Sets New Records in 2015. \u201c http:\/\/www.robotics.org\/content-detail.cfm\/Industrial-Robotics-News\/North-American-Robotics-Market-Sets-New-Records-in-2015\/content_id\/5951\u201d.  B. Doyle. 2015. North American Robotics Market Sets New Records in 2015. \u201c http:\/\/www.robotics.org\/content-detail.cfm\/Industrial-Robotics-News\/North-American-Robotics-Market-Sets-New-Records-in-2015\/content_id\/5951\u201d."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the Euro. Conference on Mobile Robotics1--6.","author":"Driewer F."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the IEEE Conference on Systems, Man and Cybernetics. IEEE, 912--918","author":"Drury J. L."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.2514\/atcq.6.1.1"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE International Conference on Robotics Automation: ECHORD Workshop. IEEE.","author":"Edwards S."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/NAECON.1988.195097"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1518\/001872095779049499"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1518\/001872095779049543"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Mica R. Endsley and Daniel J. Garland. 2000. Situation Awareness Analysis and Measurement. CRC Press.  Mica R. Endsley and Daniel J. Garland. 2000. Situation Awareness Analysis and Measurement. CRC Press.","DOI":"10.1201\/b12461"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Mica R. Endsley and Debra G. Jones. 2016. Designing for Situation Awareness: An Approach to User-centered Design. CRC Press.  Mica R. Endsley and Debra G. Jones. 2016. Designing for Situation Awareness: An Approach to User-centered Design. CRC Press.","DOI":"10.1201\/b11371"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1518\/001872095779064555"},{"key":"e_1_2_1_37_1","unstructured":"Larry P. English. 1999. Improving Data Warehouse and Business Information Quality. J. Wiley 8 Sons.  Larry P. English. 1999. Improving Data Warehouse and Business Information Quality. J. Wiley 8 Sons."},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the International Conference on Information Quality (ICIQ\u201902)","author":"Martin"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the International Conference on Safety and Industrial Automated Systems.","author":"Falco Joe","year":"2012"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the International Conference on Database Theory","volume":"3","author":"Falorsi P. D."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACC.2013.6580849"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2015.2430053"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0055392"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2016.10.518"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121251"},{"key":"e_1_2_1_46_1","volume-title":"Robotics Research","author":"Fong Terrence"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CTS.2007.4621745"},{"key":"e_1_2_1_48_1","first-page":"24","article-title":"Study on application of a human-robot collaborative system using hand-guiding in a production line","volume":"49","author":"Fujii Masakazu","year":"2016","journal-title":"IHI Eng. Rev"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.22260\/ISARC2012\/0076"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.2196\/10839"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1049\/me:20020302"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1514095.1514113"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 3942--3948","author":"Goodrich M. A."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1075\/is.8.3.10gro"},{"key":"e_1_2_1_55_1","first-page":"10","article-title":"Virtual small group dynamics: A quantitative experimental framework","volume":"1","author":"Guazzini Andrea","year":"2012","journal-title":"Rev. Psychol. Front"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1525\/aa.1963.65.5.02a00020"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193120605000909"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the Prognostic Health Management Society Conference.","author":"Heddy Gerald","year":"2015"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-010-0068-5"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 11th IEEE International Workshop on Robot and Human Communications. 399--404","author":"Helms E."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1080\/17470215208416600"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2019.2904558"},{"key":"e_1_2_1_64_1","first-page":"10","article-title":"Standardization of Japanese version of FNE and SADS. Koudou Ryouhou Kenkyu (Jap","volume":"18","author":"Ishikawa R.","year":"1992","journal-title":"J. Behav. Ther.)"},{"key":"e_1_2_1_65_1","unstructured":"ISO. 1998. ISO 9283 - Manipulating Industrial Robots - Performance Criteria and Related Test Methods. Standard. International Organization for Standardization.  ISO. 1998. ISO 9283 - Manipulating Industrial Robots - Performance Criteria and Related Test Methods. Standard. International Organization for Standardization."},{"key":"e_1_2_1_66_1","unstructured":"ISO. 2001. ISO 9126-1 - Software Engineering - Product Quality - Part 1: Quality Model. Standard. International Organization for Standardization.  ISO. 2001. ISO 9126-1 - Software Engineering - Product Quality - Part 1: Quality Model. Standard. International Organization for Standardization."},{"key":"e_1_2_1_67_1","unstructured":"ISO. 2010. ISO 9241-210 - Ergonomics of Human-system Interaction - Part 210 - Human-centered Design for Interactive Systems. Standard. International Organization for Standardization.  ISO. 2010. ISO 9241-210 - Ergonomics of Human-system Interaction - Part 210 - Human-centered Design for Interactive Systems. Standard. International Organization for Standardization."},{"key":"e_1_2_1_68_1","unstructured":"ISO. 2011a. 10218-2 - Robots and Robotic Devices - Safety Requirements - Part 2: Industrial Robot Systems and Integration. Standard. International Organization for Standardization.  ISO. 2011a. 10218-2 - Robots and Robotic Devices - Safety Requirements - Part 2: Industrial Robot Systems and Integration. Standard. International Organization for Standardization."},{"key":"e_1_2_1_69_1","unstructured":"ISO. 2011b. ISO 10218-1 - Robots and Robotic Devices - Safety Requirements - Part 1: Robots. Standard. International Organization for Standardization.  ISO. 2011b. ISO 10218-1 - Robots and Robotic Devices - Safety Requirements - Part 1: Robots. Standard. International Organization for Standardization."},{"key":"e_1_2_1_70_1","unstructured":"ISO. 2011c. ISO 25010 - Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - System and Software Quality Models. Standard. International Organization for Standardization.  ISO. 2011c. ISO 25010 - Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - System and Software Quality Models. Standard. International Organization for Standardization."},{"key":"e_1_2_1_71_1","unstructured":"ISO. 2012. ISO 8373 - Robotics and Robotic Devices - Vocabulary. Standard. International Organization for Standardization.  ISO. 2012. ISO 8373 - Robotics and Robotic Devices - Vocabulary. Standard. International Organization for Standardization."},{"key":"e_1_2_1_72_1","unstructured":"ISO. 2016. ISO\/TS 15066 - Robotics and Robotic Devices - Collaborative Robots. Standard. International Organization for Standardization.  ISO. 2016. ISO\/TS 15066 - Robotics and Robotic Devices - Collaborative Robots. Standard. International Organization for Standardization."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3029798.3038426"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-49524-6_28"},{"key":"e_1_2_1_75_1","unstructured":"J. Johnson. 2014. Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines (2nd ed.). Morgan Kaufmann.  J. Johnson. 2014. Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines (2nd ed.). Morgan Kaufmann."},{"key":"e_1_2_1_76_1","volume-title":"Situation Awareness Analysis and Measurement.","author":"Jones Debra G."},{"key":"e_1_2_1_77_1","unstructured":"Debra G. Jones and Mica R. Endsley. 1996. Sources of situation awareness errors in aviation.Aviation Space Environ. Med. 67 6 (1996).  Debra G. Jones and Mica R. Endsley. 1996. Sources of situation awareness errors in aviation.Aviation Space Environ. Med. 67 6 (1996)."},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-013-0191-1"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1080\/1463922021000054335"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcim.2016.05.002"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1115\/IMECE2018-86666"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327108ijap0303_3"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.160007"},{"key":"e_1_2_1_84_1","doi-asserted-by":"crossref","unstructured":"Shirlee-Ann Knight and Janice Burn. 2005. Developing a framework for assessing information quality on the World Wide Web.Informing Sci. 8 (2005).  Shirlee-Ann Knight and Janice Burn. 2005. Developing a framework for assessing information quality on the World Wide Web.Informing Sci. 8 (2005).","DOI":"10.28945\/493"},{"key":"e_1_2_1_85_1","doi-asserted-by":"crossref","unstructured":"Yang W. Lee Diane M. Strong Beverly K. Kahn and Richard Y. Wang. 2002. AIMQ: A methodology for information quality assessment. Info. 8 Manag. 40 2 (2002) 133--146.  Yang W. Lee Diane M. Strong Beverly K. Kahn and Richard Y. Wang. 2002. AIMQ: A methodology for information quality assessment. Info. 8 Manag. 40 2 (2002) 133--146.","DOI":"10.1016\/S0378-7206(02)00043-5"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1136\/qshc.2004.010033"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.egypro.2015.07.474"},{"key":"e_1_2_1_88_1","doi-asserted-by":"crossref","unstructured":"Honghai Liu and George M. Coghill. 2005. A model-based approach to robot fault diagnosis. Knowl.-based Syst. 18 4 (2005) 225--233.  Honghai Liu and George M. Coghill. 2005. A model-based approach to robot fault diagnosis. Knowl.-based Syst. 18 4 (2005) 225--233.","DOI":"10.1016\/j.knosys.2004.10.004"},{"key":"e_1_2_1_89_1","doi-asserted-by":"crossref","unstructured":"David Loshin. 2001. Enterprise Knowledge Management: The Data Quality Approach. Morgan Kaufmann.  David Loshin. 2001. Enterprise Knowledge Management: The Data Quality Approach. Morgan Kaufmann.","DOI":"10.1016\/B978-012455840-3.50003-0"},{"key":"e_1_2_1_90_1","doi-asserted-by":"crossref","unstructured":"Lanssie Mingyue Ma Terrence Fong Mark J. Micire Yun Kyung Kim and Karen Feigh. 2018. Human-robot teaming: Concepts and components for design. In Field and Service Robotics. Springer 649--663.  Lanssie Mingyue Ma Terrence Fong Mark J. Micire Yun Kyung Kim and Karen Feigh. 2018. Human-robot teaming: Concepts and components for design. In Field and Service Robotics. Springer 649--663.","DOI":"10.1007\/978-3-319-67361-5_42"},{"key":"e_1_2_1_91_1","doi-asserted-by":"crossref","unstructured":"Keith R. MacArthur Kimberly Stowers and P. A. Hancock. 2017. Human-robot interaction: Proximity and speed\u2014slowly back away from the robot! In Advances in Human Factors in Robots and Unmanned Systems.Springer 365--374.  Keith R. MacArthur Kimberly Stowers and P. A. Hancock. 2017. Human-robot interaction: Proximity and speed\u2014slowly back away from the robot! In Advances in Human Factors in Robots and Unmanned Systems.Springer 365--374.","DOI":"10.1007\/978-3-319-41959-6_30"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2013.2237904"},{"key":"e_1_2_1_94_1","volume-title":"Collaborative Robotics: A Gateway into Factory Automation.","author":"Marvel Jeremy A.","year":"2014"},{"key":"e_1_2_1_95_1","unstructured":"Jeremy A. Marvel. 2014b. Performance of Collaborative Robot Systems. Retrieved from https:\/\/www.nist.gov\/programs-projects\/performance-collaborative-robot-systems.  Jeremy A. Marvel. 2014b. Performance of Collaborative Robot Systems. Retrieved from https:\/\/www.nist.gov\/programs-projects\/performance-collaborative-robot-systems."},{"key":"e_1_2_1_96_1","doi-asserted-by":"crossref","unstructured":"Jeremy A. Marvel Roger Bostelman and Joe Falco. 2018. Multi-robot assembly strategies and metrics. ASM Comput. Surv. 51 1 (2018) 14:1--14:32.  Jeremy A. Marvel Roger Bostelman and Joe Falco. 2018. Multi-robot assembly strategies and metrics. ASM Comput. Surv. 51 1 (2018) 14:1--14:32.","DOI":"10.1145\/3150225"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2014.2337275"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcim.2016.08.001"},{"key":"e_1_2_1_99_1","volume-title":"Proceedings of the 41st International Symposium on Robotics. VDE, 1--6.","author":"Matthias Bj\u00f6rn","year":"2014"},{"key":"e_1_2_1_100_1","volume-title":"Proc. Ann. Conf. Int. Comm. Assoc. ERIC.","author":"McCroskey James C."},{"key":"e_1_2_1_101_1","unstructured":"Everett N. McKay. 2013. UI is Communication: How to Design Intuitive User Centered Interfaces by Focusing on Effective Communication. Newnes.  Everett N. McKay. 2013. UI is Communication: How to Design Intuitive User Centered Interfaces by Focusing on Effective Communication. Newnes."},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mechatronics.2018.08.006"},{"key":"e_1_2_1_103_1","volume-title":"Proceedings of the 16th International Conference on Intelligent User Interfaces. ACM, 145--154","author":"Micire Mark"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1080\/10580539608906992"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139408964957"},{"key":"e_1_2_1_106_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 437--441","author":"Murphy R. R."},{"key":"e_1_2_1_107_1","volume-title":"Proceedings of the IEEE International Workshop on Robot and Human Communications. IEEE, 301--306","author":"Murphy R. R."},{"key":"e_1_2_1_108_1","unstructured":"National Instruments. 2016. LabVIEW for Industrial Robotics. Retrieved from http:\/\/www.ni.com\/robotics\/industrial\/.  National Instruments. 2016. LabVIEW for Industrial Robotics. Retrieved from http:\/\/www.ni.com\/robotics\/industrial\/."},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25554-5_48"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1075\/is.7.3.14nom"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/223355.223665"},{"key":"e_1_2_1_112_1","volume-title":"Proceedings of the NIST Performance Metrics of Intelligent Systems Workshop. NIST.","author":"Olsen D. R."},{"key":"e_1_2_1_113_1","unstructured":"M. Orcutt. 2014. Robots Rising. Retrieved from https:\/\/www.technologyreview.com\/s\/529971\/robots-rising\/.  M. Orcutt. 2014. Robots Rising. Retrieved from https:\/\/www.technologyreview.com\/s\/529971\/robots-rising\/."},{"key":"e_1_2_1_114_1","first-page":"23","article-title":"Quantitative social group dynamics on a large scale","volume":"5","author":"Palla Gergely","year":"2007","journal-title":"Communities"},{"key":"e_1_2_1_115_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interactions. ACM, 41--48","author":"Perrin X."},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1109\/HUMANOIDS.2016.7803320"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1012551108"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1145\/505248.506010"},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1108\/IR-01-2016-0037"},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-016-0285-9"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62387-0"},{"key":"e_1_2_1_122_1","first-page":"247","article-title":"Delays and interruptions: A self-perpetuating paradox of communication technology use. Info","volume":"15","author":"Rennecker Julie","year":"2005","journal-title":"Organiz."},{"key":"e_1_2_1_123_1","volume-title":"Proceedings of the IEEE International Workshop on Robot and Human Communications. IEEE, 635--642","author":"Riddle D. R."},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/2163.2164"},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022279618991"},{"key":"e_1_2_1_126_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1464-0597.2004.00161.x"},{"key":"e_1_2_1_127_1","volume-title":"Robust Intelligence and Trust in Autonomous Systems","author":"Sadrfaridpour Behzad"},{"key":"e_1_2_1_128_1","volume-title":"Proceedings of the Defense and Security Conference. 96--104","author":"Salerno John J."},{"key":"e_1_2_1_129_1","first-page":"551","article-title":"The DaQuinCIS architecture: A platform for exchanging and improving data quality in cooperative information systems. Info","volume":"29","author":"Scannapieco Monica","year":"2004","journal-title":"Syst."},{"key":"e_1_2_1_130_1","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation. IEEE, 2327--2332","author":"Scholtz J."},{"key":"e_1_2_1_131_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-016-0334-3"},{"key":"e_1_2_1_132_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720809350882"},{"key":"e_1_2_1_133_1","doi-asserted-by":"publisher","DOI":"10.1145\/2393091.2393111"},{"key":"e_1_2_1_134_1","doi-asserted-by":"publisher","DOI":"10.4992\/jjpsy.87.14075"},{"key":"e_1_2_1_135_1","unstructured":"B. Shneiderman. 1998. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley.  B. Shneiderman. 1998. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley."},{"key":"e_1_2_1_136_1","doi-asserted-by":"crossref","unstructured":"Michael O. Shneier Elena R. Messina Craig I. Schlenoff Frederick M. Proctor Thomas R. Kramer and Joseph A. Falco. 2015. NISTIR 8090: Measuring and Representing the Performance of Manufacturing Assembly Robots. Technical Report. National Institute of Standards and Technology.  Michael O. Shneier Elena R. Messina Craig I. Schlenoff Frederick M. Proctor Thomas R. Kramer and Joseph A. Falco. 2015. NISTIR 8090: Measuring and Representing the Performance of Manufacturing Assembly Robots. Technical Report. National Institute of Standards and Technology.","DOI":"10.6028\/NIST.IR.8090"},{"key":"e_1_2_1_137_1","doi-asserted-by":"publisher","DOI":"10.1068\/p281059"},{"key":"e_1_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(97)01080-2"},{"key":"e_1_2_1_139_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0075814"},{"key":"e_1_2_1_140_1","doi-asserted-by":"crossref","unstructured":"David A. Sousa. 2006. How the Brain Learns. SAGE Publications Ltd.  David A. Sousa. 2006. How the Brain Learns. SAGE Publications Ltd.","DOI":"10.1002\/ace.213"},{"key":"e_1_2_1_141_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0925-7535(01)00010-8"},{"key":"e_1_2_1_142_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121249"},{"key":"e_1_2_1_143_1","volume-title":"Information Quality Management: Theory and Applications","author":"Su Ying"},{"key":"e_1_2_1_144_1","unstructured":"Nassim Nicholas Taleb. 2007. The Black Swan: The Impact of the Highly Improbable. Vol. 2. Random House.  Nassim Nicholas Taleb. 2007. The Black Swan: The Impact of the Highly Improbable. Vol. 2. Random House."},{"key":"e_1_2_1_145_1","volume-title":"Proceedings of the AAAI Spring Symposium: Emotion, Personality, and Social Behavior. 133--140","author":"Tapus Adriana"},{"key":"e_1_2_1_146_1","unstructured":"R. M. Taylor. 1990. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. AGARD Situat. Aware. Aero. Operat. AGARD-CP-478 (1990) 3:1--3:17.  R. M. Taylor. 1990. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. AGARD Situat. Aware. Aero. Operat. AGARD-CP-478 (1990) 3:1--3:17."},{"key":"e_1_2_1_147_1","doi-asserted-by":"crossref","unstructured":"Douglas S. Thomas. 2018. The Costs and Benefits of Advanced Maintenance in Manufacturing. US Department of Commerce National Institute of Standards and Technology.  Douglas S. Thomas. 2018. The Costs and Benefits of Advanced Maintenance in Manufacturing. US Department of Commerce National Institute of Standards and Technology.","DOI":"10.6028\/NIST.AMS.100-18"},{"key":"e_1_2_1_148_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139608964470"},{"key":"e_1_2_1_149_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interactions. 263--264","author":"Ullman Daniel"},{"key":"e_1_2_1_150_1","first-page":"157","article-title":"Computer based assessment systems evaluation via the ISO 9126 quality model","volume":"1","author":"Valenti S.","year":"2002","journal-title":"J. Info. Tech. Educ."},{"key":"e_1_2_1_151_1","volume-title":"Proceedings of the International Symposium on Flexible Automation.","author":"Vijayaraghavan Athulan","year":"2008"},{"key":"e_1_2_1_152_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mechatronics.2018.02.009"},{"key":"e_1_2_1_153_1","first-page":"A13","article-title":"Tools for assessing situational awareness in an operational fighter environment.Aviat., Space","volume":"65","author":"Waag Wayne L.","year":"1994","journal-title":"Envir. MD"},{"key":"e_1_2_1_154_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0029333"},{"key":"e_1_2_1_155_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0029334"},{"key":"e_1_2_1_156_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.JNR.0000387550.50458.bc"},{"key":"e_1_2_1_157_1","doi-asserted-by":"publisher","DOI":"10.1145\/269012.269022"},{"key":"e_1_2_1_158_1","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.1996.11518099"},{"key":"e_1_2_1_159_1","unstructured":"Astrid Weiss and Andreas Huber. 2016. User experience of a smart factory robot: Assembly line workers demand adaptive robots. arXiv preprint arXiv:1606.03846 (2016).  Astrid Weiss and Andreas Huber. 2016. User experience of a smart factory robot: Assembly line workers demand adaptive robots. arXiv preprint arXiv:1606.03846 (2016)."},{"key":"e_1_2_1_162_1","volume-title":"Proceedings of the Performance Metrics of Intelligent Systems Workshop. NIST, 136--145","author":"Brian"},{"key":"e_1_2_1_163_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmsy.2018.03.002"},{"key":"e_1_2_1_164_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interactions. IEEE, 75--84","author":"Welfare Katherine S."},{"key":"e_1_2_1_165_1","unstructured":"C. D. Wickens J. D. Lee Y. Liu and S. Gordon-Becker. 2004. An Introduction to Human Factors Engineering. Pearson Prentice Hall.  C. D. Wickens J. D. Lee Y. Liu and S. Gordon-Becker. 2004. An Introduction to Human Factors Engineering. Pearson Prentice Hall."},{"key":"e_1_2_1_166_1","unstructured":"L. Wood. 2014. Research and Markets: Global Articulated Robots Market Growth of 16.27% CAGR by 2019\u2014Analysis Technologies 8 Forecasts 2016--2019. Retrieved from http:\/\/www.businesswire.com\/news\/home\/20160108005449\/en\/Research-Markets-Global-Articulated-Robots-Market-Growth.  L. Wood. 2014. Research and Markets: Global Articulated Robots Market Growth of 16.27% CAGR by 2019\u2014Analysis Technologies 8 Forecasts 2016--2019. Retrieved from http:\/\/www.businesswire.com\/news\/home\/20160108005449\/en\/Research-Markets-Global-Articulated-Robots-Market-Growth."},{"key":"e_1_2_1_167_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2015.2412256"},{"key":"e_1_2_1_168_1","volume-title":"Proceedings of the Pacific Asia Conference on Information Systems (PACIS\u201918)","author":"Zhao L."},{"key":"e_1_2_1_169_1","volume-title":"Proceedings of the IEEE International Conference on Emerging Technologies in Factory Automation (ETFA\u201918)","volume":"1","author":"Zhu Li"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3385009","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3385009","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:13Z","timestamp":1750200073000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3385009"}},"subtitle":["Metrics and Measures"],"short-title":[],"issued":{"date-parts":[[2020,5,31]]},"references-count":165,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12,31]]}},"alternative-id":["10.1145\/3385009"],"URL":"https:\/\/doi.org\/10.1145\/3385009","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,31]]},"assertion":[{"value":"2018-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}