{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,17]],"date-time":"2025-12-17T08:55:03Z","timestamp":1765961703959,"version":"3.41.2"},"reference-count":78,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T00:00:00Z","timestamp":1693440000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Humans and robots will increasingly have to work together in the new industrial context. Therefore, it is necessary to improve the User Experience, Technology Acceptance, and overall wellbeing to achieve a smoother and more satisfying interaction while obtaining the maximum performance possible out of it. For this reason, it is essential to analyze these interactions to enhance User Experience. The heuristic evaluation is an easy-to-use, low-cost method that can be applied at different stages of a design process in an iterative manner. Despite these advantages, there is rarely a list of heuristics in the current literature that evaluates Human-Robot interactions both from a User Experience, Technology Acceptance, and Human-Centered approach. Such an approach should integrate key aspects like safety, trust, and perceived safety, ergonomics and workload, inclusivity, and multimodality, as well as robot characteristics and functionalities. Therefore, a new set of heuristics, namely, the HEUROBOX tool, is presented in this work in the form of the HEUROBOX tool to help practitioners and researchers in the assessment of human-robot systems in industrial environments. The HEUROBOX tool clusters design guidelines and methodologies as a logic list of heuristics for human-robot interaction and comprises four categories: Safety, Ergonomics, Functionality, and Interfaces. They include 84 heuristics in the basic evaluation, while the advanced evaluation lists a total of 228 heuristics in order to adapt the tool to the evaluation of different industrial requirements. Finally, the set of new heuristics has been validated by experts using the System Usability Scale (SUS) questionnaire and the categories has been prioritized in order of their importance in the evaluation of Human-Robot Interaction through the Analytic Hierarchy Process (AHP).<\/jats:p>","DOI":"10.3389\/frobt.2023.1227082","type":"journal-article","created":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T13:48:02Z","timestamp":1693489682000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Development of a new set of Heuristics for the evaluation of Human-Robot Interaction in industrial settings: Heuristics Robots Experience (HEUROBOX)"],"prefix":"10.3389","volume":"10","author":[{"given":"Ainhoa","family":"Apraiz","sequence":"first","affiliation":[]},{"given":"Jose Antonio","family":"Mulet Alberola","sequence":"additional","affiliation":[]},{"given":"Ganix","family":"Lasa","sequence":"additional","affiliation":[]},{"given":"Maitane","family":"Mazmela","sequence":"additional","affiliation":[]},{"given":"Hien Ngoc","family":"Nguyen","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,8,31]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1109\/thms.2014.2371048","article-title":"Usability guidelines for the design of robot teleoperation: A taxonomy","volume":"45","author":"Adamides","year":"2014","journal-title":"IEEE Trans. Human-Machine Syst."},{"key":"B2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.7160\/aol.2020.120301","article-title":"Heuristic evaluation of the user interface for a semi-autonomous agricultural robot sprayer","volume":"12","author":"Adamides","year":"2020","journal-title":"AGRIS On-Line Pap. Econ. Inf."},{"key":"B3","first-page":"380","article-title":"Towards the development of a haptics guideline in human-robot systems","author":"Andonovski","year":"2010"},{"key":"B4","article-title":"Evaluaci\u00f3n heur\u00edstica de la interacci\u00f3n persona-robot en entornos industriales","author":"Apraiz","year":"2022","journal-title":"Interacci\u00f3n"},{"key":"B5","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1007\/s12369-022-00957-z","article-title":"Evaluation of user experience in human\u2013robot interaction: A systematic literature review","volume":"15","author":"Apraiz","year":"","journal-title":"Int. J. Soc. Robotics"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.17632\/T2NWPWWSMG.1","article-title":"Experts\u2019 evaluation: datasets of rating questionnaires for the heurobox heuristics to evaluate industrial collaborative robotic environments","volume":"1","author":"Apraiz","year":"","journal-title":"Mendeley Data"},{"volume-title":"Demox: Nuevo modelo de trabajo de dise\u00f1o y evaluaci\u00f3n de la UX en interfases industriales [Tesis doctoral, Mondragon Unibertsitatea]","year":"2020","author":"Aranburu Zabalo","key":"B7"},{"key":"B8","first-page":"114","article-title":"Determining what individual SUS scores mean: adding an adjective rating scale","volume":"4","author":"Bangor","year":"2009","journal-title":"J. Usability Stud."},{"key":"B9","first-page":"24","article-title":"Heuristic rules for human-robot interaction based on principles from linguistics-asking for directions","author":"Bauer","year":"2009","journal-title":"AISB Convention-Symposium New Front. Human-Robot Interact."},{"article-title":"How the inclusion of design principles contribute to the development of social robots","year":"2018","author":"Benitez Sandoval","key":"B10"},{"key":"B11","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1080\/09540091.2016.1271400","article-title":"Principles of robotics: regulating robots in the real world","volume":"29","author":"Boden","year":"2017","journal-title":"Connect. Sci."},{"key":"B12","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1007\/s12369-019-00593-0","article-title":"Human\u2013robot collaboration acceptance model: development and comparison for germany, japan, china and the usa","volume":"11","author":"Br\u00f6hl","year":"2019","journal-title":"Int. J. Soc. Robotics"},{"key":"B13","first-page":"4","article-title":"SUS-A quick and dirty usability scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Industry"},{"key":"B14","doi-asserted-by":"publisher","first-page":"100979","DOI":"10.1016\/J.AEI.2019.100979","article-title":"A user-centric smart product-service system development approach: A case study on medication management for the elderly","volume":"42","author":"Chang","year":"2019","journal-title":"Adv. Eng. Inf."},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359616","article-title":"Trust-aware decision making for human-robot collaboration: model learning and planning","volume":"9","author":"Chen","year":"2020","journal-title":"ACM Trans. Human-Robot Interact. (THRI)"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1016\/0270-0255(87)90473-8","article-title":"Analytic Hierarchy process for survey data [R package ahpsurvey version 0.4.1]","volume":"9","author":"Cho","year":"2019","journal-title":"Math. Model."},{"key":"B17","first-page":"44","article-title":"Applying heuristic evaluation to human-robot interaction systems","author":"Clarkson","year":"2007"},{"key":"B18","doi-asserted-by":"publisher","first-page":"3048","DOI":"10.3390\/APP10093048","article-title":"Towards an ergonomic assessment framework for industrial assembly workstations\u2014a case study","volume":"10","author":"Colim","year":"2020","journal-title":"Appl. Sci."},{"key":"B19","doi-asserted-by":"publisher","first-page":"1931","DOI":"10.3390\/SU13041931","article-title":"Lean manufacturing and ergonomics integration: defining productivity and wellbeing indicators in a human\u2013robot workstation","volume":"13","author":"Colim","year":"2021","journal-title":"Sustainability"},{"key":"B20","first-page":"2761","article-title":"Gesture-based robot control: design challenges and evaluation with humans","author":"Coronado","year":"2017"},{"key":"B21","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","article-title":"Perceived usefulness, perceived ease of use, and user acceptance of information technology","volume":"13","author":"Davis","year":"1989","journal-title":"MIS Q. Manag. Inf. Syst."},{"key":"B22","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1006\/imms.1993.1022","article-title":"User acceptance of information technology: system characteristics, user perceptions and behavioral impacts","volume":"38","author":"Davis","year":"1993","journal-title":"Int. J. Man-Machine Stud."},{"key":"B23","first-page":"1540","article-title":"Design guidelines for improved human-robot interaction","author":"Drury","year":"2004"},{"volume-title":"Industry 5.0 - towards a sustainable, human-centric and resilient European industry","year":"2021","key":"B24"},{"key":"B25","first-page":"407","article-title":"Design guidelines for collaborative industrial robot user interfaces","author":"Frijns","year":"2021"},{"key":"B26","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1007\/978-981-16-0041-8_32","article-title":"Development and evaluation of usability heuristics for voice user interfaces","volume":"221","author":"Fulfagar","year":"2021","journal-title":"Int. Conf. Res. into Des. Smart Innovation, Syst. Technol."},{"key":"B27","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1016\/J.AEI.2015.04.008","article-title":"Supporting engineering design communication using a custom-built social media tool \u2013 PartBook","volume":"29","author":"Gopsill","year":"2015","journal-title":"Adv. Eng. Inf."},{"key":"B28","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1016\/J.PROCIR.2020.02.212","article-title":"From design for assembly to design for collaborative assembly - product design principles for enhancing safety, ergonomics and efficiency in human-robot collaboration","volume":"91","author":"Gualtieri","year":"","journal-title":"Procedia CIRP"},{"key":"B29","doi-asserted-by":"publisher","first-page":"3606","DOI":"10.3390\/SU12093606","article-title":"Design of human-centered collaborative assembly workstations for the improvement of operators\u2019 physical ergonomics and production efficiency: A case study","volume":"12","author":"Gualtieri","year":"","journal-title":"Sustainability"},{"key":"B30","doi-asserted-by":"publisher","first-page":"107801","DOI":"10.1016\/j.cie.2021.107801","article-title":"Development and validation of guidelines for safety in human-robot collaborative assembly systems","volume":"163","author":"Gualtieri","year":"2022","journal-title":"Comput. Industrial Eng."},{"key":"B31","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1016\/J.PROCIR.2020.02.188","article-title":"Safety, ergonomics and efficiency in human-robot collaborative assembly: design guidelines and requirements","volume":"91","author":"Gualtieri","year":"","journal-title":"Procedia CIRP"},{"key":"B32","first-page":"6167","article-title":"A framework for end-user instruction of a robot assistant for manufacturing","author":"Guerin","year":"2015"},{"key":"B33","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1177\/0018720811417254","article-title":"A meta-analysis of factors affecting trust in human-robot interaction","volume":"53","author":"Hancock","year":"2011","journal-title":"Hum. Factors"},{"key":"B34","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1108\/14635780510616025","article-title":"The importance of property-specific attributes in assessing CBD office building quality","volume":"23","author":"Ho","year":"2005","journal-title":"J. Prop. Invest. Finance"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2012.661087","article-title":"Definition and domains of ergonomics","year":"2019","journal-title":"Ergonomics"},{"volume-title":"ISO 10218: Robots and robotic devices: Safety requirements for industrial robots","year":"2011","key":"B36"},{"article-title":"Ergonomics of human-system interaction \u2014 Part 210: human-centred design for interactive systems","volume-title":"International standard","year":"2019","key":"B37"},{"key":"B38","first-page":"51","article-title":"Evaluating a methodology to establish usability heuristics","author":"Jim\u00e9nez","year":"2012"},{"key":"B39","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1075\/is.8.3.04kah","article-title":"What is a Human?","volume":"8","author":"Kahn","year":"2007","journal-title":"Interact. Stud."},{"key":"B40","first-page":"147","volume-title":"Global trend of implementation of industrial robots relating to industry 4.0. Design, Simulation, Manufacturing: The Innovation Exchange","author":"Karabegovi\u0107","year":"2020"},{"key":"B41","first-page":"3","article-title":"Structural network for the implementation of \u201cindustry 4.0\u201d in production processes","volume":"7","author":"Karabegovi\u0107","year":"2022","journal-title":"Ind. 4.0"},{"key":"B42","first-page":"169","article-title":"Applying team heuristics to future human-robot systems","author":"Keebler","year":"2012"},{"key":"B43","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1007\/978-3-319-45117-6_4","article-title":"Safety requirements in collaborative human\u2013robot cyber-physical system","author":"Khalid","year":"2017","journal-title":"Lect. Notes Logist."},{"key":"B44","doi-asserted-by":"crossref","DOI":"10.1145\/2660398.2663777","article-title":"Evaluating mobile Remote presence (MRP) robots","author":"Lewis","year":"2014"},{"key":"B45","doi-asserted-by":"publisher","first-page":"4284","DOI":"10.3390\/s20154284","article-title":"The ANEMONE: theoretical foundations for ux evaluation of action and intention recognition in human-robot interaction","volume":"20","author":"Lindblom","year":"2020","journal-title":"Sensors"},{"key":"B46","first-page":"212","article-title":"Development of a heuristic evaluation tool for voice user interfaces","author":"Maguire","year":"2019"},{"key":"B47","first-page":"17","article-title":"Heuristic evaluation of swarm metrics\u2019 effectiveness","author":"Manning","year":"2015"},{"key":"B48","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1016\/J.PROCS.2015.09.273","article-title":"European Portuguese validation of the system usability scale (SUS)","volume":"67","author":"Martins","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"B49","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3385009","article-title":"Towards effective interface designs for collaborative HRI in manufacturing: metrics and measures","volume":"9","author":"Marvel","year":"2020","journal-title":"ACM Trans. Human-Robot Interact. (THRI)"},{"key":"B50","doi-asserted-by":"publisher","first-page":"100","DOI":"10.3390\/ROBOTICS8040100","article-title":"Human\u2013robot collaboration in manufacturing applications: A review","volume":"8","author":"Matheson","year":"2019","journal-title":"Robotics"},{"key":"B51","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1016\/J.ERGON.2016.11.011","article-title":"Human-oriented design of collaborative robots","volume":"57","author":"Maurice","year":"2017","journal-title":"Int. J. Industrial Ergonomics"},{"volume-title":"Nuevo modelo para evaluar el nivel de aceptaci\u00f3n de las tecnolog\u00edas desde la perspectiva de la interacci\u00f3n y percepci\u00f3n de usuario [Mondragon Goi Eskola Politeknikoa]","year":"2020","author":"Mazmela","key":"B52"},{"key":"B53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3399433","article-title":"Friend or foe? Understanding assembly workers\u2019 acceptance of human-robot collaboration","volume":"10","author":"Meissner","year":"2020","journal-title":"ACM Trans. Human-Robot Interact. (THRI)"},{"key":"B54","doi-asserted-by":"publisher","first-page":"338","DOI":"10.1145\/77481.77486","article-title":"Improving a human-computer dialogue","volume":"33","author":"Molich","year":"1990","journal-title":"Commun. ACM"},{"key":"B55","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1007\/978-3-030-05321-5_11","article-title":"User interfaces for human-robot interaction in field robotics","volume":"128","author":"Murphy","year":"2019","journal-title":"Springer Tracts Adv. Robotics"},{"key":"B56","doi-asserted-by":"publisher","first-page":"101720","DOI":"10.1016\/j.aei.2022.101720","article-title":"Human-centered design for advanced services: A multidimensional design methodology","volume":"53","author":"Nguyen","year":"2022","journal-title":"Adv. Eng. Inf."},{"key":"B57","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/s10845-021-01796-x","article-title":"Human-centred design in industry 4.0: case study review and opportunities for future research","volume":"33","author":"Nguyen Ngoc","year":"2022","journal-title":"J. Intelligent Manuf."},{"key":"B58","first-page":"249","article-title":"Heuristic evaluation of user interfaces","author":"Nielsen","year":"1990"},{"volume-title":"Emotional design: Why we love (or hate) everyday things","year":"2004","author":"Norman","key":"B59"},{"key":"B60","doi-asserted-by":"crossref","DOI":"10.3233\/ATDE220627","volume-title":"A systematic literature review of user experience evaluation scales for human-robot collaboration","author":"Prati","year":"2022"},{"key":"B61","doi-asserted-by":"publisher","first-page":"102072","DOI":"10.1016\/J.RCIM.2020.102072","article-title":"How to include user eXperience in the design of human-robot interaction","volume":"68","author":"Prati","year":"2021","journal-title":"Robotics Computer-Integrated Manuf."},{"volume-title":"Service innovation for smart industry: Human\u2013robot collaboration","year":"2015","author":"Probst","key":"B62"},{"key":"B63","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/J.JELEKIN.2003.09.015","article-title":"Work-related musculoskeletal disorders: the epidemiologic evidence and the debate","volume":"14","author":"Punnett","year":"2004","journal-title":"J. Electromyogr. Kinesiol."},{"key":"B64","doi-asserted-by":"publisher","first-page":"561","DOI":"10.3390\/ELECTRONICS10050561","article-title":"A proposal of accessibility guidelines for human-robot interaction","volume":"10","author":"Qbilat","year":"2021","journal-title":"Electronics"},{"key":"B65","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/J.CSI.2017.03.009","article-title":"How to develop usability heuristics: A systematic literature review","volume":"53","author":"Qui\u00f1ones","year":"2017","journal-title":"Comput. Stand. Interfaces"},{"key":"B66","first-page":"432","article-title":"Diffusion of innovations","volume-title":"An integrated approach to communication theory and research","author":"Rogers","year":"2014"},{"volume-title":"Handbook of usability testing: How to plan, design and conduct effective tests","year":"2008","author":"Rubin","key":"B67"},{"key":"B68","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1007\/S11518-006-0158-Y","article-title":"Fundamentals of the analytic network process \u2014 dependence and feedback in decision-making with a single network","volume":"13","author":"Saaty","year":"2004","journal-title":"J. Syst. Sci. Syst. Eng."},{"key":"B69","doi-asserted-by":"publisher","first-page":"83","DOI":"10.3233\/OER-150226","article-title":"Human centered assistance applications for the working environment of the future","volume":"12","author":"Schmidtler","year":"2015","journal-title":"Occup. Ergon."},{"key":"B70","first-page":"193","article-title":"Developing heuristics for assistive robotics","author":"Tsui","year":"2010"},{"key":"B71","article-title":"Towards practical user experience evaluation methods","author":"V\u00e4\u00e4n\u00e4nen-Vainio-Mattila","year":"2008","journal-title":"Meaningful Meas. Valid. Useful User Exp. Meas. (VUUM)"},{"key":"B72","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1016\/J.MECHATRONICS.2018.02.009","article-title":"Survey on human\u2013robot collaboration in industrial settings: safety, intuitive interfaces and applications","volume":"55","author":"Villani","year":"2018","journal-title":"Mechatronics"},{"key":"B73","first-page":"1","article-title":"Towards modern inclusive factories: A methodology for the development of smart adaptive human-machine interfaces","author":"Villani","year":"2017"},{"key":"B74","first-page":"1","article-title":"A methodological adaptation for heuristic evaluation of HRI","author":"Weiss","year":"2010"},{"key":"B75","first-page":"75","article-title":"Consider the human work experience when integrating robotics in the workplace","author":"Welfare","year":"2019"},{"key":"B76","first-page":"454","article-title":"Heuristic evaluation and user testing with ISO 9126 in evaluating of decision support system for recommendation of outstanding marketing officer","author":"Wibowo","year":"2017"},{"key":"B77","doi-asserted-by":"publisher","first-page":"101570","DOI":"10.1016\/J.AEI.2022.101570","article-title":"Research on visual representation of icon colour in eye-controlled systems","volume":"52","author":"Ya-feng","year":"2022","journal-title":"Adv. Eng. Inf."},{"key":"B78","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1007\/S12369-010-0081-8","article-title":"Evaluating human-robot interaction: focusing on the holistic interaction experience","volume":"3","author":"Young","year":"2011","journal-title":"Int. J. Soc. Robotics"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1227082\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T13:48:12Z","timestamp":1693489692000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1227082\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,31]]},"references-count":78,"alternative-id":["10.3389\/frobt.2023.1227082"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1227082","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2023,8,31]]},"article-number":"1227082"}}