{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:37:02Z","timestamp":1760240222981,"version":"build-2065373602"},"reference-count":37,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2019,4,11]],"date-time":"2019-04-11T00:00:00Z","timestamp":1554940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>The number of scientific publications combining robotic user interfaces and mixed reality highly increased during the 21st Century. Counting the number of yearly added publications containing the keywords \u201cmixed reality\u201d and \u201crobot\u201d listed on Google Scholar indicates exponential growth. The interdisciplinary nature of mixed reality robotic user interfaces (MRRUI) makes them very interesting and powerful, but also very challenging to design and analyze. Many single aspects have already been successfully provided with theoretical structure, but to the best of our knowledge, there is no contribution combining everything into an MRRUI taxonomy. In this article, we present the results of an extensive investigation of relevant aspects from prominent classifications and taxonomies in the scientific literature. During a card sorting experiment with professionals from the field of human\u2013computer interaction, these aspects were clustered into named groups for providing a new structure. Further categorization of these groups into four different categories was obvious and revealed a memorable structure. Thus, this article provides a framework of objective, technical factors, which finds its application in a precise description of MRRUIs. An example shows the effective use of the proposed framework for precise system description, therefore contributing to a better understanding, design, and comparison of MRRUIs in this growing field of research.<\/jats:p>","DOI":"10.3390\/mti3020025","type":"journal-article","created":{"date-parts":[[2019,4,12]],"date-time":"2019-04-12T03:46:37Z","timestamp":1555040797000},"page":"25","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["IMPAct: A Holistic Framework for Mixed Reality Robotic User Interface Classification and Design"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6746-1133","authenticated-orcid":false,"given":"Dennis","family":"Krupke","sequence":"first","affiliation":[{"name":"Human-Computer Interaction (HCI), Department Informatics, University of Hamburg, 22527 Hamburg, Germany"},{"name":"Technical Aspects of Multimodal Systems (TAMS), Department Informatics, University of Hamburg, 22527 Hamburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jianwei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Technical Aspects of Multimodal Systems (TAMS), Department Informatics, University of Hamburg, 22527 Hamburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9879-7414","authenticated-orcid":false,"given":"Frank","family":"Steinicke","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction (HCI), Department Informatics, University of Hamburg, 22527 Hamburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,4,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1109\/70.768174","article-title":"Invited review: The synergy between virtual reality and robotics","volume":"15","author":"Burdea","year":"1999","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_2","unstructured":"De Barros, P.G., and Lindeman, R.W. (2009). A Survey of User Interfaces for Robot Teleoperation, Computer Science Department, Worcester Polytechnic Institute."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1109\/TSMCA.2005.850587","article-title":"Validating human\u2013robot interaction schemes in multitasking environments","volume":"35","author":"Crandall","year":"2005","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Hum."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1162\/pres.1992.1.2.229","article-title":"Synthetic Experience: A Proposed Taxonomy","volume":"1","author":"Robinett","year":"1992","journal-title":"Presence Teleoper. Virtual Environ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1162\/pres.1992.1.1.127","article-title":"Autonomy, Interaction, and Presence","volume":"1","author":"Zeltzer","year":"1992","journal-title":"Presence Teleoper. Virtual Environ."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1162\/pres.1997.6.6.603","article-title":"A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments","volume":"6","author":"Slater","year":"1997","journal-title":"Presence Teleoper. Virtual Environ."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1117\/12.197321","article-title":"Augmented reality: A class of displays on the reality-virtuality continuum","volume":"2351","author":"Milgram","year":"1995","journal-title":"Proc. SPIE"},{"key":"ref_8","first-page":"1321","article-title":"A Taxonomy of Mixed Reality Visual Displays","volume":"77","author":"Milgram","year":"1994","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_9","first-page":"1","article-title":"A taxonomy of real and virtual world display integration","volume":"Volume 1","author":"Milgram","year":"1999","journal-title":"Mixed Reality: Merging Real and Virtual Worlds"},{"key":"ref_10","unstructured":"Bowman, D., Kruijff, E., LaViola, J.J., and Poupyrev, I.P. (2004). 3D User Interfaces: Theory and Practice, CourseSmart eTextbook, Addison-Wesley."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sturman, D.J., Zeltzer, D., and Pieper, S. (1989, January 13\u201315). Hands-on Interaction with Virtual Environments. Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology, Williamsburg, VA, USA.","DOI":"10.1145\/73660.73663"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhai, S., and Milgram, P. (1994, January 24\u201328). Input Techniques for HCI in 3D Environments. Proceedings of the Conference Companion on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/259963.260059"},{"key":"ref_13","unstructured":"Lindeman, R.W. (1999). Bimanual Interaction, Passive-Haptic Feedback, 3D Widget Representation, and Simulated Surface Constraints for Interaction in Immersive Virtual Environments. [Ph.D. Thesis, George Washington University]."},{"key":"ref_14","unstructured":"Niku, S.B. (2011). Introduction to Robotics: Analysis, Control, Applications, Wiley India."},{"key":"ref_15","unstructured":"Mckerrow, P. (1991). Introduction to Robotics, Addison-Wesley Longman Publishing Co., Inc."},{"key":"ref_16","unstructured":"Murphy, R.R., and Arkin, R.C. (2000). Introduction to AI Robotics, MIT Press."},{"key":"ref_17","unstructured":"Bartneck, C., and Okada, M. (2001, January 5\u201310). Robotic user interfaces. Proceedings of the Human and Computer Conference, Aizu, Japan."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Sheridan, T.B., and Verplank, W.L. (1978). Human and Computer Control of Undersea Teleoperators, Technical Report.","DOI":"10.21236\/ADA057655"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1109\/3468.844354","article-title":"A model for types and levels of human interaction with automation","volume":"30","author":"Parasuraman","year":"2000","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Hum."},{"key":"ref_20","unstructured":"Crandall, J.W., and Goodrich, M.A. (2001, January 7\u201310). Experiments in adjustable autonomy. Proceedings of the 2001 IEEE International Conference on Systems, Man and Cybernetics, e-Systems and e-Man for Cybernetics in Cyberspace (Cat. No. 01CH37236), Tucson, AZ, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1177\/154193120304700138","article-title":"Beyond levels of automation: An architecture for more flexible human-automation collaboration","volume":"47","author":"Miller","year":"2003","journal-title":"Proc. Hum. Fact. Ergon. Soc. Annu. Meet."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"74","DOI":"10.5898\/JHRI.3.2.Beer","article-title":"Toward a Framework for Levels of Robot Autonomy in Human-robot Interaction","volume":"3","author":"Beer","year":"2014","journal-title":"J. Hum. Robot Interact."},{"key":"ref_23","unstructured":"Schilling, M., Kopp, S., Wachsmuth, S., Wrede, B., Ritter, H., Brox, T., Nebel, B., and Burgard, W. (2016, January 17\u201319). Towards a multidimensional perspective on shared autonomy. Proceedings of the 2016 AAAI Fall Symposium Series, Arlington, VA, USA."},{"key":"ref_24","unstructured":"Gunderson, J., and Gunderson, L. (2004, January 24\u201326). Intelligence = autonomy = capability. Proceedings of the Performance Metrics for Intelligent Systems, PERMIS, National Inst of Standards and Technology, Gaithersburg, MD, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bainbridge, L. (1983). Ironies of automation. Analysis, Design and Evaluation of Man\u2013Machine Systems, Elsevier.","DOI":"10.1016\/B978-0-08-029348-6.50026-9"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1007\/s101110200022","article-title":"MABA-MABA or Abracadabra? Progress on Human\u2013Automation Co-ordination","volume":"4","author":"Dekker","year":"2002","journal-title":"Cogn. Technol. Work"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Calefato, C., Montanari, R., and Tesauri, F. (2008). The adaptive automation design. Human Computer Interaction: New Developments, InTech.","DOI":"10.5772\/5878"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Scholtz, J. (2003, January 6\u20139). Theory and evaluation of human robot interactions. Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Big Island, HI, USA.","DOI":"10.1109\/HICSS.2003.1174284"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1109\/THMS.2014.2371048","article-title":"Usability Guidelines for the Design of Robot Teleoperation: A Taxonomy","volume":"45","author":"Adamides","year":"2015","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Krupke, D., Steinicke, F., Lubos, P., Jonetzko, Y., G\u00f6rner, M., and Zhang, J. (2018, January 1\u20135). Comparison of Multimodal Heading and Pointing Gestures for Co-Located Mixed Reality Human-Robot Interaction. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594043"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/5254.796083","article-title":"Mixed-initiative interaction","volume":"14","author":"Allen","year":"1999","journal-title":"IEEE Intell. Syst. Their Appl."},{"key":"ref_32","unstructured":"Craig, J.J. (2005). Introduction to robotics: mechanics and control, Pearson\/Prentice Hall."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Scholtz, J., Young, J., Drury, J.L., and Yanco, H.A. (May, January 26). Evaluation of human\u2013robot interaction awareness in search and rescue. Proceedings of the 2004 IEEE International Conference on Robotics and Automation, New Orleans, LA, USA.","DOI":"10.1109\/ROBOT.2004.1307409"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Keyes, B., Micire, M., Drury, J.L., and Yanco, H.A. (2010). Improving human\u2013robot interaction through interface evolution. Human-Robot Interaction, IntechOpen.","DOI":"10.5772\/8140"},{"key":"ref_35","unstructured":"Clarkson, E., and Arkin, R.C. (2007, January 7\u20139). Applying Heuristic Evaluation to Human-Robot Interaction Systems. Proceedings of the Flairs Conference, Key West, FL, USA."},{"key":"ref_36","unstructured":"Elara, M.R., Acosta Calderon, C., Zhou, C., Yue, P.K., and Hu, L. (December, January 29). Using heuristic evaluation for human-humanoid robot interaction in the soccer robotics domain. Proceedings of the IEEE-RAS International Conference on Humanoid Robots (Humanoids 2007), Pittsburgh, PA, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Jacko, J.A. (2012). Human Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, CRC Press.","DOI":"10.1201\/b11963"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/3\/2\/25\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:44:38Z","timestamp":1760186678000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/3\/2\/25"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,11]]},"references-count":37,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2019,6]]}},"alternative-id":["mti3020025"],"URL":"https:\/\/doi.org\/10.3390\/mti3020025","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2019,4,11]]}}}