{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T21:40:16Z","timestamp":1785447616446,"version":"3.56.0"},"reference-count":41,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,10,18]],"date-time":"2022-10-18T00:00:00Z","timestamp":1666051200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Estonian Research Council","award":["PSG453"],"award-info":[{"award-number":["PSG453"]}]},{"DOI":"10.13039\/501100002301","name":"EU Regional Development Fund","doi-asserted-by":"publisher","award":["PSG453"],"award-info":[{"award-number":["PSG453"]}],"id":[{"id":"10.13039\/501100002301","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>The adoption of Digital Twin (DT) solutions for industrial purposes is increasing among small- and medium-sized enterprises and is already being integrated into many large-scale companies. As there is an increasing need for faster production and shortening of the learning curve for new emerging technologies, Virtual Reality (VR) interfaces for enterprise manufacturing DTs seem to be a good solution. Furthermore, with the emergence of Industry 5.0 (I5.0) paradigm, human operators will be increasingly integrated in the systems interfaces though advanced interactions, pervasive sensors, real time tracking and data acquisition. This scenario is especially relevant in collaborative automated systems where the introduction of immersive VR interfaces based on production cell DTs might provide a solution for the integration of the human factors in the modern industrial scenarios. This study presents experimental results of the comparison between users controlling a physical industrial robot system via a traditional teach pendant and a DT leveraging a VR user interface. The study group involves forty subjects including experts in robotics and VR as well as non-experts. An analysis of the data gathered in both the real and the virtual use case scenario is provided. The collected information includes time for performing a task with an industrial robot, stress level evaluation, physical and mental effort, and the human subjects\u2019 perceptions of the physical and simulated robots. Additionally, operator gazes were tracked in the VR environment. In this study, VR interfaces in the DT representation are exploited to gather user centered metrics and validate efficiency and safety standards for modern collaborative industrial systems in I5.0. The goal is to evaluate how the operators perceive and respond to the virtual robot and user interface while interacting with them and detect if any degradation of user experience and task efficiency exists compared to the real robot interfaces. Results demonstrate that the use of DT VR interfaces is comparable to traditional tech pendants for the given task and might be a valuable substitute of physical interfaces. Despite improving the overall task performance and considering the higher stress levels detected while using the DT VR interface, further studies are necessary to provide a clearer validation of both interfaces and user impact assessment methods.<\/jats:p>","DOI":"10.3390\/robotics11050113","type":"journal-article","created":{"date-parts":[[2022,10,19]],"date-time":"2022-10-19T00:58:51Z","timestamp":1666141131000},"page":"113","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Digital Twin as Industrial Robots Manipulation Validation Tool"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8134-8636","authenticated-orcid":false,"given":"Vladimir","family":"Kuts","sequence":"first","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, Tallinn University of Technology, 19086 Tallinn, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy A.","family":"Marvel","sequence":"additional","affiliation":[{"name":"National Institute of Standards and Technology, Gaithersburg, MD 20899, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Murat","family":"Aksu","sequence":"additional","affiliation":[{"name":"National Institute of Standards and Technology, Gaithersburg, MD 20899, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9692-2058","authenticated-orcid":false,"given":"Simone L.","family":"Pizzagalli","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, Tallinn University of Technology, 19086 Tallinn, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8330-1989","authenticated-orcid":false,"given":"Martin\u0161","family":"Sarkans","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, Tallinn University of Technology, 19086 Tallinn, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yevhen","family":"Bondarenko","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, Tallinn University of Technology, 19086 Tallinn, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2349-0340","authenticated-orcid":false,"given":"Tauno","family":"Otto","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, Tallinn University of Technology, 19086 Tallinn, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.jmsy.2020.06.017","article-title":"Review of digital twin about concepts, technologies, and industrial applications","volume":"58","author":"Liu","year":"2020","journal-title":"J. Manuf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.mechatronics.2018.02.009","article-title":"Survey on human\u2013robot collaboration in industrial settings: Safety, intuitive interfaces and applications","volume":"55","author":"Villani","year":"2018","journal-title":"Mechatronics"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s43154-020-00005-6","article-title":"Review of Interfaces for Industrial Human-Robot Interaction","volume":"1","author":"Berg","year":"2020","journal-title":"Curr. Robot. Rep."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Krupke, D., Steinicke, F., Lubos, P., Jonetzko, Y., G\u00f6rner, M., and Zhang, J. (2018, January 1\u20135). Comparison of multimodal heading and pointing gestures for co-located mixed reality human\u2013robot interaction. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594043"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research","volume":"52","author":"Hart","year":"1988","journal-title":"Adv. Psychol."},{"key":"ref_6","first-page":"4","article-title":"SUS-A \u201cquick and dirty\u201d usability scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Ind."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hassenzahl, M., Burmester, M., and Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualit\u00e4t. Mensch & Computer 2003, Springer.","DOI":"10.1007\/978-3-322-80058-9_19"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Whitney, D., Rosen, E., Phillips, E., Konidaris, G., and Tellex, S. (2020). Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality. Robotics Research, Springer.","DOI":"10.1007\/978-3-030-28619-4_28"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tadeja, S.K., Lu, Y., Seshadri, P., and Kristensson, P.O. (2020, January 7\u201314). Digital Twin Assessments in Virtual Reality: An Explorational Study with Aeroengines. Proceedings of the 2020 IEEE Aerospace Conference, Missoula, MT, USA.","DOI":"10.1109\/AERO47225.2020.9172389"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1207\/s15327108ijap0303_3","article-title":"Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness","volume":"3","author":"Kennedy","year":"1993","journal-title":"Int. J. Aviat. Psychol."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Rheinberg, F., Engeser, S., and Vollmeyer, R. (2002, January 3\u20136). Measuring components of flow: The Flow-Short-Scale. Proceedings of the 1st International Positive Psychology Summit, Washington, DC, USA.","DOI":"10.1037\/t47787-000"},{"key":"ref_12","first-page":"115","article-title":"Igroup presence questionnaire","volume":"41","author":"Schubert","year":"2001","journal-title":"Teleoperators Virtual Environ."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"20325","DOI":"10.1109\/ACCESS.2019.2897018","article-title":"Prototyping a Digital Twin for Real Time Remote Control Over Mobile Networks: Application of Remote Surgery","volume":"7","author":"Laaki","year":"2019","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Concannon, D., Flynn, R., and Murray, N. (2019, January 18). A Quality of Experience Evaluation System and Research Challenges for Networked Virtual Reality-Based Teleoperation Applications. Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems; Association for Computing Machinery, MMVE\u201919, New York, NY, USA.","DOI":"10.1145\/3304113.3326119"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3903","DOI":"10.1007\/s00170-017-0428-5","article-title":"Effectiveness and acceptability of a virtual environment for assessing human\u2013robot collaboration in manufacturing","volume":"92","author":"Matsas","year":"2017","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.rcim.2018.07.006","article-title":"The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans","volume":"55","author":"Oyekan","year":"2019","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.promfg.2020.04.034","article-title":"Concept of a Mixed-Reality Learning Environment for Collaborative Robotics","volume":"45","author":"Sievers","year":"2020","journal-title":"Procedia Manuf."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yap, H.J., Taha, Z., Dawal, S.Z.M., and Chang, S.W. (2014). Virtual reality based support system for layout planning and programming of an industrial robotic work cell. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0109692"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1080\/0951192X.2019.1690685","article-title":"Virtual reality in manufacturing: Immersive and collaborative artificial-reality in design of human\u2013robot workspace","volume":"33","author":"Malik","year":"2020","journal-title":"Int. J. Comput. Integr. Manuf."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"P\u00e9rez, L., Rodr\u00edguez-Jim\u00e9nez, S., Rodr\u00edguez, N., Usamentiaga, R., and Garc\u00eda, D.F. (2020). Digital Twin and Virtual Reality Based Methodology for Multi-Robot Manufacturing Cell Commissioning. Appl. Sci., 10.","DOI":"10.3390\/app10103633"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"105600","DOI":"10.1016\/j.cie.2018.12.047","article-title":"Exploring the potential of Operator 4.0 interface and monitoring","volume":"139","author":"Peruzzini","year":"2020","journal-title":"Comput. Ind. Eng."},{"key":"ref_22","unstructured":"Karwowski, W. (2006). The OCRA Method: Assessment of Exposure to Occupational Repetitive Actions of the Upper Limbs. International Encyclopedia of Ergonomics and Human Factors-3 Volume Set, CRC Press."},{"key":"ref_23","first-page":"472","article-title":"Digital twin and virtual reality: A co-simulation environment for design and assessment of industrial workstations","volume":"7","author":"Havard","year":"2019","journal-title":"Prod. Manuf. Res."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"McAtamney, L., and Corlett, N. (2004). Rapid upper limb assessment (RULA). Handbook of Human Factors and Ergonomics Methods, CRC Press.","DOI":"10.1201\/9780203489925.ch7"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Rosen, E., Whitney, D., Phillips, E., Chien, G., Tompkin, J., Konidaris, G., and Tellex, S. (2020). Communicating robot arm motion intent through mixed reality head-mounted displays. Robotics Research, Springer.","DOI":"10.1007\/978-3-030-28619-4_26"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Moniri, M.M., Valcarcel, F.A.E., Merkel, D., and Sonntag, D. (2016, January 14\u201316). Human gaze and focus-of-attention in dual reality human\u2013robot collaboration. Proceedings of the 2016 12th International Conference on Intelligent Environments (IE), London, UK.","DOI":"10.1109\/IE.2016.54"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.procir.2018.01.036","article-title":"Intuitive robot programming through environment perception, augmented reality simulation and automated program verification","volume":"76","author":"Wassermann","year":"2018","journal-title":"Procedia CIRP"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1016\/j.promfg.2020.01.066","article-title":"Exploration of two safety strategies in human\u2013robot collaborative manufacturing using Virtual Reality","volume":"38","author":"Vosniakos","year":"2019","journal-title":"Procedia Manuf."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1697","DOI":"10.1016\/j.promfg.2020.01.109","article-title":"Virtual reality assisted robot programming for human collaboration","volume":"38","author":"Maragkos","year":"2019","journal-title":"Procedia Manuf."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1007\/s12008-018-0516-2","article-title":"Off-line programming of an industrial robot in a virtual reality environment","volume":"13","author":"Manou","year":"2019","journal-title":"Int. J. Interact. Des. Manuf."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.compind.2019.05.001","article-title":"Industrial robot control and operator training using virtual reality interfaces","volume":"109","author":"Diez","year":"2019","journal-title":"Comput. Ind."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.ifacol.2018.11.517","article-title":"Interactive Robot Programming Using Mixed Reality","volume":"51","author":"Ostanin","year":"2018","journal-title":"IFAC-PapersOnLine"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1016\/j.ergon.2016.02.004","article-title":"Development and evaluation of a virtual training environment for on-line robot programming","volume":"53","author":"Nathanael","year":"2016","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"101820","DOI":"10.1016\/j.rcim.2019.101820","article-title":"Augmented reality-assisted robot programming system for industrial applications","volume":"61","author":"Ong","year":"2020","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Burghardt, A., Szybicki, D., Gierlak, P., Kurc, K., Pietru\u015b, P., and Cygan, R. (2020). Programming of Industrial Robots Using Virtual Reality and Digital Twins. Appl. Sci., 10.","DOI":"10.3390\/app10020486"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"128","DOI":"10.5604\/01.3001.0013.0464","article-title":"Digital twin based synchronised control and simulation of the industrial robotic cell using virtual reality","volume":"19","author":"Kuts","year":"2019","journal-title":"J. Mach. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3385009","article-title":"Towards Effective Interface Designs for Collaborative HRI in Manufacturing: Metrics and Measures","volume":"9","author":"Marvel","year":"2020","journal-title":"ACM Trans. Hum.-Robot. Interact."},{"key":"ref_38","unstructured":"Bartneck, C., Kulic, D., and Croft, E.A. (2008, January 12\u201315). Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. Proceedings of the Metrics for Human-Robot Interaction, ACM\/IEEE International Conference on Human-Robot Interaction, Amsterdam, The Netherlands."},{"key":"ref_39","unstructured":"(2011). Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-System and Software Quality Models (Standard No. ISO 25010)."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"364","DOI":"10.3176\/proc.2019.4.03","article-title":"Synchronizing physical factory and its digital twin throughan iiot middleware: A case study","volume":"68","author":"Kuts","year":"2019","journal-title":"Proc. Est. Acad. Sci."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"53","DOI":"10.36897\/jme\/120182","article-title":"Digital Twin: Industrial robot kinematic model integration to the virtual reality environment","volume":"20","author":"Kuts","year":"2020","journal-title":"J. Mach. Eng."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/11\/5\/113\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:56:14Z","timestamp":1760144174000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/11\/5\/113"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,18]]},"references-count":41,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["robotics11050113"],"URL":"https:\/\/doi.org\/10.3390\/robotics11050113","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,18]]}}}