{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,26]],"date-time":"2025-10-26T15:08:14Z","timestamp":1761491294161,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,10,11]],"date-time":"2021-10-11T00:00:00Z","timestamp":1633910400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>(1) COVID-19 pandemic restrictions caused a dramatic shift in research activities, forcing the adoption of remote practices and methods. Despite the known benefits of remote testing, there is limited knowledge on how to prepare and conduct such studies in the industrial context where the target users are experts and company employees. (2) In this article, we detail how we organized VR user tests in five industrial cases during the pandemic, focusing on practicalities and procedures. We cover both on-site testing, including disinfecting and other safety protocols, as well as remote and hybrid setups where both remote and on-site participants were involved. Subject matter experts from eight countries were involved in a total of 22 tests. (3) We share insights for VR user test arrangements relevant to the pandemic, remote and hybrid setups, and an industrial context, among others. (4) Our work confirms that with careful planning it is possible to organize user tests remotely. There are also some limitations in remote user testing, such as reduced visibility and interaction with participants. Most importantly, we list practical recommendations for organizing hybrid user tests with safety and disinfecting procedures for on-site VR use.<\/jats:p>","DOI":"10.3390\/mti5100062","type":"journal-article","created":{"date-parts":[[2021,10,11]],"date-time":"2021-10-11T21:45:32Z","timestamp":1633988732000},"page":"62","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["There is Always a Way: Organizing VR User Tests with Remote and Hybrid Setups during a Pandemic\u2014Learnings from Five Case Studies"],"prefix":"10.3390","volume":"5","author":[{"given":"Sanni","family":"Siltanen","sequence":"first","affiliation":[{"name":"KONE Corporation, Myllykatu 3, 05830 Hyvink\u00e4\u00e4, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6672-2757","authenticated-orcid":false,"given":"Hanna","family":"Heinonen","sequence":"additional","affiliation":[{"name":"KONE Corporation, Myllykatu 3, 05830 Hyvink\u00e4\u00e4, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3799-3662","authenticated-orcid":false,"given":"Alisa","family":"Burova","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Communication Sciences, Tampere University, Kalevantie 4, 33014 Tampere, Finland"}]},{"given":"Paulina Becerril","family":"Palma","sequence":"additional","affiliation":[{"name":"KONE Corporation, Myllykatu 3, 05830 Hyvink\u00e4\u00e4, Finland"},{"name":"Faculty of Information Technology and Communication Sciences, Tampere University, Kalevantie 4, 33014 Tampere, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1966-3889","authenticated-orcid":false,"given":"Phong","family":"Truong","sequence":"additional","affiliation":[{"name":"KONE Corporation, Myllykatu 3, 05830 Hyvink\u00e4\u00e4, Finland"}]},{"given":"Viveka","family":"Opas","sequence":"additional","affiliation":[{"name":"KONE Corporation, Myllykatu 3, 05830 Hyvink\u00e4\u00e4, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7395-0769","authenticated-orcid":false,"given":"Markku","family":"Turunen","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Communication Sciences, Tampere University, Kalevantie 4, 33014 Tampere, Finland"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Fernandes, N. (2021, May 02). Economic Effects of Coronavirus Outbreak (COVID-19) on the World Economy. Available online: https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3557504.","DOI":"10.2139\/ssrn.3557504"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/s10640-020-00426-z","article-title":"The Environmental Impacts of the Coronavirus","volume":"76","author":"Helm","year":"2020","journal-title":"Environ. Resour. Econ."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ratcliffe, J., Soave, F., Hoover, M., Ortega, F.R., Bryan-Kinns, N., Tokarchuk, L., and Farkhatdinov, I. (2021, January 8). Remote XR Studies: Exploring Three Key Challenges of Remote XR Experimentation. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA \u201921), Association for Computing Machinery, New York, NY, USA.","DOI":"10.1145\/3411763.3442472"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1016\/j.dsx.2020.05.011","article-title":"Significant Applications of Virtual Reality for COVID-19 Pandemic","volume":"14","author":"Singh","year":"2020","journal-title":"Diabetes Metab. Syndr. Clin. Res. Rev."},{"key":"ref_5","unstructured":"User Research Community (2021, May 02). Doing User Research during Coronavirus (COVID-19): Choosing Face to Face or Remote Research, Available online: https:\/\/www.gov.uk\/service-manual\/user-research\/doing-user-research-during-coronavirus-covid-19-choosing-face-to-face-or-remote-research."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.chb.2018.04.019","article-title":"The State of Immersive Technology Research: A Literature Analysis","volume":"86","author":"Suh","year":"2018","journal-title":"Comput. Hum. Behav."},{"key":"ref_7","first-page":"1","article-title":"Extended Reality (XR) Remote Research: A Survey of Drawbacks and Opportunities","volume":"Volume 527","author":"Ratcliffe","year":"2021","journal-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"},{"key":"ref_8","first-page":"23","article-title":"Virtual and Augmented Reality in the Digitalo Building Project","volume":"14","author":"Woodward","year":"2007","journal-title":"Int. J. Des. Sci. Technol."},{"key":"ref_9","unstructured":"Gu, N., Watanabe, S., Erhan, H., and Haeusler, M.H. (2014, January 14\u201316). Immersive Virtual Environments: Experiments on Impacting Design and Human Building Interaction. Proceedings of the 19th International Conference on Computer Aided Architectural Design Research in Asia, Kyoto, Japan."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1186\/s40327-018-0065-6","article-title":"Virtual Reality-Integrated Workflow in BIM-enabled Projects Collaboration and Design Review: A Case Study","volume":"6","author":"Zaker","year":"2018","journal-title":"Vis. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1016\/j.procir.2019.04.016","article-title":"Collaborative Virtual Reality Decision Tool for Planning Industrial Shop Floor Layouts","volume":"81","author":"Havard","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_12","first-page":"1","article-title":"Learning Success in Immersive Virtual Reality Training Environments: Practical Evidence from Automotive Assembly","volume":"Volume 50","author":"Schwarz","year":"2020","journal-title":"Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (NordiCHI \u201920)"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1016\/j.jmsy.2020.07.007","article-title":"Applications of Virtual Reality in Maintenance During the Industrial Product Lifecycle: A Systematic Review","volume":"56","author":"Guo","year":"2020","journal-title":"J. Manuf. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Schina, L., Lazoi, M., Lombardo, R., and Corallo, A. (2016). Virtual Reality for Product Development in Manufacturing Industries. Augmented Reality, Virtual Reality, and Computer Graphics, AVR 2016, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-40621-3_15"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"011001","DOI":"10.1115\/1.4034267","article-title":"An Industry Case Study: Investigating Early Design Decision Making in Virtual Reality","volume":"17","author":"Berg","year":"2017","journal-title":"ASME J. Comput. Inf. Sci. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Maher, M.L. (2011). Designers and Collaborative Virtual Environments. Collaborative Design in Virtual Environments, Springer Netherlands.","DOI":"10.1007\/978-94-007-0605-7_1"},{"key":"ref_17","unstructured":"He, Z. (2021). Enhancing Collaboration and Productivity for Virtual and Augmented Reality. [Ph.D. Thesis, New York University]."},{"key":"ref_18","unstructured":"(2021, August 15). KONE Corporation. Available online: https:\/\/www.kone.com\/en\/."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Radecki, J., and Schonfeld, R.C. (2021, May 02). The Impacts of COVID-19 on the Research Enterprise: A Landscape Review. Available online: https:\/\/sr.ithaka.org\/wp-content\/uploads\/2020\/10\/SR-Report-Impacts-of-COVID-19-on-the-Research-Enterprise102620-1.pdf.","DOI":"10.18665\/sr.314247"},{"key":"ref_20","unstructured":"Vitae 2021 Careers Research and Advisory Centre (CRAC) (2021, May 02). The Impact of the COVID-19 Pandemic on Researchers in Universities and Research Institutes. Available online: https:\/\/www.vitae.ac.uk\/impact-and-evaluation\/covid-19-impact-on-researchers."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1038\/d41586-020-02548-2","article-title":"Pandemic Darkens Postdocs\u2019 Work and Career Hopes","volume":"585","author":"Woolston","year":"2020","journal-title":"Nature"},{"key":"ref_22","unstructured":"Rex Hartson, H., Castillo, J.C., Kelso, J., and Neale, W.C. (2006, January 22\u201327). Remote Evaluation: The Network as an Extension of the Usability Laboratory. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Castillo, J.C., Hartson, H.R., and Hix, D. (1998). Remote Usability Evaluation: Can Users Report Their Own Critical Incidents?. CHI 98 Conference Summary on Human Factors in Computing Systems (CHI \u201998), Association for Computing Machinery.","DOI":"10.1145\/286498.286736"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/971258.971264","article-title":"Remote Possibilities? International Usability Testing at a Distance","volume":"11","author":"Dray","year":"2004","journal-title":"Interactions"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Bruun, A., Gull, P., Hofmeister, L., and Stage, J. (2009, January 4\u20139). Let Your Users Do the Testing: A Comparison of Three Remote Asynchronous Usability Testing Methods. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/1518701.1518948"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Radiah, R., M\u00e4kel\u00e4, V., Prange, S., Rodriguez, S.D., Piening, R., Zhou, Y., K\u00f6hle, K., Pfeuffer, K., Abdelrahman, Y., and Hoppe, M. (2021). Remote VR Studies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs. arXiv.","DOI":"10.1145\/3472617"},{"key":"ref_27","unstructured":"Steed, A., Archer, D., Congdon, B., Friston, S., Swapp, D., and Thiel, F.J. (2021). Some Lessons Learned Running Virtual Reality Experiments Out of the Laboratory. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Burova, A., Heinonen, H., Becerril Palma, P., Keskinen, T., Hakulinen, J., Opas, V., M\u00e4kel\u00e4, J., Ronkainen, K., Siltanen, S., and Raisamo, R. (2021). Towards Efficient Academia-Industry Collaboration Methodology: A Case Study of Joint Development of VR System. Acad. Mindtrek, in print.","DOI":"10.1145\/3464327.3464367"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1207\/S15327590IJHC143&4_12","article-title":"Remote Usability Evaluation: Overview and Case Studies","volume":"14","author":"McFadden","year":"2002","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_30","unstructured":"Andreasen, M.S., Nielsen, H.V., Schr\u00f8der, S.O., and Stage, J. (May, January 28). What Happened to Remote Usability Testing? An Empirical Study of Three Methods. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201907), San Jose, CA, USA."},{"key":"ref_31","unstructured":"Madathil, K.C., and Greenstein, J.S. (2011, January 7\u201312). Synchronous Remote Usability Testing: A New Approach Facilitated by Virtual Worlds. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201911), Vancouver, BC, Canada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/j.apergo.2017.02.011","article-title":"An Investigation of the Efficacy of Collaborative Virtual Reality Systems for Moderated Remote Usability Testing","volume":"65","author":"Madathil","year":"2017","journal-title":"Appl. Ergon."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.jmir.2011.12.002","article-title":"Conducting a Research Interview","volume":"43","author":"Bolderston","year":"2012","journal-title":"J. Med. Imaging Radiat. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1037\/0021-9010.88.5.944","article-title":"Applicant Reactions to Face-to-Face and Technology-Mediated Interviews: A Field Investigation","volume":"88","author":"Chapman","year":"2003","journal-title":"J. Appl. Psychol."},{"key":"ref_35","first-page":"7","article-title":"Qualitative Interviews with Irregular Migrants in Times of COVID-19: Recourse to Remote Interview Techniques as a Possible Methodological Adjustment","volume":"22","author":"Gruber","year":"2021","journal-title":"Forum Qual. Soz. Forum Qual. Soc. Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1080\/13645579.2016.1269505","article-title":"Using Internet Video Calls in Qualitative (Longitudinal) Interviews: Some Implications for Rapport","volume":"20","author":"Weller","year":"2017","journal-title":"Int. J. Soc. Res. Methodol."},{"key":"ref_37","first-page":"1","article-title":"Cultural Differences in Smartphone User Experience Evaluation","volume":"Volume 24","author":"Walsh","year":"2010","journal-title":"Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia (MUM \u201910)"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Gruenstein, A., McGraw, I., and Badr, I. (2008, January 20\u201322). The WAMI Toolkit for Developing, Deploying, and Evaluating Web-Accessible Multimodal Interfaces. Proceedings of the 10th International Conference on Multimodal Interfaces (ICMI \u201908), Chania Crete, Greece.","DOI":"10.1145\/1452392.1452420"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3214280","article-title":"Remotion: A Motion-Based Capture and Replay Platform of Mobile Device Interaction for Remote Usability Testing","volume":"2","author":"Qian","year":"2018","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_40","unstructured":"Fidas, C., Katsanos, C., Papachristos, E., Tselios, N., and Avouris, N. (2007, January 18\u201320). Remote Usability Evaluation Methods and Tools: A Survey. Proceedings of the 11th Panhellenic Conference on Informatics, Patras, Greece."},{"key":"ref_41","unstructured":"Kittur, A., Chi, E.H., and Suh, B. (2021, May 02). \u201cCrowdsourcing for Usability: Using Micro-Task Markets for Rapid Remote and Low-Cost User Measurements\u201d, Conference on Human Factors in Computing Systems. Available online: https:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.532.2255&rep=rep1&type=pdf."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Nebeling, M., Speicher, M., and Norrie, M.C. (2013, January 24\u201327). Crowd Study: General Toolkit for Crowdsourced Evaluation of Web Interfaces. Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS \u201913), London, UK.","DOI":"10.1145\/2494603.2480303"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Henze, N., and Boll, S. (2010, January 7\u201310). Push the Study to the App Store: Evaluating off-Screen Visualizations for Maps in the Android Market. Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI \u201910), Lisbon, Portugal.","DOI":"10.1145\/1851600.1851671"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1145\/3406098","article-title":"Evaluating immersive experiences during COVID-19 and beyond","volume":"27","author":"Steed","year":"2020","journal-title":"Interactions"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Garcia Estrada, J., and Prasolova-F\u00f8rland, E. (2021). Running an XR Lab in the Context of COVID-19 pandemic: Lessons Learned From a Norwegian University. Educ. Inf. Technol., 1\u201317.","DOI":"10.1007\/s10639-021-10446-x"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bhatia, S., Bj\u00f6rling, E.A., and Budhiraja, T. (2021, January 8\u201311). Exploring Web-Based VR for Participatory Robot Design. Proceedings of the Companion of the 2021 ACM\/IEEE International Conference on Human-Robot Interaction (HRI \u201921 Companion), Boulder, CO, USA.","DOI":"10.1145\/3434074.3447139"},{"key":"ref_47","unstructured":"Mathis, F., Zhang, X., O\u2019Hagan, J., Medeiros, D., Saeghe, P., McGill, M., Brewster, S., and Khamis, M. (2021, January 8\u201313). Remote XR Studies: The Golden Future of HCI Research?. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201921), Yokohama, Japan."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"De Ponti, R., Marazzato, J., Maresca, A.M., Rovera, F., Carcano, G., and Ferrario, M.M. (2020). Pre-graduation Medical Training Including Virtual Reality during COVID-19 Pandemic: A Report on Students\u2019 Perception. BMC Med. Educ., 20.","DOI":"10.1186\/s12909-020-02245-8"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Albastaki, A., Hoggenm\u00fcller, M., Robinson, F.A., and Hespanhol, L. (2020, January 2\u20134). Augmenting Remote Interviews through Virtual Experience Prototypes. Proceedings of the 32nd Australian Conference on Human-Computer Interaction (OzCHI \u201920), Sydney, Australia.","DOI":"10.1145\/3441000.3441057"},{"key":"ref_50","unstructured":"(2021, August 15). Human Optimized XR Project. Available online: https:\/\/humanoptimizedxr.org\/wp\/."},{"key":"ref_51","unstructured":"(2021, August 15). KEKO Smart Ecosystem Project. Available online: https:\/\/kekoecosystem.com\/."},{"key":"ref_52","unstructured":"Gabriel, A. (2020). Evaluating the Utility of Multi-User VR in Product Development. [Master\u2019s Thesis, Aalto University]. Available online: http:\/\/urn.fi\/URN:NBN:fi:aalto-202008234991."},{"key":"ref_53","unstructured":"(2021, May 02). Available online: https:\/\/3dtalo.fi\/designspace\/."},{"key":"ref_54","unstructured":"(2021, May 02). Available online: https:\/\/www.glue.work\/."},{"key":"ref_55","unstructured":"Ries, E. (2011). The Lean Startup: How Today\u2019s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses, Crown Business."},{"key":"ref_56","unstructured":"THL (2021, May 02). Transmission and Protection\u2014Coronavirus. Available online: https:\/\/thl.fi\/en\/web\/infectious-diseases-and-vaccinations\/what-s-new\/coronavirus-covid-19-latest-updates\/transmission-and-protection-coronavirus\/."},{"key":"ref_57","first-page":"22","article-title":"Remote Observation Strategies for Usability Testing","volume":"22","author":"Thompson","year":"2003","journal-title":"Inf. Technol. Libr."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/10\/62\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:11:32Z","timestamp":1760166692000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/5\/10\/62"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,11]]},"references-count":57,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["mti5100062"],"URL":"https:\/\/doi.org\/10.3390\/mti5100062","relation":{},"ISSN":["2414-4088"],"issn-type":[{"type":"electronic","value":"2414-4088"}],"subject":[],"published":{"date-parts":[[2021,10,11]]}}}