{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T15:25:45Z","timestamp":1777735545489,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":41,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,9]],"date-time":"2023-10-09T00:00:00Z","timestamp":1696809600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Fundacao para a Ciencia e a Tecnologia","award":["UIDB\/00127\/2020"],"award-info":[{"award-number":["UIDB\/00127\/2020"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,9]]},"DOI":"10.1145\/3577190.3614134","type":"proceedings-article","created":{"date-parts":[[2023,10,7]],"date-time":"2023-10-07T22:30:48Z","timestamp":1696717848000},"page":"234-242","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Evaluating Outside the Box: Lessons Learned on eXtended Reality Multi-modal Experiments Beyond the Laboratory"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4454-710X","authenticated-orcid":false,"given":"Bernardo","family":"Marques","sequence":"first","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9858-8249","authenticated-orcid":false,"given":"Samuel","family":"Silva","sequence":"additional","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4331-4774","authenticated-orcid":false,"given":"Rafael","family":"Maio","sequence":"additional","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3430-5211","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Alves","sequence":"additional","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2799-643X","authenticated-orcid":false,"given":"Carlos","family":"Ferreira","sequence":"additional","affiliation":[{"name":"IEETA, DEGEIT, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3754-2749","authenticated-orcid":false,"given":"Paulo","family":"Dias","sequence":"additional","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2219-2731","authenticated-orcid":false,"given":"Beatriz Sousa","family":"Santos","sequence":"additional","affiliation":[{"name":"IEETA, DETI, LASI, University of Aveiro, Portugal"}]}],"member":"320","published-online":{"date-parts":[[2023,10,9]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3441337"},{"key":"e_1_3_2_1_2_1","volume-title":"International Conference on Human Systems Engineering and Design: Future Trends and Applications. 350\u2013355","author":"Alves Jo\u00e3o","year":"2020","unstructured":"Jo\u00e3o Alves , Bernardo Marques , Paulo Dias , and Beatriz\u00a0Sousa Santos . 2020 . Using augmented reality and step by step verification in industrial quality control . In International Conference on Human Systems Engineering and Design: Future Trends and Applications. 350\u2013355 . Jo\u00e3o Alves, Bernardo Marques, Paulo Dias, and Beatriz\u00a0Sousa Santos. 2020. Using augmented reality and step by step verification in industrial quality control. In International Conference on Human Systems Engineering and Design: Future Trends and Applications. 350\u2013355."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00170-021-07049-8"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2012.07.004"},{"key":"e_1_3_2_1_5_1","volume-title":"Grand challenges for augmented reality. Frontiers in Virtual Reality","author":"Billinghurst Mark","year":"2021","unstructured":"Mark Billinghurst . 2021. Grand challenges for augmented reality. Frontiers in Virtual Reality ( 2021 ), 12. Mark Billinghurst. 2021. Grand challenges for augmented reality. Frontiers in Virtual Reality (2021), 12."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445651"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2018.00037"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-022-00387-1"},{"key":"e_1_3_2_1_9_1","volume-title":"Henry Been-Lirn Duh, and Gregory\u00a0F Welch","author":"Kim Kangsoo","year":"2018","unstructured":"Kangsoo Kim , Mark Billinghurst , Gerd Bruder , Henry Been-Lirn Duh, and Gregory\u00a0F Welch . 2018 . Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008\u20132017). IEEE transactions on visualization and computer graphics 24, 11 (2018), 2947\u20132962. Kangsoo Kim, Mark Billinghurst, Gerd Bruder, Henry Been-Lirn Duh, and Gregory\u00a0F Welch. 2018. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008\u20132017). IEEE transactions on visualization and computer graphics 24, 11 (2018), 2947\u20132962."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-020-00346-8"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-020-00335-x"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3568444.3570591"},{"key":"e_1_3_2_1_13_1","volume-title":"International Conference on Graphics and Interaction, ICGI. 1\u20134.","author":"Maio Rafael","year":"2022","unstructured":"Rafael Maio , Andr\u00e9 Santos , Bernardo Marques , Duarte Almeida , Pedro Ramalho , Joel Batista , Paulo Dias , and Beatriz Sousa-Santos . 2022 . Supporting Human Operators in an Industrial Shop Floor through Pervasive Augmented Reality . In International Conference on Graphics and Interaction, ICGI. 1\u20134. Rafael Maio, Andr\u00e9 Santos, Bernardo Marques, Duarte Almeida, Pedro Ramalho, Joel Batista, Paulo Dias, and Beatriz Sousa-Santos. 2022. Supporting Human Operators in an Industrial Shop Floor through Pervasive Augmented Reality. In International Conference on Graphics and Interaction, ICGI. 1\u20134."},{"key":"e_1_3_2_1_14_1","volume-title":"Pervasive Augmented Reality to support logistics operators in industrial scenarios: a shop floor user study on kit assembly. The International Journal of Advanced Manufacturing Technology","author":"Maio Rafael","year":"2023","unstructured":"Rafael Maio , Andr\u00e9 Santos , Bernardo Marques , Carlos Ferreira , Duarte Almeida , Pedro Ramalho , Joel Batista , Paulo Dias , and Beatriz\u00a0Sousa Santos . 2023. Pervasive Augmented Reality to support logistics operators in industrial scenarios: a shop floor user study on kit assembly. The International Journal of Advanced Manufacturing Technology ( 2023 ), 1\u201319. Rafael Maio, Andr\u00e9 Santos, Bernardo Marques, Carlos Ferreira, Duarte Almeida, Pedro Ramalho, Joel Batista, Paulo Dias, and Beatriz\u00a0Sousa Santos. 2023. Pervasive Augmented Reality to support logistics operators in industrial scenarios: a shop floor user study on kit assembly. The International Journal of Advanced Manufacturing Technology (2023), 1\u201319."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3390\/mti6100092"},{"key":"e_1_3_2_1_16_1","volume-title":"International Conference on Graphics and Interaction, ICGI. 1\u20134.","author":"Marques Bernardo","year":"2021","unstructured":"Bernardo Marques , Carlos Ferreira , Samuel Silva , Andreia Santos , Paulo Dias , and B Sousa\u00a0Santos . 2021 . Exploring Remote Augmented Reality as a 2D Authoring Tool for Creation of Guiding Instructions . In International Conference on Graphics and Interaction, ICGI. 1\u20134. Bernardo Marques, Carlos Ferreira, Samuel Silva, Andreia Santos, Paulo Dias, and B Sousa\u00a0Santos. 2021. Exploring Remote Augmented Reality as a 2D Authoring Tool for Creation of Guiding Instructions. In International Conference on Graphics and Interaction, ICGI. 1\u20134."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12008-021-00798-6"},{"key":"e_1_3_2_1_18_1","volume-title":"IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), IEEE VR Tutorials.","author":"Marques Bernardo","year":"2022","unstructured":"Bernardo Marques , Samuel Silva , Paulo Dias , and Beatriz\u00a0Sousa Santos . 2022 . Remote Collaboration using Augmented Reality: Development and Evaluation . In IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), IEEE VR Tutorials. Bernardo Marques, Samuel Silva, Paulo Dias, and Beatriz\u00a0Sousa Santos. 2022. Remote Collaboration using Augmented Reality: Development and Evaluation. In IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), IEEE VR Tutorials."},{"key":"e_1_3_2_1_19_1","volume-title":"IEEE International Symposium on Mixed and Augmented Reality, ISMAR Tutorials.","author":"Marques Bernardo","year":"2022","unstructured":"Bernardo Marques , Samuel Silva , Paulo Dias , and Beatriz\u00a0Sousa Santos . 2022 . Remote Collaboration using Augmented Reality: Development and Evaluation . In IEEE International Symposium on Mixed and Augmented Reality, ISMAR Tutorials. Bernardo Marques, Samuel Silva, Paulo Dias, and Beatriz\u00a0Sousa Santos. 2022. Remote Collaboration using Augmented Reality: Development and Evaluation. In IEEE International Symposium on Mixed and Augmented Reality, ISMAR Tutorials."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Bernardo Marques Samuel Silva Paulo Dias and Beatriz\u00a0Sousa Santos. 2023. Evaluating Augmented Reality Based Remote Collaboration: A Contextualized Approach. In Human-Automation Interaction. 267\u2013280.  Bernardo Marques Samuel Silva Paulo Dias and Beatriz\u00a0Sousa Santos. 2023. Evaluating Augmented Reality Based Remote Collaboration: A Contextualized Approach. In Human-Automation Interaction. 267\u2013280.","DOI":"10.1007\/978-3-031-10788-7_16"},{"key":"e_1_3_2_1_21_1","volume-title":"Guidelines for Designing Mixed Reality Solutions in Remote Scenarios. In International Conference on Human-Computer Interaction. 247\u2013252","author":"Marques Bernardo","year":"2023","unstructured":"Bernardo Marques , Samuel Silva , Rafael Maio , Paulo Dias , and Beatriz Sousa\u00a0Santos . 2023 . Guidelines for Designing Mixed Reality Solutions in Remote Scenarios. In International Conference on Human-Computer Interaction. 247\u2013252 . Bernardo Marques, Samuel Silva, Rafael Maio, Paulo Dias, and Beatriz Sousa\u00a0Santos. 2023. Guidelines for Designing Mixed Reality Solutions in Remote Scenarios. In International Conference on Human-Computer Interaction. 247\u2013252."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-36004-6_35"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2021.10.009"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR-Adjunct51615.2020.00016"},{"key":"e_1_3_2_1_25_1","volume-title":"A critical analysis on remote collaboration mediated by Augmented Reality: Making a case for improved characterization and evaluation of the collaborative process. Computers & Graphics","author":"Marques Bernardo","year":"2021","unstructured":"Bernardo Marques , Ant\u00f3nio Teixeira , Samuel Silva , Jo\u00e3o Alves , Paulo Dias , and Beatriz\u00a0Sousa Santos . 2021. A critical analysis on remote collaboration mediated by Augmented Reality: Making a case for improved characterization and evaluation of the collaborative process. Computers & Graphics ( 2021 ), 1\u201317. Bernardo Marques, Ant\u00f3nio Teixeira, Samuel Silva, Jo\u00e3o Alves, Paulo Dias, and Beatriz\u00a0Sousa Santos. 2021. A critical analysis on remote collaboration mediated by Augmented Reality: Making a case for improved characterization and evaluation of the collaborative process. Computers & Graphics (2021), 1\u201317."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR50242.2020.00069"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376330"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267851.3267901"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/VRW55335.2022.00209"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445170"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3442472"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3568444.3568462"},{"key":"e_1_3_2_1_33_1","volume-title":"Case Studies: Designing and Evaluating Technologies for Use in the Wild. In Research in the Wild","author":"Rogers Yvonne","year":"2017","unstructured":"Yvonne Rogers and Paul Marshall . 2017 . Case Studies: Designing and Evaluating Technologies for Use in the Wild. In Research in the Wild . Springer , 33\u201367. Yvonne Rogers and Paul Marshall. 2017. Case Studies: Designing and Evaluating Technologies for Use in the Wild. In Research in the Wild. Springer, 33\u201367."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/1473871611413099"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300767"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3389\/fcomp.2022.966319"},{"key":"e_1_3_2_1_37_1","volume-title":"Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments. Frontiers in Virtual Reality 3","author":"Steed Anthony","year":"2022","unstructured":"Anthony Steed , Lisa Izzouzi , Klara Brandst\u00e4tter , Sebastian Friston , Ben Congdon , Otto Olkkonen , Daniele Giunchi , Nels Numan , and David Swapp . 2022 . Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments. Frontiers in Virtual Reality 3 (2022). Anthony Steed, Lisa Izzouzi, Klara Brandst\u00e4tter, Sebastian Friston, Ben Congdon, Otto Olkkonen, Daniele Giunchi, Nels Numan, and David Swapp. 2022. Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments. Frontiers in Virtual Reality 3 (2022)."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406098"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR-Adjunct57072.2022.00190"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2008.4637362"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-51369-6_11"}],"event":{"name":"ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION","location":"Paris France","acronym":"ICMI '23","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3577190.3614134","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3577190.3614134","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:01Z","timestamp":1750178221000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3577190.3614134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,9]]},"references-count":41,"alternative-id":["10.1145\/3577190.3614134","10.1145\/3577190"],"URL":"https:\/\/doi.org\/10.1145\/3577190.3614134","relation":{},"subject":[],"published":{"date-parts":[[2023,10,9]]},"assertion":[{"value":"2023-10-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}