{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,19]],"date-time":"2025-11-19T07:11:22Z","timestamp":1763536282241,"version":"3.44.0"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T00:00:00Z","timestamp":1724284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001381","name":"National Research Foundation, Singapore","doi-asserted-by":"crossref","award":["AISG Award No: AISG2-RP-2020-016"],"award-info":[{"award-number":["AISG Award No: AISG2-RP-2020-016"]}],"id":[{"id":"10.13039\/501100001381","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CityU Start-up Grant","award":["9610677"],"award-info":[{"award-number":["9610677"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,8,22]]},"abstract":"<jats:p>Pilot studies in HCI research serve as a cost-effective approach to validate potential ideas and identify impactful findings before extensive studies. Yet, the additional requirements of AR\/MR, such as multi-view observations and increased multitasking, make it challenging to conduct pilot studies effectively, hindering innovations in this field. Based on interviews with 12 AR\/MR researchers, we identified the key challenges associated with conducting AR\/MR pilot studies with Optical See-Through Head-Mounted Displays (OST-HMDs, OHMDs), including the inability to observe and record in-context user interactions, increased task load, and difficulties with in-context data analysis and discussion. To tackle these challenges, we introduce PilotAR, a desktop-based tool designed iteratively to enhance OHMD-based AR\/MR pilot studies. PilotAR facilitates data collection via live first-person and third-person views, multi-modal annotations, and flexible wizarding interfaces. It also accommodates multi-experimenter settings, streamlines the study process with configurable workflows and shortcuts, records annotated data, and eases results sharing. Formative testing, conducted using three case studies, has highlighted the significant benefits of PilotAR, as well as its potential for further development and refinement.<\/jats:p>","DOI":"10.1145\/3678576","type":"journal-article","created":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T14:36:21Z","timestamp":1725892581000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["PilotAR: Streamlining Pilot Studies with OHMDs from Concept to Insight"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2983-6808","authenticated-orcid":false,"given":"Nuwan","family":"Janaka","sequence":"first","affiliation":[{"name":"Synteraction Lab, Smart Systems Institute, National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0974-3751","authenticated-orcid":false,"given":"Runze","family":"Cai","sequence":"additional","affiliation":[{"name":"Synteraction Lab, School of Computing, National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1430-8770","authenticated-orcid":false,"given":"Ashwin","family":"Ram","sequence":"additional","affiliation":[{"name":"Synteraction Lab, School of Computing, National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0353-3534","authenticated-orcid":false,"given":"Lin","family":"Zhu","sequence":"additional","affiliation":[{"name":"Academy of Arts &amp; Design, Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7971-3107","authenticated-orcid":false,"given":"Shengdong","family":"Zhao","sequence":"additional","affiliation":[{"name":"Synteraction Lab, School of Creative Media &amp; Department of Computer Science, City University of Hong Kong, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6607-9917","authenticated-orcid":false,"given":"Kai Qi","family":"Yong","sequence":"additional","affiliation":[{"name":"Synteraction Lab, School of Computing, National University of Singapore, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1155\/2015\/271231"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376722"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","unstructured":"Ronald T. Azuma. [n. d.]. The road to ubiquitous consumer augmented reality systems. 1 1 ([n. d.]) 26--32. https:\/\/doi.org\/10.1002\/hbe2.113","DOI":"10.1002\/hbe2.113"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447310802205776"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-85607-6_6"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143518.1143522"},{"key":"e_1_2_2_7_1","unstructured":"W. I. B. Beveridge. 2020. The art of scientific investigation."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_2_9_1","volume-title":"SUS - A quick and dirty usability scale. Usability evaluation in industry 189, 194","author":"Brooke John","year":"1996","unstructured":"John Brooke. 1996. SUS - A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 7."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-019-01242-0"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3279778.3279795"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445651"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642320"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240644"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702423"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013115.1013155"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610888"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0950-"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2371574.2371608"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2800835.2807928"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2018.00037"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1056808.1056911"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MPRV.2005.93"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/SVR51698.2020.00041"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647369"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501821"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351241"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240646"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/nur.20247"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517550"},{"key":"e_1_2_2_31_1","volume-title":"Edison: A Life of Invention","author":"Israel Paul","year":"2000","unstructured":"Paul Israel. 2000. Edison: A Life of Invention (first edition ed.). John Wiley & Sons, New York, NY."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453157"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610931"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3675094.3678382"},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of Graphics Interface 2008 (GI '08)","author":"John Stasko Kang","year":"2008","unstructured":"Youn-ah Kang and John Stasko. 2008. Lightweight task\/application performance using single versus multiple monitors: a comparative study. In Proceedings of Graphics Interface 2008 (GI '08). Canadian Information Processing Society, CAN, 17--24. https:\/\/dl.acm.org\/doi\/10.5555\/1375714.1375718"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordhb\/9780199571932.013.024"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445335"},{"key":"e_1_2_2_38_1","volume-title":"Research methods in human computer interaction","author":"Lazar Jonathan","unstructured":"Jonathan Lazar. 2017. Research methods in human computer interaction (2nd edition ed.). Elsevier, Cambridge, MA.","edition":"2"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173610"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1452392.1452444"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376160"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1375761.1375762"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1029632.1029669"},{"volume-title":"Human-computer interaction: an empirical research perspective","author":"MacKenzie I. Scott","key":"e_1_2_2_44_1","unstructured":"I. Scott MacKenzie. 2013. Human-computer interaction: an empirical research perspective (first edition ed.). Morgan Kaufmann is an imprint of Elsevier, Amsterdam."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22668-2_8"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300707"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445560"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300826"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445323"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR-Adjunct.2018.00098"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376330"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1201\/b11963"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR-Adjunct.2018.00026"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3532209"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445170"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474349.3480217"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531073.3534489"},{"key":"e_1_2_2_58_1","unstructured":"Maria Rosala. 2020. The Critical Incident Technique in UX. https:\/\/www.nngroup.com\/articles\/critical-incident-technique\/"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300767"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3229089"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3279778.3279783"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-10-1"},{"key":"e_1_2_2_63_1","volume-title":"Experimentation Matters: Unlocking the Potential of New Technologies for Innovation","author":"Thomke Stefan H.","year":"2003","unstructured":"Stefan H. Thomke. 2003. Experimentation Matters: Unlocking the Potential of New Technologies for Innovation. Harvard Business Review Press, Boston, Mass."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3081016.3081020"},{"key":"e_1_2_2_65_1","unstructured":"Edward R. Tufte. 2013. The Visual Display of Quantitative Information. Cheshire Conn."},{"key":"e_1_2_2_66_1","volume-title":"van Teijlingen and Vanora Hundley","author":"Edwin","year":"2001","unstructured":"Edwin R. van Teijlingen and Vanora Hundley. 2001. The importance of pilot studies. (2001). https:\/\/aura.abdn.ac.uk\/handle\/2164\/157"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2648.2001.01757.x"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053098"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376448"},{"volume-title":"Concise oxford thesaurus in clair A-Z from","author":"Waite Maurice","key":"e_1_2_2_70_1","unstructured":"Maurice Waite. 2002. Concise oxford thesaurus in clair A-Z from. Oxford University Press. https:\/\/pks.pua.edu.eg\/social_sciences_books\/2320"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474769"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/329124.329126"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581500"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517689"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3571722"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240704"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.41.3.731"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678576","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678576","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T14:42:10Z","timestamp":1755787330000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678576"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,22]]},"references-count":77,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,8,22]]}},"alternative-id":["10.1145\/3678576"],"URL":"https:\/\/doi.org\/10.1145\/3678576","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2024,8,22]]},"assertion":[{"value":"2024-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}