{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T05:06:46Z","timestamp":1777525606222,"version":"3.51.4"},"reference-count":38,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2023,3,21]],"date-time":"2023-03-21T00:00:00Z","timestamp":1679356800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Virtual Reality (VR) technology is gaining in popularity as a research tool for studying human behavior. However, the use of VR technology for remote testing is still an emerging field. This study aimed to evaluate the feasibility of conducting remote VR behavioral experiments that require millisecond timing. Participants were recruited via an online crowdsourcing platform and accessed a task on the classic cognitive phenomenon \u201cInhibition of Return\u201d through a web browser using their own VR headset or desktop computer (68 participants in each group). The results confirm previous research that remote participants using desktop computers can be used effectively for conducting time-critical cognitive experiments. However, inhibition of return was only partially replicated for the VR headset group. Exploratory analyses revealed that technical factors, such as headset type, were likely to significantly impact variability and must be mitigated to obtain accurate results. This study demonstrates the potential for remote VR testing to broaden the research scope and reach a larger participant population. Crowdsourcing services appear to be an efficient and effective way to recruit participants for remote behavioral testing using high-end VR headsets.<\/jats:p>","DOI":"10.3390\/mti7030032","type":"journal-article","created":{"date-parts":[[2023,3,21]],"date-time":"2023-03-21T02:36:22Z","timestamp":1679366182000},"page":"32","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Online Platforms for Remote Immersive Virtual Reality Testing: An Emerging Tool for Experimental Behavioral Research"],"prefix":"10.3390","volume":"7","author":[{"given":"Tobias","family":"Loetscher","sequence":"first","affiliation":[{"name":"Cognitive Ageing and Impairment Neurosciences Lab., Justice & Society, University of South Australia, Adelaide, SA 5000, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0751-4089","authenticated-orcid":false,"given":"Nadia Siena","family":"Jurkovic","sequence":"additional","affiliation":[{"name":"Cognitive Ageing and Impairment Neurosciences Lab., Justice & Society, University of South Australia, Adelaide, SA 5000, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefan Carlo","family":"Michalski","sequence":"additional","affiliation":[{"name":"Cognitive Ageing and Impairment Neurosciences Lab., Justice & Society, University of South Australia, Adelaide, SA 5000, Australia"},{"name":"School of Psychology, University of Sydney, Sydney, NSW 2006, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Billinghurst","sequence":"additional","affiliation":[{"name":"Australian Research Centre for Interactive and Virtual Environments, STEM, University of South Australia, Adelaide, SA 5000, Australia"},{"name":"Empathic Computing Laboratory, The University of Auckland, Auckland 1010, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gun","family":"Lee","sequence":"additional","affiliation":[{"name":"Australian Research Centre for Interactive and Virtual Environments, STEM, University of South Australia, Adelaide, SA 5000, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.copsyc.2020.04.010","article-title":"Guidelines for immersive virtual reality in psychological research","volume":"36","author":"Vasser","year":"2020","journal-title":"Curr. Opin. Psychol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2086","DOI":"10.3389\/fpsyg.2018.02086","article-title":"The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature","volume":"9","author":"Cipresso","year":"2018","journal-title":"Front. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1111\/bjop.12290","article-title":"Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape","volume":"109","author":"Pan","year":"2018","journal-title":"Br. J. Psychol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1089","DOI":"10.1038\/nn948","article-title":"Virtual reality in behavioral neuroscience and beyond","volume":"5","author":"Tarr","year":"2002","journal-title":"Nat. Neurosci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"557","DOI":"10.3758\/BF03200735","article-title":"Immersive virtual environment technology as a basic research tool in psychology","volume":"31","author":"Loomis","year":"1999","journal-title":"Behav. Res. Methods Instrum. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"329","DOI":"10.3102\/0013189X17725519","article-title":"The role of MTurk in education research: Advantages, issues, and future directions","volume":"46","author":"Follmer","year":"2017","journal-title":"Educ. Res."},{"key":"ref_7","first-page":"823","article-title":"MTurk Research: Review and Recommendations","volume":"47","author":"Aguinis","year":"2020","journal-title":"J. Manag."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1380","DOI":"10.1111\/apps.12302","article-title":"Data collection via online platforms: Challenges and recommendations for future research","volume":"70","author":"Newman","year":"2021","journal-title":"Appl. Psychol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1177\/1948550619875149","article-title":"An MTurk crisis? Shifts in data quality and the impact on study results","volume":"11","author":"Chmielewski","year":"2020","journal-title":"Soc. Psychol. Personal. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1017\/psrm.2020.6","article-title":"The shape of and solutions to the MTurk quality crisis","volume":"8","author":"Kennedy","year":"2020","journal-title":"Political Sci. Res. Methods"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1643","DOI":"10.3758\/s13428-021-01694-3","article-title":"Data quality of platforms and panels for online behavioral research","volume":"54","author":"Peer","year":"2022","journal-title":"Behav. Res. Methods"},{"key":"ref_12","unstructured":"Walter, S.L., Seibert, S.E., Goering, D., and O\u2019Boyle, E.H. (2022). Key Topics in Consumer Behavior, Springer."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1145\/3472617","article-title":"Remote VR studies: A framework for running virtual reality studies remotely via participant-owned HMDs","volume":"28","author":"Radiah","year":"2021","journal-title":"ACM Trans. Comput. Hum. Interact."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ratcliffe, J., Soave, F., Bryan-Kinns, N., Tokarchuk, L., and Farkhatdinov, I. (2021, January 8\u201313). Extended Reality (XR) remote research: A survey of drawbacks and opportunities. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445170"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhao, J., Simpson, M., Sajjadi, P., Wallgr\u00fcn, J.O., Li, P., Bagher, M.M., Oprean, D., Padilla, L., and Klippel, A. (2021, January 4\u20138). Crowdxr-pitfalls and potentials of experiments with remote participants. Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bari, Italy.","DOI":"10.1109\/ISMAR52148.2021.00062"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Beacco, A., Oliva, R., Cabreira, C., Gallego, J., and Slater, M. (April, January 27). Disturbance and plausibility in a virtual rock concert: A pilot study. Proceedings of the 2021 IEEE Virtual Reality and 3D User Interfaces (VR), Lisboa, Portugal.","DOI":"10.1109\/VR50410.2021.00078"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Huber, B., and Gajos, K.Z. (2020). Conducting online virtual environment experiments with uncompensated, unsupervised samples. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0227629"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Mottelson, A., and Hornb\u00e6k, K. (2017, January 8\u201310). Virtual reality studies outside the laboratory. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden.","DOI":"10.1145\/3139131.3139141"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1406","DOI":"10.1109\/TVCG.2016.2518135","article-title":"An \u2018In the Wild\u2019 Experiment on Presence and Embodiment using Consumer Virtual Reality Equipment","volume":"22","author":"Steed","year":"2016","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Saffo, D., Di Bartolomeo, S., Yildirim, C., and Dunne, C. (2021, January 8\u201313). Remote and collaborative virtual reality experiments via social vr platforms. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445426"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"912078","DOI":"10.3389\/frvir.2022.912078","article-title":"Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments","volume":"3","author":"Steed","year":"2022","journal-title":"Front. Virtual Real."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"681482","DOI":"10.3389\/frvir.2021.681482","article-title":"Conducting Unsupervised Virtual Reality User Studies Online","volume":"2","author":"Mottelson","year":"2021","journal-title":"Front. Virtual Real."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ma, X., Cackett, M., Park, L., Chien, E., and Naaman, M. (2018, January 23\u201327). Web-based vr experiments powered by the crowd. Proceedings of the 2018 World Wide Web Conference, Lyon, France.","DOI":"10.1145\/3178876.3186034"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wiesing, M., Steink\u00f6nig, H., Vossel, S., Fink, G.R., and Weidner, R. (2022). Transferring paradigms from physical to virtual reality: Can reaction time effects be replicated in a virtual setting?. PsyArXiv.","DOI":"10.31234\/osf.io\/kmsnz"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1016\/S1364-6613(00)01452-2","article-title":"Inhibition of return","volume":"4","author":"Klein","year":"2000","journal-title":"Trends Cogn. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1080\/02643290600588095","article-title":"Inhibition of return: Twenty years after","volume":"23","author":"Klein","year":"2006","journal-title":"Cogn. Neuropsychol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1080\/02643298508252866","article-title":"Inhibition of return: Neural basis and function","volume":"2","author":"Posner","year":"1985","journal-title":"Cogn. Neuropsychol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Crump, M.J.C., McDonnell, J.V., and Gureckis, T.M. (2013). Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0057410"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"195","DOI":"10.3758\/s13428-018-01193-y","article-title":"PsychoPy2: Experiments in behavior made easy","volume":"51","author":"Peirce","year":"2019","journal-title":"Behav. Res. Methods"},{"key":"ref_30","unstructured":"(2022, January 05). The Jamovi Project Jamovi (Version 2.2.5). Available online: https:\/\/www.jamovi.org."},{"key":"ref_31","unstructured":"R Core Team (2022). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing. Available online: https:\/\/www.R-project.org\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the Tidyverse","volume":"4","author":"Wickham","year":"2019","journal-title":"J. Open Source Softw."},{"key":"ref_33","unstructured":"Grange, J. (2022, December 22). Trimr: An Implementation of Common Response Time Trimming Methods. Available online: https:\/\/CRAN.R-project.org\/package=trimr."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2815","DOI":"10.21105\/joss.02815","article-title":"Effectsize: Estimation of Effect Size Indices and Standardized Parameters","volume":"5","author":"Makowski","year":"2020","journal-title":"J. Open Source Softw."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1080\/14640749408401131","article-title":"A solution to the effect of sample size on outlier elimination","volume":"47","author":"Jolicoeur","year":"1994","journal-title":"Q. J. Exp. Psychol. Sect. A"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e9414","DOI":"10.7717\/peerj.9414","article-title":"The timing mega-study: Comparing a range of experiment generators, both lab-based and online","volume":"8","author":"Bridges","year":"2020","journal-title":"PeerJ"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1177\/0894439311415604","article-title":"Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times","volume":"30","author":"Brand","year":"2012","journal-title":"Soc. Sci. Comput. Rev."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.jesp.2017.01.006","article-title":"Beyond the Turk: Alternative platforms for crowdsourcing behavioral research","volume":"70","author":"Peer","year":"2017","journal-title":"J. Exp. Soc. Psychol."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/7\/3\/32\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:59:16Z","timestamp":1760122756000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/7\/3\/32"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,21]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["mti7030032"],"URL":"https:\/\/doi.org\/10.3390\/mti7030032","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,21]]}}}