{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,4]],"date-time":"2026-10-04T16:47:11Z","timestamp":1791132431999,"version":"4.1.0"},"reference-count":107,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2025,6,3]],"date-time":"2025-06-03T00:00:00Z","timestamp":1748908800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,3]],"date-time":"2025-06-03T00:00:00Z","timestamp":1748908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Autom Softw Eng"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Extended Reality (XR) is an emerging technology spanning diverse application domains and offering immersive user experiences. However, its unique characteristics, such as six degrees of freedom interactions, present significant testing challenges distinct from traditional 2D GUI applications, demanding novel testing techniques to build high-quality XR applications. This paper presents the first systematic mapping study on software testing for XR applications. We selected 34 studies focusing on techniques and empirical approaches in XR software testing for detailed examination. The studies are classified and reviewed to address the current research landscape, test facets, and evaluation methodologies in the XR testing domain. Additionally, we provide a repository summarising the mapping study, including datasets and tools referenced in the selected studies, to support future research and practical applications. Our study highlights open challenges in XR testing and proposes actionable future research directions to address the gaps and advance the field of XR software testing.<\/jats:p>","DOI":"10.1007\/s10515-025-00523-7","type":"journal-article","created":{"date-parts":[[2025,6,2]],"date-time":"2025-06-02T22:58:00Z","timestamp":1748905080000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Software testing for extended reality applications: a systematic mapping study"],"prefix":"10.1007","volume":"32","author":[{"given":"Ruizhen","family":"Gu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9 Miguel","family":"Rojas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donghwan","family":"Shin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,6,3]]},"reference":[{"key":"523_CR1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809163","author":"P Ammann","year":"2008","unstructured":"Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press (2008). https:\/\/doi.org\/10.1017\/CBO9780511809163","journal-title":"Cambridge University Press"},{"key":"523_CR2","doi-asserted-by":"publisher","unstructured":"Andrade, S.A., Nunes, F.L.S., Delamaro, M.E.: Towards the systematic testing of virtual reality programs. In: 2019 21st symposium on virtual and augmented reality (SVR), pp 196\u2013205 (2019). https:\/\/doi.org\/10.1109\/SVR.2019.00044","DOI":"10.1109\/SVR.2019.00044"},{"key":"523_CR3","doi-asserted-by":"publisher","unstructured":"Andrade, S.A., Quevedo AJU, Nunes FLS, et al.: Understanding vr software testing needs from stakeholders\u2019 points of view. In: Symposium on virtual and augmented reality (SVR), pp. 57\u201366 (2020). https:\/\/doi.org\/10.1109\/SVR51698.2020.00024","DOI":"10.1109\/SVR51698.2020.00024"},{"key":"523_CR4","doi-asserted-by":"publisher","unstructured":"Andrade, S.A., Nunes, F.L.S., Delamaro, M.E.: Exploiting deep reinforcement learning and metamorphic testing to automatically test virtual reality applications. Software Testing, Verification and Reliability 33(8) (2023). https:\/\/doi.org\/10.1002\/stvr.1863","DOI":"10.1002\/stvr.1863"},{"key":"523_CR5","doi-asserted-by":"publisher","unstructured":"Ardito, L., Coppola, R., Morisio, M, et al.: Espresso vs. eyeautomate: An experiment for the comparison of two generations of android gui testing. In: Proc. of the 23rd Intl. Conf. on evaluation and assessment in software engineering. ACM, EASE \u201919, p 13\u201322 (2019). https:\/\/doi.org\/10.1145\/3319008.3319022","DOI":"10.1145\/3319008.3319022"},{"key":"523_CR6","doi-asserted-by":"publisher","unstructured":"Barbieri L, Bruno F, Muzzupappa M.: Virtual museum system evaluation through user studies. J. Cult. Herit. 26:101\u2013108 (2017). https:\/\/doi.org\/10.1016\/j.culher.2017.02.005","DOI":"10.1016\/j.culher.2017.02.005"},{"issue":"5","key":"523_CR7","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1109\/TSE.2014.2372785","volume":"41","author":"ET Barr","year":"2015","unstructured":"Barr, E.T., Harman, M., McMinn, P., et al.: The oracle problem in software testing: A survey. IEEE Trans. Soft. Eng. 41(5), 507\u2013525 (2015). https:\/\/doi.org\/10.1109\/TSE.2014.2372785","journal-title":"IEEE Trans. Soft. Eng."},{"key":"523_CR8","doi-asserted-by":"publisher","unstructured":"Bierbaum, A., Hartling, P., Cruz-Neira, C.: Automated testing of virtual reality application interfaces. In: Proc. of the Workshop on Virtual Environments 2003. ACM, EGVE \u201903, p 107\u2013114 (2003). https:\/\/doi.org\/10.1145\/3551349.3561160","DOI":"10.1145\/3551349.3561160"},{"key":"523_CR9","doi-asserted-by":"publisher","unstructured":"B\u00f6rsting I, Heikamp M, Hesenius M, et al.: Software engineering for augmented reality - a research agenda. Proc ACM Hum-Comput Interact 6 (2022). https:\/\/doi.org\/10.1145\/3532205","DOI":"10.1145\/3532205"},{"key":"523_CR10","doi-asserted-by":"publisher","unstructured":"Bouvier, P., De Sorbier, F., Chaudeyrac, P., et al.: Cross benefits between virtual reality and games. In: Intl. Conf. and Industry Symposium on Computer Games, Animation, Multimedia, IPTV, Edutainment and Security (CGAT\u201908) (2008). https:\/\/doi.org\/10.5176\/978-981-08-8227-3_cgat08-26","DOI":"10.5176\/978-981-08-8227-3_cgat08-26"},{"key":"523_CR11","volume-title":"Interactive occlusion and collision of real and virtual objects in augmented reality","author":"D Breen","year":"2000","unstructured":"Breen, D., Rose, E., Whitaker, R.: Interactive occlusion and collision of real and virtual objects in augmented reality. Proc of Eurographics Poitiers, France (2000)"},{"key":"523_CR12","doi-asserted-by":"publisher","unstructured":"Brown-Johnson C, Berrean B, Cataldo J.: Development and usability evaluation of the mhealth tool for lung cancer (mhealth tlc): A virtual world health game for lung cancer patients. Patient Education and Counseling 98 (2015).https:\/\/doi.org\/10.1016\/j.pec.2014.12.006","DOI":"10.1016\/j.pec.2014.12.006"},{"issue":"2","key":"523_CR13","doi-asserted-by":"publisher","first-page":"550","DOI":"10.1109\/TDSC.2019.2907942","volume":"18","author":"P Casey","year":"2021","unstructured":"Casey, P., Baggili, I., Yarramreddy, A.: Immersive virtual reality attacks and the human joystick. IEEE Transactions on Dependable and Secure Computing 18(2), 550\u2013562 (2021). https:\/\/doi.org\/10.1109\/TDSC.2019.2907942","journal-title":"IEEE Transactions on Dependable and Secure Computing"},{"key":"523_CR14","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/j.compbiomed.2018.02.004","volume":"95","author":"O Charron","year":"2018","unstructured":"Charron, O., Lallement, A., Jarnet, D., et al.: Automatic detection and segmentation of brain metastases on multimodal mr images with a deep convolutional neural network. Comput. Biol. Med. 95, 43\u201354 (2018). https:\/\/doi.org\/10.1016\/j.compbiomed.2018.02.004","journal-title":"Comput. Biol. Med."},{"key":"523_CR15","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1016\/j.sbspro.2013.10.289","volume":"97","author":"CJ Chen","year":"2013","unstructured":"Chen, C.J., Lau, S.Y., Chuah, K.M., et al.: Group usability testing of virtual reality-based learning environments: A modified approach. Procedia - Soc. Behav. Sci. 97, 691\u2013699 (2013). https:\/\/doi.org\/10.1016\/j.sbspro.2013.10.289","journal-title":"Procedia - Soc. Behav. Sci."},{"key":"523_CR16","doi-asserted-by":"publisher","unstructured":"Cheng, S., Qu, H.: Key Issues of Real-time Collision Detection in Virtual Reality. Intl Journal of Frontiers in Engineering Technology 3(5) (2021). https:\/\/doi.org\/10.25236\/IJFET.2021.030505","DOI":"10.25236\/IJFET.2021.030505"},{"key":"523_CR17","doi-asserted-by":"publisher","unstructured":"Corr\u00eaa, C.G., Delamaro, M.E., Chaim, M.L., et al.: Software testing automation of vr-based systems with haptic interfaces. Comput. J. 64(5), 826\u2013841 (2020). . https:\/\/doi.org\/10.1093\/comjnl\/bxaa054","DOI":"10.1093\/comjnl\/bxaa054"},{"key":"523_CR18","doi-asserted-by":"publisher","unstructured":"Corr\u00eaa Souza, A.C., Nunes, F.L.S., Delamaro, M.E.: An automated functional testing approach for virtual reality applications. Software Testing, Verification and Reliability 28(8) (2018). https:\/\/doi.org\/10.1002\/stvr.1690","DOI":"10.1002\/stvr.1690"},{"key":"523_CR19","doi-asserted-by":"publisher","unstructured":"Davis, S., Nesbitt, K., Nalivaiko, E.: A systematic review of cybersickness. In: Proc. of the 2014 conference on interactive entertainment. ACM, IE2014, pp. 1\u20139 (2014). https:\/\/doi.org\/10.1145\/2677758.2677780","DOI":"10.1145\/2677758.2677780"},{"key":"523_CR20","doi-asserted-by":"publisher","unstructured":"Dey, A., Billinghurst, M., Lindeman, R.W., et al.: A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014. Frontiers in Robotics and AI 5 (2018). https:\/\/doi.org\/10.3389\/frobt.2018.00037","DOI":"10.3389\/frobt.2018.00037"},{"key":"523_CR21","doi-asserted-by":"publisher","unstructured":"Doerner, R., Broll, W., Grimm, P., et al.: (eds) Virtual and Augmented Reality (VR\/AR): Foundations and Methods of Extended Realities (XR). Springer Intl. Publishing (2022). https:\/\/doi.org\/10.1007\/978-3-030-79062-2","DOI":"10.1007\/978-3-030-79062-2"},{"key":"523_CR22","unstructured":"Emery, V., Jacko, J., Kongnakorn, T., et al.: Identifying critical interaction scenarios for innovative user modeling. In: Proc. of the 1st Intl. Conf. on universal access in human-computer interaction, pp. 481\u2013485 (2001)"},{"key":"523_CR23","doi-asserted-by":"publisher","unstructured":"Figueira, T., Gil, A.: Youkai: A cross-platform framework for testing vr\/ar apps. In: HCI Intl. 2022 \u2013 Late Breaking Papers: interacting with extended reality and artificial intelligence. Springer Nature Switzerland, pp. 3\u201312 (2022). https:\/\/doi.org\/10.1007\/978-3-031-21707-4_1","DOI":"10.1007\/978-3-031-21707-4_1"},{"issue":"8","key":"523_CR24","doi-asserted-by":"publisher","first-page":"1374","DOI":"10.1016\/j.infsof.2013.02.006","volume":"55","author":"V Garousi","year":"2013","unstructured":"Garousi, V., Mesbah, A., Betin-Can, A., et al.: A systematic mapping study of web application testing. Inf. Soft. Technol. 55(8), 1374\u20131396 (2013). https:\/\/doi.org\/10.1016\/j.infsof.2013.02.006","journal-title":"Inf. Soft. Technol."},{"key":"523_CR25","doi-asserted-by":"publisher","unstructured":"Geiger, A., Lenz, P., Stiller, C., et al.: Vision meets robotics: The kitti dataset. Intl J. Robot. Res. 32(11), 1231\u20131237 (2013). https:\/\/doi.org\/10.1177\/0278364913491297","DOI":"10.1177\/0278364913491297"},{"key":"523_CR26","doi-asserted-by":"publisher","unstructured":"Gu, R., Rojas, J.M.: An empirical study on the adoption of scripted gui testing for android apps. In: 2023 38th IEEE\/ACM international conference on automated software engineering workshops (ASEW), pp. 179\u2013182 (2023). https:\/\/doi.org\/10.1109\/ASEW60602.2023.00030","DOI":"10.1109\/ASEW60602.2023.00030"},{"key":"523_CR27","doi-asserted-by":"publisher","unstructured":"Herbold, S., Harms, P.: Autoquest \u2013 automated quality engineering of event-driven software. In: 2013 IEEE Sixth Intl. Conf. on software testing, verification and validation workshops, pp. 134\u2013139 (2013). https:\/\/doi.org\/10.1109\/ICSTW.2013.23","DOI":"10.1109\/ICSTW.2013.23"},{"key":"523_CR28","doi-asserted-by":"publisher","unstructured":"Harms, P.: Automated usability evaluation of virtual reality applications. ACM Trans Comput-Hum Interact 26(3) (2019). https:\/\/doi.org\/10.1145\/3301423","DOI":"10.1145\/3301423"},{"key":"523_CR29","doi-asserted-by":"publisher","first-page":"i","DOI":"10.2200\/S00987ED1V01Y202001HCI045","volume":"1","author":"M Hertzum","year":"2020","unstructured":"Hertzum, M.: Usability testing: A practitioner\u2019s guide to evaluating the user experience. Synthesis Lectures on Human-Centered Informatics 1, i\u2013105 (2020). https:\/\/doi.org\/10.2200\/S00987ED1V01Y202001HCI045","journal-title":"Synthesis Lectures on Human-Centered Informatics"},{"key":"523_CR30","doi-asserted-by":"publisher","unstructured":"Hesenius M, Griebe T, Gries S, et al.: Automating ui tests for mobile applications with formal gesture descriptions. In: Proc. of the 16th Intl. Conf. on human-computer interaction with mobile devices & services. ACM, MobileHCI \u201914, p 213\u2013222 (2014). https:\/\/doi.org\/10.1145\/2628363.2628391","DOI":"10.1145\/2628363.2628391"},{"issue":"10","key":"523_CR31","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1145\/2858965.2814320","volume":"50","author":"Y Hu","year":"2015","unstructured":"Hu, Y., Azim, T., Neamtiu, I.: Versatile yet lightweight record-and-replay for android. SIGPLAN Not 50(10), 349\u2013366 (2015). https:\/\/doi.org\/10.1145\/2858965.2814320","journal-title":"SIGPLAN Not"},{"key":"523_CR32","doi-asserted-by":"publisher","unstructured":"Izuazu, U.U., Kim, D.S., Lee, J.M.: Unravelling the black box: Enhancing virtual reality network security with interpretable deep learning-based intrusion detection system. In: Intl. Conf. on Information and Communication Technology Convergence (ICTC), pp 928\u2013931 (2023). https:\/\/doi.org\/10.1109\/ICTC58733.2023.10392826","DOI":"10.1109\/ICTC58733.2023.10392826"},{"issue":"1","key":"523_CR33","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1108\/AA-12-2019-0224","volume":"41","author":"Y Jin","year":"2021","unstructured":"Jin, Y., Geng, J., He, Z., et al.: A capsule-based collision detection approach of irregular objects in virtual maintenance. Assembly Autom. 41(1), 89\u2013105 (2021). https:\/\/doi.org\/10.1108\/AA-12-2019-0224","journal-title":"Assembly Autom."},{"key":"523_CR34","doi-asserted-by":"publisher","unstructured":"Jung, S.M., Oh, S.H., Whangbo, T.k.: 360$${\\circ }$$ stereo image based vr motion sickness testing system. In: 2017 Intl. Conf. on Emerging Trends & Innovation in ICT (ICEI), pp. 150\u2013153. https:\/\/doi.org\/10.1109\/ETIICT.2017.7977027","DOI":"10.1109\/ETIICT.2017.7977027"},{"key":"523_CR35","doi-asserted-by":"publisher","unstructured":"Kami\u0144ska, D., Zwoli\u0144ski, G., Laska-Le\u015bniewicz, A.: Usability testing of virtual reality applications-the pilot study. Sensors 22(4) (2022). https:\/\/doi.org\/10.3390\/s22041342","DOI":"10.3390\/s22041342"},{"issue":"2","key":"523_CR36","first-page":"85","volume":"10","author":"S Kavanagh","year":"2017","unstructured":"Kavanagh, S., Luxton-Reilly, A., Wuensche, B., et al.: A systematic review of virtual reality in education. Themes Sci. Technol. Educ. 10(2), 85\u2013119 (2017)","journal-title":"Themes Sci. Technol. Educ."},{"key":"523_CR37","doi-asserted-by":"publisher","first-page":"104107","DOI":"10.1016\/j.apergo.2023.104107","volume":"113","author":"K Kia","year":"2023","unstructured":"Kia, K., Hwang, J., Kim, J.H.: Effects of error rates and target sizes on neck and shoulder biomechanical loads during augmented reality interactions. Appl. Ergon. 113, 104107 (2023). https:\/\/doi.org\/10.1016\/j.apergo.2023.104107","journal-title":"Appl. Ergon."},{"key":"523_CR38","doi-asserted-by":"publisher","unstructured":"Kilger, F., Kabil, A., Tippmann, V., et al.: Detecting and preventing faked mixed reality. In: 2021 IEEE 4th Intl. Conf. on Multimedia Information Processing and Retrieval (MIPR), pp. 399\u2013405 (2021). https:\/\/doi.org\/10.1109\/MIPR51284.2021.00074","DOI":"10.1109\/MIPR51284.2021.00074"},{"key":"523_CR39","doi-asserted-by":"publisher","unstructured":"Kim, H.G., Baddar, W.J., Lim, H.t., et al.: Measurement of exceptional motion in vr video contents for vr sickness assessment using deep convolutional autoencoder. In: Proc. of the 23rd ACM Symposium on Virtual Reality Software and Technology. ACM, VRST \u201917 (2017). https:\/\/doi.org\/10.1145\/3139131.3139137","DOI":"10.1145\/3139131.3139137"},{"issue":"3","key":"523_CR40","doi-asserted-by":"publisher","first-page":"179","DOI":"10.5999\/aps.2017.44.3.179","volume":"44","author":"Y Kim","year":"2017","unstructured":"Kim, Y., Kim, H., Kim, Y.O.: Virtual Reality and Augmented Reality in Plastic Surgery: A Review. Arch. Plast. Surg. 44(3), 179\u2013187 (2017). https:\/\/doi.org\/10.5999\/aps.2017.44.3.179","journal-title":"Arch. Plast. Surg."},{"issue":"10","key":"523_CR41","doi-asserted-by":"publisher","first-page":"893","DOI":"10.1080\/10447318.2019.1699746","volume":"36","author":"YM Kim","year":"2020","unstructured":"Kim, Y.M., Rhiu, I., Yun, M.H.: A systematic review of a virtual reality system from the perspective of user experience. Intl Journal of Human-Computer Interaction 36(10), 893\u2013910 (2020). https:\/\/doi.org\/10.1080\/10447318.2019.1699746","journal-title":"Intl Journal of Human-Computer Interaction"},{"key":"523_CR42","unstructured":"Kitchenham, B.A., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE-2007-01, Keele University (2007)"},{"issue":"1","key":"523_CR43","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1109\/TR.2018.2865733","volume":"68","author":"P Kong","year":"2019","unstructured":"Kong, P., Li, L., Gao, J., et al.: Automated testing of android apps: A systematic literature review. IEEE Trans. Reliab. 68(1), 45\u201366 (2019). https:\/\/doi.org\/10.1109\/TR.2018.2865733","journal-title":"IEEE Trans. Reliab."},{"key":"523_CR44","doi-asserted-by":"publisher","unstructured":"Kundu, R.K., Elsaid, O.Y., Calyam, P., et al.: Vr-lens: Super learning-based cybersickness detection and explainable ai-guided deployment in virtual reality. In: Proc. of the 28th Intl. Conf. on Intelligent User Interfaces. ACM, IUI \u201923, p 819\u2013834 (2023). https:\/\/doi.org\/10.1145\/3581641.3584044","DOI":"10.1145\/3581641.3584044"},{"key":"523_CR45","doi-asserted-by":"publisher","unstructured":"Kirayeva, R.R., Khafizov, M.R., Turdiev, T.T., et\u00a0al.: Automated testing of functional requirements for virtual reality applications. In: 2023 IEEE XVI Intl. Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE), pp 1760\u20131764 (2023). https:\/\/doi.org\/10.1109\/APEIE59731.2023.10347611","DOI":"10.1109\/APEIE59731.2023.10347611"},{"key":"523_CR46","doi-asserted-by":"publisher","unstructured":"Kuri, M., Karre, S.A., Reddy, Y.R.: Understanding software quality metrics for virtual reality products - a mapping study. In: Proc. of the Innovations in Software Engineering Conference (Formerly Known as India Software Engineering Conference). ACM, ISEC \u201921 (2021). https:\/\/doi.org\/10.1145\/3452383.3452391","DOI":"10.1145\/3452383.3452391"},{"key":"523_CR47","doi-asserted-by":"publisher","unstructured":"Lam, W., Wu, Z., Li, D., et al.: Record and replay for android: are we there yet in industrial cases? In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, ESEC\/FSE 2017, pp. 854\u2013859 (2017). https:\/\/doi.org\/10.1145\/3106237.3117769","DOI":"10.1145\/3106237.3117769"},{"key":"523_CR48","doi-asserted-by":"publisher","unstructured":"Lehman, S.M., Alrumayh, A.S., Kolhe, K., et al.: Hidden in plain sight: Exploring privacy risks of mobile augmented reality applications. ACM Trans Priv Secur 25(4) (2022). https:\/\/doi.org\/10.1145\/3524020","DOI":"10.1145\/3524020"},{"key":"523_CR49","doi-asserted-by":"publisher","unstructured":"Lehman, S.M., Elezovikj, S., Ling, H., et al.: Archie++: A cloud-enabled framework for conducting ar system testing in the wild. IEEE Trans. Vis. Comput. Graph. 29(4), 2102\u20132116 (2023). https:\/\/doi.org\/10.1109\/TVCG.2022.3141029","DOI":"10.1109\/TVCG.2022.3141029"},{"issue":"1","key":"523_CR50","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s12652-011-0052-4","volume":"4","author":"A Lele","year":"2013","unstructured":"Lele, A.: Virtual reality and its military utility. J. Ambient Intell. Humanized Comput. 4(1), 17\u201326 (2013). https:\/\/doi.org\/10.1007\/s12652-011-0052-4","journal-title":"J. Ambient Intell. Humanized Comput."},{"key":"523_CR51","doi-asserted-by":"publisher","unstructured":"Li, S., Wu, Y., Liu, Y., et al.: An exploratory study of bugs in extended reality applications on the web. In: 2020 IEEE 31st Intl. symposium on software reliability engineering (ISSRE), pp. 172\u2013183 (2020). https:\/\/doi.org\/10.1109\/ISSRE5003.2020.00025","DOI":"10.1109\/ISSRE5003.2020.00025"},{"key":"523_CR52","doi-asserted-by":"publisher","unstructured":"Li, S., Gao, C., Zhang, J., et al.: Less cybersickness, please: Demystifying and detecting stereoscopic visual inconsistencies in virtual reality apps. Proc ACM Softw Eng 1(FSE) (2024). https:\/\/doi.org\/10.1145\/3660803","DOI":"10.1145\/3660803"},{"key":"523_CR53","doi-asserted-by":"publisher","unstructured":"Leykin, A., Tuceryan, M.: Automatic determination of text readability over textured backgrounds for augmented reality systems. In: IEEE\/ACM Intl. Symposium on Mixed and Augmented Reality, pp 224\u2013230 (2004). https:\/\/doi.org\/10.1109\/ISMAR.2004.22","DOI":"10.1109\/ISMAR.2004.22"},{"key":"523_CR54","unstructured":"Liu, Z., Li, C., Chen, C., et al.: Vision-driven automated mobile gui testing via multimodal large language model (2024). arXiv:2407.03037"},{"key":"523_CR55","doi-asserted-by":"publisher","unstructured":"Lu, C., Shi, J., Jia, J.: Abnormal event detection at 150 fps in matlab. In: IEEE Intl. Conf. on Computer Vision, pp. 2720\u20132727 (2013). https:\/\/doi.org\/10.1109\/ICCV.2013.338","DOI":"10.1109\/ICCV.2013.338"},{"key":"523_CR56","doi-asserted-by":"publisher","unstructured":"Mahadevan, V., Li, W., Bhalodia, V., et al.: Anomaly detection in crowded scenes. In: IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, pp. 1975\u20131981 (2010). https:\/\/doi.org\/10.1109\/CVPR.2010.5539872","DOI":"10.1109\/CVPR.2010.5539872"},{"key":"523_CR57","doi-asserted-by":"publisher","unstructured":"Milgram, P., Takemura, H., Utsumi, A., et al.: Augmented reality: A class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies 2351 (1994). https:\/\/doi.org\/10.1117\/12.197321","DOI":"10.1117\/12.197321"},{"key":"523_CR58","unstructured":"Modarressi, S.N., Maurer, F., Wang, N.: Capture and replay testing tool for xr applications. In: RealXR@AVI (2024). https:\/\/api.semanticscholar.org\/CorpusID:270563554"},{"key":"523_CR59","doi-asserted-by":"publisher","unstructured":"Minor, S., Ketoma, .K., Meixner, G.: Test automation for augmented reality applications: a development process model and case study. i-com 22(3):175\u2013192 (2023). https:\/\/doi.org\/10.1515\/icom-2023-0029","DOI":"10.1515\/icom-2023-0029"},{"key":"523_CR60","doi-asserted-by":"publisher","unstructured":"Molina, F., Gorla, A., d\u2019Amorim, M.: Test oracle automation in the era of llms. ACM Trans Softw Eng Methodol (2025). https:\/\/doi.org\/10.1145\/3715107, just Accepted","DOI":"10.1145\/3715107"},{"key":"523_CR61","doi-asserted-by":"publisher","unstructured":"Paduraru, C., Stefanescu, A., Jianu, A.: Unit test generation using large language models for unity game development. In: Proc. of the 1st ACM Intl. Workshop on Foundations of Applied Software Engineering for Games. ACM, FaSE4Games 2024, p 7\u201313 (2024). https:\/\/doi.org\/10.1145\/3663532.3664466","DOI":"10.1145\/3663532.3664466"},{"key":"523_CR62","doi-asserted-by":"publisher","unstructured":"Pastor Ric\u00f3s, F.: Scriptless testing for extended reality systems. In: Research Challenges in Information Science. Springer Intl. Publishing, pp. 786\u2013794 (2022). https:\/\/doi.org\/10.1007\/978-3-031-05760-1_56","DOI":"10.1007\/978-3-031-05760-1_56"},{"key":"523_CR63","doi-asserted-by":"publisher","unstructured":"Pastore, F., Mariani, L., Fraser, G.: Crowdoracles: Can the crowd solve the oracle problem? In: 2013 IEEE Sixth Intl. Conf. on Software Testing, Verification and Validation, pp. 342\u2013351 (2013).https:\/\/doi.org\/10.1109\/ICST.2013.13","DOI":"10.1109\/ICST.2013.13"},{"key":"523_CR64","unstructured":"Odeleye, B., Loukas, G., Heartfield, R., et al.: Detecting framerate-oriented cyber attacks on user experience in virtual reality. In: 1st Intl. Workshop on Security for XR and XR for Security (2021)"},{"key":"523_CR65","doi-asserted-by":"crossref","unstructured":"Petersen, K., Feldt, R., Mujtaba, S., et al.: Systematic mapping studies in software engineering. In: Proc. of the 12th Intl. Conf. on Evaluation and Assessment in Software Engineering. BCS Learning & Development Ltd., EASE\u201908, pp. 68\u201377 (2008)","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"523_CR66","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.infsof.2015.03.007","volume":"64","author":"K Petersen","year":"2015","unstructured":"Petersen, K., Vakkalanka, S., Kuzniarz, L.: Guidelines for conducting systematic mapping studies in software engineering: An update. Inf. Soft. Technol. 64, 1\u201318 (2015). https:\/\/doi.org\/10.1016\/j.infsof.2015.03.007","journal-title":"Inf. Soft. Technol."},{"key":"523_CR67","doi-asserted-by":"publisher","unstructured":"Politowski, C., Gu\u00e9h\u00e9neuc, Y.G., Petrillo, F.: Towards automated video game testing: still a long way to go. In: Proc. of the 6th Intl. ICSE Workshop on Games and Software Engineering: Engineering Fun, Inspiration, and Motivation. ACM, GAS \u201922, p 37\u201343 (2022). https:\/\/doi.org\/10.1145\/3524494.3527627","DOI":"10.1145\/3524494.3527627"},{"key":"523_CR68","doi-asserted-by":"publisher","unstructured":"Prasetya, I.S.W.B., Shirzadehhajimahmood, S., Ansari, S.G., et al.: An agent-based architecture for ai-enhanced automated testing for xr systems, a short paper. In: Intl. Conf. on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, pp. 213\u2013217 (2021). https:\/\/doi.org\/10.1109\/ICSTW52544.2021.00044","DOI":"10.1109\/ICSTW52544.2021.00044"},{"key":"523_CR69","unstructured":"Priem, J., Piwowar, H., Orr, R.: Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts (2022). https:\/\/arxiv.org\/abs\/2205.01833"},{"issue":"4","key":"523_CR70","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1109\/JPROC.2019.2895105","volume":"107","author":"X Qiao","year":"2019","unstructured":"Qiao, X., Ren, P., Dustdar, S., et al.: Web ar: A promising future for mobile augmented reality-state of the art, challenges, and insights. Proc. of the IEEE 107(4), 651\u2013666 (2019). https:\/\/doi.org\/10.1109\/JPROC.2019.2895105","journal-title":"Proc. of the IEEE"},{"key":"523_CR71","unstructured":"Qiu, Z., Liu, W., Feng, H., et al.: Can large language models understand symbolic graphics programs? (2024).https:\/\/arxiv.org\/abs\/2408.08313"},{"key":"523_CR72","doi-asserted-by":"publisher","unstructured":"Qu, C., Che, X., Ma, S., et al.: Bio-physiological-signals-based vr cybersickness detection. CCF Transactions on Pervasive Computing and Interaction 4 (2022). https:\/\/doi.org\/10.1007\/s42486-022-00103-8","DOI":"10.1007\/s42486-022-00103-8"},{"key":"523_CR73","doi-asserted-by":"publisher","unstructured":"Qin, X., Hassan, F.: Dytrec: A dynamic testing recommendation tool for unity-based virtual reality software. In: Proc. of the 37th IEEE\/ACM Intl. Conf. on Automated Software Engineering. ACM, New York, NY, USA (2023). https:\/\/doi.org\/10.1145\/3551349.3560510","DOI":"10.1145\/3551349.3560510"},{"key":"523_CR74","doi-asserted-by":"publisher","unstructured":"Rafi, T., Zhang, X., Wang, X.: Predart: Towards automatic oracle prediction of object placements in augmented reality testing. In: Proc. of the 37th IEEE\/ACM Intl. Conf. on automated software engineering. ACM (2023). https:\/\/doi.org\/10.1145\/3551349.3561160","DOI":"10.1145\/3551349.3561160"},{"key":"523_CR75","doi-asserted-by":"publisher","unstructured":"Ramaseri Chandra, A.N., El Jamiy, F., Reza, H.: A review on usability and performance evaluation in virtual reality systems. In: 2019 Intl. Conf. on computational science and computational intelligence (CSCI), pp. 1107\u20131114 (2019). https:\/\/doi.org\/10.1109\/CSCI49370.2019.00210","DOI":"10.1109\/CSCI49370.2019.00210"},{"key":"523_CR76","doi-asserted-by":"crossref","unstructured":"Gunawan, R., Wibisono, Y.P., Primasari, C.H., et al.: Blackbox Testing on Virtual Reality Gamelan Saron Using Equivalence Partition Method. Jurnal Buana Informatika 14(01), 11\u201319 (2023). https:\/\/doi.org\/10.24002\/jbi.v14i01.6606","DOI":"10.24002\/jbi.v14i01.6606"},{"key":"523_CR77","doi-asserted-by":"publisher","unstructured":"Robert, S., Kennedy KSBNorman. E., Lane Lilienthal, M.G.: Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The Intl J. Aviat. Psych. 3(3), 203\u2013220 (1993). https:\/\/doi.org\/10.1207\/s15327108ijap0303_3","DOI":"10.1207\/s15327108ijap0303_3"},{"key":"523_CR78","unstructured":"Roberts, J.: The ar\/vr technology stack: A central repository of software development libraries, platforms, and tools (2023). https:\/\/doi.org\/10.48550\/arXiv.2305.07842"},{"key":"523_CR79","doi-asserted-by":"publisher","unstructured":"Rodriguez, I., Wang, X.: An empirical study of open source virtual reality software projects. In: 2017 ACM\/IEEE Intl. Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 474\u2013475 (2017). https:\/\/doi.org\/10.1109\/ESEM.2017.65","DOI":"10.1109\/ESEM.2017.65"},{"key":"523_CR80","doi-asserted-by":"publisher","unstructured":"Rzig, D.E., Iqbal, N., Attisano, I., et al.: Virtual reality (vr) automated testing in the wild: A case study on unity-based vr applications. In: Proc. of the 32nd ACM SIGSOFT Intl. symposium on software testing and analysis. ACM, ISSTA 2023, p 1269\u20131281 (2023). https:\/\/doi.org\/10.1145\/3597926.3598134","DOI":"10.1145\/3597926.3598134"},{"key":"523_CR81","doi-asserted-by":"publisher","unstructured":"Scheibmeir, J., Malaiya, Y.K.: Quality model for testing augmented reality applications. In: 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp. 0219\u20130226 (2019). https:\/\/doi.org\/10.1109\/UEMCON47517.2019.8992974","DOI":"10.1109\/UEMCON47517.2019.8992974"},{"key":"523_CR82","doi-asserted-by":"publisher","unstructured":"Su, T., Meng, G., Chen, Y., et al.: Guided, stochastic model-based gui testing of android apps. In: Proc. of the Joint Meeting on Foundations of Software Engineering (ESEC\/FSE). ACM, pp. 245\u2013256 (2017). https:\/\/doi.org\/10.1145\/3106237.3106298","DOI":"10.1145\/3106237.3106298"},{"key":"523_CR83","doi-asserted-by":"publisher","unstructured":"Su, T., Yan, Y., Wang, J., et al.: Fully automated functional fuzzing of android apps for detecting non-crashing logic bugs. Proc ACM Program Lang 5(OOPSLA) (2021). https:\/\/doi.org\/10.1145\/3485533","DOI":"10.1145\/3485533"},{"key":"523_CR84","doi-asserted-by":"publisher","unstructured":"Sarupuri, B., Hoermann, S., Whitton, M.C., et al.: Lute: A locomotion usability test environmentfor virtual reality. In: 2018 10th Intl. Conf. on Virtual Worlds and Games for Serious Applications (VS-Games), pp 1\u20134 (2018). https:\/\/doi.org\/10.1109\/VS-Games.2018.8493432","DOI":"10.1109\/VS-Games.2018.8493432"},{"issue":"1280","key":"523_CR85","doi-asserted-by":"publisher","first-page":"1615","DOI":"10.1017\/aer.2020.49","volume":"124","author":"S Tadeja","year":"2020","unstructured":"Tadeja, S., Seshadri, P., Kristensson, P.: Aerovr: An immersive visualisation system for aerospace design and digital twinning in virtual reality. Aeronaut. J. 124(1280), 1615\u20131635 (2020). https:\/\/doi.org\/10.1017\/aer.2020.49","journal-title":"Aeronaut. J."},{"key":"523_CR86","doi-asserted-by":"publisher","unstructured":"Sendari, S., Firmansah, A., Aripriharta.: Performance analysis of augmented reality based on vuforia using 3d marker detection. In: 2020 4th Intl. Conf. on Vocational Education and Training (ICOVET), pp 294\u2013298 (2020). https:\/\/doi.org\/10.1109\/ICOVET50258.2020.9230276","DOI":"10.1109\/ICOVET50258.2020.9230276"},{"key":"523_CR87","unstructured":"Technologies, U.: Unity - manual: Gameobject (2024). https:\/\/docs.unity3d.com\/Manual\/class-GameObject.html, accessed: 13 Aug 2024"},{"key":"523_CR88","doi-asserted-by":"publisher","unstructured":"Thomas, P., Spielman, S., Craswell, N., et al.: Large language models can accurately predict searcher preferences. In: Proc. of the 47th Intl. ACM SIGIR conference on research and development in information retrieval. ACM, SIGIR \u201924, p 1930\u20131940 (2024). https:\/\/doi.org\/10.1145\/3626772.3657707","DOI":"10.1145\/3626772.3657707"},{"issue":"1","key":"523_CR89","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1007\/s11219-018-9418-6","volume":"27","author":"P Tramontana","year":"2019","unstructured":"Tramontana, P., Amalfitano, D., Amatucci, N., et al.: Automated functional testing of mobile applications: A systematic mapping study. Soft. Qual. J. 27(1), 149\u2013201 (2019). https:\/\/doi.org\/10.1007\/s11219-018-9418-6","journal-title":"Soft. Qual. J."},{"issue":"4","key":"523_CR90","doi-asserted-by":"publisher","first-page":"2559","DOI":"10.1109\/TSC.2022.3216539","volume":"16","author":"S Valluripally","year":"2023","unstructured":"Valluripally, S., Frailey, B., Kruse, B., et al.: Detection of security and privacy attacks disrupting user immersive experience in virtual reality learning environments. IEEE Trans. Serv. Comput. 16(4), 2559\u20132574 (2023). https:\/\/doi.org\/10.1109\/TSC.2022.3216539","journal-title":"IEEE Trans. Serv. Comput."},{"issue":"3","key":"523_CR91","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1771","volume":"31","author":"TEJ Vos","year":"2021","unstructured":"Vos, T.E.J., Aho, P., Pastor Ricos, F., et al.: testar - scriptless testing through graphical user interface. Soft. Test. Verif. Reliab. 31(3), e1771 (2021). https:\/\/doi.org\/10.1002\/stvr.1771","journal-title":"Soft. Test. Verif. Reliab."},{"issue":"7","key":"523_CR92","first-page":"17","volume":"27","author":"AE Walsh","year":"2002","unstructured":"Walsh, A.E.: Understanding scene graphs. DrDobb\u2019s Journal 27(7), 17\u201326 (2002)","journal-title":"Understanding scene graphs. DrDobb\u2019s Journal"},{"key":"523_CR93","doi-asserted-by":"publisher","unstructured":"Wang, X.: VRTest: An extensible framework for automatic testing of virtual reality scenes. In: Intl. Conf. on software engineering: companion proceedings (ICSE-Companion). ACM, pp. 232\u2013236 (2022). https:\/\/doi.org\/10.1145\/3510454.3516870","DOI":"10.1145\/3510454.3516870"},{"key":"523_CR94","doi-asserted-by":"publisher","unstructured":"Wang, X., Rafi, T., Meng, N.: Vrguide: Efficient testing of virtual reality scenes via dynamic cut coverage. In: 2023 38th IEEE\/ACM Intl. Conf. on automated software engineering (ASE). IEEE Computer Society, pp. 951\u2013962 (2023). https:\/\/doi.org\/10.1109\/ASE56229.2023.00197","DOI":"10.1109\/ASE56229.2023.00197"},{"key":"523_CR95","unstructured":"Washizaki, H. (ed) Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. IEEE Computer Society (2024) . https:\/\/www.swebok.org"},{"key":"523_CR96","doi-asserted-by":"publisher","unstructured":"Wei, Z., Xinxin, G.: The collision detection algorithm in virtual reality. In: 2012 Intl. Conf. on computer science and electronics engineering, pp. 538\u2013541 (2012). https:\/\/doi.org\/10.1109\/ICCSEE.2012.412","DOI":"10.1109\/ICCSEE.2012.412"},{"key":"523_CR97","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1007\/s00766-005-0021-6","volume":"11","author":"R Wieringa","year":"2006","unstructured":"Wieringa, R., Maiden, N., Mead, N., et al.: Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. Requir Eng 11, 102\u2013107 (2006). https:\/\/doi.org\/10.1007\/s00766-005-0021-6","journal-title":"Requir Eng"},{"key":"523_CR98","doi-asserted-by":"publisher","unstructured":"Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proc. of the 18th Intl. Conf. on evaluation and assessment in software engineering. ACM, EASE \u201914 (2014). https:\/\/doi.org\/10.1145\/2601248.2601268","DOI":"10.1145\/2601248.2601268"},{"key":"523_CR99","doi-asserted-by":"publisher","unstructured":"Xu, P., Sun, Q.: Virtual reality collision detection based on improved ant colony algorithm. Appl. Sci. 13(11) (2023). https:\/\/doi.org\/10.3390\/app13116366","DOI":"10.3390\/app13116366"},{"key":"523_CR100","unstructured":"Tramontana, P., Luca, M.D., Fasolino, A.R.: An approach for model based testing of augmented reality applications. In: RCIS Workshops (2022)"},{"issue":"1","key":"523_CR101","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s40708-022-00172-6","volume":"9","author":"AHX Yang","year":"2022","unstructured":"Yang, A.H.X., Kasabov, N., Cakmak, Y.O.: Machine learning methods for the study of cybersickness: A systematic review. Brain Inf. 9(1), 24 (2022). https:\/\/doi.org\/10.1186\/s40708-022-00172-6","journal-title":"Brain Inf."},{"key":"523_CR102","doi-asserted-by":"publisher","unstructured":"Yang, X., Zhang, X.: A study of user privacy in android mobile ar apps. in: proc. of the 37th ieee\/acm intl. conf. on Automated Software Engineering (ASE\u201922). ACM (2023). https:\/\/doi.org\/10.1145\/3551349.3560512","DOI":"10.1145\/3551349.3560512"},{"key":"523_CR103","doi-asserted-by":"publisher","unstructured":"Yang, X., Wang, Y., Rafi, T., et al.: Towards automatic oracle prediction for ar testing: Assessing virtual object placement quality under real-world scenes. In: Proc. of the 33rd ACM SIGSOFT Intl. symposium on software testing and analysis. ACM, ISSTA 2024, pp. 717\u2013729 (2024). https:\/\/doi.org\/10.1145\/3650212.3680315","DOI":"10.1145\/3650212.3680315"},{"key":"523_CR104","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1016\/j.jss.2016.03.065","volume":"117","author":"S Zein","year":"2016","unstructured":"Zein, S., Salleh, N., Grundy, J.: A systematic mapping study of mobile application testing techniques. J. Syst. Sof. 117, 334\u2013356 (2016). https:\/\/doi.org\/10.1016\/j.jss.2016.03.065","journal-title":"J. Syst. Sof."},{"key":"523_CR105","doi-asserted-by":"publisher","unstructured":"Zhang, M., Zhou, D., Lv, C., et al.: Collision detection technology based on capsule model in virtual maintenance. In: Intl. Conf. on Reliability, Maintainability and Safety (ICRMS), pp. 1150\u20131155 (2014). https:\/\/doi.org\/10.1109\/ICRMS.2014.7107384","DOI":"10.1109\/ICRMS.2014.7107384"},{"issue":"3","key":"523_CR106","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1109\/TSE.2022.3170122","volume":"49","author":"X Zhang","year":"2023","unstructured":"Zhang, X., Tao, J., Tan, K., et al.: Finding critical scenarios for automated driving systems: A systematic mapping study. IEEE Trans. Soft. Eng. 49(3), 991\u20131026 (2023). https:\/\/doi.org\/10.1109\/TSE.2022.3170122","journal-title":"IEEE Trans. Soft. Eng."},{"key":"523_CR107","doi-asserted-by":"publisher","unstructured":"Zheng, Y., Xie, X., Su, T., et al.: Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In: 2019 34th IEEE\/ACM Intl. Conf. on automated software engineering (ASE), pp. 772\u2013784 (2019). https:\/\/doi.org\/10.1109\/ASE.2019.00077","DOI":"10.1109\/ASE.2019.00077"}],"container-title":["Automated Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-025-00523-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10515-025-00523-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-025-00523-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T13:57:12Z","timestamp":1757512632000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10515-025-00523-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,3]]},"references-count":107,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["523"],"URL":"https:\/\/doi.org\/10.1007\/s10515-025-00523-7","relation":{},"ISSN":["0928-8910","1573-7535"],"issn-type":[{"value":"0928-8910","type":"print"},{"value":"1573-7535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,3]]},"assertion":[{"value":"21 April 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 June 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"56"}}