{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T07:17:30Z","timestamp":1781335050225,"version":"3.54.1"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"name":"Research Grants Council of the Hong Kong Special Administrative Region, China","award":["Theme-based Research Scheme Project Ref. No. T43-513\/23-N"],"award-info":[{"award-number":["Theme-based Research Scheme Project Ref. No. T43-513\/23-N"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 62472126"],"award-info":[{"award-number":["No. 62472126"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Shenzhen-Hong Kong Jointly Funded Project","award":["Category A, No. SGDX20230116091246007"],"award-info":[{"award-number":["Category A, No. SGDX20230116091246007"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>In recent years, the extended reality (XR) software ecosystem has emerged as the next ubiquitous computing platform, as they provide users with immersive interactive experiences. However, XR ecosystem suffers from cybersickness problems, which would greatly affect user comfort and safety, leading to symptoms like headaches, disorientation, etc. That makes effective cybersickness assessment a timely and important question. The state-of-the-art methods of assessing the cybersickness of XR software typically monitor the biological metrics of users, during XR usage, which rely heavily on manual playtesting,sufferring from limited scalability issue. User reviews on XR app stores are informative for developers to learn the cybersickness ratings of their apps and the reasons behind. Nevertheless, the large number of user reviews can hardly be analyzed by developers manually, and most current automatic user review analysis methods can only provide coarse-grained or abstract results, such as extracting several high-level key topic groups discussed by reviews. Recent advancements in LLMs may bring new opportunities. However, directly leveraging LLMs for evaluating XR cybersickness is challenging because LLMs perform poorly on a large number of short texts, and their context window is limited. To address these challenges, we introduce XRCare, a comprehensive framework designed to automate the assessment of cybersickness and root cause reasoning for XR apps by resorting to fine-grained user review analysis. XRCare mainly includes three phases: (1) Insight pool construction, when XRCare collects the cybersickness analyzing chains and corresponding analyzing results from domain experts; (2) Reasoning graph construction, when XRCare dynamically extracts, categorizes, and maintains the reasons from user reviews that make users feel cybersickness on a self-evolving hierarchical graph; and (3) Multi-agent deductive cybersickness reasoning, which utilizes a multi-agent system to simulate diverse user demographics for analyzing and rating the intensity of cybersickness, as well as the causes of cybersickness. This structured approach allows XRCare to systematically identify, categorize, and address instances of cybersickness. For experiments, we construct a large-scale dataset consisting of 685,111 user reviews from 9,667 XR apps. Our evaluation shows that XRCare enhances the F1-score by 20.63% over the best-performing baseline and by 32.27% on average across all baselines, while also offering more accurate and detailed interpretability insights.<\/jats:p>","DOI":"10.1145\/3728933","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1303-1325","source":"Crossref","is-referenced-by-count":1,"title":["Extended Reality Cybersickness Assessment via User Review Analysis"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6323-1402","authenticated-orcid":false,"given":"Shuqing","family":"Li","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7198-0866","authenticated-orcid":false,"given":"Qisheng","family":"Zheng","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4774-2434","authenticated-orcid":false,"given":"Cuiyun","family":"Gao","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5509-4724","authenticated-orcid":false,"given":"Jia","family":"Feng","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3666-5798","authenticated-orcid":false,"given":"Michael R.","family":"Lyu","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. Claude. https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet"},{"key":"e_1_2_1_2_1","unstructured":"2024. JamSpell. https:\/\/github.com\/bakwc\/JamSpell"},{"key":"e_1_2_1_3_1","unstructured":"2024. Oculus App Store. https:\/\/www.meta.com\/experiences\/"},{"key":"e_1_2_1_4_1","unstructured":"2024. Sidequest App Store. https:\/\/sidequestvr.com\/"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.3115\/1219044.1219075"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3670008"},{"key":"e_1_2_1_7_1","first-page":"993","article-title":"Latent dirichlet allocation","author":"Blei David M","year":"2003","unstructured":"David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3, Jan (2003), 993\u20131022.","journal-title":"Journal of machine Learning research, 3"},{"key":"e_1_2_1_8_1","volume-title":"Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.","author":"Bubeck S\u00e9bastien","year":"2023","unstructured":"S\u00e9bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. March, https:\/\/www.microsoft.com\/en-us\/research\/publication\/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4\/"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1778351"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622407.1622416"},{"key":"e_1_2_1_11_1","volume-title":"International journal of neural systems, 26, 02","author":"Chuang Shang-Wen","year":"2016","unstructured":"Shang-Wen Chuang, Chun-Hsiang Chuang, Yi-Hsin Yu, Jung-Tai King, and Chin-Teng Lin. 2016. EEG alpha and gamma modulators mediate motion sickness-related spectral responses. International journal of neural systems, 26, 02 (2016), 1650007."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00949"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Tamara Czinczoll Christoph H\u00f6nes Maximilian Schall and Gerard de Melo. 2024. NextLevelBERT: Investigating Masked Language Modeling with Higher-Level Representations for Long Documents. arXiv preprint arXiv:2402.17682.","DOI":"10.18653\/v1\/2024.acl-long.256"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1423"},{"key":"e_1_2_1_15_1","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang and Angela Fan. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561155"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642941"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2021.3057288"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2013.2294642"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.INFSOF.2021.106741"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3076179"},{"key":"e_1_2_1_22_1","unstructured":"Team GLM Aohan Zeng Bin Xu Bowen Wang Chenhui Zhang Da Yin Diego Rojas Guanyu Feng Hanlin Zhao and Hanyu Lai. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv preprint arXiv:2406.12793."},{"key":"e_1_2_1_23_1","unstructured":"Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794."},{"key":"e_1_2_1_24_1","volume-title":"2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 760\u2013770","author":"Gu Xiaodong","year":"2015","unstructured":"Xiaodong Gu and Sunghun Kim. 2015. \" what parts of your apps are loved by users?\"(T). In 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 760\u2013770."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380924"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Ankur Gupta Sahil Sawhney and Kashyap Kompella. 2024. The First Principles: Setting the Context for a Safe and Secure Metaverse. Comput. Surveys.","DOI":"10.1145\/3665495"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025665"},{"key":"e_1_2_1_28_1","unstructured":"Aaron Hurst Adam Lerer Adam P Goucher Adam Perelman Aditya Ramesh Aidan Clark AJ Ostrow Akila Welihinda Alan Hayes and Alec Radford. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/E17-2068"},{"key":"e_1_2_1_30_1","volume-title":"A systematic review of virtual reality in education. Themes in science and technology education, 10, 2","author":"Kavanagh Sam","year":"2017","unstructured":"Sam Kavanagh, Andrew Luxton-Reilly, Burkhard Wuensche, and Beryl Plimmer. 2017. A systematic review of virtual reality in education. Themes in science and technology education, 10, 2 (2017), 85\u2013119."},{"key":"e_1_2_1_31_1","volume-title":"Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The international journal of aviation psychology, 3, 3","author":"Kennedy Robert S","year":"1993","unstructured":"Robert S Kennedy, Norman E Lane, Kevin S Berbaum, and Michael G Lilienthal. 1993. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The international journal of aviation psychology, 3, 3 (1993), 203\u2013220."},{"key":"e_1_2_1_32_1","volume-title":"Validating an efficient method to quantify motion sickness. Human factors, 53, 4","author":"Keshavarz Behrang","year":"2011","unstructured":"Behrang Keshavarz and Heiko Hecht. 2011. Validating an efficient method to quantify motion sickness. Human factors, 53, 4 (2011), 415\u2013426."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.21037\/atm.2016.12.23"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-021-00517-2"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/333329.333344"},{"key":"e_1_2_1_36_1","volume-title":"AdCube: WebVR Ad Fraud and Practical Confinement of Third-Party Ads. In USENIX Security Symposium. USENIX Association, 2543\u20132560","author":"Lee Hyunjoo","year":"2021","unstructured":"Hyunjoo Lee, Jiyeon Lee, Daejun Kim, Suman Jana, Insik Shin, and Sooel Son. 2021. AdCube: WebVR Ad Fraud and Practical Confinement of Third-Party Ads. In USENIX Security Symposium. USENIX Association, 2543\u20132560."},{"key":"e_1_2_1_37_1","unstructured":"Quinn Leng Jacob Portes Sam Havens Matei Zaharia and Michael Carbin. 2024. Long context rag performance of large language models. arXiv preprint arXiv:2411.03538."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3660803"},{"key":"e_1_2_1_39_1","unstructured":"Shuqing Li Binchang Li Yepang Liu Cuiyun Gao Jianping Zhang Shing-Chi Cheung and Michael R Lyu. 2024. Grounded GUI Understanding for Vision Based Spatial Intelligent Agent: Exemplified by Virtual Reality Apps. arXiv preprint arXiv:2409.10811."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2308.06783"},{"key":"e_1_2_1_41_1","volume-title":"2020 IEEE 31st International symposium on software reliability engineering (ISSRE). 172\u2013183","author":"Li Shuqing","year":"2020","unstructured":"Shuqing Li, Yechang Wu, Yi Liu, Dinghua Wang, Ming Wen, Yida Tao, Yulei Sui, and Yepang Liu. 2020. An exploratory study of bugs in extended reality applications on the web. In 2020 IEEE 31st International symposium on software reliability engineering (ISSRE). 172\u2013183."},{"key":"e_1_2_1_42_1","unstructured":"Shuqing Li Chenran Zhang Cuiyun Gao and Michael R Lyu. 2024. XRZoo: A Large-Scale and Versatile Dataset of Extended Reality (XR) Applications. arXiv preprint arXiv:2412.06759."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2980\u20132988","author":"Lin Tsung-Yi","year":"2017","unstructured":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. 2017. Focal Loss for Dense Object Detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2980\u20132988."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77414-1_26"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1162\/pres.1992.1.3.311"},{"key":"e_1_2_1_46_1","volume-title":"Software Quality: The Complexity and Challenges of Software Engineering and Software Quality in the Cloud: 11th International Conference, SWQD","author":"Moser Thomas","year":"2019","unstructured":"Thomas Moser, Markus Hohlagschwandtner, Gerhard Kormann-Hainzl, Sabine P\u00f6lzlbauer, and Josef Wolfartsberger. 2019. Mixed reality applications in industry: challenges and research areas. In Software Quality: The Complexity and Challenges of Software Engineering and Software Quality in the Cloud: 11th International Conference, SWQD 2019, Vienna, Austria, January 15\u201318, 2019, Proceedings 11. 95\u2013105."},{"key":"e_1_2_1_47_1","volume-title":"The Virtual Reality Head-Mounted Display Oculus Rift Induces Motion Sickness and is Sexist in Its Effects. Experimental brain research, 235","author":"Munafo Justin","year":"2017","unstructured":"Justin Munafo, Meg Diedrick, and Thomas A Stoffregen. 2017. The Virtual Reality Head-Mounted Display Oculus Rift Induces Motion Sickness and is Sexist in Its Effects. Experimental brain research, 235 (2017), 889\u2013901."},{"key":"e_1_2_1_48_1","unstructured":"Qualcomm Developer Network. [n. d.]. Reducing Cybersickness in Virtual Reality. https:\/\/developer.qualcomm.com\/system\/files\/ebook_cybersickness_qdn_vfinal2.pdf"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0003-6870(02)00020-0"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2893171"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00052"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581315"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.18"},{"key":"e_1_2_1_54_1","doi-asserted-by":"crossref","unstructured":"Raghavendra Pappagari Piotr Zelasko Jes\u00fas Villalba Yishay Carmiel and Najim Dehak. 2019. Hierarchical transformers for long document classification. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU). 838\u2013844.","DOI":"10.1109\/ASRU46091.2019.9003958"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376847"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb046814"},{"key":"e_1_2_1_57_1","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans and Ilya Sutskever. 2018. Improving language understanding by generative pre-training."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-020-00446-6"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/361219.361220"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950299"},{"key":"e_1_2_1_61_1","unstructured":"Statista. 2022. Report of Active Virtual Reality Users Worldwide. https:\/\/www.statista.com\/statistics\/426469\/active-virtual-reality-users-worldwide\/"},{"key":"e_1_2_1_62_1","volume-title":"OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR. In USENIX Security Symposium. USENIX Association, 3789\u20133806","author":"Trimananda Rahmadi","year":"2022","unstructured":"Rahmadi Trimananda, Hieu Le, Hao Cui, and Janice Tran Ho. 2022. OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR. In USENIX Security Symposium. USENIX Association, 3789\u20133806."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510201"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","unstructured":"Miroslav Tushev Fahimeh Ebrahimi and Anas Mahmoud. 2022. Domain-Specific Analysis of Mobile App Reviews Using Keyword-Assisted Topic Models. In ICSE. ACM 762\u2013773. https:\/\/doi.org\/10.1145\/3510003.3510201 10.1145\/3510003.3510201","DOI":"10.1145\/3510003.3510201"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-89817-5_7"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","unstructured":"Lili Wei Yepang Liu and Shing-Chi Cheung. 2017. OASIS: prioritizing static analysis warnings for Android apps based on app user reviews. In ESEC\/FSE. ACM 672\u2013682. https:\/\/doi.org\/10.1145\/3106237.3106294 10.1145\/3106237.3106294","DOI":"10.1145\/3106237.3106294"},{"key":"e_1_2_1_67_1","unstructured":"David Matthew Whittinghill Bradley Ziegler T Case and B Moore. 2015. Nasum virtualis: A simple technique for reducing simulator sickness. In Games developers conference (GDC). 74."},{"key":"e_1_2_1_68_1","volume-title":"Reducing head mounted display VR sickness through dynamic field of view constriction. Ph. D. Dissertation","author":"Yaqub Hashim","unstructured":"Hashim Yaqub. 2018. Reducing head mounted display VR sickness through dynamic field of view constriction. Ph. D. Dissertation. University of Bath, Somerset, UK. https:\/\/ethos.bl.uk\/OrderDetails.do?uin=uk.bl.ethos.761035"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642143"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728933","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:51:46Z","timestamp":1752684706000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728933"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":69,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728933"],"URL":"https:\/\/doi.org\/10.1145\/3728933","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}