{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T15:03:39Z","timestamp":1781708619579,"version":"3.54.5"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"name":"National Key Research and Development Program of China","award":["2024YFB2505604"],"award-info":[{"award-number":["2024YFB2505604"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62232008 and 62172200"],"award-info":[{"award-number":["62232008 and 62172200"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"IIE, Chinese Academy of Science","award":["U24A20236"],"award-info":[{"award-number":["U24A20236"]}]},{"name":"CAS Project for Young Scientists in Basic Research","award":["YSBR-118"],"award-info":[{"award-number":["YSBR-118"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62572256"],"award-info":[{"award-number":["62572256"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>With the increasing adoption of smart speakers, virtual personal assistant (VPA) applications have become integral to daily life, enabling users to access news, entertainment, and smart device control through voice user interfaces (VUI). However, many VPA apps suffer from quality issues, such as unexpected terminations and failures to process common user commands, highlighting the urgent need for systematic and efficient VUI testing. Existing chatbot-style and model-based testing approaches lack global and semantic awareness, resulting in ineffective test case generation and inefficient state exploration.<\/jats:p>\n                  <jats:p>To address these challenges, we introduce Elevate, a model-enhanced, large language model (LLM)-driven VPA testing framework that employs a multi-agent architecture to enhance VUI behavior testing. Elevate comprises three specialized LLM agents\u2014Observer, Generator, and Planner\u2014that collaboratively perform state extraction, test case generation, and guided state exploration. Additionally, a deterministic finite automaton (DFA)-based behavior model is designed to abstract app behavior and provide structured guidance to LLM agents, enhancing testing performance. Elevate also incorporates a feedback mechanism that refines testing strategies based on observed behaviors, ensuring continuous improvement.<\/jats:p>\n                  <jats:p>Implemented using GPT-4-Turbo and DeepSeek-R1, Elevate has been evaluated on problem detection, sentence\/semantic coverage, and large-scale testing. Experimental results show that Elevate outperforms state-of-the-art methods (Vitas and LLM-based chatbots), detecting at least 18 and 37 more problems, respectively, and achieving over 10% and 30% higher state coverage. In a large-scale evaluation on 4,000 Alexa skills, Elevate further demonstrated 15% higher coverage than Vitas, confirming its effectiveness, scalability, and potential for widespread application in VUI testing.<\/jats:p>","DOI":"10.1145\/3771555","type":"journal-article","created":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:50:12Z","timestamp":1760104212000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["VUI Testing of VPA Apps via Behavior Model-Enhanced LLM Agents"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1202-0220","authenticated-orcid":false,"given":"Suwan","family":"Li","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0517-7801","authenticated-orcid":false,"given":"Lei","family":"Bu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5598-4006","authenticated-orcid":false,"given":"Shangqing","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6390-9890","authenticated-orcid":false,"given":"Guangdong","family":"Bai","sequence":"additional","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5446-3081","authenticated-orcid":false,"given":"Fuman","family":"Xie","sequence":"additional","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5624-2987","authenticated-orcid":false,"given":"Kai","family":"Chen","sequence":"additional","affiliation":[{"name":"Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8589-8490","authenticated-orcid":false,"given":"Chang","family":"Yue","sequence":"additional","affiliation":[{"name":"Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,17]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Amazon.com Inc. or its affiliates. 2014. Amazon.com: Alexa Skills. Retrieved from https:\/\/www.amazon.com\/alexa-skills\/"},{"key":"e_1_3_3_3_2","unstructured":"Amazon.com Inc. or its affiliates. 2017. Alexa Simulator Limitations. Retrieved from https:\/\/developer.amazon.com\/en-US\/docs\/alexa\/devconsole\/test-your-skill.html#use-simulator"},{"key":"e_1_3_3_4_2","unstructured":"Willamette Week. 2018. Portland Family Says Their Amazon Alexa Recorded Private Conversations. Retrieved from https:\/\/www.wweek.com\/news\/2018\/05\/26\/portland-family-says-their-amazon-alexa-recorded-private-conversations-and-sent-them-to-a-random-contact-in-seattle\/"},{"key":"e_1_3_3_5_2","unstructured":"Google for Developers. 2021. Scenes \\( | \\) Conversational Actions \\( | \\) Google Developers. Retrieved from https:\/\/developers.google.com\/assistant\/conversational\/scenes"},{"key":"e_1_3_3_6_2","unstructured":"Statista. 2022. Total Number of Amazon Alexa Skills in Selected Countries as of January 2021. Retrieved from https:\/\/www.statista.com\/statistics\/917900\/selected-countries-amazon-alexa-skill-count\/"},{"key":"e_1_3_3_7_2","unstructured":"Fuman Xie Yanjun Zhang Hanlin Wei and Guangdong Bai. 2023. Amazon-Alexa-UQ-AAS21-Datasets. Retrieved from https:\/\/github.com\/xie00059\/Amazon-Alexa-UQ-AAS21-datasets\/tree\/main"},{"key":"e_1_3_3_8_2","unstructured":"OpenAI. 2023. GPT-4. Retrieved from https:\/\/openai.com\/gpt-4"},{"key":"e_1_3_3_9_2","unstructured":"Ming Xu. 2024. text2vec. Retrieved from https:\/\/github.com\/shibing624\/text2vec"},{"key":"e_1_3_3_10_2","unstructured":"DeepSeek. 2025. DeepSeek-R1. Retrieved from https:\/\/github.com\/deepseek-ai\/DeepSeek-R1"},{"key":"e_1_3_3_11_2","unstructured":"Suwan Li Lei Bu Shangqing Liu Guangdong Bai Fuman Xie Kai Chen and Chang Yue. 2025. Elevate. Retrieved from https:\/\/github.com\/Elevate-12\/Elevate-2025"},{"key":"e_1_3_3_12_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Chen Bei","year":"2023","unstructured":"Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code generation with generated tests. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372297.3423339"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598067"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623343"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3654438"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Marcia Ford and William Palmer. 2019. Alexa are you listening to me? An analysis of Alexa voice service network traffic. Personal and Ubiquitous Computing 23 1 (2019) 67\u201379.","DOI":"10.1007\/s00779-018-1174-x"},{"key":"e_1_3_3_18_2","first-page":"1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201922)","volume":"96","author":"Guglielmi Emanuela","year":"2022","unstructured":"Emanuela Guglielmi, Giovanni Rosa, Simone Scalabrino, Gabriele Bavota, and Rocco Oliveto. 2022. Sorry, I don\u2019t understand: Improving voice user interface testing. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201922). ACM, Article 96, 1\u201312."},{"key":"e_1_3_3_19_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu Y. K. Li et al. 2024. DeepSeek-Coder: When the large language model meets programming\u2014The rise of code intelligence. arXiv:2401.14196. Retrieved from https:\/\/arxiv.org\/abs\/2401.14196"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623306"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Qi Guo Xiaofei Xie Shangqing Liu Ming Hu Xiaohong Li and Lei Bu. 2025. Intention is all you need: Refining your code from your intention. arXiv:2502.08172. Retrieved from https:\/\/arxiv.org\/abs\/2502.08172","DOI":"10.1109\/ICSE55347.2025.00191"},{"key":"e_1_3_3_22_2","first-page":"2649","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIX Security \u201920)","author":"Guo Zhixiu","year":"2020","unstructured":"Zhixiu Guo, Zijin Lin, Pan Li, and Kai Chen. 2020. SkillExplorer: Understanding the behavior of skills in large scale. In Proceedings of the 29th USENIX Security Symposium (USENIX Security \u201920). Srdjan Capkun and Franziska Roesner (Eds.), USENIX Association, 2649\u20132666."},{"key":"e_1_3_3_23_2","unstructured":"J. Mathav Raj V. M. Kushala Harikrishna Warrier and Yogesh Gupta. 2024. Fine tuning LLM for Enterprise: Practical guidelines and recommendations. arXiv:2404.10779. Retrieved from https:\/\/arxiv.org\/abs\/2404.10779"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.5555\/3600270.3601883"},{"key":"e_1_3_3_25_2","unstructured":"Jiaolong Kong Mingfei Cheng Xiaofei Xie Shangqing Liu Xiaoning Du and Qi Guo. 2024. ContrastRepair: Enhancing conversation-based automated program repair via contrastive test case pairs. arXiv:2403.01971. Retrieved from https:\/\/arxiv.org\/abs\/2403.01971"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3729390"},{"key":"e_1_3_3_27_2","first-page":"33","volume-title":"Proceedings of the 27th USENIX Security Symposium (USENIX Security \u201918)","author":"Kumar Deepak","year":"2018","unstructured":"Deepak Kumar, Riccardo Paccagnella, Paul Murley, Eric Hennenfent, Joshua Mason, Adam Bates, and Michael Bailey. 2018. Skill squatting attacks on amazon Alexa. In Proceedings of the 27th USENIX Security Symposium (USENIX Security \u201918). William Enck and Adrienne Porter Felt (Eds.), USENIX Association, 33\u201347."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556957"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576915.3616650"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639180"},{"key":"e_1_3_3_33_2","unstructured":"Lezhi Ma Shangqing Liu Lei Bu Shangru Li Yida Wang and Yang Liu. 2024. SpecEval: Evaluating code comprehension in large language models via program specifications. arXiv:2409.12866. Retrieved from https:\/\/arxiv.org\/abs\/2409.12866"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624724"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3432233"},{"key":"e_1_3_3_36_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639107"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560416"},{"key":"e_1_3_3_39_2","first-page":"1113","volume-title":"Proceedings of the 31st USENIX Security Symposium (USENIX Security \u201922)","author":"Young Jeffrey","year":"2022","unstructured":"Jeffrey Young, Song Liao, Long Cheng, Hongxin Hu, and Huixing Deng. 2022. SkillDetective: Automated policy-violation detection of voice assistant applications in the wild. In Proceedings of the 31st USENIX Security Symposium (USENIX Security \u201922). Kevin R. B. Butler and Kurt Thomas (Eds.), USENIX Association, 1113\u20131130."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2019.00016"},{"key":"e_1_3_3_41_2","article-title":"Dangerous skills: Understanding and mitigating security risks of voice-controlled third-party functions on virtual personal assistant systems","author":"Zhang Nan","year":"2019","unstructured":"Nan Zhang, Xianghang Mi, Xuan Feng, Xiao Feng Wang, Yuan Tian, and Feng Qian. 2019. Dangerous skills: Understanding and mitigating security risks of voice-controlled third-party functions on virtual personal assistant systems. In Proceedings of the IEEE Symposium on Security and Privacy.","journal-title":"Proceedings of the IEEE Symposium on Security and Privacy"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23525"},{"key":"e_1_3_3_43_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3771555","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T14:50:38Z","timestamp":1781707838000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3771555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,17]]},"references-count":42,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3771555"],"URL":"https:\/\/doi.org\/10.1145\/3771555","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,17]]},"assertion":[{"value":"2025-03-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}