{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T08:48:53Z","timestamp":1782204533840,"version":"3.54.5"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T00:00:00Z","timestamp":1777680000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T00:00:00Z","timestamp":1777680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100012305","name":"Universit\u00e0 degli Studi del Molise","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100012305","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Voice-based virtual assistants enable hands-free operation, allowing users to perform tasks, access information, and control smart home devices through simple voice commands. Their growing ubiquity in smartphones, smart speakers, and other devices led to the flourish of more and more apps taking advantage of a Voice User Interface (VUI). VUI testing is far from trivial due to the wide variability in human speech (e.g.,\u00a0different accents, dialects, speech patterns), and the fact that users can express the same command in numerous ways, using different (but semantically equivalent) wordings and phrases. For this reason, techniques have been proposed to support VUI testing. The basic idea behind these specialized approaches is to generate paraphrases for the set of voice commands for which developers implemented support in the VUI. Preliminary results from a recent study suggest that specialized models can outperform a general-purpose LLM (ChatGPT). However, a simple prompt and interaction strategy with ChatGPT has been adopted. In other words, it is still unknown whether optimizing the LLM usage allows to obtain better results. In this paper, we aim to thoroughly study to what extent LLMs (ChatGPT, specifically) can be adopted to test VUIs. We focused on optimizing the used prompt and the interaction with the model. Our results show that an optimized use of LLMs results in new state-of-the-art performance for VUI testing in terms of number of correct and bug-revealing paraphrases. While introducing the generated paraphrases into the Voice Interaction Models of the skills allows to fix some bugs, we observe that many bugs remain, and some are even introduced by the generated paraphrases. Our results call for specialized approaches for fixing bugs in VUIs.<\/jats:p>","DOI":"10.1007\/s10664-026-10859-7","type":"journal-article","created":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T06:56:29Z","timestamp":1777704989000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Using large language models to test voice user interfaces"],"prefix":"10.1007","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5443-1303","authenticated-orcid":false,"given":"Emanuela","family":"Guglielmi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angelica","family":"Spina","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriele","family":"Bavota","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rocco","family":"Oliveto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simone","family":"Scalabrino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,2]]},"reference":[{"key":"10859_CR1","unstructured":"Alexa A (2025) OUse LLMs to improve accuracy in your Interaction Model skill. https:\/\/developer.amazon.com\/en-US\/alexa\/alexa-haus\/LLM-utterances"},{"key":"10859_CR2","unstructured":"Amazon (2018) Amazon official documentation. https:\/\/www.amazon.com\/alexa-skills\/b?ie=UTF8&node=13727921011"},{"key":"10859_CR3","unstructured":"Chen J, Zhenhao L, Xing H, Xin X (2024) NLPerturbator: Studying the robustness of code LLMs to natural language variations. ACM Trans Softw Eng Methodol"},{"key":"10859_CR4","doi-asserted-by":"crossref","unstructured":"Denny P, Kumar V, Giacaman N (2023) Conversing with copilot: Exploring prompt engineering for solving CS1 problems using natural language. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp 1136\u20131142","DOI":"10.1145\/3545945.3569823"},{"key":"10859_CR5","doi-asserted-by":"crossref","unstructured":"Egli A (2023) ChatGPT, GPT-4, and other large language models: The next revolution for clinical microbiology? Clinical Infectious Diseases p ciad407","DOI":"10.1093\/cid\/ciad407"},{"key":"10859_CR6","doi-asserted-by":"crossref","unstructured":"Guglielmi E, Rosa G, Scalabrino S, Bavota G, Oliveto R (2022) Sorry, I Don\u2019t understand: Improving voice user interface testing. In: 37th IEEE\/ACM International conference on automated software engineering, pp 1\u201312","DOI":"10.1145\/3551349.3556934"},{"key":"10859_CR7","doi-asserted-by":"crossref","unstructured":"Guglielmi E, Rosa G, Scalabrino S, Bavota G, Oliveto R (2023) Help them understand: Testing and improving voice user interfaces. ACM Trans Softw Eng Methodol","DOI":"10.1145\/3654438"},{"key":"10859_CR8","unstructured":"Guglielmi E, Spina A, Bavota G, Oliveto R, Scalabrino S (2024) Replication Package. https:\/\/figshare.com\/s\/1d7d555e54c466fd4d2e"},{"key":"10859_CR9","doi-asserted-by":"crossref","unstructured":"Guichard J, Ruane E, Smith R, Bean D, Ventresque A (2019) Assessing the robustness of conversational agents using paraphrases. In: 2019 IEEE International Conference On Artificial Intelligence Testing (AITest), IEEE, pp 55\u201362","DOI":"10.1109\/AITest.2019.000-7"},{"key":"10859_CR10","unstructured":"Guo Z, Lin Z, Li P, Chen K (2020) $$\\{\\text{SkillExplorer}\\}$$: Understanding the behavior of skills in large scale. In: 29th USENIX Security Symposium (USENIX Security 20), pp 2649\u20132666"},{"key":"10859_CR11","unstructured":"Hegde C, Patil S (2020) Unsupervised paraphrase generation using pre-trained language models. arXiv:2006.05477"},{"key":"10859_CR12","doi-asserted-by":"crossref","unstructured":"Li S, Bu L, Bai G, Guo Z, Chen K, Wei H (2022) VITAS: Guided model-based VUI testing of VPA Apps. In: 37th IEEE\/ACM International conference on automated software engineering, pp 1\u201312","DOI":"10.1145\/3551349.3556957"},{"key":"10859_CR13","unstructured":"Li S, Bu L, Bai G, Xie F, Chen K, Yue C (2024) Model-enhanced LLM-driven VUI testing of VPA Apps. arXiv:2407.02791"},{"key":"10859_CR14","doi-asserted-by":"crossref","unstructured":"Liao S, Cheng L, Cai H, Guo L, Hu H (2023) SkillScanner: Detecting policy-violating voice applications through static analysis at the development phase. In: Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pp 2321\u20132335","DOI":"10.1145\/3576915.3616650"},{"key":"10859_CR15","doi-asserted-by":"crossref","unstructured":"Liu V, Chilton LB (2022) Design guidelines for prompt engineering text-to-image generative models. In: Proceedings of the 2022 CHI conference on human factors in computing systems, pp 1\u201323","DOI":"10.1145\/3491102.3501825"},{"key":"10859_CR16","unstructured":"Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692"},{"issue":"4","key":"10859_CR17","first-page":"203","volume":"27","author":"LS Lo","year":"2023","unstructured":"Lo LS (2023) The art and science of prompt engineering: a new literacy in the information age. Internet Ref Serv Q 27(4):203\u2013210","journal-title":"Internet Ref Serv Q"},{"issue":"2","key":"10859_CR18","doi-asserted-by":"publisher","first-page":"545","DOI":"10.11144\/Javeriana.upsy10-2.cdcp","volume":"10","author":"G Macbeth","year":"2011","unstructured":"Macbeth G, Razumiejczyk E, Ledesma RD (2011) Cliff\u2019s delta calculator: A non-parametric effect size program for two groups of observations. Universitas Psychologica 10(2):545\u2013555","journal-title":"Universitas Psychologica"},{"key":"10859_CR19","unstructured":"Meier D, Wahle JP, Ruas T, Gipp B (2024) Towards human understanding of paraphrase types in ChatGPT. arXiv:2407.02302"},{"key":"10859_CR20","unstructured":"OpenAI (2024a) ntroducing OpenAI o1. https:\/\/openai.com\/o1\/"},{"key":"10859_CR21","unstructured":"OpenAI (2024b) OpenAI o3-mini. https:\/\/openai.com\/index\/openai-o3-mini\/"},{"key":"10859_CR22","doi-asserted-by":"crossref","unstructured":"Reynolds L, McDonell K (2021) Prompt programming for large language models: Beyond the few-shot paradigm. In: Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp 1\u20137","DOI":"10.1145\/3411763.3451760"},{"issue":"6","key":"10859_CR23","doi-asserted-by":"publisher","first-page":"192","DOI":"10.3390\/fi15060192","volume":"15","author":"KI Roumeliotis","year":"2023","unstructured":"Roumeliotis KI, Tselikas ND (2023) ChatGPT and Open-AI models: A preliminary review. Future Internet 15(6):192","journal-title":"Future Internet"},{"issue":"9","key":"10859_CR24","doi-asserted-by":"publisher","first-page":"805","DOI":"10.1109\/TSE.2016.2532875","volume":"42","author":"S Segura","year":"2016","unstructured":"Segura S, Fraser G, Sanchez AB, Ruiz-Cort\u00e9s A (2016) A survey on metamorphic testing. IEEE Trans Software Eng 42(9):805\u2013824","journal-title":"IEEE Trans Software Eng"},{"issue":"4","key":"10859_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3432233","volume":"4","author":"FH Shezan","year":"2020","unstructured":"Shezan FH, Hu H, Wang G, Tian Y (2020) VerHealth: Vetting medical voice applications through policy enforcement. Proceed ACM on Interact Mobile Wearable Ubiquit Technol 4(4):1\u201321","journal-title":"Proceed ACM on Interact Mobile Wearable Ubiquit Technol"},{"key":"10859_CR26","unstructured":"Wang S, Li Z, Qian H, Yang C, Wang Z, Shang M, Kumar V, Tan S, Ray B, Bhatia P et\u00a0al (2022) ReCode: Robustness evaluation of code generation models. arXiv:2212.10264"},{"key":"10859_CR27","unstructured":"Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, Yogatama D, Bosma M, Zhou D, Metzler D et\u00a0al (2022) Emergent abilities of large language models. arXiv:2206.07682"},{"key":"10859_CR28","doi-asserted-by":"crossref","unstructured":"Witteveen S, Andrews M (2019) Paraphrasing with large language models. arXiv:1911.09661","DOI":"10.18653\/v1\/D19-5623"},{"key":"10859_CR29","doi-asserted-by":"crossref","unstructured":"Woolson RF (2007) Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials pp 1\u20133","DOI":"10.1002\/9780471462422.eoct979"},{"key":"10859_CR30","doi-asserted-by":"crossref","unstructured":"Xu Z, Wang C, Qiu M, Luo F, Xu R, Huang S, Huang J (2023) Making pre-trained language models end-to-end few-shot learners with contrastive prompt tuning. In: Proceedings of the Sixteenth ACM international conference on web search and data mining, pp 438\u2013446","DOI":"10.1145\/3539597.3570398"},{"issue":"12","key":"10859_CR31","doi-asserted-by":"publisher","first-page":"7165","DOI":"10.1007\/s10115-024-02199-z","volume":"66","author":"G Yigit","year":"2024","unstructured":"Yigit G, Amasyali MF (2024) From text to multimodal: a survey of adversarial example generation in question answering systems. Knowl Inf Syst 66(12):7165\u20137204","journal-title":"Knowl Inf Syst"},{"key":"10859_CR32","unstructured":"Young J, Liao S, Cheng L, Hu H, Deng H (2022) $$\\{\\text{ SkillDetective }\\}$$: Automated $$\\{\\text{ Policy-Violation }\\}$$ detection of voice assistant applications in the wild. In: 31st USENIX security symposium (USENIX Security 22)"},{"key":"10859_CR33","doi-asserted-by":"crossref","unstructured":"Zhang Y, Xu L, Mendoza A, Yang G, Chinprutthiwong P, Gu G (2019) Life after speech recognition: Fuzzing semantic misinterpretation for voice assistant applications. In: Proc. of the Network and Distributed System Security Symposium (NDSS\u201919)","DOI":"10.14722\/ndss.2019.23525"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10859-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-026-10859-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10859-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T08:07:33Z","timestamp":1782202053000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-026-10859-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,2]]},"references-count":33,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["10859"],"URL":"https:\/\/doi.org\/10.1007\/s10664-026-10859-7","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,2]]},"assertion":[{"value":"24 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 March 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}}],"article-number":"132"}}