{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:04:31Z","timestamp":1750309471868,"version":"3.41.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,11,18]],"date-time":"2024-11-18T00:00:00Z","timestamp":1731888000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>Extracting skill utterances is a crucial step in testing and evaluating smart speaker skills. Previous works have proposed various techniques for extracting utterances from the skill web page to test and evaluate smart speaker skills. In this article, we evaluate the effectiveness and correctness of different utterance-extracting techniques proposed in previous works. Our evaluation reveals that some techniques can capture incorrect utterances that the skill will not accept as spoken utterances. We also find that all of the proposed techniques would never capture the total utterances supported by skills, and combining these techniques yields the best results in terms of text parsing. To address these limitations, we propose a new technique that combines the strengths of previous techniques and leverages human to interact with a small set of skills to expand the coverage for testing other skills. We evaluate the effectiveness of our technique and demonstrate that it can capture a higher number of utterances supported by skills. We delved into the impact assessment of utterance extraction, aiming to enhance the thoroughness and effectiveness of skill testing. We conducted an impact study on 11 skills to assess the importance of utterance extraction in the context of skills testing. Our findings demonstrate that our technique captures a higher number of utterances compared to previous techniques. Through our evaluation, we provide insights into the significance of discovering more utterances in the testing context, and demonstrate the effectiveness of our proposed technique in capturing more utterances for skill testing and evaluation.<\/jats:p>","DOI":"10.1145\/3698200","type":"journal-article","created":{"date-parts":[[2024,9,28]],"date-time":"2024-09-28T09:07:43Z","timestamp":1727514463000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Enhancing Alexa Skill Testing Through Improved Utterance Discovery"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-5100","authenticated-orcid":false,"given":"Hassan A.","family":"Shafei","sequence":"first","affiliation":[{"name":"CIS, Temple University, Philadelphia, United States and College of Engineering and Computer Sciences, Jazan University, Jazan, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5758-6394","authenticated-orcid":false,"given":"Chiu C.","family":"Tan","sequence":"additional","affiliation":[{"name":"Temple University, Philadelphia, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,11,18]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Amazon. 2022. Alexa Store. Retrieved September 30 2024 from https:\/\/www.amazon.com\/alexa-skills"},{"key":"e_1_3_1_3_2","first-page":"96","volume-title":"Proceedings of the Computer-Supported Collaborative Learning Conference (CSCL\u201919)","volume":"1","author":"Beirl Diana","year":"2019","unstructured":"Diana Beirl, Y. Rogers, and Nicola Yuill. 2019. Using voice assistant skills in family life. In Proceedings of the Computer-Supported Collaborative Learning Conference (CSCL\u201919), Vol. 1. 96\u2013103."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376344"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3380993"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.2196\/11510"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.3390\/s21030734"},{"key":"e_1_3_1_8_2","unstructured":"Bret Kinsella. 2022. Google Home Added 600 000 More U.S. Users in 2018 Than Amazon Echo But Amazon Echo Dot Is Still the Most Owned Smart Speaker. Retrieved September 30 2024 from https:\/\/voicebot.ai\/2019\/03\/07\/google-home-added-600000-more-u-s-users-in-2018-than-amazon-echo-but-amazon-echo-dot-is-still-the-most-owned-smart-speaker\/"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555164"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376569"},{"key":"e_1_3_1_11_2","unstructured":"Google Cloud. 2023. Text-to-Speech. Retrieved September 30 2024 from https:\/\/cloud.google.com\/text-to-speech"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5935"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3098279.3098539"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocz152"},{"key":"e_1_3_1_15_2","unstructured":"Amazon Documentation. 2022. Building Alexa Skill. Retrieved September 30 2024 from https:\/\/developer.amazon.com\/en-US\/docs\/alexa\/custom-skills\/steps-to-build-a-custom-skill.html"},{"key":"e_1_3_1_16_2","unstructured":"Amazon Documentation. 2022. Skill Certification Requirements. Retrieved September 30 2024 from https:\/\/developer.amazon.com\/en-US\/docs\/alexa\/custom-skills\/certification-requirements-for-custom-skills.html#submission-checklist"},{"key":"e_1_3_1_17_2","article-title":"SkillVet: Automated traceability analysis of Amazon Alexa skills","author":"Edu Jide","year":"2021","unstructured":"Jide Edu, Xavi Ferrer Aran, Jose Such, and Guillermo Suarez-Tangil. 2021. SkillVet: Automated traceability analysis of Amazon Alexa skills. IEEE Transactions on Dependable and Secure Computing. Published Online, March 3, 2021.","journal-title":"IEEE Transactions on Dependable and Secure Computing."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3342775.3342809"},{"key":"e_1_3_1_19_2","unstructured":"Yolanda Parks. 2022. How Parents Can Use a Smart Speaker as a Homework Helper. Retrieved September 30 2024 from https:\/\/www.verizon.com\/about\/parenting\/smart-speaker-new-homework-helper#"},{"key":"e_1_3_1_20_2","unstructured":"Department of Health and Social Care. 2019. NHS Health Information Available Through Amazon\u2019s Alexa. Retrieved September 30 2024 from https:\/\/www.gov.uk\/government\/news\/nhs-health-information-available-through-amazon-s-alexa"},{"key":"e_1_3_1_21_2","unstructured":"Gecko. 2022. Spare the Air. Retrieved September 30 2024 from https:\/\/www.amazon.com\/Gecko-spare-the-air\/dp\/B01JG73CJI\/ref=sr_1_1"},{"key":"e_1_3_1_22_2","unstructured":"Google. 2022. Google Assistant. Retrieved September 30 2024 from https:\/\/assistant.google.com\/explore?hl=en_us"},{"key":"e_1_3_1_23_2","first-page":"2649","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920)","author":"Guo Zhixiu","unstructured":"Zhixiu Guo, Zijin Lin, Pan Li, and Kai Chen. 2020. SkillExplorer: Understanding the behavior of skills in large scale. In Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920). 2649\u20132666."},{"key":"e_1_3_1_24_2","volume-title":"Natural Language Processing: Python and NLTK","author":"Hardeniya Nitin","year":"2016","unstructured":"Nitin Hardeniya, Jacob Perkins, Deepti Chopra, Nisheeth Joshi, and Iti Mathur. 2016. Natural Language Processing: Python and NLTK. Packt Publishing Ltd."},{"key":"e_1_3_1_25_2","article-title":"The accent gap: How Amazon\u2019s and Google\u2019s smart speakers leave certain voices behind","volume":"19","author":"Harwell Drew","year":"2018","unstructured":"Drew Harwell. 2018. The accent gap: How Amazon\u2019s and Google\u2019s smart speakers leave certain voices behind. Washington Post 19 (2018).","journal-title":"Washington Post"},{"key":"e_1_3_1_26_2","unstructured":"Healthcare IT News. 2022. Special Report: AI Voice Assistants Making an Impact in Healthcare. Retrieved September 30 2024 from https:\/\/www.healthcareitnews.com\/news\/special-report-ai-voice-assistants-making-impact-healthcare"},{"key":"e_1_3_1_27_2","unstructured":"Kelly Crowe. 2018. Think twice before consulting Dr. Google researchers Say. Retrieved September 30 2024 from https:\/\/www.cbc.ca\/news\/health\/second-opinion-online-medical180407-1.4608410"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1080\/02763869.2018.1404391"},{"key":"e_1_3_1_29_2","volume-title":"Inequity in Popular Voice Recognition Systems Regarding African Accents","author":"Ike Chinaemere","year":"2020","unstructured":"Chinaemere Ike. 2020. Inequity in Popular Voice Recognition Systems Regarding African Accents. Ph.D. Dissertation. Texas A&M University."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1177\/2167702619894904"},{"key":"e_1_3_1_31_2","unstructured":"Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing. Pearson Prentice Hall."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5043"},{"key":"e_1_3_1_33_2","unstructured":"Christopher Lambert. 2017. Alexa Skill Database. Retrieved September 30 2024 from https:\/\/www.kaggle.com\/datasets\/theriley106\/alexa-skill-database"},{"key":"e_1_3_1_34_2","unstructured":"Bret Kinsella. 2021. Alexa skill counts surpass 80K in US Spain adds the most skills New skill introduction rate continues to fall across countries. Voicebot.ai. Retrieved September 30 2024 from https:\/\/voicebot.ai\/2021\/01\/14\/alexa-skill-counts-surpass-80k-in-us-spain-adds-the-most-skills-new-skill-introduction-rate-continues-to-fall-across-countries\/"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274371"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539609"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3427228.3427250"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-15742-5_9"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451595"},{"key":"e_1_3_1_40_2","unstructured":"David Lazarus. 2019. Column: Alexa may be key to Amazon\u2019s looming domination of the healthcare market. Los Angeles Times. Retrieved September 30 2024 from https:\/\/www.latimes.com\/business\/lazarus\/la-fi-lazarus-amazon-healthcare-privacy-20190716-story.html"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_3_1_42_2","first-page":"1","volume-title":"Proceedings of the 2019 IEEE International Conference on E-health Networking, Application, and Services (HealthCom\u201919)","author":"Maharjan Bikesh","year":"2019","unstructured":"Bikesh Maharjan, Juan Li, Jun Kong, and Cui Tao. 2019. Alexa, what should I eat? A personalized virtual nutrition coach for Native American diabetes patients using Amazon\u2019s smart speaker technology. In Proceedings of the 2019 IEEE International Conference on E-health Networking, Application, and Services (HealthCom\u201919). IEEE, 1\u20136."},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2016.0400"},{"key":"e_1_3_1_44_2","doi-asserted-by":"crossref","unstructured":"Souheila Moussalli and Walcir Cardoso. 2016. Are commercial \u2018personal robots\u2019 ready for language learning? Focus on second language speech. In CALL Communities and Culture\u2014Short Papers from EUROCALL 2016 L. Bradley and S. Thouesny (Eds.). Research-publishing.net 325\u2013329.","DOI":"10.14705\/rpnet.2016.eurocall2016.583"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1080\/09588221.2019.1595664"},{"key":"e_1_3_1_46_2","unstructured":"NLTK. 2023. Natural Language Toolkit. Retrieved September 30 2024 from https:\/\/www.nltk.org"},{"key":"e_1_3_1_47_2","article-title":"Nearly half of Americans use digital voice assistants, mostly on their smartphones","volume":"12","author":"Olmstead Kenneth","year":"2017","unstructured":"Kenneth Olmstead. 2017. Nearly half of Americans use digital voice assistants, mostly on their smartphones. Pew Research Center 12 (2017).","journal-title":"Pew Research Center"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3549015.3554213"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2021.669971"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0133-x"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3410530.3414374"},{"key":"e_1_3_1_52_2","unstructured":"Bhagyashree R. 2022. Amazon\u2019s Partnership with NHS to Make Alexa Offer Medical Advice Raises Privacy Concerns and Public Backlash. Retrieved September 30 2024 from https:\/\/hub.packtpub.com\/amazons-partnership-with-nhs-to-make-alexa-offer-medical-advice-raises-privacy-concerns-and-public-backlash\/"},{"key":"e_1_3_1_53_2","first-page":"29","volume-title":"Proceedings of the 1st Instructional Conference on Machine Learning","volume":"242","year":"2003","unstructured":"Juan Ramos. 2003. Using TF-IDF to determine word relevance in document queries. In Proceedings of the 1st Instructional Conference on Machine Learning, Vol. 242. 29\u201348."},{"key":"e_1_3_1_54_2","unstructured":"Elisabeth Rosenthal. 2022. Analysis: Why Alexa\u2019s Bedside manner is Bad for Health care. KFF Health News. Retrieved September 30 2024 from https:\/\/kffhealthnews.org\/news\/analysis-why-alexas-bedside-manner-is-bad-for-health-care\/"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517510"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196772"},{"key":"e_1_3_1_57_2","unstructured":"Selenium. 2022. The Selenium Browser. Retrieved September 30 2024 from https:\/\/www.selenium.dev"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587828.3587858"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00332-0"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3432233"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380179"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3052775"},{"key":"e_1_3_1_63_2","unstructured":"Beautiful Soup. 2022. Python. Retrieved September 30 2024 from https:\/\/pypi.org\/project\/beautifulsoup4\/"},{"key":"e_1_3_1_64_2","article-title":"\u201cAre you home alone?\u201d \u201cYes\u201d disclosing security and privacy vulnerabilities in Alexa skills","author":"Su Dan","year":"2020","unstructured":"Dan Su, Jiqiang Liu, Sencun Zhu, Xiaoyang Wang, and Wei Wang. 2020. \u201cAre you home alone?\u201d \u201cYes\u201d disclosing security and privacy vulnerabilities in Alexa skills. arXiv preprint arXiv:2010.10788 (2020).","journal-title":"arXiv preprint arXiv:2010.10788"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3479858"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3369807"},{"key":"e_1_3_1_67_2","unstructured":"Amazon Alexa Team. 2022. Introducing New Alexa Healthcare Skills. Retrieved September 30 2024 from https:\/\/developer.amazon.com\/blogs\/alexa\/post\/ff33dbc7-6cf5-4db8-b203-99144a251a21\/introducing-new-alexa-healthcare-skills"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376760"},{"key":"e_1_3_1_69_2","first-page":"305","volume-title":"Proceedings of the 20th Annual ACM Interaction Design and Children Conference (IDC\u201921)","author":"Brummelen Jessica Van","year":"2021","unstructured":"Jessica Van Brummelen, Viktoriya Tabunshchyk, and Tommy Heng. 2021. \u201cAlexa, can I program you?\u201d: Student perceptions of conversational artificial intelligence before and after programming Alexa. In Proceedings of the 20th Annual ACM Interaction Design and Children Conference (IDC\u201921). 305\u2013313."},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3313090"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2015.02.024"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-95405-5_12"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560416"},{"key":"e_1_3_1_74_2","volume-title":"Proceedings of the USENIX Security Symposium","author":"Young Jeffrey","year":"2022","unstructured":"Jeffrey Young, Song Liao, Long Cheng, Hongxin Hu, and Huixing Deng. 2022. SkillDetective: Automated policy-violation detection of voice assistant applications in the wild. In Proceedings of the USENIX Security Symposium."},{"key":"e_1_3_1_75_2","first-page":"65","volume-title":"Proceedings of the 13th Symposium on Usable Privacy and Security (SOUPS\u201917)","author":"Zeng Eric","year":"2017","unstructured":"Eric Zeng, Shrirang Mare, and Franziska Roesner. 2017. End user security and privacy concerns with smart homes. In Proceedings of the 13th Symposium on Usable Privacy and Security (SOUPS\u201917). 65\u201380."},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-020-00424-4"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3698200","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3698200","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:18Z","timestamp":1750295838000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3698200"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,18]]},"references-count":75,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3698200"],"URL":"https:\/\/doi.org\/10.1145\/3698200","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"type":"print","value":"1533-5399"},{"type":"electronic","value":"1557-6051"}],"subject":[],"published":{"date-parts":[[2024,11,18]]},"assertion":[{"value":"2023-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}