{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T17:23:32Z","timestamp":1778779412102,"version":"3.51.4"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,3,27]],"date-time":"2023-03-27T00:00:00Z","timestamp":1679875200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,3,27]]},"abstract":"<jats:p>Nowadays, voice assistants help users complete tasks on the smartphone with voice commands, replacing traditional touchscreen interactions when such interactions are inhibited. However, the usability of those tools remains moderate due to the problems in understanding rich language variations in human commands, along with efficiency and comprehensibility issues. Therefore, we introduce Voicify, an Android virtual assistant that allows users to interact with on-screen elements in mobile apps through voice commands. Using a novel deep learning command parser, Voicify interprets human verbal input and performs matching with UI elements. In addition, the tool can directly open a specific feature from installed applications by fetching application code information to explore the set of in-app components. Our command parser achieved 90% accuracy on the human command dataset. Furthermore, the direct feature invocation module achieves better feature coverage in comparison to Google Assistant. The user study demonstrates the usefulness of Voicify in real-world scenarios.<\/jats:p>","DOI":"10.1145\/3581998","type":"journal-article","created":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T14:57:51Z","timestamp":1680015471000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Voicify Your UI"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4798-8701","authenticated-orcid":false,"given":"Minh Duc","family":"Vu","sequence":"first","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7862-6677","authenticated-orcid":false,"given":"Han","family":"Wang","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9808-9992","authenticated-orcid":false,"given":"Zhuang","family":"Li","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7326-8380","authenticated-orcid":false,"given":"Gholamreza","family":"Haffari","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO's Data61 &amp; Australian National University, Canberra, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2011-9618","authenticated-orcid":false,"given":"Chunyang","family":"Chen","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,3,28]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"2022. Voice Access - Apps on Google Play. https:\/\/play.google.com\/store\/apps\/details?id=com.google.android.apps.accessibility.voiceaccess&hl=en_AU&gl=US"},{"key":"e_1_2_2_2_1","unstructured":"Voice Access. 2022. Troubleshoot Voice Access. https:\/\/support.google.com\/accessibility\/android\/answer\/6377053?hl=en#:~:text=If%20you%20have%20trouble%20starting Access%20from%20the%20lock%20screen."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.2976556"},{"key":"e_1_2_2_4_1","unstructured":"Just An Application. 2022. Android Binary XML. https:\/\/justanapplication.wordpress.com\/category\/android\/android-binary-xml\/"},{"key":"e_1_2_2_5_1","volume-title":"App-Based Task Shortcuts for Virtual Assistants. In The 34th Annual ACM Symposium on User Interface Software and Technology. 1089--1099","author":"Arsan Deniz","year":"2021","unstructured":"Deniz Arsan, Ali Zaidi, Aravind Sagar, and Ranjitha Kumar. 2021. App-Based Task Shortcuts for Virtual Assistants. In The 34th Annual ACM Symposium on User Interface Software and Technology. 1089--1099."},{"key":"e_1_2_2_6_1","unstructured":"Google Assistant. 2022. Assistant. https:\/\/assistant.google.com\/intl\/en_au\/platforms\/phones\/"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2906388.2906416"},{"key":"e_1_2_2_8_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Bahdanau Dzmitry","year":"2015","unstructured":"Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3529299.3531493"},{"key":"e_1_2_2_10_1","article-title":"Smart Voice Assistant: a universal voice control solution for non-visual access to the Android operating system","volume":"4","author":"Bhalerao Aditi","year":"2017","unstructured":"Aditi Bhalerao, Samira Bhilare, Anagha Bondade, and Monal Shingade. 2017. Smart Voice Assistant: a universal voice control solution for non-visual access to the Android operating system. Int. Res. J. Eng. Technol 4, 2 (2017).","journal-title":"Int. Res. J. Eng. Technol"},{"key":"e_1_2_2_11_1","volume-title":"Sus: a \"quick and dirty'usability. Usability evaluation in industry 189, 3","author":"Brooke John","year":"1996","unstructured":"John Brooke. 1996. Sus: a \"quick and dirty'usability. Usability evaluation in industry 189, 3 (1996)."},{"key":"e_1_2_2_12_1","volume-title":"Hierarchical phrase-based translation. computational linguistics 33, 2","author":"Chiang David","year":"2007","unstructured":"David Chiang. 2007. Hierarchical phrase-based translation. computational linguistics 33, 2 (2007), 201--228."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i09.7073"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2935334.2935386"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_2_2_16_1","volume-title":"Wiley StatsRef: Statistics Reference Online","author":"DePuy Venita","year":"2014","unstructured":"Venita DePuy and Vance W Berger. 2014. Counterbalancing. Wiley StatsRef: Statistics Reference Online (2014)."},{"key":"e_1_2_2_17_1","unstructured":"Android Developers. 2022. AccessibilityNodeInfo. https:\/\/developer.android.com\/reference\/android\/view\/accessibility\/AccessibilityNodeInfo.AccessibilityAction"},{"key":"e_1_2_2_18_1","unstructured":"Android Developers. 2022. AccessibilityService. https:\/\/developer.android.com\/guide\/topics\/ui\/accessibility\/service"},{"key":"e_1_2_2_19_1","unstructured":"Android Developers. 2022. Deep Linking. https:\/\/developer.android.com\/training\/app-links\/deep-linking"},{"key":"e_1_2_2_20_1","unstructured":"Android Developers. 2022. SpeechRecognizer. https:\/\/developer.android.com\/reference\/android\/speech\/SpeechRecognizer"},{"key":"e_1_2_2_21_1","unstructured":"Android developers. 2022. Text To Speech. https:\/\/developer.android.com\/reference\/android\/speech\/tts\/TextToSpeech"},{"key":"e_1_2_2_22_1","volume-title":"Coarse-to-Fine Decoding for Neural Semantic Parsing. In 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 731--742","author":"Dong Li","year":"2018","unstructured":"Li Dong and Mirella Lapata. 2018. Coarse-to-Fine Decoding for Neural Semantic Parsing. In 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 731--742."},{"key":"e_1_2_2_23_1","volume-title":"A Comparison of LSTM and BERT for Small Corpus. arXiv preprint arXiv:2009.05451","author":"Ezen-Can Aysu","year":"2020","unstructured":"Aysu Ezen-Can. 2020. A Comparison of LSTM and BERT for Small Corpus. arXiv preprint arXiv:2009.05451 (2020)."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2072221.2072233"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053166"},{"key":"e_1_2_2_26_1","volume-title":"Flask web development: developing web applications with python. \" O'Reilly Media","author":"Grinberg Miguel","unstructured":"Miguel Grinberg. 2018. Flask web development: developing web applications with python. \" O'Reilly Media, Inc.\"."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1154"},{"key":"e_1_2_2_28_1","doi-asserted-by":"crossref","unstructured":"Sonal Gupta Rushin Shah Mrinal Mohit Anuj Kumar and Mike Lewis. 2018. Semantic Parsing for Task Oriented Dialog using Hierarchical Representations. In EMNLP.","DOI":"10.18653\/v1\/D18-1300"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1037\/e577632012-009"},{"key":"e_1_2_2_30_1","volume-title":"Long short-term memory. Neural computation 9, 8","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780."},{"key":"e_1_2_2_31_1","volume-title":"an introduction to voice assistants. Medical reference services quarterly 37, 1","author":"Hoy Matthew B","year":"2018","unstructured":"Matthew B Hoy. 2018. Alexa, Siri, Cortana, and more: an introduction to voice assistants. Medical reference services quarterly 37, 1 (2018), 81--88."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.02.012"},{"key":"e_1_2_2_33_1","unstructured":"Tom Hume. 2020. Use Voice Access to control your Android device with your voice. https:\/\/blog.google\/outreach-initiatives\/accessibility\/voice-access-updates\/."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289595.3289597"},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of NAACL-HLT. 4171--4186","author":"Ming-Wei Chang Jacob Devlin","year":"2019","unstructured":"Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171--4186."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.48084\/etasr.1895"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381009"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1979742.1979749"},{"key":"e_1_2_2_39_1","first-page":"163","article-title":"Zigbee based voice control system for smart home","volume":"3","author":"Bala Krishna Y","year":"2012","unstructured":"Y Bala Krishna and S Nagendram. 2012. Zigbee based voice control system for smart home. International Journal on Computer Technology and Applications 3, 1 (2012), 163--168.","journal-title":"International Journal on Computer Technology and Applications"},{"key":"e_1_2_2_40_1","first-page":"416","article-title":"Personal assistant with voice recognition intelligence","volume":"10","author":"Kulhalli Kshama V","year":"2017","unstructured":"Kshama V Kulhalli, Kotrappa Sirbi, and Abhijit J Patankar. 2017. Personal assistant with voice recognition intelligence. International Journal of Engineering Research and Technology 10, 1 (2017), 416--419.","journal-title":"International Journal of Engineering Research and Technology"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.729"},{"key":"e_1_2_2_42_1","first-page":"e2","article-title":"Data flow analysis on Android platform with fragment lifecycle modeling and callbacks","volume":"4","author":"Li Yongfeng","year":"2017","unstructured":"Yongfeng Li, Jinbing Ouyang, Bing Mao, Kai Ma, and Shanqing Guo. 2017. Data flow analysis on Android platform with fragment lifecycle modeling and callbacks. EAI Endorsed Transactions on Security and Safety 4, 11 (2017), e2.","journal-title":"EAI Endorsed Transactions on Security and Safety"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.226"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.109"},{"key":"e_1_2_2_45_1","volume-title":"Voice helper: A mobile assistive system for visually impaired persons. In 2015 IEEE International Conference on Computer and Information Technology","author":"Liu Kuei-Chun","unstructured":"Kuei-Chun Liu, Ching-Hung Wu, Shau-Yin Tseng, and Yin-Te Tsai. 2015. Voice helper: A mobile assistive system for visually impaired persons. In 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing. IEEE, 1400--1405."},{"key":"e_1_2_2_46_1","volume-title":"Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.42"},{"key":"e_1_2_2_48_1","volume-title":"DroidLink: Automated generation of deep links for Android apps. arXiv preprint arXiv:1605.06928","author":"Ma Yun","year":"2016","unstructured":"Yun Ma, Xuanzhe Liu, Ruogu Du, Ziniu Hu, Yi Liu, Meihua Yu, and Gang Huang. 2016. DroidLink: Automated generation of deep links for Android apps. arXiv preprint arXiv:1605.06928 (2016)."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.3390\/su10020488"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445455"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.23919\/FUSION49751.2022.9841287"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.677"},{"key":"e_1_2_2_53_1","first-page":"17","article-title":"The zephyr abstract syntax description language","volume":"97","author":"Wang Daniel C","year":"1997","unstructured":"Daniel C Wang, Andrew W Appel, Jeffrey L Korn, and Christopher S Serra. 1997. The zephyr abstract syntax description language.. In DSL, Vol. 97. 17--17.","journal-title":"DSL"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1129"},{"key":"e_1_2_2_55_1","volume-title":"Natural language processing for government: Problems and potential","author":"Wijeratne Yudhanjaya","year":"2019","unstructured":"Yudhanjaya Wijeratne, Nisansa de Silva, and Yashothara Shanmugarajah. 2019. Natural language processing for government: Problems and potential. International Development Research Centre (Canada) 1 (2019)."},{"key":"e_1_2_2_56_1","volume-title":"Vijay patel, and Kerwin","author":"Bam Swarup","year":"2022","unstructured":"Xaif, Swarup Bam, Satya Ram, Ajinkya, Vijay patel, and Kerwin. 2022. Android 12: Google assistant not working [8 fixes]. https:\/\/devsjournal.com\/android-12-google-assistant-not-working-8-fixes.html"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1881\/3\/032071"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2015.11.004"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411974"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.31"},{"key":"e_1_2_2_61_1","unstructured":"Kannon Yamada. 2020. How to control your Android device entirely with your voice. https:\/\/www.makeuseof.com\/tag\/control-android-device-entirely-voice\/"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415841"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445186"},{"key":"e_1_2_2_64_1","volume-title":"JustSpeak. Proceedings of the 11th Web for All Conference on - W4A 14","author":"Zhong Yu","year":"2014","unstructured":"Yu Zhong, T. V. Raman, Casey Burkhardt, Fadi Biadsy, and Jeffrey P. Bigham. 2014. JustSpeak. Proceedings of the 11th Web for All Conference on - W4A 14 (2014)."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3581998","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3581998","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:43:39Z","timestamp":1752468219000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3581998"}},"subtitle":["Towards Android App Control with Voice Commands"],"short-title":[],"issued":{"date-parts":[[2023,3,27]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3,27]]}},"alternative-id":["10.1145\/3581998"],"URL":"https:\/\/doi.org\/10.1145\/3581998","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,27]]},"assertion":[{"value":"2023-03-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}