{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:29:51Z","timestamp":1750220991125,"version":"3.41.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"EICS","license":[{"start":{"date-parts":[[2019,6,13]],"date-time":"2019-06-13T00:00:00Z","timestamp":1560384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1350883 and CNS-1618531"],"award-info":[{"award-number":["CNS-1350883 and CNS-1618531"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,6,13]]},"abstract":"<jats:p>Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture\/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.<\/jats:p>","DOI":"10.1145\/3300964","type":"journal-article","created":{"date-parts":[[2019,6,14]],"date-time":"2019-06-14T12:36:28Z","timestamp":1560515788000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Gesto"],"prefix":"10.1145","volume":"3","author":[{"given":"Chang Min","family":"Park","sequence":"first","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taeyeon","family":"Ki","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ali J.","family":"Ben Ali","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nikhil Sunil","family":"Pawar","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karthik","family":"Dantu","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steven Y.","family":"Ko","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lukasz","family":"Ziarek","sequence":"additional","affiliation":[{"name":"University at Buffalo, The State University of New York, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,6,13]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Cited May 2018. Accessibility Services. https:\/\/developer.android.com\/reference\/android\/accessibilityservice\/AccessibilityService.html.  Cited May 2018. Accessibility Services. https:\/\/developer.android.com\/reference\/android\/accessibilityservice\/AccessibilityService.html."},{"key":"e_1_2_1_2_1","unstructured":"Cited May 2018. Apktool. https:\/\/ibotpeaches.github.io\/Apktool\/.  Cited May 2018. Apktool. https:\/\/ibotpeaches.github.io\/Apktool\/."},{"key":"e_1_2_1_3_1","unstructured":"Cited May 2018. appetizer-toolkit. https:\/\/github.com\/appetizerio\/appetizer-toolkit.  Cited May 2018. appetizer-toolkit. https:\/\/github.com\/appetizerio\/appetizer-toolkit."},{"key":"e_1_2_1_4_1","unstructured":"Cited May 2018. Bot-bot. http:\/\/imaginea.github.io\/bot-bot\/index.html.  Cited May 2018. Bot-bot. http:\/\/imaginea.github.io\/bot-bot\/index.html."},{"key":"e_1_2_1_5_1","unstructured":"Cited May 2018. Culebra. https:\/\/github.com\/dtmilano\/AndroidViewClient\/wiki\/culebra.  Cited May 2018. Culebra. https:\/\/github.com\/dtmilano\/AndroidViewClient\/wiki\/culebra."},{"key":"e_1_2_1_6_1","volume-title":"Espresso Test Recorder","author":"Cited May","year":"2017","unstructured":"Cited May 2018. Espresso Test Recorder , 2017 . h ps:\/\/developer.android.com\/studio\/test\/espresso-test-recorder.html. Cited May 2018. Espresso Test Recorder, 2017. h ps:\/\/developer.android.com\/studio\/test\/espresso-test-recorder.html."},{"key":"e_1_2_1_7_1","unstructured":"Cited May 2018. Frep. http:\/\/strai.x0.com\/frep\/.  Cited May 2018. Frep. http:\/\/strai.x0.com\/frep\/."},{"key":"e_1_2_1_8_1","unstructured":"Cited May 2018. monkeyrunner. https:\/\/developer.android.com\/studio\/test\/monkeyrunner\/index.html.  Cited May 2018. monkeyrunner. https:\/\/developer.android.com\/studio\/test\/monkeyrunner\/index.html."},{"key":"e_1_2_1_9_1","unstructured":"Cited May 2018. Ranorex. http:\/\/www.ranorex.com\/mobile-automation-testing.html.  Cited May 2018. Ranorex. http:\/\/www.ranorex.com\/mobile-automation-testing.html."},{"key":"e_1_2_1_10_1","unstructured":"Cited May 2018. Robotium Recorder. https:\/\/robotium.com\/products\/robotium-recorder.  Cited May 2018. Robotium Recorder. https:\/\/robotium.com\/products\/robotium-recorder."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1999995.1999998"},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_12_1","unstructured":"Amazon. Cited December 2017 . Alexa . https:\/\/developer.amazon.com\/alexa. Amazon. Cited December 2017. Alexa. https:\/\/developer.amazon.com\/alexa."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_13_1","unstructured":"Amazon. Cited December 2017 . Alexa SDK. https:\/\/developer.amazon.com\/alexa-voice-service\/sdk. Amazon. Cited December 2017. Alexa SDK. https:\/\/developer.amazon.com\/alexa-voice-service\/sdk."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_14_1","unstructured":"Apple. Cited December 2017 . Siri . https:\/\/www.apple.com\/ios\/siri\/. Apple. Cited December 2017. Siri. https:\/\/www.apple.com\/ios\/siri\/."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_15_1","unstructured":"Apple. Cited December 2017 . SiriKit . https:\/\/developer.apple.com\/sirikit\/. Apple. Cited December 2017. SiriKit. https:\/\/developer.apple.com\/sirikit\/."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2906388.2906416"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2632090"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3057862"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2184489.2184507"},{"key":"e_1_2_1_20_1","volume-title":"2013 35th International Conference on. IEEE, 72--81","author":"Gomez Lorenzo","year":"2013","unstructured":"Lorenzo Gomez , Iulian Neamtiu , Tanzirul Azim , and Todd Millstein . 2013 . Reran: Timing-and touch-sensitive record and replay for android. In So ware Engineering (ICSE) , 2013 35th International Conference on. IEEE, 72--81 . Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for android. In So ware Engineering (ICSE), 2013 35th International Conference on. IEEE, 72--81."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_21_1","unstructured":"Google. Cited December 2017 . Class android.view. View. https:\/\/developer.android.com\/reference\/android\/view\/ View .html. Google. Cited December 2017. Class android.view. View. https:\/\/developer.android.com\/reference\/android\/view\/ View.html."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_22_1","unstructured":"Google. Cited December 2017 . Google Assistant . https:\/\/assistant.google.com\/. Google. Cited December 2017. Google Assistant. https:\/\/assistant.google.com\/."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_23_1","unstructured":"Google. Cited December 2017 . Google Assistant SDK. https:\/\/developers.google.com\/assistant\/sdk\/. Google. Cited December 2017. Google Assistant SDK. https:\/\/developers.google.com\/assistant\/sdk\/."},{"key":"e_1_2_1_24_1","volume-title":"2015 IEEE International Symposium on. IEEE, 215--224","author":"Halpern Ma","year":"2015","unstructured":"Ma hew Halpern , Yuhao Zhu , Ramesh Peri , and Vijay Janapa Reddi . 2015 . Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosystem. In Performance Analysis of Systems and So ware (ISPASS) , 2015 IEEE International Symposium on. IEEE, 215--224 . Ma hew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosystem. In Performance Analysis of Systems and So ware (ISPASS), 2015 IEEE International Symposium on. IEEE, 215--224."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2814270.2814320"},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_26_1","unstructured":"Intel. Cited December 2017 . Intel Context Sensing SDK | Intel Software . https:\/\/software.intel.com\/en-us\/context-sensing-sdk. Intel. Cited December 2017. Intel Context Sensing SDK | Intel Software. https:\/\/software.intel.com\/en-us\/context-sensing-sdk."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/2616448.2616477"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2933242.2933261"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3117769"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866067"},{"volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17)","author":"Jia-Jun Li Toby","key":"e_1_2_1_31_1","unstructured":"Toby Jia-Jun Li , Amos Azaria , and Brad A. Myers . 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, 21 New York, NY, USA, 6038--6049. Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 21 New York, NY, USA, 6038--6049."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2009.07.007"},{"key":"e_1_2_1_33_1","volume-title":"Cited","author":"Open Handset Alliance Google","year":"2017","unstructured":"Google LLC and Open Handset Alliance . Cited December 2017 . SpeechRecognizer | Android Developers . https:\/\/developer.android.com\/reference\/android\/speech\/SpeechRecognizer.html. Google LLC and Open Handset Alliance. Cited December 2017. SpeechRecognizer | Android Developers. https:\/\/developer.android.com\/reference\/android\/speech\/SpeechRecognizer.html."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-C.2017.107"},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_35_1","unstructured":"Microsoft. Cited December 2017 . Cortana . https:\/\/www.microsoft.com\/en-us\/windows\/cortana. Microsoft. Cited December 2017. Cortana. https:\/\/www.microsoft.com\/en-us\/windows\/cortana."},{"volume-title":"Cited","year":"2017","key":"e_1_2_1_36_1","unstructured":"Microsoft. Cited December 2017 . Cortana SDK. https:\/\/developer.microsoft.com\/en-us\/cortana. Microsoft. Cited December 2017. Cortana SDK. https:\/\/developer.microsoft.com\/en-us\/cortana."},{"key":"e_1_2_1_37_1","volume-title":"Cited","author":"Mobile AVG","year":"2017","unstructured":"AVG Mobile . Cited December 2017 . AVG Cleaner . https:\/\/play.google.com\/store\/apps\/details?id=com.avg.cleaner&hl=en. AVG Mobile. Cited December 2017. AVG Cleaner. https:\/\/play.google.com\/store\/apps\/details?id=com.avg.cleaner&hl=en."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594368.2594379"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070942.2070969"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884854"},{"key":"e_1_2_1_41_1","volume-title":"2010 International Symposium). IEEE.","author":"Raa Giuseppe","year":"2010","unstructured":"Giuseppe Raa , Jinwon Lee , and Lama Nachman . 2010 . Don't slow me down: Bringing energy efficiency to continuous gesture recognition (Wearable Computers (ISWC) , 2010 International Symposium). IEEE. Giuseppe Raa, Jinwon Lee, and Lama Nachman. 2010. Don't slow me down: Bringing energy efficiency to continuous gesture recognition (Wearable Computers (ISWC), 2010 International Symposium). IEEE."},{"key":"e_1_2_1_42_1","unstructured":"Margaret Rouse. Cited November 2017. What is deep link? http:\/\/searchmicroservices.techtarget.com\/definition\/deep-link.  Margaret Rouse. Cited November 2017. What is deep link? http:\/\/searchmicroservices.techtarget.com\/definition\/deep-link."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804260"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2906388.2906407"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2494603.2480307"},{"key":"e_1_2_1_46_1","unstructured":"Nishant Srivastava. 2016. GitHub - nisrulz\/sensey: {Android Library} Play with sensor events & detect gestures in a snap. https:\/\/github.com\/nisrulz\/sensey.  Nishant Srivastava. 2016. GitHub - nisrulz\/sensey: {Android Library} Play with sensor events & detect gestures in a snap. https:\/\/github.com\/nisrulz\/sensey."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON '99)","author":"Rai Raja Valle","year":"1999","unstructured":"Raja Valle e- Rai , Phong Co , Etienne Gagnon , Laurie Hendren , Patrick Lam , and Vijay Sundaresan . 1999 . Soot - a Java bytecode Optimization Framework . In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON '99) . Raja Valle e-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java bytecode Optimization Framework. In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON '99)."},{"volume-title":"Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14","author":"Xu Chao","key":"e_1_2_1_48_1","unstructured":"Chao Xu , Parth H Pathak , and Prasant Mohapatra . {n. d.}. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch . In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14 . Chao Xu, Parth H Pathak, and Prasant Mohapatra. {n. d.}. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2742647.2742662"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3300964","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3300964","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3300964","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:25:23Z","timestamp":1750206323000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3300964"}},"subtitle":["Mapping UI Events to Gestures and Voice Commands"],"short-title":[],"issued":{"date-parts":[[2019,6,13]]},"references-count":49,"journal-issue":{"issue":"EICS","published-print":{"date-parts":[[2019,6,13]]}},"alternative-id":["10.1145\/3300964"],"URL":"https:\/\/doi.org\/10.1145\/3300964","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2019,6,13]]},"assertion":[{"value":"2019-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}