{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T13:59:04Z","timestamp":1758981544366,"version":"3.41.0"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,6,15]],"date-time":"2020-06-15T00:00:00Z","timestamp":1592179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003130","name":"Fonds Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["G0E7317N"],"award-info":[{"award-number":["G0E7317N"]}],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2020,6,15]]},"abstract":"<jats:p>We present Rataplan, a robust and resilient pixel-based approach for linking multi-modal proxies to automated sequences of actions in graphical user interfaces (GUIs). With Rataplan, users demonstrate a sequence of actions and answer human-readable follow-up questions to clarify their desire for automation. After demonstrating a sequence, the user can link a proxy input control to the action which can then be used as a shortcut for automating a sequence. Alternatively, output proxies use a notification model in which content is pushed when it becomes available. As an example use case, Rataplan uses keyboard shortcuts and tangible user interfaces (TUIs) as input proxies, and TUIs as output proxies. Instead of relying on available APIs, Rataplan automates GUIs using pixel-based reverse engineering. This ensures our approach can be used with all applications that offer a GUI, including web applications. We implemented a set of important strategies to support robust automation of modern interfaces that have a flat and minimal style, have frequent data and state changes, and have dynamic viewports.<\/jats:p>","DOI":"10.1145\/3397329","type":"journal-article","created":{"date-parts":[[2020,6,15]],"date-time":"2020-06-15T22:30:37Z","timestamp":1592260237000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Rataplan"],"prefix":"10.1145","volume":"4","author":[{"given":"Tom","family":"Veuskens","sequence":"first","affiliation":[{"name":"Hasselt University - tUL - Flanders Make Expertise Centre for Digital Media, Diepenbeek, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kris","family":"Luyten","sequence":"additional","affiliation":[{"name":"Hasselt University - tUL - Flanders Make Expertise Centre for Digital Media, Diepenbeek, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raf","family":"Ramakers","sequence":"additional","affiliation":[{"name":"Hasselt University - tUL - Flanders Make Expertise Centre for Digital Media, Diepenbeek, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,6,15]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Khalid Alharbi and Tom Yeh. 2019. Sikuli Slides. http:\/\/slides.sikuli.org. Accessed: 2019-07-30.  Khalid Alharbi and Tom Yeh. 2019. Sikuli Slides. http:\/\/slides.sikuli.org. Accessed: 2019-07-30."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3075222"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983990.2984020"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1347390.1347404"},{"key":"e_1_2_2_5_1","volume-title":"Human-Computer Interaction - INTERACT","author":"Burmistrov Ivan","year":"2015","unstructured":"Ivan Burmistrov , Tatiana Zlokazova , Anna Izmalkova , and Anna Leonova . 2015. Flat Design vs Traditional Design: Comparative Experimental Study . In Human-Computer Interaction - INTERACT 2015 , Julio Abascal, Simone Barbosa , Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing , Cham, 106--114. Ivan Burmistrov, Tatiana Zlokazova, Anna Izmalkova, and Anna Leonova. 2015. Flat Design vs Traditional Design: Comparative Experimental Study. In Human-Computer Interaction - INTERACT 2015, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 106--114."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047228"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753554"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979086"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647412"},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11)","author":"Eagan James R.","year":"2047","unstructured":"James R. Eagan , Michel Beaudouin-Lafon , and Wendy E. Mackay . 2011. Cracking the Cocoa Nut: User Interface Programming at Runtime . In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11) . ACM, New York, NY, USA, 225--234. https:\/\/doi.org\/10.1145\/ 2047 196.2047226 10.1145\/2047196.2047226 James R. Eagan, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2011. Cracking the Cocoa Nut: User Interface Programming at Runtime. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 225--234. https:\/\/doi.org\/10.1145\/2047196.2047226"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/263407.263537"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/571985.571991"},{"volume-title":"Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16)","author":"Guo Anhong","key":"e_1_2_2_13_1","unstructured":"Anhong Guo , Xiang \u00e2\u0102 &IJlig; Anthony\u00e2\u0102\u0130 Chen , Haoran Qi , Samuel White , Suman Ghosh , Chieko Asakawa , and Jeffrey P. Bigham . 2016. VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World . In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16) . Association for Computing Machinery, New York, NY, USA, 651--664. https:\/\/doi.org\/10.1145\/2984511.2984518 10.1145\/2984511.2984518 Anhong Guo, Xiang \u00e2\u0102&IJlig;Anthony\u00e2\u0102\u0130 Chen, Haoran Qi, Samuel White, Suman Ghosh, Chieko Asakawa, and Jeffrey P. Bigham. 2016. VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). Association for Computing Machinery, New York, NY, USA, 651--664. https:\/\/doi.org\/10.1145\/2984511.2984518"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1719970.1719973"},{"key":"e_1_2_2_15_1","volume-title":"Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM","author":"Intharah Thanapong","year":"1885","unstructured":"Thanapong Intharah , Michael Firman , and Gabriel J. Brostow . 2018. RecurBot: Learn to Auto-complete GUI Tasks From Human Demonstrations . In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM , New York, NY, USA, Article LBW595, 6 pages. https:\/\/doi.org\/10.1145\/3170427.3 1885 32 10.1145\/3170427.3188532 Thanapong Intharah, Michael Firman, and Gabriel J. Brostow. 2018. RecurBot: Learn to Auto-complete GUI Tasks From Human Demonstrations. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW595, 6 pages. https:\/\/doi.org\/10.1145\/3170427.3188532"},{"volume-title":"Proceedings of the 22Nd International Conference on Intelligent User Interfaces (IUI '17)","author":"Intharah Thanapong","key":"e_1_2_2_16_1","unstructured":"Thanapong Intharah , Daniyar Turmukhambetov , and Gabriel J. Brostow . 2017. Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions . In Proceedings of the 22Nd International Conference on Intelligent User Interfaces (IUI '17) . ACM, New York, NY, USA, 233--243. https:\/\/doi.org\/10.1145\/3025171.3025176 10.1145\/3025171.3025176 Thanapong Intharah, Daniyar Turmukhambetov, and Gabriel J. Brostow. 2017. Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions. In Proceedings of the 22Nd International Conference on Intelligent User Interfaces (IUI '17). ACM, New York, NY, USA, 233--243. https:\/\/doi.org\/10.1145\/3025171.3025176"},{"key":"e_1_2_2_17_1","unstructured":"JitBit. 2019. Macro Recorder Macro Program Keyboard Macros & Mouse Macros. https:\/\/www.jitbit.com\/macro-recorder\/. Accessed: 2019-07-01.  JitBit. 2019. Macro Recorder Macro Program Keyboard Macros & Mouse Macros. https:\/\/www.jitbit.com\/macro-recorder\/. Accessed: 2019-07-01."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357323"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753432"},{"volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17)","author":"Jia-Jun Li Toby","key":"e_1_2_2_20_1","unstructured":"Toby Jia-Jun Li , Amos Azaria , and Brad A. Myers . 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA, 6038--6049. https:\/\/doi.org\/10.1145\/3025453.3025483 10.1145\/3025453.3025483 Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6038--6049. https:\/\/doi.org\/10.1145\/3025453.3025483"},{"key":"e_1_2_2_21_1","unstructured":"AutoHotkey Foundation LLC. 2020. AutoHotKey. https:\/\/www.autohotkey.com. Accessed: 2020-02-13.  AutoHotkey Foundation LLC. 2020. AutoHotKey. https:\/\/www.autohotkey.com. Accessed: 2020-02-13."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557349"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303038"},{"key":"e_1_2_2_24_1","unstructured":"Macro Recorder. 2019. Mouse and Keyboard Capture. https:\/\/www.macrorecorder.com. Accessed: 2019-07-01.  Macro Recorder. 2019. Mouse and Keyboard Capture. https:\/\/www.macrorecorder.com. Accessed: 2019-07-01."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557016"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132525.3132557"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.3390\/mti2020031"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1166253.1166301"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807466"},{"key":"e_1_2_2_30_1","volume-title":"Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13)","author":"Voelker Simon","year":"2013","unstructured":"Simon Voelker , Kosuke Nakajima , Christian Thoresen , Yuichi Itoh , Kjell Ivar Overgard , and Jan Borchers . 2013 . PUCs: Detecting Transparent, Passive Untouched Capacitive Widgets on Unmodified Multi-touch Displays . In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13) . ACM, New York, NY, USA, 101--104. https:\/\/doi.org\/10.1145\/2512349.2512791 10.1145\/2512349.2512791 Simon Voelker, Kosuke Nakajima, Christian Thoresen, Yuichi Itoh, Kjell Ivar Overgard, and Jan Borchers. 2013. PUCs: Detecting Transparent, Passive Untouched Capacitive Widgets on Unmodified Multi-touch Displays. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13). ACM, New York, NY, USA, 101--104. https:\/\/doi.org\/10.1145\/2512349.2512791"},{"volume-title":"Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09)","author":"Yeh Tom","key":"e_1_2_2_31_1","unstructured":"Tom Yeh , Tsung-Hsiang Chang , and Robert C. Miller . 2009. Sikuli: Using GUI Screenshots for Search and Automation . In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09) . ACM, New York, NY, USA, 183--192. https:\/\/doi.org\/10.1145\/1622176.1622213 10.1145\/1622176.1622213 Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, New York, NY, USA, 183--192. https:\/\/doi.org\/10.1145\/1622176.1622213"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397329","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397329","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:47Z","timestamp":1750201967000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397329"}},"subtitle":["Resilient Automation of User Interface Actions with Multi-modal Proxies"],"short-title":[],"issued":{"date-parts":[[2020,6,15]]},"references-count":31,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,15]]}},"alternative-id":["10.1145\/3397329"],"URL":"https:\/\/doi.org\/10.1145\/3397329","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2020,6,15]]},"assertion":[{"value":"2020-06-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}