{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T05:18:14Z","timestamp":1755926294597,"version":"3.41.0"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"2-3","license":[{"start":{"date-parts":[[2019,3,18]],"date-time":"2019-03-18T00:00:00Z","timestamp":1552867200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Ministry of Science and Technology of Thailand Scholarship and EPSRC","award":["EP\/K023578\/1 and EP\/K015664\/1"],"award-info":[{"award-number":["EP\/K023578\/1 and EP\/K015664\/1"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>Creating automation scripts for tasks involving Graphical User Interface (GUI) interactions is hard. It is challenging because not all software applications allow access to a program\u2019s internal state, nor do they all have accessibility APIs. Although much of the internal state is exposed to the user through the GUI, it is hard to programmatically operate the GUI\u2019s widgets.<\/jats:p>\n          <jats:p>\n            To that end, we developed a system prototype that learns by demonstration, called\n            <jats:italic>HILC<\/jats:italic>\n            (Help, It Looks Confusing). Users, both programmers and non-programmers, train HILC to synthesize a task script by demonstrating the task. A demonstration produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions.\n          <\/jats:p>\n          <jats:p>We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. Although pure programming by demonstration is still unrealistic due to a computer\u2019s limited understanding of user intentions, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system, to help with confusing parts of the demonstrations. Our models of events and appearances are surprisingly simple but are combined effectively to cope with varying amounts of supervision.<\/jats:p>\n          <jats:p>The best available baseline, Sikuli Slides, struggled to assist users in the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple applications. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.<\/jats:p>","DOI":"10.1145\/3234508","type":"journal-article","created":{"date-parts":[[2019,3,19]],"date-time":"2019-03-19T12:11:29Z","timestamp":1552997489000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["HILC"],"prefix":"10.1145","volume":"9","author":[{"given":"Thanapong","family":"Intharah","sequence":"first","affiliation":[{"name":"University College London and Khon Kaen University, Thailand"}]},{"given":"Daniyar","family":"Turmukhambetov","sequence":"additional","affiliation":[{"name":"University College London, United Kingdom"}]},{"given":"Gabriel J.","family":"Brostow","sequence":"additional","affiliation":[{"name":"University College London, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2019,3,18]]},"reference":[{"volume-title":"Retrieved","year":"2006","key":"e_1_2_1_1_1","unstructured":"GitHub. 2006 . Tesseract . Retrieved February 24, 2019 from https:\/\/github.com\/tesseract-ocr\/tesseract\/. GitHub. 2006. Tesseract. Retrieved February 24, 2019 from https:\/\/github.com\/tesseract-ocr\/tesseract\/."},{"key":"e_1_2_1_2_1","volume-title":"Retrieved","author":"Lab Sikuli","year":"2014","unstructured":"Sikuli Lab . 2014 . Sikuli Slides . Retrieved February 24, 2019 from http:\/\/slides.sikuli.org\/. Sikuli Lab. 2014. Sikuli Slides. Retrieved February 24, 2019 from http:\/\/slides.sikuli.org\/."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380129"},{"volume-title":"Pattern Recognition and Machine Learning","author":"Bishop Christopher M.","key":"e_1_2_1_4_1","unstructured":"Christopher M. Bishop . 2006. Pattern Recognition and Machine Learning . Springer . Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2009.06.008"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047228"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753555"},{"volume-title":"Watch What I Do: Programming by Demonstration","author":"Cypher Allen","key":"e_1_2_1_9_1","unstructured":"Allen Cypher and Daniel Conrad Halbert . 1993. Watch What I Do: Programming by Demonstration . MIT Press , Cambridge, MA . Allen Cypher and Daniel Conrad Halbert. 1993. Watch What I Do: Programming by Demonstration. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979086"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647412"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.167"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531372"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866054"},{"volume-title":"Verification and Synthesis of Correct and Secure Systems","author":"Gulwani Sumit","key":"e_1_2_1_15_1","unstructured":"Sumit Gulwani . 2016. Programming by examples (and its applications in data wrangling) . In Verification and Synthesis of Correct and Secure Systems . IOS Press . Sumit Gulwani. 2016. Programming by examples (and its applications in data wrangling). In Verification and Synthesis of Correct and Secure Systems. IOS Press."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1950.tb00463.x"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995470"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1719970.1719973"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025176"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the CHI 2008 Workshop on Usable AI.","author":"Lau Tessa","year":"2008","unstructured":"Tessa Lau . 2008 . Why PBD systems fail: Lessons learned for usable AI . In Proceedings of the CHI 2008 Workshop on Usable AI. Tessa Lau. 2008. Why PBD systems fail: Lessons learned for usable AI. In Proceedings of the CHI 2008 Workshop on Usable AI."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/964442.964464"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357323"},{"key":"e_1_2_1_23_1","first-page":"120","article-title":"Fast normalized cross-correlation","volume":"10","author":"Lewis John P.","year":"1995","unstructured":"John P. Lewis . 1995 . Fast normalized cross-correlation . In Proceedings of Vision Interface , Vol. 10. 120 -- 123 . John P. Lewis. 1995. Fast normalized cross-correlation. In Proceedings of Vision Interface, Vol. 10. 120--123.","journal-title":"Proceedings of Vision Interface"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025483"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240767"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493190.2493216"},{"volume-title":"Automating Iterative Tasks With Programming by Demonstration. Master\u2019s Thesis","author":"Paynter Gordon W.","key":"e_1_2_1_27_1","unstructured":"Gordon W. Paynter . 2000. Automating Iterative Tasks With Programming by Demonstration. Master\u2019s Thesis . University of Waikato , Hamilton, New Zealand. Gordon W. Paynter. 2000. Automating Iterative Tasks With Programming by Demonstration. Master\u2019s Thesis. University of Waikato, Hamilton, New Zealand."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047213"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976604773135104"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-010-0384-0"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition.","author":"Shi Qinfeng","year":"2008","unstructured":"Qinfeng Shi , Li Wang , Li Cheng , and Alex Smola . 2008 . Discriminative human action segmentation and recognition using semi-Markov model . In Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition. Qinfeng Shi, Li Wang, Li Cheng, and Alex Smola. 2008. Discriminative human action segmentation and recognition using semi-Markov model. In Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1054010"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557407"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1622176.1622213"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047214"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/291080.291087"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3234508","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3234508","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:06Z","timestamp":1750268946000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3234508"}},"subtitle":["Domain-Independent PbD System Via Computer Vision and Follow-Up Questions"],"short-title":[],"issued":{"date-parts":[[2019,3,18]]},"references-count":36,"journal-issue":{"issue":"2-3","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3234508"],"URL":"https:\/\/doi.org\/10.1145\/3234508","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2019,3,18]]},"assertion":[{"value":"2017-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}