{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T15:52:18Z","timestamp":1778169138880,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":36,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T00:00:00Z","timestamp":1651017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,27]]},"DOI":"10.1145\/3491101.3504030","type":"proceedings-article","created":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T16:49:48Z","timestamp":1651250988000},"page":"1-6","source":"Crossref","is-referenced-by-count":23,"title":["Computational Approaches for Understanding, Generating, and Adapting User Interfaces"],"prefix":"10.1145","author":[{"given":"Yue","family":"Jiang","sequence":"first","affiliation":[{"name":"Max Planck Institute for Informatics, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuwen","family":"Lu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Notre Dame, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey","family":"Nichols","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wolfgang","family":"Stuerzlinger","sequence":"additional","affiliation":[{"name":"School of Interactive Arts + Technology (SIAT), Simon Fraser University, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chun","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Computer science and Technology, Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christof","family":"Lutteroth","sequence":"additional","affiliation":[{"name":"Computer Science, University of Bath, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Li","sequence":"additional","affiliation":[{"name":"Google Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ranjitha","family":"Kumar","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Toby Jia-Jun","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Notre Dame, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,4,28]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/504704.504705"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276526"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/27623.29354"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753554"},{"key":"e_1_3_2_1_6_1","volume-title":"Preference Elicitation for Interface Optimization. UIST: Proceedings of the Annual ACM Symposium on User Interface Softaware and Technology, 173\u2013182","author":"Gajos Krzysztof","year":"2005","unstructured":"Krzysztof Gajos and Daniel Weld . 2005 . Preference Elicitation for Interface Optimization. UIST: Proceedings of the Annual ACM Symposium on User Interface Softaware and Technology, 173\u2013182 . https:\/\/doi.org\/10.1145\/1095034.1095063 Krzysztof Gajos and Daniel Weld. 2005. Preference Elicitation for Interface Optimization. UIST: Proceedings of the Annual ACM Symposium on User Interface Softaware and Technology, 173\u2013182. https:\/\/doi.org\/10.1145\/1095034.1095063"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2010.05.005"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357250"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300334"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300334"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300643"},{"key":"e_1_3_2_1_12_1","volume-title":"ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints","author":"Jiang Yue","unstructured":"Yue Jiang , Wolfgang Stuerzlinger , and Christof Lutteroth . 2021. ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints . Association for Computing Machinery , New York, NY, USA . https:\/\/doi.org\/10.1145\/3411764.3445043 Yue Jiang, Wolfgang Stuerzlinger, and Christof Lutteroth. 2021. ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445043"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376610"},{"key":"e_1_3_2_1_14_1","volume-title":"Neural Design Network: Graphic Layout Generation with Constraints. arXiv e-prints","author":"Lee Hsin-Ying","year":"2019","unstructured":"Hsin-Ying Lee , Lu Jiang , Irfan Essa , Phuong\u00a0 B Le , Haifeng Gong , Ming-Hsuan Yang , and Weilong Yang . 2019. Neural Design Network: Graphic Layout Generation with Constraints. arXiv e-prints ( 2019 ), arXiv\u20131912. Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong\u00a0B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. 2019. Neural Design Network: Graphic Layout Generation with Constraints. arXiv e-prints (2019), arXiv\u20131912."},{"key":"e_1_3_2_1_15_1","volume-title":"Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767(2019).","author":"Li Jianan","year":"2019","unstructured":"Jianan Li , Jimei Yang , Aaron Hertzmann , Jianming Zhang , and Tingfa Xu . 2019 . Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767(2019). Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767(2019)."},{"key":"e_1_3_2_1_16_1","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Jia-Jun Li Toby","year":"2017","unstructured":"Toby Jia-Jun Li , Amos Azaria , and Brad\u00a0 A. Myers . 2017 . SUGILITE: Creating Multimodal Smartphone Automation by Demonstration . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems ( Denver, Colorado, USA) (CHI \u201917). Association for Computing Machinery, New York, NY, USA, 6038\u20136049. https:\/\/doi.org\/10.1145\/3025453.3025483 Toby Jia-Jun Li, Amos Azaria, and Brad\u00a0A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI \u201917). Association for Computing Machinery, New York, NY, USA, 6038\u20136049. https:\/\/doi.org\/10.1145\/3025453.3025483"},{"key":"e_1_3_2_1_17_1","volume-title":"2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). 105\u2013114","author":"Jia-Jun Li Toby","year":"2018","unstructured":"Toby Jia-Jun Li , Igor Labutov , Xiaohan\u00a0Nancy Li , Xiaoyi Zhang , Wenze Shi , Wanling Ding , Tom\u00a0 M. Mitchell , and Brad\u00a0 A. Myers . 2018 . APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions . In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). 105\u2013114 . https:\/\/doi.org\/10.1109\/VLHCC.2018.8506506 Toby Jia-Jun Li, Igor Labutov, Xiaohan\u00a0Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom\u00a0M. Mitchell, and Brad\u00a0A. Myers. 2018. APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). 105\u2013114. https:\/\/doi.org\/10.1109\/VLHCC.2018.8506506"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Online, 215\u2013223","author":"Jia-Jun Li Toby","year":"2020","unstructured":"Toby Jia-Jun Li , Tom Mitchell , and Brad Myers . 2020 . Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Online, 215\u2013223 . https:\/\/doi.org\/10.18653\/v1\/2020.acl-demos.25 Toby Jia-Jun Li, Tom Mitchell, and Brad Myers. 2020. Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Online, 215\u2013223. https:\/\/doi.org\/10.18653\/v1\/2020.acl-demos.25"},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","author":"Jia-Jun Li Toby","year":"2021","unstructured":"Toby Jia-Jun Li , Lindsay Popowski , Tom Mitchell , and Brad\u00a0 A Myers . 2021 . Screen2Vec: Semantic Embedding of GUI Screens and GUI Components . In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems ( Yokohama, Japan) (CHI \u201921). Association for Computing Machinery, New York, NY, USA, Article 578, 15\u00a0pages. https:\/\/doi.org\/10.1145\/3411764.3445049 Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad\u00a0A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI \u201921). Association for Computing Machinery, New York, NY, USA, Article 578, 15\u00a0pages. https:\/\/doi.org\/10.1145\/3411764.3445049"},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology","author":"Jia-Jun Li Toby","year":"2019","unstructured":"Toby Jia-Jun Li , Marissa Radensky , Justin Jia , Kirielle Singarajah , Tom\u00a0 M. Mitchell , and Brad\u00a0 A. Myers . 2019 . PUMICE: A Multi-Modal Agent That Learns Concepts and Conditionals from Natural Language and Demonstrations . In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology ( New Orleans, LA, USA) (UIST \u201919). Association for Computing Machinery, New York, NY, USA, 577\u2013589. https:\/\/doi.org\/10.1145\/3332165.3347899 Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom\u00a0M. Mitchell, and Brad\u00a0A. Myers. 2019. PUMICE: A Multi-Modal Agent That Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST \u201919). Association for Computing Machinery, New York, NY, USA, 577\u2013589. https:\/\/doi.org\/10.1145\/3332165.3347899"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys","author":"Jia-Jun Li Toby","year":"2018","unstructured":"Toby Jia-Jun Li and Oriana Riva . 2018 . KITE: Building conversational bots from mobile apps . In Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018). ACM. Toby Jia-Jun Li and Oriana Riva. 2018. KITE: Building conversational bots from mobile apps. In Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018). ACM."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.729"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.443"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242650"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the Ninth Conference on Australasian User Interface -","volume":"76","author":"Lutteroth Christof","year":"2008","unstructured":"Christof Lutteroth . 2008 . Automated Reverse Engineering of Hard-Coded GUI Layouts . In Proceedings of the Ninth Conference on Australasian User Interface - Volume 76 (Wollongong, Australia) (AUIC \u201908). Australian Computer Society, Inc., AUS, 65\u201373. https:\/\/doi.org\/10.5555\/1378337.1378350 Christof Lutteroth. 2008. Automated Reverse Engineering of Hard-Coded GUI Layouts. In Proceedings of the Ninth Conference on Australasian User Interface - Volume 76 (Wollongong, Australia) (AUIC \u201908). Australian Computer Society, Inc., AUS, 65\u201373. https:\/\/doi.org\/10.5555\/1378337.1378350"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/571985.572008"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.485844"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415848"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377515"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1166253.1166301"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025506"},{"key":"e_1_3_2_1_32_1","volume-title":"Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints","author":"Swearngin Amanda","year":"2020","unstructured":"Amanda Swearngin , Chenglong Wang , Alannah Oleson , James Fogarty , and Amy\u00a0 J. Ko . 2020 . Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints . Association for Computing Machinery , New York, NY, USA , 1\u201313. https:\/\/doi.org\/10.1145\/3313831.3376593 Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy\u00a0J. Ko. 2020. Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. Association for Computing Machinery, New York, NY, USA, 1\u201313. https:\/\/doi.org\/10.1145\/3313831.3376593"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169305"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. (2021).  Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. (2021).","DOI":"10.1145\/3472749.3474765"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97248"},{"key":"e_1_3_2_1_36_1","volume-title":"Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels","author":"Zhang Xiaoyi","year":"2021","unstructured":"Xiaoyi Zhang , Lilian de Greef , Amanda Swearngin , Samuel White , Kyle Murray , Lisa Yu , Qi Shan , Jeffrey Nichols , Jason Wu , Chris Fleizach , Aaron Everitt , and Jeffrey\u00a0 P Bigham . 2021 . Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels . Association for Computing Machinery , New York, NY, USA . https:\/\/doi.org\/10.1145\/3411764.3445186 Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey\u00a0P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445186"}],"event":{"name":"CHI '22: CHI Conference on Human Factors in Computing Systems","location":"New Orleans LA USA","acronym":"CHI '22","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["CHI Conference on Human Factors in Computing Systems Extended Abstracts"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491101.3504030","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3491101.3504030","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:09Z","timestamp":1750188669000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491101.3504030"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,27]]},"references-count":36,"alternative-id":["10.1145\/3491101.3504030","10.1145\/3491101"],"URL":"https:\/\/doi.org\/10.1145\/3491101.3504030","relation":{},"subject":[],"published":{"date-parts":[[2022,4,27]]}}}