{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:14:40Z","timestamp":1776111280401,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":41,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,3,22]],"date-time":"2022-03-22T00:00:00Z","timestamp":1647907200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,3,22]]},"DOI":"10.1145\/3490099.3511109","type":"proceedings-article","created":{"date-parts":[[2022,3,21]],"date-time":"2022-03-21T23:24:13Z","timestamp":1647905053000},"page":"447-458","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Understanding Screen Relationships from Screenshots of Smartphone Applications"],"prefix":"10.1145","author":[{"given":"Shirin","family":"Feiz","sequence":"first","affiliation":[{"name":"Computer Science department, Stony Brook University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jason","family":"Wu","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoyi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Apple Inc, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amanda","family":"Swearngin","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Titus","family":"Barik","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey","family":"Nichols","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,3,22]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"(accessed)2021. Image Hash Library. Available at https:\/\/github.com\/JohannesBuchner\/imagehash.  (accessed)2021. Image Hash Library. Available at https:\/\/github.com\/JohannesBuchner\/imagehash."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2014.55"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445762"},{"key":"e_1_3_2_2_4_1","volume-title":"TensorFlow Similarity: A Usable","author":"Bursztein Elie","year":"2021","unstructured":"Elie Bursztein , James Long , Shun Lin , Owen Vallis , and Francois Chollet . 2021. TensorFlow Similarity: A Usable , High-Performance Metric Learning Library . Fixme ( 2021 ). Elie Bursztein, James Long, Shun Lin, Owen Vallis, and Francois Chollet. 2021. TensorFlow Similarity: A Usable, High-Performance Metric Learning Library. Fixme (2021)."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359282"},{"key":"#cr-split#-e_1_3_2_2_6_1.1","doi-asserted-by":"crossref","unstructured":"Jia Chen Ge Han Shanqing Guo and Wenrui Diao. 2018. FragDroid: Automated User Interface Interaction with Activity and Fragment Analysis in Android Applications. In 2018 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN). 398-409. https:\/\/doi.org\/10.1109\/DSN.2018.00049 10.1109\/DSN.2018.00049","DOI":"10.1109\/DSN.2018.00049"},{"key":"#cr-split#-e_1_3_2_2_6_1.2","doi-asserted-by":"crossref","unstructured":"Jia Chen Ge Han Shanqing Guo and Wenrui Diao. 2018. FragDroid: Automated User Interface Interaction with Activity and Fragment Analysis in Android Applications. In 2018 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN). 398-409. https:\/\/doi.org\/10.1109\/DSN.2018.00049","DOI":"10.1109\/DSN.2018.00049"},{"key":"e_1_3_2_2_7_1","volume-title":"StoryDroid: Automated Generation of Storyboard for Android Apps. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 596\u2013607","author":"Chen Sen","year":"2019","unstructured":"Sen Chen , Lingling Fan , Chunyang Chen , Ting Su , Wenhe Li , Yang Liu , and Lihua Xu . 2019 . StoryDroid: Automated Generation of Storyboard for Android Apps. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 596\u2013607 . https:\/\/doi.org\/10.1109\/ICSE.2019.00070 10.1109\/ICSE.2019.00070 Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. StoryDroid: Automated Generation of Storyboard for Android Apps. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 596\u2013607. https:\/\/doi.org\/10.1109\/ICSE.2019.00070"},{"key":"e_1_3_2_2_8_1","unstructured":"Agnese Chiatti Dolzodmaa Davaasuren Nilam Ram Prasenjit Mitra Byron Reeves and Thomas Robinson. 2019. Guess What\u2019s on my Screen? Clustering Smartphone Screenshots with Active Learning. arXiv preprint arXiv:1901.02701(2019).  Agnese Chiatti Dolzodmaa Davaasuren Nilam Ram Prasenjit Mitra Byron Reeves and Thomas Robinson. 2019. Guess What\u2019s on my Screen? Clustering Smartphone Screenshots with Active Learning. arXiv preprint arXiv:1901.02701(2019)."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984581"},{"key":"e_1_3_2_2_11_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018)."},{"key":"e_1_3_2_2_12_1","volume-title":"Puppetdroid: A user-centric ui exerciser for automatic dynamic analysis of similar android applications. arXiv preprint arXiv:1402.4826(2014).","author":"Gianazza Andrea","year":"2014","unstructured":"Andrea Gianazza , Federico Maggi , Aristide Fattori , Lorenzo Cavallaro , and Stefano Zanero . 2014 . Puppetdroid: A user-centric ui exerciser for automatic dynamic analysis of similar android applications. arXiv preprint arXiv:1402.4826(2014). Andrea Gianazza, Federico Maggi, Aristide Fattori, Lorenzo Cavallaro, and Stefano Zanero. 2014. Puppetdroid: A user-centric ui exerciser for automatic dynamic analysis of similar android applications. arXiv preprint arXiv:1402.4826(2014)."},{"key":"e_1_3_2_2_13_1","volume-title":"2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906)","author":"Hadsell Raia","year":"2006","unstructured":"Raia Hadsell , Sumit Chopra , and Yann LeCun . 2006 . Dimensionality reduction by learning an invariant mapping . In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906) , Vol.\u00a02. IEEE, 1735\u20131742. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201906), Vol.\u00a02. IEEE, 1735\u20131742."},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594368.2594390"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236055"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01001"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300334"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2018.00037"},{"key":"e_1_3_2_2_20_1","volume-title":"Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services","author":"Leiva A.","year":"2020","unstructured":"Luis\u00a0 A. Leiva , Asutosh Hota , and Antti Oulasvirta . 2020 . Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services ( Oldenburg, Germany) (MobileHCI \u201920). Association for Computing Machinery, New York, NY, USA, Article 9, 4\u00a0pages. https:\/\/doi.org\/10.1145\/3406324.3410710 10.1145\/3406324.3410710 Luis\u00a0A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2020. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI \u201920). Association for Computing Machinery, New York, NY, USA, Article 9, 4\u00a0pages. https:\/\/doi.org\/10.1145\/3406324.3410710"},{"key":"e_1_3_2_2_21_1","volume-title":"Screen2Vec: Semantic Embedding of GUI Screens and GUI Components","author":"Jia-Jun Li Toby","unstructured":"Toby Jia-Jun Li , Lindsay Popowski , Tom Mitchell , and Brad\u00a0 A Myers . 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components . Association for Computing Machinery , New York, NY, USA . https:\/\/doi.org\/10.1145\/3411764.3445049 10.1145\/3411764.3445049 Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad\u00a0A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445049"},{"key":"e_1_3_2_2_22_1","volume-title":"2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23\u201326","author":"Li Yuanchun","year":"2017","unstructured":"Yuanchun Li , Ziyue Yang , Yao Guo , and Xiangqun Chen . 2017 . Droidbot: a lightweight ui-guided test input generator for android . In 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23\u201326 . Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23\u201326."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00104"},{"key":"e_1_3_2_2_24_1","volume-title":"11th USENIX symposium on networked systems design and implementation (NSDI 14)","author":"Liu Bin","year":"2014","unstructured":"Bin Liu , Suman Nath , Ramesh Govindan , and Jie Liu . 2014 . DECAF: Detecting and characterizing ad fraud in mobile apps . In 11th USENIX symposium on networked systems design and implementation (NSDI 14) . 57\u201370. Bin Liu, Suman Nath, Ramesh Govindan, and Jie Liu. 2014. DECAF: Detecting and characterizing ad fraud in mobile apps. In 11th USENIX symposium on networked systems design and implementation (NSDI 14). 57\u201370."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2018.10205"},{"key":"e_1_3_2_2_26_1","volume-title":"Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology","author":"Liu F.","year":"2018","unstructured":"Thomas\u00a0 F. Liu , Mark Craft , Jason Situ , Ersin Yumer , Radomir Mech , and Ranjitha Kumar . 2018 . Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology ( Berlin, Germany) (UIST \u201918). ACM, New York, NY, USA, 569\u2013579. https:\/\/doi.org\/10.1145\/3242587.3242650 10.1145\/3242587.3242650 Thomas\u00a0F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST \u201918). ACM, New York, NY, USA, 569\u2013579. https:\/\/doi.org\/10.1145\/3242587.3242650"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3029806.3029819"},{"key":"e_1_3_2_2_28_1","volume-title":"Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 ( 2015 ), 91\u201399. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015), 91\u201399."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"crossref","unstructured":"Amanpreet Singh Vivek Natarajan Meet Shah Yu Jiang Xinlei Chen Dhruv Batra Devi Parikh and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8317\u20138326.  Amanpreet Singh Vivek Natarajan Meet Shah Yu Jiang Xinlei Chen Dhruv Batra Devi Parikh and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8317\u20138326.","DOI":"10.1109\/CVPR.2019.00851"},{"key":"e_1_3_2_2_30_1","volume-title":"Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490(2019).","author":"Tan Hao","year":"2019","unstructured":"Hao Tan and Mohit Bansal . 2019 . Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490(2019). Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490(2019)."},{"key":"e_1_3_2_2_31_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan\u00a0N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998\u20136008.  Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan\u00a0N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998\u20136008."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.03.003"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468554"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474763"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380416"},{"key":"e_1_3_2_2_36_1","volume-title":"Fundamental Approaches to Software Engineering, Vittorio Cortellessa and D\u00e1niel Varr\u00f3 (Eds.)","author":"Yang Wei","unstructured":"Wei Yang , Mukul\u00a0 R. Prasad , and Tao Xie . 2013. A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications . In Fundamental Approaches to Software Engineering, Vittorio Cortellessa and D\u00e1niel Varr\u00f3 (Eds.) . Springer Berlin Heidelberg, Berlin , Heidelberg , 250\u2013265. Wei Yang, Mukul\u00a0R. Prasad, and Tao Xie. 2013. A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications. In Fundamental Approaches to Software Engineering, Vittorio Cortellessa and D\u00e1niel Varr\u00f3 (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 250\u2013265."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00090"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445186"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242616"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00030"}],"event":{"name":"IUI '22: 27th International Conference on Intelligent User Interfaces","location":"Helsinki Finland","acronym":"IUI '22","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["27th International Conference on Intelligent User Interfaces"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490099.3511109","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490099.3511109","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:23Z","timestamp":1750188683000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490099.3511109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,22]]},"references-count":41,"alternative-id":["10.1145\/3490099.3511109","10.1145\/3490099"],"URL":"https:\/\/doi.org\/10.1145\/3490099.3511109","relation":{},"subject":[],"published":{"date-parts":[[2022,3,22]]},"assertion":[{"value":"2022-03-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}