{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T15:55:59Z","timestamp":1783007759059,"version":"3.54.5"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","funder":[{"name":"Research Grants Council, University Grants Committee","award":["SRFS2425-4S03"],"award-info":[{"award-number":["SRFS2425-4S03"]}]},{"name":"Ministry of Education - Singapore","award":["Academic Research Fund (AcRF) Tier 1 grant"],"award-info":[{"award-number":["Academic Research Fund (AcRF) Tier 1 grant"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Websites are critical in today\u2019s digital world, with over 1.11 billion currently active and approximately 252,000 new sites launched daily. Converting website layout design into functional UI code is a time-consuming yet indispensable step of website development. Manual methods of converting visual designs into functional code present significant challenges, especially for non-experts. To explore automatic design-to-code solutions, we first conduct a motivating study on GPT-4o and identify three types of issues in generating UI code: element omission, element distortion, and element misarrangement. We further reveal that a focus on smaller visual segments can help multimodal large language models (MLLMs) mitigate these failures in the generation process.<\/jats:p>\n          <jats:p>In this paper, we propose DCGen, a divide-and-conquer-based approach to automate the translation of webpage design to UI code. DCGen starts by dividing screenshots into manageable segments, generating code for each segment, and then reassembling them into complete UI code for the entire screenshot. We conduct extensive testing with a dataset comprised of real-world websites and various MLLMs and demonstrate that DCGen achieves up to a 15% improvement in visual similarity and 8% in code similarity for large input images. Human evaluations show that DCGen can help developers implement webpages significantly faster and more similar to the UI designs. To the best of our knowledge, DCGen is the first segment-aware MLLM-based approach for generating UI code directly from screenshots.<\/jats:p>","DOI":"10.1145\/3729364","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:15:34Z","timestamp":1750346134000},"page":"2099-2122","source":"Crossref","is-referenced-by-count":12,"title":["Divide-and-Conquer: Generating UI Code from Screenshots"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-6739-4675","authenticated-orcid":false,"given":"Yuxuan","family":"Wan","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3935-7328","authenticated-orcid":false,"given":"Chaozheng","family":"Wang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4456-072X","authenticated-orcid":false,"given":"Yi","family":"Dong","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9803-8204","authenticated-orcid":false,"given":"Wenxuan","family":"Wang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6323-1402","authenticated-orcid":false,"given":"Shuqing","family":"Li","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8798-5667","authenticated-orcid":false,"given":"Yintong","family":"Huo","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3666-5798","authenticated-orcid":false,"given":"Michael","family":"Lyu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. 17+ Surprising WordPress Statistics You Should Not Miss [2024]. WPDeveloper https:\/\/wpdeveloper.com\/wordpress-statistics-2024 Accessed: 2024-05-30"},{"key":"e_1_2_1_2_1","unstructured":"2024. Differences Between Mobile App and Web Development. KnowledgeHut https:\/\/www.knowledgehut.com\/blog\/programming\/differences-mobile-app-web-development"},{"key":"e_1_2_1_3_1","unstructured":"2024. How Many Websites Are There in 2024? (13 Latest Statistics). TechJury https:\/\/techjury.net\/blog\/how-many-websites-are-there\/ Accessed: 2024-05-30"},{"key":"e_1_2_1_4_1","unstructured":"2024. Website & Mobile App Design: Differences In 2024? Inkbot Design https:\/\/inkbotdesign.com\/differences-website-mobile-app-design\/"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-sen.2019.0378"},{"key":"e_1_2_1_6_1","volume-title":"Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35","author":"Alayrac Jean-Baptiste","year":"2022","unstructured":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, and Malcolm Reynolds. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35 (2022), 23716\u201323736."},{"key":"e_1_2_1_7_1","unstructured":"Anthropic. 2024. Claude. https:\/\/www.anthropic.com\/claude Accessed: 2024-06-06"},{"key":"e_1_2_1_8_1","unstructured":"Anthropic. 2024. Vision Documentation. https:\/\/docs.anthropic.com\/en\/docs\/vision Accessed: 2024-06-06"},{"key":"e_1_2_1_9_1","volume-title":"Exploring Distributional Shifts in Large Language Models for Code Analysis. In Conference on Empirical Methods in Natural Language Processing. https:\/\/api.semanticscholar.org\/CorpusID:257557735","author":"Arakelyan Shushan","year":"2023","unstructured":"Shushan Arakelyan, Rocktim Jyoti Das, Yi Mao, and Xiang Ren. 2023. Exploring Distributional Shifts in Large Language Models for Code Analysis. In Conference on Empirical Methods in Natural Language Processing. https:\/\/api.semanticscholar.org\/CorpusID:257557735"},{"key":"e_1_2_1_10_1","volume-title":"2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT), 1\u20134. https:\/\/api.semanticscholar.org\/CorpusID:195223316","author":"Asiroglu Batuhan","year":"2019","unstructured":"Batuhan Asiroglu, B\u00fc\u015fta R\u00fcmeysa Mete, Eyy\u00fcp Yildiz, Yagiz Nal\u00e7akan, Alper Sezen, Mustafa Da\u011ftekin, and Tolga Ensari. 2019. Automatic HTML Code Generation from Mock-Up Images Using Machine Learning Techniques. 2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT), 1\u20134. https:\/\/api.semanticscholar.org\/CorpusID:195223316"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3220134.3220135"},{"key":"e_1_2_1_12_1","volume-title":"Beong woo Kwak, Moohyeon Kim, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, and Jinyoung Yeo.","author":"Chae Hyungjoo","year":"2024","unstructured":"Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu iunn Ong, Beong woo Kwak, Moohyeon Kim, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, and Jinyoung Yeo. 2024. Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models. ArXiv, abs\/2404.02575 (2024), https:\/\/api.semanticscholar.org\/CorpusID:268876471"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 40th International Conference on Software Engineering. 665\u2013676","author":"Chen C.","unstructured":"C. Chen, T. Su, G. Meng, Z. Xing, and Y. Liu. 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. In Proceedings of the 40th International Conference on Software Engineering. 665\u2013676."},{"key":"e_1_2_1_14_1","volume-title":"On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages. 2022 IEEE\/ACM 30th International Conference on Program Comprehension (ICPC), 401\u2013412","author":"Chen Fuxiang","year":"2022","unstructured":"Fuxiang Chen, Fateme Moradian Fard, David Lo, and Timofey Bryksin. 2022. On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages. 2022 IEEE\/ACM 30th International Conference on Program Comprehension (ICPC), 401\u2013412. https:\/\/api.semanticscholar.org\/CorpusID:248266381"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 18030\u201318040","author":"Chen Jun","year":"2022","unstructured":"Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 18030\u201318040."},{"key":"e_1_2_1_16_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harrison Edwards Yura Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such David W. Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Igor Babuschkin Suchir Balaji Shantanu Jain Andrew Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew M. Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. ArXiv abs\/2107.03374 (2021) https:\/\/api.semanticscholar.org\/CorpusID:235755472"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00530-021-00804-7"},{"key":"e_1_2_1_18_1","volume-title":"Teaching Large Language Models to Self-Debug. ArXiv, abs\/2304.05128","author":"Chen Xinyun","year":"2023","unstructured":"Xinyun Chen, Maxwell Lin, Nathanael Sch\u00e4rli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. ArXiv, abs\/2304.05128 (2023), https:\/\/api.semanticscholar.org\/CorpusID:258059885"},{"key":"e_1_2_1_19_1","unstructured":"CSS Chopper. 2024. How to Convert PSD to HTML : A Complete Guide. https:\/\/www.csschopper.com\/blog\/how-to-convert-psd-to-html-a-complete-guide\/ Accessed: 31-05-2024"},{"key":"e_1_2_1_20_1","first-page":"1","article-title":"Scaling instruction-finetuned language models","volume":"25","author":"Chung Hyung Won","year":"2024","unstructured":"Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, and Siddhartha Brahma. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25, 70 (2024), 1\u201353.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_21_1","volume-title":"Web","author":"Cizotto A. A. J.","unstructured":"A. A. J. Cizotto, R. C. T. de Souza, V. C. Mariani, and L. dos Santos Coelho. 2023. Web pages from mockup design based on convolutional neural network and class activation mapping. Multimedia Tools and Applications, 1\u201327."},{"key":"e_1_2_1_22_1","volume-title":"Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi.","author":"Dai Wenliang","year":"2023","unstructured":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. ArXiv, abs\/2305.06500 (2023), https:\/\/api.semanticscholar.org\/CorpusID:258615266"},{"key":"e_1_2_1_23_1","volume-title":"Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers. ArXiv, abs\/2210.16494","author":"Dibia Victor C.","year":"2022","unstructured":"Victor C. Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2022. Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers. ArXiv, abs\/2210.16494 (2022), https:\/\/api.semanticscholar.org\/CorpusID:253237523"},{"key":"e_1_2_1_24_1","volume-title":"Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang.","author":"Ding Hantian","year":"2023","unstructured":"Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Robert Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. A Static Evaluation of Code Completion by Large Language Models. ArXiv, abs\/2306.03203 (2023), https:\/\/api.semanticscholar.org\/CorpusID:259088657"},{"key":"e_1_2_1_25_1","volume-title":"Self-collaboration Code Generation via ChatGPT. ArXiv, abs\/2304.07590","author":"Dong Yihong","year":"2023","unstructured":"Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. ArXiv, abs\/2304.07590 (2023), https:\/\/api.semanticscholar.org\/CorpusID:258179537"},{"key":"e_1_2_1_26_1","volume-title":"PaLM-E: An Embodied Multimodal Language Model. In International Conference on Machine Learning. 8469\u20138488","author":"Driess Danny","year":"2023","unstructured":"Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, and Tianhe Yu. 2023. PaLM-E: An Embodied Multimodal Language Model. In International Conference on Machine Learning. 8469\u20138488."},{"key":"e_1_2_1_27_1","volume-title":"ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. ArXiv, abs\/2308.01861","author":"Du Xueying","year":"2023","unstructured":"Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. ArXiv, abs\/2308.01861 (2023), https:\/\/api.semanticscholar.org\/CorpusID:260439062"},{"key":"e_1_2_1_28_1","volume-title":"AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. ArXiv, abs\/2305.14387","author":"Dubois Yann","year":"2023","unstructured":"Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. ArXiv, abs\/2305.14387 (2023), https:\/\/api.semanticscholar.org\/CorpusID:258865545"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608137"},{"key":"e_1_2_1_30_1","volume-title":"Lyu","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Xinjie Wen, Cuiyun Gao, Wenxuan Wang, and Michael R. Lyu. 2023. Constructing Effective In-Context Demonstration for Code Intelligence Tasks: An Empirical Study. ArXiv, abs\/2304.07575 (2023), https:\/\/api.semanticscholar.org\/CorpusID:263867793"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Henry Gilbert Michael Sandborn Douglas C. Schmidt Jesse Spencer-Smith and Jules White. 2023. Semantic Compression with Large Language Models. 2023 Tenth International Conference on Social Networks Analysis Management and Security (SNAMS) 1\u20138. https:\/\/api.semanticscholar.org\/CorpusID:258309482","DOI":"10.1109\/SNAMS60348.2023.10375400"},{"key":"e_1_2_1_32_1","unstructured":"Google. 2024. Gemini API. https:\/\/ai.google.dev\/gemini-api Accessed: 2024-06-06"},{"key":"e_1_2_1_33_1","unstructured":"Google AI. 2024. API Overview. https:\/\/ai.google.dev\/gemini-api\/docs\/api-overview Accessed: 2024-06-06"},{"key":"e_1_2_1_34_1","volume-title":"Assemble Foundation Models for Automatic Code Summarization. 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 935\u2013946","author":"Gu Jian","unstructured":"Jian Gu, Pasquale Salza, and Harald C. Gall. 2022. Assemble Foundation Models for Automatic Code Summarization. 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 935\u2013946. https:\/\/api.semanticscholar.org\/CorpusID:245986582"},{"key":"e_1_2_1_35_1","volume-title":"Levenshtein Distance Technique in Dictionary Lookup Methods: An Improved Approach. ArXiv, abs\/1101.1232","author":"Haldar Rishin","year":"2011","unstructured":"Rishin Haldar and Debajyoti Mukhopadhyay. 2011. Levenshtein Distance Technique in Dictionary Lookup Methods: An Improved Approach. ArXiv, abs\/1101.1232 (2011), https:\/\/api.semanticscholar.org\/CorpusID:1049293"},{"key":"e_1_2_1_36_1","volume-title":"Large Language Models for Software Engineering: A Systematic Literature Review. ArXiv, abs\/2308.10620","author":"Hou Xinying","year":"2023","unstructured":"Xinying Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John C. Grundy, and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. ArXiv, abs\/2308.10620 (2023), https:\/\/api.semanticscholar.org\/CorpusID:261048648"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.08.028"},{"key":"e_1_2_1_38_1","volume-title":"SelfEvolve: A Code Evolution Framework via Large Language Models. ArXiv, abs\/2306.02907","author":"Jiang Shuyang","year":"2023","unstructured":"Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. SelfEvolve: A Code Evolution Framework via Large Language Models. ArXiv, abs\/2306.02907 (2023), https:\/\/api.semanticscholar.org\/CorpusID:259076266"},{"key":"e_1_2_1_39_1","volume-title":"Likert scale: Explored and explained. British journal of applied science & technology, 7, 4","author":"Joshi Ankur","year":"2015","unstructured":"Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology, 7, 4 (2015), 396\u2013403."},{"key":"e_1_2_1_40_1","first-page":"1","volume-title":"Designing User Experience for Mobile Apps: Long-Term Product Owner Perspective. 2013 20th Asia-Pacific Software Engineering Conference (APSEC)","author":"Kuusinen Kati","year":"2013","unstructured":"Kati Kuusinen and Tommi Mikkonen. 2013. Designing User Experience for Mobile Apps: Long-Term Product Owner Perspective. 2013 20th Asia-Pacific Software Engineering Conference (APSEC), 1 (2013), 535\u2013540. https:\/\/api.semanticscholar.org\/CorpusID:18632493"},{"key":"e_1_2_1_41_1","volume-title":"Classifying and Qualifying GUI Defects. 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), 1\u201310","author":"Lelli Val\u00e9ria","year":"2015","unstructured":"Val\u00e9ria Lelli, Arnaud Blouin, and Beno\u00eet Baudry. 2015. Classifying and Qualifying GUI Defects. 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), 1\u201310. https:\/\/api.semanticscholar.org\/CorpusID:2288032"},{"key":"e_1_2_1_42_1","volume-title":"International conference on machine learning. 19730\u201319742","author":"Li Junnan","year":"2023","unstructured":"Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. 19730\u201319742."},{"key":"e_1_2_1_43_1","volume-title":"Enabling Programming Thinking in Large Language Models Toward Code Generation. ArXiv, abs\/2305.06599","author":"Li Jia","year":"2023","unstructured":"Jia Li, Ge Li, Yongming Li, and Zhi Jin. 2023. Enabling Programming Thinking in Large Language Models Toward Code Generation. ArXiv, abs\/2305.06599 (2023), https:\/\/api.semanticscholar.org\/CorpusID:263896057"},{"key":"e_1_2_1_44_1","volume-title":"2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE), 14\u201326","author":"Li Tsz On","unstructured":"Tsz On Li, Wen yi Zong, Yibo Wang, Haoye Tian, Y. Wang, and S. C. Cheung. 2023. Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting. 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE), 14\u201326. https:\/\/api.semanticscholar.org\/CorpusID:258298446"},{"key":"e_1_2_1_45_1","volume-title":"CCTEST: Testing and Repairing Code Completion Systems. 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE), 1238\u20131250","author":"Li Zongjie","year":"2022","unstructured":"Zongjie Li, Chaozheng Wang, Zhibo Liu, Hao Wang, Shuai Wang, and Cuiyun Gao. 2022. CCTEST: Testing and Repairing Code Completion Systems. 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE), 1238\u20131250. https:\/\/api.semanticscholar.org\/CorpusID:251623193"},{"key":"e_1_2_1_46_1","volume-title":"2022 IEEE\/ACM 44th International Conference on Software Engineering (ICSE), 2279\u20132290","author":"Mastropaolo Antonio","year":"2022","unstructured":"Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using Deep Learning to Generate Complete Log Statements. 2022 IEEE\/ACM 44th International Conference on Software Engineering (ICSE), 2279\u20132290. https:\/\/api.semanticscholar.org\/CorpusID:245906103"},{"key":"e_1_2_1_47_1","volume-title":"2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE), 336\u2013347","author":"Mastropaolo Antonio","year":"2021","unstructured":"Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader-Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE), 336\u2013347. https:\/\/api.semanticscholar.org\/CorpusID:231786586"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3059968"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2844788"},{"key":"e_1_2_1_50_1","volume-title":"Automated Reporting of GUI Design Violations for Mobile Apps. 2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE), 165\u2013175","author":"Moran Kevin","year":"2018","unstructured":"Kevin Moran, Boyang Li, Carlos Bernal-C\u00e1rdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. 2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE), 165\u2013175. https:\/\/api.semanticscholar.org\/CorpusID:3634687"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. 224\u2013235","author":"Natarajan S.","unstructured":"S. Natarajan and C. Csallner. 2018. P2A: A tool for converting pixels to animated mobile application user interfaces. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. 224\u2013235."},{"key":"e_1_2_1_52_1","volume-title":"2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 248\u2013259","author":"Nguyen T. A.","unstructured":"T. A. Nguyen and C. Csallner. 2015. Reverse engineering mobile application user interfaces with remaui (t). In 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 248\u2013259."},{"key":"e_1_2_1_53_1","volume-title":"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations. https:\/\/api.semanticscholar.org\/CorpusID:252668917","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations. https:\/\/api.semanticscholar.org\/CorpusID:252668917"},{"key":"e_1_2_1_54_1","unstructured":"OpenAI. [n. d.]. GPT-4 System Card. https:\/\/openai.com\/index\/gpt-4v-system-card\/ Accessed: 2024-05-30"},{"key":"e_1_2_1_55_1","unstructured":"OpenAI. 2024. Hello GPT-4o. https:\/\/openai.com\/index\/hello-gpt-4o\/ Accessed: 2024-06-06"},{"key":"e_1_2_1_56_1","unstructured":"OpenAI. 2024. Vision Guide. https:\/\/platform.openai.com\/docs\/guides\/vision Accessed: 2024-06-06"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings 2019 Network and Distributed System Security Symposium, https:\/\/api.semanticscholar.org\/CorpusID:56367624","author":"Pochat Victor Le","year":"2018","unstructured":"Victor Le Pochat, Tom van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczy\u0144ski, and Wouter Joosen. 2018. Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation. Proceedings 2019 Network and Distributed System Security Symposium, https:\/\/api.semanticscholar.org\/CorpusID:56367624"},{"key":"e_1_2_1_58_1","volume-title":"Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning. https:\/\/api.semanticscholar.org\/CorpusID:231591445","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning. https:\/\/api.semanticscholar.org\/CorpusID:231591445"},{"key":"e_1_2_1_59_1","unstructured":"RapidFuzz. 2024. Levenshtein Distance \u2014 RapidFuzz. https:\/\/rapidfuzz.github.io\/Levenshtein\/levenshtein.html Accessed: 2024-02-26"},{"key":"e_1_2_1_60_1","volume-title":"Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779\u2013788","author":"Redmon Joseph","year":"2015","unstructured":"Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2015. You Only Look Once: Unified, Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779\u2013788. https:\/\/api.semanticscholar.org\/CorpusID:206594738"},{"key":"e_1_2_1_61_1","unstructured":"Raj Sanghvi. 2024. Differences Between Web and Mobile App Development. BitCot https:\/\/www.bitcot.com\/blog\/differences-between-web-and-mobile-app-development\/"},{"key":"e_1_2_1_62_1","volume-title":"Design2Code: How Far Are We From Automating Front-End Engineering? ArXiv, abs\/2403.03163","author":"Si Chenglei","year":"2024","unstructured":"Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024. Design2Code: How Far Are We From Automating Front-End Engineering? ArXiv, abs\/2403.03163 (2024), https:\/\/api.semanticscholar.org\/CorpusID:268248801"},{"key":"e_1_2_1_63_1","volume-title":"Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. ArXiv, abs\/2401.06209","author":"Tong Shengbang","year":"2024","unstructured":"Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. ArXiv, abs\/2401.06209 (2024), https:\/\/api.semanticscholar.org\/CorpusID:266976992"},{"key":"e_1_2_1_64_1","first-page":"200","article-title":"Multimodal few-shot learning with frozen language models","volume":"34","author":"Tsimpoukelli Maria","year":"2021","unstructured":"Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34 (2021), 200\u2013212.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_65_1","volume-title":"Lyu","author":"Wan Yuxuan","year":"2024","unstructured":"Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, and Michael R. Lyu. 2024. MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs. ArXiv, abs\/2412.15310 (2024), https:\/\/api.semanticscholar.org\/CorpusID:274965541"},{"key":"e_1_2_1_66_1","volume-title":"F. Xia, Quoc Le, and Denny Zhou.","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv, abs\/2201.11903 (2022), https:\/\/api.semanticscholar.org\/CorpusID:246411621"},{"key":"e_1_2_1_67_1","volume-title":"Chi, Quoc V Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35 (2022), 24824\u201324837."},{"key":"e_1_2_1_68_1","volume-title":"The 34th Annual ACM Symposium on User Interface Software and Technology. 470\u2013483","author":"Wu J.","unstructured":"J. Wu, X. Zhang, J. Nichols, and J. P. Bigham. 2021. Screen parsing: Towards reverse engineering of UI models from screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology. 470\u2013483."},{"key":"e_1_2_1_69_1","volume-title":"Lyu","author":"Xiao Jingyu","year":"2024","unstructured":"Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zhiyao Xu, and Michael R. Lyu. 2024. Interaction2Code: How Far Are We From Automatic Interactive Webpage Generation? ArXiv, abs\/2411.03292 (2024), https:\/\/api.semanticscholar.org\/CorpusID:273821629"},{"key":"e_1_2_1_70_1","first-page":"e2369","article-title":"image2emmet: Automatic code generation from web user interface image","volume":"33","author":"Xu Y.","year":"2021","unstructured":"Y. Xu, L. Bo, X. Sun, B. Li, J. Jiang, and W. Zhou. 2021. image2emmet: Automatic code generation from web user interface image. Journal of Software: Evolution and Process, 33, 8 (2021), e2369.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_2_1_71_1","volume-title":"The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision). ArXiv, abs\/2309.17421","author":"Yang Zhengyuan","year":"2023","unstructured":"Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision). ArXiv, abs\/2309.17421 (2023), https:\/\/api.semanticscholar.org\/CorpusID:263310951"},{"key":"e_1_2_1_72_1","volume-title":"A Survey on Multimodal Large Language Models. ArXiv, abs\/2306.13549","author":"Yin Shukang","year":"2023","unstructured":"Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A Survey on Multimodal Large Language Models. ArXiv, abs\/2306.13549 (2023), https:\/\/api.semanticscholar.org\/CorpusID:259243718"},{"key":"e_1_2_1_73_1","volume-title":"CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In International Conference on Software Engineering. https:\/\/api.semanticscholar.org\/CorpusID:256459413","author":"Yu Hao","year":"2023","unstructured":"Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yu Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang. 2023. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In International Conference on Software Engineering. https:\/\/api.semanticscholar.org\/CorpusID:256459413"},{"key":"e_1_2_1_74_1","volume-title":"Evaluating Direct Manipulation Operations for Constraint-Based Layout. In IFIP TC13 International Conference on Human-Computer Interaction. https:\/\/api.semanticscholar.org\/CorpusID:8243987","author":"Zeidler Clemens","year":"2013","unstructured":"Clemens Zeidler, Christof Lutteroth, Wolfgang Stuerzlinger, and Gerald Weber. 2013. Evaluating Direct Manipulation Operations for Constraint-Based Layout. In IFIP TC13 International Conference on Human-Computer Interaction. https:\/\/api.semanticscholar.org\/CorpusID:8243987"},{"key":"e_1_2_1_75_1","volume-title":"LIMA: Less Is More for Alignment. ArXiv, abs\/2305.11206","author":"Zhou Chunting","year":"2023","unstructured":"Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment. ArXiv, abs\/2305.11206 (2023), https:\/\/api.semanticscholar.org\/CorpusID:258822910"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729364","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:21:18Z","timestamp":1750346478000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729364"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":75,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3729364"],"URL":"https:\/\/doi.org\/10.1145\/3729364","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}