{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:58:08Z","timestamp":1791460688914,"version":"4.3.4"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","license":[{"start":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T00:00:00Z","timestamp":1750550400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Converting user interfaces into code (UI2Code) is a crucial step in website development, which is time-consuming and labor-intensive. The automation of UI2Code is essential to streamline this task, beneficial for improving the development efficiency. There exist deep learning-based methods for the task; however, they heavily rely on a large amount of labeled training data and struggle with generalizing to real-world, unseen web page designs. The advent of Multimodal Large Language Models (MLLMs) presents potential for alleviating the issue, but they are difficult to comprehend the complex layouts in UIs and generate the accurate code with layout preserved. To address these issues, we propose LayoutCoder, a novel MLLM-based framework generating UI code from real-world webpage images, which includes three key modules: (1) Element Relation Construction, which aims at capturing UI layout by identifying and grouping components with similar structures; (2) UI Layout Parsing, which aims at generating UI layout trees for guiding the subsequent code generation process; and (3) Layout-Guided Code Fusion, which aims at producing the accurate code with layout preserved. For evaluation, we build a new benchmark dataset which involves 350 real-world websites named Snap2Code, divided into seen and unseen parts for mitigating the data leakage issue, besides the popular dataset Design2Code. Extensive evaluation shows the superior performance of Layout Coder over the state-of-the-art approaches. Compared with the best-performing baseline, Layout Coder improves 10.14% in the BLEU score and 3.95% in the CLIP score on average across all datasets.<\/jats:p>","DOI":"10.1145\/3728925","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1123-1145","source":"Crossref","is-referenced-by-count":15,"title":["MLLM-Based UI2Code Automation Guided by UI Layout Information"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-9090-1832","authenticated-orcid":false,"given":"Fan","family":"Wu","sequence":"first","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4774-2434","authenticated-orcid":false,"given":"Cuiyun","family":"Gao","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6323-1402","authenticated-orcid":false,"given":"Shuqing","family":"Li","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2115-9921","authenticated-orcid":false,"given":"Xin-Cheng","family":"Wen","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1012-5301","authenticated-orcid":false,"given":"Qing","family":"Liao","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_3_1_2_2","volume-title":"Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS \u201922)","author":"Alayrac Jean-Baptiste","year":"2024","unstructured":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, 2024. Flamingo: a visual language model for few-shot learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS \u201922). Curran Associates Inc., Red Hook, NY, USA, Article 1723, 21 pages."},{"key":"e_1_3_1_3_2","unstructured":"Anthropic. 2024. Claude 3.5. https:\/\/www.anthropic.com. Accessed: 2024-10-30."},{"key":"e_1_3_1_4_2","first-page":"1","volume-title":"2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT)","author":"A\u015firo\u011flu Batuhan","year":"2019","unstructured":"Batuhan A\u015firo\u011flu, B\u00fc\u015fra R\u00fcmeysa Mete, Eyy\u00fcp Yildiz, Ya\u011fiz Nal\u00e7akan, Alper Sezen, Mustafa Da\u011ftekin, and Tolga Ensari. 2019. Automatic HTML code generation from mock-up images using machine learning techniques. In 2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT). Ieee, 1\u20134."},{"key":"e_1_3_1_5_2","unstructured":"Microsoft Azure. 2018. Turn your whiteboard sketches to working code in seconds with sketch2code. https:\/\/azure.microsoft.com\/en-us\/blog\/turn-your-whiteboard-sketches-to-working-code-in-seconds-with-sketch2code\/ Accessed: 2024-10-30."},{"key":"e_1_3_1_6_2","unstructured":"Jinze Bai Shuai Bai Shusheng Yang Shijie Wang Sinan Tan Peng Wang Junyang Lin Chang Zhou and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding Localization Text Reading and Beyond. arXiv:2308.12966 [cs.CV] https:\/\/arxiv.org\/abs\/2308.12966"},{"key":"e_1_3_1_7_2","unstructured":"Daniel Baul\u00e9 Christiane Gresse von Wangenheim Aldo von Wangenheim Jean C. R. Hauck and Edson C. Vargas J\u00fanior. 2021. Automatic code generation from sketches of mobile applications in end-user development using Deep Learning. arXiv:2103.05704 [cs.HC] https:\/\/arxiv.org\/abs\/2103.05704"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3220134.3220135"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180240"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00530-021-00804-7"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","unstructured":"Andr\u00e9 Armstrong Janino Cizotto Rodrigo Clemente Thom de Souza Viviana Cocco Mariani and Leandro dos Santos Coelho. 2023. Web pages from mockup design based on convolutional neural network and class activation mapping. 82 25 (March 2023) 38771\u201338797. doi:10.1007\/s11042-023-15108-3","DOI":"10.1007\/s11042-023-15108-3"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_3_1_13_2","unstructured":"Martin Ester Hans-Peter Kriegel J\u00f6rg Sander and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise (KDD\u201996). AAAI Press 226\u2013231."},{"key":"e_1_3_1_14_2","unstructured":"Jack Hessel Ari Holtzman Maxwell Forbes Ronan Le Bras and Yejin Choi. 2022. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. arXiv:2104.08718 [cs.CV] https:\/\/arxiv.org\/abs\/2104.08718"},{"key":"e_1_3_1_15_2","unstructured":"Vanita Jain Piyush Agrawal Subham Banga Rishabh Kapoor and Shashwat Gulyani. 2019. Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network. arXiv:1910.08930 [cs.CV] https:\/\/arxiv.org\/abs\/1910.08930"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01765"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i1.19994"},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","unstructured":"Alexander Kirillov Eric Mintun Nikhila Ravi Hanzi Mao Chloe Rolland Laura Gustafson Tete Xiao Spencer Whitehead Alexander C. Berg Wan-Yen Lo Piotr Doll\u00e1r and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 [cs.CV] https:\/\/arxiv.org\/abs\/2304.02643","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"e_1_3_1_19_2","unstructured":"Hugo Lauren\u00e7on L\u00e9o Tronchon and Victor Sanh. 2024. Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset. arXiv:2403.09029 [cs.HC] https:\/\/arxiv.org\/abs\/2403.09029"},{"key":"e_1_3_1_20_2","volume-title":"Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML\u201923)","author":"Lee Kenton","year":"2023","unstructured":"Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, . 2023. Pix2Struct: screenshot parsing as pretraining for visual language understanding. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML\u201923). JMLR.org, Article 780, 20 pages."},{"key":"e_1_3_1_21_2","unstructured":"Junnan Li Dongxu Li Silvio Savarese and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597 [cs.CV] https:\/\/arxiv.org\/abs\/2301.12597"},{"key":"e_1_3_1_22_2","unstructured":"Jianan Li Jimei Yang Aaron Hertzmann Jianming Zhang and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. arXiv:1901.06767 [cs.CV] https:\/\/arxiv.org\/abs\/1901.06767"},{"key":"e_1_3_1_23_2","unstructured":"Haotian Liu Chunyuan Li Qingyang Wu and Yong Jae Lee. 2023. Visual Instruction Tuning. ArXiv abs\/2304.08485 (2023). https:\/\/api.semanticscholar.org\/CorpusID:258179774"},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1109\/TSE.2018.2844788","article-title":"Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps.","volume":"46","author":"Moran Kevin","year":"2018","unstructured":"Kevin Moran, Carlos Bernal-C\u00e1rdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46 (2018), 196\u2013221. https:\/\/api.semanticscholar.org\/CorpusID:3629347","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","unstructured":"Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 248\u2013259. https:\/\/api.semanticscholar.org\/CorpusID:7499368","DOI":"10.1109\/ASE.2015.32"},{"key":"e_1_3_1_26_2","unstructured":"OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad . 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_1_28_2","first-page":"8748","volume-title":"International conference on machine learning","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, . 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748\u20138763."},{"key":"e_1_3_1_29_2","unstructured":"Alex Robinson. 2019. Sketch2code: Generating a website from a paper mockup. arXiv:1905.13750 [cs.CV] https:\/\/arxiv.org\/abs\/1905.13750"},{"key":"e_1_3_1_30_2","unstructured":"Andy Rutledge. 2009. Gestalt Principles \u2013 3: Proximity Uniform Connectedness and Good Continuation. https:\/\/andyrutledge.com\/gestalt-principles-3.html. Accessed: 2024-10-30."},{"key":"e_1_3_1_31_2","unstructured":"Chenglei Si Yanzhe Zhang Zhengyuan Yang Ruibo Liu and Diyi Yang. 2024. Design2Code: How Far Are We From Automating Front-End Engineering? arXiv preprint arXiv:2403.03163 (2024)."},{"key":"e_1_3_1_32_2","unstructured":"Digital Silk. 2024. How Many Websites Are There In 2024? https:\/\/www.digitalsilk.com\/digital-trends\/how-manywebsites-are-there\/. Accessed: 2024-10-31."},{"key":"e_1_3_1_33_2","unstructured":"Yuxuan Wan Chaozheng Wang Yi Dong Wenxuan Wang Shuqing Li Yintong Huo and Michael R. Lyu. 2024. Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach. arXiv:2406.16386 [cs.SE] https:\/\/arxiv.org\/abs\/2406.16386"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"Weihan Wang Qingsong Lv Wenmeng Yu Wenyi Hong Ji Qi Yan Wang Junhui Ji Zhuoyi Yang Lei Zhao Xixuan Song . 2023. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 (2023).","DOI":"10.52202\/079017-3860"},{"key":"e_1_3_1_35_2","unstructured":"Fan Wu. 2025. MLLM-Based UI2Code Automation Guided by UI Layout Information. https:\/\/github.com\/ay7u1009\/LayoutCoder\/. Accessed: 2025-04-05."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474763"},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"Shuhong Xiao Yunnong Chen Jiazhi Li Liuqing Chen Lingyun Sun and Tingting Zhou. 2024. Prototype2Code: End-toend Front-end Code Generation from UI Design Prototypes. arXiv:2405.04975 [cs.SE] https:\/\/arxiv.org\/abs\/2405.04975","DOI":"10.1115\/DETC2024-143139"},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Shuhong Xiao Yunnong Chen Yaxuan Song Liuqing Chen Lingyun Sun Yankun Zhen and Yanfang Chang. 2024. UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface. arXiv:2403.04984 [cs.SE] https:\/\/arxiv.org\/abs\/2403.04984","DOI":"10.1016\/j.displa.2024.102679"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417940"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549138"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","unstructured":"Jianwei Yang Jiasen Lu Stefan Lee Dhruv Batra and Devi Parikh. 2018. Graph R-CNN for Scene Graph Generation. In Proceedings of the European Conference on Computer Vision (ECCV).","DOI":"10.1007\/978-3-030-01246-5_41"},{"key":"e_1_3_1_42_2","unstructured":"Hao Zhang Feng Li Shilong Liu Lei Zhang Hang Su Jun Zhu Lionel M. Ni and Heung-Yeung Shum. 2022. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. arXiv:2203.03605 [cs.CV] https:\/\/arxiv.org\/abs\/2203.03605"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","unstructured":"Junyi Zhang Jiaqi Guo Shizhao Sun Jian-Guang Lou and D. Zhang. 2023. LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models. In 2023 IEEE\/CVF International Conference on Computer Vision (ICCV). 7192\u20137202. https:\/\/api.semanticscholar.org\/CorpusID:257636725","DOI":"10.1109\/ICCV51070.2023.00664"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","unstructured":"Wei Zhang Shangmin Luan Liqin Tian and Nima Jafari Navimipour. 2022. A Rapid Combined Model for Automatic Generating Web UI Codes. Wirel. Commun. Mob. Comput. 2022 (Jan. 2022) 10 pages. doi:10.1155\/2022\/4415479","DOI":"10.1155\/2022\/4415479"},{"key":"e_1_3_1_45_2","first-page":"275:1","volume-title":"CHI","author":"Zhang Xiaoyi","year":"2021","unstructured":"Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, . 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels.. In CHI, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bj\u00f8rn, and Steven Mark Drucker (Eds.). ACM, 275:1\u2013275:15. http:\/\/dblp.uni-trier.de\/db\/conf\/chi\/chi2021.html#ZhangGSWMYSNWFE21"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3290610"},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","unstructured":"Xu Zhong Jianbin Tang and Antonio Jimeno-Yepes. 2019. PubLayNet: Largest Dataset Ever for Document Layout Analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR). 1015\u20131022. https:\/\/api.semanticscholar.org\/CorpusID:201124789","DOI":"10.1109\/ICDAR.2019.00166"},{"key":"e_1_3_1_48_2","unstructured":"Ting Zhou Yanjie Zhao Xinyi Hou Xiaoyu Sun Kai Chen and Haoyu Wang. 2024. Bridging Design and Development with Automated Declarative UI Code Generation. arXiv:2409.11667 [cs.SE] https:\/\/arxiv.org\/abs\/2409.11667"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728925","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T13:52:02Z","timestamp":1787493122000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728925"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":47,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728925"],"URL":"https:\/\/doi.org\/10.1145\/3728925","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}