{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:56:29Z","timestamp":1774540589856,"version":"3.50.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>Manually translating web designs into code is a costly and time-consuming process, particularly due to the frequent iterations and refinements between designers and developers. Deep learning techniques, which possess the capability to automatically translate designs into functional code using an encoder-decoder architecture, have emerged as a promising solution to enhance this tedious process. However, many current methods depend on simplistic datasets that do not capture the diversity of components found in modern websites. Additionally, the potential of transformer-based models, which have enabled significant progress in vision and language modeling tasks due to their scalability and ability to handle cross-modal relationships, has not been investigated in this context. Addressing these limitations, this paper contributes with: 1) a web scraping methodology to automatically collect and process a diverse dataset of real-world websites with reduced noise and complexity, 2) a synthetic dataset of webpage mockups along with their sketched conversions, and 3) an evaluation of two recent multimodal transformer architectures on these proposed datasets. Results on synthetic and sketch-based datasets demonstrate the architectures potential as effective design-to-code automation solutions, while identifying remaining challenges in modeling real-world website complexity.<\/jats:p>","DOI":"10.1145\/3734190","type":"journal-article","created":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T11:35:59Z","timestamp":1751024159000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Advancing Code Generation from Visual Designs through Transformer-Based Architectures and Specialized Datasets"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3200-2348","authenticated-orcid":false,"given":"Tommaso","family":"Cal\u00f2","sequence":"first","affiliation":[{"name":"Dipartimento di Automatica e Informatica","place":["Torino, Italy"]},{"name":"Politecnico Di Torino","place":["Torino, Italy"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7647-6652","authenticated-orcid":false,"given":"Luigi","family":"De Russis","sequence":"additional","affiliation":[{"name":"Dipartimento di Automatica e Informatica","place":["Torino, Italy"]},{"name":"Politecnico di Torino","place":["Torino, Italy"]}]}],"member":"320","published-online":{"date-parts":[[2025,6,27]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"2023. Majestic Million. Available at: https:\/\/majestic.com\/reports\/majestic-million."},{"key":"e_1_3_3_3_2","first-page":"248","volume-title":"2015 30th IEEE\/ACM International Conference on Automated Software Engineering, ASE","author":"Anh Nguyen\u00a0Tuan","year":"2015","unstructured":"Nguyen\u00a0Tuan Anh and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with REMAUI. In 2015 30th IEEE\/ACM International Conference on Automated Software Engineering, ASE. IEEE, 248\u2013259."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/EBBT.2019.8741736"},{"key":"e_1_3_3_5_2","first-page":"64","volume-title":"Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems","author":"Bada Kim","year":"2018","unstructured":"Kim Bada, Sangmin Park, Taeyeon Won, Junyoung Heo, and Bongjae Kim. 2018. Deeplearning based web UI automatic programming. In Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems. ACM, 64\u201365."},{"key":"e_1_3_3_6_2","first-page":"601","volume-title":"2018 33rd IEEE\/ACM International Conference on Automated Software Engineering, ASE","author":"Bajammal M.","year":"2018","unstructured":"M. Bajammal, D. Mazinanian, and A. Mesbah. 2018. Generating reusable web components from mock-ups. In 2018 33rd IEEE\/ACM International Conference on Automated Software Engineering, ASE. IEEE, 601\u2013611."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197231.3197244"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3220134.3220135"},{"key":"e_1_3_3_9_2","volume-title":"Hack your design sprint: Wireframes to prototype in under 5 minutes","author":"Beltramelli T.","year":"2019","unstructured":"T. Beltramelli. 2019. Hack your design sprint: Wireframes to prototype in under 5 minutes. https:\/\/uizard.io\/"},{"key":"e_1_3_3_10_2","unstructured":"Bookmark. 2025. Bookmark AI Website Builder. https:\/\/www.bookmark.com\/. Accessed: 2025-04-27 (Inferred URL official site)."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208740"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180240"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/AI4Mobile.2019.8672718"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_3_3_15_2","first-page":"980","volume-title":"International Conference on Machine Learning","author":"Deng Y.","year":"2017","unstructured":"Y. Deng, A. Kanervisto, J. Ling, and A.M. Rush. 2017. Image-to-markup generation with coarse-to-fine attention. In International Conference on Machine Learning. PMLR, 980\u2013989."},{"key":"e_1_3_3_16_2","unstructured":"Elegant Themes Blog. 2025. Wix ADI Review 2025: Is It Really That Powerful? https:\/\/www.elegantthemes.com\/blog\/business\/wix-adi-review. Accessed: 2025-04-27."},{"key":"e_1_3_3_17_2","first-page":"6059","volume-title":"Advances in Neural Information Processing Systems","author":"Ellis K.","year":"2018","unstructured":"K. Ellis, D. Ritchie, A. Solar-Lezama, and J. Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. In Advances in Neural Information Processing Systems. 6059\u20136068."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.5220\/0010193600480058"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00060"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1062745.1062763"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"A. Halbe and A.R. Joshi. 2015. A novel approach to HTML page creation using neural network. Procedia Comput. Sci. 45 197\u2013204. http:\/\/www.sciencedirect.com\/science\/article\/pii\/S1877050915003580 International Conference on Advanced Computing Technologies and Applications (ICACTA).","DOI":"10.1016\/j.procs.2015.03.122"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292448.3292455"},{"key":"e_1_3_3_23_2","volume-title":"The UX Book: Process and Guidelines for Ensuring a Quality User Experience (2 ed.)","author":"Hartson R.","year":"2019","unstructured":"R. Hartson and P.S. Pyla. 2019. The UX Book: Process and Guidelines for Ensuring a Quality User Experience (2 ed.). Morgan Kaufmann."},{"key":"e_1_3_3_24_2","first-page":"155","volume-title":"SBM","author":"Hashimoto Y.","year":"2015","unstructured":"Y. Hashimoto and T. Igarashi. 2015. Retrieving web page layouts using sketches to support example-based web design. In SBM. 155\u2013164."},{"key":"e_1_3_3_25_2","unstructured":"Saad Hassan Manan Arya Ujjwal Bhardwaj and Silica Kole. 2018. Extraction and classification of user interface components from an image. Int. J. Pure Appl. Math. 118 24 (2018) 1\u201316."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_3_27_2","unstructured":"HeyLeia. 2025. Leia AI Website Builder. https:\/\/heyleia.com\/. Accessed: 2025-04-27 (URL from StackSocial deal)."},{"key":"e_1_3_3_28_2","first-page":"278","volume-title":"Proceedings of 3rd International Conference on Document Analysis and Recognition","author":"Ho T.K.","year":"1995","unstructured":"T.K. Ho. 1995. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognition , Vol.\u00a01. IEEE, 278\u2013282."},{"key":"e_1_3_3_29_2","unstructured":"Hostinger. 2025. Hostinger Website Builder (formerly Zyro). https:\/\/www.hostinger.com\/website-builder. Accessed: 2025-04-27 (Reflecting Zyro\u2019s integration)."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300334"},{"key":"e_1_3_3_31_2","volume-title":"The Elements of User Experience: User-Centered Design for the Web and beyond (2 ed.)","author":"James G.J.","year":"2010","unstructured":"G.J. James. 2010. The Elements of User Experience: User-Centered Design for the Web and beyond (2 ed.). Pearson Education."},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","unstructured":"James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei\u00a0A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska et\u00a0al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114 13 (2017) 3521\u20133526.","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_3_3_33_2","volume-title":"Automated front-end development using deep learning","author":"Kumar A.","year":"2018","unstructured":"A. Kumar. 2018. Automated front-end development using deep learning. https:\/\/blog.insightdatascience.com\/automated-front-end-development-usingdeep-learning-3169dd086e82"},{"key":"e_1_3_3_34_2","unstructured":"Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang and Kristina Toutanova. 2023. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. arxiv:https:\/\/arXiv.org\/abs\/2210.03347\u00a0[cs.CL]"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/AUTEEE.2018.8720784"},{"key":"e_1_3_3_36_2","unstructured":"Matchbox Design Group. 2023. The Mechanics Of AI Website Builders: How Do They Work? https:\/\/matchboxdesigngroup.com\/blog\/the-mechanics-of-ai-website-builders-how-do-they-work\/. Accessed: 2025-04-27."},{"key":"e_1_3_3_37_2","unstructured":"T. Memmel and H. Reiterer. 2009. Support Collaboration Model-based and prototyping-driven user interface specification to and creativity. Int. J. Univ. Comput. Sci. 14 19 (2009) 3217\u20133235."},{"key":"e_1_3_3_38_2","unstructured":"Microsoft AI labs. 2019. Sketch2Code. https:\/\/sketch2code.azurewebsites.net\/"},{"key":"e_1_3_3_39_2","unstructured":"K.P. Moran C. Bernal-C\u00e1rdenas M. Curcio R. Bonett and D. Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans. Softw. Eng. (2018)."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197231.3197249"},{"key":"e_1_3_3_41_2","unstructured":"Neo. 2025. Zyro AI | Set-up Pricing Templates and Review. https:\/\/www.neo.space\/blog\/zyro-ai. Accessed: 2025-04-27."},{"key":"e_1_3_3_42_2","unstructured":"Pagecloud Blog. 2024. What happened to The Grid? Does the AI site builder still exist? https:\/\/www.pagecloud.com\/blog\/what-happened-to-the-grid. Accessed: 2025-04-27."},{"key":"e_1_3_3_43_2","unstructured":"Product Hunt. 2025. Bookmark | AI Website Builder Product Information and Latest Updates (2025). https:\/\/www.producthunt.com\/products\/bookmark-ai-website-builder. Accessed: 2025-04-27."},{"key":"e_1_3_3_44_2","unstructured":"Product Hunt. 2025. Firedrop Product Information and Latest Updates (2025). https:\/\/www.producthunt.com\/posts\/firedrop-2. Accessed: 2025-04-27."},{"key":"e_1_3_3_45_2","unstructured":"Product Hunt. 2025. Leia: Website Builder Product Information and Latest Updates (2025). https:\/\/www.producthunt.com\/products\/leia-website-builder. Accessed: 2025-04-27."},{"key":"e_1_3_3_46_2","doi-asserted-by":"crossref","unstructured":"\u00d3.S. Ram\u00f3n J.S. Cuadrado J.G. Molina and J. Vanderdonckt. 2016. A layout inference algorithm for graphical user interfaces. Inf. Softw. Technol. 70 (2016) 155\u2013175.","DOI":"10.1016\/j.infsof.2015.10.005"},{"key":"e_1_3_3_47_2","first-page":"28","volume-title":"Advances in Neural Information Processing Systems","author":"Ren S.","year":"2015","unstructured":"S. Ren, K. He, R. Girshick, and J. Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems. 28."},{"key":"e_1_3_3_48_2","first-page":"13","volume-title":"International Conference on Web Engineering","author":"Rivero J.M.","year":"2010","unstructured":"J.M. Rivero, G. Rossi, J. Grigera, J. Burella, E.R. Luna, and S. Gordillo. 2010. From mock-ups to user interface models: An extensible model driven approach. In International Conference on Web Engineering (Berlin, Heidelberg). Springer, 13\u201324."},{"key":"e_1_3_3_49_2","first-page":"589","volume-title":"International Conference on Software Engineering & Knowledge Engineering","author":"Ruozi Huang","year":"2016","unstructured":"Huang Ruozi, Yonghao Long, and Xiangping Chen. 2016. Automatically generating web page from a mock-up. In International Conference on Software Engineering & Knowledge Engineering (USA). 589\u2013594."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397482.3450725"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491427"},{"key":"e_1_3_3_52_2","unstructured":"Smart Tools AI. 2024. Bookmark AiDA - AI-Powered Website Builder. https:\/\/www.smart-tools.ai\/product\/BookmarkAiDA. Accessed: 2025-04-27."},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","unstructured":"Andr\u00e9s Soto H\u00e9ctor Mora and Jaime\u00a0A. Riascos. 2022. Web Generator: An open-source software for synthetic web-based user interface dataset generation. SoftwareX 17 (2022) 100985. 10.1016\/j.softx.2022.100985","DOI":"10.1016\/j.softx.2022.100985"},{"key":"e_1_3_3_54_2","unstructured":"StackSocial. 2025. Leia AI Website Builder: Lifetime Subscription (Business LITE). https:\/\/www.stacksocial.com\/sales\/leia-business-lite-lifetime-subscription. Accessed: 2025-04-27."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3312994"},{"key":"e_1_3_3_56_2","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan\u00a0N Gomez, \u0141\u00a0ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems , I.\u00a0Guyon, U.\u00a0Von Luxburg, S.\u00a0Bengio, H.\u00a0Wallach, R.\u00a0Fergus, S.\u00a0Vishwanathan, and R.\u00a0Garnett (Eds.), Vol.\u00a030. Curran Associates, Inc.https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"e_1_3_3_57_2","unstructured":"WebsitePlanet. 2025. Recensione di Zyro 2025: pu\u00f2 competere con Wix? https:\/\/www.websiteplanet.com\/it\/website-builders\/zyro\/. Accessed: 2025-04-27 (Notes Zyro\u2019s integration into Hostinger)."},{"key":"e_1_3_3_58_2","unstructured":"Wix Blog. 2024. Wix ADI: How Design AI Elevates Website Creation for Everyone. https:\/\/www.wix.com\/blog\/wix-artificial-design-intelligence. Accessed: 2025-04-27."},{"key":"e_1_3_3_59_2","unstructured":"Wix.com. 2025. Creare sito web con intelligenza artificiale | AI per creare siti web. https:\/\/it.wix.com\/ai-website-builder. Accessed: 2025-04-27."},{"key":"e_1_3_3_60_2","unstructured":"WowSlider. 2024. Segnalibro Costruttore di Siti Web AI - Recensioni Tutorial Alternative. https:\/\/wowslider.com\/ai-builder\/it\/bookmark.html. Accessed: 2025-04-27."},{"key":"e_1_3_3_61_2","unstructured":"WowSlider. 2024. Zyro AI Costruttore di Siti Web - Recensioni Tutorial Alternative. https:\/\/wowslider.com\/ai-builder\/it\/zyro.html. Accessed: 2025-04-27."},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","unstructured":"Jason Wu Siyan Wang Siman Shen Yi-Hao Peng Jeffrey Nichols and Jeffrey\u00a0P Bigham. 2023. WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics(CHI \u201923). Association for Computing Machinery New York NY USA. 10.1145\/3544548.3581158","DOI":"10.1145\/3544548.3581158"},{"key":"e_1_3_3_63_2","doi-asserted-by":"crossref","unstructured":"Y. Xu L. Bo X. Sun B. Li J. Jiang and W. Zhou. 2021. image2emmet: Automatic code generation from web user interface image. J. Softw.: Evol. Process 33 8 (2021) e2369.","DOI":"10.1002\/smr.2369"},{"key":"e_1_3_3_64_2","first-page":"240","volume-title":"European Conference on Computer Vision","author":"You Keen","year":"2024","unstructured":"Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision. Springer, 240\u2013255."},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-0311-1_16"},{"key":"e_1_3_3_66_2","first-page":"181","volume-title":"Asian Conference on Computer Vision","author":"Zhu Z.","year":"2018","unstructured":"Z. Zhu, Z. Xue, and Z. Yuan. 2018. Automatic graphics program generation using attention-based hierarchical decoder. In Asian Conference on Computer Vision. Springer, Cham, 181\u2013196."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3734190","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,28]],"date-time":"2025-06-28T09:30:03Z","timestamp":1751103003000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3734190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,27]]},"references-count":65,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3734190"],"URL":"https:\/\/doi.org\/10.1145\/3734190","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,27]]},"assertion":[{"value":"2025-06-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}