{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T00:13:55Z","timestamp":1761178435904,"version":"build-2065373602"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Des. Autom. Electron. Syst."],"published-print":{"date-parts":[[2025,11,30]]},"abstract":"<jats:p>Designing and verifying hardware circuits using a Hardware Description Language (HDL) is an essential but time-consuming part of hardware design. Generating the desired correct circuit and testbench code usually requires a significant engineering effort. Recently, Large Language Models (LLMs) have claimed to have strong code generation capabilities to reduce such engineering costs. Existing work has provided quantitative evaluations using LLMs for single-module, simple circuit generation. However, it is still unclear whether modern LLMs are useful in production workflows, e.g., generating correct hierarchical circuits with testbenches. And if they are capable, what are the best prompt engineering practices for hardware design? In this article, we evaluate LLMs for HDL generation by exploring a 3-dimensional design space: commercial and open-source language models, single-module and hierarchical circuits, and prompting methods with varying complexity. We propose a 3-step design space exploration methodology to answer the two aforementioned questions. First, we explore the best prompt engineering practices across generating simple, middle, and hard single-module circuits with testbenches on CodeLLama-34B. We also define two fine-grained checklists to evaluate the circuit and testbench quality from a user\u2019s perspective. Second, we benchmark 11 LLMs with prompt adaptation on 4 single-module circuits that CodeLLama-34B has trouble with to further find models that may be useful in a production workflow. Third, we apply the learned prompt practices on four top-level models to generate simple 2 to 4-module and more complex multi-module hierarchical circuits and testbenches. As a result, we find that some of the latest LLMs can generate correct simple hierarchical circuits and testbenches with given proper prompts, but still struggle with complex hierarchical circuits. We further provide useful guidelines from an end-user\u2019s perspective on leveraging LLMs for hardware design.<\/jats:p>","DOI":"10.1145\/3742430","type":"journal-article","created":{"date-parts":[[2025,6,4]],"date-time":"2025-06-04T07:30:11Z","timestamp":1749022211000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring Large Language Models for Hierarchical Hardware Circuit and Testbench Generation"],"prefix":"10.1145","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-8075-4327","authenticated-orcid":false,"given":"Samuel","family":"Gomes Lopes","sequence":"first","affiliation":[{"name":"D-INFK, ETH Zurich","place":["Z\u00fcrich, Switzerland"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2094-7643","authenticated-orcid":false,"given":"Shien","family":"Zhu","sequence":"additional","affiliation":[{"name":"D-INFK, ETH Zurich","place":["Z\u00fcrich, Switzerland"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4396-6695","authenticated-orcid":false,"given":"Gustavo","family":"Alonso","sequence":"additional","affiliation":[{"name":"D-INFK, ETH Zurich","place":["Z\u00fcrich, Switzerland"]}]}],"member":"320","published-online":{"date-parts":[[2025,10,22]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Anthropic. 2024. Claude 3.5 Sonnet. Retrieved from https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet. (2024)."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.420"},{"key":"e_1_3_2_4_2","unstructured":"Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang Xiaodong Deng Yang Fan Wenbin Ge Yu Han Fei Huang et al. 2023. Qwen technical report. arXiv:2309.16609. Retrieved from https:\/\/arxiv.org\/abs\/2309.16609"},{"key":"e_1_3_2_5_2","first-page":"1","volume-title":"Proceedings of the 2023 ACM\/IEEE 5th Workshop on Machine Learning for CAD","author":"Blocklove Jason","year":"2023","unstructured":"Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023. Chip-chat: Challenges and opportunities in conversational hardware design. In Proceedings of the 2023 ACM\/IEEE 5th Workshop on Machine Learning for CAD. IEEE, 1\u20136."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","unstructured":"Kaiyan Chang Kun Wang Nan Yang Ying Wang Dantong Jin Wenlong Zhu Zhirong Chen Cangyuan Li Hao Yan Yunhao Zhou et\u00a0al. 2024. Data is all you need: Finetuning llms for chip design via an automated design-data augmentation framework. In Proceedings of the 61st ACM\/IEEE Design Automation Conference. 1\u20136.","DOI":"10.1145\/3649329.3657356"},{"key":"e_1_3_2_7_2","unstructured":"Guangyao Chen Siwei Dong Yu Shu Ge Zhang Jaward Sesay B\u00f6rje Karlsson Jie Fu and Yemin Shi. 2024. AutoAgents: A framework for automatic agent generation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 22\u201330."},{"key":"e_1_3_2_8_2","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et\u00a0al. 2025. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)."},{"key":"e_1_3_2_9_2","unstructured":"Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan et\u00a0al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)."},{"key":"e_1_3_2_10_2","unstructured":"Qihao Zhu Daya Guo Zhihong Shao Dejian Yang Peiyi Wang Runxin Xu Y. Wu Yukun Li Huazuo Gao Shirong Ma et\u00a0al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931 (2024)."},{"key":"e_1_3_2_11_2","unstructured":"Matthew DeLorenzo Animesh Basak Chowdhury Vasudev Gohil Shailja Thakur Ramesh Karri Siddharth Garg and Jeyavijayan Rajendran. 2024. Make every move count: Llm-based high-quality rtl code generation using mcts. arXiv preprint arXiv:2402.03289 (2024)."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Mingzhe Gao Jieru Zhao Zhe Lin Wenchao Ding Xiaofeng Hou Yu Feng Chao Li and Minyi Guo. 2024. AutoVCoder: A systematic framework for automated verilog code generation using LLMs. In 2024 IEEE 42nd International Conference on Computer Design (ICCD). IEEE 162\u2013169.","DOI":"10.1109\/ICCD63220.2024.00033"},{"key":"e_1_3_2_13_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu Y. K. Li et\u00a0al. 2024. DeepSeek-Coder: When the large language model meets programming\u2013The rise of code intelligence. arXiv preprint arXiv:2401.14196 (2024)."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3451179"},{"key":"e_1_3_2_15_2","unstructured":"Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li and Huaming Chen. 2024. From LLMs to LLM-based Agents for Software Engineering: A Survey of Current Challenges and Future. arXiv preprint arXiv:2408.02479 (2024)."},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","unstructured":"Rahul Kande Hammond Pearce Benjamin Tan Brendan Dolan-Gavitt Shailja Thakur Ramesh Karri and Jeyavijayan Rajendran. 2024. (Security) assertions by large language models. IEEE Transactions on Information Forensics and Security (2024).","DOI":"10.1109\/TIFS.2024.3372809"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD57390.2023.10323812"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Shang Liu Wenji Fang Yao Lu Qijun Zhang Hongce Zhang and Zhiyao Xie. 2024. Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. In 2024 IEEE LLM Aided Design Workshop (LAD). IEEE 1\u20135.","DOI":"10.1109\/LAD62341.2024.10691788"},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the 2024 IEEE\/ACM International Conference on Computer-Aided Design","author":"Liu Shang","year":"2024","unstructured":"Shang Liu, Yao Lu, Wenji Fang, Mengming Li, and Zhiyao Xie. 2024. OpenLLM-RTL: Open dataset and benchmark for LLM-Aided design RTL generation (invited). In Proceedings of the 2024 IEEE\/ACM International Conference on Computer-Aided Design. ACM."},{"key":"e_1_3_2_20_2","first-page":"710","volume-title":"Proceedings of the 2024 2nd International Symposium of Electronics Design Automation","author":"Liu Tianyang","year":"2024","unstructured":"Tianyang Liu, Qi Tian, Jianmin Ye, LikTung Fu, Shengchu Su, Junyan Li, Gwok-Waa Wan, Layton Zhang, Sam-Zaak Wong, Xi Wang, et\u00a0al. 2024. ChatChisel: Enabling agile hardware design with large language models. In Proceedings of the 2024 2nd International Symposium of Electronics Design Automation. IEEE, 710\u2013716."},{"key":"e_1_3_2_21_2","first-page":"722","volume-title":"Proceedings of the 2024 29th Asia and South Pacific Design Automation Conference.","author":"Lu Yao","year":"2024","unstructured":"Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. 2024. RTLLM: An open-source benchmark for design RTL generation with large language model. In Proceedings of the 2024 29th Asia and South Pacific Design Automation Conference.IEEE, 722\u2013727."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","unstructured":"Javier Moya Mario Ruiz and Gustavo Alonso. 2024. fpgasystems\/hdev: HACC Development. Zenodo. 10.5281\/zenodo.14202998","DOI":"10.5281\/zenodo.14202998"},{"key":"e_1_3_2_23_2","unstructured":"Aaron Hurst Adam Lerer Adam P. Goucher Adam Perelman Aditya Ramesh Aidan Clark A. J. Ostrow Akila Welihinda Alan Hayes Alec Radford et\u00a0al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)."},{"key":"e_1_3_2_24_2","unstructured":"OpenAI. 2024. Hello GPT-4o. Retrieved from https:\/\/openai.com\/index\/hello-gpt-4o\/. (2024)."},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Pei Zehua","year":"2024","unstructured":"Zehua Pei, Huiling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. 2024. BetterV: Controlled verilog generation with discriminative guidance. In Proceedings of the 41st International Conference on Machine Learning. Retrieved from https:\/\/openreview.net\/forum?id=jKnW7r7de1"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Nathaniel Pinckney Christopher Batten Mingjie Liu Haoxing Ren and Brucek Khailany. 2025. Revisiting verilogeval: A year of improvements in large-language models for hardware code generation. ACM Transactions on Design Automation of Electronic Systems (2025).","DOI":"10.1145\/3718088"},{"key":"e_1_3_2_27_2","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez et\u00a0al. 2023. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_3_2_28_2","unstructured":"Wenhao Sun Bing Li Grace Li Zhang Xunzhao Yin Cheng Zhuo and Ulf Schlichtmann. 2024. Classification-based automatic HDL code generation using LLMs. arXiv preprint arXiv:2407.18326 (2024)."},{"key":"e_1_3_2_29_2","unstructured":"Gemini Team Petko Georgiev Ving Ian Lei Ryan Burnell Libin Bai Anmol Gulati Garrett Tanzer Damien Vincent Zhufeng Pan Shibo Wang et\u00a0al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)."},{"key":"e_1_3_2_30_2","unstructured":"Qwen Team. 2024. Code with CodeQwen1.5. Retrieved from https:\/\/qwenlm.github.io\/blog\/codeqwen1.5\/. (2024)."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643681"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Shailja Thakur Baleegh Ahmad Hammond Pearce Benjamin Tan Brendan Dolan-Gavitt Ramesh Karri and Siddharth Garg. 2024. VeriGen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems 29 3 (2024) 1\u201331.","DOI":"10.1145\/3643681"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649329.3658493"},{"key":"e_1_3_2_34_2","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Fei Xia Ed Chi Quoc V. Le Denny Zhou et\u00a0al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022) 24824\u201324837."},{"key":"e_1_3_2_35_2","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382 (2023)."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661308"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"Yang Zhao Di Huang Chongxiao Li Pengwei Jin Ziyuan Nan Tianyun Ma Lei Qi Yansong Pan Zhenxing Zhang Rui Zhang et\u00a0al. 2024. Codev: Empowering llms for verilog generation through multi-level summarization. arXiv preprint arXiv:2407.10424 (2024).","DOI":"10.1109\/TCAD.2025.3604320"}],"container-title":["ACM Transactions on Design Automation of Electronic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3742430","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T12:21:04Z","timestamp":1761135664000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3742430"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,22]]},"references-count":36,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,11,30]]}},"alternative-id":["10.1145\/3742430"],"URL":"https:\/\/doi.org\/10.1145\/3742430","relation":{},"ISSN":["1084-4309","1557-7309"],"issn-type":[{"type":"print","value":"1084-4309"},{"type":"electronic","value":"1557-7309"}],"subject":[],"published":{"date-parts":[[2025,10,22]]},"assertion":[{"value":"2024-10-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}