{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T09:02:09Z","timestamp":1783069329361,"version":"3.54.6"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T00:00:00Z","timestamp":1781222400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["Project-ID 504518248"],"award-info":[{"award-number":["Project-ID 504518248"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100018930","name":"TUM International Graduate School of Science and Engineering","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100018930","id-type":"DOI","asserted-by":"crossref"}]},{"name":"German Ministry for Research, Technology and Space","award":["ExaVerse 01IW25003 and ECXL 01IW22002"],"award-info":[{"award-number":["ExaVerse 01IW25003 and ECXL 01IW22002"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Des. Autom. Electron. Syst."],"published-print":{"date-parts":[[2026,11,30]]},"abstract":"<jats:p>Functional simulation is an essential step in digital hardware design. Recently, there has been a growing interest in leveraging Large Language Models (LLMs) for hardware testbench generation tasks. However, the inherent instability associated with LLMs often leads to functional errors in the generated testbenches. Previous methods do not incorporate automatic functional correction mechanisms without human intervention and still suffer from low success rates, especially for sequential tasks. To address this issue, we propose ConfiBench, an automatic testbench generation framework with functional self-validation, self-correction, scenario masking and an ensemble of multiple testbenches. Utilizing only the RTL specification in natural language, the proposed approach can validate the correctness of the generated testbenches, perform functional self-correction on the generated testbenches and construct an ensemble of them with effective masks. The comparative analysis demonstrates that our method achieves a pass rate of 72.22% across all evaluated tasks, compared with the previous LLM-based testbench generation framework\u2019s 52.18% and a direct LLM-based generation method\u2019s 33.33%. Specifically in sequential circuits, our work\u2019s performance achieves a testbench pass rate that is 21.06 percentage points higher than previous work AutoBench in sequential tasks and almost 5 times the pass rate of the direct method. More importantly, ConfiBench significantly improves the correctness of generated testbenches under golden RTLs, particularly for sequential circuits, which had been a major weakness in the previous work, CorrectBench. The codes and experimental results are open-sourced at the link: https:\/\/github.com\/AutoBench\/ConfiBench.<\/jats:p>","DOI":"10.1145\/3773087","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T11:25:19Z","timestamp":1761650719000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["ConfiBench: Automatic Testbench Generation with Confidence-Based Scenario Mask and Testbench Ensemble using LLMs for HDL Design"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-5828-4567","authenticated-orcid":false,"given":"Ruidi","family":"Qiu","sequence":"first","affiliation":[{"name":"Chair of Electronic Design Automation, Technical University of Munich","place":["Munich, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8289-9288","authenticated-orcid":false,"given":"Grace Li","family":"Zhang","sequence":"additional","affiliation":[{"name":"TU Darmstadt","place":["Darmstadt, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9872-1740","authenticated-orcid":false,"given":"Rolf","family":"Drechsler","sequence":"additional","affiliation":[{"name":"University of Bremen\/DFKI","place":["Bremen, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7348-5625","authenticated-orcid":false,"given":"Tsungyi","family":"Ho","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong","place":["Hong Kong, Hong Kong"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4431-7619","authenticated-orcid":false,"given":"Ulf","family":"Schlichtmann","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Technische Universitat Munchen","place":["Munchen, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9752-7201","authenticated-orcid":false,"given":"Bing","family":"Li","sequence":"additional","affiliation":[{"name":"University of Siegen","place":["Siegen, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,12]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2071356.2071363"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSD.2014.100"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSC55427.2022.9826162"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD.2007.4397275"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD58807.2023.10299874"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD62225.2024.10740262"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD62225.2024.10740247"},{"key":"e_1_3_2_9_2","unstructured":"Kaiyan Chang Ying Wang Haimeng Ren Mengdi Wang Shengwen Liang Yinhe Han Huawei Li and Xiaowei Li. 2023. ChipGPT: How far are we from natural language hardware design. arXiv:2305.14019. Retrieved from https:\/\/arxiv.org\/abs\/2305.14019"},{"key":"e_1_3_2_10_2","volume-title":"International Conference on Computer Aided Design (ICCAD)","author":"Liu Mingjie","year":"2023","unstructured":"Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023. VerilogEval: Evaluating large language models for verilog code generation. In International Conference on Computer Aided Design (ICCAD)."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/MDAT.2022.3216799"},{"key":"e_1_3_2_12_2","unstructured":"Wenhao Sun Bing Li Grace Li Zhang Xunzhao Yin Cheng Zhuo and Ulf Schlichtmann. 2024. Classification-based automatic HDL code generation using LLMs. Retrieved from https:\/\/arxiv.org\/abs\/2407.18326"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Zhuorui Zhao Ruidi Qiu Ing-Chao Lin Grace Li Zhang Bing Li and Ulf Schlichtmann. 2025. VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency. (2025). arXiv:2502.00028. Retrieved from https:\/\/arxiv.org\/abs\/2502.00028","DOI":"10.1109\/ISQED65160.2025.11014398"},{"key":"e_1_3_2_14_2","unstructured":"Kangwei Xu Bing Li Grace Li Zhang and Ulf Schlichtmann. 2025. HLSTester: Efficient Testing of Behavioral Discrepancies with LLMs for High-Level Synthesis. (2025). arXiv:2504.14641. Retrieved from https:\/\/arxiv.org\/abs\/2504.14641"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","unstructured":"Kangwei Xu Grace Li Zhang Xunzhao Yin Cheng Zhuo Ulf Schlichtmann and Bing Li. 2025. HLSRewriter: Efficient refactoring and optimization of C\/C++ code with LLMs for high-level synthesis. ACM Trans. Des. Autom. Electron. Syst. (July 2025). DOI:10.1145\/3749986","DOI":"10.1145\/3749986"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics14010120"},{"key":"e_1_3_2_17_2","unstructured":"YunDa Tsai Mingjie Liu and Haoxing Ren. 2023. RTLFixer: Automatically fixing RTL syntax errors with large language models. arXiv:2311.16543. Retrieved from https:\/\/arxiv.org\/abs\/2311.16543"},{"key":"e_1_3_2_18_2","unstructured":"Shailja Thakur Jason Blocklove Hammond Pearce Benjamin Tan Siddharth Garg and Ramesh Karri. 2023. AutoChip: Automating HDL generation using LLM feedback. arXiv:2311.04887. Retrieved from https:\/\/arxiv.org\/abs\/2311.04887"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD62225.2024.10740250"},{"key":"e_1_3_2_20_2","unstructured":"Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng Haotian Wang Qianglong Chen Weihua Peng Xiaocheng Feng Bing Qin and Ting Liu. 2023. A survey on hallucination in large language models: Principles taxonomy challenges and open questions. arXiv:2311.05232. Retrieved from https:\/\/arxiv.org\/abs\/2311.05232"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.284"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.23919\/DATE64628.2025.10992873"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICET.2016.7813256"},{"key":"e_1_3_2_24_2","unstructured":"Jitendra Bhandari Johann Knechtel Ramesh Narayanaswamy Siddharth Garg and Ramesh Karri. 2024. LLM-aided testbench generation and bug detection for finite-state machines. arXiv:2406.17132. Retrieved from https:\/\/arxiv.org\/abs\/2406.17132"},{"key":"e_1_3_2_25_2","unstructured":"Stephen Williams. 2024. The ICARUS verilog compilation system. (2024). Retrieved from https:\/\/github.com\/steveicarus\/iverilog"},{"key":"e_1_3_2_26_2","unstructured":"Henry Wong. 2019. Problem sets - HDLBits. (2019). Retrieved from https:\/\/hdlbits.01xz.net\/wiki\/Problemsets"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.5555\/1622737.1622748"}],"container-title":["ACM Transactions on Design Automation of Electronic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773087","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T11:48:24Z","timestamp":1781264904000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773087"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,12]]},"references-count":26,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,11,30]]}},"alternative-id":["10.1145\/3773087"],"URL":"https:\/\/doi.org\/10.1145\/3773087","relation":{},"ISSN":["1084-4309","1557-7309"],"issn-type":[{"value":"1084-4309","type":"print"},{"value":"1557-7309","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,12]]},"assertion":[{"value":"2025-03-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}