{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T12:00:33Z","timestamp":1781265633394,"version":"3.54.1"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022YFB2901100"],"award-info":[{"award-number":["2022YFB2901100"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62404021, U22B2038, 62192784"],"award-info":[{"award-number":["62404021, U22B2038, 62192784"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Elite Scientists Sponsorship Program","award":["2023QNRC001"],"award-info":[{"award-number":["2023QNRC001"]}]},{"name":"CAST, the Beijing Natural Science Foundation","award":["4244107, QY24216, QY24204, QY25329"],"award-info":[{"award-number":["4244107, QY24216, QY24204, QY25329"]}]},{"name":"State Key Lab of Processors, Institute of Computing Technology, CAS","award":["CLQ202504"],"award-info":[{"award-number":["CLQ202504"]}]},{"name":"Fundamental Research Funds for the Beijing University of Posts and Telecommunications","award":["2025AI4S20"],"award-info":[{"award-number":["2025AI4S20"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Des. Autom. Electron. Syst."],"published-print":{"date-parts":[[2026,11,30]]},"abstract":"<jats:p>As a key step in digital integrated circuit (IC) design, logic synthesis involves various logic optimization algorithms, where the quality of results (QoR) depends heavily on the optimization sequence used. Exploring the optimization space is challenging as the number of potential optimal permutations grows exponentially. Traditional methods rely on manual adjustments by experts, but are difficult to deal with complex and different circuits, leading to significant optimality gaps. Many automatic methods have been introduced, but they still face problems of low generalization and low efficiency.<\/jats:p>\n                  <jats:p>In this work, we propose ORL-LO, a two-stage learning framework that leverages an offline reinforcement learning (RL) method (i.e., advantage-weighted actor-critic, AWAC) to pretrain on expert trajectories and finetune on target test circuits, enabling efficient exploration of optimal synthesis flows. Firstly, ORL-LO uses AWAC to pretrain a fast and high-performance intelligent policy to fully leverage the offline knowledge of a large corpus of high-quality expert trajectories. Then, the pretrained policy is finetuned for target circuits using AWAC and policy distillation to obtain better results. Compared with the state-of-the-art (SOTA) methods, our framework can effectively improve the quality of logic optimization and significantly speed up the exploration time.<\/jats:p>","DOI":"10.1145\/3774323","type":"journal-article","created":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T11:31:04Z","timestamp":1762428664000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["ORL-LO: Offline Reinforcement Learning for Pretraining and Finetuning in Logic Optimization"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3570-4109","authenticated-orcid":false,"given":"Guande","family":"Dong","sequence":"first","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1527-621X","authenticated-orcid":false,"given":"Hongtao","family":"Cheng","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1581-3536","authenticated-orcid":false,"given":"Jianwang","family":"Zhai","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8340-9104","authenticated-orcid":false,"given":"Xiao","family":"Yang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3734-0266","authenticated-orcid":false,"given":"Chuan","family":"Shi","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0502-8523","authenticated-orcid":false,"given":"Kang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1007\/978-3-642-14295-6_5","volume-title":"Proceedings of the International Conference on Computer Aided Verification","author":"Brayton Robert","year":"2010","unstructured":"Robert Brayton and Alan Mishchenko. 2010. ABC: An academic industrial-strength verification tool. In Proceedings of the International Conference on Computer Aided Verification. 24\u201340."},{"key":"e_1_3_1_3_2","first-page":"1","volume-title":"Proceedings of the ACM\/IEEE Design Automation Conference","author":"Yu Cunxi","year":"2018","unstructured":"Cunxi Yu, Houping Xiao, and Giovanni De Micheli. 2018. Developing synthesis flows without human knowledge. In Proceedings of the ACM\/IEEE Design Automation Conference. 1\u20136."},{"issue":"8","key":"e_1_3_1_4_2","doi-asserted-by":"crossref","first-page":"2580","DOI":"10.1109\/TCAD.2022.3226668","article-title":"Bulls-eye: Active few-shot learning guided logic synthesis","volume":"42","author":"Chowdhury Animesh Basak","year":"2023","unstructured":"Animesh Basak Chowdhury, Benjamin Tan, Ryan Carey, Tushit Jain, Ramesh Karri, and Siddharth Garg. 2023. Bulls-eye: Active few-shot learning guided logic synthesis. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42, 8 (2023), 2580\u20132590.","journal-title":"IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems"},{"key":"e_1_3_1_5_2","first-page":"581","volume-title":"Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference","author":"Hosny Abdelrahman","year":"2020","unstructured":"Abdelrahman Hosny, Soheil Hashemi, Mohamed Shalan, and Sherief Reda. 2020. DRiLLS: Deep reinforcement learning for logic synthesis. In Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference. 581\u2013586."},{"key":"e_1_3_1_6_2","first-page":"145","volume-title":"Proceedings of the ACM\/IEEE Workshop on Machine Learning CAD","author":"Zhu Keren","year":"2020","unstructured":"Keren Zhu, Mingjie Liu, Hao Chen, Zheng Zhao, and David Z. Pan. 2020. Exploring logic optimizations with reinforcement learning and graph convolutional network. In Proceedings of the ACM\/IEEE Workshop on Machine Learning CAD. 145\u2013150."},{"key":"e_1_3_1_7_2","first-page":"159","volume-title":"Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference","author":"Zhou Guanglei","year":"2023","unstructured":"Guanglei Zhou and Jason H. Anderson. 2023. Area-driven FPGA logic synthesis using reinforcement learning. In Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference. 159\u2013165."},{"key":"e_1_3_1_8_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM Proceedings Design, Automation and Test in Europe","author":"Liu Fangzhou","year":"2024","unstructured":"Fangzhou Liu, Zehua Pei, Ziyang Yu, Haisheng Zheng, Zhuolun He, Tinghuan Chen, and Bei Yu. 2024. CBTune: Contextual bandit tuning for logic synthesis. In Proceedings of the IEEE\/ACM Proceedings Design, Automation and Test in Europe. 1\u20136."},{"key":"e_1_3_1_9_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design","author":"Pei Zehua","year":"2023","unstructured":"Zehua Pei, Fangzhou Liu, Zhuolun He, Guojin Chen, Haisheng Zheng, Keren Zhu, and Bei Yu. 2023. AlphaSyn: Logic synthesis optimization with efficient monte carlo tree search. In Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design. 1\u20139."},{"key":"e_1_3_1_10_2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1145\/3658617.3697786","volume-title":"Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference","author":"Dong Guande","year":"2025","unstructured":"Guande Dong, Jianwang Zhai, Hongtao Cheng, Xiao Yang, Chuan Shi, and Kang Zhao. 2025. PIRLLS: Pretraining with imitation and rl finetuning for logic synthesis. In Proceedings of the IEEE\/ACM Asia and South Pacific Design Automation Conference. 65\u201371."},{"key":"e_1_3_1_11_2","first-page":"2052","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Fujimoto Scott","year":"2018","unstructured":"Scott Fujimoto, David Meger, and Doina Precup. 2018. Off-policy deep reinforcement learning without exploration. In Proceedings of the International Conference on Machine Learning. 2052\u20132062."},{"key":"e_1_3_1_12_2","first-page":"11784","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems","author":"Kumar Aviral","year":"2019","unstructured":"Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. In Proceedings of the Annual Conference on Neural Information Processing Systems. 11784\u201311794."},{"key":"e_1_3_1_13_2","unstructured":"Yifan Wu G. Tucker and Ofir Nachum. 2019. Behavior regularized offline reinforcement learning. arXiv:1911.11361."},{"key":"e_1_3_1_14_2","first-page":"532","volume-title":"Proceedings of the ACM\/IEEE Design Automation Conference","author":"Mishchenko A.","year":"2006","unstructured":"A. Mishchenko, S. Chatterjee, and R. Brayton. 2006. DAG-aware AIG rewriting: A fresh look at combinational logic synthesis. In Proceedings of the ACM\/IEEE Design Automation Conference. 532\u2013535."},{"key":"e_1_3_1_15_2","volume-title":"Proceedings of the IEEE\/ACM International Workshop on Logic Synthesis","author":"Mishchenko Alan","year":"2006","unstructured":"Alan Mishchenko and Robert K. Brayton. 2006. Scalable logic synthesis using a simple circuit structure. In Proceedings of the IEEE\/ACM International Workshop on Logic Synthesis."},{"key":"e_1_3_1_16_2","article-title":"Offline RL Distributional Shift","year":"2026","unstructured":"APXML. 2026. Offline RL Distributional Shift. Retrieved October 24, 2025 from https:\/\/apxml.com\/zh\/courses\/advanced-reinforcement-learning\/chapter-7-offline-reinforcement-learning\/offline-rl-distributional-shift ([n. d.]).","journal-title":"Retrieved October 24, 2025 from https:\/\/apxml.com\/zh\/courses\/advanced-reinforcement-learning\/chapter-7-offline-reinforcement-learning\/offline-rl-distributional-shift"},{"key":"e_1_3_1_17_2","first-page":"677","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems","author":"Huang Rui","year":"2021","unstructured":"Rui Huang, Andrew Geng, and Yixuan Li. 2021. On the importance of gradients for detecting distributional shifts in the wild. In Proceedings of the Annual Conference on Neural Information Processing Systems. 677\u2013689."},{"key":"e_1_3_1_18_2","first-page":"5637","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Koh Pang Wei","year":"2021","unstructured":"Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Jordan Phillips, Tony Gao, et\u00a0al. 2021. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the International Conference on Machine Learning. 5637\u20135664."},{"key":"e_1_3_1_19_2","unstructured":"Ashvin Nair Abhishek Gupta Murtaza Dalal and Sergey Levine. 2021. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv:2006.09359."},{"key":"e_1_3_1_20_2","unstructured":"Andrei A. Rusu Sergio Gomez Colmenarejo \u00c7aglar G\u00fcl\u00e7ehre Guillaume Desjardins James Kirkpatrick Razvan Pascanu Volodymyr Mnih Koray Kavukcuoglu and Raia Hadsell. 2015. Policy distillation. (2015). arXiv:1511.06295."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD52597.2021.9531070"},{"key":"e_1_3_1_23_2","first-page":"20313","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems","author":"Yang Shuwen","year":"2022","unstructured":"Shuwen Yang, Zhihao Yang, Dong Li, Yingxue Zhang, Zhanguang Zhang, Guojie Song, and Jianye Hao. 2022. Versatile multi-stage graph neural network for circuit representation. In Proceedings of the Annual Conference on Neural Information Processing Systems. 20313\u201320324."},{"key":"e_1_3_1_24_2","first-page":"667","volume-title":"Proceedings of the ACM\/IEEE Design Automation Conference","author":"Li Min","year":"2022","unstructured":"Min Li, Sadaf Khan, Zhengyuan Shi, Naixing Wang, Huang Yu, and Qiang Xu. 2022. DeepGate: Learning neural representations of logic gates. In Proceedings of the ACM\/IEEE Design Automation Conference. 667\u2013672."},{"key":"e_1_3_1_25_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design","author":"Shi Zhengyuan","year":"2023","unstructured":"Zhengyuan Shi, Hongyang Pan, Sadaf Khan, Min Li, Yi Liu, Junhua Huang, Hui-Ling Zhen, Mingxuan Yuan, Zhufei Chu, and Qiang Xu. 2023. DeepGate2: Functionality-aware circuit representation learning. In Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design. 1\u20139."},{"key":"e_1_3_1_26_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design","author":"Liu Jiawei","year":"2025","unstructured":"Jiawei Liu, Jianwang Zhai, Mingyu Zhao, Zhe Lin, Bei Yu, and Chuan Shi. 2025. PolarGate: Breaking the functionality representation bottleneck of and-inverter graph neural network. In Proceedings of the IEEE\/ACM International Conference on Computer-Aided Design. 1\u20139."},{"key":"e_1_3_1_27_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM Proceedings Design, Automation and Test in Europe","author":"Liu Jiawei","year":"2025","unstructured":"Jiawei Liu, Zhiyan Liu, Xun He, Jianwang Zhai, Zhengyuan Shi, Qiang Xu, Bei Yu, and Chuan Shi. 2025. WideGate: Beyond directed acyclic graph learning in subcircuit boundary prediction. In Proceedings of the IEEE\/ACM Proceedings Design, Automation and Test in Europe. 1\u20137."},{"key":"e_1_3_1_28_2","first-page":"4171","volume-title":"Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics. 4171\u20134186."},{"key":"e_1_3_1_29_2","first-page":"11","volume-title":"Proceedings of the 2022 IEEE 33rd International Conference on Application-specific Systems, Architectures and Processors","author":"Wu Nan","year":"2022","unstructured":"Nan Wu, Jiwon Lee, Yuan Xie, and Cong Hao. 2022. Lostin: Logic optimization via spatio-temporal information with hybrid graph models. In Proceedings of the 2022 IEEE 33rd International Conference on Application-specific Systems, Architectures and Processors. IEEE, 11\u201318."},{"key":"e_1_3_1_30_2","first-page":"64","volume-title":"Proceedings of the ACM Great Lakes Symposium on VLSI","author":"Zhao Mingyu","year":"2025","unstructured":"Mingyu Zhao, Jiawei Liu, Jianwang Zhai, and Chuan Shi. 2025. MILS: Modality interaction driven learning for logic synthesis. In Proceedings of the ACM Great Lakes Symposium on VLSI. 64\u201370."},{"key":"e_1_3_1_31_2","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:1606.01540."},{"key":"e_1_3_1_32_2","first-page":"677","volume-title":"Proceedings of the IEEE International Symposium on Circuits and Systems","author":"Brglez F.","year":"1985","unstructured":"F. Brglez and H. Fujiwara. 1985. A neutral netlist of 10 combinational benchmark circuits and a target translator in fortran. In Proceedings of the IEEE International Symposium on Circuits and Systems. 677\u2013692."},{"key":"e_1_3_1_33_2","first-page":"1929\u20131934 vol.3","volume-title":"Proceedings of the IEEE International Symposium on Circuits and Systems","author":"Brglez F.","year":"1989","unstructured":"F. Brglez, D. Bryan, and K. Kozminski. 1989. Combinational profiles of sequential benchmark circuits. In Proceedings of the IEEE International Symposium on Circuits and Systems. 1929\u20131934 vol.3."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/54.867894"},{"key":"e_1_3_1_35_2","volume-title":"Logic Synthesis and Optimization Benchmarks","author":"Yang S.","year":"1988","unstructured":"S. Yang. 1988. Logic Synthesis and Optimization Benchmarks. Technical Report. 1989 MCNC International Workshop on Logic Synthesis."},{"key":"e_1_3_1_36_2","volume-title":"Logic Synthesis and Optimization Benchmarks User Guide: Version 3.0","author":"Yang S.","year":"1991","unstructured":"S. Yang. 1991. Logic Synthesis and Optimization Benchmarks User Guide: Version 3.0. Technical Report. Microelectronics Center of North Carolina (MCNC)."},{"key":"e_1_3_1_37_2","volume-title":"IWLS\u201993 Benchmark Set: Version 4.0","author":"McElvain K.","year":"1993","unstructured":"K. McElvain. 1993. IWLS\u201993 Benchmark Set: Version 4.0. Technical Report. a part of IWLS\u201993 benchmark set."},{"key":"e_1_3_1_38_2","volume-title":"IWLS 2005 Benchmarks","author":"Albrecht C.","year":"2005","unstructured":"C. Albrecht. 2005. IWLS 2005 Benchmarks. Technical Report. IWLS."},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","unstructured":"J. Cong and K. Minkovich. 2007. Optimality study of logic synthesis for LUT-based FPGAs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 26 2 (2007) 230\u2013239.","DOI":"10.1109\/TCAD.2006.887922"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.123"},{"key":"e_1_3_1_41_2","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics."},{"key":"e_1_3_1_42_2","unstructured":"Marcin Andrychowicz Anton Raichuk Piotr Sta\u0144czyk Manu Orsini Sertan Girgin Raphael Marinier L\u00e9onard Hussenot Matthieu Geist Olivier Pietquin Marcin Michalski et\u00a0al. 2020. What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv:2006.05990."},{"key":"e_1_3_1_43_2","unstructured":"Luca Amaru Pierre-Emmanuel Gaillardon and Giovanni De Micheli. 2015. The EPFL Combinational Benchmark Suite. In IEEE\/ACM International Workshop on Logic Synthesis no. CONF."}],"container-title":["ACM Transactions on Design Automation of Electronic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3774323","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T11:47:59Z","timestamp":1781264879000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3774323"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,12]]},"references-count":42,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,11,30]]}},"alternative-id":["10.1145\/3774323"],"URL":"https:\/\/doi.org\/10.1145\/3774323","relation":{},"ISSN":["1084-4309","1557-7309"],"issn-type":[{"value":"1084-4309","type":"print"},{"value":"1557-7309","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,12]]},"assertion":[{"value":"2025-03-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}