{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T13:41:27Z","timestamp":1777038087559,"version":"3.51.4"},"reference-count":22,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T00:00:00Z","timestamp":1744416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2020YFA0711900, 2020YFA0711901"],"award-info":[{"award-number":["2020YFA0711900, 2020YFA0711901"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62350610270, 62374034, 62235009, 62474050, 62304052, 92373207 and 62090025"],"award-info":[{"award-number":["62350610270, 62374034, 62235009, 62474050, 62304052, 92373207 and 62090025"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"The Innovation Program of Shanghai Municipal Education Commission","award":["2021-01-07-00-07-E00077"],"award-info":[{"award-number":["2021-01-07-00-07-E00077"]}]},{"name":"Natural Science Foundation of Shanghai","award":["22ZR1403500, 24JD1401000"],"award-info":[{"award-number":["22ZR1403500, 24JD1401000"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Des. Autom. Electron. Syst."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>Design automation of complex analog circuits (CAC) with multiple sub-blocks is challenging mainly due to large design search space, uncertain intermediate subgoal creation, and lengthy CAC simulation runtime. In this work, we propose a hierarchical and heterogeneous integration framework as a fully automated and time-efficient CAC design optimization solution. In Particularly, we (i) decompose CAC into two levels hierarchically and for the first time introduce hierarchical RL agents with hindsight and subgoal testing to automate the subgoal creation between these two levels. The subgoal converges to the optimal value through algorithm interactions. (ii) We enable high-level design space dimensionality reduction, minimize CAC simulation runs through a buffer hold, and employ low-level sub-block execution parallelization to reduce overall runtime. (iii) We construct a heterogeneous integration of different RL algorithms and black-box optimization algorithms in hierarchy to further boost the speed by benefiting both from the hierarchical structure and the advantages of each different algorithm. Experiments on four CAC topologies demonstrate that this framework achieves a maximum of 11.4\u00d7 speed up compared to existing methods at the desired figure-of-merit. This work opens up a time efficient design automation route for complex analog circuits and systems.<\/jats:p>","DOI":"10.1145\/3723162","type":"journal-article","created":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T11:26:21Z","timestamp":1742210781000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Hierarchical Integration of Reinforcement Learning and Optimization Algorithms for Time-Efficient Design Automation of Complex Analog Circuit"],"prefix":"10.1145","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-0034-1400","authenticated-orcid":false,"given":"Xingwei","family":"Feng","sequence":"first","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8529-4840","authenticated-orcid":false,"given":"Yifan","family":"Xu","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2551-7044","authenticated-orcid":false,"given":"Zhangcheng","family":"Huang","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3828-5037","authenticated-orcid":false,"given":"Wuyi","family":"Xu","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7315-3150","authenticated-orcid":false,"given":"Zhaori","family":"Bi","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2164-8175","authenticated-orcid":false,"given":"Fan","family":"Yang","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8097-4053","authenticated-orcid":false,"given":"Xuan","family":"Zeng","sequence":"additional","affiliation":[{"name":"Microelectronics, Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9054-2644","authenticated-orcid":false,"given":"Ye","family":"Lu","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,4,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.vlsi.2009.09.001"},{"key":"e_1_3_1_3_2","first-page":"3306","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Lyu Wenlong","year":"2018","unstructured":"Wenlong Lyu, Fan Yang, Changhao Yan, Dian Zhou, and Xuan Zeng. 2018. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In Proceedings of the International Conference on Machine Learning. PMLR, 3306\u20133314."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.23919\/DATE.2019.8714788"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1146909.1146919"},{"key":"e_1_3_1_6_2","first-page":"1","volume-title":"Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE\u201914)","author":"Afacan Engin","year":"2014","unstructured":"Engin Afacan, Simge Ay, Francisco V Fernandez, G\u00fcnhan D\u00fcndar, Faik Bas, et\u00a0al. 2014. Model based hierarchical optimization strategies for analog design automation. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE\u201914). IEEE, 1\u20134."},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1146909.1146920"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECS.2016.7841150"},{"key":"e_1_3_1_9_2","unstructured":"Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv:1509.02971 [cs.LG]."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/DAC18072.2020.9218757"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.23919\/DATE48585.2020.9116200"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/DAC18074.2021.9586189"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/DAC56929.2023.10247909"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453160"},{"key":"e_1_3_1_15_2","first-page":"3540","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Vezhnevets Alexander Sasha","year":"2017","unstructured":"Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 3540\u20133549."},{"key":"e_1_3_1_16_2","unstructured":"Andrew Levy George Konidaris Robert Platt and Kate Saenko. 2017. Learning multi-level hierarchies with hindsight. arXiv:1712.00948 [cs.AI]."},{"key":"e_1_3_1_17_2","unstructured":"Abhishek Gupta Vikash Kumar Corey Lynch Sergey Levine and KarolHausman. 2019. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv:1910.11956 [cs.LG]."},{"key":"e_1_3_1_18_2","first-page":"1312","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Schaul Tom","year":"2015","unstructured":"Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. 2015. Universal value function approximators. In Proceedings of the International Conference on Machine Learning. PMLR, 1312\u20131320."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(89)90020-8"},{"key":"e_1_3_1_20_2","unstructured":"Haihong Guo Fengxin Li Jiao Li and Hongyan Liu. 2024. KAN vs MLP for offline reinforcement learning. arXiv:2409.09653 [cs.LG]."},{"key":"e_1_3_1_21_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347 [cs.LG]."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.13643"},{"issue":"2","key":"e_1_3_1_23_2","first-page":"182","article-title":"A 20-kHz\u223c 16-MHz programmable-bandwidth 4th order active filter using gain-boosted OPAMP with negative resistance in 65-nm CMOS","volume":"66","author":"Lim Jiye","year":"2018","unstructured":"Jiye Lim and Jintae Kim. 2018. A 20-kHz\u223c 16-MHz programmable-bandwidth 4th order active filter using gain-boosted OPAMP with negative resistance in 65-nm CMOS. IEEE Trans. Circ. Syst. II: Expr. Briefs 66, 2 (2018), 182\u2013186.","journal-title":"IEEE Trans. Circ. Syst. II: Expr. Briefs"}],"container-title":["ACM Transactions on Design Automation of Electronic Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3723162","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3723162","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:56:42Z","timestamp":1750298202000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3723162"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,12]]},"references-count":22,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3723162"],"URL":"https:\/\/doi.org\/10.1145\/3723162","relation":{},"ISSN":["1084-4309","1557-7309"],"issn-type":[{"value":"1084-4309","type":"print"},{"value":"1557-7309","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,12]]},"assertion":[{"value":"2024-09-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-24","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}