{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T18:08:25Z","timestamp":1757614105764,"version":"3.44.0"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:p>Query optimizers are crucial for the performance of database systems. Recently, many learned query optimizers (LQOs) have demonstrated significant performance improvements over traditional optimizers. However, most of them operate under a limited assumption: a static query environment. This limitation prevents them from effectively handling complex, dynamic query environments in real-world scenarios. Extensive retraining can lead to the well-known catastrophic forgetting problem which reduces the LQO generalizability over time. In this paper, we address this limitation and introduce LIMAO (Lifelong Modular Learned Query Optimizer), a framework for lifelong learning of plan cost prediction that can be seamlessly integrated into existing LQOs. LIMAO leverages a modular lifelong learning technique, an attention-based neural network composition architecture, and an efficient training paradigm designed to retain prior knowledge while continuously adapting to new environments. We implement LIMAO in two LQOs, showing that our approach is agnostic to underlying engines. Experimental results show that LIMAO significantly enhances the performance of LQOs, achieving up to a 40% improvement in query execution time and reducing the variance of execution time by up to 60% under dynamic workloads. By leveraging a precise and self-consistent design, LIMAO effectively mitigates catastrophic forgetting, ensuring stable and reliable plan quality over time. Compared to Postgres, LIMAO achieves up to a 4\u00d7 speedup on selected benchmarks, highlighting its practical advantages in real-world query optimization.<\/jats:p>","DOI":"10.14778\/3749646.3749712","type":"journal-article","created":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T17:55:06Z","timestamp":1757008506000},"page":"4546-4559","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["LIMAO: A Framework for Lifelong Modular Learned Query Optimization"],"prefix":"10.14778","volume":"18","author":[{"given":"Qihan","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Southern California, Los Angeles, USA"}]},{"given":"Shaolin","family":"Xie","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, USA"}]},{"given":"Ibrahim","family":"Sabek","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,9,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3076264"},{"key":"e_1_2_1_2_1","volume-title":"Neural Module Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 39\u201348","author":"Andreas Jacob","year":"2016","unstructured":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural Module Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 39\u201348."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.12"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/3298483.3298491"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.115303"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of ICML Workshop on Unsupervised and Transfer Learning.","author":"Bengio Yoshua","year":"2012","unstructured":"Yoshua Bengio. 2012. Deep Learning of Representations for Unsupervised and Transfer Learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2021.3126456"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/3587136.3587150"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/3598581.3598597"},{"key":"e_1_2_1_10_1","volume-title":"Robust Listwise Learning-to-Rank Approach for Database Query Optimizer. In 2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS). 1165\u20131172","author":"Chen Xue","year":"2023","unstructured":"Xue Chen, Rui Xi, Yiheng Tang, and Mengshu Hou. 2023. Robust Listwise Learning-to-Rank Approach for Database Query Optimizer. In 2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS). 1165\u20131172."},{"key":"e_1_2_1_11_1","unstructured":"Transaction Processing Performance Council(TPC). 2021. TPC-H Vesion 2 and Version 3. http:\/\/www.tpc.org\/tpch\/"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10148-x"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Coline Devin Abhishek Gupta Trevor Darrell Pieter Abbeel and Sergey Levine. 2017. Learning modular neural network policies for multi-task and multi-robot transfer. In ICRA. 2169\u20132176.","DOI":"10.1109\/ICRA.2017.7989250"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389711"},{"key":"e_1_2_1_15_1","volume-title":"Kepler: Robust Learning for Parametric Query Optimization. In SIGMOD.","author":"Doshi Lyric","year":"2023","unstructured":"Lyric Doshi, Vincent Zhuang, Gaurav Jain, Ryan Marcus, Haoyu Huang, Deniz Altinb\u00fcken, Eugene Brevdo, and Campbell Fraser. 2023. Kepler: Robust Learning for Parametric Query Optimization. In SIGMOD."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.05.014"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407820"},{"key":"e_1_2_1_18_1","volume-title":"Recurrent Independent Mechanisms. In International Conference on Learning Representations.","author":"Goyal Anirudh","year":"2021","unstructured":"Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Sch\u00f6lkopf. 2021. Recurrent Independent Mechanisms. In International Conference on Learning Representations."},{"key":"e_1_2_1_19_1","first-page":"19","article-title":"The Cascades Framework for Query Optimization","volume":"18","author":"Graefe Goetz","year":"1995","unstructured":"Goetz Graefe. 1995. The Cascades Framework for Query Optimization. IEEE Data Eng. Bull. 18, 3 (1995), 19\u201329.","journal-title":"IEEE Data Eng. Bull."},{"volume-title":"Proceedings of IEEE 9th International Conference on Data Engineering. 209\u2013218","author":"Graefe Goetz","key":"e_1_2_1_20_1","unstructured":"Goetz Graefe and William J. McKenna. 1993. The Volcano optimizer generator: extensibility and efficient search. In Proceedings of IEEE 9th International Conference on Data Engineering. 209\u2013218."},{"key":"e_1_2_1_21_1","volume-title":"Attention mechanisms in computer vision: A survey. Computational visual media 8, 3","author":"Guo Meng-Hao","year":"2022","unstructured":"Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. 2022. Attention mechanisms in computer vision: A survey. Computational visual media 8, 3 (2022), 331\u2013368."},{"key":"e_1_2_1_22_1","volume-title":"12th Conference on Innovative Data Systems Research, CIDR 2022","author":"Hilprecht Benjamin","year":"2022","unstructured":"Benjamin Hilprecht and Carsten Binnig. 2022. One Model to Rule them All: Towards Zero-Shot Learning for Databases. In 12th Conference on Innovative Data Systems Research, CIDR 2022, Chaminade, CA, USA, January 9\u201312, 2022."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/3551793.3551799"},{"key":"e_1_2_1_24_1","volume-title":"International Conference on Sensor Systems and Software. Springer, 191\u2013201","author":"Hu Songqiao","year":"2022","unstructured":"Songqiao Hu, Zeyi Liu, and Xiao He. 2022. CADM: Confusion model-based detection method for real-drift in chunk data stream. In International Conference on Sensor Systems and Software. Springer, 191\u2013201."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings Of 1st Pacific-Asia Conference on Knowledge Discovery And Data Mining Conference, 21\u201334","author":"Huang Zhexue","year":"1997","unstructured":"Zhexue Huang. 1997. Clustering Large Data Sets With Mixed Numeric And Categorical Values,\". Proceedings Of 1st Pacific-Asia Conference on Knowledge Discovery And Data Mining Conference, 21\u201334."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Ronald Kemker Marc McClure Angelina Abitino Tyler L. Hayes and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (New Orleans Louisiana USA) (AAAI'18\/IAAI'18\/EAAI'18). AAAI Press Article 415 9 pages.","DOI":"10.1609\/aaai.v32i1.11651"},{"key":"e_1_2_1_27_1","volume-title":"Seo, Wook-Shin Han, Kangwoo Choi, and Jaehyok Chong.","author":"Kim Kyoungmin","year":"2022","unstructured":"Kyoungmin Kim, Jisung Jung, In Seo, Wook-Shin Han, Kangwoo Choi, and Jaehyok Chong. 2022. Learned Cardinality Estimation: An In-depth Study. In SIGMOD. Association for Computing Machinery, 1214\u20131227."},{"key":"e_1_2_1_28_1","volume-title":"Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In CIDR.","author":"Kipf Andreas","year":"2019","unstructured":"Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In CIDR."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196909"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/3659437.3659449"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526179"},{"key":"e_1_2_1_34_1","first-page":"2","article-title":"ALECE: An Attention-based Learned Cardinality Estimator for SPJ Queries on Dynamic Workloads","volume":"17","author":"Li Pengfei","year":"2023","unstructured":"Pengfei Li, Wenqing Wei, Rong Zhu, Bolin Ding, Jingren Zhou, and Hua Lu. 2023. ALECE: An Attention-based Learned Cardinality Estimator for SPJ Queries on Dynamic Workloads. VLDB 17, 2 (Oct. 2023), 197\u2013210.","journal-title":"VLDB"},{"key":"e_1_2_1_35_1","first-page":"77","article-title":"Matchmaker: Data drift mitigation in machine learning for large-scale systems","volume":"4","author":"Mallick Ankur","year":"2022","unstructured":"Ankur Mallick, Kevin Hsieh, Behnaz Arzani, and Gauri Joshi. 2022. Matchmaker: Data drift mitigation in machine learning for large-scale systems. Proceedings of Machine Learning and Systems 4 (2022), 77\u201394.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_36_1","volume-title":"Bao: Making learned query optimization practical. In SIGMOD. 1275\u20131288.","author":"Marcus Ryan","year":"2021","unstructured":"Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. 2021. Bao: Making learned query optimization practical. In SIGMOD. 1275\u20131288."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342644"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342646"},{"key":"e_1_2_1_39_1","volume-title":"Mendez and Eric Eaton","author":"Jorge","year":"2023","unstructured":"Jorge A. Mendez and Eric Eaton. 2023. How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition. Trans. Mach. Learn. Res. (2023)."},{"key":"e_1_2_1_40_1","unstructured":"Jorge A. Mendez Harm van Seijen and Eric Eaton. 2022. Modular Lifelong Reinforcement Learning via Neural Composition. In ICLR."},{"key":"e_1_2_1_41_1","unstructured":"Jorge Mendez Mendez. 2023. Embodied lifelong learning for decision making: Opportunities brought on by modularity."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research), Jie Tan, Marc Toussaint, and Kourosh Darvish (Eds.)","volume":"229","author":"Mendez-Mendez Jorge","year":"2023","unstructured":"Jorge Mendez-Mendez, Leslie Pack Kaelbling, and Tom\u00e1s Lozano-P\u00e9rez. 2023. Embodied Lifelong Learning for Task and Motion Planning. In Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research), Jie Tan, Marc Toussaint, and Kourosh Darvish (Eds.), Vol. 229. PMLR, 2134\u20132150."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321211.1321245"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3190376"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626734"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10139"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476259"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/3583140.3583164"},{"key":"e_1_2_1_49_1","unstructured":"PostgreSQL Global Development Group. [n.d.]. https:\/\/www.postgresql.org\/."},{"key":"e_1_2_1_50_1","volume-title":"Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. login Usenix Mag. 39, 6","author":"Ricci Robert","year":"2014","unstructured":"Robert Ricci, Eric Eide, and CloudLab Team. 2014. Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. login Usenix Mag. 39, 6 (2014), 36\u201338."},{"key":"e_1_2_1_51_1","volume-title":"Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning. ICLR abs\/1711.01239","author":"Rosenbaum Clemens","year":"2017","unstructured":"Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2017. Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning. ICLR abs\/1711.01239 (2017)."},{"key":"e_1_2_1_52_1","volume-title":"Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning. In International Conference on Learning Representations.","author":"Rosenbaum Clemens","year":"2018","unstructured":"Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2018. Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning. In International Conference on Learning Representations."},{"key":"e_1_2_1_53_1","unstructured":"Paul Ruvolo and Eric Eaton. 2013. ELLA: an efficient lifelong learning algorithm. In ICML. JMLR.org I-507\u2013I-515."},{"key":"e_1_2_1_54_1","volume-title":"Tenzin Samten Ukyab, and Tim Kraska","author":"Sabek Ibrahim","year":"2022","unstructured":"Ibrahim Sabek, Tenzin Samten Ukyab, and Tim Kraska. 2022. LSched: A Workload-Aware Learned Query Scheduler for Analytical Database Systems. In SIGMOD. ACM, 1228\u20131242."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/582095.582099"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368296"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.14778\/3485450.3485459"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517843"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402707.3402724"},{"volume-title":"Attention is all you need","author":"Vaswani Ashish","key":"e_1_2_1_60_1","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. Curran Associates Inc., 6000\u20136010."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.14778\/3641204.3641205"},{"key":"e_1_2_1_62_1","volume-title":"SIGMOD (Virtual Event, China)","author":"Wu Peizhi","year":"2009","unstructured":"Peizhi Wu and Gao Cong. 2021. A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation. In SIGMOD (Virtual Event, China). Association for Computing Machinery, 2009\u20132022."},{"key":"e_1_2_1_63_1","volume-title":"Proc. ACM Manag. Data 2, 1, Article 38 (March","author":"Wu Peizhi","year":"2024","unstructured":"Peizhi Wu and Zachary G. Ives. 2024. Modeling Shifting Workloads for Learned Database Systems. Proc. ACM Manag. Data 2, 1, Article 38 (March 2024), 27 pages."},{"volume-title":"Multi-task reinforcement learning with soft modularization","author":"Yang Ruihan","key":"e_1_2_1_64_1","unstructured":"Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. 2020. Multi-task reinforcement learning with soft modularization. In NIPS. Curran Associates Inc., Article 400, 11 pages."},{"key":"e_1_2_1_65_1","volume-title":"Balsa: Learning a Query Optimizer Without Expert Demonstrations. In SIGMOD, Zachary G. Ives, Angela Bonifati, and Amr El Abbadi (Eds.). ACM, 931\u2013944.","author":"Yang Zongheng","year":"2022","unstructured":"Zongheng Yang, Wei-Lin Chiang, Sifei Luan, Gautam Mittal, Michael Luo, and Ion Stoica. 2022. Balsa: Learning a Query Optimizer Without Expert Demonstrations. In SIGMOD, Zachary G. Ives, Angela Bonifati, and Amr El Abbadi (Eds.). ACM, 931\u2013944."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517885"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.14778\/3421424.3421432"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368294"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.14778\/3565838.3565846"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.14778\/3583140.3583160"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.14778\/3461535.3461539"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3749646.3749712","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,5]],"date-time":"2025-09-05T03:34:56Z","timestamp":1757043296000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3749646.3749712"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":71,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.14778\/3749646.3749712"],"URL":"https:\/\/doi.org\/10.14778\/3749646.3749712","relation":{},"ISSN":["2150-8097"],"issn-type":[{"type":"print","value":"2150-8097"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"2025-09-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}