{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T14:26:10Z","timestamp":1777299970489,"version":"3.51.4"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T00:00:00Z","timestamp":1725926400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T00:00:00Z","timestamp":1725926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002386","name":"Cairo University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002386","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Data Sci Anal"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Join query optimization aims to find the best join order for tables in a query, which is critical for query processing performance. Recently, reinforcement learning models have been proposed to solve the challenges existing with query processing and join query optimization. However, changes in the data distribution can turn the trained reinforcement learning models into obsolete models, resulting in longer execution times. In this paper, we propose a new training strategy in order to extend the existing reinforcement learning models and improve their adaptation when the data distribution changes. The experiments show that the proposed strategy has a significant benefit in decreasing training time for the models given the changes in the data distribution.<\/jats:p>","DOI":"10.1007\/s41060-024-00628-4","type":"journal-article","created":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T08:02:12Z","timestamp":1725955332000},"page":"2711-2720","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Towards online training for RL-based query optimizer"],"prefix":"10.1007","volume":"20","author":[{"given":"Mohamed","family":"Ramadan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hoda M. O.","family":"Mokhtar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ibrahim","family":"Sobh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ayman","family":"El-Kilany","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,9,10]]},"reference":[{"key":"628_CR1","volume-title":"Inside the SQL Server Query Optimizer","author":"B Nevarez","year":"2011","unstructured":"Nevarez, B.: Inside the SQL Server Query Optimizer. Red Gate Books, Newnham House, Cambridge Business Park, Cambridge, United Kingdom (2011)"},{"key":"628_CR2","first-page":"1312","volume":"1","author":"S Vellevt","year":"2009","unstructured":"Vellevt, S.: Review of algorithms for join ordering problem in database query optimization. Inf. Technol. Control. 1, 1312\u20132622 (2009)","journal-title":"Inf. Technol. Control."},{"key":"628_CR3","doi-asserted-by":"publisher","unstructured":"Chaudhuri, S.: An overview of query optimization in relational systems. In: Proceedings of the Seventeenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pp. 34\u201343. ACM, Seattle Washington USA (1998). https:\/\/doi.org\/10.1145\/275487.275492","DOI":"10.1145\/275487.275492"},{"key":"628_CR4","doi-asserted-by":"publisher","first-page":"70502","DOI":"10.1109\/ACCESS.2022.3187102","volume":"10","author":"M Ramadan","year":"2022","unstructured":"Ramadan, M., El-Kilany, A., Mokhtar, H.M.O., Sobh, I.: Rl_qoptimizer: a reinforcement learning based query optimizer. IEEE Access 10, 70502\u201370515 (2022). https:\/\/doi.org\/10.1109\/ACCESS.2022.3187102","journal-title":"IEEE Access"},{"key":"628_CR5","doi-asserted-by":"publisher","unstructured":"Marcus, R., Papaemmanouil, O.: Deep reinforcement learning for join order enumeration. In: Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management, pp. 1\u20134 (2018). https:\/\/doi.org\/10.1145\/3211954.3211957","DOI":"10.1145\/3211954.3211957"},{"key":"628_CR6","doi-asserted-by":"publisher","unstructured":"Ortiz, J., Balazinska, M., Gehrke, J., Keerthi, S.: Learning state representations for query optimization with deep reinforcement learning. In: In Workshop on Data Management for End-To-End Machine Learning, pp. 1\u20134 (2018). https:\/\/doi.org\/10.1145\/3209889.3209890","DOI":"10.1145\/3209889.3209890"},{"issue":"11","key":"628_CR7","doi-asserted-by":"publisher","first-page":"1705","DOI":"10.14778\/3342263.3342644","volume":"12","author":"R Marcus","year":"2019","unstructured":"Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., Papaemmanouil, O., Tatbul, N.: Neo: a learned query optimizer. Proc. VLDB Endowment. 12(11), 1705\u20131718 (2019)","journal-title":"Proc. VLDB Endowment."},{"key":"628_CR8","unstructured":"Ivanov, O., Bartunov, S.: Adaptive cardinality estimation. In: Preprint at ArXiv (2017)"},{"key":"628_CR9","unstructured":"Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., Stoica, I.: Learning to optimize join queries with deep reinforcement learning. Preprint at arXiv:1808.03196 (2018)"},{"key":"628_CR10","unstructured":"Tzoumas, K., Sellis, T., Jensen, C.: A reinforcement learning approach for adaptive query processing (2008)"},{"key":"628_CR11","doi-asserted-by":"publisher","unstructured":"Guo, R., Daudjee, K.: Research challenges in deep reinforcement learning-based join query optimization. In: aiDM \u201920: Proceedings of the Third International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (2020). https:\/\/doi.org\/10.1145\/3401071.3401657","DOI":"10.1145\/3401071.3401657"},{"key":"628_CR12","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.1998.712192","volume-title":"Introduction to Reinforcement Learning","author":"RS Sutton","year":"1998","unstructured":"Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)"},{"issue":"4","key":"628_CR13","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1512\/iumj.1957.6.56038","volume":"6","author":"R Bellman","year":"1957","unstructured":"Bellman, R.: A Markovian decision process. Indiana Univ. Math. J. 6(4), 679\u2013684 (1957)","journal-title":"Indiana Univ. Math. J."},{"key":"628_CR14","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. in deep learning. In: Neural Information Processing Systems Workshop (2013)"},{"key":"628_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-018-1296-x","author":"X Zhao","year":"2019","unstructured":"Zhao, X., Ding, S., Yuexuan, A., Jia, W.: Applications of asynchronous deep reinforcement learning based on dynamic updating weights. Appl. Intell. (2019). https:\/\/doi.org\/10.1007\/s10489-018-1296-x","journal-title":"Appl. Intell."},{"key":"628_CR16","unstructured":"Mnih, V., Badia, A., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning (2016)"},{"key":"628_CR17","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-018-1241-z","author":"X Zhao","year":"2018","unstructured":"Zhao, X., Ding, S., Yuexuan, A., Jia, W.: Asynchronous reinforcement learning algorithms for solving discrete space path planning problems. Appl. Intell. (2018). https:\/\/doi.org\/10.1007\/s10489-018-1241-z","journal-title":"Appl. Intell."},{"key":"628_CR18","doi-asserted-by":"publisher","unstructured":"Zhao, X., Ding, S., Yuexuan, A.: A new asynchronous architecture for tabular reinforcement learning algorithms, pp. 172\u2013180 (2019). https:\/\/doi.org\/10.1007\/978-3-030-01520-6_15","DOI":"10.1007\/978-3-030-01520-6_15"},{"key":"628_CR19","doi-asserted-by":"publisher","unstructured":"Ding, S., Zhao, X., Xu, X., Sun, T., Jia, W.: An effective asynchronous framework for small scale reinforcement learning problems. Appl. Intell. (2019). https:\/\/doi.org\/10.1007\/s10489-019-01501-9","DOI":"10.1007\/s10489-019-01501-9"},{"key":"628_CR20","unstructured":"Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)"},{"key":"628_CR21","doi-asserted-by":"publisher","unstructured":"Guo, S., Zou, L., Chen, H., Qu, B., Chi, H., Yu, P., Chang, Y.: Sample efficient offline-to-online reinforcement learning. In: IEEE Transactions on Knowledge and Data Engineering, pp. 1\u201312 (2023). https:\/\/doi.org\/10.1109\/TKDE.2023.3302804","DOI":"10.1109\/TKDE.2023.3302804"},{"key":"628_CR22","doi-asserted-by":"publisher","unstructured":"Nair, A., Gupta, A., Dalal, M., Levine, S.: Awac: accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 (2020). https:\/\/doi.org\/10.48550\/arXiv.2006.09359","DOI":"10.48550\/arXiv.2006.09359"},{"key":"628_CR23","doi-asserted-by":"publisher","first-page":"11372","DOI":"10.1609\/aaai.v37i9.26345","volume":"37","author":"H Zheng","year":"2023","unstructured":"Zheng, H., Luo, X., Wei, P., Song, X., Li, D., Jiang, J.: Adaptive policy learning for offline-to-online reinforcement learning. Proc. AAAI Conf. Artif. Intell. 37, 11372\u201311380 (2023). https:\/\/doi.org\/10.1609\/aaai.v37i9.26345","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"628_CR24","doi-asserted-by":"publisher","unstructured":"Marcus, R., Negi, P., Mao, H., Tatbul, N., Alizadeh, M., Kraska, T.: Bao: Making learned query optimization practical. In: Proceedings of the 2021 International Conference on Management of Data, pp. 1275\u20131288 (2021). https:\/\/doi.org\/10.1145\/3448016.3452838","DOI":"10.1145\/3448016.3452838"},{"key":"628_CR25","doi-asserted-by":"crossref","unstructured":"Trummer, e.a. Immanuel: Skinnerdb: Regret-bounded query evaluation via reinforcement learning. In: Proceedings of the 2019 International Conference on Management of Data (2019)","DOI":"10.1145\/3299869.3300088"},{"key":"628_CR26","unstructured":"Marcus, R., Papaemmanouil, O.: Towards ahands-free query optimizer through deep learning. In: The 9th Biennial Conference on Innovative Data Systems Research, CIDR (2019)"},{"key":"628_CR27","unstructured":"Heitz, J., Stockinger, K.: Join query optimization with deep reinforcement learning algorithms. arXiv Preprint at arXiv:1911.11689 (2019)"},{"key":"628_CR28","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/BF00992698","volume":"8","author":"CJ Watkins","year":"1992","unstructured":"Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8, 279\u2013292 (1992). https:\/\/doi.org\/10.1007\/BF00992698","journal-title":"Mach. Learn."},{"issue":"7540","key":"628_CR29","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529\u2013533 (2015). https:\/\/doi.org\/10.1038\/nature14236","journal-title":"Nature"},{"key":"628_CR30","doi-asserted-by":"publisher","first-page":"204","DOI":"10.14778\/2850583.2850594","volume":"9","author":"V Leis","year":"2015","unstructured":"Leis, V., Gubichev, A., Mirchev, A., Boncz, P., Kemper, A., Neumann, T.: How good are query optimizers, really? Proc. VLDB Endow. 9, 204\u2013215 (2015). https:\/\/doi.org\/10.14778\/2850583.2850594","journal-title":"Proc. VLDB Endow."},{"key":"628_CR31","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man\u00e9, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Vi\u00e9gas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org (2015). http:\/\/tensorflow.org\/"},{"key":"628_CR32","unstructured":"Chollet, F.: Keras. https:\/\/keras.io\/ (2015)"},{"key":"628_CR33","unstructured":"Kingma, D.P., Adam, J.B.: A method for stochastic optimization. In: In 3rd International Conference for Learning Representations, ICLR. Ithaca, NY 15, San Diego, CA (2015)"},{"key":"628_CR34","unstructured":"IMDB (1990). https:\/\/www.imdb.com\/"}],"container-title":["International Journal of Data Science and Analytics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41060-024-00628-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41060-024-00628-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41060-024-00628-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,5]],"date-time":"2025-09-05T22:01:22Z","timestamp":1757109682000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41060-024-00628-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,10]]},"references-count":34,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["628"],"URL":"https:\/\/doi.org\/10.1007\/s41060-024-00628-4","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-3689202\/v1","asserted-by":"object"}]},"ISSN":["2364-415X","2364-4168"],"issn-type":[{"value":"2364-415X","type":"print"},{"value":"2364-4168","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,10]]},"assertion":[{"value":"30 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"All authors consent to the publication of this manuscript.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}