{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T15:24:15Z","timestamp":1774452255416,"version":"3.50.1"},"reference-count":56,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T00:00:00Z","timestamp":1774310400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012190","name":"Ministry of Science and Higher Education of the Russian Federation","doi-asserted-by":"publisher","award":["FSWE-2024-0001"],"award-info":[{"award-number":["FSWE-2024-0001"]}],"id":[{"id":"10.13039\/501100012190","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006769","name":"Russian Science Foundation","doi-asserted-by":"publisher","award":["25-61-00027"],"award-info":[{"award-number":["25-61-00027"]}],"id":[{"id":"10.13039\/501100006769","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computation"],"abstract":"<jats:p>In this article, we explore the possibility of using reinforcement learning to create convective flow approximation schemes that maintain accuracy and stability at high Courant-Friedrichs-Lewy (CFL) numbers in the finite-volume discretization of advection equations. Unlike most existing data-driven discretization methods, which primarily concentrate on spatial grid refinement, this work emphasizes increasing the allowable time step without compromising solution accuracy. This approach reduces the total number of time integration steps, thereby enabling faster computation. A neural network is used as a surrogate model for reconstructing the convective flow, which takes as input local information about the flow, scalars, and geometry and predicts scalar values at node points. Reinforcement learning is used for training and is formulated as a policy optimization problem, where the long-term reward is defined as the difference between the numerical and reference solutions over the entire simulation period. Both the genetic algorithm and the Deep Deterministic Policy Gradient (DDPG) method are investigated. The effectiveness of the approach is evaluated using a one-dimensional nonlinear advection problem with a constant velocity field. Despite the simplicity of the test case, the results demonstrate that the trained convective flux approximation scheme achieves accuracy comparable to or better than the classical second-order linear upwind (LUD) scheme, while operating at CFL numbers 2\u201350 times higher than the optimal CFL for LUD, thereby reducing the simulation time by the same factor. This allows for a wider range of stability and accuracy in the finite-volume method and the use of larger time steps without compromising the quality of the solution. The study is intentionally limited to a single spatial dimension and serves as a basic analysis of the method\u2019s applicability. The results demonstrate that reinforcement learning can successfully find more convective flow approximation schemes that improve efficiency at high CFL numbers than conventional explicit second-order schemes, establishing a framework that is subsequently extended in our follow-up work to improve training methods and three-dimensional complex transport problems. The proposed method improves the spatial discretization of convective fluxes, which is independent of the choice of time integration scheme. Therefore, the neural reconstruction can in principle be used in both explicit and implicit finite-volume solvers.<\/jats:p>","DOI":"10.3390\/computation14040075","type":"journal-article","created":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T13:57:34Z","timestamp":1774447054000},"page":"75","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Reinforcement-Learning-Based Optimization of Convective Fluxes for High-CFL Finite-Volume Schemes"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7232-3058","authenticated-orcid":false,"given":"Andrey","family":"Rozhkov","sequence":"first","affiliation":[{"name":"Nizhny Novgorod State Technical University n.a. R.E. Alekseev, Nizhny Novgorod 603155, Russia"},{"name":"Russian Federal Nuclear Center\u2014All-Russian Scientific Research Institute of Experimental Physics (RFNC-VNIIEF), Sarov 607188, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3247-0835","authenticated-orcid":false,"given":"Andrey","family":"Kozelkov","sequence":"additional","affiliation":[{"name":"Nizhny Novgorod State Technical University n.a. R.E. Alekseev, Nizhny Novgorod 603155, Russia"},{"name":"Russian Federal Nuclear Center\u2014All-Russian Scientific Research Institute of Experimental Physics (RFNC-VNIIEF), Sarov 607188, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vadim","family":"Kurulin","sequence":"additional","affiliation":[{"name":"Nizhny Novgorod State Technical University n.a. R.E. Alekseev, Nizhny Novgorod 603155, Russia"},{"name":"Russian Federal Nuclear Center\u2014All-Russian Scientific Research Institute of Experimental Physics (RFNC-VNIIEF), Sarov 607188, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7408-724X","authenticated-orcid":false,"given":"Maxim","family":"Shishlenin","sequence":"additional","affiliation":[{"name":"Institute of Computational Mathematics and Mathematical Geophysics, Novosibirsk 630090, Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.neunet.2022.03.037","article-title":"Deep learning, reinforcement learning, and world models","volume":"152","author":"Matsuo","year":"2022","journal-title":"Neural Netw."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"075117","DOI":"10.1063\/5.0095270","article-title":"Physics-informed neural networks for solving Reynolds-averaged Navier\u2013stokes equations","volume":"34","author":"Eivazi","year":"2022","journal-title":"Phys. Fluids"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"055130","DOI":"10.1063\/5.0149551","article-title":"Super-resolution reconstruction of turbulent flows with a transformer-based deep learning framework","volume":"35","author":"Xu","year":"2023","journal-title":"Phys. Fluids"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2529","DOI":"10.1038\/s41598-023-29525-9","article-title":"A deep-learning approach for reconstructing 3D turbulent flows from 2D observation data","volume":"13","author":"Yousif","year":"2023","journal-title":"Sci. Rep."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Shishlenin, M., Kozelkov, A., and Novikov, N. (2024). Nonlinear Medical Ultrasound Tomography: 3D Modeling of Sound Wave Propagation in Human Tissues. Mathematics, 12.","DOI":"10.3390\/math12020212"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"116996","DOI":"10.1016\/j.cam.2025.116996","article-title":"Applying A Synthetic Turbulence Generator to An Unmatched RANS-LES Interface","volume":"475","author":"Korotkov","year":"2025","journal-title":"J. Comput. Appl. Math."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"110374","DOI":"10.1016\/j.jcp.2021.110374","article-title":"CPU-time and RAM memory optimization for solving dynamic inverse problems using gradient-based approach","volume":"439","author":"Klyuchinskiy","year":"2021","journal-title":"J. Comput. Phys."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Guo, X., Li, W., and Iorio, F. (2016). Convolutional Neural Networks for Steady Flow Approximation. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201816), Association for Computing Machinery.","DOI":"10.1145\/2939672.2939738"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"105644","DOI":"10.1016\/j.icheatmasstransfer.2021.105644","article-title":"Application of artificial neural network for mixed convection in a square lid-driven cavity with double vertical or horizontal oriented rectangular blocks","volume":"129","author":"Filali","year":"2021","journal-title":"Int. Commun. Heat Mass Transf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"107665","DOI":"10.1016\/j.neunet.2025.107665","article-title":"Physics-informed neural networks for solving inverse problems in phase field models","volume":"190","author":"Zhao","year":"2025","journal-title":"Neural Netw."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"111427","DOI":"10.1016\/j.engappai.2025.111427","article-title":"Can foundation language models predict fluid dynamics?","volume":"158","author":"Wang","year":"2025","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1093\/imanum\/drac085","article-title":"Error estimates for physics-informed neural networks approximating the Navier\u2013Stokes equations","volume":"44","author":"Jagtap","year":"2024","journal-title":"IMA J. Numer. Anal."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"115895","DOI":"10.1016\/j.rser.2025.115895","article-title":"Simulation of multiphase flow with thermochemical reactions: A review of computational fluid dynamics (CFD) theory to AI integration","volume":"221","author":"Zhang","year":"2025","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Heiland, J., Benner, P., and Bahmani, R. (2022). Convolutional Neural Networks for Very Low-Dimensional LPV Approximations of Incompressible Navier-Stokes Equations. Front. Appl. Math. Stat., 8.","DOI":"10.3389\/fams.2022.879140"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1007\/s00521-014-1762-2","article-title":"Artificial neural network method for solving the Navier\u2013Stokes equations","volume":"26","author":"Baymani","year":"2015","journal-title":"Neural Comput. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2845","DOI":"10.1007\/s00521-019-04157-1","article-title":"Integrated intelligent computing for heat transfer and thermal radiation-based two-phase MHD nanofluid flow model","volume":"32","author":"Raja","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/s40323-023-00242-2","article-title":"POD-Galerkin reduced order models and physics-informed neural networks for solving inverse problems for the Navier\u2013Stokes equations","volume":"10","author":"Hijazi","year":"2023","journal-title":"Adv. Model. Simul. Eng. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"105729","DOI":"10.1016\/j.cnsns.2021.105729","article-title":"An exploratory study on machine learning to couple numerical solutions of partial differential equations","volume":"97","author":"Tang","year":"2021","journal-title":"Commun. Nonlinear Sci. Numer. Simul."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/j.matcom.2025.04.031","article-title":"Application of machine learning and convex limiting to subgrid flux modeling in the shallow-water equations","volume":"238","author":"Timofeyev","year":"2025","journal-title":"Math. Comput. Simul."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1038\/s41467-022-28957-7","article-title":"Scientific multi-agent reinforcement learning for wall-models of turbulent flows","volume":"13","author":"Bae","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"111972","DOI":"10.1016\/j.jcp.2023.111972","article-title":"A physics-informed diffusion model for high-fidelity flow field reconstruction","volume":"478","author":"Shu","year":"2023","journal-title":"J. Comput. Phys."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1134\/S096554251507009X","article-title":"Eddy resolving numerical scheme for simulation of turbulent incompressible flows","volume":"55","author":"Kozelkov","year":"2015","journal-title":"Comput. Math. Math. Phys."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Weinman, K.A., and Valentino, M. (2010). Comparison of Hybrid RANS-LES Calculations within the Framework of Compressible and Incompressible Unstructured Solvers. Progress in Hybrid RANS-LES Modelling, Springer.","DOI":"10.1007\/978-3-642-14168-3_28"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1007\/s10915-015-0075-7","article-title":"Comparison of convective flux discretization schemes in detached-eddy simulation of turbulent flows on unstructured meshes","volume":"67","author":"Kozelkov","year":"2016","journal-title":"J. Sci. Comput."},{"key":"ref_25","first-page":"106","article-title":"Evaluation of numerical diffusion of the finite volume method in surface wave modeling","volume":"24","author":"Tyatyushkina","year":"2019","journal-title":"J. Comput. Technol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kozelkov, A., Kurkin, A., Kurulin, V., Plygunova, K., and Krutyakova, O. (2023). Validation of the LOGOS Software Package Methods for the Numerical Simulation of Cavitational Flows. Fluids, 8.","DOI":"10.3390\/fluids8030104"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tyatyushkina, E.S., Kozelkov, A.S., Kurkin, A.A., Pelinovsky, E.N., Kurulin, V.V., Plygunova, K.S., and Utkin, D.A. (2020). Verification of the LOGOS Software Package for Tsunami Simulations. Geosciences, 10.","DOI":"10.3390\/geosciences10100385"},{"key":"ref_28","first-page":"943","article-title":"DNS of Turbulent Channel Flow","volume":"11","author":"Mozer","year":"1999","journal-title":"Phys. Fluids"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1002\/fld.1650080602","article-title":"Curvature-compensated convective-transport\u2014SMART, A new boundedness-preserving transport algorithm","volume":"8","author":"Gaskell","year":"1988","journal-title":"Int. J. Numer. Methods Fluids"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1002\/(SICI)1097-0363(19990930)31:2<431::AID-FLD884>3.0.CO;2-T","article-title":"High resolution NVD differencing scheme for arbitrarily unstructured meshes","volume":"31","author":"Jasak","year":"1999","journal-title":"Int. J. Numer. Methods Fluids"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/s43588-022-00264-7","article-title":"Enhancing computational fluid dynamics with machine learning","volume":"2","author":"Ricardo","year":"2022","journal-title":"Nat. Comput. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1146\/annurev-fluid-010518-040547","article-title":"Turbulence modeling in the age of data","volume":"51","author":"Duraisamy","year":"2019","journal-title":"Annu. Rev. Fluid Mech."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"A27","DOI":"10.1017\/jfm.2021.812","article-title":"Convolutional-network models to predict wall-bounded turbulence from wall quantities","volume":"928","author":"Guastoni","year":"2021","journal-title":"J. Fluid Mech."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1017\/jfm.2016.615","article-title":"Reynolds averaged turbulence modelling using deep neural networks with embedded invariance","volume":"807","author":"Ling","year":"2016","journal-title":"J. Fluid Mech."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1699","DOI":"10.1007\/s00366-024-02057-0","article-title":"Reducing spatial discretization error on coarse CFD simulations using an openFOAM-embedded deep learning framework","volume":"41","author":"Pardo","year":"2025","journal-title":"Eng. Comput."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"064605","DOI":"10.1103\/PhysRevFluids.6.064605","article-title":"Learned discretizations for passive scalar advection in a two-dimensional turbulent flow","volume":"6","author":"Zhuang","year":"2021","journal-title":"Phys. Rev. Fluids"},{"key":"ref_37","unstructured":"Illarramendi, E.A., Alguacil, A., Bauerheim, M., Misdariis, A., Cuenot, B., and Benazera, E. (2020, January 15\u201319). Towards a hybrid computational strategy based on deep learning for incompressible flows. Proceedings of the AIAA Aviation Forum, Virtual."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"124900","DOI":"10.1016\/j.ijheatmasstransfer.2023.124900","article-title":"Residual-based physics-informed transfer learning: A hybrid method for accelerating long-term cfd simulations via deep learning","volume":"220","author":"Jeon","year":"2024","journal-title":"Int. J. Heat Mass Transf."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Obiols-Sales, O., Vishnu, A., Malaya, N., and Chandramowliswharan, A. (2020). CFDNet: A deep learning-based accelerator for fluid simulations. Proceedings of the 34th ACM International Conference on Supercomputing, Association for Computing Machinery.","DOI":"10.1145\/3392717.3392772"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"105132","DOI":"10.1063\/5.0106940","article-title":"Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence","volume":"34","author":"Kim","year":"2022","journal-title":"Phys. Fluids"},{"key":"ref_41","first-page":"279","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"15344","DOI":"10.1073\/pnas.1814058116","article-title":"Learning data-driven discretizations for partial differential equations","volume":"116","author":"Hoyer","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"e2101784118","DOI":"10.1073\/pnas.2101784118","article-title":"Machine learning\u2013accelerated computational fluid dynamics","volume":"118","author":"Kochkov","year":"2021","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_44","unstructured":"Somasekharan, N., and Pan, S. (2023, January 19\u201321). Warp-DG: A Differentiable Discontinuous Galerkin Solver for Compressible Flows. Proceedings of the 76th Annual Meeting of the APS Division of Fluid Dynamics, Washington, DC, USA. Available online: https:\/\/meetings.aps.org\/Meeting\/DFD23\/Session\/J15.4."},{"key":"ref_45","unstructured":"Jasak, H. (1996). Error Analysis and Estimation for the Finite Volume Method with Applications to Fluid Flows. [Doctoral Thesis, Imperial College of Science]."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/0021-9991(81)90145-5","article-title":"Volume of fluid (VOF) method for the dynamics of free boundaries","volume":"39","author":"Hirt","year":"1981","journal-title":"J. Comput. Phys."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/0045-7825(79)90034-3","article-title":"A stable and accurate convective modeling procedure based on quadratic upstream interpolation","volume":"19","author":"Leonard","year":"1979","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"ref_48","unstructured":"Yasnitskiy, L.N. (2008). Introduction to Artificial Intelligence: A Textbook for Students of Higher Educational Institutions, Publishing Center Academy. [2nd ed.]."},{"key":"ref_49","unstructured":"Aurelien, G. (2017). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O\u2019Reilly Media, Inc."},{"key":"ref_50","unstructured":"Parascandol, G., Huttunen, H., and Virtanen, T. (2017, January 24\u201326). Taming the waves: Sine as activation function in deep neural networks. Proceedings of the ICLR 2017, Toulon, France."},{"key":"ref_51","unstructured":"Sitzmann, V., Julien Martel, N.P., Bergman, A.W., Lindell, D.B., and Wetzstein, G. (2020). Implicit neural representations with periodic activation functions. Neural Information Processing Systems 33 (NeurIPS 2020), NeurIPS."},{"key":"ref_52","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). Deterministic Policy Gradient Algorithms. ICML-14 Proceedings of the 31st International Conference on International Conference on Machine Learning, JMLR."},{"key":"ref_53","unstructured":"Yemelyanov, V.V., Kureichik, V.V., and Kureichik, V.M. (2003). Theory and Practice of Evolutionary Modeling, Fizmatlit."},{"key":"ref_54","unstructured":"Rosenblatt, F. (1965). Principles of Neurodynamics: Perceptrons and Theory of Brain Mechanisms, Mir."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_56","first-page":"1889","article-title":"Trust region policy optimization","volume":"Volume 37","author":"Schulman","year":"2015","journal-title":"Proceedings of the 32nd International Conference on Machine Learning"}],"container-title":["Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-3197\/14\/4\/75\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T14:23:55Z","timestamp":1774448635000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-3197\/14\/4\/75"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,24]]},"references-count":56,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["computation14040075"],"URL":"https:\/\/doi.org\/10.3390\/computation14040075","relation":{},"ISSN":["2079-3197"],"issn-type":[{"value":"2079-3197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,24]]}}}