{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T13:37:54Z","timestamp":1774964274112,"version":"3.50.1"},"reference-count":54,"publisher":"IOP Publishing","issue":"2","license":[{"start":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T00:00:00Z","timestamp":1618358400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T00:00:00Z","timestamp":1618358400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100006224","name":"Argonne National Laboratory","doi-asserted-by":"crossref","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}],"id":[{"id":"10.13039\/100006224","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Several applications in the scientific simulation of physical systems can be formulated as control\/optimization problems. The computational models for such systems generally contain hyperparameters, which control solution fidelity and computational expense. The tuning of these parameters is non-trivial and the general approach is to manually \u2018spot-check\u2019 for good combinations. This is because optimal hyperparameter configuration search becomes intractable when the parameter space is large and when they may vary dynamically. To address this issue, we present a framework based on deep reinforcement learning (RL) to train a deep neural network agent that controls a model solve by varying parameters dynamically. First, we validate our RL framework for the problem of controlling chaos in chaotic systems by dynamically changing the parameters of the system. Subsequently, we illustrate the capabilities of our framework for accelerating the convergence of a steady-state computational fluid dynamics solver by automatically adjusting the relaxation factors of the discretized Navier\u2013Stokes equations during run-time. The results indicate that the run-time control of the relaxation factors by the learned policy leads to a significant reduction in the number of iterations for convergence compared to the random selection of the relaxation factors. Our results point to potential benefits for learning adaptive hyperparameter learning strategies across different geometries and boundary conditions with implications for reduced computational campaign expenses<jats:xref ref-type=\"fn\" rid=\"mlstabdaf8fn2\">\n                     <jats:sup>4<\/jats:sup>\n                  <\/jats:xref>\n                  <jats:fn id=\"mlstabdaf8fn2\">\n                     <jats:label>4<\/jats:label>\n                     <jats:p>Data and codes available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/Romit-Maulik\/PAR-RL\" xlink:type=\"simple\">https:\/\/github.com\/Romit-Maulik\/PAR-RL<\/jats:ext-link>.<\/jats:p>\n                  <\/jats:fn>.<\/jats:p>","DOI":"10.1088\/2632-2153\/abdaf8","type":"journal-article","created":{"date-parts":[[2021,1,14]],"date-time":"2021-01-14T07:51:28Z","timestamp":1610610688000},"page":"025029","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Distributed deep reinforcement learning for simulation control"],"prefix":"10.1088","volume":"2","author":[{"given":"Suraj","family":"Pawar","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9731-8936","authenticated-orcid":false,"given":"Romit","family":"Maulik","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2021,4,14]]},"reference":[{"key":"mlstabdaf8bib1","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1146\/annurev-fluid-010719-060214","article-title":"Machine learning for fluid mechanics","volume":"52","author":"Brunton","year":"2019","journal-title":"Annu. Rev. Fluid Mech."},{"key":"mlstabdaf8bib2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevFluids.4.100501","article-title":"Perspective on machine learning for advancing fluid mechanics","volume":"4","author":"Brenner","year":"2019","journal-title":"Phys. Rev. Fluids"},{"key":"mlstabdaf8bib3","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1146\/annurev-fluid-010518-040547","article-title":"Turbulence modeling in the age of data","volume":"51","author":"Duraisamy","year":"2019","journal-title":"Annu. Rev. Fluid Mech."},{"key":"mlstabdaf8bib4","doi-asserted-by":"publisher","first-page":"A13","DOI":"10.1017\/jfm.2019.822","article-title":"Nonlinear mode decomposition with convolutional neural networks for fluid dynamics","volume":"882","author":"Murata","year":"2020","journal-title":"J. Fluid Mech."},{"key":"mlstabdaf8bib5","doi-asserted-by":"crossref","DOI":"10.1109\/SC41405.2020.00013","article-title":"MeshfreeFlowNet: a physics-constrained deep continuous space-time super-resolution framework","author":"Jiang","year":"2020"},{"key":"mlstabdaf8bib6","doi-asserted-by":"publisher","first-page":"A18","DOI":"10.1017\/jfm.2019.814","article-title":"Prediction of turbulent heat transfer using convolutional neural networks","volume":"882","author":"Kim","year":"2020","journal-title":"J. Fluid Mech."},{"key":"mlstabdaf8bib7","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","article-title":"Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","volume":"378","author":"Raissi","year":"2019","journal-title":"J. Comput. Phys."},{"key":"mlstabdaf8bib8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2019.108925","article-title":"PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network","volume":"399","author":"Long","year":"2019","journal-title":"J. Comput. Phys."},{"key":"mlstabdaf8bib9","author":"Sutton","year":"2018"},{"key":"mlstabdaf8bib10","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1017\/jfm.2019.62","article-title":"Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control","volume":"865","author":"Rabault","year":"2019","journal-title":"J. Fluid Mech."},{"key":"mlstabdaf8bib11","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"mlstabdaf8bib12","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1109\/ADCONIP.2017.7983780","article-title":"Deep reinforcement learning approaches for process control","author":"Spielberg","year":"2017"},{"key":"mlstabdaf8bib13","doi-asserted-by":"publisher","first-page":"5849","DOI":"10.1073\/pnas.1800923115","article-title":"Efficient collective swimming by harnessing vortices through deep reinforcement learning","volume":"115","author":"Verma","year":"2018"},{"key":"mlstabdaf8bib14","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1038\/s41586-018-0533-0","article-title":"Glider soaring via reinforcement learning in the field","volume":"562","author":"Reddy","year":"2018","journal-title":"Nature"},{"key":"mlstabdaf8bib15","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevFluids.4.093902","article-title":"Controlled gliding and perching through deep-reinforcement-learning","volume":"4","author":"Novati","year":"2019","journal-title":"Phys. Rev. Fluids"},{"key":"mlstabdaf8bib16","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.118.158004","article-title":"Flow navigation by smart microswimmers via reinforcement learning","volume":"118","author":"Colabrese","year":"2017","journal-title":"Phys. Rev. Lett."},{"key":"mlstabdaf8bib17","doi-asserted-by":"publisher","first-page":"497","DOI":"10.1007\/s00162-016-0392-y","article-title":"A statistical learning strategy for closed-loop control of fluid flows","volume":"30","author":"Gu\u00e9niat","year":"2016","journal-title":"Theor. Comput. Fluid Dyn."},{"key":"mlstabdaf8bib18","doi-asserted-by":"publisher","first-page":"26091","DOI":"10.1073\/pnas.2004939117","article-title":"Reinforcement learning for bluff body active flow control in experiments and simulations","volume":"117","author":"Fan","year":"2020","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstabdaf8bib19","doi-asserted-by":"publisher","first-page":"3120","DOI":"10.23919\/ACC.2017.7963427","article-title":"Deep reinforcement learning for partial differential equation control","author":"Farahmand","year":"2017"},{"key":"mlstabdaf8bib20","article-title":"Computational model discovery with reinforcement learning","author":"Bassenne","year":"2019"},{"key":"mlstabdaf8bib21","article-title":"Optimization and passive flow control using single-step deep reinforcement learning","author":"Ghraieb","year":"2020"},{"key":"mlstabdaf8bib22","article-title":"A view on deep reinforcement learning in system optimization","author":"Haj-Ali","year":"2019"},{"key":"mlstabdaf8bib23","article-title":"Automating turbulence modeling by multi-agent reinforcement learning","author":"Novati","year":"2020"},{"key":"mlstabdaf8bib24","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1080\/10407790490487677","article-title":"A fuzzy logic algorithm for acceleration of convergence in solving turbulent flow and heat transfer problems","volume":"46","author":"Dragojlovic","year":"2004","journal-title":"Numer. Heat Transfer B"},{"key":"mlstabdaf8bib25","doi-asserted-by":"publisher","first-page":"3811","DOI":"10.1016\/S0017-9310(01)00029-1","article-title":"Tuning of a fuzzy rule set for controlling convergence of a CFD solver in turbulent flow","volume":"44","author":"Dragojlovic","year":"2001","journal-title":"Int. J. Heat Mass Transfer"},{"key":"mlstabdaf8bib26","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/TFUZZ.2004.839656","article-title":"Control of convergence in a computational fluid dynamics simulation using ANFIS","volume":"13","author":"Ryoo","year":"2005","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"mlstabdaf8bib27","article-title":"Reinforcement learning for learning rate control","author":"Xu","year":"2017"},{"key":"mlstabdaf8bib28","doi-asserted-by":"publisher","DOI":"10.1063\/1.5116415","article-title":"Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach","volume":"31","author":"Rabault","year":"2019","journal-title":"Phys. Fluids"},{"key":"mlstabdaf8bib29","first-page":"3053","article-title":"RLlib: abstractions for distributed reinforcement learning","author":"Liang","year":"2018"},{"key":"mlstabdaf8bib30","author":"Schaarschmidt","year":"2017"},{"key":"mlstabdaf8bib31","article-title":"SEED RL: scalable and efficient deep-RL with accelerated central inference","author":"Espeholt","year":"2019"},{"key":"mlstabdaf8bib32","article-title":"Acme: a research framework for distributed reinforcement learning","author":"Hoffman","year":"2020"},{"key":"mlstabdaf8bib33","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017"},{"key":"mlstabdaf8bib34","first-page":"pp 1057","article-title":"Policy gradient methods for reinforcement learning with function approximation","author":"Sutton","year":"2000"},{"key":"mlstabdaf8bib35","first-page":"pp 1008","author":"Konda","year":"2000"},{"key":"mlstabdaf8bib36","first-page":"1889","article-title":"Trust region policy optimization","author":"Schulman","year":"2015"},{"key":"mlstabdaf8bib37","article-title":"High-dimensional continuous control using generalized advantage estimation","author":"Schulman","year":"2015"},{"key":"mlstabdaf8bib38","doi-asserted-by":"publisher","DOI":"10.1063\/5.0002047","article-title":"Restoring chaos using deep reinforcement learning","volume":"30","author":"Vashishtha","year":"2020","journal-title":"Chaos"},{"key":"mlstabdaf8bib39","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/0167-2789(83)90126-4","article-title":"Crises, sudden changes in chaotic attractors and transient chaos","volume":"7","author":"Grebogi","year":"1983","journal-title":"Physica D:"},{"key":"mlstabdaf8bib40","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1146\/annurev.fl.22.010190.001231","article-title":"Mixing, chaotic advection and turbulence","volume":"22","author":"Ottino","year":"1990","journal-title":"Annu. Rev. Fluid Mech."},{"key":"mlstabdaf8bib41","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1016\/0167-6911(89)90072-8","article-title":"Towards a theory of voltage collapse in electric power systems","volume":"13","author":"Dobson","year":"1989","journal-title":"Syst. Control Lett."},{"key":"mlstabdaf8bib42","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1103\/PhysRevE.51.102","article-title":"Preserving chaos: control strategies to preserve complex dynamics with potential relevance to biological disorders","volume":"51","author":"Yang","year":"1995","journal-title":"Phys. Rev. E"},{"key":"mlstabdaf8bib43","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1175\/1520-0469(1963)0200130:DNF2.0.CO;2","article-title":"Deterministic nonperiodic flow","volume":"20","author":"Lorenz","year":"1963","journal-title":"J. Atmos. Sci."},{"key":"mlstabdaf8bib44","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1007\/BF01221359","article-title":"Preturbulence: a regime observed in a fluid flow model of lorenz","volume":"67","author":"Kaplan","year":"1979","journal-title":"Commun. Math. Phys."},{"key":"mlstabdaf8bib45","article-title":"Population based training of neural networks","author":"Jaderberg","year":"2017"},{"key":"mlstabdaf8bib46","author":"Saad","year":"2003"},{"key":"mlstabdaf8bib47","author":"Patankar","year":"2018"},{"key":"mlstabdaf8bib48","doi-asserted-by":"publisher","first-page":"163","DOI":"10.2514\/3.8890","article-title":"Features of a reattaching turbulent shear layer in divergent channelflow","volume":"23","author":"Driver","year":"1985","journal-title":"AIAA J."},{"key":"mlstabdaf8bib49","first-page":"p 2927","article-title":"Recent developments on the turbulence modeling resource website","author":"Rumsey","year":"2015"},{"key":"mlstabdaf8bib50","first-page":"1","volume":"1000","author":"Jasak","year":"2007"},{"key":"mlstabdaf8bib51","first-page":"625","article-title":"Ten years of industrial experience with the SST turbulence model","volume":"4","author":"Menter","year":"2003","journal-title":"Turbul. Heat Mass Transfer"},{"key":"mlstabdaf8bib52","doi-asserted-by":"publisher","DOI":"10.1088\/1367-2630\/abadb3","article-title":"Turbulence closure modeling with data-driven techniques: physical compatibility and consistency considerations","volume":"22","author":"Taghizadeh","year":"2020","journal-title":"New J. Phys."},{"key":"mlstabdaf8bib53","doi-asserted-by":"publisher","DOI":"10.1063\/1.4896941","article-title":"Sensitivity of aerodynamic forces in laminar and turbulent flow past a square cylinder","volume":"26","author":"Meliga","year":"2014","journal-title":"Phys. Fluids"},{"key":"mlstabdaf8bib54","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1017\/jfm.2013.323","article-title":"The drag-adjoint field of a circular cylinder wake at Reynolds numbers 20, 100 and 500","volume":"730","author":"Wang","year":"2013","journal-title":"J. Fluid Mech."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,6]],"date-time":"2022-02-06T01:07:04Z","timestamp":1644109624000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abdaf8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,14]]},"references-count":54,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,4,14]]},"published-print":{"date-parts":[[2021,6,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/abdaf8","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,14]]},"assertion":[{"value":"Distributed deep reinforcement learning for simulation control","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-09-25","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-01-12","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-04-14","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}