{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T16:24:09Z","timestamp":1775579049988,"version":"3.50.1"},"reference-count":57,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"crossref","award":["ScaDS.AI"],"award-info":[{"award-number":["ScaDS.AI"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001659","name":"German Research Foundation","doi-asserted-by":"crossref","award":["EXC-2068"],"award-info":[{"award-number":["EXC-2068"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2022,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We characterize and remedy a failure mode that may arise from multi-scale dynamics with scale imbalances during training of deep neural networks, such as physics informed neural networks (PINNs). PINNs are popular machine-learning templates that allow for seamless integration of physical equation models with data. Their training amounts to solving an optimization problem over a weighted sum of data-fidelity and equation-fidelity objectives. Conflicts between objectives can arise from scale imbalances, heteroscedasticity in the data, stiffness of the physical equation, or from catastrophic interference during sequential training. We explain the training pathology arising from this and propose a simple yet effective inverse Dirichlet weighting strategy to alleviate the issue. We compare with Sobolev training of neural networks, providing the baseline of analytically <jats:italic>\u03b5<\/jats:italic>-optimal training. We demonstrate the effectiveness of inverse Dirichlet weighting in various applications, including a multi-scale model of active turbulence, where we show orders of magnitude improvement in accuracy and convergence over conventional PINN training. For inverse modeling using sequential training, we find that inverse Dirichlet weighting protects a PINN against catastrophic forgetting.<\/jats:p>","DOI":"10.1088\/2632-2153\/ac3712","type":"journal-article","created":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T22:23:46Z","timestamp":1636151026000},"page":"015026","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":45,"title":["Inverse Dirichlet weighting enables reliable training of physics informed neural networks"],"prefix":"10.1088","volume":"3","author":[{"given":"Suryanarayana","family":"Maddu","sequence":"first","affiliation":[]},{"given":"Dominik","family":"Sturm","sequence":"additional","affiliation":[]},{"given":"Christian L","family":"M\u00fcller","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4414-4340","authenticated-orcid":true,"given":"Ivo F","family":"Sbalzarini","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"key":"mlstac3712bib1","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"mlstac3712bib2","article-title":"WaveNet: a generative model for raw audio","author":"van den Oord","year":"2016"},{"key":"mlstac3712bib3","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"mlstac3712bib4","doi-asserted-by":"publisher","first-page":"1339","DOI":"10.1016\/j.jcp.2018.08.029","article-title":"Dgm: a deep learning algorithm for solving partial differential equations","volume":"375","author":"Sirignano","year":"2018","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib5","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.jcp.2018.10.045","article-title":"Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","volume":"378","author":"Raissi","year":"2019","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib6","doi-asserted-by":"publisher","first-page":"803","DOI":"10.1142\/S0219530519410021","article-title":"Error bounds for approximations with deep ReLU neural networks in Ws,p norms","volume":"18","author":"G\u00fchring","year":"2020","journal-title":"Anal. Appl."},{"key":"mlstac3712bib7","first-page":"pp 4278","article-title":"Sobolev training for neural networks","author":"Czarnecki","year":"2017"},{"key":"mlstac3712bib8","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1017\/jfm.2018.872","article-title":"Deep learning of vortex-induced vibrations","volume":"861","author":"Raissi","year":"2019","journal-title":"J. Fluid Mech."},{"key":"mlstac3712bib9","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2963375","article-title":"Deep physical informed neural networks for metamaterial design","volume":"8","author":"Fang","year":"2019","journal-title":"IEEE Access"},{"key":"mlstac3712bib10","doi-asserted-by":"publisher","DOI":"10.1115\/1.4044400","article-title":"Multi-fidelity physics-constrained neural network and its application in materials modeling","volume":"141","author":"Liu","year":"2019","journal-title":"J. Mech. Des."},{"key":"mlstac3712bib11","doi-asserted-by":"publisher","DOI":"10.1364\/OE.384875","article-title":"Physics-informed neural networks for inverse problems in nano-optics and metamaterials","volume":"28","author":"Chen","year":"2020","journal-title":"Opt. Express"},{"key":"mlstac3712bib12","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.1126\/science.aaw4741","article-title":"Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations","volume":"367","author":"Raissi","year":"2020","journal-title":"Science"},{"key":"mlstac3712bib13","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2019.112732","article-title":"Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data","volume":"361","author":"Sun","year":"2020","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"mlstac3712bib14","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2019.112789","article-title":"Physics-informed neural networks for high-speed flows","volume":"360","author":"Mao","year":"2020","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"mlstac3712bib15","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevFluids.4.124501","article-title":"Deep learning of turbulent scalar mixing","volume":"4","author":"Raissi","year":"2019","journal-title":"Phys. Rev. Fluids"},{"key":"mlstac3712bib16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2020.109951","article-title":"NSFnets (Navier-Stokesflow nets): physics-informed neural networks for the incompressible Navier-Stokes equations","volume":"426","author":"Jin","year":"2020","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1007575","article-title":"Systems biology informed deep learning for inferring parameters and hidden dynamics","volume":"16","author":"Yazdani","year":"2020","journal-title":"PLoS Computat. Biol."},{"key":"mlstac3712bib18","doi-asserted-by":"publisher","first-page":"42","DOI":"10.3389\/fphy.2020.00042","article-title":"Physics-informed neural networks for cardiac activation mapping","volume":"8","author":"Sahli Costabal","year":"2020","journal-title":"Front. Phys."},{"key":"mlstac3712bib19","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2019.112623","article-title":"Machine learning in cardiovascular flows modeling: predicting arterial blood pressure from non-invasive 4D flowMRI data using physics-informed neural networks","volume":"358","author":"Kissas","year":"2020","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"mlstac3712bib20","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2020.109676","article-title":"Physics-informed semantic inpainting: application to geostatistical modeling","volume":"419","author":"Zheng","year":"2020","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib21","doi-asserted-by":"publisher","DOI":"10.1061\/(ASCE)EM.1943-7889.0001947","article-title":"Physics-informed deep learning for computational elastodynamics without labeled data","volume":"147","author":"Rao","year":"2021","journal-title":"J. Eng. Mech."},{"key":"mlstac3712bib22","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2020.109913","article-title":"B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data","volume":"425","author":"Yang","year":"2020","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib23","doi-asserted-by":"publisher","first-page":"A639","DOI":"10.1137\/19M1260141","article-title":"Learning in modal space: solving time-dependent stochastic PDEs using physics-informed neural networks","volume":"42","author":"Zhang","year":"2020","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstac3712bib24","doi-asserted-by":"publisher","first-page":"8505","DOI":"10.1073\/pnas.1718942115","article-title":"Solving high-dimensional partial differential equations using deep learning","volume":"115","author":"Han","year":"2018","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac3712bib25","doi-asserted-by":"publisher","first-page":"A2603","DOI":"10.1137\/18M1229845","article-title":"fPINNs: fractional physics-informed neural networks","volume":"41","author":"Pang","year":"2019","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstac3712bib26","first-page":"pp 5301","article-title":"On the spectral bias of neural networks","author":"Rahaman","year":"2019"},{"key":"mlstac3712bib27","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1007\/978-3-030-36708-4_22","article-title":"Training behavior of deep neural network in frequency domain","author":"Xu","year":"2019"},{"key":"mlstac3712bib28","article-title":"Frequency principle: Fourier analysis sheds light on deep neural networks","author":"Xu","year":"2019"},{"key":"mlstac3712bib29","article-title":"Understanding training and generalization in deep learning by Fourier analysis","author":"Xu","year":"2018"},{"key":"mlstac3712bib30","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2021.110768","article-title":"When and why PINNs fail to train: a neural tangent kernel perspective","volume":"449","author":"Wang","year":"2021","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib31","doi-asserted-by":"publisher","first-page":"7482","DOI":"10.1109\/CVPR.2018.00781","article-title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","author":"Kendall","year":"2018"},{"key":"mlstac3712bib32","first-page":"pp 794","article-title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","author":"Chen","year":"2018"},{"key":"mlstac3712bib33","doi-asserted-by":"publisher","first-page":"525","DOI":"10.5555\/3326943.3326992","article-title":"Multi-task learning as multi-objective optimization","volume":"31","author":"Sener","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"mlstac3712bib34","article-title":"Understanding and mitigating gradient pathologies in physics-informed neural networks","author":"Wang","year":"2020"},{"key":"mlstac3712bib35","article-title":"On the Pareto front of physics-informed neural networks","author":"Rohrhofer","year":"2021"},{"key":"mlstac3712bib36","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1109\/4235.797969","article-title":"Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach","volume":"3","author":"Zitzler","year":"1999","journal-title":"IEEE Trans. Evolutionary Computat."},{"key":"mlstac3712bib37","doi-asserted-by":"publisher","author":"Boyd","year":"2004","DOI":"10.1017\/CBO9780511804441"},{"key":"mlstac3712bib38","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2015"},{"key":"mlstac3712bib39","first-page":"pp 249","article-title":"Understanding the difficulty of training deep feedforward neural networks","author":"Glorot","year":"2010"},{"key":"mlstac3712bib40","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1006\/jcph.2002.6995","article-title":"Exponential time differencing for stiff systems","volume":"176","author":"Cox","year":"2002","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib41","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.crma.2012.03.014","article-title":"Multiple-gradient descent algorithm (MGDA) for multiobjective optimization","volume":"350","author":"D\u00e9sid\u00e9ri","year":"2012","journal-title":"C. R. Math."},{"key":"mlstac3712bib42","article-title":"Optimally weighted loss functions for solving PDEs with neural networks","author":"van der Meer","year":"2020"},{"key":"mlstac3712bib43","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1202032109","article-title":"Meso-scale turbulence in living fluids","volume":"109","author":"Wensink","year":"2012","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac3712bib44","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.110.228102","article-title":"Fluid dynamics of bacterial turbulence","volume":"110","author":"Dunkel","year":"2013","journal-title":"Phys. Rev. Lett."},{"key":"mlstac3712bib45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/srep20838","article-title":"Activity induces traveling waves, vortices and spatiotemporal chaos in a model actomyosin layer","volume":"6","author":"Ramaswamy","year":"2016","journal-title":"Sci. Rep."},{"key":"mlstac3712bib46","doi-asserted-by":"publisher","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"Kirkpatrick","year":"2017","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac3712bib47","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1007\/978-3-030-66415-2_21","article-title":"DenoiSeg: joint denoising and segmentation","author":"Buchholz","year":"2020"},{"key":"mlstac3712bib48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2020.109985","article-title":"Deepmod: deep learning for model discovery in noisy data","volume":"428","author":"Both","year":"2021","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib49","first-page":"1","article-title":"Stochastic gradient descent as approximate Bayesian inference","volume":"18","author":"Mandt","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"mlstac3712bib50","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1112\/jlms\/s1-6.1.44","article-title":"A note on Parseval\u2019s theorem for Fourier transforms","volume":"1","author":"Hardy","year":"1931","journal-title":"J. London Math. Soc."},{"key":"mlstac3712bib51","doi-asserted-by":"publisher","first-page":"1214","DOI":"10.1137\/S1064827502410633","article-title":"Fourth-order time-stepping for stiff PDEs","volume":"26","author":"Kassam","year":"2005","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstac3712bib52","doi-asserted-by":"publisher","author":"Osher","year":"2006","DOI":"10.1007\/b98879"},{"key":"mlstac3712bib53","doi-asserted-by":"publisher","first-page":"7046","DOI":"10.1073\/pnas.92.15.7046","article-title":"Image processing via level set curvature flow","volume":"92","author":"Malladi","year":"1995","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac3712bib54","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1006\/jcph.1996.0167","article-title":"A variational level set approach to multiphase motion","volume":"127","author":"Zhao","year":"1996","journal-title":"J. Comput. Phys."},{"key":"mlstac3712bib55","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1007\/s00285-009-0315-2","article-title":"A Lagrangian particle method for reaction-diffusion systems on deforming surfaces","volume":"61","author":"Bergdorf","year":"2010","journal-title":"J. Math. Biol."},{"key":"mlstac3712bib56","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1007\/BF00375127","article-title":"Axioms and fundamental equations of image processing","volume":"123","author":"Alvarez","year":"1993","journal-title":"Arch. Ration. Mech. Anal."},{"key":"mlstac3712bib57","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2019.109184","article-title":"A variational level set methodology without reinitialization for the prediction of equilibrium interfaces over arbitrary solid surfaces","volume":"406","author":"Alam\u00e9","year":"2020","journal-title":"J. Comput. Phys."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T10:20:27Z","timestamp":1644920427000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3712"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":57,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,2,15]]},"published-print":{"date-parts":[[2022,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ac3712","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,15]]},"assertion":[{"value":"Inverse Dirichlet weighting enables reliable training of physics informed neural networks","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2022 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2021-08-03","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-11-05","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-02-15","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}