{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T09:52:34Z","timestamp":1753869154521,"version":"3.41.2"},"reference-count":21,"publisher":"Wiley","issue":"5","license":[{"start":{"date-parts":[[2025,3,23]],"date-time":"2025-03-23T00:00:00Z","timestamp":1742688000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Journal of Forecasting"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>ABSTRACT<\/jats:title><jats:p>Real\u2010world systems are often formulated as constrained optimization problems. Techniques to incorporate constraints into neural networks (NN), such as neural ordinary differential equations (Neural ODEs), have been used. However, these introduce hyperparameters that require manual tuning through trial and error, raising doubts about the successful incorporation of constraints into the generated model. This paper describes in detail the two\u2010stage training method for Neural ODEs, a simple, effective, and penalty parameter\u2010free approach to model constrained systems. In this approach, the constrained optimization problem is rewritten as two optimization subproblems that are solved in two stages. The first stage aims at finding feasible NN parameters by minimizing a measure of constraints violation. The second stage aims to find the optimal NN parameters by minimizing the loss function while keeping inside the feasible region. We experimentally demonstrate that our method produces models that satisfy the constraints and also improves their predictive performance, thus ensuring compliance with critical system properties and also contributing to reducing data quantity requirements. Furthermore, we show that the proposed method improves the convergence to an optimal solution and improves the explainability of Neural ODE models. Our proposed two\u2010stage training method can be used with any NN architectures.<\/jats:p>","DOI":"10.1002\/for.3270","type":"journal-article","created":{"date-parts":[[2025,3,24]],"date-time":"2025-03-24T02:05:06Z","timestamp":1742781906000},"page":"1785-1805","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Two\u2010Stage Training Method for Modeling Constrained Systems With Neural Networks"],"prefix":"10.1002","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-4502-937X","authenticated-orcid":false,"given":"C.","family":"Coelho","sequence":"first","affiliation":[{"name":"Centre of Mathematics (CMAT) University of Minho  Braga Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6235-286X","authenticated-orcid":false,"given":"M.\u00a0Fernanda\u00a0P.","family":"Costa","sequence":"additional","affiliation":[{"name":"Centre of Mathematics (CMAT) University of Minho  Braga Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5477-3226","authenticated-orcid":false,"given":"L.L.","family":"Ferr\u00e1s","sequence":"additional","affiliation":[{"name":"Centre of Mathematics (CMAT) University of Minho  Braga Portugal"},{"name":"Department of Mechanical Engineering (Section of Mathematics) and CEFT \u2010 FEUP University of Porto  Porto Portugal"},{"name":"ALiCE, Faculdade de Engenharia University of Porto  Porto Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2025,3,23]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/nme.947"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.camwa.2010.08.018"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-04167-0"},{"key":"e_1_2_9_5_1","unstructured":"Coelho C. M. F. P.Costa andL. L.Ferr\u00e1s.2023. \u201cPrior Knowledge Meets Neural ODEs: A Two\u2010Stage Training Method for Improved Explainability.\u201d InThe First Tiny Papers Track at ICLR 2023 Tiny Papers @ ICLR 2023 Kigali Rwanda May 5 2023 edited byK.Maughan R.Liu andT. F.Burns.https:\/\/openreview.net\/pdf?id=p7sHcNt_tqo."},{"key":"e_1_2_9_6_1","unstructured":"Coelho C. M. F. P.Costa andL. L.Ferr\u00e1s.2023. \u201cSynthetic Chemical Reaction.\u201d Kaggle."},{"key":"e_1_2_9_7_1","unstructured":"Coelho C. M. F. P.Costa andL. L.Ferr\u00e1s.2023. \u201cWorld Population Growth.\u201d Kaggle."},{"key":"e_1_2_9_8_1","doi-asserted-by":"crossref","unstructured":"Fioretto F. P.Van\u00a0Hentenryck TWKMak et\u00a0al.2021. \u201cLagrangian Duality for Constrained Deep Learning.\u201d InMachine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference ECML PKDD 2020 Ghent Belgium September 14\u201318 2020 Proceedings Part V 118\u2013135. Springer.","DOI":"10.1007\/978-3-030-67670-4_8"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s101070100244"},{"key":"e_1_2_9_10_1","doi-asserted-by":"crossref","unstructured":"Hwang H. J. andH.Son.2021. \u201cLagrangian Dual Framework for Conservative Neural Network Solutions of Kinetic Equations.\u201d arXiv preprint arXiv:2106.12147.","DOI":"10.3934\/krm.2021046"},{"key":"e_1_2_9_11_1","unstructured":"Lim Y. H. andM. F.Kasim.2022. \u201cUnifying Physical Systems' Inductive Biases in Neural ODE Using Dynamics Constraints.\u201d Trans. Mach. Learn. Res.https:\/\/doi.org\/10.48550\/arXiv.2208.02632."},{"key":"e_1_2_9_12_1","series-title":"Springer Series in Operation Research and Financial Engineering","volume-title":"Numerical Optimization","author":"Nocedal J.","year":"2006"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30504-7_7"},{"key":"e_1_2_9_14_1","unstructured":"Raissi M. P.Perdikaris andG. E.Karniadakis.2017. \u201cPhysics Informed Deep Learning (Part I): Data\u2010driven Solutions of Nonlinear Partial Differential Equations.\u201dhttps:\/\/doi.org\/10.48550\/arXiv.1711.10561."},{"key":"e_1_2_9_15_1","doi-asserted-by":"crossref","unstructured":"Rodrigues D. D. S. andE. B.Hauser.2014. \u201cModelo Log\u00edstico de Verhulst e M\u00e9todos Num\u00e9ricos na An\u00e1lise do Censo Populacional Mundial.\u201d InCMAC Sul \u2013 Congresso de Matem\u00e1tica Aplicada e Computacional.","DOI":"10.5540\/03.2014.002.01.0068"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2020.12.2182"},{"key":"e_1_2_9_17_1","first-page":"25400","article-title":"Constrained Optimization to Train Neural Networks on Critical and Under\u2010Represented Classes","volume":"34","author":"Sangalli S.","year":"2021","journal-title":"Advances in neural information processing systems"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2023.126424"},{"key":"e_1_2_9_19_1","doi-asserted-by":"crossref","unstructured":"Su X. W.Ji J.An et\u00a0al.2022. \u201cKinetics Parameter Optimization via Neural Ordinary Differential Equations.\u201d arXiv:2209.01862.","DOI":"10.1016\/j.combustflame.2023.112732"},{"key":"e_1_2_9_20_1","unstructured":"Tuor A. J.Drgona andD.Vrabie.2020. \u201cConstrained Neural Ordinary Differential Equations With Stability Guarantees.\u201d arXiv preprint arXiv:2004.10883."},{"key":"e_1_2_9_21_1","doi-asserted-by":"crossref","unstructured":"Xing Y. H.Ye X.Zhang et\u00a0al.2022. \u201cA Continuous Glucose Monitoring Measurements Forecasting Approach via Sporadic Blood Glucose Monitoring.\u201d In2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 860\u2013863.","DOI":"10.1109\/BIBM55620.2022.9995522"},{"key":"e_1_2_9_22_1","unstructured":"Zhong Y. D. B.Dey andA.Chakraborty.2020. \u201cDissipative SymODEN: Encoding Hamiltonian Dynamics With Dissipation and Control Into Deep Learning. arXiv.\u201d Comment: Published at ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations (DeepDiffEq)."}],"container-title":["Journal of Forecasting"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/for.3270","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,2]],"date-time":"2025-07-02T02:04:44Z","timestamp":1751421884000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/for.3270"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,23]]},"references-count":21,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["10.1002\/for.3270"],"URL":"https:\/\/doi.org\/10.1002\/for.3270","archive":["Portico"],"relation":{},"ISSN":["0277-6693","1099-131X"],"issn-type":[{"type":"print","value":"0277-6693"},{"type":"electronic","value":"1099-131X"}],"subject":[],"published":{"date-parts":[[2025,3,23]]},"assertion":[{"value":"2024-06-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}