{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T18:41:20Z","timestamp":1791916880537,"version":"4.3.4"},"reference-count":71,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"6","funder":[{"DOI":"10.13039\/100010665","name":"H2020 Marie Sk\u0142odowska-Curie Actions","doi-asserted-by":"publisher","award":["860124"],"award-info":[{"award-number":["860124"]}],"id":[{"id":"10.13039\/100010665","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012112","name":"Isaac Newton Institute for Mathematical Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012112","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/R014604\/1"],"award-info":[{"award-number":["EP\/R014604\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Sci. Comput."],"published-print":{"date-parts":[[2023,12,31]]},"abstract":"<jats:p>Abstract.<\/jats:p>\n                  <jats:p>Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a nonautonomous ODE and build neural networks using a suitable, structure-preserving, numerical time discretization. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modeling procedure allows a better theoretical understanding of their behavior. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.<\/jats:p>\n                  <jats:p>Reproducibility of computational results. This paper has been awarded the \u201cSIAM Reproducibility Badge: code and data available\u201d, as a recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available in https:\/\/github.com\/davidemurari\/StructuredNeuralNetworks OR in the Supplementary Materials ( 123366_2_supp_531986_rxfx2s_sc.pdf [736KB]). [Formula: see text]<\/jats:p>","DOI":"10.1137\/22m1527337","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T13:58:00Z","timestamp":1702389480000},"page":"A3071-A3094","source":"Crossref","is-referenced-by-count":12,"title":["Dynamical Systems\u2013Based Neural Networks"],"prefix":"10.1137","volume":"45","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2863-2603","authenticated-orcid":true,"given":"Elena","family":"Celledoni","sequence":"first","affiliation":[{"name":"Department of Mathematical Sciences, NTNU, N-7491 Trondheim, Norway."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1095-6685","authenticated-orcid":true,"given":"Davide","family":"Murari","sequence":"additional","affiliation":[{"name":"Department of Mathematical Sciences, NTNU, N-7491 Trondheim, Norway."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6662-9704","authenticated-orcid":true,"given":"Brynjulf","family":"Owren","sequence":"additional","affiliation":[{"name":"Department of Mathematical Sciences, NTNU, N-7491 Trondheim, Norway."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carola-Bibiane","family":"Sch\u00f6nlieb","sequence":"additional","affiliation":[{"name":"Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge CB3 0WA, UK."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4809-7254","authenticated-orcid":true,"given":"Ferdia","family":"Sherry","sequence":"additional","affiliation":[{"name":"Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge CB3 0WA, UK."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1007\/s10883-021-09561-2"},{"key":"ref2","first-page":"949","volume":"17","author":"Alpcan T.","year":"2010","journal-title":"Dyn. Contin. Discrete Impulsive Syst. Ser. A Math. Anal."},{"key":"ref3","unstructured":"C. Anil, J. Lucas, and R. Grosse, Sorting out Lipschitz function approximation, in International Conference on Machine Learning, PMLR, 2019, pp. 291\u2013301."},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492919000059"},{"key":"ref5","first-page":"688","volume":"32","author":"Bai S.","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref6","unstructured":"J. Baj\u0101rs, Locally-symplectic Neural Networks for Learning Volume-Preserving Dynamics, preprint, https:\/\/arxiv.org\/abs\/2109.09151, 2021."},{"key":"ref7","first-page":"6240","volume":"30","author":"Bartlett P. L.","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref8","first-page":"10665","volume":"34","author":"Bashivan P.","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref9","doi-asserted-by":"crossref","unstructured":"J. Berner, P. Grohs, G. Kutyniok, and P. Petersen, The Modern Mathematics of Deep Learning, preprint, https:\/\/arxiv.org\/abs\/2105.04026, 2021.","DOI":"10.1017\/9781009025096.002"},{"key":"ref10","unstructured":"S. Blanes, A. Iserles, and S. Macnamara, Positivity-Preserving Methods for Population Models, preprint, https:\/\/arxiv.org\/abs\/2102.08242, 2021."},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1007\/s005260100144"},{"key":"ref12","unstructured":"M. M. Bronstein, J. Bruna, T. Cohen, and P. Veli\u010dkovi\u0107, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, preprint, https:\/\/arxiv.org\/abs\/2104.13478, 2021."},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2693418"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6587\/abcbaa"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1016\/S0168-9274(03)00101-6"},{"key":"ref16","doi-asserted-by":"crossref","unstructured":"N. Carlini and D. Wagner, Towards evaluating the robustness of neural networks, in 2017 IEEE Symposium on Security and Privacy, IEEE, 2017, pp. 39\u201357.","DOI":"10.1109\/SP.2017.49"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1017\/S0956792521000139"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1016\/j.cam.2022.114608"},{"key":"ref19","unstructured":"B. Chang, M. Chen, E. Haber, and E. H. Chi, AntisymmetricRNN: A dynamical system view on recurrent neural networks, in International Conference on Learning Representations, 2019."},{"key":"ref20","first-page":"6571","volume":"31","author":"Chen R. T.","year":"2018","journal-title":"Adv. Neur. Inf. Process. Syst."},{"key":"ref21","unstructured":"Z. Chen, J. Zhang, M. Arjovsky, and L. Bottou, Symplectic recurrent neural networks, in International Conference on Learning Representations, 2020."},{"key":"ref22","unstructured":"G. Dahlquist and R. Jeltsch, Generalized Disks of Contractivity for Explicit and Implicit Runge-Kutta Methods, Technical report, Department of Numerical Analysis and Computational Science, Royal Institute of Technology, Stockholm, 1979."},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1007\/s10543-022-00909-z"},{"key":"ref24","doi-asserted-by":"crossref","unstructured":"G. Fran\u00e7a, A. Barp, M. Girolami, and M. I. Jordan, Optimization on Manifolds: A Symplectic Approach, preprint, https:\/\/arxiv.org\/abs\/2107.11231, 2021.","DOI":"10.1088\/1742-5468\/abf5d4"},{"key":"ref25","unstructured":"C. L. Galimberti, L. Furieri, L. Xu, and G. Ferrari-Trecate, Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design, preprint, https:\/\/arxiv.org\/abs\/2105.13205, 2021."},{"key":"ref26","first-page":"2214","volume":"30","author":"Gomez A. N.","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref27","unstructured":"I. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples, in ICLR 2015, San Diego, CA, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, eds. 2015."},{"key":"ref28","volume-title":"Geometric Numerical Integration: Structure-preserving Algorithms for Ordinary Differential Equations","author":"Haier E.","year":"2006"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecolmodel.2010.06.014"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.laa.2021.09.004"},{"key":"ref31","series-title":"Grundlehren Math. Wiss. 305","volume-title":"Convex Analysis and Minimization Algorithms I: Fundamentals","author":"Hiriart-Urruty J.-B.","year":"2013"},{"key":"ref32","first-page":"2698","volume":"33","author":"Huang K.","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2020.08.017"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03819-2"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1989.1.4.541"},{"key":"ref36","series-title":"Cambridge Monogr. Appl. Comput. Math. 14","volume-title":"Simulating Hamiltonian Dynamics","author":"Leimkuhler B.","year":"2004"},{"key":"ref37","unstructured":"B. Leimkuhler, T. J. Vlaar, T. Pouchon, and A. Storkey, Better training using weight-constrained stochastic dynamics, in Proceedings of the 38th International Conference on Machine Learning, PMLR, 2021, pp. 6200\u20136211."},{"key":"ref38","first-page":"15390","volume":"32","author":"Li Q.","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.4171\/JEMS\/1221"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0017-8"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1109\/37.793443"},{"key":"ref42","unstructured":"A. Look, S. Doneva, M. Kandemir, R. Gemulla, and J. Peters, Differentiable Implicit Layers, preprint, https:\/\/arxiv.org\/abs\/2010.07078, 2020."},{"key":"ref43","unstructured":"Y. Lu, A. Zhong, Q. Li, and B. Dong, Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations, in International Conference on Machine Learning, PMLR, 2018, pp. 3276\u20133285."},{"key":"ref44","unstructured":"A. L. Maas, A. Y. Hannun, and A. Y. Ng, Rectifier nonlinearities improve neural network acoustic models, in Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, 2013, p. 3."},{"key":"ref45","unstructured":"A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, Towards deep learning models resistant to Adversarial Attacks, in International Conference on Learning Representations, 2018."},{"key":"ref46","first-page":"3952","volume":"33","author":"Massaroli S.","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492902000053"},{"key":"ref48","unstructured":"L. Meunier, B. Delattre, A. Araujo, and A. Allauzen, Scalable Lipschitz Residual Networks with Convex Potential Flows, preprint, https:\/\/arxiv.org\/abs\/2110.12690, 2021."},{"key":"ref49","unstructured":"M. Ozay and T. Okatani, Optimization on Submanifolds of Convolution Kernels in CNNs, preprint, https:\/\/arxiv.org\/abs\/1610.07008, 2016."},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492900002919"},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.1016\/S0960-0779(02)00023-1"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2018.10.045"},{"key":"ref53","unstructured":"J. Rauber, W. Brendel, and M. Bethge, Foolbox: A Python Toolbox to Benchmark the Robustness of Machine Learning Models, preprint, https:\/\/arxiv.org\/abs\/1707.04131, 2017."},{"key":"ref54","first-page":"178","volume-title":"Numerical Analysis: An Introduction","author":"Robertson H.","year":"1966"},{"key":"ref55","unstructured":"D. Ruiz-Balet and E. Zuazua, Neural ODE Control for Classification, Approximation and Transport, preprint, https:\/\/arxiv.org\/abs\/2104.05278, 2021."},{"key":"ref56","unstructured":"T. K. Rusch and S. Mishra, UnICORNN: A recurrent model for learning very long time dependencies, in International Conference on Machine Learning, PMLR, 2021, pp. 9168\u20139178."},{"key":"ref57","doi-asserted-by":"publisher","DOI":"10.1007\/s10851-019-00903-1"},{"key":"ref58","unstructured":"A. M. Saxe, J. L. McClelland, and S. Ganguli, Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks, preprint, https:\/\/arxiv.org\/abs\/1312.6120, 2014."},{"key":"ref59","unstructured":"R. Shwartz-Ziv and N. Tishby, Opening the Black Box of Deep Neural Networks via Information, preprint, https:\/\/arxiv.org\/abs\/1703.00810, 2017."},{"key":"ref60","doi-asserted-by":"publisher","DOI":"10.1007\/s10851-022-01114-x"},{"key":"ref61","unstructured":"C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, Intriguing Properties of Neural Networks, preprint, https:\/\/arxiv.org\/abs\/1312.6199, 2014."},{"key":"ref62","unstructured":"T. Teshima, K. Tojo, M. Ikeda, I. Ishikawa, and K. Oono, Universal Approximation Property of Neural Ordinary Differential Equations, preprint, https:\/\/arxiv.org\/abs\/2012.02414, 2020."},{"key":"ref63","unstructured":"M. Thorpe and Y. van Gennip, Deep Limits of Residual Neural Networks, preprint, https:\/\/arxiv.org\/abs\/1810.11741, 2018."},{"key":"ref64","unstructured":"A. Trockman and J. Z. Kolter, Orthogonalizing Convolutional Layers with the Cayley Transform, in International Conference on Learning Representations, 2021."},{"key":"ref65","first-page":"6542","volume":"31","author":"Tsuzuku Y.","year":"2018","journal-title":"Adv. Neural Inf. Process. Syt."},{"key":"ref66","doi-asserted-by":"crossref","unstructured":"J. Wang, Y. Chen, R. Chakraborty, and S. X. Yu, Orthogonal convolutional neural networks, in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11505\u201311515.","DOI":"10.1109\/CVPR42600.2020.01152"},{"key":"ref67","first-page":"1","volume":"1","author":"Weinan E.","year":"2017","journal-title":"Commun. Math. Stat."},{"key":"ref68","unstructured":"L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington, Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks, in International Conference on Machine Learning, PMLR, 2018, pp. 5393\u20135402."},{"key":"ref69","doi-asserted-by":"crossref","unstructured":"M. Zakwan, L. Xu, and G. Ferrari-Trecate, On Robust Classification Using Contractive Hamiltonian Neural ODEs, preprint, https:\/\/arxiv.org\/abs\/2203.11805, 2022.","DOI":"10.1109\/LCSYS.2022.3186959"},{"key":"ref70","unstructured":"H. Zhang, X. Gao, J. Unterman, and T. Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in International Conference on Machine Learning, PMLR, 2020, pp. 11086\u201311095."},{"key":"ref71","unstructured":"A. Zhu, P. Jin, and Y. Tang, Deep Hamiltonian Networks Based on Symplectic Integrators, preprint, https:\/\/arxiv.org\/abs\/2004.13830, 2020."}],"container-title":["SIAM Journal on Scientific Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/22M1527337","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:20:28Z","timestamp":1787332828000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/22M1527337"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,12]]},"references-count":71,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,31]]}},"alternative-id":["10.1137\/22M1527337"],"URL":"https:\/\/doi.org\/10.1137\/22m1527337","relation":{},"ISSN":["1064-8275","1095-7197"],"issn-type":[{"value":"1064-8275","type":"print"},{"value":"1095-7197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,12]]}}}