{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T23:49:19Z","timestamp":1783640959090,"version":"3.55.0"},"reference-count":44,"publisher":"IOP Publishing","issue":"2","license":[{"start":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T00:00:00Z","timestamp":1717027200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T00:00:00Z","timestamp":1717027200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"crossref","award":["N00014-18-1-2865"],"award-info":[{"award-number":["N00014-18-1-2865"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and <jats:italic>L<\/jats:italic>\n                  <jats:sub>2<\/jats:sub> regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal manifold coordinate system, and provide the mapping functions between the ambient space and manifold space, allowing for out-of-sample projections. We validate our framework\u2019s ability to estimate the manifold dimension for a series of datasets from dynamical systems of varying complexities and compare to other state-of-the-art estimators. We analyze the training dynamics of the network to glean insight into the mechanism of low-rank learning and find that collectively each of the implicit regularizing layers compound the low-rank representation and even self-correct during training. Analysis of gradient descent dynamics for this architecture in the linear case reveals the role of the internal linear layers in leading to faster decay of a \u2018collective weight variable\u2019 incorporating all layers, and the role of weight decay in breaking degeneracies and thus driving convergence along directions in which no decay would occur in its absence. We show that this framework can be naturally extended for applications of state-space modeling and forecasting by generating a data-driven dynamic model of a spatiotemporally chaotic partial differential equation using only the manifold coordinates. Finally, we demonstrate that our framework is robust to hyperparameter choices.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad4ba5","type":"journal-article","created":{"date-parts":[[2024,5,14]],"date-time":"2024-05-14T23:02:27Z","timestamp":1715727747000},"page":"025053","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6856-3489","authenticated-orcid":true,"given":"Kevin","family":"Zeng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6043-1716","authenticated-orcid":true,"given":"Carlos E P\u00e9rez","family":"De Jes\u00fas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3967-1317","authenticated-orcid":true,"given":"Andrew J","family":"Fox","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4983-4949","authenticated-orcid":true,"given":"Michael D","family":"Graham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,5,30]]},"reference":[{"key":"mlstad4ba5bib1","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1002\/cpa.3160010401","article-title":"A mathematical example displaying features of turbulence","volume":"1","author":"Hopf","year":"1948-12","journal-title":"Commun. Pure Appl. Math."},{"key":"mlstad4ba5bib2","author":"Temam","year":"1977"},{"key":"mlstad4ba5bib3","article-title":"Attractors. Then and now","author":"Zelik","year":"2022"},{"key":"mlstad4ba5bib4","volume":"vol 218","author":"Lee","year":"2012"},{"key":"mlstad4ba5bib5","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1038\/s42256-022-00575-4","article-title":"Data-driven discovery of intrinsic dynamics","volume":"4","author":"Floryan","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"mlstad4ba5bib6","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.102.074102","article-title":"Hyperbolicity and the effective dimension of spatially extended dissipative systems","volume":"102","author":"Yang","year":"2009","journal-title":"Phys. Rev. Lett."},{"key":"mlstad4ba5bib7","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.108.154101","article-title":"Geometry of inertial manifolds probed via a Lyapunov projection method","volume":"108","author":"Yang","year":"2012","journal-title":"Phys. Rev. Lett."},{"key":"mlstad4ba5bib8","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.117.024101","article-title":"Estimating the dimension of an inertial manifold from unstable periodic orbits","volume":"117","author":"Ding","year":"2016","journal-title":"Phys. Rev. Lett."},{"key":"mlstad4ba5bib9","author":"Jolliffe","year":"1986"},{"key":"mlstad4ba5bib10","doi-asserted-by":"publisher","first-page":"262","DOI":"10.1198\/106186006X113430","article-title":"Sparse principal component analysis","volume":"15","author":"Zou","year":"2004","journal-title":"J. Comput. Graph. Stat."},{"key":"mlstad4ba5bib11","article-title":"Bayesian PCA","volume":"vol 11","author":"Bishop","year":"1998"},{"key":"mlstad4ba5bib12","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1016\/0893-6080(94)90060-4","article-title":"Representation and separation of signals using nonlinear PCA type learning","volume":"7","author":"Karhunen","year":"1994","journal-title":"Neural Netw."},{"key":"mlstad4ba5bib13","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.101.062209","article-title":"Deep learning to discover and predict dynamics on an inertial manifold","volume":"101","author":"Linot","year":"2020","journal-title":"Phys. Rev. E"},{"key":"mlstad4ba5bib14","doi-asserted-by":"publisher","DOI":"10.1063\/5.0069536","article-title":"Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations","volume":"32","author":"Linot","year":"2022","journal-title":"Chaos"},{"key":"mlstad4ba5bib15","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1038\/s42256-022-00464-w","article-title":"Multiscale simulations of complex systems by learning their effective dynamics","volume":"4","author":"Vlachas","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"mlstad4ba5bib16","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1038\/s43588-022-00281-6","article-title":"Automated discovery of fundamental variables hidden in experimental data","volume":"2","author":"Chen","year":"2022","journal-title":"Nat. Comput. Sci."},{"key":"mlstad4ba5bib17","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevFluids.8.044402","article-title":"Data-driven low-dimensional dynamic model of Kolmogorov flow","volume":"8","author":"P\u00e9rez De Jes\u00fas","year":"2023","journal-title":"Phys. Rev. Fluids"},{"key":"mlstad4ba5bib18","doi-asserted-by":"publisher","first-page":"A42","DOI":"10.1017\/jfm.2023.720","article-title":"Dynamics of a data-driven low-dimensional model of turbulent minimal Couette flow","volume":"973","author":"Linot","year":"2023","journal-title":"J. Fluid Mech."},{"key":"mlstad4ba5bib19","doi-asserted-by":"publisher","first-page":"3403","DOI":"10.1103\/PhysRevA.45.3403","article-title":"Determining embedding dimension for phase-space reconstruction using a geometrical construction","volume":"45","author":"Kennel","year":"1992","journal-title":"Phys. Rev. A"},{"key":"mlstad4ba5bib20","article-title":"Deep reconstruction of strange attractors from time series","author":"Gilpin","year":"2020"},{"key":"mlstad4ba5bib21","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.1109\/TETCI.2022.3146332","article-title":"Self-consistent learning of neural dynamical systems from noisy time series","volume":"6","author":"Wang","year":"2022","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"mlstad4ba5bib22","doi-asserted-by":"publisher","first-page":"1373","DOI":"10.1162\/089976603321780317","article-title":"Laplacian eigenmaps for dimensionality reduction and data representation","volume":"15","author":"Belkin","year":"2003","journal-title":"Neural Comput."},{"key":"mlstad4ba5bib23","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"van der Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad4ba5bib24","doi-asserted-by":"publisher","first-page":"2319","DOI":"10.1126\/science.290.5500.2319","article-title":"A global geometric framework for nonlinear dimensionality reduction","volume":"290","author":"Tenenbaum","year":"2000","journal-title":"Science"},{"key":"mlstad4ba5bib25","doi-asserted-by":"publisher","first-page":"2323","DOI":"10.1126\/science.290.5500.2323","article-title":"Nonlinear dimensionality reduction by locally linear embedding","volume":"290","author":"Roweis","year":"2000","journal-title":"Science"},{"key":"mlstad4ba5bib26","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.ins.2015.08.029","article-title":"Intrinsic dimension estimation: advances and open problems","volume":"328","author":"Camastra","year":"2016","journal-title":"Inf. Sci."},{"key":"mlstad4ba5bib27","article-title":"Maximum likelihood estimation of intrinsic dimension","volume":"vol 17","author":"Levina","year":"2004"},{"key":"mlstad4ba5bib28","first-page":"pp 26","article-title":"Multiscale estimation of intrinsic dimensionality of data sets","volume":"vol FS-09-04","author":"Little","year":"2009"},{"key":"mlstad4ba5bib29","article-title":"Implicit regularization in matrix factorization","volume":"vol 30","author":"Gunasekar","year":"2017"},{"key":"mlstad4ba5bib30","article-title":"Implicit regularization in deep matrix factorization","volume":"vol 32","author":"Arora","year":"2019"},{"key":"mlstad4ba5bib31","first-page":"pp 21174","article-title":"Implicit regularization in deep learning may not be explainable by norms","volume":"vol 33","author":"Razin","year":"2020"},{"key":"mlstad4ba5bib32","first-page":"pp 14736","article-title":"Implicit rank-minimizing autoencoder","volume":"vol 33","author":"Jing","year":"2020"},{"key":"mlstad4ba5bib33","article-title":"Neural networks efficiently learn low-dimensional representations with SGD","author":"Mousavi-Hosseini","year":"2023"},{"key":"mlstad4ba5bib34","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2019"},{"key":"mlstad4ba5bib35","author":"Zeng","year":"2024"},{"key":"mlstad4ba5bib36","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1175\/1520-0469(1963)020<0130:DNF>2.0.CO;2","article-title":"Deterministic nonperiodic flow","volume":"20","author":"Lorenz","year":"1963","journal-title":"J. Atmos. Sci."},{"key":"mlstad4ba5bib37","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.84.046214","article-title":"Hyperbolic decoupling of tangent space and effective dimension of dissipative systems","volume":"84","author":"Takeuchi","year":"2011","journal-title":"Phys. Rev. E"},{"key":"mlstad4ba5bib38","doi-asserted-by":"publisher","first-page":"22445","DOI":"10.1073\/pnas.1906995116","article-title":"Data-driven discovery of coordinates and governing equations","volume":"116","author":"Champion","year":"2019","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstad4ba5bib39","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2023.116204","article-title":"Adaptive learning of effective dynamics for online modeling of complex systems","volume":"415","author":"Ki\u010di\u0107","year":"2023","journal-title":"Comput. Methods Appl. Mech. Eng."},{"key":"mlstad4ba5bib40","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1146\/annurev-fluid-122316-045241","article-title":"Wall-Modeled Large-Eddy simulation for complex turbulent flows","volume":"50","author":"Bose","year":"2018","journal-title":"Ann. Rev. Fluid Mech."},{"key":"mlstad4ba5bib41","first-page":"pp 6572","article-title":"Neural ordinary differential equations","author":"Qi Chen","year":"2018"},{"key":"mlstad4ba5bib42","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1007\/s00397-023-01412-0","article-title":"Scattering-Informed Microstructure Prediction during Lagrangian Evolution (SIMPLE)-a data-driven framework for modeling complex fluids in flow","volume":"62","author":"Young","year":"2023","journal-title":"Rheol. Acta"},{"key":"mlstad4ba5bib43","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1016\/0098-1354(95)00040-2","article-title":"Alternative approaches to the Karhunen-Loeve decomposition for model reduction and data analysis","volume":"20","author":"Graham","year":"1996","journal-title":"Comput. Chem. Eng."},{"key":"mlstad4ba5bib44","author":"Hinch","year":"1991"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T11:05:09Z","timestamp":1717067109000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad4ba5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,30]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,5,30]]},"published-print":{"date-parts":[[2024,6,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad4ba5","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,30]]},"assertion":[{"value":"Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2023-04-28","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-05-14","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-05-30","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}