{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T14:59:07Z","timestamp":1777733947672,"version":"3.51.4"},"reference-count":56,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2018,2,26]],"date-time":"2018-02-26T00:00:00Z","timestamp":1519603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computation"],"abstract":"<jats:p>There are multiple ways in which a stochastic system can be out of statistical equilibrium. It might be subject to time-varying forcing; or be in a transient phase on its way towards equilibrium; it might even be in equilibrium without us noticing it, due to insufficient observations; and it even might be a system failing to admit an equilibrium distribution at all. We review some of the approaches that model the effective statistical behavior of equilibrium and non-equilibrium dynamical systems, and show that both cases can be considered under the unified framework of optimal low-rank approximation of so-called transfer operators. Particular attention is given to the connection between these methods, Markov state models, and the concept of metastability, further to the estimation of such reduced order models from finite simulation data. All these topics bear an important role in, e.g., molecular dynamics, where Markov state models are often and successfully utilized, and which is the main motivating application in this paper. We illustrate our considerations by numerical examples.<\/jats:p>","DOI":"10.3390\/computation6010022","type":"journal-article","created":{"date-parts":[[2018,2,27]],"date-time":"2018-02-27T03:36:12Z","timestamp":1519702572000},"page":"22","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Optimal Data-Driven Estimation of Generalized Markov State Models for Non-Equilibrium Dynamics"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8127-6804","authenticated-orcid":false,"given":"P\u00e9ter","family":"Koltai","sequence":"first","affiliation":[{"name":"Department of Mathematics and Computer Science, Freie Universit\u00e4t Berlin, 14195 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Freie Universit\u00e4t Berlin, 14195 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4169-9324","authenticated-orcid":false,"given":"Frank","family":"No\u00e9","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Freie Universit\u00e4t Berlin, 14195 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christof","family":"Sch\u00fctte","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Freie Universit\u00e4t Berlin, 14195 Berlin, Germany"},{"name":"Zuse Institute Berlin, 14195 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,2,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Sch\u00fctte, C., and Sarich, M. (2013). Metastability and Markov State Models in Molecular Dynamics: Modeling, Analysis, Algorithmic Approaches, American Mathematical Society.","DOI":"10.1090\/cln\/024"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"174105","DOI":"10.1063\/1.3565032","article-title":"Markov models of molecular kinetics: Generation and validation","volume":"134","author":"Prinz","year":"2011","journal-title":"J. Chem. Phys."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bowman, G.R., Pande, V.S., and No\u00e9, F. (2014). An Introduction to Markov State Models and Their Application to Long Timescale Molecular Simulation. Advances in Experimental Medicine and Biology, Springer.","DOI":"10.1007\/978-94-007-7606-7"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5525","DOI":"10.1021\/acs.jctc.5b00743","article-title":"PyEMMA 2: A software package for estimation, validation, and analysis of Markov models","volume":"11","author":"Scherer","year":"2015","journal-title":"J. Chem. Theory Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.bpj.2016.10.042","article-title":"MSMBuilder: Statistical Models for Biomolecular Dynamics","volume":"112","author":"Harrigan","year":"2017","journal-title":"Biophys. J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"204105","DOI":"10.1063\/1.3590108","article-title":"Markov State Models based on Milestoning","volume":"134","author":"Lu","year":"2011","journal-title":"J. Chem. Phys."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1154","DOI":"10.1137\/090764049","article-title":"On the approximation quality of Markov state models","volume":"8","author":"Sarich","year":"2010","journal-title":"Multiscale Model. Simul."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1137\/100798910","article-title":"Estimating the eigenvalue error of Markov State Models","volume":"10","author":"Djurdjevac","year":"2012","journal-title":"Multiscale Model. Simul."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1137\/110858616","article-title":"A variational approach to modeling slow processes in stochastic dynamical systems","volume":"11","year":"2013","journal-title":"Multiscale Model. Simul."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1739","DOI":"10.1021\/ct4009156","article-title":"Variational approach to molecular kinetics","volume":"10","author":"Keller","year":"2014","journal-title":"J. Chem. Theory Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1006\/jcph.1999.6231","article-title":"A direct approach to conformational dynamics based on hybrid Monte Carlo","volume":"151","author":"Fischer","year":"1999","journal-title":"J. Comput. Phys."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.laa.2004.10.026","article-title":"Robust Perron cluster analysis in conformation dynamics","volume":"398","author":"Deuflhard","year":"2005","journal-title":"Linear Algebra Its Appl."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"E3221","DOI":"10.1073\/pnas.1525092113","article-title":"Multiensemble Markov models of molecular thermodynamics and kinetics","volume":"113","author":"Wu","year":"2016","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"06B612","DOI":"10.1063\/1.3592152","article-title":"Dynamical reweighting: Improved estimates of dynamical properties from simulations at multiple temperatures","volume":"134","author":"Chodera","year":"2011","journal-title":"J. Chem. Phys."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"043116","DOI":"10.1063\/1.3502450","article-title":"Transport in time-dependent dynamical systems: Finite-time coherent sets","volume":"20","author":"Froyland","year":"2010","journal-title":"Chaos Interdiscip. J. Nonlinear Sci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physd.2013.01.013","article-title":"An analytic framework for identifying finite-time coherent sets in time-dependent dynamical systems","volume":"250","author":"Froyland","year":"2013","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"174103","DOI":"10.1063\/1.4966157","article-title":"On metastability and Markov state models for non-stationary molecular dynamics","volume":"145","author":"Koltai","year":"2016","journal-title":"J. Chem. Phys."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"154104","DOI":"10.1063\/1.4979344","article-title":"Variational Koopman models: Slow collective variables and molecular kinetics from short off-equilibrium simulations","volume":"146","author":"Wu","year":"2017","journal-title":"J. Chem. Phys."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1021\/ct500997y","article-title":"Building Markov State Models for Periodically Driven Non-Equilibrium Systems","volume":"11","author":"Wang","year":"2015","journal-title":"J. Chem. Theory Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1948","DOI":"10.1088\/1361-6544\/aa6693","article-title":"Estimating long-term behavior of periodically driven flows without trajectory integration","volume":"30","author":"Froyland","year":"2017","journal-title":"Nonlinearity"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"10007","DOI":"10.1209\/0295-5075\/89\/10007","article-title":"Fluctuation-dissipation theorem in nonequilibrium steady states","volume":"89","author":"Seifert","year":"2010","journal-title":"EPL Europhys. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"022134","DOI":"10.1103\/PhysRevE.96.022134","article-title":"Nonequilibrium steady states in Langevin thermal systems","volume":"96","author":"Lee","year":"2017","journal-title":"Phys. Rev. E"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"174106","DOI":"10.1063\/1.4802007","article-title":"Hierarchical Nystroem methods for constructing Markov state models for conformational dynamics","volume":"138","author":"Yao","year":"2013","journal-title":"J. Chem. Phys."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"121905","DOI":"10.1063\/1.4812768","article-title":"Quantitative comparison of alternative methods for coarse-graining biological networks","volume":"139","author":"Bowman","year":"2013","journal-title":"J. Chem. Phys."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"115004","DOI":"10.1088\/1367-2630\/17\/11\/115004","article-title":"Cycle representatives for the coarse-graining of systems driven into a non-equilibrium steady state","volume":"17","author":"Knoch","year":"2015","journal-title":"New J. Phys."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1143\/PTP.33.423","article-title":"Transport, collective motion, and Brownian motion","volume":"33","author":"Mori","year":"1965","journal-title":"Prog. Theor. Phys."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/BF01008729","article-title":"Nonlinear generalized Langevin equations","volume":"9","author":"Zwanzig","year":"1973","journal-title":"J. Stat. Phys."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2968","DOI":"10.1073\/pnas.97.7.2968","article-title":"Optimal prediction and the Mori\u2013Zwanzig representation of irreversible processes","volume":"97","author":"Chorin","year":"2000","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1016\/S0167-2789(02)00446-3","article-title":"Optimal prediction with memory","volume":"166","author":"Chorin","year":"2002","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_30","unstructured":"Wu, H., and No\u00e9, F. (arXiv, 2017). Variational approach for learning Markov processes from time series data, arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/0167-7152(94)00085-M","article-title":"Rates of convergence for everywhere-positive Markov chains","volume":"22","author":"Baxter","year":"1995","journal-title":"Stat. Probab. Lett."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1080\/10618600.1992.10477008","article-title":"On the convergence of successive substitution sampling","volume":"1","author":"Schervish","year":"1992","journal-title":"J. Comput. Graph. Stat."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Klus, S., N\u00fcske, F., Koltai, P., Wu, H., Kevrekidis, I., Sch\u00fctte, C., and No\u00e9, F. (2018). Data-Driven Model Reduction and Transfer Operator Approximation. J. Nonlinear Sci., 1\u201326.","DOI":"10.1007\/s00332-017-9437-7"},{"key":"ref_34","first-page":"199","article-title":"Geometric ergodicity of some hypo-elliptic diffusions for particle motions","volume":"8","author":"Mattingly","year":"2002","journal-title":"Markov Process. Relat. Fields"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/S0304-4149(02)00150-3","article-title":"Ergodicity for SDEs and approximations: Locally Lipschitz vector fields and degenerate noise","volume":"101","author":"Mattingly","year":"2002","journal-title":"Stoch. Process. Their Appl."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bittracher, A., Koltai, P., Klus, S., Banisch, R., Dellnitz, M., and Sch\u00fctte, C. (2017). Transition Manifolds of Complex Metastable Systems. J. Nonlinear Sci., 1\u201342.","DOI":"10.1007\/s00332-017-9415-0"},{"key":"ref_37","unstructured":"Denner, A. (2017). Coherent Structures and Transfer Operators. [Ph.D. Thesis, Technische Universit\u00e4t M\u00fcnchen]."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hsing, T., and Eubank, R. (2015). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators, John Wiley & Sons.","DOI":"10.1002\/9781118762547"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.ocemod.2012.05.001","article-title":"Three-dimensional characterization and tracking of an Agulhas Ring","volume":"52\u201353","author":"Froyland","year":"2012","journal-title":"Ocean Model."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"083119","DOI":"10.1063\/1.4927830","article-title":"Studying an Agulhas ring\u2019s long-term pathway and decay with finite-time coherent sets","volume":"25","author":"Froyland","year":"2015","journal-title":"Chaos"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1307","DOI":"10.1007\/s00332-015-9258-5","article-title":"A data\u2013driven approximation of the Koopman operator: Extending dynamic mode decomposition","volume":"25","author":"Williams","year":"2015","journal-title":"J. Nonlinear Sci."},{"key":"ref_42","first-page":"51","article-title":"On the numerical approximation of the Perron\u2013Frobenius and Koopman operator","volume":"3","author":"Klus","year":"2016","journal-title":"J. Comput. Dyn."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Korda, M., and Mezi\u0107, I. (2017). On convergence of extended dynamic mode decomposition to the Koopman operator. J. Nonlinear Sci., 1\u201324.","DOI":"10.1007\/s00332-017-9423-0"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"015102","DOI":"10.1063\/1.4811489","article-title":"Identification of slow molecular order parameters for Markov model construction","volume":"139","author":"Paul","year":"2013","journal-title":"J. Chem. Phys."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2000","DOI":"10.1021\/ct300878a","article-title":"Improvements in Markov state model construction reveal many non-native interactions in the folding of NTL9","volume":"9","author":"Schwantes","year":"2013","journal-title":"J. Chem. Theory Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"3634","DOI":"10.1103\/PhysRevLett.72.3634","article-title":"Separation of a mixture of independent signals using time delayed correlations","volume":"72","author":"Molgedey","year":"1994","journal-title":"Phys. Rev. Lett."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1111\/j.2517-6161.1954.tb00145.x","article-title":"Poor man\u2019s Monte Carlo","volume":"16","author":"Hammersley","year":"1954","journal-title":"J. R. Stat. Soc. Ser. B Methodol."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1063\/1.1741967","article-title":"Monte Carlo calculation of the average extension of molecular chains","volume":"23","author":"Rosenbluth","year":"1955","journal-title":"J. Chem. Phys."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2690","DOI":"10.1103\/PhysRevLett.78.2690","article-title":"Nonequilibrium equality for free energy differences","volume":"78","author":"Jarzynski","year":"1997","journal-title":"Phys. Rev. Lett."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2721","DOI":"10.1103\/PhysRevE.60.2721","article-title":"Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences","volume":"60","author":"Crooks","year":"1999","journal-title":"Phys. Rev. E"},{"key":"ref_51","unstructured":"Bucklew, J. (2013). Introduction to Rare Event Simulation, Springer Science & Business Media."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"P11004","DOI":"10.1088\/1742-5468\/2012\/11\/P11004","article-title":"Efficient rare event simulation by optimal nonequilibrium forcing","volume":"2012","author":"Hartmann","year":"2012","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Hartmann, C., Richter, L., Sch\u00fctte, C., and Zhang, W. (2017). Variational Characterization of Free Energy: Theory and Algorithms. Entropy, 19.","DOI":"10.3390\/e19110626"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1137\/S0036142996313002","article-title":"On the approximation of complicated dynamical behavior","volume":"36","author":"Dellnitz","year":"1999","journal-title":"SIAM J. Numer. Anal."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1319","DOI":"10.1137\/15M1032272","article-title":"Finding dominant structures of nonreversible Markov processes","volume":"14","author":"Weber","year":"2016","journal-title":"Multiscale Model. Simul."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3934\/jcd.2015.2.1","article-title":"Modularity of directed networks: Cycle decomposition approach","volume":"2","author":"Conrad","year":"2015","journal-title":"J. Comput. Dyn."}],"container-title":["Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-3197\/6\/1\/22\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:56:28Z","timestamp":1760194588000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-3197\/6\/1\/22"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,26]]},"references-count":56,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,3]]}},"alternative-id":["computation6010022"],"URL":"https:\/\/doi.org\/10.3390\/computation6010022","relation":{},"ISSN":["2079-3197"],"issn-type":[{"value":"2079-3197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,26]]}}}