{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T05:23:42Z","timestamp":1761110622225,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T00:00:00Z","timestamp":1642377600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100017649","name":"Government of Hong Kong","doi-asserted-by":"publisher","award":["11305318"],"award-info":[{"award-number":["11305318"]}],"id":[{"id":"10.13039\/501100017649","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"publisher","award":["11871486"],"award-info":[{"award-number":["11871486"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We propose a reinforcement learning (RL) approach to compute the expression of quasi-stationary distribution. Based on the fixed-point formulation of quasi-stationary distribution, we minimize the KL-divergence of two Markovian path distributions induced by candidate distribution and true target distribution. To solve this challenging minimization problem by gradient descent, we apply a reinforcement learning technique by introducing the reward and value functions. We derive the corresponding policy gradient theorem and design an actor-critic algorithm to learn the optimal solution and the value function. The numerical examples of finite state Markov chain are tested to demonstrate the new method.<\/jats:p>","DOI":"10.3390\/e24010133","type":"journal-article","created":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T08:20:42Z","timestamp":1642407642000},"page":"133","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Learn Quasi-Stationary Distributions of Finite State Markov Chain"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5920-6471","authenticated-orcid":false,"given":"Zhiqiang","family":"Cai","sequence":"first","affiliation":[{"name":"School of Data Science, City University of Hong Kong, Tat Chee Ave, Kowloon, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0559-2319","authenticated-orcid":false,"given":"Ling","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Mathematics, Sun Yat-sen University, Guangzhou 510275, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3835-3894","authenticated-orcid":false,"given":"Xiang","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Data Science, City University of Hong Kong, Tat Chee Ave, Kowloon, Hong Kong, China"},{"name":"Department of Mathematics, City University of Hong Kong, Tat Chee Ave, Kowloon, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,17]]},"reference":[{"doi-asserted-by":"crossref","unstructured":"Collet, P., Mart\u00ednez, S., and Mart\u00edn, J.S. (2012). Quasi-Stationary Distributions: Markov Chains, Diffusions and Dynamical Systems, Springer Science & Business Media.","key":"ref_1","DOI":"10.1007\/978-3-642-33131-2"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2526","DOI":"10.1016\/j.ecolmodel.2010.02.017","article-title":"Analytical methods for a stochastic mainland\u2013island metapopulation model","volume":"221","author":"Buckley","year":"2010","journal-title":"Ecol. Model."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1080\/15326340802437728","article-title":"Population dynamics and random genealogies","volume":"24","author":"Lambert","year":"2008","journal-title":"Stoch. Model."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1016\/j.physa.2004.06.155","article-title":"Quasi-stationary distributions for models of heterogeneous catalysis","volume":"343","author":"Dickman","year":"2004","journal-title":"Phys. Stat. Mech. Appl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"966","DOI":"10.1063\/1.469796","article-title":"Statistical distribution and stochastic resonance in a periodically driven chemical system","volume":"103","author":"Dykman","year":"1995","journal-title":"J. Chem. Phys."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"799","DOI":"10.1007\/s00285-012-0570-5","article-title":"Stochastic epidemic models with random environment: Quasi-stationarity, extinction and final size","volume":"67","author":"Artalejo","year":"2013","journal-title":"J. Math. Biol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1007\/s11009-010-9177-8","article-title":"Approximating the quasi-stationary distribution of the sis model for endemic infection","volume":"13","author":"Clancy","year":"2011","journal-title":"Methodol. Comput. Appl. Probab."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1016\/j.mbs.2006.09.024","article-title":"Stochastic models for the spread of hiv in a mobile heterosexual population","volume":"208","author":"Sani","year":"2007","journal-title":"Math. Biosci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1177\/0272989X08330121","article-title":"Quantitative risk stratification in markov chains with limiting conditional distributions","volume":"29","author":"Chan","year":"2009","journal-title":"Med. Decis. Mak."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2303","DOI":"10.1088\/0951-7715\/25\/8\/2303","article-title":"Mixed-mode oscillations and interspike interval statistics in the stochastic fitzhugh\u2013nagumo model","volume":"25","author":"Berglund","year":"2012","journal-title":"Nonlinearity"},{"unstructured":"Landon, D. (2012). Perturbation et Excitabilit\u00e9 Dans des Modeles Stochastiques de Transmission de l\u2019Influx Nerveux. [Ph.D. Thesis, Universit\u00e9 d\u2019Orl\u00e9ans].","key":"ref_11"},{"key":"ref_12","first-page":"469","article-title":"Jump markov models and transition state theory: The quasi-stationary distribution approach","volume":"195","author":"Peutrec","year":"2017","journal-title":"Faraday Discuss."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"561","DOI":"10.2140\/apde.2015.8.561","article-title":"Low temperature asymptotics for quasistationary distributions in a bounded domain","volume":"8","author":"Nier","year":"2015","journal-title":"Anal. PDE"},{"unstructured":"Pollock, M., Fearnhead, P., Johansen, A.M., and Roberts, G.O. (2016). The scalable langevin exact algorithm: Bayesian inference for big data. arXiv.","key":"ref_14"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3193","DOI":"10.1016\/j.spa.2019.09.010","article-title":"An approximation scheme for quasi-stationary distributions of killed diffusions","volume":"130","author":"Wang","year":"2020","journal-title":"Stoch. Process. Appl."},{"unstructured":"Watkins, D.S. (2004). Fundamentals of Matrix Computations, John Wiley & Sons.","key":"ref_16"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1545","DOI":"10.1016\/S0167-8191(97)89286-1","article-title":"Parallel implementation of an aggregation\/disaggregation method for evaluating quasi-stationary behavior in continuous-time markov chains","volume":"23","author":"Bebbington","year":"1997","journal-title":"Parallel Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"68","DOI":"10.2307\/1427580","article-title":"An efficient procedure for computing quasi-stationary distributions of markov chains by sparse transition structure","volume":"26","author":"Pollett","year":"1994","journal-title":"Adv. Appl. Probab."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"911","DOI":"10.2307\/3215316","article-title":"Quasi-stationary distributions for a brownian motion with drift and associated limit laws","volume":"31","author":"Martinez","year":"1994","journal-title":"J. Appl. Probab."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1017\/S026996480000084X","article-title":"Two applications of urn processes the fringe analysis of search trees and the simulation of quasi-stationary distributions of markov chains","volume":"2","author":"Aldous","year":"1988","journal-title":"Probab. Eng. Inform. Sci."},{"key":"ref_21","first-page":"1","article-title":"A stochastic approximation approach to quasi-stationary distributions on finite spaces","volume":"20","author":"Cloez","year":"2015","journal-title":"Electron. Commun. Probab."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"016129","DOI":"10.1103\/PhysRevE.71.016129","article-title":"How to simulate the quasistationary state","volume":"71","author":"Dickman","year":"2005","journal-title":"Phys. Rev. E"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"792","DOI":"10.1017\/apr.2016.28","article-title":"Analysis of a stochastic approximation algorithm for computing quasi-stationary distributions","volume":"48","author":"Blanchet","year":"2016","journal-title":"Adv. Appl. Probab."},{"unstructured":"Zheng, S. (2014). Stochastic Approximation Algorithms in the Estimation of Quasi-Stationary Distribution of Finite and General State Space Markov Chains. [Ph.D. Thesis, Columbia University].","key":"ref_24"},{"unstructured":"Kushner, H., and Yin, G.G. (2003). Stochastic Approximation and Recursive Algorithms and Applications, Springer Science & Business Media.","key":"ref_25"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"838","DOI":"10.1137\/0330046","article-title":"Acceleration of stochastic approximation by averaging","volume":"30","author":"Polyak","year":"1992","journal-title":"SIAM J. Control. Optim."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1080\/01621459.2017.1285773","article-title":"Variational inference: A review for statisticians","volume":"112","author":"Blei","year":"2017","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1023\/A:1007665907178","article-title":"An Introduction to Variational Methods for Graphical Models","volume":"37","author":"Jordan","year":"1999","journal-title":"Mach. Learn."},{"unstructured":"Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (2016). Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in Neural Information Processing Systems, Curran Associates, Inc.","key":"ref_29"},{"unstructured":"Bach, F., and Blei, D. (2015, January 7\u20139). Variational inference with normalizing flows. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.","key":"ref_30"},{"unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.","key":"ref_31"},{"unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.","key":"ref_32"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"eaap7885","DOI":"10.1126\/sciadv.aap7885","article-title":"Deep reinforcement learning for de novo drug design","volume":"4","author":"Popova","year":"2018","journal-title":"Sci. Adv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"013013","DOI":"10.1088\/1367-2630\/abd7bd","article-title":"A reinforcement learning approach to rare trajectory sampling","volume":"23","author":"Rose","year":"2021","journal-title":"New J. Phys."},{"key":"ref_36","first-page":"340","article-title":"Quasi-stationary distributions and population processes","volume":"9","author":"Villemonais","year":"2012","journal-title":"Probab. Surv."},{"doi-asserted-by":"crossref","unstructured":"Sch\u00fctze, O., Coello, C.A.C., Tantar, A.-A., Tantar, E., Bouvry, P., Moral, P.D., and Legrand, P. (2013). Empirical analysis of a stochastic approximation approach for computing quasi-stationary distributions. EVOLVE\u2014A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation II, Springer.","key":"ref_37","DOI":"10.1007\/978-3-642-31519-0"},{"doi-asserted-by":"crossref","unstructured":"Boyd, S., Boyd, S.P., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.","key":"ref_38","DOI":"10.1017\/CBO9780511804441"},{"unstructured":"Wang, W., and Carreira-Perpin\u00e1n, M.A. (2013). Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. arXiv.","key":"ref_39"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/133\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:02:16Z","timestamp":1760133736000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,17]]},"references-count":39,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["e24010133"],"URL":"https:\/\/doi.org\/10.3390\/e24010133","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,1,17]]}}}