{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:49:34Z","timestamp":1760237374670,"version":"build-2065373602"},"reference-count":37,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2020,5,11]],"date-time":"2020-05-11T00:00:00Z","timestamp":1589155200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate thermodynamic quantities such as entropy, free energy, and internal energy as a function of the training epoch. We demonstrate the growth of the correlation between the visible and hidden layers via the subadditivity of entropies as the training proceeds. Using the Monte-Carlo simulation of trajectories of the visible and hidden vectors in the configuration space, we also calculate the distribution of the work done on the restricted Boltzmann machine by switching the parameters of the energy function. We discuss the Jarzynski equality which connects the path average of the exponential function of the work and the difference in free energies before and after training.<\/jats:p>","DOI":"10.3390\/e22050538","type":"journal-article","created":{"date-parts":[[2020,5,11]],"date-time":"2020-05-11T12:26:30Z","timestamp":1589199990000},"page":"538","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Entropy, Free Energy, and Work of Restricted Boltzmann Machines"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3983-9877","authenticated-orcid":false,"given":"Sangchul","family":"Oh","sequence":"first","affiliation":[{"name":"Qatar Environment and Energy Research Institute, Hamad Bin Khalifa University, Qatar Foundation, 5825 Doha, Qatar"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8742-5519","authenticated-orcid":false,"given":"Abdelkader","family":"Baggag","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar Foundation, 5825 Doha, Qatar"}]},{"given":"Hyunchul","family":"Nha","sequence":"additional","affiliation":[{"name":"Department of Physics, Texas A&amp;M University at Qatar, Education City, 23874 Doha, Qatar"}]}],"member":"1968","published-online":{"date-parts":[[2020,5,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rumelhart, D., and McLelland, J. (1986). Information processing in dynamical systems: Foundations of harmony theory. Parallel Distributed Processing: Explorations in The Microstructure of Cognition, MIT Press.","DOI":"10.7551\/mitpress\/5236.001.0001"},{"key":"ref_2","unstructured":"Montavon, G., Orr, G.B., and M\u00fcller, K.R. (2012). A Practical Guide to Training Restricted Boltzmann Machines. Neural Networks: Tricks of the Trade: Second Edition, Springer Berlin Heidelberg."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.patcog.2013.05.025","article-title":"Training restricted Boltzmann machines: An introduction","volume":"47","author":"Fischer","year":"2014","journal-title":"Pattern Recognit."},{"key":"ref_4","first-page":"1","article-title":"How to Center Deep Boltzmann Machines","volume":"17","author":"Melchior","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physrep.2019.03.001","article-title":"A high-bias, low-variance introduction to Machine Learning for physicists","volume":"810","author":"Mehta","year":"2019","journal-title":"Phys. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1126\/science.aag2302","article-title":"Solving the quantum many-body problem with artificial neural networks","volume":"355","author":"Carleo","year":"2017","journal-title":"Science"},{"key":"ref_7","first-page":"041006","article-title":"Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines","volume":"8","author":"Tramel","year":"2018","journal-title":"Phys. Rev. X"},{"key":"ref_8","first-page":"021050","article-title":"Quantum Boltzmann Machine","volume":"8","author":"Amin","year":"2018","journal-title":"Phys. Rev. X"},{"key":"ref_9","unstructured":"Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., and Garnett, R. (2016). Supervised Learning with Tensor Networks. Advances in Neural Information Processing Systems 29, Curran Associates, Inc."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1038\/s41467-017-00705-2","article-title":"Efficient representation of quantum many-body states with deep neural networks","volume":"8","author":"Gao","year":"2017","journal-title":"Nat. Commun."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"085104","DOI":"10.1103\/PhysRevB.97.085104","article-title":"Equivalence of restricted Boltzmann machines and tensor network states","volume":"97","author":"Chen","year":"2018","journal-title":"Phys. Rev. B"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1063\/PT.3.4164","article-title":"Machine learning meets quantum physics","volume":"72","author":"Deng","year":"2019","journal-title":"Phys. Today"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"024001","DOI":"10.1088\/2058-9565\/aaea94","article-title":"Towards quantum machine learning with tensor networks","volume":"4","author":"Huggins","year":"2019","journal-title":"Quantum Sci. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4195","DOI":"10.1038\/s41467-018-06598-z","article-title":"Quantum machine learning for electronic structure calculations","volume":"9","author":"Xia","year":"2018","journal-title":"Nat. Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On Information and Sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Statist."},{"key":"ref_16","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elementary Information Theory, Wiley. [2 ed.]."},{"key":"ref_17","unstructured":"Nielsen, M.A., and Chuang, I.L. (2000). Quantum Computation and Quantum Information, Cambridge University Press."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Rumelhart, D.E., and McLelland, J.L. (1986). Learning and relearning in Boltzmann machines. Parallel Distributed Processing: Explorations in The Microstructure of Cognition, MIT Press.","DOI":"10.7551\/mitpress\/5236.001.0001"},{"key":"ref_19","unstructured":"MacKay, D.J.C. (2002). Information Theory, Inference & Learning Algorithms, Cambridge University Press."},{"key":"ref_20","unstructured":"Reif, F. (1965). Fundamentals of Statistical and Thermal Physics, McGraw Hill."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1007\/BF01646092","article-title":"Entropy inequalities","volume":"18","author":"Araki","year":"1970","journal-title":"Commun. Math. Phys."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2690","DOI":"10.1103\/PhysRevLett.78.2690","article-title":"Nonequilibrium Equality for Free Energy Differences","volume":"78","author":"Jarzynski","year":"1997","journal-title":"Phys. Rev. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1146\/annurev-conmatphys-062910-140506","article-title":"Equalities and Inequalities: Irreversibility and the Second Law of Thermodynamics at the Nanoscale","volume":"2","author":"Jarzynski","year":"2011","journal-title":"Annu. Rev. Condens. Matter Phys."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1063\/1.1699114","article-title":"Equation of State Calculations by Fast Computing Machines","volume":"21","author":"Metropolis","year":"1953","journal-title":"J. Chem. Phys."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1093\/biomet\/57.1.97","article-title":"Monte Carlo sampling methods using Markov chains and their applications","volume":"57","author":"Hastings","year":"1970","journal-title":"Biometrika"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1023\/A:1023208217925","article-title":"Nonequilibrium Measurements of Free Energy Differences for Microscopically Reversible Markovian Systems","volume":"90","author":"Crooks","year":"1998","journal-title":"J. Stat. Phys."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"046105","DOI":"10.1103\/PhysRevE.73.046105","article-title":"Rare events and the convergence of exponentially averaged work values","volume":"73","author":"Jarzynski","year":"2006","journal-title":"Phys. Rev. E"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"180602","DOI":"10.1103\/PhysRevLett.89.180602","article-title":"Theory of a Systematic Computational Error in Free Energy Differences","volume":"89","author":"Zuckerman","year":"2002","journal-title":"Phys. Rev. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"044113","DOI":"10.1063\/1.2162874","article-title":"Equilibrium free energies from fast-switching trajectories with large time steps","volume":"124","author":"Lechner","year":"2006","journal-title":"J. Chem. Phys."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"P04001","DOI":"10.1088\/1742-5468\/2007\/04\/P04001","article-title":"On the efficiency of path sampling methods for the calculation of free energies from non-equilibrium simulations","volume":"2007","author":"Lechner","year":"2007","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"052144","DOI":"10.1103\/PhysRevE.93.052144","article-title":"Number of trials required to estimate a free-energy difference, using fluctuation relations","volume":"93","author":"Jarzynski","year":"2016","journal-title":"Phys. Rev. E"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"5974","DOI":"10.1063\/1.1353552","article-title":"A fast growth method of computing free energy differences","volume":"114","author":"Hendrix","year":"2001","journal-title":"J. Chem. Phys."},{"key":"ref_33","unstructured":"LeCun, Y., Cortes, C., and Burges, C. (2020, March 15). MNIST Handwritten Digit Database. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2721","DOI":"10.1103\/PhysRevE.60.2721","article-title":"Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences","volume":"60","author":"Crooks","year":"1999","journal-title":"Phys. Rev. E"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2361","DOI":"10.1103\/PhysRevE.61.2361","article-title":"Path-ensemble averages in systems driven far from equilibrium","volume":"61","author":"Crooks","year":"2000","journal-title":"Phys. Rev. E"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"180602","DOI":"10.1103\/PhysRevLett.100.180602","article-title":"Optimized Free Energies from Bidirectional Single-Molecule Force Spectroscopy","volume":"100","author":"Minh","year":"2008","journal-title":"Phys. Rev. Lett."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3658","DOI":"10.1073\/pnas.071034098","article-title":"Free energy reconstruction from nonequilibrium single-molecule pulling experiments","volume":"98","author":"Hummer","year":"2001","journal-title":"Proc. Natl. Acad. Sci. USA"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/5\/538\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:27:48Z","timestamp":1760174868000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/5\/538"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,11]]},"references-count":37,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,5]]}},"alternative-id":["e22050538"],"URL":"https:\/\/doi.org\/10.3390\/e22050538","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2020,5,11]]}}}