{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T23:10:31Z","timestamp":1787008231309,"version":"build-2736575974"},"reference-count":47,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2021,5,13]],"date-time":"2021-05-13T00:00:00Z","timestamp":1620864000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,5,13]],"date-time":"2021-05-13T00:00:00Z","timestamp":1620864000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We define a neural network as a septuple consisting of (1) a state vector, (2) an input projection, (3) an output projection, (4) a weight matrix, (5) a bias vector, (6) an activation map and (7) a loss function. We argue that the loss function can be imposed either on the boundary (i.e. input and\/or output neurons) or in the bulk (i.e. hidden neurons) for both supervised and unsupervised systems. We apply the principle of maximum entropy to derive a canonical ensemble of the state vectors subject to a constraint imposed on the bulk loss function by a Lagrange multiplier (or an inverse temperature parameter). We show that in an equilibrium the canonical partition function must be a product of two factors: a function of the temperature, and a function of the bias vector and weight matrix. Consequently, the total Shannon entropy consists of two terms which represent, respectively, a thermodynamic entropy and a complexity of the neural network. We derive the first and second laws of learning: during learning the total entropy must decrease until the system reaches an equilibrium (i.e. the second law), and the increment in the loss function must be proportional to the increment in the thermodynamic entropy plus the increment in the complexity (i.e. the first law). We calculate the entropy destruction to show that the efficiency of learning is given by the Laplacian of the total free energy, which is to be maximized in an optimal neural architecture, and explain why the optimization condition is better satisfied in a deep network with a large number of hidden layers. The key properties of the model are verified numerically by training a supervised feedforward neural network using the stochastic gradient descent method. We also discuss a possibility that the entire Universe at its most fundamental level is a neural network.<\/jats:p>","DOI":"10.1088\/2632-2153\/abe6d7","type":"journal-article","created":{"date-parts":[[2021,2,16]],"date-time":"2021-02-16T17:34:23Z","timestamp":1613496863000},"page":"035012","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Toward a theory of machine learning"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9496-9212","authenticated-orcid":false,"given":"Vitaly","family":"Vanchurin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2021,5,13]]},"reference":[{"key":"mlstabe6d7bib1","article-title":"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks","author":"Saxe","year":"2014"},{"key":"mlstabe6d7bib2","article-title":"The loss surfaces of multilayer networks","volume":"vol 38","author":"Choromanska","year":"2015"},{"key":"mlstabe6d7bib3","article-title":"Optimal architectures in a solvable model of deep networks","author":"Kadmon","year":"2016"},{"key":"mlstabe6d7bib4","article-title":"Opening the black box of deep neural networks via information","author":"Shwartz-Ziv","year":"2017"},{"key":"mlstabe6d7bib5","article-title":"High-dimensional dynamics of generalization error in neural networks","author":"Advani","year":"2017"},{"key":"mlstabe6d7bib6","doi-asserted-by":"publisher","first-page":"1223","DOI":"10.1007\/s10955-017-1836-5","article-title":"Why does deep and cheap learning work so well?","volume":"168","author":"Lin","year":"2017","journal-title":"J. Stat. Phys."},{"key":"mlstabe6d7bib7","first-page":"p 396","author":"Galushkin","year":"2007"},{"key":"mlstabe6d7bib8","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: an overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Netw."},{"key":"mlstabe6d7bib9","author":"Haykin","year":"1999"},{"key":"mlstabe6d7bib10","doi-asserted-by":"publisher","first-page":"2554","DOI":"10.1073\/pnas.79.8.2554","article-title":"Neural networks and physical systems with emergent collective computational abilities","volume":"79","author":"Hopfield","year":"1982","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstabe6d7bib11","doi-asserted-by":"publisher","first-page":"3088","DOI":"10.1073\/pnas.81.10.3088","article-title":"Neurons with graded response have collective computational properties like those of two-state neurons","volume":"81","author":"Hopfield","year":"1984","journal-title":"Proc. Natl Acad. Sci"},{"key":"mlstabe6d7bib12","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.91.045002","article-title":"Machine learning and the physical sciences","volume":"91","author":"Carleo","year":"2019","journal-title":"Rev. Mod. Phys."},{"key":"mlstabe6d7bib13","doi-asserted-by":"publisher","first-page":"620","DOI":"10.1103\/PhysRev.106.620","article-title":"Information theory and statistical mechanics","volume":"106","author":"Jaynes","year":"1957","journal-title":"Phys. Rev. II"},{"key":"mlstabe6d7bib14","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1103\/PhysRev.108.171","article-title":"Information theory and statistical mechanics II","volume":"108","author":"Jaynes","year":"1957","journal-title":"Phys. Rev. II"},{"key":"mlstabe6d7bib15","article-title":"Etude thermodynamique des ph\u00e9nom\u00e9nes irr\u00e9versibles","author":"Prigogine","year":"1947"},{"key":"mlstabe6d7bib16","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1103\/PhysRev.96.250","article-title":"Principle of minimum entropy production","volume":"96","author":"Klein","year":"1954","journal-title":"Phys. Rev."},{"key":"mlstabe6d7bib17","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1007\/s10701-019-00315-6","article-title":"Entropic mechanics: towards a stochastic description of quantum mechanics","volume":"50","author":"Vanchurin","year":"2019","journal-title":"Found. Phys."},{"key":"mlstabe6d7bib18","doi-asserted-by":"publisher","DOI":"10.1142\/S0217751X18450197","article-title":"Covariant information theory and emergent gravity","volume":"33","author":"Vanchurin","year":"2018","journal-title":"Int. J. Mod. Phys. A"},{"key":"mlstabe6d7bib19","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/BF02551274","article-title":"Approximations by superpositions of sigmoidal functions","volume":"2","author":"Cybenko","year":"1989","journal-title":"Math. Control, Signals, Syst."},{"key":"mlstabe6d7bib20","first-page":"251","article-title":"Approximation capabilities of multilayer feedforward networks","volume":"4","author":"Hornik","year":"1991"},{"key":"mlstabe6d7bib21","first-page":"pp 265","article-title":"Tensorflow: a system for large- scale machine learning","author":"Abadi","year":"2016"},{"key":"mlstabe6d7bib22","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"mlstabe6d7bib23","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.116.191301","article-title":"Holographic complexity equals bulk action?","volume":"116","author":"Brown","year":"2016","journal-title":"Phys. Rev. Lett."},{"key":"mlstabe6d7bib24","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.93.086006","article-title":"Complexity, action and black holes","volume":"93","author":"Brown","year":"2016","journal-title":"Phys. Rev. D"},{"key":"mlstabe6d7bib25","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.97.086015","article-title":"Second law of quantum complexity","volume":"97","author":"Brown","year":"2018","journal-title":"Phys. Rev. D"},{"key":"mlstabe6d7bib26","doi-asserted-by":"publisher","first-page":"1210","DOI":"10.3390\/e22111210","article-title":"The world as a neural network","volume":"22","author":"Vanchurin","year":"2020","journal-title":"Entropy"},{"key":"mlstabe6d7bib27","author":"Katsnelson"},{"key":"mlstabe6d7bib28","first-page":"083C01","article-title":"Review of particle physics","volume":"2020","author":"Zyla","year":"2020","journal-title":"PTEP"},{"key":"mlstabe6d7bib29","doi-asserted-by":"crossref","first-page":"A1","DOI":"10.1051\/0004-6361\/201833880","article-title":"Planck 2018 results. I. Overview and the cosmological legacy of Planck","volume":"641","author":"Aghanim","year":"2020","journal-title":"Astron. Astrophys."},{"key":"mlstabe6d7bib30","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1023\/A:1026654312961","article-title":"The large N limit of superconformal field theories and supergravity","volume":"38","author":"Maldacena","year":"1999","journal-title":"Int. J. Theor. Phys."},{"key":"mlstabe6d7bib31","doi-asserted-by":"publisher","first-page":"253","DOI":"10.4310\/ATMP.1998.v2.n2.a2","article-title":"Anti-de Sitter space and holography","volume":"2","author":"Witten","year":"1998","journal-title":"Adv. Theor. Math. Phys."},{"key":"mlstabe6d7bib32","doi-asserted-by":"publisher","first-page":"6377","DOI":"10.1063\/1.531249","article-title":"The world as a hologram","volume":"36","author":"Susskind","year":"1995","journal-title":"J. Math. Phys."},{"key":"mlstabe6d7bib33","doi-asserted-by":"publisher","first-page":"1260","DOI":"10.1103\/PhysRevLett.75.1260","article-title":"Thermodynamics of space-time: the Einstein equation of state","volume":"75","author":"Jacobson","year":"1995","journal-title":"Phys. Rev. Lett."},{"key":"mlstabe6d7bib34","doi-asserted-by":"publisher","first-page":"029","DOI":"10.1007\/JHEP04(2011)029","article-title":"On the origin of gravity and the laws of Newton","volume":"1104","author":"Verlinde","year":"2011","journal-title":"JHEP"},{"key":"mlstabe6d7bib35","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.96.181602","article-title":"Holographic derivation of entanglement entropy from AdS\/CFT","volume":"96","author":"Ryu","year":"2006","journal-title":"Phys. Rev. Lett."},{"key":"mlstabe6d7bib36","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.86.065007","article-title":"Entanglement renormalization and holography","volume":"86","author":"Swingle","year":"2012","journal-title":"Phys. Rev. D"},{"key":"mlstabe6d7bib37","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1007\/JHEP04(2015)163","article-title":"Bulk locality and quantum error correction in AdS\/CFT","volume":"1504","author":"Almheiri","year":"2015","journal-title":"JHEP"},{"key":"mlstabe6d7bib38","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1103\/RevModPhys.29.454","article-title":"Relative state formulation of quantum mechanics","volume":"29","author":"Everett","year":"1957","journal-title":"Rev. Mod. Phys."},{"key":"mlstabe6d7bib39","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1103\/PhysRev.85.166","article-title":"A suggested interpretation of the quantum theory in terms of \u2018hidden variables\u2019 I","volume":"85","author":"Bohm","year":"1952","journal-title":"Phys. Rev."},{"key":"mlstabe6d7bib40","doi-asserted-by":"publisher","first-page":"580?590","DOI":"10.1038\/121580a0","article-title":"The quantum postulate and the recent development of atomic theory","volume":"121","author":"Bohr","year":"1928","journal-title":"Nature"},{"key":"mlstabe6d7bib41","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1103\/PhysicsPhysiqueFizika.1.195","article-title":"On the Einstein Podolsky Rosen paradox","volume":"1","author":"Bell","year":"1964","journal-title":"Physics"},{"key":"mlstabe6d7bib42","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.61.083507","article-title":"Predictability crisis in inflationary cosmology and its resolution","volume":"61","author":"Vanchurin","year":"2000","journal-title":"Phys. Rev. D"},{"key":"mlstabe6d7bib43","doi-asserted-by":"publisher","first-page":"216","DOI":"10.2307\/25470707","article-title":"Consciousness as integrated information: a provisional manifesto","volume":"215","author":"Tononi","year":"2008","journal-title":"Biol. Bull."},{"key":"mlstabe6d7bib44","doi-asserted-by":"publisher","DOI":"10.1002\/prop.201800007","article-title":"Black holes as brains: neural networks with area law entropy","volume":"66","author":"Dvali","year":"2018","journal-title":"Fortsch. Phys."},{"key":"mlstabe6d7bib45","doi-asserted-by":"publisher","first-page":"636","DOI":"10.1007\/s10701-018-0166-z","article-title":"Information graph flow: a geometric approximation of quantum and statistical systems","volume":"48","author":"Vanchurin","year":"2018","journal-title":"Found. Phys."},{"key":"mlstabe6d7bib46","first-page":"pp 347","article-title":"A quantum-classical duality and emergent space-time","author":"Vanchurin","year":"2019"},{"key":"mlstabe6d7bib47","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1103\/PhysRev.37.405","article-title":"Reciprocal relations in irreversible processes, I","volume":"37","author":"Onsager","year":"1931","journal-title":"Phys. Rev."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,1,22]],"date-time":"2022-01-22T20:12:28Z","timestamp":1642882348000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abe6d7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,13]]},"references-count":47,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,5,13]]},"published-print":{"date-parts":[[2021,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/abe6d7","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,13]]},"assertion":[{"value":"Toward a theory of machine learning","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-09-25","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-02-16","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-05-13","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}