{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T05:58:54Z","timestamp":1771912734168,"version":"3.50.1"},"reference-count":63,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2022,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Message-passing algorithms based on the belief propagation (BP) equations constitute a well-known distributed computational scheme. They yield exact marginals on tree-like graphical models and have also proven to be effective in many problems defined on loopy graphs, from inference to optimization, from signal processing to clustering. The BP-based schemes are fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present and adapt to mini-batch training on GPUs a family of BP-based message-passing algorithms with a reinforcement term that biases distributions towards locally entropic solutions. These algorithms are capable of training multi-layer neural networks with performance comparable to SGD heuristics in a diverse set of experiments on natural datasets including multi-class image classification and continual learning, while being capable of yielding improved performances on sparse networks. Furthermore, they allow to make approximate Bayesian predictions that have higher accuracy than point-wise ones.<\/jats:p>","DOI":"10.1088\/2632-2153\/ac7d3b","type":"journal-article","created":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T22:31:30Z","timestamp":1656541890000},"page":"035005","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Deep learning via message passing algorithms based on belief propagation"],"prefix":"10.1088","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0837-9783","authenticated-orcid":true,"given":"Carlo","family":"Lucibello","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fabrizio","family":"Pittorino","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gabriele","family":"Perugini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Riccardo","family":"Zecchina","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2022,7,15]]},"reference":[{"key":"mlstac7d3bbib1","article-title":"Tullio.jl julia package","author":"Abbott","year":"2021"},{"key":"mlstac7d3bbib2","first-page":"pp 139","article-title":"Memory aware synapses: learning what (not) to forget","author":"Aljundi","year":"2018"},{"key":"mlstac7d3bbib3","article-title":"Sparsely-connected neural networ VLSI implementation of deep neural networks","author":"Ardakani","year":"2017"},{"key":"mlstac7d3bbib4","doi-asserted-by":"publisher","first-page":"1156","DOI":"10.1109\/TIT.2020.3033985","article-title":"The spiked matrix model with generative priors","volume":"67","author":"Aubin","year":"2021","journal-title":"IEEE Trans. Inf. Theory"},{"key":"mlstac7d3bbib5","doi-asserted-by":"publisher","first-page":"E7655","DOI":"10.1073\/pnas.1608103113","article-title":"Unreasonable effectiveness of learning neural networks: from accessible states and robust ensembles to basic algorithmic schemes","volume":"113","author":"Baldassi","year":"2016","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib6","doi-asserted-by":"publisher","first-page":"11079","DOI":"10.1073\/pnas.0700324104","article-title":"Efficient supervised learning in networks with binary synapses","volume":"104","author":"Baldassi","year":"2007","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib7","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.93.052313","article-title":"Learning may need only a few bits of synaptic precision","volume":"93","author":"Baldassi","year":"2016","journal-title":"Phys. Rev. E"},{"key":"mlstac7d3bbib8","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.115.128101","article-title":"Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses","volume":"115","author":"Baldassi","year":"2015","journal-title":"Phys. Rev. Lett."},{"key":"mlstac7d3bbib9","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1073\/pnas.1908636117","article-title":"Shaping the learning landscape in neural networks around wide flat minima","volume":"117","author":"Baldassi","year":"2020","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib10","doi-asserted-by":"publisher","first-page":"5451","DOI":"10.1073\/pnas.1802705116","article-title":"Optimal errors and phase transitions in high-dimensional generalized linear models","volume":"116","author":"Barbier","year":"2019","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib11","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1098\/rspa.1935.0122","article-title":"Statistical theory of superlattices","volume":"150","author":"Bethe","year":"1935","journal-title":"Proc. R. Soc. A"},{"key":"mlstac7d3bbib12","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.96.030201","article-title":"Learning by message passing in networks of discrete synapses","volume":"96","author":"Braunstein","year":"2006","journal-title":"Phys. Rev. Lett."},{"key":"mlstac7d3bbib13","article-title":"Entropy-sgd: biasing gradient descent into wide valleys","author":"Chaudhari","year":"2017"},{"key":"mlstac7d3bbib14","article-title":"Multi-prize lottery ticket hypothesis: finding accurate binary neural networks by pruning a randomly weighted network","author":"Diffenderfer","year":"2021"},{"key":"mlstac7d3bbib15","doi-asserted-by":"publisher","first-page":"18914","DOI":"10.1073\/pnas.0909892106","article-title":"Message-passing algorithms for compressed sensing","volume":"106","author":"Donoho","year":"2009","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib16","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2015617118","article-title":"The inverse variance\u2013flatness relation in stochastic gradient descent is critical for finding flat minima","volume":"118","author":"Feng","year":"2021","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib17","first-page":"pp 1884","article-title":"Inference in deep networks in high dimensions","author":"Fletcher","year":"2018"},{"key":"mlstac7d3bbib18","article-title":"Pruning neural networks at initialization: why are we missing the mark?","author":"Frankle","year":"2021"},{"key":"mlstac7d3bbib19","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1016\/j.neuron.2005.02.001","article-title":"Cascade models of synaptically stored memories","volume":"45","author":"Fusi","year":"2005","journal-title":"Neuron"},{"key":"mlstac7d3bbib20","doi-asserted-by":"publisher","DOI":"10.1088\/1751-8121\/ab7f65","article-title":"Mean-field inference methods for neural networks","volume":"53","author":"Gabri\u00e9","year":"2020","journal-title":"J. Phys. A: Math. Theor."},{"key":"mlstac7d3bbib21","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/ab3430","article-title":"Entropy and mutual information in models of deep neural networks","volume":"2019","author":"Gabrie","year":"2019","journal-title":"J. Stat. Mech."},{"key":"mlstac7d3bbib22","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1109\/TIT.1962.1057683","article-title":"Low-density parity-check codes","volume":"8","author":"Gallager","year":"1962","journal-title":"IRE Trans. Inf. Theory"},{"key":"mlstac7d3bbib23","article-title":"Loss surfaces, mode connectivity and fast ensembling of dnns","volume":"vol 31","author":"Garipov","year":"2018"},{"key":"mlstac7d3bbib24","first-page":"pp 249","article-title":"Understanding the difficulty of training deep feedforward neural networks","volume":"vol 9","author":"Glorot","year":"2010"},{"key":"mlstac7d3bbib25","doi-asserted-by":"publisher","DOI":"10.1103\/physrevx.10.041044","article-title":"Modeling the influence of data structure on learning in neural networks: the hidden manifold model","volume":"10","author":"Goldt","year":"2020","journal-title":"Phys. Rev. X"},{"key":"mlstac7d3bbib26","article-title":"An empirical investigation of catastrophic forgetting in gradient-based neural networks","author":"Goodfellow","year":"2013"},{"key":"mlstac7d3bbib27","article-title":"Deep compression: compressing deep neural network with pruning, trained quantization and huffman coding","author":"Han","year":"2016"},{"key":"mlstac7d3bbib28","article-title":"Probabilistic backpropagation for scalable learning of bayesian neural networks","volume":"vol 37","author":"Hern\u00e1ndez-Lobato","year":"2015"},{"key":"mlstac7d3bbib29","article-title":"Binarized neural networks","volume":"vol 29","author":"Hubara","year":"2016"},{"key":"mlstac7d3bbib30","article-title":"Fantastic generalization measures and where to find them","author":"Jiang","year":"2020"},{"key":"mlstac7d3bbib31","doi-asserted-by":"publisher","first-page":"4228","DOI":"10.1109\/TIT.2016.2556702","article-title":"Phase transitions and sample complexity in bayes-optimal matrix factorization","volume":"62","author":"Kabashima","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"mlstac7d3bbib32","doi-asserted-by":"publisher","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"Kirkpatrick","year":"2017","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstac7d3bbib33","first-page":"pp 667","article-title":"Belief propagation neural networks","volume":"vol 33","author":"Kuck","year":"2020"},{"key":"mlstac7d3bbib34","doi-asserted-by":"publisher","first-page":"2549","DOI":"10.1038\/s41467-021-22768-y","article-title":"Synaptic metaplasticity in binarized neural networks","volume":"12","author":"Laborieux","year":"2021","journal-title":"Nat. Commun."},{"key":"mlstac7d3bbib35","article-title":"Visualizing the loss landscape of neural nets","volume":"vol 31","author":"Li","year":"2018"},{"key":"mlstac7d3bbib36","first-page":"pp 6936","article-title":"How do adam and training strategies help bnns optimization","volume":"vol 139","author":"Liu","year":"2021"},{"key":"mlstac7d3bbib37","article-title":"Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising","author":"Maillard","year":"2021"},{"key":"mlstac7d3bbib38","first-page":"pp 2098","article-title":"Multi-layer generalized linear estimation","author":"Manoel","year":"2017"},{"key":"mlstac7d3bbib39","first-page":"pp 1048","article-title":"Streaming bayesian inference: theoretical limits and mini-batch approximate message-passing","author":"Manoel","year":"2017"},{"key":"mlstac7d3bbib40","first-page":"pp 109","volume":"vol 24","author":"McCloskey","year":"1989"},{"key":"mlstac7d3bbib41","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.95.022117","article-title":"Mean-field message-passing equations in the hopfield model and its generalizations","volume":"95","author":"M\u00e9zard","year":"2017","journal-title":"Phys. Rev. E"},{"key":"mlstac7d3bbib42","author":"M\u00e9zard","year":"2009"},{"key":"mlstac7d3bbib43","volume":"vol 9","author":"M\u00e9zard","year":"1987"},{"key":"mlstac7d3bbib44","first-page":"pp 362","article-title":"Expectation propagation for approximate bayesian inference","author":"Minka","year":"2001"},{"key":"mlstac7d3bbib45","article-title":"Bilinear generalized approximate message passing","author":"Parker","year":"2013"},{"key":"mlstac7d3bbib46","doi-asserted-by":"publisher","first-page":"5839","DOI":"10.1109\/TSP.2014.2357776","article-title":"Bilinear generalized approximate message passing-part I: derivation","volume":"62","author":"Parker","year":"2014","journal-title":"IEEE Trans. Signal Process."},{"key":"mlstac7d3bbib47","article-title":"Reverend Bayes on inference engines: a distributed hierarchical approach","author":"Pearl","year":"1982"},{"key":"mlstac7d3bbib48","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1017\/S0305004100019174","article-title":"On ising\u2019s model of ferromagnetism","volume":"32","author":"Peierls","year":"1936","journal-title":"Math. Proc. Camb. Phil. Soc."},{"key":"mlstac7d3bbib49","doi-asserted-by":"crossref","DOI":"10.1088\/1742-5468\/ac3ae8","article-title":"Entropic gradient descent algorithms and wide flat minima","author":"Pittorino","year":"2021"},{"key":"mlstac7d3bbib50","doi-asserted-by":"publisher","first-page":"6664","DOI":"10.1109\/TIT.2019.2916359","article-title":"Vector approximate message passing","volume":"65","author":"Rangan","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"mlstac7d3bbib51","first-page":"pp 239","author":"Rao","year":"2007"},{"key":"mlstac7d3bbib52","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1080\/09540099550039318","article-title":"Catastrophic forgetting, rehearsal and pseudorehearsal","volume":"7","author":"Robins","year":"1995","journal-title":"Connect. Sci."},{"key":"mlstac7d3bbib53","first-page":"pp 685","article-title":"Neural enhanced belief propagation on factor graphs","author":"Satorras","year":"2021"},{"key":"mlstac7d3bbib54","first-page":"p 2","article-title":"Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights","volume":"vol 1","author":"Soudry","year":"2014"},{"key":"mlstac7d3bbib55","article-title":"Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights","volume":"vol 27","author":"Soudry","year":"2014"},{"key":"mlstac7d3bbib56","article-title":"Critical initialisation in continuous approximations of binary neural networks","author":"Stamatescu","year":"2020"},{"key":"mlstac7d3bbib57","article-title":"Training neural networks with fixed sparse masks","author":"Sung","year":"2021"},{"key":"mlstac7d3bbib58","first-page":"pp 7873","article-title":"Clip-q: deep network compression learning by in-parallel pruning-quantization","author":"Tung","year":"2018"},{"key":"mlstac7d3bbib59","article-title":"Deterministic variational inference for robust bayesian neural networks","author":"Wu","year":"2018"},{"key":"mlstac7d3bbib60","first-page":"pp 239","author":"Yedidia","year":"2003"},{"key":"mlstac7d3bbib61","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1080\/00018732.2016.1211393","article-title":"Statistical physics of inference: thresholds and algorithms","volume":"65","author":"Zdeborov\u00e1","year":"2016","journal-title":"Adv. Phys."},{"key":"mlstac7d3bbib62","first-page":"pp 3987","article-title":"Continual learning through synaptic intelligence","author":"Zenke","year":"2017"},{"key":"mlstac7d3bbib63","doi-asserted-by":"publisher","first-page":"4529","DOI":"10.1109\/TSP.2021.3100305","article-title":"Multi-layer bilinear generalized approximate message passing","volume":"69","author":"Zou","year":"2021","journal-title":"IEEE Trans. Signal Process."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T08:38:14Z","timestamp":1657874294000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac7d3b"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,15]]},"references-count":63,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,7,15]]},"published-print":{"date-parts":[[2022,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ac7d3b","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,15]]},"assertion":[{"value":"Deep learning via message passing algorithms based on belief propagation","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2022 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2022-01-29","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-06-29","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-07-15","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}