{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T19:35:07Z","timestamp":1775936107225,"version":"3.50.1"},"reference-count":34,"publisher":"Proceedings of the National Academy of Sciences","issue":"48","license":[{"start":{"date-parts":[[2016,11,15]],"date-time":"2016-11-15T00:00:00Z","timestamp":1479168000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.pnas.org\/site\/misc\/userlicense.xhtml"}],"funder":[{"DOI":"10.13039\/501100000781","name":"EC | European Research Council","doi-asserted-by":"publisher","award":["267915"],"award-info":[{"award-number":["267915"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2016,11,29]]},"abstract":"<jats:title>Significance<\/jats:title><jats:p>Artificial neural networks are some of the most widely used tools in data science. Learning is, in principle, a hard problem in these systems, but in practice heuristic algorithms often find solutions with good generalization properties. We propose an explanation of this good performance in terms of a nonequilibrium statistical physics framework: We show that there are regions of the optimization landscape that are both robust and accessible and that their existence is crucial to achieve good performance on a class of particularly difficult learning problems. Building on these results, we introduce a basic algorithmic scheme that improves existing optimization algorithms and provides a framework for further research on learning in neural networks.<\/jats:p>","DOI":"10.1073\/pnas.1608103113","type":"journal-article","created":{"date-parts":[[2016,11,16]],"date-time":"2016-11-16T13:17:27Z","timestamp":1479302247000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":106,"title":["Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes"],"prefix":"10.1073","volume":"113","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5451-8388","authenticated-orcid":false,"given":"Carlo","family":"Baldassi","sequence":"first","affiliation":[{"name":"Department of Applied Science and Technology, Politecnico di Torino, I-10129 Torino, Italy;"},{"name":"Human Genetics Foundation-Torino, I-10126 Torino, Italy;"}]},{"given":"Christian","family":"Borgs","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, MA 02142;"}]},{"given":"Jennifer T.","family":"Chayes","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, MA 02142;"}]},{"given":"Alessandro","family":"Ingrosso","sequence":"additional","affiliation":[{"name":"Department of Applied Science and Technology, Politecnico di Torino, I-10129 Torino, Italy;"},{"name":"Human Genetics Foundation-Torino, I-10126 Torino, Italy;"}]},{"given":"Carlo","family":"Lucibello","sequence":"additional","affiliation":[{"name":"Department of Applied Science and Technology, Politecnico di Torino, I-10129 Torino, Italy;"},{"name":"Human Genetics Foundation-Torino, I-10126 Torino, Italy;"}]},{"given":"Luca","family":"Saglietti","sequence":"additional","affiliation":[{"name":"Department of Applied Science and Technology, Politecnico di Torino, I-10129 Torino, Italy;"},{"name":"Human Genetics Foundation-Torino, I-10126 Torino, Italy;"}]},{"given":"Riccardo","family":"Zecchina","sequence":"additional","affiliation":[{"name":"Department of Applied Science and Technology, Politecnico di Torino, I-10129 Torino, Italy;"},{"name":"Human Genetics Foundation-Torino, I-10126 Torino, Italy;"},{"name":"Collegio Carlo Alberto, I-10024 Moncalieri, Italy"}]}],"member":"341","published-online":{"date-parts":[[2016,11,15]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun Y","year":"2015","unstructured":"Y LeCun, Y Bengio, G Hinton, Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"e_1_3_4_2_2","unstructured":"J Ngiam On optimization methods for deep learning. Proceedings of the 28th International Conference on Machine Learning (ICML-11) (International Machine Learning Society) pp 265\u2013272. (2011)."},{"key":"e_1_3_4_3_2","doi-asserted-by":"crossref","first-page":"3725","DOI":"10.1038\/ncomms4725","article-title":"Fractal free energy landscapes in structural glasses","volume":"5","author":"Charbonneau P","year":"2014","unstructured":"P Charbonneau, J Kurchan, G Parisi, P Urbani, F Zamponi, Fractal free energy landscapes in structural glasses. Nat Commun 5, 3725 (2014).","journal-title":"Nat Commun"},{"key":"e_1_3_4_4_2","doi-asserted-by":"crossref","first-page":"P09001","DOI":"10.1088\/1742-5468\/2009\/09\/P09001","article-title":"On the cavity method for decimated random constraint satisfaction problems and the analysis of belief propagation guided decimation algorithms","volume":"2009","author":"Ricci-Tersenghi F","year":"2009","unstructured":"F Ricci-Tersenghi, G Semerjian, On the cavity method for decimated random constraint satisfaction problems and the analysis of belief propagation guided decimation algorithms. J Stat Mech Theor Exp 2009, P09001 (2009).","journal-title":"J Stat Mech Theor Exp"},{"key":"e_1_3_4_5_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-08488-6","volume-title":"Stochastic Processes in Cell Biology","author":"Bressloff PC","year":"2014","unstructured":"PC Bressloff Stochastic Processes in Cell Biology (Springer, Berlin) Vol 41 (2014)."},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511761942","volume-title":"Networks, Crowds, and Markets: Reasoning About a Highly Connected World","author":"Easley D","year":"2010","unstructured":"D Easley, J Kleinberg Networks, Crowds, and Markets: Reasoning About a Highly Connected World (Cambridge Univ Press, Cambridge, UK, 2010)."},{"key":"e_1_3_4_7_2","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1038\/nrn2699","article-title":"Experience-dependent structural synaptic plasticity in the mammalian brain","volume":"10","author":"Holtmaat A","year":"2009","unstructured":"A Holtmaat, K Svoboda, Experience-dependent structural synaptic plasticity in the mammalian brain. Nat Rev Neurosci 10, 647\u2013658 (2009).","journal-title":"Nat Rev Neurosci"},{"key":"e_1_3_4_8_2","first-page":"685","volume-title":"Advances in Neural Information Processing Systems 28","author":"Zhang S","year":"2015","unstructured":"S Zhang, AE Choromanska, Y LeCun, Deep learning with elastic averaging SGD. Advances in Neural Information Processing Systems 28, eds C Cortes, ND Lawrence, DD Lee, M Sugiyama, R Garnett (Curran Associates, Red Hook, NY), pp. 685\u2013693 (2015)."},{"key":"e_1_3_4_9_2","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by simmulated annealing","volume":"220","author":"Kirkpatrick S","year":"1983","unstructured":"S Kirkpatrick, Jr CD Gelatt, MP Vecchi, Optimization by simmulated annealing. Science 220, 671\u2013680 (1983).","journal-title":"Science"},{"key":"e_1_3_4_10_2","doi-asserted-by":"crossref","first-page":"812","DOI":"10.1126\/science.1073287","article-title":"Analytic and algorithmic solution of random satisfiability problems","volume":"297","author":"M\u00e9zard M","year":"2002","unstructured":"M M\u00e9zard, G Parisi, R Zecchina, Analytic and algorithmic solution of random satisfiability problems. Science 297, 812\u2013815 (2002).","journal-title":"Science"},{"key":"e_1_3_4_11_2","doi-asserted-by":"crossref","first-page":"10318","DOI":"10.1073\/pnas.0703685104","article-title":"Gibbs states and the set of solutions of random constraint satisfaction problems","volume":"104","author":"Krzakala F","year":"2007","unstructured":"F Krzakala, A Montanari, F Ricci-Tersenghi, G Semerjian, L Zdeborova, Gibbs states and the set of solutions of random constraint satisfaction problems. Proc Natl Acad Sci USA 104, 10318\u201310323 (2007).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_12_2","doi-asserted-by":"crossref","first-page":"078702","DOI":"10.1103\/PhysRevLett.101.078702","article-title":"Locked constraint satisfaction problems","volume":"101","author":"Zdeborov\u00e1 L","year":"2008","unstructured":"L Zdeborov\u00e1, M M\u00e9zard, Locked constraint satisfaction problems. Phys Rev Lett 101, 078702 (2008).","journal-title":"Phys Rev Lett"},{"key":"e_1_3_4_13_2","doi-asserted-by":"crossref","first-page":"128101","DOI":"10.1103\/PhysRevLett.115.128101","article-title":"Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses","volume":"115","author":"Baldassi C","year":"2015","unstructured":"C Baldassi, A Ingrosso, C Lucibello, L Saglietti, R Zecchina, Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses. Phys Rev Lett 115, 128101 (2015).","journal-title":"Phys Rev Lett"},{"key":"e_1_3_4_14_2","doi-asserted-by":"crossref","first-page":"052813","DOI":"10.1103\/PhysRevE.90.052813","article-title":"Origin of the computational hardness for learning with binary synapses","volume":"90","author":"Huang H","year":"2014","unstructured":"H Huang, Y Kabashima, Origin of the computational hardness for learning with binary synapses. Phys Rev E Stat Nonlin Soft Matter Phys. 90, 052813 (2014).","journal-title":"Phys Rev E Stat Nonlin Soft Matter Phys."},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","first-page":"P023301","DOI":"10.1088\/1742-5468\/2016\/02\/023301","article-title":"Local entropy as a measure for sampling solutions in constraint satisfaction problems","volume":"2016","author":"Baldassi C","year":"2016","unstructured":"C Baldassi, A Ingrosso, C Lucibello, L Saglietti, R Zecchina, Local entropy as a measure for sampling solutions in constraint satisfaction problems. J Stat Mech Theor Exp 2016, P023301 (2016).","journal-title":"J Stat Mech Theor Exp"},{"key":"e_1_3_4_16_2","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780198570837.001.0001","volume-title":"Information, Physics, and Computation","author":"M\u00e9zard M","year":"2009","unstructured":"M M\u00e9zard, A Montanari Information, Physics, and Computation (Oxford Univ Press, New York, 2009)."},{"key":"e_1_3_4_17_2","doi-asserted-by":"crossref","first-page":"052313","DOI":"10.1103\/PhysRevE.93.052313","article-title":"Learning may need only a few bits of synaptic precision","volume":"93","author":"Baldassi C","year":"2016","unstructured":"C Baldassi, F Gerace, C Lucibello, L Saglietti, R Zecchina, Learning may need only a few bits of synaptic precision. Phys Rev E 93, 052313 (2016).","journal-title":"Phys Rev E"},{"key":"e_1_3_4_18_2","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780199233212.001.0001","volume-title":"The Nature of Computation","author":"Moore C","year":"2011","unstructured":"C Moore, S Mertens The Nature of Computation (Oxford Univ Press, New York, 2011)."},{"key":"e_1_3_4_19_2","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart DE","year":"1988","unstructured":"DE Rumelhart, GE Hinton, RJ Williams, Learning representations by back-propagating errors. Nature 323, 533\u2013536 (1988).","journal-title":"Nature"},{"key":"e_1_3_4_20_2","unstructured":"S Hochreiter Untersuchungen zu dynamischen neuronalen netzen. Master\u2019s thesis (Institut fur Informatik Technische Universitat Munich). (1991)."},{"key":"e_1_3_4_21_2","doi-asserted-by":"crossref","first-page":"11079","DOI":"10.1073\/pnas.0700324104","article-title":"Efficient supervised learning in networks with binary synapses","volume":"104","author":"Baldassi C","year":"2007","unstructured":"C Baldassi, A Braunstein, N Brunel, R Zecchina, Efficient supervised learning in networks with binary synapses. Proc Natl Acad Sci USA 104, 11079\u201311084 (2007).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","first-page":"902","DOI":"10.1007\/s10955-009-9822-1","article-title":"Generalization learning in a perceptron with binary synapses","volume":"136","author":"Baldassi C","year":"2009","unstructured":"C Baldassi, Generalization learning in a perceptron with binary synapses. J Stat Phys 136, 902\u2013916 (2009).","journal-title":"J Stat Phys"},{"key":"e_1_3_4_23_2","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun Y","year":"1998","unstructured":"Y LeCun, L Bottou, Y Bengio, P Haffner, Gradient-based learning applied to document recognition. Proc IEEE 86, 2278\u20132324 (1998).","journal-title":"Proc IEEE"},{"key":"e_1_3_4_24_2","first-page":"3105","volume-title":"Advances in Neural Information Processing Systems 28","author":"Courbariaux M","year":"2015","unstructured":"M Courbariaux, Y Bengio, JP David, Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems 28, eds C Cortes, ND Lawrence, DD Lee, M Sugiyama, R Garnett (Curran Associates, Red Hook, NY), pp. 3105\u20133113 (2015)."},{"key":"e_1_3_4_25_2","unstructured":"Courbariaux I Matthieu Hubara D Soudry R El-Yaniv Y Bengio Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv:1602.02830. (2016)."},{"key":"e_1_3_4_26_2","unstructured":"S Zhang Distributed stochastic optimization for deep learning. Ph.D. thesis (New York University New York). arXiv:1605.02216. (2016)."},{"key":"e_1_3_4_27_2","volume-title":"Information Theory, Inference and Learning Algorithms","author":"MacKay DJ","year":"2003","unstructured":"DJ MacKay Information Theory, Inference and Learning Algorithms (Cambridge Univ Press, New York, 2003)."},{"key":"e_1_3_4_28_2","doi-asserted-by":"crossref","first-page":"2282","DOI":"10.1109\/TIT.2005.850085","article-title":"Constructing free-energy approximations and generalized belief propagation algorithms","volume":"51","author":"Yedidia JS","year":"2005","unstructured":"JS Yedidia, WT Freeman, Y Weiss, Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Trans Inform Theor 51, 2282\u20132312 (2005).","journal-title":"IEEE Trans Inform Theor"},{"key":"e_1_3_4_29_2","doi-asserted-by":"crossref","first-page":"030201","DOI":"10.1103\/PhysRevLett.96.030201","article-title":"Learning by message-passing in neural networks with material synapses","volume":"96","author":"Braunstein A","year":"2006","unstructured":"A Braunstein, R Zecchina, Learning by message-passing in neural networks with material synapses. Phys Rev Lett 96, 030201 (2006).","journal-title":"Phys Rev Lett"},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1073\/pnas.1004751108","article-title":"Finding undetected protein associations in cell signaling by belief propagation","volume":"108","author":"Bailly-Bechet M","year":"2011","unstructured":"M Bailly-Bechet, , Finding undetected protein associations in cell signaling by belief propagation. Proc Natl Acad Sci USA 108, 882\u2013887 (2011).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","first-page":"2133","DOI":"10.1143\/JPSJ.74.2133","article-title":"Replicated bethe free energy: A variational principle behind survey propagation","volume":"74","author":"Kabashima Y","year":"2005","unstructured":"Y Kabashima, Replicated bethe free energy: A variational principle behind survey propagation. J Phys Soc Jpn 74, 2133\u20132136 (2005).","journal-title":"J Phys Soc Jpn"},{"key":"e_1_3_4_32_2","doi-asserted-by":"crossref","first-page":"053401","DOI":"10.1088\/1742-5468\/2016\/05\/053401","article-title":"The large deviations of the whitening process in random constraint satisfaction problems","volume":"2016","author":"Braunstein A","year":"2016","unstructured":"A Braunstein, L Dall\u2019Asta, G Semerjian, L Zdeborov\u00e1, The large deviations of the whitening process in random constraint satisfaction problems. J Stat Mech Theor Exp 2016, 053401 (2016).","journal-title":"J Stat Mech Theor Exp"},{"key":"e_1_3_4_33_2","doi-asserted-by":"crossref","unstructured":"R Marino G Parisi F Ricci-Tersenghi The backtracking survey propagation algorithm for solving random K-SAT problems. arXiv:1508.05117. (2015).","DOI":"10.1038\/ncomms12996"},{"key":"e_1_3_4_34_2","doi-asserted-by":"crossref","first-page":"031118","DOI":"10.1103\/PhysRevE.77.031118","article-title":"Entropy landscape and non-gibbs solutions in constraint satisfaction problems","volume":"77","author":"Dall\u2019Asta L","year":"2008","unstructured":"L Dall\u2019Asta, A Ramezanpour, R Zecchina, Entropy landscape and non-gibbs solutions in constraint satisfaction problems. Phys Rev E 77, 031118 (2008).","journal-title":"Phys Rev E"}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.pnas.org\/syndication\/doi\/10.1073\/pnas.1608103113","content-type":"unspecified","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.1608103113","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,20]],"date-time":"2023-08-20T22:54:43Z","timestamp":1692572083000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.1608103113"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,11,15]]},"references-count":34,"journal-issue":{"issue":"48","published-print":{"date-parts":[[2016,11,29]]}},"alternative-id":["10.1073\/pnas.1608103113"],"URL":"https:\/\/doi.org\/10.1073\/pnas.1608103113","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,11,15]]},"assertion":[{"value":"2016-11-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}