{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T11:56:23Z","timestamp":1770897383329,"version":"3.50.1"},"reference-count":20,"publisher":"MIT Press - Journals","issue":"5","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the \u201ceffective order parameter\u201d studied in the statistical mechanics literature is exactly equivalent to the infinite-width neural network gaussian process kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.<\/jats:p>","DOI":"10.1162\/neco_a_01494","type":"journal-article","created":{"date-parts":[[2022,3,28]],"date-time":"2022-03-28T23:22:00Z","timestamp":1648509720000},"page":"1136-1142","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["On Neural Network Kernels and the Storage Capacity Problem"],"prefix":"10.1162","volume":"34","author":[{"given":"Jacob A.","family":"Zavatone-Veth","sequence":"first","affiliation":[{"name":"Department of Physics and Center for Brain Science, Harvard University, Cambridge, MA 02138, U.S.A. jzavatoneveth@g.harvard.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cengiz","family":"Pehlevan","sequence":"additional","affiliation":[{"name":"Center for Brain Science and John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, U.S.A. cpehlevan@seas.harvard.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,4,15]]},"reference":[{"issue":"17","key":"2022042221320890500_B1","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.123.170602","article-title":"Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations","volume":"123","author":"Baldassi","year":"2019","journal-title":"Physical Review Letters"},{"issue":"6","key":"2022042221320890500_B2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.45.4146","article-title":"Broken symmetries in multilayered perceptrons","volume":"45","author":"Barkai","year":"1992","journal-title":"Physical Review A"},{"key":"2022042221320890500_B3","doi-asserted-by":"crossref","DOI":"10.1090\/surv\/062","volume-title":"Gaussian measures","author":"Bogachev","year":"1998"},{"key":"2022042221320890500_B4","article-title":"Kernel methods for deep learning","volume-title":"Advances in neural information processing systems","author":"Cho","year":"2009"},{"key":"2022042221320890500_B5","first-page":"2253","article-title":"Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity","volume-title":"Advances in neural information processing systems","author":"Daniely","year":"2016"},{"issue":"10","key":"2022042221320890500_B6","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.45.7590","article-title":"Storage capacity and learning algorithms for two-layer neural networks","volume":"45","author":"Engel","year":"1992","journal-title":"Physical Review A"},{"issue":"1","key":"2022042221320890500_B7","article-title":"The space of interactions in neural network models","volume":"21","author":"Gardner","year":"1988","journal-title":"Journal of Physics A: Mathematical and General"},{"issue":"1","key":"2022042221320890500_B8","article-title":"Optimal storage properties of neural network models","volume":"21","author":"Gardner","year":"1988","journal-title":"Journal of Physics A: Mathematical and General"},{"key":"2022042221320890500_B9","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139020411","volume-title":"Matrix analysis","author":"Horn","year":"2012"},{"key":"2022042221320890500_B10","author":"Hron","year":"2020","journal-title":"Exact posterior distributions of wide Bayesian neural networks"},{"key":"2022042221320890500_B11","article-title":"Neural tangent kernel: Convergence and generalization in neural networks","volume-title":"Advances in neural information processing systems","author":"Jacot","year":"2018"},{"key":"2022042221320890500_B12","article-title":"Deep neural networks as gaussian processes","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Lee","year":"2018"},{"key":"2022042221320890500_B13","article-title":"Gaussian process behaviour in wide deep neural networks","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Matthews","year":"2018"},{"key":"2022042221320890500_B14","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1007\/978-1-4612-0745-0_2","volume-title":"Bayesian learning for neural networks","author":"Neal","year":"1996"},{"issue":"2","key":"2022042221320890500_B15","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/abd485","article-title":"How isotropic kernels perform on simple invariants","volume":"2","author":"Paccolat","year":"2021","journal-title":"Machine Learning: Science and Technology"},{"key":"2022042221320890500_B16","article-title":"Effect of activation functions on the training of overparameterized neural nets","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Panigrahi","year":"2020"},{"key":"2022042221320890500_B17","first-page":"3360","article-title":"Exponential expressivity in deep neural networks through transient chaos","volume-title":"Advances in neural information processing systems, 29","author":"Poole","year":"2016"},{"key":"2022042221320890500_B18","first-page":"295","article-title":"Computing with infinite networks","volume-title":"Advances in neural information processing systems","author":"Williams","year":"1997"},{"key":"2022042221320890500_B19","volume-title":"Advances in neural information processing systems","author":"Zavatone-Veth","year":"2021"},{"key":"2022042221320890500_B20","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.103.L020301","article-title":"Activation function dependence of the storage capacity of treelike neural networks","volume":"103","author":"Zavatone-Veth","year":"2021","journal-title":"Physical Review E"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/5\/1136\/2008621\/neco_a_01494.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/5\/1136\/2008621\/neco_a_01494.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,22]],"date-time":"2022-04-22T21:33:00Z","timestamp":1650663180000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/34\/5\/1136\/110043\/On-Neural-Network-Kernels-and-the-Storage-Capacity"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,15]]},"references-count":20,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,4,15]]},"published-print":{"date-parts":[[2022,4,15]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01494","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5]]},"published":{"date-parts":[[2022,4,15]]}}}