{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T03:17:10Z","timestamp":1775099830369,"version":"3.50.1"},"reference-count":41,"publisher":"Proceedings of the National Academy of Sciences","issue":"16","license":[{"start":{"date-parts":[[2019,3,29]],"date-time":"2019-03-29T00:00:00Z","timestamp":1553817600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2019,4,16]]},"abstract":"<jats:p>It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb\u2019s idea that change of the synapse strength should be local\u2014i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.<\/jats:p>","DOI":"10.1073\/pnas.1820458116","type":"journal-article","created":{"date-parts":[[2019,3,30]],"date-time":"2019-03-30T00:15:30Z","timestamp":1553904930000},"page":"7723-7731","update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":126,"title":["Unsupervised learning by competing hidden units"],"prefix":"10.1073","volume":"116","author":[{"given":"Dmitry","family":"Krotov","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology\u2013International Business Machines (IBM) Watson Artificial Intelligence Laboratory, IBM Research, Cambridge, MA 02142;"},{"name":"Institute for Advanced Study, Princeton, NJ 08540;"}]},{"given":"John J.","family":"Hopfield","sequence":"additional","affiliation":[{"name":"Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544"}]}],"member":"341","published-online":{"date-parts":[[2019,3,29]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun Y","year":"2015","unstructured":"Y LeCun, Y Bengio, G Hinton, Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"e_1_3_4_2_2","first-page":"818","volume-title":"European Conference on Computer Vision","author":"Zeiler MD","year":"2014","unstructured":"MD Zeiler, R Fergus, Visualizing and understanding convolutional networks. European Conference on Computer Vision, eds D Fleet, T Pajdla, B Schiele, T Tuytelaars (Springer, Cham, Switzerland), pp. 818\u2013833 (2014)."},{"key":"e_1_3_4_3_2","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton GE","year":"2006","unstructured":"GE Hinton, RR Salakhutdinov, Reducing the dimensionality of data with neural networks. Science 313, 504\u2013507 (2006).","journal-title":"Science"},{"key":"e_1_3_4_4_2","unstructured":"TP Lillicrap D Cownden DB Tweed CJ Akerman Random feedback weights support learning in deep neural networks. arXiv:1411.0247. Preprint posted November 2 2014. (2014)."},{"key":"e_1_3_4_5_2","doi-asserted-by":"crossref","first-page":"13276","DOI":"10.1038\/ncomms13276","article-title":"Random synaptic feedback weights support error backpropagation for deep learning","volume":"7","author":"Lillicrap TP","year":"2016","unstructured":"TP Lillicrap, D Cownden, DB Tweed, CJ Akerman, Random synaptic feedback weights support error backpropagation for deep learning. Nat Commun 7, 13276 (2016).","journal-title":"Nat Commun"},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","unstructured":"DH Lee S Zhang A Fischer Y Bengio Difference target propagation. Joint European Conference on Machine Learning and Knowledge Discovery in Databases eds Appice A. et al. (Springer Cham Switzerland) pp 498\u2013515. (2015).","DOI":"10.1007\/978-3-319-23528-8_31"},{"key":"e_1_3_4_7_2","doi-asserted-by":"crossref","first-page":"24","DOI":"10.3389\/fncom.2017.00024","article-title":"Equilibrium propagation: Bridging the gap between energy-based models and backpropagation","volume":"11","author":"Scellier B","year":"2017","unstructured":"B Scellier, Y Bengio, Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Front Comput Neurosci 11, 24 (2017).","journal-title":"Front Comput Neurosci"},{"key":"e_1_3_4_8_2","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1162\/NECO_a_00949","article-title":"An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity","volume":"29","author":"Whittington JC","year":"2017","unstructured":"JC Whittington, R Bogacz, An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Comput 29, 1229\u20131262 (2017).","journal-title":"Neural Comput"},{"key":"e_1_3_4_9_2","unstructured":"A Choromanska Beyond backprop: Alternating minimization with co-activation memory. arXiv:1806.09077. Preprint posted February 1 2019. (2018)."},{"key":"e_1_3_4_10_2","unstructured":"Y Bengio Deep learning and backprop in the brain (online video). Available at https:\/\/www.youtube.com\/watch?v=FhRW77rZUS8. Accessed October 13 2017. (2017)."},{"key":"e_1_3_4_11_2","unstructured":"G Hinton Can sensory cortex do backpropagation? (online video). Available at https:\/\/www.youtube.com\/watch?v=cBLk5baHbZ8. Accessed April 16 2016. (2016)."},{"key":"e_1_3_4_12_2","unstructured":"J Sacramento RP Costa Y Bengio W Senn Dendritic error backpropagation in deep cortical microcircuits. arXiv:1801.00062. Preprint posted December 30 2017. (2017)."},{"key":"e_1_3_4_13_2","doi-asserted-by":"crossref","first-page":"1461","DOI":"10.1162\/NECO_a_00745","article-title":"A Hebbian\/anti-Hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data","volume":"27","author":"Pehlevan C","year":"2015","unstructured":"C Pehlevan, T Hu, DB Chklovskii, A Hebbian\/anti-Hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data. Neural Comput 27, 1461\u20131495 (2015).","journal-title":"Neural Comput"},{"key":"e_1_3_4_14_2","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1162\/neco_a_01018","article-title":"Why do similarity matching objectives lead to Hebbian\/anti-Hebbian networks?","volume":"30","author":"Pehlevan C","year":"2018","unstructured":"C Pehlevan, AM Sengupta, DB Chklovskii, Why do similarity matching objectives lead to Hebbian\/anti-Hebbian networks? Neural Comput 30, 84\u2013124 (2018).","journal-title":"Neural Comput"},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","first-page":"769","DOI":"10.1109\/ACSSC.2014.7094553","volume-title":"2014 48th Asilomar Conference on Signals, Systems and Computers","author":"Pehlevan C","year":"2014","unstructured":"C Pehlevan, DB Chklovskii, A Hebbian\/anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features. 2014 48th Asilomar Conference on Signals, Systems and Computers, ed MB Matthews (IEEE, New York), pp. 769\u2013775 (2014)."},{"key":"e_1_3_4_16_2","doi-asserted-by":"crossref","unstructured":"A Sengupta C Pehlevan M Tepper A Genkin D Chklovskii Manifold-tiling localized receptive fields are optimal in similarity-preserving neural networks. Advances in Neural Information Processing Systems eds Bengio S et al. (Neural Information Processing Systems Foundation Inc. La Jolla CA) pp 7077\u20137087. (2018).","DOI":"10.1101\/338947"},{"key":"e_1_3_4_17_2","first-page":"62","volume-title":"The Organization of Behavior: A Neurophysiological Approach","author":"Hebb DO","year":"1949","unstructured":"DO Hebb The Organization of Behavior: A Neurophysiological Approach (Wiley, New York), pp. 62 (1949)."},{"key":"e_1_3_4_18_2","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1098\/rsnr.1976.0015","article-title":"From electrical to chemical transmission in the central nervous system","volume":"30","author":"Eccles J","year":"1976","unstructured":"J Eccles, From electrical to chemical transmission in the central nervous system. Notes Rec R Soc Lond 30, 219\u2013230 (1976).","journal-title":"Notes Rec R Soc Lond"},{"key":"e_1_3_4_19_2","volume-title":"Biophysics of Computation: Information Processing in Single Neurons","author":"Koch C","year":"2004","unstructured":"C Koch Biophysics of Computation: Information Processing in Single Neurons (Oxford Univ Press, New York, 2004)."},{"key":"e_1_3_4_20_2","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107447615","volume-title":"Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition","author":"Gerstner W","year":"2014","unstructured":"W Gerstner, WM Kistler, R Naud, L Paninski Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition (Cambridge Univ Press, Cambridge, UK, 2014)."},{"key":"e_1_3_4_21_2","doi-asserted-by":"crossref","first-page":"4732","DOI":"10.1073\/pnas.95.8.4732","article-title":"All-or-none potentiation at CA3-CA1 synapses","volume":"95","author":"Petersen CC","year":"1998","unstructured":"CC Petersen, RC Malenka, RA Nicoll, JJ Hopfield, All-or-none potentiation at CA3-CA1 synapses. Proc Natl Acad Sci USA 95, 4732\u20134737 (1998).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","DOI":"10.1201\/9781317553830","volume-title":"Principles of Neurobiology","author":"Luo L","year":"2015","unstructured":"L Luo Principles of Neurobiology (Garland Science, Taylor & Francis Group, LLC, New York, 2015)."},{"key":"e_1_3_4_23_2","first-page":"1172","volume-title":"Advances in Neural Information Processing Systems","author":"Krotov D","year":"2016","unstructured":"D Krotov, JJ Hopfield, Dense associative memory for pattern recognition. Advances in Neural Information Processing Systems, eds DD Lee, M Sugiyama, UV Luxburg, I Guyon, R Garnett (Neural Information Processing Systems Foundation, Inc., La Jolla, CA), pp. 1172\u20131180 (2016)."},{"key":"e_1_3_4_24_2","doi-asserted-by":"crossref","first-page":"3151","DOI":"10.1162\/neco_a_01143","article-title":"Dense associative memory is robust to adversarial inputs","volume":"30","author":"Krotov D","year":"2018","unstructured":"D Krotov, J Hopfield, Dense associative memory is robust to adversarial inputs. Neural Comput 30, 3151\u20133167 (2018).","journal-title":"Neural Comput"},{"key":"e_1_3_4_25_2","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/BF00288907","article-title":"Self-organization of orientation sensitive cells in the striate cortex","volume":"14","author":"Von der Malsburg C","year":"1973","unstructured":"C Von der Malsburg, Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85\u2013100 (1973).","journal-title":"Kybernetik"},{"key":"e_1_3_4_26_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1523\/JNEUROSCI.02-01-00032.1982","article-title":"Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex","volume":"2","author":"Bienenstock EL","year":"1982","unstructured":"EL Bienenstock, LN Cooper, PW Munro, Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J Neurosci 2, 32\u201348 (1982).","journal-title":"J Neurosci"},{"key":"e_1_3_4_27_2","doi-asserted-by":"crossref","first-page":"7508","DOI":"10.1073\/pnas.83.19.7508","article-title":"From basic network principles to neural architecture: Emergence of spatial-opponent cells","volume":"83","author":"Linsker R","year":"1986","unstructured":"R Linsker, From basic network principles to neural architecture: Emergence of spatial-opponent cells. Proc Natl Acad Sci USA 83, 7508\u20137512 (1986).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_28_2","doi-asserted-by":"crossref","first-page":"8390","DOI":"10.1073\/pnas.83.21.8390","article-title":"From basic network principles to neural architecture: Emergence of orientation-selective cells","volume":"83","author":"Linsker R","year":"1986","unstructured":"R Linsker, From basic network principles to neural architecture: Emergence of orientation-selective cells. Proc Natl Acad Sci USA 83, 8390\u20138394 (1986).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_29_2","doi-asserted-by":"crossref","first-page":"8779","DOI":"10.1073\/pnas.83.22.8779","article-title":"From basic network principles to neural architecture: Emergence of orientation columns","volume":"83","author":"Linsker R","year":"1986","unstructured":"R Linsker, From basic network principles to neural architecture: Emergence of orientation columns. Proc Natl Acad Sci USA 83, 8779\u20138783 (1986).","journal-title":"Proc Natl Acad Sci USA"},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja E","year":"1982","unstructured":"E Oja, Simplified neuron model as a principal component analyzer. J Math Biol 15, 267\u2013273 (1982).","journal-title":"J Math Biol"},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1207\/s15516709cog0901_5","article-title":"Feature discovery by competitive learning","volume":"9","author":"Rumelhart DE","year":"1985","unstructured":"DE Rumelhart, D Zipser, Feature discovery by competitive learning. Cogn Sci 9, 75\u2013112 (1985).","journal-title":"Cogn Sci"},{"key":"e_1_3_4_32_2","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/S0734-189X(87)80014-2","article-title":"A massively parallel architecture for a self-organizing neural pattern recognition machine","volume":"37","author":"Carpenter GA","year":"1987","unstructured":"GA Carpenter, S Grossberg, A massively parallel architecture for a self-organizing neural pattern recognition machine. Comput Vision Graphics Image Process 37, 54\u2013115 (1987).","journal-title":"Comput Vision Graphics Image Process"},{"key":"e_1_3_4_33_2","doi-asserted-by":"crossref","first-page":"1464","DOI":"10.1109\/5.58325","article-title":"The self-organizing map","volume":"78","author":"Kohonen T","year":"1990","unstructured":"T Kohonen, The self-organizing map. Proc IEEE 78, 1464\u20131480 (1990).","journal-title":"Proc IEEE"},{"key":"e_1_3_4_34_2","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1007\/BF02331346","article-title":"Forming sparse representations by local anti-Hebbian learning","volume":"64","author":"F\u00f6ldiak P","year":"1990","unstructured":"P F\u00f6ldiak, Forming sparse representations by local anti-Hebbian learning. Biol Cybernetics 64, 165\u2013170 (1990).","journal-title":"Biol Cybernetics"},{"key":"e_1_3_4_35_2","unstructured":"HS Seung J Zung A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation anti-Hebbian inhibition and synapse elimination. arXiv:1704.00646. Preprint posted April 3 2017. (2017)."},{"key":"e_1_3_4_36_2","unstructured":"D Krotov JJ Hopfield Data from \u201cBiological Learning.\u201d GitHub. Available at https:\/\/github.com\/DimaKrotov\/Biological_Learning. Deposited January 28 2019. (2019)."},{"key":"e_1_3_4_37_2","first-page":"1037","volume-title":"Advances in Neural Information Processing Systems","author":"N\u00f8kland A","year":"2016","unstructured":"A N\u00f8kland, Direct feedback alignment provides learning in deep neural networks. Advances in Neural Information Processing Systems, eds DD Lee, M Sugiyama, UV Luxburg, I Guyon, R Garnett (Neural Information Processing Systems Foundation, Inc., La Jolla, CA), pp. 1037\u20131045 (2016)."},{"key":"e_1_3_4_38_2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.3389\/fncom.2015.00099","article-title":"Unsupervised learning of digit recognition using spike-timing-dependent plasticity","volume":"9","author":"Diehl PU","year":"2015","unstructured":"PU Diehl, M Cook, Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front Comput Neurosci 9, 99 (2015).","journal-title":"Front Comput Neurosci"},{"key":"e_1_3_4_39_2","unstructured":"S Bartunov A Santoro BA Richards GE Hinton T Lillicrap Assessing the scalability of biologically-motivated deep learning algorithms and architectures. arXiv:1807.04587. Preprint posted November 20 2018. (2018)."},{"key":"e_1_3_4_40_2","unstructured":"Y Tang Deep learning using linear support vector machines. arXiv:1306.0239. Preprint posted February 21 2015. (2013)."},{"key":"e_1_3_4_41_2","unstructured":"A Rasmus M Berglund M Honkala H Valpola T Raiko Semi-supervised learning with ladder networks. Advances in Neural Information Processing Systems pp 3546\u20133554. (2015)."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.1820458116","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,7]],"date-time":"2022-06-07T19:08:47Z","timestamp":1654628927000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.1820458116"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,29]]},"references-count":41,"journal-issue":{"issue":"16","published-print":{"date-parts":[[2019,4,16]]}},"alternative-id":["10.1073\/pnas.1820458116"],"URL":"https:\/\/doi.org\/10.1073\/pnas.1820458116","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,29]]},"assertion":[{"value":"2019-03-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}