{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T20:21:55Z","timestamp":1776889315472,"version":"3.51.2"},"reference-count":73,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":14,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"tdm","delay-in-days":14,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Neuromorph. Comput. Eng."],"published-print":{"date-parts":[[2022,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Hebbian plasticity in winner-take-all (WTA) networks is highly attractive for neuromorphic on-chip learning, owing to its efficient, local, unsupervised, and on-line nature. Moreover, its biological plausibility may help overcome important limitations of artificial algorithms, such as their susceptibility to adversarial attacks, and their high demands for training-example quantity and repetition. However, Hebbian WTA learning has found little use in machine learning, likely because it has been missing an optimization theory compatible with deep learning (DL). Here we show rigorously that WTA networks constructed by standard DL elements, combined with a Hebbian-like plasticity that we derive, maintain a Bayesian generative model of the data. Importantly, without any supervision, our algorithm, SoftHebb, minimizes cross-entropy, i.e. a common loss function in supervised DL. We show this theoretically and in practice. The key is a \u2018soft\u2019 WTA where there is no absolute \u2018hard\u2019 winner neuron. Strikingly, in shallow-network comparisons with backpropagation, SoftHebb shows advantages beyond its Hebbian efficiency. Namely, it converges in fewer iterations, and is significantly more robust to noise and adversarial attacks. Notably, attacks that maximally confuse SoftHebb are also confusing to the human eye, potentially linking human perceptual robustness, with Hebbian WTA circuits of cortex. Finally, SoftHebb can generate synthetic objects as interpolations of real object classes. All in all, Hebbian efficiency, theoretical underpinning, cross-entropy-minimization, and surprising empirical advantages, suggest that SoftHebb may inspire highly neuromorphic and radically different, but practical and advantageous learning algorithms and hardware accelerators.<\/jats:p>","DOI":"10.1088\/2634-4386\/aca710","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T06:23:58Z","timestamp":1669789438000},"page":"044017","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6521-0717","authenticated-orcid":true,"given":"Timoleon","family":"Moraitis","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dmitry","family":"Toichkin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adrien","family":"Journ\u00e9","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yansong","family":"Chua","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qinghai","family":"Guo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"key":"nceaca710bib1","first-page":"324","article-title":"Hebbian learning meets deep convolutional neural networks","author":"Amato","year":"2019"},{"key":"nceaca710bib2","article-title":"VICReg: variance-invariance-covariance regularization for self-supervised learning","author":"Bardes","year":"2021"},{"key":"nceaca710bib3","doi-asserted-by":"publisher","first-page":"1129","DOI":"10.1162\/neco.1995.7.6.1129","article-title":"An information-maximization approach to blind separation and blind deconvolution","volume":"7","author":"Bell","year":"1995","journal-title":"Neural Comput."},{"key":"nceaca710bib4","article-title":"Towards biologically plausible deep learning","author":"Bengio","year":"2015"},{"key":"nceaca710bib5","article-title":"Understanding and improving interpolation in autoencoders via an adversarial regularizer","author":"Berthelot","year":"2018"},{"key":"nceaca710bib6","doi-asserted-by":"publisher","first-page":"68","DOI":"10.3389\/fncom.2014.00068","article-title":"Learning and stabilization of winner-take-all dynamics through interacting excitatory and inhibitory plasticity","volume":"8","author":"Binas","year":"2014","journal-title":"Front. Comput. Neurosci."},{"key":"nceaca710bib7","doi-asserted-by":"publisher","first-page":"8441","DOI":"10.1523\/JNEUROSCI.1400-04.2004","article-title":"A quantitative map of the circuit of cat primary visual cortex","volume":"24","author":"Binzegger","year":"2004","journal-title":"J. Neurosci."},{"key":"nceaca710bib8","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2022.865897","article-title":"A surrogate gradient spiking baseline for speech command recognition","volume":"16","author":"Bittar","year":"2022","journal-title":"Front. Neurosci."},{"key":"nceaca710bib9","article-title":"Optimizing the latent space of generative networks","author":"Bojanowski","year":"2017"},{"key":"nceaca710bib10","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1111\/ejn.12453","article-title":"Neurosystems: brain rhythms and cognitive processing","volume":"39","author":"Cannon","year":"2014","journal-title":"Eur. J. Neurosci."},{"key":"nceaca710bib11","first-page":"1597","article-title":"A simple framework for contrastive learning of visual representations","author":"Chen","year":"2020"},{"key":"nceaca710bib12","doi-asserted-by":"publisher","first-page":"1269","DOI":"10.1613\/jair.1.13643","article-title":"HEBO: pushing the limits of sample-efficient hyper-parameter optimisation","volume":"74","author":"Cowen-Rivers","year":"2022","journal-title":"J. Artif. Intell. Res."},{"key":"nceaca710bib13","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1109\/MSP.2017.2765202","article-title":"Generative adversarial networks: an overview","volume":"35","author":"Creswell","year":"2018","journal-title":"IEEE Signal Process. Mag."},{"key":"nceaca710bib14","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1038\/337129a0","article-title":"The recent excitement about neural networks","volume":"337","author":"Crick","year":"1989","journal-title":"Nature"},{"key":"nceaca710bib15","first-page":"904","article-title":"Understanding synthetic gradients and decoupled neural interfaces","author":"Czarnecki","year":"2017"},{"key":"nceaca710bib16","article-title":"Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018"},{"key":"nceaca710bib17","doi-asserted-by":"publisher","first-page":"99","DOI":"10.3389\/fncom.2015.00099","article-title":"Unsupervised learning of digit recognition using spike-timing-dependent plasticity","volume":"9","author":"Diehl","year":"2015","journal-title":"Front. Comput. Neurosci."},{"key":"nceaca710bib18","article-title":"Learning and inferring relations in cortical networks","author":"Diehl","year":"2016"},{"key":"nceaca710bib19","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1146\/annurev.neuro.27.070203.144152","article-title":"Neuronal circuits of the neocortex","volume":"27","author":"Douglas","year":"2004","journal-title":"Annu. Rev. Neurosci."},{"key":"nceaca710bib20","article-title":"Equilibrium propagation with continual weight updates","author":"Ernoult","year":"2020"},{"key":"nceaca710bib21","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/BF02331346","article-title":"Forming sparse representations by local anti-Hebbian learning","volume":"64","author":"F\u00f6ldiak","year":"1990","journal-title":"Biol. Cybern."},{"key":"nceaca710bib22","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1109\/IJCNN.1989.118615","article-title":"Adaptive network for optimal linear feature extraction","volume":"1","author":"F\u00f6ldi\u00e1k","year":"1989","journal-title":"1989 International Joint Conference on Neural Networks"},{"key":"nceaca710bib23","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2021.629892","article-title":"Learning without feedback: fixed random learning signals allow for feedforward training of deep neural networks","volume":"15","author":"Frenkel","year":"2021","journal-title":"Front. Neurosci."},{"key":"nceaca710bib24","first-page":"18704","article-title":"Short-term plasticity neurons learning to learn and forget","author":"Garcia Rodriguez","year":"2022"},{"key":"nceaca710bib25","article-title":"Generative adversarial networks","author":"Goodfellow","year":"2014"},{"key":"nceaca710bib26","article-title":"Local unsupervised learning for image analysis","author":"Grinberg","year":"2019"},{"key":"nceaca710bib27","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1111\/j.1551-6708.1987.tb00862.x","article-title":"Competitive learning: from interactive activation to adaptive resonance","volume":"11","author":"Grossberg","year":"1987","journal-title":"Cogn. Sci."},{"key":"nceaca710bib28","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.22901","article-title":"Towards deep learning with segregated dendrites","volume":"6","author":"Guerguiev","year":"2017","journal-title":"ELife"},{"key":"nceaca710bib29","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1038\/11219","article-title":"Feedback interactions between neuronal pointers and maps for attentional processing","volume":"2","author":"Hahnloser","year":"1999","journal-title":"Nat. Neurosci."},{"key":"nceaca710bib30","article-title":"Distilling the knowledge in a neural network","author":"Hinton","year":"2015"},{"key":"nceaca710bib31","first-page":"613","article-title":"A Hebbian\/anti-Hebbian network for online sparse dictionary learning derived from symmetric matrix factorization","author":"Hu","year":"2014"},{"key":"nceaca710bib32","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1113\/jphysiol.1962.sp006837","article-title":"Receptive fields, binocular interaction and functional architecture in the cat\u2019s visual cortex","volume":"160","author":"Hubel","year":"1962","journal-title":"J. Physiol."},{"key":"nceaca710bib33","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.neunet.2019.06.001","article-title":"Biologically plausible deep learning\u2014but how far can we go with shallow networks?","volume":"118","author":"Illing","year":"2019","journal-title":"Neural Netw."},{"key":"nceaca710bib34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-018-20082-0","article-title":"Error-gated Hebbian rule: a local learning rule for principal and independent component analysis","volume":"8","author":"Isomura","year":"2018","journal-title":"Sci. Rep."},{"key":"nceaca710bib35","article-title":"Spike-inspired rank coding for fast and accurate recurrent neural networks","author":"Jeffares","year":"2022"},{"key":"nceaca710bib36","article-title":"Hebbian deep learning without feedback","author":"Journ\u00e9","year":"2022"},{"key":"nceaca710bib37","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2015"},{"key":"nceaca710bib38","author":"Krizhevsky","year":"2009"},{"key":"nceaca710bib39","first-page":"7723","article-title":"Unsupervised learning by competing hidden units","volume":"116","author":"Krotov","year":"2019"},{"key":"nceaca710bib40","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1016\/j.neunet.2021.08.003","article-title":"Hebbian semi-supervised learning in a sample efficiency setting","volume":"143","author":"Lagani","year":"2021","journal-title":"Neural Netw."},{"key":"nceaca710bib41","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1162\/089976699300016719","article-title":"Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources","volume":"11","author":"Lee","year":"1999","journal-title":"Neural Comput."},{"key":"nceaca710bib42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/ncomms13276","article-title":"Random synaptic feedback weights support error backpropagation for deep learning","volume":"7","author":"Lillicrap","year":"2016","journal-title":"Nat. Commun."},{"key":"nceaca710bib43","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1162\/neco.1992.4.5.691","article-title":"Local synaptic learning rules suffice to maximize mutual information in a linear network","volume":"4","author":"Linsker","year":"1992","journal-title":"Neural Comput."},{"key":"nceaca710bib44","doi-asserted-by":"publisher","first-page":"2519","DOI":"10.1162\/089976600300014827","article-title":"On the computational power of winner-take-all","volume":"12","author":"Maass","year":"2000","journal-title":"Neural Comput."},{"key":"nceaca710bib45","article-title":"Towards deep learning models resistant to adversarial attacks","author":"Madry","year":"2017"},{"key":"nceaca710bib46","article-title":"Predictive coding approximates backprop along arbitrary computation graphs","author":"Millidge","year":"2020"},{"key":"nceaca710bib47","article-title":"Optimality of short-term synaptic plasticity in modelling certain dynamic environments","author":"Moraitis","year":"2020"},{"key":"nceaca710bib48","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003037","article-title":"Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity","volume":"9","author":"Nessler","year":"2013","journal-title":"PLoS Comput. Biol."},{"key":"nceaca710bib49","first-page":"1357","article-title":"STDP enables spiking neurons to detect hidden causes of their inputs","volume":"vol 22","author":"Nessler","year":"2009"},{"key":"nceaca710bib50","article-title":"Direct feedback alignment provides learning in deep neural networks","volume":"vol 29","author":"N\u00f8kland","year":"2016"},{"key":"nceaca710bib51","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja","year":"1982","journal-title":"J. Math. Biol."},{"key":"nceaca710bib52","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1038\/381607a0","article-title":"Emergence of simple-cell receptive field properties by learning a sparse code for natural images","volume":"381","author":"Olshausen","year":"1996","journal-title":"Nature"},{"key":"nceaca710bib53","doi-asserted-by":"publisher","first-page":"3311","DOI":"10.1016\/S0042-6989(97)00169-7","article-title":"Sparse coding with an overcomplete basis set: a strategy employed by V1?","volume":"37","author":"Olshausen","year":"1997","journal-title":"Vis. Res."},{"key":"nceaca710bib54","doi-asserted-by":"publisher","first-page":"1010","DOI":"10.1038\/s41593-021-00857-x","article-title":"Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits","volume":"24","author":"Payeur","year":"2021","journal-title":"Nat. Neurosci."},{"key":"nceaca710bib55","first-page":"769","article-title":"A Hebbian\/anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features","author":"Pehlevan","year":"2014"},{"key":"nceaca710bib56","article-title":"A normative theory of adaptive dimensionality reduction in neural networks","volume":"vol 28","author":"Pehlevan","year":"2015","edition":"ed"},{"key":"nceaca710bib57","first-page":"593","article-title":"A clustering neural network model of insect olfaction","author":"Pehlevan","year":"2017"},{"key":"nceaca710bib58","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2018.00774","article-title":"Deep learning with spiking neurons: opportunities and challenges","volume":"12","author":"Pfeiffer","year":"2018","journal-title":"Front. Neurosci."},{"key":"nceaca710bib59","article-title":"Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks","author":"Pogodin","year":"2020"},{"key":"nceaca710bib60","article-title":"Towards biologically plausible convolutional networks","author":"Pogodin","year":"2021"},{"key":"nceaca710bib61","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1038\/s41583-020-0301-7","article-title":"Illuminating dendritic function with computational models","volume":"21","author":"Poirazi","year":"2020","journal-title":"Nat. Rev. Neurosci."},{"key":"nceaca710bib62","article-title":"Deflecting adversarial attacks","author":"Qin","year":"2020"},{"key":"nceaca710bib63","article-title":"Unsupervised representation learning with deep convolutional generative adversarial networks","author":"Radford","year":"2015"},{"key":"nceaca710bib64","article-title":"Foolbox: a python toolbox to benchmark the robustness of machine learning models","author":"Rauber","year":"2017"},{"key":"nceaca710bib65","doi-asserted-by":"publisher","first-page":"735","DOI":"10.1162\/NECO_a_00091","article-title":"Collective stability of networks of winner-take-all circuits","volume":"23","author":"Rutishauser","year":"2011","journal-title":"Neural Comput."},{"key":"nceaca710bib66","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1016\/0893-6080(89)90044-0","article-title":"Optimal unsupervised learning in a single-layer linear feedforward neural network","volume":"2","author":"Sanger","year":"1989","journal-title":"Neural Netw."},{"key":"nceaca710bib67","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-022-29870-9","article-title":"Chalcogenide optomemristors for multi-factor neuromorphic computation","volume":"13","author":"Sarwat","year":"2022","journal-title":"Nat. Commun."},{"key":"nceaca710bib68","doi-asserted-by":"publisher","first-page":"24","DOI":"10.3389\/fncom.2017.00024","article-title":"Equilibrium propagation: bridging the gap between energy-based models and backpropagation","volume":"11","author":"Scellier","year":"2017","journal-title":"Front. Comput. Neurosci."},{"key":"nceaca710bib69","article-title":"Self-supervised learning through efference copies","author":"Scherr","year":"2022"},{"key":"nceaca710bib70","first-page":"0033","article-title":"The unreasonable effectiveness of deep learning in artificial intelligence","volume":"117","author":"Sejnowski","year":"2020"},{"key":"nceaca710bib71","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1007\/BF00288907","article-title":"Self-organization of orientation sensitive cells in the striate cortex","volume":"14","author":"Von der Malsburg","year":"1973","journal-title":"Kybernetik"},{"key":"nceaca710bib72","article-title":"Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms","author":"Xiao","year":"2017"},{"key":"nceaca710bib73","article-title":"Toward next-generation artificial intelligence: catalyzing the NeuroAI revolution","author":"Zador","year":"2022"}],"container-title":["Neuromorphic Computing and Engineering"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T13:48:17Z","timestamp":1699883297000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/aca710"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,1]]},"references-count":73,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12,15]]},"published-print":{"date-parts":[[2022,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2634-4386\/aca710","relation":{},"ISSN":["2634-4386"],"issn-type":[{"value":"2634-4386","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,1]]},"assertion":[{"value":"SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks","name":"article_title","label":"Article Title"},{"value":"Neuromorphic Computing and Engineering","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2022 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2022-07-22","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-11-29","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-12-15","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}