{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T14:22:33Z","timestamp":1783520553297,"version":"3.55.0"},"reference-count":77,"publisher":"MIT Press","issue":"9","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid\/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.<\/jats:p>","DOI":"10.1162\/neco_a_01684","type":"journal-article","created":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T18:25:29Z","timestamp":1724178329000},"page":"1669-1712","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["Hebbian Descent: A Unified View on Log-Likelihood Learning"],"prefix":"10.1162","volume":"36","author":[{"given":"Jan","family":"Melchior","sequence":"first","affiliation":[{"name":"Ruhr University Bochum, 44801 Bochum, Germany jan.melchior@ini.rub.de"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robin","family":"Schiewer","sequence":"additional","affiliation":[{"name":"Ruhr University Bochum, 44801 Bochum, Germany robin.schiewer@ini.rub.de"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laurenz","family":"Wiskott","sequence":"additional","affiliation":[{"name":"Ruhr University Bochum, 44801 Bochum, Germany laurenz.wiskott@ini.rub.de"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,8,19]]},"reference":[{"issue":"12","key":"2024082018250189000_bib1","first-page":"989","article-title":"Avoiding catastrophic forgetting by coupling two reverberating neural networks","volume":"320","author":"Ans","year":"1997","journal-title":"Comptes Rendus de l\u2019Acad\u00e9mie des Sciences, Series III: Sciences de la Vie"},{"key":"2024082018250189000_bib2","author":"Bergstra","year":"2009","journal-title":"Quadratic polynomials learn better image features."},{"issue":"3","key":"2024082018250189000_bib3","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1088\/0305-4470\/28\/3\/018","article-title":"Learning by on-line gradient descent","volume":"28","author":"Biehl","year":"1995","journal-title":"Journal of Physics A: Mathematical and General"},{"issue":"1","key":"2024082018250189000_bib4","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1523\/JNEUROSCI.02-01-00032.1982","article-title":"Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex","volume":"2","author":"Bienenstock","year":"1982","journal-title":"Journal of Neuroscience"},{"key":"2024082018250189000_bib5","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1007\/978-3-642-76153-9_28","article-title":"Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition","volume-title":"Neurocomputing","author":"Bridle","year":"1990"},{"key":"2024082018250189000_bib6","author":"Carlile","year":"2017","journal-title":"Improving deep learning by inverse square root linear units (ISRLUs)."},{"key":"2024082018250189000_bib7","first-page":"601","article-title":"Stepsize variation methods for accelerating the back propagation algorithm","volume-title":"Proceedings of the International Joint Conference on Neural Networks","author":"Chen","year":"1990"},{"key":"2024082018250189000_bib8","first-page":"3246","article-title":"Parallel tempering is efficient for learning restricted Boltzmann machines","volume-title":"Proceedings of the International Joint Conference on Neural Networks","author":"Cho","year":"2010"},{"key":"2024082018250189000_bib9","author":"Clevert","year":"2015","journal-title":"Fast and accurate deep network learning by exponential linear units (ELUS)."},{"key":"2024082018250189000_bib10","article-title":"Parallel tempering for training of restricted Boltzmann machines","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Desjardins","year":"2010"},{"issue":"7","key":"2024082018250189000_bib11","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi","year":"2011","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2024082018250189000_bib12","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/s13218-018-0564-6","article-title":"A brief survey on forgetting from a knowledge representation and reasoning perspective","volume":"33","author":"Eiter","year":"2019","journal-title":"K\u00fcnstliche Intelligenz"},{"key":"2024082018250189000_bib13","first-page":"38","article-title":"Faster learning variations of back propagation: An empirical study","volume-title":"Proceedings of the 1988 Connectionist Models Summer School","author":"Fahlman","year":"1988"},{"key":"2024082018250189000_bib14","article-title":"Catastrophic forgetting in connectionist networks","author":"French","year":"1991","journal-title":"Encyclopedia of Cognitive Science"},{"issue":"4","key":"2024082018250189000_bib15","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1080\/095400997116595","article-title":"Pseudo-recurrent connectionist networks: An approach to the \u201csensitivity-stability\u201d dilemma","volume":"9","author":"French","year":"1997","journal-title":"Connection Science"},{"key":"2024082018250189000_bib16","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/978-3-642-46466-9_18","article-title":"Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition","volume-title":"Competition and cooperation in neural nets","author":"Fukushima","year":"1982"},{"issue":"2","key":"2024082018250189000_bib17","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1109\/72.80230","article-title":"Perceptron-based learning algorithms","volume":"1","author":"Gallant","year":"1990","journal-title":"IEEE Transactions on neural networks"},{"key":"2024082018250189000_bib18","article-title":"Understanding the difficulty of training deep feedforward neuralnetworks","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Glorot","year":"2010"},{"key":"2024082018250189000_bib19","first-page":"315","article-title":"Deep sparse rectifier neural networks","volume-title":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","author":"Glorot","year":"2011"},{"key":"2024082018250189000_bib20","first-page":"1783","article-title":"Learning one convolutional layer with overlapping patches","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Goel","year":"2018"},{"issue":"6789","key":"2024082018250189000_bib21","doi-asserted-by":"publisher","first-page":"947","DOI":"10.1038\/35016072","article-title":"Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit","volume":"405","author":"Hahnloser","year":"2000","journal-title":"Nature"},{"key":"2024082018250189000_bib22","first-page":"217","article-title":"Permitted and forbidden sets in symmetric threshold-linear networks","volume-title":"Advances in neural information processing systems","author":"Hahnloser","year":"2001"},{"key":"2024082018250189000_bib23","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2024082018250189000_bib24","volume-title":"The organization of behavior: A neuropsychological approach","author":"Hebb","year":"1949"},{"issue":"6","key":"2024082018250189000_bib25","doi-asserted-by":"publisher","first-page":"1321","DOI":"10.1109\/72.641455","article-title":"Nonlinear backpropagation: Doing backpropagation without derivatives of the activation function","volume":"8","author":"Hertz","year":"1997","journal-title":"IEEE Transactions on Neural Networks"},{"issue":"1\u20133","key":"2024082018250189000_bib26","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/0004-3702(89)90049-0","article-title":"Connectionist learning procedures","volume":"40","author":"Hinton","year":"1989","journal-title":"Artificial Intelligence"},{"key":"2024082018250189000_bib27","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1162\/089976602760128018","article-title":"Training products of experts by minimizing contrastive divergence","volume":"14","author":"Hinton","year":"2002","journal-title":"Neural Computation"},{"key":"2024082018250189000_bib28","author":"Hochreiter","year":"1991","journal-title":"Untersuchungen zu dynamischen neuronalen netzen."},{"issue":"8","key":"2024082018250189000_bib29","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2024082018250189000_bib30","author":"Ioffe","year":"2015","journal-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift."},{"key":"2024082018250189000_bib31","first-page":"927","article-title":"Efficient learning of generalized linear and single index models with isotonic regression","volume-title":"Advances in neural information processing systems","author":"Kakade","year":"2011"},{"issue":"10","key":"2024082018250189000_bib32","doi-asserted-by":"publisher","first-page":"947","DOI":"10.2514\/8.5282","article-title":"Gradient theory of optimal flight paths","volume":"30","author":"Kelley","year":"1960","journal-title":"ARS Journal"},{"key":"2024082018250189000_bib33","author":"Kingma","year":"2014","journal-title":"Adam: A method for stochastic optimization."},{"issue":"13","key":"2024082018250189000_bib34","doi-asserted-by":"publisher","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"Kirkpatrick","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024082018250189000_bib35","first-page":"972","article-title":"Self-normalizing neural networks","volume-title":"Advances in neural information processing systems","author":"Klambauer","year":"2017"},{"key":"2024082018250189000_bib36","article-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky","year":"2009"},{"key":"2024082018250189000_bib37","doi-asserted-by":"publisher","first-page":"193","DOI":"10.14445\/23488379\/IJEEE-V2I11P104","article-title":"ANN training: A review of soft computing approaches","volume":"2","author":"Kumar","year":"2015","journal-title":"International Journal of Electrical and Electronics Engineering"},{"key":"2024082018250189000_bib38","doi-asserted-by":"publisher","first-page":"2285","DOI":"10.1162\/NECO_a_00014","article-title":"Tractable multivariate binary density estimation and the restricted Boltzmann forest","volume":"22","author":"Larochelle","year":"2010","journal-title":"Neural Computation"},{"key":"2024082018250189000_bib39","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/978-3-642-35289-8_3","article-title":"Efficient BackProp","volume-title":"Neural networks: Tricks of the trade","author":"LeCun","year":"2012"},{"issue":"5","key":"2024082018250189000_bib40","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1016\/S0893-6080(05)80116-9","article-title":"An analysis of premature saturation in back propagation learning","volume":"6","author":"Lee","year":"1993","journal-title":"Neural Networks"},{"issue":"4","key":"2024082018250189000_bib41","doi-asserted-by":"crossref","first-page":"1216","DOI":"10.1214\/aoap\/1028903378","article-title":"On the storage capacity of Hopfield models with correlated patterns","volume":"8","author":"L\u00f6we","year":"1998","journal-title":"Annals of Applied Probability"},{"key":"2024082018250189000_bib42","article-title":"Rectifier nonlinearities improve neural network acoustic models","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Maas","year":"2013"},{"key":"2024082018250189000_bib43","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/978-1-4684-6775-8_5","article-title":"Simple memory: A theory for archicortex","volume-title":"From the retina to the neocortex","author":"Marr","year":"1991"},{"issue":"3","key":"2024082018250189000_bib44","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1037\/0033-295X.102.3.419","article-title":"Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory","volume":"102","author":"McClelland","year":"1995","journal-title":"Psychological Review"},{"key":"2024082018250189000_bib45","first-page":"109","article-title":"Catastrophic interference in connectionist networks: The sequential learning problem","volume-title":"Psychology of learning and motivation","author":"McCloskey","year":"1989"},{"key":"2024082018250189000_bib46","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4899-3242-6","volume-title":"Generalized linear models","author":"McCullagh","year":"1989"},{"key":"2024082018250189000_bib47","author":"Melchior","year":"2021","journal-title":"On the importance of centering in artificial neural networks"},{"issue":"99","key":"2024082018250189000_bib48","first-page":"1","article-title":"How to center deep Boltzmann machines","volume":"17","author":"Melchior","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"2024082018250189000_bib49","author":"Mikolov","year":"2014","journal-title":"Learning longer memory in recurrent neural networks"},{"key":"2024082018250189000_bib50","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1007\/978-3-642-35289-8_33","article-title":"Deep Boltzmann machines and the centering trick","volume-title":"Lecture Notes in Computer Science","author":"Montavon","year":"2012"},{"key":"2024082018250189000_bib51","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/B978-1-4832-1448-1.50007-X","article-title":"Contrastive Hebbian learning in the continuous Hopfield model","volume-title":"Connectionist models","author":"Movellan","year":"1991"},{"issue":"5","key":"2024082018250189000_bib52","doi-asserted-by":"publisher","first-page":"e1004250","DOI":"10.1371\/journal.pcbi.1004250","article-title":"Memory storage fidelity in the hippocampal circuit: The role of subregions and input statistics","volume":"11","author":"Neher","year":"2015","journal-title":"PLOS Computational Biology"},{"key":"2024082018250189000_bib53","volume-title":"Generalized linear models","author":"Nelder","year":"1972"},{"key":"2024082018250189000_bib54","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1109\/IJCNN.2003.1223698","article-title":"Fast convergence for backpropagation network with magnified gradient function","volume-title":"Proceedings of the International Joint Conference on Neural Networks, 2003","author":"Ng","year":"2003"},{"key":"2024082018250189000_bib55","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1023\/A:1018611626332","article-title":"Fast convergent generalized backpropagation algorithm with constant learning rate","volume":"9","author":"Ng","year":"1999","journal-title":"Neural Processing Letters"},{"issue":"3","key":"2024082018250189000_bib56","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja","year":"1982","journal-title":"Journal of Mathematical Biology"},{"key":"2024082018250189000_bib57","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1016\/0893-6080(92)90008-7","article-title":"Improving the convergence of the backpropagation algorithm","volume":"5","author":"Ooyen","year":"1992","journal-title":"Neural Networks"},{"key":"2024082018250189000_bib58","article-title":"On the difficulty of training recur- rent neural networks","volume-title":"Proceedings of the 30th International Conference on Machine Learning","author":"Pascanu","year":"2013"},{"issue":"2","key":"2024082018250189000_bib59","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1037\/0033-295X.97.2.285","article-title":"Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions","volume":"97","author":"Ratcliff","year":"1990","journal-title":"Psychological Review"},{"key":"2024082018250189000_bib60","doi-asserted-by":"crossref","first-page":"586","DOI":"10.1109\/ICNN.1993.298623","article-title":"A direct adaptive method for faster backpropagation learning: The RPROP algorithm","volume-title":"Proceedings of the IEEE International Conference on Neural Networks","author":"Riedmiller","year":"1993"},{"key":"2024082018250189000_bib61","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/978-1-4612-5110-1_10","article-title":"A convergence theorem for non negative almost supermartingales and some applications","volume-title":"Herbert Robbins selected papers","author":"Robbins","year":"1985"},{"issue":"2","key":"2024082018250189000_bib62","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1080\/09540099550039318","article-title":"Catastrophic forgetting, rehearsal and pseudorehearsal","volume":"7","author":"Robins","year":"1995","journal-title":"Connection Science"},{"issue":"6","key":"2024082018250189000_bib63","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1037\/h0042519","article-title":"The perceptron: A probabilistic model for information storage and organization in the brain","volume":"65","author":"Rosenblatt","year":"1958","journal-title":"Psychological Review"},{"key":"2024082018250189000_bib64","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"2024082018250189000_bib65","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/5236.001.0001","volume-title":"Parallel distributed processing: Explorations in the microstructure of cognition","author":"Rumelhart","year":"1986"},{"key":"2024082018250189000_bib66","article-title":"Online algorithms and stochastic approximations","volume":"5","author":"Saad","year":"1998","journal-title":"Online Learning"},{"key":"2024082018250189000_bib67","author":"Salehinejad","year":"2018","journal-title":"Recent advances in recurrent neural networks"},{"issue":"6","key":"2024082018250189000_bib68","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1016\/0893-6080(89)90044-0","article-title":"Optimal unsupervised learning in a single-layer linear feedforward neural network","volume":"2","author":"Sanger","year":"1989","journal-title":"Neural Networks"},{"key":"2024082018250189000_bib69","author":"Santoro","year":"2016","journal-title":"One-shot learning with memory-augmented neural networks"},{"key":"2024082018250189000_bib70","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/3-540-49430-8_11","article-title":"Centering neural network gradient factors","volume-title":"Neural networks: Tricks of the trade","author":"Schraudolph","year":"1998"},{"key":"2024082018250189000_bib71","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/B978-0-12-148955-7.50010-2","article-title":"The Hebb rule for synaptic plasticity: Algorithms and implementations","volume-title":"Neural models of plasticity: Experimental and theoretical approaches","author":"Sejnowski","year":"1989"},{"key":"2024082018250189000_bib72","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1145\/1390156.1390290","article-title":"Training restricted Boltzmann machines using approximations to the likelihood gradient","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Tieleman","year":"2008"},{"key":"2024082018250189000_bib73","article-title":"Enhanced backpropagation training algorithm for transient event identification","volume-title":"Transactions of the American Nuclear Society","author":"Vitela","year":"1993"},{"issue":"3","key":"2024082018250189000_bib74","first-page":"193","article-title":"A survey on backpropagation algorithms for feedforward neural networks","volume":"1","author":"Vora","year":"2013","journal-title":"International Journal of Engineering Development and Research"},{"key":"2024082018250189000_bib75","first-page":"96","article-title":"Adaptive switching circuits","volume-title":"1960 IRE WESCON Convention Record, Part 4","author":"Widrow","year":"1960"},{"issue":"2","key":"2024082018250189000_bib76","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1162\/089976603762552988","article-title":"Equivalence of backpropagation and contrastive Hebbian learning in a layered network","volume":"15","author":"Xie","year":"2003","journal-title":"Neural Computation"},{"key":"2024082018250189000_bib77","first-page":"645","article-title":"An adaptive activation function for multilayer feedforward neural networks","volume-title":"Proceedings of the 2002 IEEE Region 10 Conference on Computers, Communications, Control and Power Engineering","author":"Yu","year":"2002"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/9\/1669\/2465982\/neco_a_01684.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/9\/1669\/2465982\/neco_a_01684.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T18:26:08Z","timestamp":1724178368000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/36\/9\/1669\/124060\/Hebbian-Descent-A-Unified-View-on-Log-Likelihood"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,19]]},"references-count":77,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,8,19]]},"published-print":{"date-parts":[[2024,8,19]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01684","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,8,19]]}}}