{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T17:25:22Z","timestamp":1783531522619,"version":"3.55.0"},"reference-count":74,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2024,12,22]],"date-time":"2024-12-22T00:00:00Z","timestamp":1734825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Dutch Ministry of Education","award":["024.005.022"],"award-info":[{"award-number":["024.005.022"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Learning is a fundamental property of intelligent systems, observed across biological organisms and engineered systems. While modern intelligent systems typically rely on gradient descent for learning, the need for exact gradients and complex information flow makes its implementation in biological and neuromorphic systems challenging. This has motivated the exploration of alternative learning mechanisms that can operate locally and do not rely on exact gradients. In this work, we introduce a novel approach that leverages noise in the parameters of the system and global reinforcement signals. Using an Ornstein\u2013Uhlenbeck process with adaptive dynamics, our method balances exploration and exploitation during learning, driven by deviations from error predictions, akin to reward prediction error. Operating in continuous time, Ornstein\u2013Uhlenbeck adaptation (OUA) is proposed as a general mechanism for learning in dynamic, time-evolving environments. We validate our approach across a range of different tasks, including supervised learning and reinforcement learning in feedforward and recurrent systems. Additionally, we demonstrate that it can perform meta-learning, adjusting hyper-parameters autonomously. Our results indicate that OUA provides a promising alternative to traditional gradient-based methods, with potential applications in neuromorphic computing. It also hints at a possible mechanism for noise-driven learning in the brain, where stochastic neurotransmitter release may guide synaptic adjustments.<\/jats:p>","DOI":"10.3390\/e26121125","type":"journal-article","created":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T02:39:56Z","timestamp":1734921596000},"page":"1125","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Ornstein\u2013Uhlenbeck Adaptation as a Mechanism for Learning in Brains and Machines"],"prefix":"10.3390","volume":"26","author":[{"given":"Jes\u00fas","family":"Garc\u00eda Fern\u00e1ndez","sequence":"first","affiliation":[{"name":"Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6500HB Nijmegen, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nasir","family":"Ahmad","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6500HB Nijmegen, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2206-9098","authenticated-orcid":false,"given":"Marcel","family":"van Gerven","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6500HB Nijmegen, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1098\/rsif.2008.0344","article-title":"Molecular circuits for associative learning in single-celled organisms","volume":"6","author":"Fernando","year":"2009","journal-title":"J. R. Soc. Interface"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gagliano, M., Vyazovskiy, V.V., Borb\u00e9ly, A.A., Grimonprez, M., and Depczynski, M. (2016). Learning by association in plants. Sci. Rep., 6.","DOI":"10.1038\/srep38427"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/j.funbio.2021.02.001","article-title":"Hyphal and mycelial consciousness: The concept of the fungal mind","volume":"125","author":"Money","year":"2021","journal-title":"Fungal Biol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.conb.2012.09.005","article-title":"Behavioral plasticity, learning, and memory in C. Elegans","volume":"23","author":"Sasakura","year":"2013","journal-title":"Curr. Opin. Neurobiol."},{"key":"ref_5","unstructured":"Sutton, R.S. (2018). Reinforcement Learning: An Introduction, MIT Press. A Bradford Book."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Lecun","year":"2015","journal-title":"Nature"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Brunton, S.L., and Kutz, J.N. (2022). Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control, Cambridge University Press.","DOI":"10.1017\/9781009089517"},{"key":"ref_8","unstructured":"Linnainmaa, S. (1970). The Representation of the Cumulative Rounding Error of an Algorithm as a Taylor Expansion of the Local Rounding Errors. [Master\u2019s Thesis, University of Helsinki]. (In Finnish)."},{"key":"ref_9","unstructured":"Werbos, P. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. [Ph.D. Thesis, Harvard University]."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1985). Learning Internal Representations by Error Propagation, California Univ San Diego La Jolla Inst for Cognitive Science. Technical Report.","DOI":"10.21236\/ADA164453"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1038\/s41583-020-0277-3","article-title":"Backpropagation and the brain","volume":"21","author":"Lillicrap","year":"2020","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.tics.2018.12.005","article-title":"Theories of error back-propagation in the brain","volume":"23","author":"Whittington","year":"2019","journal-title":"Trends Cogn. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1629","DOI":"10.1109\/5.58356","article-title":"Neuromorphic Electronic Systems","volume":"78","author":"Mead","year":"1990","journal-title":"Proc. IEEE"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1145\/1978542.1978559","article-title":"Cognitive computing","volume":"54","author":"Modha","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4911","DOI":"10.1038\/s41467-023-40533-1","article-title":"Toward a formal theory for computing machines made out of whatever physics offers","volume":"14","author":"Jaeger","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1109\/JPROC.2021.3067593","article-title":"Advancing neuromorphic computing with loihi: A survey of results and outlook","volume":"109","author":"Davies","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja","year":"1982","journal-title":"J. Math. Biol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1523\/JNEUROSCI.02-01-00032.1982","article-title":"Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex","volume":"2","author":"Bienenstock","year":"1982","journal-title":"J. Neurosci."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Scellier, B., and Bengio, Y. (2017). Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Front. Comput. Neurosci., 11.","DOI":"10.3389\/fncom.2017.00024"},{"key":"ref_20","unstructured":"Bengio, Y. (2014). How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1162\/NECO_a_00949","article-title":"An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity","volume":"29","author":"Whittington","year":"2017","journal-title":"Neural Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1126\/science.275.5297.213","article-title":"Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs","volume":"275","author":"Markram","year":"1997","journal-title":"Science"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3625","DOI":"10.1038\/s41467-020-17236-y","article-title":"A solution to the learning dilemma for recurrent networks of spiking neurons","volume":"11","author":"Bellec","year":"2020","journal-title":"Nat. Commun."},{"key":"ref_24","first-page":"17839","article-title":"Latent equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons","volume":"34","author":"Haider","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1010","DOI":"10.1038\/s41593-021-00857-x","article-title":"Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits","volume":"24","author":"Payeur","year":"2021","journal-title":"Nat. Neurosci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1109\/9.119632","article-title":"Multivariate stochastic approximation using a simultaneous perturbation gradient approximation","volume":"37","author":"Spall","year":"1992","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1109\/5.58323","article-title":"30 years of adaptive neural networks: Perceptron, madaline, and backpropagation","volume":"78","author":"Widrow","year":"1990","journal-title":"Proc. IEEE"},{"key":"ref_28","first-page":"1197","article-title":"Learning curves for stochastic gradient descent in linear feedforward networks","volume":"16","author":"Werfel","year":"2003","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","first-page":"212","article-title":"Summed weight neuron perturbation: An O(n) improvement over weight perturbation","volume":"5","author":"Flower","year":"1992","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_30","first-page":"31929","article-title":"On the stability and scalability of node perturbation learning","volume":"35","author":"Hiratani","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"048104","DOI":"10.1103\/PhysRevLett.97.048104","article-title":"Gradient learning in spiking neural networks by dynamic perturbation of conductances","volume":"97","author":"Fiete","year":"2006","journal-title":"Phys. Rev. Lett."},{"key":"ref_33","unstructured":"Dalm, S., Offergeld, J., Ahmad, N., and van Gerven, M. (2024). Efficient deep learning with decorrelated backpropagation. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez, J.G., Keemink, S., and van Gerven, M. (2024). Gradient-free training of recurrent neural networks using random perturbations. Front. Neurosci., 18.","DOI":"10.3389\/fnins.2024.1439155"},{"key":"ref_35","first-page":"789","article-title":"Analog VLSI implementation of multi-dimensional gradient descent","volume":"5","author":"Kirk","year":"1992","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","first-page":"803","article-title":"A study of parallel perturbative gradient descent","volume":"7","author":"Lippe","year":"1994","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_37","first-page":"021006","article-title":"Weight versus node perturbation learning in temporally extended tasks: Weight perturbation often performs similarly or better","volume":"13","author":"Klos","year":"2023","journal-title":"Phys. Rev. X"},{"key":"ref_38","first-page":"244","article-title":"A fast stochastic error-descent algorithm for supervised learning and optimization","volume":"5","author":"Cauwenberghs","year":"1992","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1109\/72.80205","article-title":"Model-free distributed learning","volume":"1","author":"Dembo","year":"1990","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"8400","DOI":"10.1523\/JNEUROSCI.4284-09.2010","article-title":"A reward-modulated hebbian learning rule can explain experimentally observed network reorganization in a brain control task","volume":"30","author":"Legenstein","year":"2010","journal-title":"J. Neurosci."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"e20899","DOI":"10.7554\/eLife.20899","article-title":"Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks","volume":"6","author":"Miconi","year":"2017","journal-title":"eLife"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1103\/PhysRev.36.823","article-title":"On the theory of the Brownian motion","volume":"36","author":"Uhlenbeck","year":"1930","journal-title":"Phys. Rev."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"351","DOI":"10.2307\/1968873","article-title":"The Brownian movement and stochastic equations","volume":"43","author":"Doob","year":"1942","journal-title":"Ann. Math."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"Schultz","year":"1997","journal-title":"Science"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1038\/s42256-022-00452-0","article-title":"Biological underpinnings for lifelong learning machines","volume":"4","author":"Kudithipudi","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"ref_46","unstructured":"Tzen, B., and Raginsky, M. (2019). Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. arXiv."},{"key":"ref_47","unstructured":"Wan, Y., Naik, A., and Sutton, R.S. (2021, January 18\u201324). Learning and planning in average-reward Markov decision processes. Proceedings of the International Conference on Machine Learning, PMLR."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1109\/37.954521","article-title":"Na\u00efve control of the double integrator","volume":"21","author":"Rao","year":"2001","journal-title":"IEEE Control Syst. Mag."},{"key":"ref_49","unstructured":"Kidger, P. (2021). On Neural Differential Equations. [Ph.D. Thesis, University of Oxford]."},{"key":"ref_50","unstructured":"Morrill, J., Kidger, P., Yang, L., and Lyons, T. (2021). Neural controlled differential equations for online prediction tasks. arXiv."},{"key":"ref_51","unstructured":"Ahmad, N., Schrader, E., and van Gerven, M. (2022). Constrained parameter inference as a principle for learning. arXiv."},{"key":"ref_52","unstructured":"Ahmad, N. (2024). Correlations are ruining your gradient descent. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"3327","DOI":"10.1016\/S0042-6989(97)00121-1","article-title":"The \u201cindependent components\u201d of natural scenes are edge filters","volume":"37","author":"Bell","year":"1997","journal-title":"Vis. Res."},{"key":"ref_54","unstructured":"Schmidhuber, J. (1987). Evolutionary Principles in Self-Referential Learning. [Ph.D. Thesis, Technische Universit\u00e4t M\u00fcnchen]."},{"key":"ref_55","first-page":"1240","article-title":"Dual control theory, I","volume":"21","author":"Feldbaum","year":"1960","journal-title":"Avtomat. Telemekh."},{"key":"ref_56","unstructured":"Sutton, R.S., and Barto, A.G. (2015). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]."},{"key":"ref_57","unstructured":"Xu, W., Chen, R.T.Q., Li, X., and Duvenaud, D. (2022). Infinitely deep Bayesian neural networks with stochastic differential equations. arXiv."},{"key":"ref_58","unstructured":"McMahan, H.B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20\u201322). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR, Fort Lauderdale, FL, USA."},{"key":"ref_59","first-page":"2951","article-title":"Practical Bayesian optimization of machine learning algorithms","volume":"25","author":"Snoek","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","article-title":"Taking the human out of the loop: A review of Bayesian optimization","volume":"104","author":"Shahriari","year":"2015","journal-title":"Proc. IEEE"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1023\/A:1015059928466","article-title":"Evolution strategies\u2014A comprehensive introduction","volume":"1","author":"Beyer","year":"2002","journal-title":"Nat. Comput."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1038\/nrn2258","article-title":"Noise in the nervous system","volume":"9","author":"Faisal","year":"2008","journal-title":"Nat. Rev.. Neurosci."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"6454","DOI":"10.1038\/ncomms7454","article-title":"Choice-correlated activity fluctuations underlie learning of neuronal category representation","volume":"6","author":"Engel","year":"2015","journal-title":"Nat. Commun."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1038\/nrn2634","article-title":"The probability of neurotransmitter release: Variability and feedback control at single synapses","volume":"10","author":"Branco","year":"2009","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Rolls, E.T., and Deco, G. (2010). The Noisy Brain: Stochastic Dynamics as a Principle of Brain Function, Oxford University Press.","DOI":"10.1093\/acprof:oso\/9780199587865.001.0001"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"23","DOI":"10.31887\/DCNS.2016.18.1\/wschultz","article-title":"Dopamine reward prediction error coding","volume":"18","author":"Schultz","year":"2016","journal-title":"Dialogues Clin. Neurosci."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1038\/s42254-020-0208-2","article-title":"Physics for neuromorphic computing","volume":"2","author":"Markovic","year":"2020","journal-title":"Nat. Rev. Phys."},{"key":"ref_68","first-page":"021045","article-title":"Supervised learning in physical networks: From machine learning to learning machines","volume":"11","author":"Stern","year":"2021","journal-title":"Phys. Rev. X"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"7847","DOI":"10.1038\/s41467-022-35216-2","article-title":"Physical deep learning with biologically inspired training method: Gradient-free approach for physical hardware","volume":"13","author":"Nakajima","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_70","first-page":"031020","article-title":"Self-learning machines based on Hamiltonian echo backpropagation","volume":"13","author":"Marquardt","year":"2023","journal-title":"Phys. Rev. X"},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1126\/science.adi8474","article-title":"Backpropagation-free training of deep physical neural networks","volume":"382","author":"Momeni","year":"2023","journal-title":"Science"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"8999","DOI":"10.1126\/sciadv.ado8999","article-title":"Hardware implementation of backpropagation using progressive gradient descent for in situ training of multilayer neural networks","volume":"10","author":"Stevens","year":"2024","journal-title":"Sci. Adv."},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1038\/s41586-021-03453-y","article-title":"The rise of intelligent matter","volume":"594","author":"Kaspar","year":"2021","journal-title":"Nature"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"100323","DOI":"10.1016\/j.array.2023.100323","article-title":"Advancements in spiking neural network communication and synchronization techniques for event-driven neuromorphic systems","volume":"20","author":"Shahsavari","year":"2023","journal-title":"Array"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/12\/1125\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:57:58Z","timestamp":1760115478000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/12\/1125"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,22]]},"references-count":74,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["e26121125"],"URL":"https:\/\/doi.org\/10.3390\/e26121125","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,22]]}}}