{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T16:55:26Z","timestamp":1790528126569,"version":"4.1.0"},"reference-count":129,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T00:00:00Z","timestamp":1652227200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["ARO W911NF-19-1-0402"],"award-info":[{"award-number":["ARO W911NF-19-1-0402"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["URAP Supplement"],"award-info":[{"award-number":["URAP Supplement"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000055","name":"National Institute on Deafness and Other Communication Disorders","doi-asserted-by":"publisher","award":["R01DC012947"],"award-info":[{"award-number":["R01DC012947"]}],"id":[{"id":"10.13039\/100000055","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000055","name":"National Institute on Deafness and Other Communication Disorders","doi-asserted-by":"publisher","award":["R01DC012947"],"award-info":[{"award-number":["R01DC012947"]}],"id":[{"id":"10.13039\/100000055","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24EB028998"],"award-info":[{"award-number":["U24EB028998"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000070","name":"National Institute of Biomedical Imaging and Bioengineering","doi-asserted-by":"publisher","award":["U01EB017695"],"award-info":[{"award-number":["U01EB017695"]}],"id":[{"id":"10.13039\/100000070","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1904444-1042C"],"award-info":[{"award-number":["1904444-1042C"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","award":["Cloud Platform Research Credits"],"award-info":[{"award-number":["Cloud Platform Research Credits"]}],"id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","award":["Cloud Platform Research Credits"],"award-info":[{"award-number":["Cloud Platform Research Credits"]}],"id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual\/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.<\/jats:p>","DOI":"10.1371\/journal.pone.0265808","type":"journal-article","created":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T13:59:23Z","timestamp":1652277563000},"page":"e0265808","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":13,"title":["Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3079-4812","authenticated-orcid":true,"given":"Haroon","family":"Anwar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0057-1060","authenticated-orcid":true,"given":"Simon","family":"Caby","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salvador","family":"Dura-Bernal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9665-4401","authenticated-orcid":true,"given":"David","family":"D\u2019Onofrio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2274-2999","authenticated-orcid":true,"given":"Daniel","family":"Hasegan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7494-4555","authenticated-orcid":true,"given":"Matt","family":"Deible","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara","family":"Grunblatt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George L.","family":"Chadderdon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cliff C.","family":"Kerr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Lakatos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William W.","family":"Lytton","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1446-1628","authenticated-orcid":true,"given":"Hananel","family":"Hazan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3646-5195","authenticated-orcid":true,"given":"Samuel A.","family":"Neymotin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,5,11]]},"reference":[{"key":"pone.0265808.ref001","doi-asserted-by":"crossref","unstructured":"Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning. Proceedings of the AAAI conference on artificial intelligence. 2016. https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/10295","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"pone.0265808.ref002","volume-title":"Reinforcement learning: An introduction","author":"RS Sutton","year":"1998"},{"key":"pone.0265808.ref003","unstructured":"Witty S, Lee JK, Tosch E, Atrey A, Littman M, Jensen D. Measuring and Characterizing Generalization in Deep Reinforcement Learning. arXiv [cs.LG]. 2018. http:\/\/arxiv.org\/abs\/1812.02868"},{"key":"pone.0265808.ref004","unstructured":"Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N. Dueling Network Architectures for Deep Reinforcement Learning. In: Balcan MF, Weinberger KQ, editors. Proceedings of The 33rd International Conference on Machine Learning. New York, New York, USA: PMLR; 2016. pp. 1995\u20132003."},{"key":"pone.0265808.ref005","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"V Mnih","year":"2015","journal-title":"Nature"},{"key":"pone.0265808.ref006","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1016\/j.tics.2019.02.006","article-title":"Reinforcement Learning, Fast and Slow","volume":"23","author":"M Botvinick","year":"2019","journal-title":"Trends Cogn Sci"},{"key":"pone.0265808.ref007","article-title":"The unreasonable effectiveness of deep learning in artificial intelligence","author":"TJ Sejnowski","year":"2020","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0265808.ref008","doi-asserted-by":"crossref","unstructured":"Hayes TL, Krishnan GP, Bazhenov M, Siegelmann HT, Sejnowski TJ, Kanan C. Replay in Deep Learning: Current Approaches and Missing Biological Elements. arXiv [q-bio.NC]. 2021. http:\/\/arxiv.org\/abs\/2104.04132","DOI":"10.1162\/neco_a_01433"},{"key":"pone.0265808.ref009","doi-asserted-by":"crossref","first-page":"94","DOI":"10.3389\/fncom.2016.00094","article-title":"Toward an Integration of Deep Learning and Neuroscience","volume":"10","author":"AH Marblestone","year":"2016","journal-title":"Front Comput Neurosci"},{"key":"pone.0265808.ref010","unstructured":"Gilra A, Gerstner W. Non-linear motor control by local learning in spiking neural networks. In: Dy J, Krause A, editors. Proceedings of the 35th International Conference on Machine Learning. PMLR; 2018. pp. 1773\u20131782."},{"key":"pone.0265808.ref011","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1016\/j.cell.2015.09.029","article-title":"Reconstruction and Simulation of Neocortical Microcircuitry","volume":"163","author":"H Markram","year":"2015","journal-title":"Cell"},{"key":"pone.0265808.ref012","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1038\/nn.4241","article-title":"Building functional networks of spiking model neurons","volume":"19","author":"LF Abbott","year":"2016","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref013","doi-asserted-by":"crossref","first-page":"e47251","DOI":"10.1371\/journal.pone.0047251","article-title":"Reinforcement learning of targeted movement in a spiking neuronal model of motor cortex","volume":"7","author":"GL Chadderdon","year":"2012","journal-title":"PLoS One"},{"key":"pone.0265808.ref014","doi-asserted-by":"crossref","first-page":"3263","DOI":"10.1162\/NECO_a_00521","article-title":"Reinforcement learning of two-joint virtual arm reaching in a computer model of sensorimotor cortex","volume":"25","author":"SA Neymotin","year":"2013","journal-title":"Neural Comput"},{"key":"pone.0265808.ref015","doi-asserted-by":"crossref","first-page":"e1004792","DOI":"10.1371\/journal.pcbi.1004792","article-title":"Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework","volume":"12","author":"HF Song","year":"2016","journal-title":"PLoS Comput Biol"},{"key":"pone.0265808.ref016","doi-asserted-by":"crossref","first-page":"1468","DOI":"10.1162\/neco.2007.19.6.1468","article-title":"Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity","volume":"19","author":"RV Florian","year":"2007","journal-title":"Neural Comput"},{"key":"pone.0265808.ref017","doi-asserted-by":"crossref","DOI":"10.1523\/ENEURO.0427-20.2020","article-title":"PsychRNN: An Accessible and Flexible Python Package for Training Recurrent Neural Network Models on Cognitive Tasks","volume":"8","author":"DB Ehrlich","year":"2021","journal-title":"eNeuro"},{"key":"pone.0265808.ref018","unstructured":"Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY. On optimization methods for deep learning. ICML. 2011. https:\/\/openreview.net\/pdf?id=Sk4lD3W_bB"},{"key":"pone.0265808.ref019","doi-asserted-by":"crossref","unstructured":"Stork DG. Is backpropagation biologically plausible? International 1989 Joint Conference on Neural Networks. ieeexplore.ieee.org; 1989. pp. 241\u2013246 vol.2.","DOI":"10.1109\/IJCNN.1989.118705"},{"key":"pone.0265808.ref020","doi-asserted-by":"crossref","first-page":"R496","DOI":"10.1016\/j.cub.2007.04.024","article-title":"Recurrent neuronal circuits in the neocortex","volume":"17","author":"RJ Douglas","year":"2007","journal-title":"Curr Biol"},{"key":"pone.0265808.ref021","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1038\/nn.3917","article-title":"The neocortical circuit: themes and variations","volume":"18","author":"KD Harris","year":"2015","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref022","first-page":"114","volume-title":"The Quarterly Review of Biology","author":"RW Stackman","year":"2001"},{"key":"pone.0265808.ref023","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1177\/1073858416645088","article-title":"Sensorimotor Convergence in Circuitry of the Motor Cortex","volume":"23","author":"BM Hooks","year":"2017","journal-title":"Neuroscientist"},{"key":"pone.0265808.ref024","doi-asserted-by":"crossref","first-page":"e1000572","DOI":"10.1371\/journal.pbio.1000572","article-title":"Laminar analysis of excitatory local circuits in vibrissal motor and sensory cortical areas","volume":"9","author":"BM Hooks","year":"2011","journal-title":"PLoS Biol"},{"key":"pone.0265808.ref025","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.neuron.2011.07.029","article-title":"Long-range neuronal circuits underlying the interaction between sensory and motor cortex","volume":"72","author":"T Mao","year":"2011","journal-title":"Neuron"},{"key":"pone.0265808.ref026","doi-asserted-by":"crossref","first-page":"17373","DOI":"10.1523\/JNEUROSCI.2515-13.2013","article-title":"Distinct balance of excitation and inhibition in an interareal feedforward and feedback circuit of mouse visual cortex","volume":"33","author":"W Yang","year":"2013","journal-title":"J Neurosci"},{"key":"pone.0265808.ref027","doi-asserted-by":"crossref","first-page":"748","DOI":"10.1523\/JNEUROSCI.4338-12.2013","article-title":"Organization of cortical and thalamic input to pyramidal neurons in mouse motor cortex","volume":"33","author":"BM Hooks","year":"2013","journal-title":"J Neurosci"},{"key":"pone.0265808.ref028","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/978-1-4613-2149-1_4","volume-title":"Sensory-Motor Areas and Aspects of Cortical Connectivity","author":"EG Jones","year":"1986"},{"key":"pone.0265808.ref029","doi-asserted-by":"crossref","first-page":"e144","DOI":"10.1371\/journal.pcbi.0020144","article-title":"Mapping information flow in sensorimotor networks","volume":"2","author":"M Lungarella","year":"2006","journal-title":"PLoS Comput Biol"},{"key":"pone.0265808.ref030","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1038\/nature02169","article-title":"Bayesian integration in sensorimotor learning","volume":"427","author":"KP K\u00f6rding","year":"2004","journal-title":"Nature"},{"key":"pone.0265808.ref031","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1038\/nrn3112","article-title":"Principles of sensorimotor learning","volume":"12","author":"DM Wolpert","year":"2011","journal-title":"Nat Rev Neurosci"},{"key":"pone.0265808.ref032","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1146\/annurev-psych-122414-033625","article-title":"Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework","volume":"68","author":"SJ Gershman","year":"2017","journal-title":"Annu Rev Psychol"},{"key":"pone.0265808.ref033","doi-asserted-by":"crossref","first-page":"2443","DOI":"10.1093\/cercor\/bhl152","article-title":"Solving the distal reward problem through linkage of STDP and dopamine signaling","volume":"17","author":"EM Izhikevich","year":"2007","journal-title":"Cereb Cortex"},{"key":"pone.0265808.ref034","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","article-title":"Deep Reinforcement Learning: A Brief Survey","volume":"34","author":"K Arulkumaran","year":"2017","journal-title":"IEEE Signal Process Mag"},{"key":"pone.0265808.ref035","first-page":"101","volume-title":"Advances in Neural Information Processing Systems 7","author":"K Doya","year":"1995"},{"key":"pone.0265808.ref036","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1162\/089976600300015961","article-title":"Reinforcement learning in continuous time and space","volume":"12","author":"K. Doya","year":"2000","journal-title":"Neural Comput"},{"key":"pone.0265808.ref037","first-page":"1008","article-title":"Actor-critic algorithms","author":"VR Konda","year":"2000","journal-title":"Advances in neural information processing systems"},{"key":"pone.0265808.ref038","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"RJ Williams","year":"1992","journal-title":"Mach Learn"},{"key":"pone.0265808.ref039","unstructured":"Schulman J, Moritz P, Levine S, Jordan M, Abbeel P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv [cs.LG]. 2015. http:\/\/arxiv.org\/abs\/1506.02438"},{"key":"pone.0265808.ref040","doi-asserted-by":"crossref","unstructured":"Dura-Bernal S, Chadderdon GL. Virtual musculoskeletal arm and robotic arm driven by a biomimetic model of sensorimotor cortex with reinforcement learning. Signal Processing in. 2013. http:\/\/ieeexplore.ieee.org\/abstract\/document\/6736768\/","DOI":"10.1109\/SPMB.2013.6736768"},{"key":"pone.0265808.ref041","article-title":"In silico hippocampal modeling for multi-target pharmacotherapy in schizophrenia","author":"MA Sherif","journal-title":"npj Schizophrenia"},{"key":"pone.0265808.ref042","article-title":"Ketamine disrupts theta modulation of gamma in a computer model of hippocampus","author":"SA Neymotin","year":"2011","journal-title":"Journal of"},{"key":"pone.0265808.ref043","doi-asserted-by":"crossref","first-page":"740597","DOI":"10.7554\/eLife.51214","article-title":"Human Neocortical Neurosolver (HNN), a new software tool for interpreting the cellular and network origin of human MEG\/EEG data","volume":"9","author":"SA Neymotin","year":"2020","journal-title":"Elife"},{"key":"pone.0265808.ref044","unstructured":"Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, et al. OpenAI Gym. arXiv [cs.LG]. 2016. http:\/\/arxiv.org\/abs\/1606.01540"},{"key":"pone.0265808.ref045","doi-asserted-by":"crossref","first-page":"28","DOI":"10.3389\/fnins.2016.00028","article-title":"Restoring Behavior via Inverse Neurocontroller in a Lesioned Cortical Spiking Model Driving a Virtual Arm","volume":"10","author":"S Dura-Bernal","year":"2016","journal-title":"Front Neurosci"},{"key":"pone.0265808.ref046","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1162\/jocn.2006.18.2.242","article-title":"A large-scale neurocomputational model of task-oriented behavior selection and working memory in prefrontal cortex","volume":"18","author":"GL Chadderdon","year":"2006","journal-title":"J Cogn Neurosci"},{"key":"pone.0265808.ref047","doi-asserted-by":"crossref","first-page":"1616","DOI":"10.1126\/science.1255514","article-title":"A critical time window for dopamine actions on the structural plasticity of dendritic spines","volume":"345","author":"S Yagishita","year":"2014","journal-title":"Science"},{"key":"pone.0265808.ref048","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1016\/j.neuron.2015.09.037","article-title":"Distinct Eligibility Traces for LTP and LTD in Cortical Synapses","volume":"88","author":"K He","year":"2015","journal-title":"Neuron"},{"key":"pone.0265808.ref049","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.09685","article-title":"Retroactive modulation of spike timing-dependent plasticity by dopamine","volume":"4","author":"Z Brzosko","year":"2015","journal-title":"Elife"},{"key":"pone.0265808.ref050","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1126\/science.aan3846","article-title":"Behavioral time scale synaptic plasticity underlies CA1 place fields","volume":"357","author":"KC Bittner","year":"2017","journal-title":"Science"},{"key":"pone.0265808.ref051","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.27756","article-title":"Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation","volume":"6","author":"Z Brzosko","year":"2017","journal-title":"Elife"},{"key":"pone.0265808.ref052","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1111\/ejn.13921","article-title":"A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum","volume":"49","author":"T Shindou","year":"2019","journal-title":"Eur J Neurosci"},{"key":"pone.0265808.ref053","doi-asserted-by":"crossref","first-page":"19","DOI":"10.3389\/fncom.2011.00019","article-title":"Emergence of physiological oscillation frequencies in a computer model of neocortex","volume":"5","author":"SA Neymotin","year":"2011","journal-title":"Front Comput Neurosci"},{"key":"pone.0265808.ref054","first-page":"47","article-title":"A rule-based firing model for neural networks","volume":"7","author":"WW Lytton","year":"2005","journal-title":"Int J Bioelectromagn"},{"key":"pone.0265808.ref055","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1016\/j.neucom.2005.12.066","article-title":"Rule-based firing for network simulations","volume":"69","author":"WW Lytton","year":"2006","journal-title":"Neurocomputing"},{"key":"pone.0265808.ref056","doi-asserted-by":"crossref","first-page":"2745","DOI":"10.1162\/neco.2008.10-07-622","article-title":"Just-in-time connectivity for large spiking networks","volume":"20","author":"WW Lytton","year":"2008","journal-title":"Neural Comput"},{"key":"pone.0265808.ref057","article-title":"The NEURON simulation environment","volume":"9","author":"ML Hines","year":"2006","journal-title":"Neuron"},{"key":"pone.0265808.ref058","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.44494","article-title":"NetPyNE, a tool for data-driven multiscale modeling of brain circuits","volume":"8","author":"S Dura-Bernal","year":"2019","journal-title":"Elife"},{"key":"pone.0265808.ref059","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: an overview","volume":"61","author":"J. Schmidhuber","year":"2015","journal-title":"Neural Netw"},{"key":"pone.0265808.ref060","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2011266118","article-title":"Prioritized experience replays on a hippocampal predictive map for learning","volume":"118","author":"H Igata","year":"2021","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0265808.ref061","doi-asserted-by":"crossref","first-page":"13","DOI":"10.3389\/fnbot.2015.00013","article-title":"Cortical Spiking Network Interfaced with Virtual Musculoskeletal Arm and Robotic Arm","volume":"9","author":"S Dura-Bernal","year":"2015","journal-title":"Front Neurorobot"},{"key":"pone.0265808.ref062","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/1744-9081-1-6","article-title":"Dopamine, uncertainty and TD learning","volume":"1","author":"Y Niv","year":"2005","journal-title":"Behav Brain Funct"},{"key":"pone.0265808.ref063","first-page":"688622","article-title":"Interleaved training prevents catastrophic forgetting in spiking neural networks","author":"R Golden","year":"2019","journal-title":"bioRxiv"},{"key":"pone.0265808.ref064","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1038\/nn2049","article-title":"Top-down laminar organization of the excitatory network in motor cortex","volume":"11","author":"N Weiler","year":"2008","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref065","doi-asserted-by":"crossref","first-page":"39","DOI":"10.3389\/fncom.2014.00039","article-title":"Electrostimulation to reduce synaptic scaling driven progression of Alzheimer\u2019s disease","volume":"8","author":"MS Rowan","year":"2014","journal-title":"Front Comput Neurosci"},{"key":"pone.0265808.ref066","doi-asserted-by":"crossref","unstructured":"Rowan M, Neymotin S. Synaptic scaling balances learning in a spiking model of neocortex. International Conference on Adaptive and Natural. 2013. https:\/\/link.springer.com\/chapter\/10.1007\/978-3-642-37213-1_3","DOI":"10.1007\/978-3-642-37213-1_3"},{"key":"pone.0265808.ref067","doi-asserted-by":"crossref","first-page":"6424","DOI":"10.1073\/pnas.0700622104","article-title":"A feedforward architecture accounts for rapid categorization","volume":"104","author":"T Serre","year":"2007","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0265808.ref068","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1109\/TPAMI.2007.56","article-title":"Robust object recognition with cortex-like mechanisms","volume":"29","author":"T Serre","year":"2007","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"pone.0265808.ref069","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1016\/j.neuron.2012.01.010","article-title":"How does the brain solve visual object recognition?","volume":"73","author":"JJ DiCarlo","year":"2012","journal-title":"Neuron"},{"key":"pone.0265808.ref070","doi-asserted-by":"crossref","first-page":"8107","DOI":"10.1523\/JNEUROSCI.1462-12.2012","article-title":"What visual information is processed in the human dorsal stream?","author":"MN Hebart","year":"2012","journal-title":"The Journal of neuroscience: the official journal of the Society for Neuroscience"},{"key":"pone.0265808.ref071","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0896-6273(94)90455-3","article-title":"Neural mechanisms of form and motion processing in the primate visual system","volume":"13","author":"DC Van Essen","year":"1994","journal-title":"Neuron"},{"key":"pone.0265808.ref072","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1126\/science.1181936","article-title":"How Good Are Neuron Models?","volume":"326","author":"W Gerstner","year":"2009","journal-title":"Science"},{"key":"pone.0265808.ref073","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1002\/aja.1001800203","article-title":"Brain sizes, surfaces, and neuronal sizes of the cortex cerebri: a stereological investigation of man and his variability and a comparison with some mammals (primates, whales, marsupials, insectivores, and one elephant)","volume":"180","author":"H. Haug","year":"1987","journal-title":"Am J Anat"},{"key":"pone.0265808.ref074","first-page":"575","article-title":"Computational models for generic cortical microcircuits","volume":"18","author":"W Maass","year":"2004","journal-title":"Computational neuroscience: A comprehensive approach"},{"key":"pone.0265808.ref075","doi-asserted-by":"crossref","first-page":"8441","DOI":"10.1523\/JNEUROSCI.1400-04.2004","article-title":"A quantitative map of the circuit of cat primary visual cortex","volume":"24","author":"T Binzegger","year":"2004","journal-title":"J Neurosci"},{"key":"pone.0265808.ref076","article-title":"Reservoir computing approaches to recurrent neural network training","author":"M Luko\u0161evi\u010dius","year":"2009","journal-title":"Computer Science Review"},{"key":"pone.0265808.ref077","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1038\/s41586-020-03157-9","article-title":"First return, then explore","volume":"590","author":"A Ecoffet","year":"2021","journal-title":"Nature"},{"key":"pone.0265808.ref078","doi-asserted-by":"crossref","unstructured":"Cornford J, Kalajdzievski D, Leite M, Lamarquette A, Kullmann DM, Richards BA. Learning to live with Dale\u2019s principle: ANNs with separate excitatory and inhibitory units. 2020. https:\/\/openreview.net\/pdf?id=eU776ZYxEpz","DOI":"10.1101\/2020.11.02.364968"},{"key":"pone.0265808.ref079","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.neuron.2004.09.007","article-title":"Spike timing-dependent plasticity of neural circuits","volume":"44","author":"Y Dan","year":"2004","journal-title":"Neuron"},{"key":"pone.0265808.ref080","first-page":"20719","article-title":"Meta-learning through Hebbian plasticity in random networks","volume":"33","author":"E Najarro","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"pone.0265808.ref081","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1038\/nature14467","article-title":"Impermanence of dendritic spines in live adult CA1 hippocampus","volume":"523","author":"A Attardo","year":"2015","journal-title":"Nature"},{"key":"pone.0265808.ref082","doi-asserted-by":"crossref","first-page":"774","DOI":"10.3389\/fnins.2018.00774","article-title":"Deep Learning With Spiking Neurons: Opportunities and Challenges","volume":"12","author":"M Pfeiffer","year":"2018","journal-title":"Front Neurosci"},{"key":"pone.0265808.ref083","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.neuron.2005.01.003","article-title":"Transient and persistent dendritic spines in the neocortex in vivo","volume":"45","author":"AJGD Holtmaat","year":"2005","journal-title":"Neuron"},{"key":"pone.0265808.ref084","doi-asserted-by":"crossref","first-page":"12535","DOI":"10.1523\/JNEUROSCI.2917-14.2015","article-title":"Predicting the Dynamics of Network Connectivity in the Neocortex","volume":"35","author":"Y Loewenstein","year":"2015","journal-title":"J Neurosci"},{"key":"pone.0265808.ref085","doi-asserted-by":"crossref","first-page":"13592","DOI":"10.1523\/JNEUROSCI.0603-08.2008","article-title":"Principles of long-term dynamics of dendritic spines","volume":"28","author":"N Yasumatsu","year":"2008","journal-title":"J Neurosci"},{"key":"pone.0265808.ref086","doi-asserted-by":"crossref","first-page":"9481","DOI":"10.1523\/JNEUROSCI.6130-10.2011","article-title":"Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo","volume":"31","author":"Y Loewenstein","year":"2011","journal-title":"J Neurosci"},{"key":"pone.0265808.ref087","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.conb.2019.08.005","article-title":"Causes and consequences of representational drift","volume":"58","author":"ME Rule","year":"2019","journal-title":"Curr Opin Neurobiol"},{"key":"pone.0265808.ref088","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1016\/j.conb.2017.06.006","article-title":"Intrinsic volatility of synaptic connections\u2014a challenge to the synaptic trace theory of memory","author":"G Mongillo","year":"2017","journal-title":"Current Opinion in Neurobiology"},{"key":"pone.0265808.ref089","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.tins.2017.12.003","article-title":"Synaptic Tenacity or Lack Thereof: Spontaneous Remodeling of Synapses","volume":"41","author":"NE Ziv","year":"2018","journal-title":"Trends Neurosci"},{"key":"pone.0265808.ref090","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.62912","article-title":"Optimal plasticity for memory maintenance during ongoing synaptic change","volume":"10","author":"DV Raman","year":"2021","journal-title":"Elife"},{"key":"pone.0265808.ref091","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.conb.2020.12.017","article-title":"Frozen algorithms: how the brain\u2019s wiring facilitates learning","volume":"67","author":"DV Raman","year":"2021","journal-title":"Curr Opin Neurobiol"},{"key":"pone.0265808.ref092","doi-asserted-by":"crossref","DOI":"10.1523\/ENEURO.0301-17.2018","article-title":"A Dynamic Connectome Supports the Emergence of Stable Computational Function of Neural Circuits through Reward-Based Learning","volume":"5","author":"D Kappel","year":"2018","journal-title":"eNeuro"},{"key":"pone.0265808.ref093","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1152\/jn.00534.2018","article-title":"Stable memory and computation in randomly rewiring neural networks","volume":"122","author":"D Acker","year":"2019","journal-title":"J Neurophysiol"},{"key":"pone.0265808.ref094","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.43717","article-title":"Self-organized reactivation maintains and reinforces memories despite synaptic turnover","volume":"8","author":"MJ Fauth","year":"2019","journal-title":"Elife"},{"key":"pone.0265808.ref095","article-title":"Multi-timescale biological learning algorithms train spiking neuronal network motor control","author":"D Hasegan","year":"2021","journal-title":"bioRxiv"},{"key":"pone.0265808.ref096","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1016\/j.neuron.2016.10.029","article-title":"Circuit Mechanisms of Sensorimotor Learning","volume":"92","author":"H Makino","year":"2016","journal-title":"Neuron"},{"key":"pone.0265808.ref097","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1177\/1073858420939552","article-title":"Multiple Motor Learning Processes in Humans: Defining Their Neurophysiological Bases","volume":"27","author":"D Spampinato","year":"2021","journal-title":"Neuroscientist"},{"key":"pone.0265808.ref098","doi-asserted-by":"crossref","first-page":"12008","DOI":"10.1038\/s41598-020-68928-w","article-title":"Activation of cerebellum and basal ganglia during the observation and execution of manipulative actions","volume":"10","author":"A Errante","year":"2020","journal-title":"Sci Rep"},{"key":"pone.0265808.ref099","doi-asserted-by":"crossref","first-page":"e0214926","DOI":"10.1371\/journal.pone.0214926","article-title":"The interplay between cerebellum and basal ganglia in motor adaptation: A modeling study","volume":"14","author":"DI Todorov","year":"2019","journal-title":"PLoS One"},{"key":"pone.0265808.ref100","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.jmp.2008.12.005","article-title":"Reinforcement learning in the brain","volume":"53","author":"Y. Niv","year":"2009","journal-title":"J Math Psychol"},{"key":"pone.0265808.ref101","doi-asserted-by":"crossref","first-page":"16311","DOI":"10.1073\/pnas.0706111104","article-title":"Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning","volume":"104","author":"MJ Frank","year":"2007","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0265808.ref102","doi-asserted-by":"crossref","first-page":"2366","DOI":"10.1038\/npp.2014.84","article-title":"Role of dopamine D2 receptors in human reinforcement learning","volume":"39","author":"C Eisenegger","year":"2014","journal-title":"Neuropsychopharmacology"},{"key":"pone.0265808.ref103","article-title":"Heads for learning, tails for memory: reward, reinforcement and a role of dopamine in determining behavioral relevance across multiple timescales","author":"M Baudonnat","year":"2013","journal-title":"Frontiers in Neuroscience"},{"key":"pone.0265808.ref104","doi-asserted-by":"crossref","first-page":"1211","DOI":"10.1523\/JNEUROSCI.1901-15.2016","article-title":"Variability in Dopamine Genes Dissociates Model-Based and Model-Free Reinforcement Learning","volume":"36","author":"BB Doll","year":"2016","journal-title":"J Neurosci"},{"key":"pone.0265808.ref105","unstructured":"Krishnan GP, Tadros T, Ramyaa R, Bazhenov M. Biologically inspired sleep algorithm for artificial neural networks. arXiv [cs.NE]. 2019. http:\/\/arxiv.org\/abs\/1908.02240"},{"key":"pone.0265808.ref106","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1038\/s41586-021-03993-3","article-title":"The mouse cortico\u2013basal ganglia\u2013thalamic network","volume":"598","author":"NN Foster","year":"2021","journal-title":"Nature"},{"key":"pone.0265808.ref107","doi-asserted-by":"crossref","first-page":"3200","DOI":"10.1016\/j.cub.2020.05.077","article-title":"Input Connectivity Reveals Additional Heterogeneity of Dopaminergic Reinforcement in Drosophila","volume":"30","author":"N Otto","year":"2020","journal-title":"Curr Biol"},{"key":"pone.0265808.ref108","doi-asserted-by":"crossref","first-page":"e04580","DOI":"10.7554\/eLife.04580","article-title":"Mushroom body output neurons encode valence and guide memory-based action selection in Drosophila","volume":"3","author":"Y Aso","year":"2014","journal-title":"Elife"},{"key":"pone.0265808.ref109","doi-asserted-by":"crossref","first-page":"3500","DOI":"10.1016\/j.neuron.2021.08.024","article-title":"A dopamine gradient controls access to distributed working memory in the large-scale monkey cortex","volume":"109","author":"S Froudist-Walsh","year":"2021","journal-title":"Neuron"},{"key":"pone.0265808.ref110","doi-asserted-by":"crossref","first-page":"e1005705","DOI":"10.1371\/journal.pcbi.1005705","article-title":"Multi-layer network utilizing rewarded spike time dependent plasticity to learn a foraging task","volume":"13","author":"P Sanda","year":"2017","journal-title":"PLoS Comput Biol"},{"key":"pone.0265808.ref111","doi-asserted-by":"crossref","first-page":"3648","DOI":"10.1152\/jn.00364.2007","article-title":"Reinforcement Learning With Modulated Spike Timing\u2013Dependent Synaptic Plasticity","volume":"98","author":"MA Farries","year":"2007","journal-title":"J Neurophysiol"},{"key":"pone.0265808.ref112","doi-asserted-by":"crossref","first-page":"8812","DOI":"10.1523\/JNEUROSCI.20-23-08812.2000","article-title":"Stable Hebbian learning from spike timing-dependent plasticity","volume":"20","author":"MCW Van Rossum","year":"2000","journal-title":"Journal of neuroscience"},{"key":"pone.0265808.ref113","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1016\/j.cell.2008.10.008","article-title":"The self-tuning neuron: synaptic scaling of excitatory synapses","volume":"135","author":"GG Turrigiano","year":"2008","journal-title":"Cell"},{"key":"pone.0265808.ref114","doi-asserted-by":"crossref","first-page":"2274","DOI":"10.1016\/j.cub.2021.03.024","article-title":"Homeostatic synaptic scaling establishes the specificity of an associative memory","volume":"31","author":"C-H Wu","year":"2021","journal-title":"Curr Biol"},{"key":"pone.0265808.ref115","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1038\/nn.3616","article-title":"Temporal structure of motor variability is dynamically regulated and predicts motor learning ability","volume":"17","author":"HG Wu","year":"2014","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref116","doi-asserted-by":"crossref","first-page":"1525","DOI":"10.1098\/rstb.1997.0139","article-title":"Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles","volume":"352","author":"YL Qin","year":"1997","journal-title":"Philos Trans R Soc Lond B Biol Sci"},{"key":"pone.0265808.ref117","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1038\/nn1825","article-title":"Coordinated memory replay in the visual cortex and hippocampus during sleep","volume":"10","author":"D Ji","year":"2007","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref118","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.1038\/nn1961","article-title":"Forward and reverse hippocampal place-cell sequences during ripples","volume":"10","author":"K Diba","year":"2007","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref119","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1038\/nn.2732","article-title":"Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval","volume":"14","author":"MF Carr","year":"2011","journal-title":"Nat Neurosci"},{"key":"pone.0265808.ref120","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1016\/j.neuron.2010.01.034","article-title":"Hippocampal replay is not a simple function of experience","volume":"65","author":"AS Gupta","year":"2010","journal-title":"Neuron"},{"key":"pone.0265808.ref121","doi-asserted-by":"crossref","first-page":"4069","DOI":"10.1038\/s41467-020-17866-2","article-title":"Brain-inspired replay for continual learning with artificial neural networks","volume":"11","author":"GM van de Ven","year":"2020","journal-title":"Nat Commun"},{"key":"pone.0265808.ref122","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","article-title":"Continual lifelong learning with neural networks: A review","volume":"113","author":"GI Parisi","year":"2019","journal-title":"Neural Netw"},{"key":"pone.0265808.ref123","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.51005","article-title":"Can sleep protect memories from catastrophic forgetting?","volume":"9","author":"OC Gonz\u00e1lez","year":"2020","journal-title":"Elife"},{"key":"pone.0265808.ref124","doi-asserted-by":"crossref","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"J Kirkpatrick","year":"2017","journal-title":"Proceedings of the national academy of sciences"},{"key":"pone.0265808.ref125","unstructured":"Kaplanis C, Shanahan M, Clopath C. Continual reinforcement learning with complex synapses. arXiv preprint arXiv:180207239. 2018. http:\/\/arxiv.org\/abs\/1802.07239"},{"key":"pone.0265808.ref126","article-title":"A rapid and efficient learning rule for biological neural circuits","author":"E Sezener","year":"2021","journal-title":"bioRxiv"},{"key":"pone.0265808.ref127","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.neuron.2005.05.020","article-title":"Midbrain dopamine neurons encode a quantitative reward prediction error signal","volume":"47","author":"HM Bayer","year":"2005","journal-title":"Neuron"},{"key":"pone.0265808.ref128","doi-asserted-by":"crossref","first-page":"e1006897","DOI":"10.1371\/journal.pcbi.1006897","article-title":"Deep convolutional models improve predictions of macaque V1 responses to natural images","volume":"15","author":"SA Cadena","year":"2019","journal-title":"PLoS Comput Biol"},{"key":"pone.0265808.ref129","article-title":"Data-driven model of auditory thalamocortical system rhythms","volume":"2019","author":"E. Y. Griffith","year":"2019","journal-title":"Society for Neuroscience"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0265808","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,4]],"date-time":"2023-02-04T18:04:08Z","timestamp":1675533848000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0265808"}},"subtitle":[],"editor":[{"given":"Gennady S.","family":"Cymbalyuk","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,5,11]]},"references-count":129,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5,11]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0265808","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.07.29.454361","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,11]]}}}