{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:23:04Z","timestamp":1772173384631,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012554","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000}}],"reference-count":32,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T00:00:00Z","timestamp":1730073600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Nottingham Vice Chancellor\u2019s International Scholarship","award":["R00700"],"award-info":[{"award-number":["R00700"]}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Synaptic plasticity enables animals to adapt to their environment, but memory formation can require a substantial amount of metabolic energy, potentially impairing survival. Hence, a neuro-economic dilemma arises whether learning is a profitable investment or not, and the brain must therefore judiciously regulate learning. Indeed, in experiments it was observed that during starvation, Drosophila suppress formation of energy-intensive aversive memories. Here we include energy considerations in a reinforcement learning framework. Simulated flies learned to avoid noxious stimuli through synaptic plasticity in either the energy expensive long-term memory (LTM) pathway, or the decaying anesthesia-resistant memory (ARM) pathway. The objective of the flies is to maximize their lifespan, which is calculated with a hazard function. We find that strategies that switch between the LTM and ARM pathways, based on energy reserve and reward prediction error, prolong lifespan. Our study highlights the significance of energy-regulation of memory pathways and dopaminergic control for adaptive learning and survival. It might also benefit engineering applications of reinforcement learning under resources constraints.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012554","type":"journal-article","created":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T13:46:49Z","timestamp":1730123209000},"page":"e1012554","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":4,"title":["Reinforcement learning when your life depends on it: A neuro-economic theory of learning"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0915-8184","authenticated-orcid":true,"given":"Jiamu","family":"Jiang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4178-0220","authenticated-orcid":true,"given":"Emilie","family":"Foyard","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6525-6814","authenticated-orcid":true,"given":"Mark C. W.","family":"van Rossum","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2024,10,28]]},"reference":[{"issue":"5725","key":"pcbi.1012554.ref001","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1126\/science.1111331","article-title":"A cost of long-term memory in Drosophila","volume":"308","author":"F Mery","year":"2005","journal-title":"Science"},{"issue":"6118","key":"pcbi.1012554.ref002","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1126\/science.1226018","article-title":"To favor survival under food shortage, the brain disables costly memory","volume":"339","author":"PY Pla\u00e7ais","year":"2013","journal-title":"Science"},{"issue":"1","key":"pcbi.1012554.ref003","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/ncomms15510","article-title":"Upregulated energy metabolism in the Drosophila mushroom body is the trigger for long-term memory","volume":"8","author":"PY Placais","year":"2017","journal-title":"Nature Communications"},{"issue":"5673","key":"pcbi.1012554.ref004","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1126\/science.1094932","article-title":"Exclusive consolidated memory phases in drosophila","volume":"304","author":"G Isabel","year":"2004","journal-title":"Science"},{"issue":"1","key":"pcbi.1012554.ref005","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/0092-8674(94)90398-0","article-title":"Genetic dissection of consolidated memory in Drosophila","volume":"79","author":"T Tully","year":"1994","journal-title":"Cell"},{"issue":"6","key":"pcbi.1012554.ref006","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1111\/j.1467-9280.1991.tb00174.x","article-title":"Reflections of the environment in memory","volume":"2","author":"JR Anderson","year":"1991","journal-title":"Psychological science"},{"issue":"6118","key":"pcbi.1012554.ref007","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1126\/science.1226018","article-title":"To favor survival under food shortage, the brain disables costly memory","volume":"339","author":"PY Pla\u00e7ais","year":"2013","journal-title":"Science"},{"key":"pcbi.1012554.ref008","doi-asserted-by":"crossref","first-page":"e2113961119","DOI":"10.1073\/pnas.2113961119","article-title":"Mice exhibit stochastic and efficient action switching during probabilistic decision making","volume":"119","author":"CC Beron","year":"2022","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"key":"pcbi.1012554.ref009","volume-title":"Reliability Engineering and Risk Analysis: A Practical Guide","author":"M Modarres","year":"1999"},{"issue":"2","key":"pcbi.1012554.ref010","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1038\/sj.bjc.6601118","article-title":"Survival analysis part I: basic concepts and first analyses","volume":"89","author":"TG Clark","year":"2003","journal-title":"British journal of cancer"},{"key":"pcbi.1012554.ref011","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107447615","volume-title":"Neuronal Dynamics","author":"W Gerstner","year":"2014"},{"key":"pcbi.1012554.ref012","doi-asserted-by":"crossref","unstructured":"Girard M, Jiang J, van Rossum MCW. Estimating the energy requirements for long term memory formation. arxiv. 2023; p. 2301.09565.","DOI":"10.1101\/2023.01.16.524203"},{"issue":"71","key":"pcbi.1012554.ref013","first-page":"e50068","article-title":"Measurement of lifespan in Drosophila melanogaster","author":"NJ Linford","year":"2013","journal-title":"JoVE (Journal of Visualized Experiments)"},{"issue":"5","key":"pcbi.1012554.ref014","doi-asserted-by":"crossref","first-page":"1482","DOI":"10.1073\/pnas.80.5.1482","article-title":"Reward learning in normal and mutant Drosophila","volume":"80","author":"BL Tempel","year":"1983","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"pcbi.1012554.ref015","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-021-22592-4","article-title":"Learning with reinforcement prediction errors in a model of the Drosophila mushroom body","volume":"12","author":"JE Bennett","year":"2021","journal-title":"Nature communications"},{"key":"pcbi.1012554.ref016","doi-asserted-by":"crossref","first-page":"e04580","DOI":"10.7554\/eLife.04580","article-title":"Mushroom body output neurons encode valence and guide memory-based action selection in Drosophila","volume":"3","author":"Y Aso","year":"2014","journal-title":"Elife"},{"issue":"3","key":"pcbi.1012554.ref017","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.conb.2013.01.005","article-title":"Reinforcement signalling in Drosophila; dopamine does it all after all","volume":"23","author":"S Waddell","year":"2013","journal-title":"Current opinion in neurobiology"},{"key":"pcbi.1012554.ref018","doi-asserted-by":"crossref","first-page":"e04577","DOI":"10.7554\/eLife.04577","article-title":"The neuronal architecture of the mushroom body provides a logic for associative learning","volume":"3","author":"Y Aso","year":"2014","journal-title":"elife"},{"issue":"7","key":"pcbi.1012554.ref019","doi-asserted-by":"crossref","first-page":"1742","DOI":"10.1016\/j.cell.2015.11.019","article-title":"Coordinated and compartmentalized neuromodulation shapes sensory processing in Drosophila","volume":"163","author":"R Cohn","year":"2015","journal-title":"Cell"},{"issue":"2","key":"pcbi.1012554.ref020","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1016\/j.neuron.2015.03.025","article-title":"Activity of defined mushroom body output neurons underlies learned olfactory behavior in Drosophila","volume":"86","author":"D Owald","year":"2015","journal-title":"Neuron"},{"issue":"5","key":"pcbi.1012554.ref021","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1016\/j.neuron.2015.11.003","article-title":"Heterosynaptic plasticity underlies aversive olfactory learning in Drosophila","volume":"88","author":"T Hige","year":"2015","journal-title":"Neuron"},{"key":"pcbi.1012554.ref022","doi-asserted-by":"crossref","first-page":"e50804","DOI":"10.7554\/eLife.50804","article-title":"Energy efficient synaptic plasticity","volume":"9","author":"H Li","year":"2020","journal-title":"Elife"},{"key":"pcbi.1012554.ref023","article-title":"Competitive plasticity to reduce the energetic costs of learning","author":"MCW van Rossum","journal-title":"PloS Comp Bio"},{"issue":"1","key":"pcbi.1012554.ref024","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.neuron.2011.03.012","article-title":"Traces of Drosophila memory","volume":"70","author":"RL Davis","year":"2011","journal-title":"Neuron"},{"issue":"19","key":"pcbi.1012554.ref025","doi-asserted-by":"crossref","first-page":"1647","DOI":"10.1016\/j.cub.2011.08.032","article-title":"Parallel Processing of Appetitive Short- and Long-Term Memories In Drosophila","volume":"21","author":"S Trannoy","year":"2011","journal-title":"Current Biology"},{"issue":"6","key":"pcbi.1012554.ref026","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1016\/j.neuron.2020.03.013","article-title":"Spaced training forms complementary long-term memories of opposite valence in Drosophila","volume":"106","author":"PF Jacob","year":"2020","journal-title":"Neuron"},{"issue":"2","key":"pcbi.1012554.ref027","doi-asserted-by":"crossref","first-page":"e8996","DOI":"10.1371\/journal.pone.0008996","article-title":"Metabolic regulation of neuronal plasticity by the energy sensor AMPK","volume":"5","author":"WB Potter","year":"2010","journal-title":"PloS one"},{"issue":"3","key":"pcbi.1012554.ref028","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1016\/S0163-1047(86)80022-X","article-title":"Glucose modulation of memory storage processing","volume":"45","author":"PE Gold","year":"1986","journal-title":"Behavioral and neural biology"},{"issue":"3","key":"pcbi.1012554.ref029","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1016\/j.neubiorev.2010.09.008","article-title":"Glucose enhancement of human memory: a comprehensive research review of the glucose memory facilitation effect","volume":"35","author":"MA Smith","year":"2011","journal-title":"Neuroscience & Biobehavioral Reviews"},{"issue":"6","key":"pcbi.1012554.ref030","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1101\/lm.321006","article-title":"Dopaminergic modulation of the persistence of one-trial hippocampus-dependent memory","volume":"13","author":"CM O\u2019Carroll","year":"2006","journal-title":"Learning & memory"},{"issue":"5","key":"pcbi.1012554.ref031","doi-asserted-by":"crossref","first-page":"1610","DOI":"10.1523\/JNEUROSCI.2721-09.2010","article-title":"Dopamine and memory: modulation of the persistence of memory for novel hippocampal NMDA receptor-dependent paired associates","volume":"30","author":"I Bethus","year":"2010","journal-title":"Journal of Neuroscience"},{"key":"pcbi.1012554.ref032","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1016\/j.tins.2011.07.006","article-title":"A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP","volume":"34","author":"J Lisman","year":"2011","journal-title":"Trends in neurosciences"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012554","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012554","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T13:47:49Z","timestamp":1730987269000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012554"}},"subtitle":[],"editor":[{"given":"Barbara","family":"Webb","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,10,28]]},"references-count":32,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,10,28]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012554","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.05.08.593165","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,28]]}}}