{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T07:58:42Z","timestamp":1779350322469,"version":"3.51.4"},"reference-count":45,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,5,19]],"date-time":"2022-05-19T00:00:00Z","timestamp":1652918400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP17H06039"],"award-info":[{"award-number":["JP17H06039"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP18KT0021"],"award-info":[{"award-number":["JP18KT0021"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP19H04998"],"award-info":[{"award-number":["JP19H04998"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP20H00001"],"award-info":[{"award-number":["JP20H00001"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP20H00625"],"award-info":[{"award-number":["JP20H00625"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP16H06568"],"award-info":[{"award-number":["JP16H06568"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP18H02542"],"award-info":[{"award-number":["JP18H02542"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP21H05694"],"award-info":[{"award-number":["JP21H05694"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","award":["JPMJCR16E2"],"award-info":[{"award-number":["JPMJCR16E2"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","award":["JPMJCR21P4"],"award-info":[{"award-number":["JPMJCR21P4"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009619","name":"Japan Agency for Medical Research and Development","doi-asserted-by":"publisher","award":["JP21wm0425010"],"award-info":[{"award-number":["JP21wm0425010"]}],"id":[{"id":"10.13039\/100009619","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009666","name":"Salt Science Research Foundation","doi-asserted-by":"publisher","award":["No. 2137"],"award-info":[{"award-number":["No. 2137"]}],"id":[{"id":"10.13039\/100009666","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004330","name":"Smoking Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004330","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008667","name":"SENSHIN Medical Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100008667","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurosci."],"abstract":"<jats:p>Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation <jats:italic>via<\/jats:italic> sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.<\/jats:p>","DOI":"10.3389\/fnins.2022.857009","type":"journal-article","created":{"date-parts":[[2022,5,19]],"date-time":"2022-05-19T05:06:35Z","timestamp":1652936795000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite"],"prefix":"10.3389","volume":"16","author":[{"given":"Yuuki","family":"Uchida","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Takatoshi","family":"Hikida","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuichi","family":"Yamashita","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2022,5,19]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/j.cell.2019.11.040","article-title":"Neural control and modulation of thirst, sodium appetite, and hunger.","volume":"180","author":"Augustine","year":"2020","journal-title":"Cell"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1111\/j.1460-9568.2012.07990.x","article-title":"From prediction error to incentive salience: mesolimbic computation of reward motivation.","volume":"35","author":"Berridge","year":"2012","journal-title":"Eur. J. Neurosci."},{"key":"B3","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1038\/nrn2400","article-title":"Central mechanisms of osmosensation and systemic osmoregulation.","volume":"9","author":"Bourque","year":"2008","journal-title":"Nat. Rev. Neurosci."},{"key":"B4","doi-asserted-by":"publisher","first-page":"1240","DOI":"10.1038\/nn.3767","article-title":"Central amygdala PKC-\u03b4+ neurons mediate the influence of multiple anorexigenic signals.","volume":"17","author":"Cai","year":"2014","journal-title":"Nat. Neurosci."},{"key":"B5","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1152\/physrev.1929.9.3.399","article-title":"Organization for physiological homeostasis.","volume":"9","author":"Cannon","year":"1929","journal-title":"Physiol. Rev."},{"key":"B6","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1016\/S0091-6773(78)90803-9","article-title":"Salivary sodium and potassium concentrations in adrenalectomized rats.","volume":"24","author":"Catalanotto","year":"1978","journal-title":"Behav. Biol."},{"key":"B7","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1038\/nature08783","article-title":"The cells and peripheral representation of sodium taste in mice.","volume":"464","author":"Chandrashekar","year":"2010","journal-title":"Nature"},{"key":"B8","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1037\/h0040857","article-title":"Performance as a joint function of amount of reinforcement and inter-reinforcement interval.","volume":"57","author":"Collier","year":"1959","journal-title":"J. Exp. Psychol."},{"key":"B9","doi-asserted-by":"publisher","first-page":"1943","DOI":"10.1073\/pnas.151964311","article-title":"Physiological state gates acquisition and expression of mesolimbic reward prediction signals","volume":"113","author":"Cone","year":"2016","journal-title":"Proc. Natl. Acad. Sci. U.S.A."},{"key":"B10","author":"Eaton","year":"2009","journal-title":"Vander\u2019s Renal Physiology."},{"key":"B11","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/chemse\/bjt059","article-title":"Taste bud homeostasis in health, disease, and aging.","volume":"39","author":"Feng","year":"2014","journal-title":"Chem. Sens."},{"key":"B12","doi-asserted-by":"publisher","first-page":"778.e","DOI":"10.1016\/j.neuron.2020.03.009","article-title":"Postingestive modulation of food seeking depends on vagus-mediated dopamine neuron activity.","volume":"106","author":"Fernandes","year":"2020","journal-title":"Neuron"},{"key":"B13","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1038\/nrn2787","article-title":"The free-energy principle: A unified brain theory?","volume":"11","author":"Friston","year":"2010","journal-title":"Nat. Rev. Neurosci."},{"key":"B14","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/0166-4328(93)90146-H","article-title":"Lesions of the central nucleus of the amygdala I: effects on taste reactivity, taste aversion learning and sodium appetite.","volume":"59","author":"Galaverna","year":"1993","journal-title":"Behav. Brain Res."},{"key":"B15","doi-asserted-by":"publisher","first-page":"1037.e","DOI":"10.1016\/j.neuron.2019.03.037","article-title":"Central amygdala Prepronociceptin-expressing neurons mediate palatable food consumption and reward.","volume":"102","author":"Hardaway","year":"2019","journal-title":"Neuron"},{"key":"B16","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1037\/h0044372","article-title":"Saccharine reinforcement in a T maze.","volume":"50","author":"Hughes","year":"1957","journal-title":"J. Comp. Physiol. Psychol."},{"key":"B17","author":"Hull","year":"1943","journal-title":"Principles of Behavior: An Introduction to Behavior Theory."},{"key":"B18","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.plrev.2019.07.005","article-title":"Neurocomputational theories of homeostatic control.","volume":"31","author":"Hulme","year":"2019","journal-title":"Phys. Life Rev."},{"key":"B19","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1038\/s41586-021-04359-5","article-title":"Sensory representation and detection mechanisms of gut osmolality change.","volume":"602","author":"Ichiki","year":"2022","journal-title":"Nature"},{"key":"B20","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1037\/rev0000046","article-title":"Cocaine addiction as a homeostatic reinforcement learning disorder.","volume":"124","author":"Keramati","year":"2017","journal-title":"Psychol. Rev."},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04811","article-title":"Homeostatic reinforcement learning for integrating reward collection and physiological stability.","volume":"3","author":"Keramati","year":"2014","journal-title":"eLife"},{"key":"B22","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1038\/s41586-019-1053-2","article-title":"Chemosensory modulation of neural circuits for sodium appetite.","volume":"568","author":"Lee","year":"2019","journal-title":"Nature"},{"key":"B23","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262514620.001.0001","author":"Marr","year":"2010","journal-title":"Vision: A Computational Investigation Into the Human Representation and Processing of Visual Information."},{"key":"B24","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1038\/nn.4463","article-title":"Distinct neural mechanisms for the control of thirst and salt appetite in the subfornical organ.","volume":"20","author":"Matsuda","year":"2017","journal-title":"Nat. Neurosci."},{"key":"B25","doi-asserted-by":"publisher","first-page":"987","DOI":"10.1016\/0031-9384(69)90054-7","article-title":"Separation of satiating and rewarding consequences of drinking.","volume":"4","author":"McFarland","year":"1969","journal-title":"Physiol. Behav."},{"key":"B26","author":"Melmed","year":"2019","journal-title":"Williams Textbook of Endocrinology"},{"key":"B27","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.neuroscience.2014.04.026","article-title":"Distinct dopaminergic control of the direct and indirect pathways in reward-based and avoidance learning behaviors.","volume":"282","author":"Nakanishi","year":"2014","journal-title":"Neuroscience"},{"key":"B28","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1038\/nature11905","article-title":"High salt recruits aversive taste pathways.","volume":"494","author":"Oka","year":"2013","journal-title":"Nature"},{"key":"B29","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1038\/s41593-019-0573-2","article-title":"A neural basis for tonic suppression of sodium appetite.","volume":"23","author":"Park","year":"2020","journal-title":"Nat. Neurosci."},{"key":"B30","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.tins.2020.09.012","article-title":"Computational models of interoception and body regulation.","volume":"44","author":"Petzschner","year":"2021","journal-title":"Trends Neurosci."},{"key":"B31","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.pneurobio.2015.09.001","article-title":"Active Inference, homeostatic regulation and adaptive behavioural control.","volume":"134","author":"Pezzulo","year":"2015","journal-title":"Prog. Neurobiol."},{"key":"B32","doi-asserted-by":"publisher","first-page":"190.e","DOI":"10.1016\/j.neuron.2017.09.014","article-title":"Aldosterone-sensing neurons in the NTS exhibit state-dependent pacemaker activity and drive sodium appetite via synergy with angiotensin II signaling.","volume":"96","author":"Resch","year":"2017","journal-title":"Neuron"},{"key":"B33","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1152\/ajplegacy.1936.115.1.155","article-title":"Increased salt appetite in adrenalectomized rats.","volume":"115","author":"Richter","year":"1936","journal-title":"Am. J. Physiol. Legacy Content"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1186\/1475-2891-9-15","article-title":"Influence of taste disorders on dietary behaviors in cancer patients under chemotherapy.","volume":"9","author":"S\u00e1nchez-Lara","year":"2010","journal-title":"Nutr. J."},{"key":"B35","doi-asserted-by":"publisher","first-page":"ENEURO.0064","DOI":"10.1523\/ENEURO.0064-18.2018","article-title":"Phasic stimulation of midbrain dopamine neuron activity reduces salt consumption.","volume":"5","author":"Sandhu","year":"2018","journal-title":"eNeuro"},{"key":"B36","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1152\/physrev.00023.2014","article-title":"Neuronal reward and decision signals: from theories to data.","volume":"95","author":"Schultz","year":"2015","journal-title":"Physiol. Rev."},{"key":"B37","doi-asserted-by":"publisher","first-page":"1899","DOI":"10.1200\/JCO.2008.19.2690","article-title":"Qualitative and quantitative assessment of taste and smell changes in patients undergoing chemotherapy for breast cancer or gynecologic malignancies.","volume":"27","author":"Steinbach","year":"2009","journal-title":"J. Clin. Oncol."},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2016.00550","article-title":"Allostatic self-efficacy: A metacognitive theory of dyshomeostasis-induced fatigue and depression.","volume":"10","author":"Stephan","year":"2016","journal-title":"Front. Hum. Neurosci."},{"key":"B39","author":"Sutton","year":"2018","journal-title":"Reinforcement Learning: An Introduction"},{"key":"B40","doi-asserted-by":"publisher","first-page":"12220","DOI":"10.1523\/JNEUROSCI.2499-09.2009","article-title":"Dynamic computation of incentive salience: \u201cwanting\u201d what was never \u201cliked.\u201d.","volume":"29","author":"Tindell","year":"2009","journal-title":"J. Neurosci."},{"key":"B41","doi-asserted-by":"publisher","first-page":"2399","DOI":"10.1152\/jn.00576.2006","article-title":"Ventral pallidum firing codes hedonic reward: when a bad taste turns good.","volume":"96","author":"Tindell","year":"2006","journal-title":"J. Neurophysiol."},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1101\/2021.04.20.440596.","article-title":"Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite.","author":"Uchida","year":"2021","journal-title":"bioRxiv."},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-37405-w","article-title":"Limbic control over the homeostatic need for sodium.","volume":"9","author":"Verharen","year":"2019","journal-title":"Sci. Rep."},{"key":"B44","doi-asserted-by":"crossref","first-page":"7743","DOI":"10.1523\/JNEUROSCI.20-20-07743.2000","article-title":"Na v2\/NaG channel is involved in control of salt-intake behavior in the CNS.","volume":"20","author":"Watanabe","year":"2000","journal-title":"J. Neurosci."},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000437","article-title":"A neural computational model of incentive salience.","volume":"5","author":"Zhang","year":"2009","journal-title":"PLoS Comput. Biol."}],"container-title":["Frontiers in Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2022.857009\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,19]],"date-time":"2022-05-19T05:06:56Z","timestamp":1652936816000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2022.857009\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,19]]},"references-count":45,"alternative-id":["10.3389\/fnins.2022.857009"],"URL":"https:\/\/doi.org\/10.3389\/fnins.2022.857009","relation":{},"ISSN":["1662-453X"],"issn-type":[{"value":"1662-453X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,19]]},"article-number":"857009"}}