{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T08:59:43Z","timestamp":1790067583555,"version":"4.0.1"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,12,5]],"date-time":"2017-12-05T00:00:00Z","timestamp":1512432000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2017,12,5]],"date-time":"2017-12-05T00:00:00Z","timestamp":1512432000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To flexibly adapt to the demands of their environment, animals are constantly exposed to the conflict resulting from having to choose between predictably rewarding familiar options (exploitation) and risky novel options, the value of which essentially consists of obtaining new information about the space of possible rewards (exploration). Despite extensive research, the mechanisms that subtend the manner in which animals solve this exploitation-exploration dilemma are still poorly understood. Here, we investigate human decision-making in a gambling task in which the informational value of each trial and the reward potential were separately manipulated. To better characterize the mechanisms that underlined the observed behavioural choices, we introduce a computational model that augments the standard reward-based reinforcement learning formulation by associating a value to information. We find that both reward and information gained during learning influence the balance between exploitation and exploration, and that this influence was dependent on the reward context. Our results shed light on the mechanisms that underpin decision-making under uncertainty, and suggest new approaches for investigating the exploration-exploitation dilemma throughout the animal kingdom.<\/jats:p>","DOI":"10.1038\/s41598-017-17237-w","type":"journal-article","created":{"date-parts":[[2017,11,29]],"date-time":"2017-11-29T09:58:41Z","timestamp":1511949521000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["Learning the value of information and reward over time when solving exploration-exploitation problems"],"prefix":"10.1038","volume":"7","author":[{"given":"Irene","family":"Cogliati Dezza","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela J.","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Axel","family":"Cleeremans","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Alexander","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2017,12,5]]},"reference":[{"key":"17237_CR1","doi-asserted-by":"crossref","unstructured":"Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction. (MIT Press, 1998).","DOI":"10.1109\/TNN.1998.712192"},{"key":"17237_CR2","doi-asserted-by":"publisher","first-page":"2074","DOI":"10.1037\/a0038199","volume":"143","author":"RC Wilson","year":"2014","unstructured":"Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A. & Cohen, J. D. Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of experimental psychology. General 143, 2074\u20132081, https:\/\/doi.org\/10.1037\/a0038199 (2014).","journal-title":"Journal of experimental psychology. General"},{"key":"17237_CR3","unstructured":"Gittins, J. & Jones, D. In Progress in statisics (ed J. Gans) 241\u2013266 (The Netherlands: North-Holland, 1974)."},{"key":"17237_CR4","doi-asserted-by":"publisher","first-page":"e1004164","DOI":"10.1371\/journal.pcbi.1004164","volume":"11","author":"BB Averbeck","year":"2015","unstructured":"Averbeck, B. B. Theory of choice in bandit, information sampling and foraging tasks. PLoS computational biology 11, e1004164, https:\/\/doi.org\/10.1371\/journal.pcbi.1004164 (2015).","journal-title":"PLoS computational biology"},{"key":"17237_CR5","doi-asserted-by":"publisher","first-page":"876","DOI":"10.1038\/nature04766","volume":"441","author":"ND Daw","year":"2006","unstructured":"Daw, N. D., O\u2019Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876\u2013879, https:\/\/doi.org\/10.1038\/nature04766 (2006).","journal-title":"Nature"},{"key":"17237_CR6","doi-asserted-by":"publisher","unstructured":"Payzan-LeNestour, E. & Bossaerts, P. Risk, unexpected uncertainty,and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, e1001048, https:\/\/doi.org\/10.1371\/journal.pcbi.1001048 (2011).","DOI":"10.1371\/journal.pcbi.1001048"},{"key":"17237_CR7","doi-asserted-by":"publisher","first-page":"503","DOI":"10.2307\/1907921","volume":"21","author":"M Allais","year":"1953","unstructured":"Allais, M. Le Comportement de l\u2019Homme Rationnel devant le Risque: Critique des Postulats et Axiomes de l\u2032Ecole Americaine. Econometrica 21, 503\u2013546 (1953).","journal-title":"Econometrica"},{"key":"17237_CR8","doi-asserted-by":"publisher","first-page":"2575","DOI":"10.1093\/cercor\/bhr332","volume":"22","author":"JF Cavanagh","year":"2012","unstructured":"Cavanagh, J. F., Figueroa, C. M., Cohen, M. X. & Frank, M. J. Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb Cortex 22, 2575\u20132586, https:\/\/doi.org\/10.1093\/cercor\/bhr332 (2012).","journal-title":"Cereb Cortex"},{"key":"17237_CR9","doi-asserted-by":"publisher","first-page":"1704","DOI":"10.1038\/nn1560","volume":"8","author":"ND Daw","year":"2005","unstructured":"Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8, 1704\u20131711, https:\/\/doi.org\/10.1038\/nn1560 (2005).","journal-title":"Nat Neurosci"},{"key":"17237_CR10","doi-asserted-by":"publisher","first-page":"263","DOI":"10.2307\/1914185","volume":"47","author":"D Kahneman","year":"1979","unstructured":"Kahneman, D. & Tversky, A. Prospect theory: An analysis of decision under risk. Econometrica 47, 263\u2013291 (1979).","journal-title":"Econometrica"},{"key":"17237_CR11","doi-asserted-by":"publisher","first-page":"1214","DOI":"10.1038\/nn1954","volume":"10","author":"TE Behrens","year":"2007","unstructured":"Behrens, T. E., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. Learning the value of information in an uncertain world. Nat Neurosci 10, 1214\u20131221, https:\/\/doi.org\/10.1038\/nn1954 (2007).","journal-title":"Nat Neurosci"},{"key":"17237_CR12","unstructured":"Bossaerts, P., Preuschoff, K. & Hsu, M. In Neuroeconomics: Decision Making and the Brain (eds Glimcher, P. W., Camerer, C. F., Fehr, E. & Poldrack, R. A.) (Academic Press, 2008)."},{"key":"17237_CR13","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/j.conb.2008.07.010","volume":"18","author":"B Seymour","year":"2008","unstructured":"Seymour, B. & McClure, S. M. Anchors, scales and the relative coding of value in the brain. Curr Opin Neurobiol 18, 173\u2013178, https:\/\/doi.org\/10.1016\/j.conb.2008.07.010 (2008).","journal-title":"Curr Opin Neurobiol"},{"key":"17237_CR14","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1037\/0033-2909.130.5.769","volume":"130","author":"L Green","year":"2004","unstructured":"Green, L. & Myerson, J. A discounting framework for choice with delayed and probabilistic rewards. Psychol Bull 130, 769\u2013792, https:\/\/doi.org\/10.1037\/0033-2909.130.5.769 (2004).","journal-title":"Psychol Bull"},{"key":"17237_CR15","doi-asserted-by":"crossref","unstructured":"Stephens, D. W. & Krebs, J. R. Foraging Theory. (Princeton University Press, 1986).","DOI":"10.1515\/9780691206790"},{"key":"17237_CR16","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1080\/00031305.1981.10479327","volume":"35","author":"WJ Conover","year":"1981","unstructured":"Conover, W. J. & Iman, R. L. Rank transformations as a bridge between parametric and nonparametric statistics. American Statistician 35, 124\u2013129 (1981).","journal-title":"American Statistician"},{"key":"17237_CR17","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1017\/S1930297500005659","volume":"12","author":"PM Krueger","year":"2017","unstructured":"Krueger, P. M. Strategies for exploration in the domain of losses. Judgement and Decision Making 12, 104\u2013117 (2017).","journal-title":"Judgement and Decision Making"},{"key":"17237_CR18","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1214\/aos\/1176344136","volume":"6","author":"G Schwarz","year":"1978","unstructured":"Schwarz, G. Estimating the dimension of a model. Ann. Stat 6, 461\u2013464 (1978).","journal-title":"Ann. Stat"},{"key":"17237_CR19","doi-asserted-by":"crossref","unstructured":"Akaike, H. A new look at the statistical model identification. IEEE Trans. Automatic Control, 716\u2013723 (1974).","DOI":"10.1109\/TAC.1974.1100705"},{"key":"17237_CR20","unstructured":"Kass, R. E. & Raftery, A. E. Bayes factors and model uncertainty. (University of Washington, 1993)."},{"key":"17237_CR21","doi-asserted-by":"publisher","unstructured":"Zajkowski, W. K., Kossut, M. & Wilson, R. C. A causal role for right frontopolar cortex in directed, but not random, exploration. Elife 6, https:\/\/doi.org\/10.7554\/eLife.27430 (2017).","DOI":"10.7554\/eLife.27430"},{"key":"17237_CR22","doi-asserted-by":"publisher","first-page":"1062","DOI":"10.1038\/nn.2342","volume":"12","author":"MJ Frank","year":"2009","unstructured":"Frank, M. J., Doll, B. B., Oas-Terpstra, J. & Moreno, F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 12, 1062\u20131068, https:\/\/doi.org\/10.1038\/nn.2342 (2009).","journal-title":"Nat Neurosci"},{"key":"17237_CR23","doi-asserted-by":"publisher","unstructured":"Payzan-LeNestour, E. & Bossaerts, P. Do not bet on the unknown versus try to find out more: Estimation uncertainty and \u201cunexpected uncertainty\u201d both modulate exploration. Frontiers in Neuroscience, 150, https:\/\/doi.org\/10.3389\/fnins.2012.00150 (2012).","DOI":"10.3389\/fnins.2012.00150"},{"key":"17237_CR24","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1016\/j.tics.2009.09.004","volume":"13","author":"R Hertwig","year":"2009","unstructured":"Hertwig, R. & Erev, I. The description-experience gap in risky choice. Trends Cogn Sci 13, 517\u2013523, https:\/\/doi.org\/10.1016\/j.tics.2009.09.004 (2009).","journal-title":"Trends Cogn Sci"},{"key":"17237_CR25","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1038\/nn2062","volume":"11","author":"ML Platt","year":"2008","unstructured":"Platt, M. L. & Huettel, S. A. Risky business: the neuroeconomics of decision making under uncertainty. Nat Neurosci 11, 398\u2013403, https:\/\/doi.org\/10.1038\/nn2062 (2008).","journal-title":"Nat Neurosci"},{"key":"17237_CR26","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1037\/pag0000146","volume":"32","author":"JA Cooper","year":"2017","unstructured":"Cooper, J. A., Blanco, N. J. & Maddox, W. T. Framing matters: Effects of framing on older adults\u2019 exploratory decision-making. Psychol Aging 32, 60\u201368, https:\/\/doi.org\/10.1037\/pag0000146 (2017).","journal-title":"Psychol Aging"},{"key":"17237_CR27","unstructured":"Stephens, D. W., Couzin, I. & Giraldeau, L. In Cognitive search: evolution, algorithms, and the brain (eds Todd, P. M., Hills, T. T. & Robbins, T.W.) (2012)."},{"key":"17237_CR28","unstructured":"Garivier, A. & Moulines, E. In ALT (ed J. Kivinen, Szepesv\u00e1ri, C., Ukkonen, E., Zeugmann, T.) 174\u2013188 (Springer, Berlin, Heidelberg) (2011)."},{"key":"17237_CR29","doi-asserted-by":"publisher","first-page":"933","DOI":"10.1098\/rstb.2007.2098","volume":"362","author":"JD Cohen","year":"2007","unstructured":"Cohen, J. D., McClure, S. M. & Yu, A. J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc Lond B Biol Sci 362, 933\u2013942, https:\/\/doi.org\/10.1098\/rstb.2007.2098 (2007).","journal-title":"Philos Trans R Soc Lond B Biol Sci"},{"key":"17237_CR30","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1037\/dec0000033","volume":"2","author":"K Mehlhorn","year":"2015","unstructured":"Mehlhorn, K. et al. Unpacking the exploration\u2013exploitation tradeoff: A synthesis of human and animal literatures. Decision 2, 191\u2013215 (2015).","journal-title":"Decision"},{"key":"17237_CR31","unstructured":"Rescorla, R. A. & Wagner, A. R. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. Classical conditioning: Current research and theory 64\u201399 (1972)."},{"key":"17237_CR32","doi-asserted-by":"publisher","first-page":"105","DOI":"10.3389\/fnins.2013.00105","volume":"7","author":"JX O\u2019Reilly","year":"2013","unstructured":"O\u2019Reilly, J. X. Making predictions in a changing world-inference, uncertainty, and learning. Front Neurosci 7, 105, https:\/\/doi.org\/10.3389\/fnins.2013.00105 (2013).","journal-title":"Front Neurosci"},{"key":"17237_CR33","doi-asserted-by":"publisher","first-page":"189","DOI":"10.3389\/fnhum.2011.00189","volume":"5","author":"RC Wilson","year":"2011","unstructured":"Wilson, R. C. & Niv, Y. Inferring relevance in a changing world. Front Hum Neurosci 5, 189, https:\/\/doi.org\/10.3389\/fnhum.2011.00189 (2011).","journal-title":"Front Hum Neurosci"},{"key":"17237_CR34","doi-asserted-by":"publisher","first-page":"9","DOI":"10.3389\/fnins.2012.00009","volume":"6","author":"MD Humphries","year":"2012","unstructured":"Humphries, M. D., Khamassi, M. & Gurney, K. Dopaminergic Control of the Exploration-Exploitation Trade-Off via the Basal Ganglia. Front Neurosci 6, 9, https:\/\/doi.org\/10.3389\/fnins.2012.00009 (2012).","journal-title":"Front Neurosci"},{"key":"17237_CR35","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1016\/B978-0-444-62604-2.00022-8","volume":"202","author":"M Khamassi","year":"2013","unstructured":"Khamassi, M., Enel, P., Dominey, P. F. & Procyk, E. Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. Prog Brain Res 202, 441\u2013464, https:\/\/doi.org\/10.1016\/B978-0-444-62604-2.00022-8 (2013).","journal-title":"Prog Brain Res"}],"updated-by":[{"DOI":"10.1038\/s41598-018-22685-z","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2018,3,6]],"date-time":"2018-03-06T00:00:00Z","timestamp":1520294400000}}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17237-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17237-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17237-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,29]],"date-time":"2024-06-29T05:49:13Z","timestamp":1719640153000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17237-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,5]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,12]]}},"alternative-id":["17237"],"URL":"https:\/\/doi.org\/10.1038\/s41598-017-17237-w","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,12,5]]},"assertion":[{"value":"7 August 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 November 2017","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 December 2017","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2018","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has been fixed in the paper.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"16919"}}