{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T08:09:43Z","timestamp":1781165383795,"version":"3.54.1"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,12,15]],"date-time":"2017-12-15T00:00:00Z","timestamp":1513296000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2017,12,15]],"date-time":"2017-12-15T00:00:00Z","timestamp":1513296000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>When making a choice with limited information, we explore new features through trial-and-error to learn how they are related. However, few studies have investigated exploratory behaviour when information is limited. In this study, we address, at both the behavioural and neural level, how, when, and why humans explore new feature dimensions to learn a new policy for choosing a state-space. We designed a novel multi-dimensional reinforcement learning task to encourage participants to explore and learn new features, then used a reinforcement learning algorithm to model policy exploration and learning behaviour. Our results provide the first evidence that, when humans explore new feature dimensions, their values are transferred from the previous policy to the new online (active) policy, as opposed to being learned from scratch. We further demonstrated that exploration may be regulated by the level of cognitive ambiguity, and that this process might be controlled by the frontopolar cortex. This opens up new possibilities of further understanding how humans explore new features in an open-space with limited information.<\/jats:p>","DOI":"10.1038\/s41598-017-17687-2","type":"journal-article","created":{"date-parts":[[2017,12,11]],"date-time":"2017-12-11T15:43:14Z","timestamp":1513006994000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Exploring Feature Dimensions to Learn a New Policy in an Uninformed Reinforcement Learning Task"],"prefix":"10.1038","volume":"7","author":[{"given":"Oh-hyeon","family":"Choung","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sang Wan","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5907-3787","authenticated-orcid":false,"given":"Yong","family":"Jeong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2017,12,15]]},"reference":[{"key":"17687_CR1","doi-asserted-by":"publisher","first-page":"876","DOI":"10.1038\/nature04766","volume":"441","author":"ND Daw","year":"2006","unstructured":"Daw, N. D., O\u2019Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature \n                           441, 876\u2013879, https:\/\/doi.org\/10.1038\/nature04766 (2006).","journal-title":"Nature"},{"key":"17687_CR2","doi-asserted-by":"publisher","first-page":"14544","DOI":"10.1523\/JNEUROSCI.2322-15.2015","volume":"35","author":"AR Beharelle","year":"2015","unstructured":"Beharelle, A. R., Polan\u00eda, R., Hare, T. A. & Ruff, C. C. Transcranial Stimulation over Frontopolar Cortex Elucidates the Choice Attributes and Neural Mechanisms Used to Resolve Exploration\u2013Exploitation Trade-Offs. Journal of Neuroscience \n                           35, 14544\u201314556 (2015).","journal-title":"Journal of Neuroscience"},{"key":"17687_CR3","doi-asserted-by":"publisher","first-page":"1481","DOI":"10.1126\/science.1252254","volume":"344","author":"M Donoso","year":"2014","unstructured":"Donoso, M., Collins, A. G. & Koechlin, E. Human cognition. Foundations of human reasoning in the prefrontal cortex. Science \n                           344, 1481\u20131486, https:\/\/doi.org\/10.1126\/science.1252254 (2014).","journal-title":"Science"},{"key":"17687_CR4","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1016\/j.neuron.2015.03.015","volume":"86","author":"NW Schuck","year":"2015","unstructured":"Schuck, N. W. et al. Medial prefrontal cortex predicts internally driven strategy shifts. Neuron \n                           86, 331\u2013340, https:\/\/doi.org\/10.1016\/j.neuron.2015.03.015 (2015).","journal-title":"Neuron"},{"key":"17687_CR5","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1101\/lm.45202","volume":"9","author":"MA Gluck","year":"2002","unstructured":"Gluck, M. A., Shohamy, D. & Myers, C. How do people solve the \u201cweather prediction\u201d task? Individual variability in strategies for probabilistic category learning. Learning & Memory \n                           9, 408\u2013418 (2002).","journal-title":"Learning & Memory"},{"key":"17687_CR6","doi-asserted-by":"publisher","first-page":"8145","DOI":"10.1523\/JNEUROSCI.2978-14.2015","volume":"35","author":"Y Niv","year":"2015","unstructured":"Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J Neurosci \n                           35, 8145\u20138157, https:\/\/doi.org\/10.1523\/JNEUROSCI.2978-14.2015 (2015).","journal-title":"J Neurosci"},{"key":"17687_CR7","doi-asserted-by":"publisher","first-page":"e1004558","DOI":"10.1371\/journal.pcbi.1004558","volume":"11","author":"D Markovic","year":"2015","unstructured":"Markovic, D., Glascher, J., Bossaerts, P., O\u2019Doherty, J. & Kiebel, S. J. Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism. PLoS Comput Biol \n                           11, e1004558, https:\/\/doi.org\/10.1371\/journal.pcbi.1004558 (2015).","journal-title":"PLoS Comput Biol"},{"key":"17687_CR8","doi-asserted-by":"publisher","first-page":"1613","DOI":"10.1038\/nn.3836","volume":"17","author":"LT Hunt","year":"2014","unstructured":"Hunt, L. T., Dolan, R. J. & Behrens, T. E. Hierarchical competitions subserving multi-attribute choice. Nat Neurosci \n                           17, 1613\u20131622, https:\/\/doi.org\/10.1038\/nn.3836 (2014).","journal-title":"Nat Neurosci"},{"key":"17687_CR9","doi-asserted-by":"publisher","first-page":"2087","DOI":"10.1016\/j.visres.2009.05.020","volume":"49","author":"KC Aberg","year":"2009","unstructured":"Aberg, K. C., Tartaglia, E. M. & Herzog, M. H. Perceptual learning with Chevrons requires a minimal number of trials, transfers to untrained directions, but does not require sleep. Vision research \n                           49, 2087\u20132094 (2009).","journal-title":"Vision research"},{"key":"17687_CR10","doi-asserted-by":"publisher","first-page":"13214","DOI":"10.1523\/JNEUROSCI.1246-11.2011","volume":"31","author":"S-L Lim","year":"2011","unstructured":"Lim, S.-L., O\u2019Doherty, J. P. & Rangel, A. The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention. Journal of Neuroscience \n                           31, 13214\u201313223 (2011).","journal-title":"Journal of Neuroscience"},{"key":"17687_CR11","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/j.neuron.2013.11.028","volume":"81","author":"SW Lee","year":"2014","unstructured":"Lee, S. W., Shimojo, S. & O\u2019Doherty, J. P. Neural computations underlying arbitration between model-based and model-free learning. Neuron \n                           81, 687\u2013699, https:\/\/doi.org\/10.1016\/j.neuron.2013.11.028 (2014).","journal-title":"Neuron"},{"key":"17687_CR12","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1016\/j.neuroimage.2013.02.063","volume":"76","author":"O Bartra","year":"2013","unstructured":"Bartra, O., McGuire, J. T. & Kable, J. W. The valuation system: a coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value. Neuroimage \n                           76, 412\u2013427 (2013).","journal-title":"Neuroimage"},{"key":"17687_CR13","doi-asserted-by":"publisher","first-page":"503","DOI":"10.1126\/science.1100907","volume":"306","author":"SM McClure","year":"2004","unstructured":"McClure, S. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. Separate neural systems value immediate and delayed monetary rewards. Science \n                           306, 503\u2013507 (2004).","journal-title":"Science"},{"key":"17687_CR14","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1126\/science.1094285","volume":"304","author":"J O\u2019doherty","year":"2004","unstructured":"O\u2019doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science \n                           304, 452\u2013454 (2004).","journal-title":"science"},{"key":"17687_CR15","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1016\/j.neuron.2006.05.006","volume":"50","author":"W Yoshida","year":"2006","unstructured":"Yoshida, W. & Ishii, S. Resolution of uncertainty in prefrontal cortex. Neuron \n                           50, 781\u2013789 (2006).","journal-title":"Neuron"},{"key":"17687_CR16","first-page":"64","volume":"2","author":"RA Rescorla","year":"1972","unstructured":"Rescorla, R. A. & Wagner, A. R. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. Classical conditioning II: Current research and theory \n                           2, 64\u201399 (1972).","journal-title":"Classical conditioning II: Current research and theory"},{"key":"17687_CR17","unstructured":"Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction. Vol. 1 (MIT press Cambridge, 1998)."},{"key":"17687_CR18","doi-asserted-by":"publisher","first-page":"1554","DOI":"10.1214\/aoms\/1177699147","volume":"37","author":"LE Baum","year":"1966","unstructured":"Baum, L. E. & Petrie, T. Statistical inference for probabilistic functions of finite state Markov chains. The annals of mathematical statistics \n                           37, 1554\u20131563 (1966).","journal-title":"The annals of mathematical statistics"},{"key":"17687_CR19","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1090\/S0002-9904-1967-11751-8","volume":"73","author":"LE Baum","year":"1967","unstructured":"Baum, L. E. & Eagon, J. A. An inequality with applications to statistical estimation for probabilistic functions of Markov processes and to a model for ecology. Bull. Amer. Math. Soc \n                           73, 360\u2013363 (1967).","journal-title":"Bull. Amer. Math. Soc"},{"key":"17687_CR20","doi-asserted-by":"publisher","first-page":"211","DOI":"10.2140\/pjm.1968.27.211","volume":"27","author":"LE Baum","year":"1968","unstructured":"Baum, L. E. & Sell, G. Growth transformations for functions on manifolds. Pacific Journal of Mathematics \n                           27, 211\u2013227 (1968).","journal-title":"Pacific Journal of Mathematics"},{"key":"17687_CR21","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1214\/aoms\/1177697196","volume":"41","author":"LE Baum","year":"1970","unstructured":"Baum, L. E., Petrie, T., Soules, G. & Weiss, N. A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The annals of mathematical statistics \n                           41, 164\u2013171 (1970).","journal-title":"The annals of mathematical statistics"},{"key":"17687_CR22","doi-asserted-by":"publisher","first-page":"716","DOI":"10.1109\/TAC.1974.1100705","volume":"19","author":"H Akaike","year":"1974","unstructured":"Akaike, H. A new look at the statistical model identification. IEEE transactions on automatic control \n                           19, 716\u2013723 (1974).","journal-title":"IEEE transactions on automatic control"},{"key":"17687_CR23","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1214\/aos\/1176344136","volume":"6","author":"G Schwarz","year":"1978","unstructured":"Schwarz, G. Estimating the dimension of a model. The annals of statistics \n                           6, 461\u2013464 (1978).","journal-title":"The annals of statistics"},{"key":"17687_CR24","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1016\/S0893-6080(05)80154-6","volume":"7","author":"S Kaski","year":"1994","unstructured":"Kaski, S. & Kohonen, T. Winner-take-all networks for physiological models of competitive learning. Neural Networks \n                           7, 973\u2013984 (1994).","journal-title":"Neural Networks"},{"key":"17687_CR25","doi-asserted-by":"publisher","first-page":"1141","DOI":"10.1016\/0893-6080(96)00019-6","volume":"9","author":"Y Fang","year":"1996","unstructured":"Fang, Y., Cohen, M. A. & Kincaid, T. G. Dynamics of a winner-take-all neural network. Neural Networks \n                           9, 1141\u20131154 (1996).","journal-title":"Neural Networks"},{"key":"17687_CR26","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","volume":"22","author":"SJ Pan","year":"2010","unstructured":"Pan, S. J. & Yang, Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering \n                           22, 1345\u20131359 (2010).","journal-title":"IEEE Transactions on knowledge and data engineering"},{"key":"17687_CR27","doi-asserted-by":"crossref","unstructured":"Marthi, B. Automatic shaping and decomposition of reward functions. In proceedings of the 24th International Conference on Machine learning. 601\u2013608 (ACM) (2007).","DOI":"10.1145\/1273496.1273572"},{"key":"17687_CR28","doi-asserted-by":"publisher","first-page":"15727","DOI":"10.1523\/JNEUROSCI.3489-09.2009","volume":"29","author":"J Peters","year":"2009","unstructured":"Peters, J. & B\u00fcchel, C. Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. Journal of Neuroscience \n                           29, 15727\u201315734 (2009).","journal-title":"Journal of Neuroscience"},{"key":"17687_CR29","doi-asserted-by":"publisher","first-page":"17199","DOI":"10.1073\/pnas.0901077106","volume":"106","author":"K Wunderlich","year":"2009","unstructured":"Wunderlich, K., Rangel, A. & O\u2019Doherty, J. P. Neural computations underlying action-based decision making in the human brain. Proceedings of the National Academy of Sciences \n                           106, 17199\u201317204 (2009).","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"17687_CR30","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1016\/S0306-4522(98)00697-6","volume":"91","author":"RE Suri","year":"1999","unstructured":"Suri, R. E. & Schultz, W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience \n                           91, 871\u2013890 (1999).","journal-title":"Neuroscience"},{"key":"17687_CR31","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1016\/S0377-2217(97)00147-1","volume":"107","author":"SH Zanakis","year":"1998","unstructured":"Zanakis, S. H., Solomon, A., Wishart, N. & Dublish, S. Multi-attribute decision making: A simulation comparison of select methods. European journal of operational research \n                           107, 507\u2013529 (1998).","journal-title":"European journal of operational research"},{"key":"17687_CR32","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1016\/j.neuron.2010.04.016","volume":"66","author":"J Glascher","year":"2010","unstructured":"Glascher, J., Daw, N., Dayan, P. & O\u2019Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron \n                           66, 585\u2013595, https:\/\/doi.org\/10.1016\/j.neuron.2010.04.016 (2010).","journal-title":"Neuron"},{"key":"17687_CR33","doi-asserted-by":"publisher","first-page":"14220","DOI":"10.1523\/JNEUROSCI.2277-15.2015","volume":"35","author":"WM Pauli","year":"2015","unstructured":"Pauli, W. M. et al. Distinct contributions of ventromedial and dorsolateral subregions of the human substantia nigra to appetitive and aversive learning. Journal of Neuroscience \n                           35, 14220\u201314233 (2015).","journal-title":"Journal of Neuroscience"},{"key":"17687_CR34","doi-asserted-by":"publisher","first-page":"189","DOI":"10.3389\/fnhum.2011.00189","volume":"5","author":"RC Wilson","year":"2011","unstructured":"Wilson, R. C. & Niv, Y. Inferring relevance in a changing world. Front Hum Neurosci \n                           5, 189, https:\/\/doi.org\/10.3389\/fnhum.2011.00189 (2011).","journal-title":"Front Hum Neurosci"},{"key":"17687_CR35","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/j.neuron.2016.12.040","volume":"93","author":"YC Leong","year":"2017","unstructured":"Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V. & Niv, Y. Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments. Neuron \n                           93, 451\u2013463, https:\/\/doi.org\/10.1016\/j.neuron.2016.12.040 (2017).","journal-title":"Neuron"},{"key":"17687_CR36","unstructured":"Chen, X. et al. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172\u20132180 (2016)."},{"key":"17687_CR37","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2921\u20132929."},{"key":"17687_CR38","doi-asserted-by":"crossref","unstructured":"Bacon, P.-L., Harb, J. & Precup, D. The option-critic architecture. arXiv preprint arXiv:1609.05140 (2016).","DOI":"10.1609\/aaai.v31i1.10916"},{"key":"17687_CR39","doi-asserted-by":"crossref","unstructured":"Nassar, M. R., Helmers, J. C. & Frank, M. J. Chunking as a rational strategy for lossy data compression in visual working memory tasks. bioRxiv, 098939 (2017).","DOI":"10.1101\/098939"},{"key":"17687_CR40","doi-asserted-by":"publisher","first-page":"8360","DOI":"10.1523\/JNEUROSCI.1010-06.2006","volume":"26","author":"AN Hampton","year":"2006","unstructured":"Hampton, A. N., Bossaerts, P. & O\u2019Doherty, J. P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci \n                           26, 8360\u20138367, https:\/\/doi.org\/10.1523\/JNEUROSCI.1010-06.2006 (2006).","journal-title":"J Neurosci"},{"key":"17687_CR41","doi-asserted-by":"publisher","first-page":"786","DOI":"10.1038\/nn.3068","volume":"15","author":"K Wunderlich","year":"2012","unstructured":"Wunderlich, K., Dayan, P. & Dolan, R. J. Mapping value based planning and extensively trained choice in the human brain. Nature neuroscience \n                           15, 786\u2013791 (2012).","journal-title":"Nature neuroscience"},{"key":"17687_CR42","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1038\/nrn2357","volume":"9","author":"A Rangel","year":"2008","unstructured":"Rangel, A., Camerer, C. & Montague, P. R. A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience \n                           9, 545\u2013556 (2008).","journal-title":"Nature Reviews Neuroscience"},{"key":"17687_CR43","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/nn2066","volume":"11","author":"MF Rushworth","year":"2008","unstructured":"Rushworth, M. F. & Behrens, T. E. Choice, uncertainty and value in prefrontal and cingulate cortex. Nature neuroscience \n                           11, 389\u2013397 (2008).","journal-title":"Nature neuroscience"},{"key":"17687_CR44","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1152\/jn.00762.2005","volume":"95","author":"PN Tobler","year":"2006","unstructured":"Tobler, P. N., O\u2019Doherty, J. P., Dolan, R. J. & Schultz, W. Human neural learning depends on reward prediction errors in the blocking paradigm. Journal of Neurophysiology \n                           95, 301\u2013310 (2006).","journal-title":"Journal of Neurophysiology"},{"key":"17687_CR45","doi-asserted-by":"publisher","first-page":"1054","DOI":"10.1016\/j.neuron.2011.05.014","volume":"70","author":"MF Rushworth","year":"2011","unstructured":"Rushworth, M. F., Noonan, M. P., Boorman, E. D., Walton, M. E. & Behrens, T. E. Frontal cortex and reward-guided learning and decision-making. Neuron \n                           70, 1054\u20131069, https:\/\/doi.org\/10.1016\/j.neuron.2011.05.014 (2011).","journal-title":"Neuron"},{"key":"17687_CR46","unstructured":"Jones, D. M. & Gittins, J. C. A dynamic allocation index for the sequential design of experiments. (University of Cambridge, Department of Engineering, 1972)."},{"key":"17687_CR47","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.jneumeth.2006.11.017","volume":"162","author":"JW Peirce","year":"2007","unstructured":"Peirce, J. W. PsychoPy\u2014psychophysics software in Python. Journal of neuroscience methods \n                           162, 8\u201313 (2007).","journal-title":"Journal of neuroscience methods"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17687-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17687-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17687-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T16:39:10Z","timestamp":1671813550000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-17687-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,15]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,12]]}},"alternative-id":["17687"],"URL":"https:\/\/doi.org\/10.1038\/s41598-017-17687-2","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,12,15]]},"assertion":[{"value":"19 June 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2017","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2017","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"17676"}}