{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,26]],"date-time":"2026-01-26T10:42:15Z","timestamp":1769424135572,"version":"3.49.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010580","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,10,13]],"date-time":"2022-10-13T00:00:00Z","timestamp":1665619200000}}],"reference-count":46,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T00:00:00Z","timestamp":1664755200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["GU 1845\/1-1"],"award-info":[{"award-number":["GU 1845\/1-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["STE 1430\/9-1"],"award-info":[{"award-number":["STE 1430\/9-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100017268","name":"Berlin Institute of Health","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100017268","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Berlin School of Mind and Brain, Humboldt-Universit\u00e4t zu Berlin"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Reinforcement learning algorithms have a long-standing success story in explaining the dynamics of instrumental conditioning in humans and other species. While normative reinforcement learning models are critically dependent on external feedback, recent findings in the field of perceptual learning point to a crucial role of internally generated reinforcement signals based on subjective confidence, when external feedback is not available. Here, we investigated the existence of such confidence-based learning signals in a key domain of reinforcement-based learning: instrumental conditioning. We conducted a value-based decision making experiment which included phases with and without external feedback and in which participants reported their confidence in addition to choices. Behaviorally, we found signatures of self-reinforcement in phases without feedback, reflected in an increase of subjective confidence and choice consistency. To clarify the mechanistic role of confidence in value-based learning, we compared a family of confidence-based learning models with more standard models predicting either no change in value estimates or a devaluation over time when no external reward is provided. We found that confidence-based models indeed outperformed these reference models, whereby the learning signal of the winning model was based on the prediction error between current confidence and a stimulus-unspecific average of previous confidence levels. Interestingly, individuals with more volatile reward-based value updates in the presence of feedback also showed more volatile confidence-based value updates when feedback was not available. Together, our results provide evidence that confidence-based learning signals affect instrumentally learned subjective values in the absence of external feedback.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010580","type":"journal-article","created":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T17:56:28Z","timestamp":1664819788000},"page":"e1010580","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":13,"title":["The value of confidence: Confidence prediction errors drive value-based learning in the absence of external feedback"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8267-7239","authenticated-orcid":true,"given":"Lena Esther","family":"Ptasczynski","sequence":"first","affiliation":[]},{"given":"Isa","family":"Steinecker","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4687-2317","authenticated-orcid":true,"given":"Philipp","family":"Sterzer","sequence":"additional","affiliation":[]},{"given":"Matthias","family":"Guggenmos","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,10,3]]},"reference":[{"key":"pcbi.1010580.ref001","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"1998"},{"issue":"1","key":"pcbi.1010580.ref002","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1037\/h0048826","article-title":"Perceptual learning; differentiation or enrichment?","volume":"62","author":"JJ Gibson","year":"1955","journal-title":"Psychol Rev."},{"issue":"3","key":"pcbi.1010580.ref003","doi-asserted-by":"crossref","first-page":"258","DOI":"10.3758\/BF03206097","article-title":"Improvement in vernier acuity with practice.","volume":"24","author":"SP McKee","year":"1978","journal-title":"Percept Psychophys"},{"issue":"11","key":"pcbi.1010580.ref004","doi-asserted-by":"crossref","first-page":"4966","DOI":"10.1073\/pnas.88.11.4966","article-title":"Where practice makes perfect in texture discrimination: evidence for primary visual cortex plasticity","volume":"88","author":"A Karni","year":"1991","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"15","key":"pcbi.1010580.ref005","doi-asserted-by":"crossref","first-page":"2133","DOI":"10.1016\/S0042-6989(97)00043-6","article-title":"The role of feedback in learning a vernier discrimination task","volume":"37","author":"MH Herzog","year":"1997","journal-title":"Vision Res"},{"key":"pcbi.1010580.ref006","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s004220050418","article-title":"Modeling perceptual learning: difficulties and how they can be overcome.","volume":"78","author":"MH Herzog","year":"1998","journal-title":"Biol Cybern"},{"issue":"6858","key":"pcbi.1010580.ref007","doi-asserted-by":"crossref","first-page":"844","DOI":"10.1038\/35101601","article-title":"Perceptual learning without perception","volume":"413","author":"T Watanabe","year":"2001","journal-title":"Nature"},{"issue":"March","key":"pcbi.1010580.ref008","first-page":"2003","article-title":"Is subliminal learning really passive?","volume":"422","author":"AR Seitz","year":"2003","journal-title":"Nature"},{"issue":"7","key":"pcbi.1010580.ref009","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/j.tics.2005.05.010","article-title":"A unified model for perceptual learning.","volume":"9","author":"AR Seitz","year":"2005","journal-title":"Trends Cogn Sci.Jul"},{"issue":"4","key":"pcbi.1010580.ref010","doi-asserted-by":"crossref","first-page":"3457","DOI":"10.1016\/j.neuroimage.2011.11.058","article-title":"Striatal activations signal prediction errors on confidence in the absence of external feedback.","volume":"59","author":"R Daniel","year":"2012","journal-title":"NeuroImage."},{"key":"pcbi.1010580.ref011","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.nlm.2014.05.002","article-title":"A universal role of the ventral striatum in reward-based learning: Evidence from human studies.","volume":"114","author":"R Daniel","year":"2014","journal-title":"Neurobiol Learn Mem."},{"key":"pcbi.1010580.ref012","doi-asserted-by":"crossref","first-page":"1","DOI":"10.7554\/eLife.13388","article-title":"Mesolimbic confidence signals guide perceptual learning in the absence of external feedback.","volume":"5","author":"M Guggenmos","year":"2016","journal-title":"eLife"},{"issue":"1","key":"pcbi.1010580.ref013","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1093\/cercor\/bhu181","article-title":"The Relationship between Perceptual Decision Variables and Confidence in the Human Brain","volume":"26","author":"M Hebart","year":"2016","journal-title":"Cereb Cortex"},{"issue":"7","key":"pcbi.1010580.ref014","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1016\/j.neubiorev.2013.03.023","article-title":"Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies.","volume":"37","author":"J Garrison","year":"2013","journal-title":"Neurosci Biobehav Rev"},{"key":"pcbi.1010580.ref015","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neuropsychologia.2015.04.011","article-title":"Goal- and retrieval-dependent activity in the striatum during memory recognition.","volume":"72","author":"M Clos","year":"2015","journal-title":"Neuropsychologia."},{"key":"pcbi.1010580.ref016","doi-asserted-by":"crossref","DOI":"10.1515\/9781503620766","volume-title":"A Theory of Cognitive Dissonance","author":"L. Festinger","year":"1957"},{"issue":"3","key":"pcbi.1010580.ref017","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1037\/h0041006","article-title":"Postdecision changes in the desirability of alternatives.","volume":"52","author":"JW Brehm","year":"1956","journal-title":"J Abnorm Soc Psychol"},{"key":"pcbi.1010580.ref018","article-title":"Rationalization and Cognitive Dissonance: Do Choices Affect or Reflect Preferences?","author":"MK Chen","year":"2008","journal-title":"Cowles Found Discuss Pap No 1669."},{"issue":"4","key":"pcbi.1010580.ref019","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1037\/a0020217","article-title":"How Choice Affects and Reflects Preferences: Revisiting the Free-Choice Paradigm.","volume":"99","author":"MK Chen","year":"2010","journal-title":"J Pers Soc Psychol."},{"issue":"4","key":"pcbi.1010580.ref020","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1177\/0956797610364115","article-title":"I\u2019m no longer torn after choice: How explicit choices implicitly shape preferences of odors.","volume":"21","author":"G Coppin","year":"2010","journal-title":"Psychol Sci."},{"issue":"6","key":"pcbi.1010580.ref021","doi-asserted-by":"crossref","first-page":"e37857","DOI":"10.1371\/journal.pone.0037857","article-title":"When Flexibility Is Stable: Implicit Long-Term Shaping of Olfactory Preferences","volume":"7","author":"G Coppin","year":"2012","journal-title":"PLoS ONE."},{"issue":"9","key":"pcbi.1010580.ref022","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0956797610379235","article-title":"Do decisions shape preference? Evidence from blind choice.","volume":"21","author":"T Sharot","year":"2010","journal-title":"Psychol Sci."},{"issue":"10","key":"pcbi.1010580.ref023","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1177\/0956797612438733","article-title":"Is Choice-Induced Preference Change Long Lasting?","volume":"23","author":"T Sharot","year":"2012","journal-title":"Psychol Sci."},{"issue":"8","key":"pcbi.1010580.ref024","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0072071","article-title":"I Choose, Therefore I Like: Preference for Faces Induced by Arbitrary Choice.","volume":"8","author":"K Nakamura","year":"2013","journal-title":"PLoS ONE."},{"issue":"3","key":"pcbi.1010580.ref025","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1002\/bdm.1807","article-title":"Choice Blindness and Preference Change: You Will Like This Paper Better If You (Believe You) Chose to Read It!: Choice Blindness and Preference Change.","volume":"27","author":"P Johansson","year":"2014","journal-title":"J Behav Decis Mak."},{"issue":"3","key":"pcbi.1010580.ref026","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pone.0119682","article-title":"Action and valence modulate choice and choice-induced preference change.","volume":"10","author":"R Koster","year":"2015","journal-title":"PLoS ONE."},{"issue":"2","key":"pcbi.1010580.ref027","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1002\/bdm.1967","article-title":"The Spreading of Alternatives: Is it the Perceived Choice or Actual Choice that Changes our Preference?: Perceived Choice and Actual Choice in our Preference.","volume":"30","author":"J Luo","year":"2017","journal-title":"J Behav Decis Mak."},{"issue":"1","key":"pcbi.1010580.ref028","doi-asserted-by":"crossref","first-page":"3318","DOI":"10.1038\/s41467-020-17192-7","article-title":"Decisions bias future choices by modifying hippocampal associative memories.","volume":"11","author":"L Luettgau","year":"2020","journal-title":"Nat Commun."},{"key":"pcbi.1010580.ref029","article-title":"A confidence-based reinforcement learning model for perceptual learning.","author":"M Guggenmos","year":"2017","journal-title":"BioRxiv"},{"issue":"1","key":"pcbi.1010580.ref030","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1523\/JNEUROSCI.2205-09.2010","article-title":"Comparing the neural basis of monetary reward and cognitive feedback during information-integration category learning","volume":"30","author":"R Daniel","year":"2010","journal-title":"J Neurosci"},{"issue":"1","key":"pcbi.1010580.ref031","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1146\/annurev.neuro.23.1.473","article-title":"Neuronal Coding of Prediction Errors","volume":"23","author":"W Schultz","year":"2000","journal-title":"Annu Rev Neurosci"},{"issue":"48","key":"pcbi.1010580.ref032","doi-asserted-by":"crossref","first-page":"15104","DOI":"10.1523\/JNEUROSCI.3524-09.2009","article-title":"Dopaminergic Drugs Modulate Learning Rates and Perseveration in Parkinson\u2019s Patients in a Dynamic Foraging Task","volume":"29","author":"RB Rutledge","year":"2009","journal-title":"J Neurosci"},{"key":"pcbi.1010580.ref033","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.jmp.2018.09.002","article-title":"The statistical structures of reinforcement learning with asymmetric value updates.","volume":"87","author":"K. Katahira","year":"2018","journal-title":"J Math Psychol."},{"issue":"6","key":"pcbi.1010580.ref034","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1109\/TAC.1974.1100705","article-title":"A new look at the statistical model identification","volume":"19","author":"H Akaike","year":"1974","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"5","key":"pcbi.1010580.ref035","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1037\/0278-7393.20.5.1063","article-title":"Remembering can cause forgetting: Retrieval dynamics in long-term memory.","volume":"20","author":"MC Anderson","year":"1994","journal-title":"J Exp Psychol Learn Mem Cogn"},{"issue":"10","key":"pcbi.1010580.ref036","doi-asserted-by":"crossref","first-page":"3994","DOI":"10.1093\/cercor\/bhu284","article-title":"Neural Differentiation Tracks Improved Recall of Competing Memories Following Interleaved Study and Retrieval Practice","volume":"25","author":"JC Hulbert","year":"2015","journal-title":"Cereb Cortex"},{"issue":"4","key":"pcbi.1010580.ref037","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1038\/nn.3973","article-title":"Retrieval induces adaptive forgetting of competing memories via cortical pattern suppression","volume":"18","author":"M Wimber","year":"2015","journal-title":"Nat Neurosci"},{"key":"pcbi.1010580.ref038","first-page":"1","article-title":"Metacognition about the past and future: quantifying common and distinct influences on prospective and retrospective judgments of self-performance.","author":"SM Fleming","year":"2016","journal-title":"Neurosci Conscious."},{"issue":"january","key":"pcbi.1010580.ref039","doi-asserted-by":"crossref","first-page":"0035","DOI":"10.1038\/s41562-016-0035","article-title":"Perceptual learning alters post-sensory processing in human decision-making.","volume":"1","author":"JA Diaz","year":"2017","journal-title":"Nat Hum Behav"},{"issue":"5","key":"pcbi.1010580.ref040","doi-asserted-by":"crossref","first-page":"e0231081","DOI":"10.1371\/journal.pone.0231081","article-title":"Choosing what we like vs liking what we choose: How choice-induced preference change might actually be instrumental to decision-making.","volume":"15","author":"D Lee","year":"2020","journal-title":"PLOS ONE."},{"key":"pcbi.1010580.ref041","unstructured":"Skipper S, Perktold J. statsmodels: Econometric and statistical modeling with python. In: 9th Python in Science Conference. 2010."},{"issue":"1","key":"pcbi.1010580.ref042","doi-asserted-by":"crossref","first-page":"195","DOI":"10.3758\/s13428-018-01193-y","article-title":"PsychoPy2: Experiments in behavior made easy.","volume":"51","author":"J Peirce","year":"2019","journal-title":"Behav Res Methods."},{"key":"pcbi.1010580.ref043","first-page":"1","author":"RC Wilson","year":"2019","journal-title":"Ten simple rules for the computational modeling of behavioral data"},{"issue":"3","key":"pcbi.1010580.ref044","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in Python.","volume":"17","author":"P Virtanen","year":"2020","journal-title":"Nat Methods"},{"issue":"1","key":"pcbi.1010580.ref045","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1093\/imamat\/6.1.76","article-title":"The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations.","volume":"6","author":"CG Broyden","year":"1970","journal-title":"IMA J Appl Math"},{"issue":"2","key":"pcbi.1010580.ref046","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1093\/comjnl\/7.2.155","article-title":"An efficient method for finding the minimum of a function of several variables without calculating derivatives.","volume":"7","author":"MJD Powell","year":"1964","journal-title":"Comput J."}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010580","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,10,13]],"date-time":"2022-10-13T00:00:00Z","timestamp":1665619200000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010580","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,13]],"date-time":"2022-10-13T18:12:04Z","timestamp":1665684724000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010580"}},"subtitle":[],"editor":[{"given":"Stefano","family":"Palminteri","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,10,3]]},"references-count":46,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10,3]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010580","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1010580","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,3]]}}}