{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T05:30:39Z","timestamp":1767850239412,"version":"3.49.0"},"reference-count":41,"publisher":"MIT Press","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2010,5]]},"abstract":"<jats:p>Reinforcement learning models generally assume that a stimulus is presented that allows a learner to unambiguously identify the state of nature, and the reward received is drawn from a distribution that depends on that state. However, in any natural environment, the stimulus is noisy. When there is state uncertainty, it is no longer immediately obvious how to perform reinforcement learning, since the observed reward cannot be unambiguously allocated to a state of the environment. This letter addresses the problem of incorporating state uncertainty in reinforcement learning models. We show that simply ignoring the uncertainty and allocating the reward to the most likely state of the environment results in incorrect value estimates. Furthermore, using only the information that is available before observing the reward also results in incorrect estimates. We therefore introduce a new technique, posterior weighted reinforcement learning, in which the estimates of state probabilities are updated according to the observed rewards (e.g., if a learner observes a reward usually associated with a particular state, this state becomes more likely). We show analytically that this modified algorithm can converge to correct reward estimates and confirm this with numerical experiments. The algorithm is shown to be a variant of the expectation-maximization algorithm, allowing rigorous convergence analyses to be carried out. A possible neural implementation of the algorithm in the cortico-basal-ganglia-thalamic network is presented, and experimental predictions of our model are discussed.<\/jats:p>","DOI":"10.1162\/neco.2010.01-09-948","type":"journal-article","created":{"date-parts":[[2010,1,25]],"date-time":"2010-01-25T18:21:01Z","timestamp":1264443661000},"page":"1149-1179","source":"Crossref","is-referenced-by-count":17,"title":["Posterior Weighted Reinforcement Learning with State Uncertainty"],"prefix":"10.1162","volume":"22","author":[{"given":"Tobias","family":"Larsen","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Bristol, Bristol, BS8 1UB, U.K."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David S.","family":"Leslie","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Bristol, Bristol, BS8 1TW, U.K"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edmund J.","family":"Collins","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Bristol, Bristol, BS8 1TW, U.K"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rafal","family":"Bogacz","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Bristol, Bristol, BS8 1UB, U.K"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"B1","volume-title":"Information and exponential families in statistical theory","author":"Barndoff-Nielsen O.","year":"1978"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1038\/nn1954"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0096509"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-374620-7.00018-2"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2007.19.2.442"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.12-12-04745.1992"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1037\/0096-1523.14.2.281"},{"key":"B8","first-page":"4393","volume":"4","author":"Cao X.-R.","year":"2004","journal-title":"Proceedings of the 43rd IEEE Conference on Decision and Control"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2009.00698.x"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1126\/science.1150605"},{"key":"B11","volume-title":"Theoretical neuroscience","author":"Dayan P.","year":"2001"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1080\/03772063.2003.11416335"},{"key":"B13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","volume":"39","author":"Dempster A.","year":"1977","journal-title":"Journal of the Royal Statistical Society"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.2976\/1.2732246\/10.2976\/1"},{"key":"B15","volume-title":"Proceedings of the 20th International Conference on Machine Learning","author":"Duff M.","year":"2003"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1126\/science.1102941"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X01004149"},{"key":"B18","volume-title":"Advances in neural information processing systems","volume":"7","author":"Jaakkola T.","year":"1995"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(98)00023-X"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1038\/nature07200"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.4761-07.2008"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-2696-8"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.16-05-01936.1996"},{"key":"B24","volume-title":"Proceedings of the Tenth International Symposium on Artificial Intelligence and Mathematics","author":"Poupart P.","year":"2008"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143932"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.95.2.238"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogpsych.2005.10.002"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.22-21-09475.2002"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1152\/jn.1998.80.1.1"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.93.2.628"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1152\/jn.2001.86.4.1916"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007678930559"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1007\/BF00115009"},{"key":"B34","volume-title":"Reinforcement learning: An introduction","author":"Sutton R.","year":"1998"},{"key":"B35","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1111\/j.2517-6161.1984.tb01296.x","volume":"46","author":"Titterington D.","year":"1984","journal-title":"Journal of the Royal Statistical Society, B"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1126\/science.1105370"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1126\/science.1093360"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.108.3.550"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1016\/j.spl.2006.05.017"},{"key":"B40","volume-title":"Advances in neural information processing systems","volume":"21","author":"Yu A.","year":"2009"},{"key":"B41","volume-title":"Advances in neural information processing Systems","author":"Yu A.","year":"2003"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco.2010.01-09-948","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,16]],"date-time":"2025-02-16T23:32:03Z","timestamp":1739748723000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/22\/5\/1149-1179\/7541"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,5]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2010,5]]}},"alternative-id":["10.1162\/neco.2010.01-09-948"],"URL":"https:\/\/doi.org\/10.1162\/neco.2010.01-09-948","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,5]]}}}