{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T18:22:32Z","timestamp":1781374952765,"version":"3.54.1"},"reference-count":107,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:p> Active inference offers a first principle account of sentient behavior, from which special and important cases\u2014for example, reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design\u2014can be derived. Active inference finesses the exploitation-exploration dilemma in relation to prior preferences by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this letter, we consider a sophisticated kind of active inference using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs and beliefs about those latent states. In other words, we move from simply considering beliefs about \u201cwhat would happen if I did that\u201d to \u201cwhat I would believe about what would happen if I did that.\u201d The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states as opposed to states per se. We illustrate the competence of this scheme using numerical simulations of deep decision problems. <\/jats:p>","DOI":"10.1162\/neco_a_01351","type":"journal-article","created":{"date-parts":[[2021,2,24]],"date-time":"2021-02-24T22:12:20Z","timestamp":1614204740000},"page":"713-763","source":"Crossref","is-referenced-by-count":98,"title":["Sophisticated Inference"],"prefix":"10.1162","volume":"33","author":[{"given":"Karl","family":"Friston","sequence":"first","affiliation":[{"name":"Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lancelot","family":"Da Costa","sequence":"additional","affiliation":[{"name":"Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K., and Department of Mathematics, Imperial College London, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danijar","family":"Hafner","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada, and Google Research, Brain Team, Toronto, ON MSH 153, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Casper","family":"Hesp","sequence":"additional","affiliation":[{"name":"Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K., and Amsterdam Brain and Cognition Center, University of Amsterdam, Amsterdam 1001 NK, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Parr","sequence":"additional","affiliation":[{"name":"Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London WC1N 3AR, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017746"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1016\/0022-247X(65)90154-X"},{"key":"B3","author":"Attias H.","year":"2003","journal-title":"Proceedings of the 9th Int. Workshop on Artificial Intelligence and Statistics"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.3390\/e17042432"},{"key":"B5","first-page":"217","volume-title":"Sensory communication","author":"Barlow H.","year":"1961"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1068\/p030123"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00907"},{"key":"B8","author":"Beal M. J.","year":"2003","journal-title":"Variational algorithms for approximate Bayesian inference"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.38.8.716"},{"key":"B10","author":"Berger J. O.","year":"2011","journal-title":"Statistical decision theory and Bayesian analysis"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1162\/089976601753195969"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2012.08.006"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1162\/0033553041502225"},{"key":"B14","author":"Catal O.","year":"2019","journal-title":"Bayesian policy selection using active inference"},{"key":"B15","author":"\u00c7atal O.","year":"2020","journal-title":"Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing"},{"key":"B16","author":"\u00c7atal O.","year":"2020","journal-title":"Deep active inference for autonomous robot navigation"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1063\/1.4905960"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2007.2098"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0262.00239"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.99.100602"},{"key":"B21","author":"Da Costa L.","year":"2020","journal-title":"Active inference on discrete state-spaces: A synthesis"},{"key":"B22","author":"Da Costa L.","year":"2020","journal-title":"Natural selection finds natural gradient"},{"key":"B23","author":"Da Costa L.","year":"2020","journal-title":"The relationship between dynamic programming and active inference: The discrete, finite-horizon case"},{"key":"B24","author":"Dauwels J.","year":"2007","journal-title":"On variational message passing on factor graphs"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2011.02.027"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1993.5.4.613"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1995.7.5.889"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0087619"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(99)00046-5"},{"key":"B30","author":"Duff M. O.","year":"2002","journal-title":"Optimal learning: Computational procedures for Bayes-adaptive Markov decision processes"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1026915623"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2013.0475"},{"key":"B33","author":"Friston K.","year":"2019","journal-title":"A free energy principle for a particular physics"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2016.06.022"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00912"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1162\/neco_a_00999"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1162\/NETN_a_00018"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1080\/17588928.2015.1020053"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2017.04.009"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000532"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1016\/j.cobeha.2017.05.025"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007425814087"},{"key":"B43","author":"Ghavamzadeh M.","year":"2016","journal-title":"Bayesian reinforcement learning: A survey"},{"key":"B44","author":"Hafez-Kolahi H.","year":"2019","journal-title":"Information bottleneck and its applications in deep learning"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00075-2_7"},{"key":"B46","author":"Hesp C.","year":"2019","journal-title":"Deeply felt affect: The emergence of valence in deep active inference"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1111\/nous.12062"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1109\/TSSC.1966.300074"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1162\/0899766041336477"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1016\/j.visres.2008.09.007"},{"issue":"4","key":"B51","first-page":"620","volume":"106","author":"Jaynes E. T.","year":"1957","journal-title":"Physical Review Series II"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.2307\/1914185"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-018-0753-2"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1995.10476572"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.2307\/2226451"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1609094113"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.3390\/e20080574"},{"key":"B58","first-page":"1:128","author":"Klyubin A. S.","year":"2005","journal-title":"Proceedings of the IEEE Congress on Evolutionary Computation"},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005753"},{"key":"B60","author":"Lillicrap T. P.","year":"2015","journal-title":"Continuous control with deep reinforcement learning"},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177728069"},{"key":"B62","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ne.13.030190.001353"},{"key":"B63","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1992.4.4.590"},{"key":"B64","author":"MacKay D. J. C.","year":"2003","journal-title":"Information theory, inference and learning algorithms"},{"key":"B65","author":"Millidge B.","year":"2019","journal-title":"Deep active inference as variational policy gradients"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-017-0180-8"},{"key":"B67","doi-asserted-by":"publisher","DOI":"10.1152\/jn.1987.57.1.162"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3062"},{"issue":"124","key":"B69","first-page":"1","volume":"20","author":"Osband I.","year":"2019","journal-title":"Journal of Machine Learning Research"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.3389\/neuro.12.006.2007"},{"issue":"2164","key":"B71","volume":"378","author":"Parr T.","year":"2020","journal-title":"Philosophical Transactions of the Royal Society A"},{"key":"B72","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-15249-0"},{"key":"B73","doi-asserted-by":"publisher","DOI":"10.1016\/j.copsyc.2018.10.006"},{"key":"B74","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-019-00805-w"},{"key":"B75","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-38246-3"},{"key":"B76","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-444-63356-9.00010-8"},{"key":"B77","author":"Rikhye R. V.","year":"2019","journal-title":"Memorize-generalize: An online algorithm for learning higher-order sequential structure with cloned hidden Markov models"},{"key":"B78","volume-title":"Advances in neural information processing systems","volume":"20","author":"Ross S.","year":"2008"},{"key":"B79","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005768"},{"key":"B80","author":"Ryan R.","year":"1985","journal-title":"Intrinsic motivation and self-determination in human behavior"},{"key":"B81","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.1991.170605"},{"key":"B82","doi-asserted-by":"publisher","DOI":"10.1080\/09540090600768658"},{"key":"B83","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2010.2056368"},{"key":"B84","author":"Schrittwieser J.","year":"2019","journal-title":"Mastering Atari, go, chess and shogi by planning with a learned model"},{"key":"B85","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00710"},{"key":"B86","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.41703"},{"key":"B87","unstructured":"Sengupta, B. & Friston, K. (2018). How robust are deep neural networks? arXiv:1804.11313."},{"key":"B88","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1505483112"},{"key":"B89","doi-asserted-by":"publisher","DOI":"10.1007\/s12064-011-0142-z"},{"key":"B90","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.109.120604"},{"key":"B91","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2016.7727309"},{"key":"B92","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22887-2_5"},{"key":"B93","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.88.2.135"},{"key":"B94","author":"Sutton R. S.","year":"1998","journal-title":"Reinforcement learning: An introduction"},{"key":"B95","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/25.3-4.285"},{"key":"B96","first-page":"368","author":"Tishby N.","year":"1999","journal-title":"Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing"},{"key":"B97","author":"Tishby N.","year":"2010","journal-title":"Perception-reason-action cycle: Models, algorithms and systems"},{"key":"B98","author":"Tishby N.","year":"2015","journal-title":"Deep learning and the information bottleneck principle"},{"key":"B99","author":"Todorov E.","year":"2008","journal-title":"Proceedings of the IEEE Conference on Decision and Control"},{"key":"B100","author":"Toussaint M.","year":"2006","journal-title":"Proceedings of the 23rd Int. Conf. on Machine Learning"},{"key":"B101","author":"Tribus M.","year":"1961","journal-title":"Thermodynamics and thermostatics: An introduction to energy, information and states of matter, with engineering applications"},{"key":"B102","author":"Tschantz A.","year":"2019","journal-title":"Scaling active inference"},{"key":"B103","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-018-0785-7"},{"key":"B104","first-page":"1","volume":"6","author":"van den Broek J. L.","year":"2010","journal-title":"Uncertainty in Artificial Intelligence"},{"key":"B105","author":"Von Neumann J.","year":"1944","journal-title":"Theory of games and economic behavior"},{"key":"B106","first-page":"661","volume":"6","author":"Winn J.","year":"2005","journal-title":"Journal of Machine Learning Research"},{"key":"B107","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2005.850085"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01351","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:44:00Z","timestamp":1615585440000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/3\/713-763\/97487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3]]},"references-count":107,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["10.1162\/neco_a_01351"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01351","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3]]}}}