{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T11:31:45Z","timestamp":1793446305135,"version":"4.5.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011678","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T00:00:00Z","timestamp":1705449600000}}],"reference-count":49,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T00:00:00Z","timestamp":1704326400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["2536\/20"],"award-info":[{"award-number":["2536\/20"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Tel Aviv University Center for AI and Data Science","award":["TAD"],"award-info":[{"award-number":["TAD"]}]},{"DOI":"10.13039\/501100003977","name":"the Israeli Science Foundation","doi-asserted-by":"crossref","award":["2923\/20"],"award-info":[{"award-number":["2923\/20"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Reinforcement learning (RL) models are used extensively to study human behavior. These rely on normative models of behavior and stress interpretability over predictive capabilities. More recently, neural network models have emerged as a descriptive modeling paradigm that is capable of high predictive power yet with limited interpretability. Here, we seek to augment the expressiveness of theoretical RL models with the high flexibility and predictive power of neural networks. We introduce a novel framework, which we term theoretical-RNN (t-RNN), whereby a recurrent neural network is trained to predict trial-by-trial behavior and to infer theoretical RL parameters using artificial data of RL agents performing a two-armed bandit task. In three studies, we then examined the use of our approach to dynamically predict unseen behavior along with time-varying theoretical RL parameters. We first validate our approach using synthetic data with known RL parameters. Next, as a proof-of-concept, we applied our framework to two independent datasets of humans performing the same task. In the first dataset, we describe differences in theoretical RL parameters dynamic among clinical psychiatric vs. healthy controls. In the second dataset, we show that the exploration strategies of humans varied dynamically in response to task phase and difficulty. For all analyses, we found better performance in the prediction of actions for t-RNN compared to the stationary maximum-likelihood RL method. We discuss the use of neural networks to facilitate the estimation of latent RL parameters underlying choice behavior.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011678","type":"journal-article","created":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T13:26:07Z","timestamp":1704374767000},"page":"e1011678","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":18,"title":["Harnessing the flexibility of neural networks to predict dynamic theoretical parameters underlying human choice behavior"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4847-0146","authenticated-orcid":true,"given":"Yoav","family":"Ger","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eliya","family":"Nachmani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lior","family":"Wolf","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1364-6738","authenticated-orcid":true,"given":"Nitzan","family":"Shahar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,1,4]]},"reference":[{"issue":"1","key":"pcbi.1011678.ref001","article-title":"Trial-by-trial data analysis using computational models","volume":"23","author":"ND Daw","year":"2011","journal-title":"Decision making, affect, and learning: Attention and performance XXIII"},{"key":"pcbi.1011678.ref002","doi-asserted-by":"crossref","first-page":"e49547","DOI":"10.7554\/eLife.49547","article-title":"Ten simple rules for the computational modeling of behavioral data","volume":"8","author":"RC Wilson","year":"2019","journal-title":"Elife"},{"key":"pcbi.1011678.ref003","doi-asserted-by":"crossref","first-page":"e75474","DOI":"10.7554\/eLife.75474","article-title":"The interpretation of computational model parameters depends on the context","volume":"11","author":"MK Eckstein","year":"2022","journal-title":"Elife"},{"issue":"5306","key":"pcbi.1011678.ref004","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1126\/science.275.5306.1593","article-title":"A neural substrate of prediction and reward","volume":"275","author":"W Schultz","year":"1997","journal-title":"Science"},{"issue":"6","key":"pcbi.1011678.ref005","doi-asserted-by":"crossref","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-based influences on humans\u2019 choices and striatal prediction errors","volume":"69","author":"ND Daw","year":"2011","journal-title":"Neuron"},{"issue":"1","key":"pcbi.1011678.ref006","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.tics.2011.11.018","article-title":"Computational psychiatry","volume":"16","author":"PR Montague","year":"2012","journal-title":"Trends in cognitive sciences"},{"issue":"4","key":"pcbi.1011678.ref007","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3758\/CABN.8.4.429","article-title":"Decision theory, reinforcement learning, and the brain","volume":"8","author":"P Dayan","year":"2008","journal-title":"Cognitive, Affective, & Behavioral Neuroscience"},{"issue":"3","key":"pcbi.1011678.ref008","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.jmp.2008.12.005","article-title":"Reinforcement learning in the brain","volume":"53","author":"Y Niv","year":"2009","journal-title":"Journal of Mathematical Psychology"},{"issue":"6","key":"pcbi.1011678.ref009","doi-asserted-by":"crossref","first-page":"e1006903","DOI":"10.1371\/journal.pcbi.1006903","article-title":"Models that learn how humans learn: the case of decision-making and its disorders","volume":"15","author":"A Dezfouli","year":"2019","journal-title":"PLoS computational biology"},{"issue":"5","key":"pcbi.1011678.ref010","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","author":"K Hornik","year":"1989","journal-title":"Neural networks"},{"key":"pcbi.1011678.ref011","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1145\/130385.130432","volume-title":"Proceedings of the fifth annual workshop on Computational learning theory","author":"HT Siegelmann","year":"1992"},{"issue":"6","key":"pcbi.1011678.ref012","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1177\/1745691617693393","article-title":"Choosing prediction over explanation in psychology: Lessons from machine learning","volume":"12","author":"T Yarkoni","year":"2017","journal-title":"Perspectives on Psychological Science"},{"issue":"3","key":"pcbi.1011678.ref013","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1016\/j.neuron.2019.12.002","article-title":"Direct fit to nature: an evolutionary perspective on biological and artificial neural networks","volume":"105","author":"U Hasson","year":"2020","journal-title":"Neuron"},{"issue":"5752","key":"pcbi.1011678.ref014","doi-asserted-by":"crossref","first-page":"1337","DOI":"10.1126\/science.1115270","article-title":"Representation of action-specific reward values in the striatum","volume":"310","author":"K Samejima","year":"2005","journal-title":"Science"},{"issue":"7095","key":"pcbi.1011678.ref015","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1038\/nature04766","article-title":"Cortical substrates for exploratory decisions in humans","volume":"441","author":"ND Daw","year":"2006","journal-title":"Nature"},{"issue":"9","key":"pcbi.1011678.ref016","doi-asserted-by":"crossref","first-page":"1214","DOI":"10.1038\/nn1954","article-title":"Learning the value of information in an uncertain world","volume":"10","author":"TE Behrens","year":"2007","journal-title":"Nature neuroscience"},{"key":"pcbi.1011678.ref017","article-title":"Practical Bayesian optimization for model fitting with Bayesian adaptive direct search","volume":"30","author":"L Acerbi","year":"2017","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1011678.ref018","unstructured":"Song M, Niv Y, Cai M. Using Recurrent Neural Networks to Understand Human Reward Learning. In: Proceedings of the Annual Meeting of the Cognitive Science Society. vol. 43; 2021. p. 1388\u20131394."},{"issue":"6547","key":"pcbi.1011678.ref019","doi-asserted-by":"crossref","first-page":"1209","DOI":"10.1126\/science.abe2629","article-title":"Using large-scale experiments and machine learning to discover theories of human decision-making","volume":"372","author":"JC Peterson","year":"2021","journal-title":"Science"},{"key":"pcbi.1011678.ref020","first-page":"4584","article-title":"Reverse-engineering recurrent neural network solutions to a hierarchical inference task for mice","volume":"33","author":"R. Schaeffer","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"pcbi.1011678.ref021","article-title":"Disentangled behavioural representations","volume":"32","author":"A Dezfouli","year":"2019","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1011678.ref022","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.cognition.2017.12.014","article-title":"Deconstructing the human algorithms for exploration","volume":"173","author":"SJ Gershman","year":"2018","journal-title":"Cognition"},{"key":"pcbi.1011678.ref023","volume-title":"DSM-IV sourcebook","author":"T. A. Widiger","year":"1997"},{"issue":"6","key":"pcbi.1011678.ref024","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1037\/a0038199","article-title":"Humans use directed and random exploration to solve the explore\u2013exploit dilemma","volume":"143","author":"RC Wilson","year":"2014","journal-title":"Journal of Experimental Psychology: General"},{"key":"pcbi.1011678.ref025","doi-asserted-by":"crossref","unstructured":"Cho K, Van Merri\u00ebnboer B, Bahdanau D, Bengio Y. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:14091259. 2014;.","DOI":"10.3115\/v1\/W14-4012"},{"key":"pcbi.1011678.ref026","unstructured":"Ranjan R, Castillo CD, Chellappa R. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:170309507. 2017;."},{"key":"pcbi.1011678.ref027","author":"L Fei-Fei","year":"2015","journal-title":"Stanford\u2019s cs231n class notes"},{"key":"pcbi.1011678.ref028","article-title":"Estimating internal variables and paramters of a learning agent by a particle filter","volume":"16","author":"K Samejima","year":"2003","journal-title":"Advances in neural information processing systems"},{"issue":"3","key":"pcbi.1011678.ref029","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"CJ Watkins","year":"1992","journal-title":"Machine learning"},{"key":"pcbi.1011678.ref030","volume-title":"Reinforcement learning: An introduction","author":"RS Sutton","year":"2018"},{"issue":"17","key":"pcbi.1011678.ref031","doi-asserted-by":"crossref","first-page":"5833","DOI":"10.1523\/JNEUROSCI.0053-12.2012","article-title":"Serotonin selectively modulates reward value in human decision-making","volume":"32","author":"B Seymour","year":"2012","journal-title":"Journal of Neuroscience"},{"key":"pcbi.1011678.ref032","volume-title":"Pattern recognition and machine learning","author":"CM Bishop","year":"2006"},{"issue":"3-4","key":"pcbi.1011678.ref033","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1093\/biomet\/25.3-4.285","article-title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","volume":"25","author":"WR Thompson","year":"1933","journal-title":"Biometrika"},{"key":"pcbi.1011678.ref034","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-time analysis of the multiarmed bandit problem","volume":"47","author":"P Auer","year":"2002","journal-title":"Machine learning"},{"key":"pcbi.1011678.ref035","first-page":"1","article-title":"Modelling human behaviour in cognitive tasks with latent dynamical systems","author":"PI Jaffe","year":"2023","journal-title":"Nature Human Behaviour"},{"issue":"1","key":"pcbi.1011678.ref036","doi-asserted-by":"crossref","first-page":"4736","DOI":"10.1038\/s41598-022-08863-0","article-title":"Using deep learning to predict human decisions and using cognitive models to explain deep learning models","volume":"12","author":"M Fintz","year":"2022","journal-title":"Scientific reports"},{"key":"pcbi.1011678.ref037","article-title":"Stress, noradrenaline, and realistic prediction of mouse behaviour using reinforcement learning","volume":"21","author":"C Sandi","year":"2008","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"9","key":"pcbi.1011678.ref038","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1038\/nn.2374","article-title":"Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning","volume":"12","author":"G Luksys","year":"2009","journal-title":"Nature neuroscience"},{"key":"pcbi.1011678.ref039","article-title":"Efficient inference for time-varying behavior during learning","volume":"31","author":"NA Roy","year":"2018","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1011678.ref040","first-page":"3442","article-title":"Inferring learning rules from animal decision-making","volume":"33","author":"Z Ashwood","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"pcbi.1011678.ref041","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1038\/s41593-021-01007-z","article-title":"Mice alternate between discrete strategies during perceptual decision-making","volume":"25","author":"ZC Ashwood","year":"2022","journal-title":"Nature Neuroscience"},{"key":"pcbi.1011678.ref042","first-page":"29663","article-title":"Dynamic Inverse Reinforcement Learning for Characterizing Animal Behavior","volume":"35","author":"Z Ashwood","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"31","key":"pcbi.1011678.ref043","doi-asserted-by":"crossref","first-page":"9861","DOI":"10.1523\/JNEUROSCI.6157-08.2009","article-title":"Validation of decision-making models and analysis of decision variables in the rat basal ganglia","volume":"29","author":"M Ito","year":"2009","journal-title":"Journal of Neuroscience"},{"issue":"1","key":"pcbi.1011678.ref044","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"N Srivastava","year":"2014","journal-title":"The journal of machine learning research"},{"key":"pcbi.1011678.ref045","first-page":"461129","article-title":"From predictive models to cognitive models: Separable behavioral processes underlying reward learning in the rat","author":"K Miller","year":"2021","journal-title":"bioRxiv"},{"key":"pcbi.1011678.ref046","doi-asserted-by":"crossref","first-page":"e56261","DOI":"10.7554\/eLife.56261","article-title":"Training deep neural density estimators to identify mechanistic models of neural dynamics","volume":"9","author":"PJ Gon\u00e7alves","year":"2020","journal-title":"eLife"},{"key":"pcbi.1011678.ref047","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"A Paszke","year":"2019","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1011678.ref048","unstructured":"Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980. 2014;."},{"issue":"3","key":"pcbi.1011678.ref049","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in Python","volume":"17","author":"P Virtanen","year":"2020","journal-title":"Nature methods"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011678","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T00:00:00Z","timestamp":1705449600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011678","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T13:36:31Z","timestamp":1705498591000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011678"}},"subtitle":[],"editor":[{"given":"Lusha","family":"Zhu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,1,4]]},"references-count":49,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1,4]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011678","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.04.21.537666","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,4]]}}}