{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T16:42:45Z","timestamp":1783096965800,"version":"3.54.6"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1008342","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2020,11,18]],"date-time":"2020-11-18T00:00:00Z","timestamp":1605657600000}}],"reference-count":46,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T00:00:00Z","timestamp":1604361600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002367","name":"Chinese Academy of Sciences","doi-asserted-by":"crossref","award":["XDB32070100"],"award-info":[{"award-number":["XDB32070100"]}],"id":[{"id":"10.13039\/501100002367","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shanghai Municipal Science and Technology Major Project","award":["2018SHZDZX05"],"award-info":[{"award-number":["2018SHZDZX05"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["31771179"],"award-info":[{"award-number":["31771179"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>The brain makes flexible and adaptive responses in a complicated and ever-changing environment for an organism\u2019s survival. To achieve this, the brain needs to understand the contingencies between its sensory inputs, actions, and rewards. This is analogous to the statistical inference that has been extensively studied in the natural language processing field, where recent developments of recurrent neural networks have found many successes. We wonder whether these neural networks, the gated recurrent unit (GRU) networks in particular, reflect how the brain solves the contingency problem. Therefore, we build a GRU network framework inspired by the statistical learning approach of NLP and test it with four exemplar behavior tasks previously used in empirical studies. The network models are trained to predict future events based on past events, both comprising sensory, action, and reward events. We show the networks can successfully reproduce animal and human behavior. The networks generalize the training, perform Bayesian inference in novel conditions, and adapt their choices when event contingencies vary. Importantly, units in the network encode task variables and exhibit activity patterns that match previous neurophysiology findings. Our results suggest that the neural network approach based on statistical sequence learning may reflect the brain\u2019s computational principle underlying flexible and adaptive behaviors and serve as a useful approach to understand the brain.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1008342","type":"journal-article","created":{"date-parts":[[2020,11,3]],"date-time":"2020-11-03T14:06:35Z","timestamp":1604412395000},"page":"e1008342","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":18,"title":["A recurrent neural network framework for flexible and adaptive decision making based on sequence learning"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5841-4239","authenticated-orcid":true,"given":"Zhewei","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huzi","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6976-9246","authenticated-orcid":true,"given":"Tianming","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2020,11,3]]},"reference":[{"key":"pcbi.1008342.ref001","volume-title":"Foundations of Statistical Natural Language Processing","author":"CD Manning","year":"1999"},{"key":"pcbi.1008342.ref002","author":"W Yin","year":"2017","journal-title":"Comparative Study of CNN and RNN for Natural Language Processing"},{"key":"pcbi.1008342.ref003","author":"K Cho","year":"2014","journal-title":"Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation"},{"key":"pcbi.1008342.ref004","author":"J Chung","year":"2014","journal-title":"Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling"},{"key":"pcbi.1008342.ref005","article-title":"An empirical exploration of recurrent network architectures","author":"R Jozefowicz","year":"2015","journal-title":"Proceedings of the 32nd"},{"key":"pcbi.1008342.ref006","first-page":"1","article-title":"LSTM: A search space odyssey.","author":"K Greff","year":"2016","journal-title":"IEEE Trans"},{"key":"pcbi.1008342.ref007","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory.","volume":"9","author":"S Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"pcbi.1008342.ref008","article-title":"Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia","author":"RC O\u2019Reilly","year":"2006","journal-title":"Neural Comput."},{"key":"pcbi.1008342.ref009","author":"RP Costa","year":"2017","journal-title":"Cortical microcircuits as gated-recurrent neural networks"},{"key":"pcbi.1008342.ref010","first-page":"6","article-title":"Reword-based training of recurrent neural networks for cognitive and value-based tasks","author":"HF Song","year":"2017","journal-title":"Elife"},{"key":"pcbi.1008342.ref011","first-page":"1","article-title":"Task representations in neural networks trained to perform many cognitive tasks","author":"GR Yang","year":"2019","journal-title":"Nat Neurosci"},{"key":"pcbi.1008342.ref012","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1038\/s41592-018-0109-9","article-title":"Inferring single-trial neural population dynamics using sequential auto-encoders","volume":"15","author":"C Pandarinath","year":"2018","journal-title":"Nat Methods"},{"key":"pcbi.1008342.ref013","author":"DP Kingma","year":"2014","journal-title":"Adam: A Method for Stochastic Optimization"},{"key":"pcbi.1008342.ref014","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.neuron.2015.01.007","article-title":"A neural implementation of Wald\u2019s sequential probability ratio test","volume":"85","author":"S Kira","year":"2015","journal-title":"Neuron"},{"key":"pcbi.1008342.ref015","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1214\/aoms\/1177730197","article-title":"Optimum Character of the Sequential Probability Ratio Test","volume":"19","author":"A Wald","year":"1948","journal-title":"Ann Math Stat."},{"key":"pcbi.1008342.ref016","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1038\/5739","article-title":"Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque","volume":"2","author":"J-N Kim","year":"1999","journal-title":"Nat Neurosci"},{"key":"pcbi.1008342.ref017","doi-asserted-by":"crossref","first-page":"9475","DOI":"10.1523\/JNEUROSCI.22-21-09475.2002","article-title":"Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task","volume":"22","author":"JD Roitman","year":"2002","journal-title":"J Neurosci"},{"key":"pcbi.1008342.ref018","doi-asserted-by":"crossref","first-page":"1075","DOI":"10.1038\/nature05852","article-title":"Probabilistic reasoning by neurons","volume":"447","author":"T Yang","year":"2007","journal-title":"Nature"},{"key":"pcbi.1008342.ref019","doi-asserted-by":"crossref","first-page":"15747","DOI":"10.1523\/JNEUROSCI.2894-10.2010","article-title":"Caudate encodes multiple computations for perceptual decisions","volume":"30","author":"L Ding","year":"2010","journal-title":"J Neurosci"},{"key":"pcbi.1008342.ref020","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1016\/j.neuron.2017.07.039","article-title":"The Basal Ganglia Do Not Select Reach Targets but Control the Urgency of Commitment","volume":"95","author":"D Thura","year":"2017","journal-title":"Neuron"},{"key":"pcbi.1008342.ref021","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1016\/j.neuron.2010.12.037","article-title":"Variance as a Signature of Neural Computations during Decision Making","volume":"69","author":"AK Churchland","year":"2011","journal-title":"Neuron"},{"key":"pcbi.1008342.ref022","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.02260","article-title":"A neural mechanism of speed-accuracy tradeoff in macaque area LIP","volume":"3","author":"T Hanks","year":"2014","journal-title":"Elife"},{"key":"pcbi.1008342.ref023","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1016\/j.neuron.2012.08.030","article-title":"Neural mechanisms of speed-accuracy tradeoff","volume":"76","author":"RP Heitz","year":"2012","journal-title":"Neuron"},{"key":"pcbi.1008342.ref024","doi-asserted-by":"crossref","first-page":"865","DOI":"10.1016\/j.neuron.2012.07.021","article-title":"Separate, causal roles of the caudate in saccadic choice and execution in a perceptual decision task","volume":"75","author":"L Ding","year":"2012","journal-title":"Neuron"},{"key":"pcbi.1008342.ref025","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.1038\/nn.2191","article-title":"Neural correlates of multisensory cue integration in macaque MSTd","volume":"11","author":"Y Gu","year":"2008","journal-title":"Nat Neurosci"},{"key":"pcbi.1008342.ref026","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1126\/science.1169405","article-title":"Representation of confidence associated with a decision by neurons in the parietal cortex","volume":"324","author":"R Kiani","year":"2009","journal-title":"Science"},{"key":"pcbi.1008342.ref027","doi-asserted-by":"crossref","first-page":"1204","DOI":"10.1016\/j.neuron.2011.02.027","article-title":"Model-Based Influences on Humans\u2019 Choices and Striatal Prediction Errors","volume":"69","author":"ND Daw","year":"2011","journal-title":"Neuron"},{"key":"pcbi.1008342.ref028","doi-asserted-by":"crossref","first-page":"e1004648","DOI":"10.1371\/journal.pcbi.1004648","article-title":"Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task","volume":"11","author":"T Akam","year":"2015","journal-title":"PLOS Comput Biol."},{"key":"pcbi.1008342.ref029","first-page":"336","volume-title":"Annals of the New York Academy of Sciences","author":"CD Salzman","year":"2007"},{"key":"pcbi.1008342.ref030","first-page":"1","article-title":"Causal contribution and dynamical encoding in the striatum during evidence accumulation","volume":"7","author":"MM Yartsev","year":"2018"},{"key":"pcbi.1008342.ref031","doi-asserted-by":"crossref","first-page":"137","DOI":"10.3758\/CABN.1.2.137","article-title":"Interactions between frontal cortex and basal ganglia in working memory: a computational model","volume":"1","author":"MJ Frank","year":"2001","journal-title":"Cogn Affect Behav Neurosci"},{"key":"pcbi.1008342.ref032","first-page":"456194","article-title":"Specialized and spatially organized coding of sensory, motor, and cognitive variables in midbrain dopamine neurons","author":"B Engelhard","year":"2018"},{"key":"pcbi.1008342.ref033","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1016\/j.neuron.2018.01.049","article-title":"Activation of Striatal Neurons Causes a Perceptual Decision Bias during Visual Change Detection in Mice","volume":"97","author":"L Wang","year":"2018","journal-title":"Neuron"},{"key":"pcbi.1008342.ref034","doi-asserted-by":"crossref","first-page":"140","DOI":"10.3389\/fnbeh.2017.00140","article-title":"Networks of VTA Neurons Encode Real-Time Information about Uncertain Numbers of Actions Executed to Earn a Reward","volume":"11","author":"J Wood","year":"2017","journal-title":"Front Behav Neurosci."},{"key":"pcbi.1008342.ref035","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1038\/nature09263","article-title":"Start\/stop signals emerge in nigrostriatal circuits during sequence learning","volume":"466","author":"X Jin","year":"2010","journal-title":"Nature"},{"key":"pcbi.1008342.ref036","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/j.cell.2018.06.012","article-title":"Optogenetic Editing Reveals the Hierarchical Organization of Learned Action Sequences","volume":"174","author":"CE Geddes","year":"2018","journal-title":"Cell"},{"key":"pcbi.1008342.ref037","doi-asserted-by":"crossref","first-page":"a021691","DOI":"10.1101\/cshperspect.a021691","article-title":"The striatum: where skills and habits meet","volume":"7","author":"AM Graybiel","year":"2015","journal-title":"Cold Spring Harb Perspect Biol"},{"key":"pcbi.1008342.ref038","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1016\/0959-4388(95)80100-6","article-title":"Building action repertoires: memory and learning functions of the basal ganglia","volume":"5","author":"AM Graybiel","year":"1995","journal-title":"Curr Opin Neurobiol"},{"key":"pcbi.1008342.ref039","doi-asserted-by":"crossref","first-page":"1468","DOI":"10.1162\/neco.2007.19.6.1468","article-title":"Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity","volume":"19","author":"V Florian R","year":"2007","journal-title":"Neural Comput"},{"key":"pcbi.1008342.ref040","doi-asserted-by":"crossref","first-page":"e1005925","DOI":"10.1371\/journal.pcbi.1005925","article-title":"A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning","volume":"14","author":"Z Zhang","year":"2018","journal-title":"PLOS Comput Biol"},{"key":"pcbi.1008342.ref041","doi-asserted-by":"crossref","first-page":"860","DOI":"10.1038\/s41593-018-0147-8","article-title":"Prefrontal cortex as a meta-reinforcement learning system","volume":"21","author":"JX Wang","year":"2018","journal-title":"Nat Neurosci"},{"key":"pcbi.1008342.ref042","doi-asserted-by":"crossref","first-page":"672","DOI":"10.3758\/BF03196324","article-title":"Operant variability: Evidence, functions, and theory","volume":"9","author":"A Neuringer","year":"2002","journal-title":"Psychon Bull Rev"},{"key":"pcbi.1008342.ref043","first-page":"1593","volume":"275","author":"W Schultz","year":"1997","journal-title":"Neural Substrate of Prediction and. Science"},{"key":"pcbi.1008342.ref044","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1016\/j.biopsych.2011.12.019","article-title":"Phasic Mesolimbic Dopamine Signaling Precedes and Predicts Performance of a Self-Initiated Action Sequence Task","volume":"71","author":"KM Wassum","year":"2012","journal-title":"Biol Psychiatry"},{"key":"pcbi.1008342.ref045","first-page":"1","article-title":"Scientific collaboration networks. II. Shortest paths, weighted networks, and centralit","volume":"64","author":"MEJ Newman","year":"2001"},{"key":"pcbi.1008342.ref046","volume-title":"Network Flows: Theory, Algorithms, and Applications | Pearson","author":"RK Ahuja","year":"1993"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1008342","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2020,11,18]],"date-time":"2020-11-18T00:00:00Z","timestamp":1605657600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,18]],"date-time":"2020-11-18T14:23:16Z","timestamp":1605709396000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008342"}},"subtitle":[],"editor":[{"given":"Alireza","family":"Soltani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2020,11,3]]},"references-count":46,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,11,3]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1008342","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/555862","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,3]]}}}