{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T16:22:14Z","timestamp":1759335734894,"version":"3.37.3"},"reference-count":66,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2024,3,21]],"date-time":"2024-03-21T00:00:00Z","timestamp":1710979200000},"content-version":"vor","delay-in-days":20,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,21]],"date-time":"2024-03-21T00:00:00Z","timestamp":1710979200000},"content-version":"tdm","delay-in-days":20,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"Chicago Prize Postdoctoral Fellowship"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The recent success of neural networks in natural language processing has drawn renewed attention to learning sequence-to-sequence (seq2seq) tasks. While there exists a rich literature that studies classification and regression tasks using solvable models of neural networks, seq2seq tasks have not yet been studied from this perspective. Here, we propose a simple model for a seq2seq task that has the advantage of providing explicit control over the degree of memory, or non-Markovianity, in the sequences\u2014the stochastic switching-Ornstein\u2013Uhlenbeck (SSOU) model. We introduce a measure of non-Markovianity to quantify the amount of memory in the sequences. For a minimal auto-regressive (AR) learning model trained on this task, we identify two learning regimes corresponding to distinct phases in the stationary state of the SSOU process. These phases emerge from the interplay between two different time scales that govern the sequence statistics. Moreover, we observe that while increasing the integration window of the AR model always improves performance, albeit with diminishing returns, increasing the non-Markovianity of the input sequences can improve or degrade its performance. Finally, we perform experiments with recurrent and convolutional neural networks that show that our observations carry over to more complicated neural network architectures.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad2feb","type":"journal-article","created":{"date-parts":[[2024,3,4]],"date-time":"2024-03-04T22:24:01Z","timestamp":1709591041000},"page":"015053","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["The impact of memory on learning sequence-to-sequence tasks"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5419-5999","authenticated-orcid":false,"given":"Alireza","family":"Seif","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5946-5684","authenticated-orcid":false,"given":"Sarah A M","family":"Loos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gennaro","family":"Tucci","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7196-8404","authenticated-orcid":false,"given":"\u00c9dgar","family":"Rold\u00e1n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5799-7644","authenticated-orcid":true,"given":"Sebastian","family":"Goldt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2024,3,21]]},"reference":[{"article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","year":"2019","author":"Devlin","key":"mlstad2febbib1"},{"year":"2018","author":"Howard","key":"mlstad2febbib2"},{"article-title":"Improving language understanding by generative pre-training","year":"2018","author":"Radford","key":"mlstad2febbib3"},{"key":"mlstad2febbib4","first-page":"pp 1877","article-title":"Language models are few-shot learners","volume":"vol 33","author":"Brown","year":"2020"},{"article-title":"Gpt-4 technical report","year":"2023","author":"OpenAI","key":"mlstad2febbib5"},{"volume":"vol 7","year":"2004","author":"Kantz","key":"mlstad2febbib6"},{"year":"2015","author":"Box","key":"mlstad2febbib7"},{"key":"mlstad2febbib8","doi-asserted-by":"publisher","first-page":"1983","DOI":"10.1088\/0305-4470\/22\/12\/004","article-title":"Three unfinished works on the optimal storage capacity of networks","volume":"22","author":"Gardner","year":"1989","journal-title":"J. Phys. A: Math. Gen."},{"key":"mlstad2febbib9","doi-asserted-by":"publisher","first-page":"6056","DOI":"10.1103\/PhysRevA.45.6056","article-title":"Statistical mechanics of learning from examples","volume":"45","author":"Seung","year":"1992","journal-title":"Phys. Rev. A"},{"year":"2001","author":"Engel","key":"mlstad2febbib10"},{"year":"2009","author":"Mezard","key":"mlstad2febbib11"},{"key":"mlstad2febbib12","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.91.045002","article-title":"Machine learning and the physical sciences","volume":"91","author":"Carleo","year":"2019","journal-title":"Rev. Mod. Phys."},{"key":"mlstad2febbib13","first-page":"pp 1097","article-title":"ImageNet classification with deep convolutional neural networks","author":"Krizhevsky","year":"2012"},{"article-title":"Very deep convolutional networks for large-scale image recognition","year":"2015","author":"Simonyan","key":"mlstad2febbib14"},{"key":"mlstad2febbib15","first-page":"pp 770","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"article-title":"An image is worth 16x16 words: transformers for image recognition at scale","year":"2021","author":"Dosovitskiy","key":"mlstad2febbib16"},{"article-title":"The intrinsic dimension of images and its impact on learning","year":"2021","author":"Pope","key":"mlstad2febbib17"},{"key":"mlstad2febbib18","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevX.8.031003","article-title":"Classification and geometry of general perceptual manifolds","volume":"8","author":"Chung","year":"2018","journal-title":"Phys. Rev. X"},{"key":"mlstad2febbib19","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevX.10.041044","article-title":"Modeling the influence of data structure on learning in neural networks: the hidden manifold model","volume":"10","author":"Goldt","year":"2020","journal-title":"Phys. Rev. X"},{"key":"mlstad2febbib20","first-page":"pp 426","article-title":"The Gaussian equivalence of generative models for learning with shallow neural networks","author":"Goldt","year":"2022"},{"key":"mlstad2febbib21","article-title":"When do neural networks outperform kernel methods?","volume":"vol 33","author":"Ghorbani","year":"2020"},{"key":"mlstad2febbib22","first-page":"pp 3889","article-title":"Asymptotics of ridge(less) regression under general source condition","author":"Richards","year":"2021"},{"key":"mlstad2febbib23","first-page":"pp 1305","article-title":"Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss","author":"Chizat","year":"2020"},{"key":"mlstad2febbib24","first-page":"pp 8936","article-title":"Classifying high-dimensional Gaussian mixtures: where kernel methods fail and neural networks succeed","author":"Refinetti","year":"2021"},{"key":"mlstad2febbib25","article-title":"Learning Gaussian mixtures with generalized linear models: precise asymptotics in high-dimensions","volume":"vol 34","author":"Loureiro","year":"2021"},{"key":"mlstad2febbib26","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/abc61d","article-title":"Asymptotic learning curves of kernel methods: empirical data versus teacher\u2013student paradigm","author":"Spigler","year":"2020","journal-title":"J. Stat. Mech."},{"key":"mlstad2febbib27","first-page":"pp 8506","article-title":"On the interplay between data structure and loss function in classification problems","volume":"vol 34","author":"d\u2019Ascoli","year":"2021"},{"key":"mlstad2febbib28","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2018422118","article-title":"Place cells may simply be memory cells: memory compression leads to spatial tuning and history dependence","volume":"118","author":"Benna","year":"2021","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstad2febbib29","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac4f3f","article-title":"Probing transfer learning with a model of synthetic correlated datasets","volume":"3","author":"Gerace","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad2febbib30","first-page":"pp 9111","article-title":"Limitations of lazy training of two-layers neural network","volume":"vol 32","author":"Ghorbani","year":"2019"},{"key":"mlstad2febbib31","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.129.030603","article-title":"Modeling active non-Markovian oscillations","volume":"129","author":"Tucci","year":"2022","journal-title":"Phys. Rev. Lett."},{"key":"mlstad2febbib32","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.96.012101","article-title":"Finite-time generalization of the thermodynamic uncertainty relation","volume":"96","author":"Pietzonka","year":"2017","journal-title":"Phys. Rev. E"},{"article-title":"Variance sum rule for entropy production","year":"2023","author":"Di Terlizzi","key":"mlstad2febbib33"},{"year":"1992","author":"Van Kampen","key":"mlstad2febbib34"},{"key":"mlstad2febbib35","doi-asserted-by":"publisher","first-page":"5522","DOI":"10.1364\/AO.51.005522","article-title":"Force mapping of an optical trap using an acousto-optical deflector in a time-sharing regime","volume":"51","author":"Martinez","year":"2012","journal-title":"Appl. Opt."},{"key":"mlstad2febbib36","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.87.032159","article-title":"Effective heating to several thousand kelvins of an optically trapped sphere in a liquid","volume":"87","author":"Mart\u00ednez","year":"2013","journal-title":"Phys. Rev. E"},{"key":"mlstad2febbib37","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1162\/neco.1989.1.4.541","article-title":"Backpropagation applied to handwritten zip code recognition","volume":"1","author":"LeCun","year":"1989","journal-title":"Neural Comput."},{"year":"2016","author":"Goodfellow","key":"mlstad2febbib38"},{"key":"mlstad2febbib39","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1109\/TSSC.1969.300225","article-title":"Visual feature extraction by a multilayered network of analog threshold elements","volume":"5","author":"Fukushima","year":"1969","journal-title":"IEEE Trans. Syst. Sci. Cybern."},{"key":"mlstad2febbib40","first-page":"pp 103","article-title":"On the properties of neural machine translation: encoder\u2013decoder approaches","author":"Cho","year":"2014"},{"article-title":"Adam: a method for stochastic optimization","year":"2014","author":"Kingma","key":"mlstad2febbib41"},{"article-title":"Code for data generation and training and testing machine learning models","year":"2022","author":"Seif","key":"mlstad2febbib42"},{"first-page":"pp 103","year":"1992","author":"Kloeden","key":"mlstad2febbib43"},{"key":"mlstad2febbib44","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevResearch.3.L022018","article-title":"Toolbox for quantifying memory in dynamics along reaction coordinates","volume":"3","author":"Lapolla","year":"2021","journal-title":"Phys. Rev. Res."},{"key":"mlstad2febbib45","doi-asserted-by":"publisher","first-page":"182","DOI":"10.3389\/fphy.2019.00182","article-title":"Manifestations of projection-induced memory: general theory and the tilted single file","volume":"7","author":"Lapolla","year":"2019","journal-title":"Front. Phys."},{"key":"mlstad2febbib46","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.81.062115","article-title":"Measure for the non-Markovianity of quantum processes","volume":"81","author":"Laine","year":"2010","journal-title":"Phys. Rev. A"},{"key":"mlstad2febbib47","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.89.042120","article-title":"Canonical form of master equations and characterization of non-Markovianity","volume":"89","author":"Hall","year":"2014","journal-title":"Phys. Rev. A"},{"key":"mlstad2febbib48","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.105.050403","article-title":"Entanglement and non-Markovianity of quantum evolutions","volume":"105","author":"Rivas","year":"2010","journal-title":"Phys. Rev. Lett."},{"key":"mlstad2febbib49","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevA.104.032212","article-title":"Quantifying non-Markovianity via conditional mutual information","volume":"104","author":"Huang","year":"2021","journal-title":"Phys. Rev. A"},{"key":"mlstad2febbib50","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.121.040601","article-title":"Response functions as quantifiers of non-Markovianity","volume":"121","author":"Strasberg","year":"2018","journal-title":"Phys. Rev. Lett."},{"key":"mlstad2febbib51","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1002\/j.1537-2197.1987.tb08741.x","article-title":"Effect of seed dimorphism on the density-dependent dynamics of experimental populations of atriplex triangularis (chenopodiaceae)","volume":"74","author":"Ellison","year":"1987","journal-title":"Am. J. Bot."},{"key":"mlstad2febbib52","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"mlstad2febbib53","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1103\/PhysRevLett.61.259","article-title":"Chaos in random neural networks","volume":"61","author":"Sompolinsky","year":"1988","journal-title":"Phys. Rev. Lett."},{"key":"mlstad2febbib54","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.neuron.2009.07.018","article-title":"Generating coherent patterns of activity from chaotic neural networks","volume":"63","author":"Sussillo","year":"2009","journal-title":"Neuron"},{"key":"mlstad2febbib55","doi-asserted-by":"publisher","first-page":"609","DOI":"10.1016\/j.neuron.2018.07.003","article-title":"Linking connectivity, dynamics and computations in low-rank recurrent neural networks","volume":"99","author":"Mastrogiuseppe","year":"2018","journal-title":"Neuron"},{"key":"mlstad2febbib56","first-page":"pp 13352","article-title":"The interplay between randomness and structure during learning in RNNs","volume":"vol 33","author":"Schuessler","year":"2020"},{"key":"mlstad2febbib57","doi-asserted-by":"publisher","DOI":"10.1063\/1.4986932","article-title":"Nonlinear dynamics in the study of birdsong","volume":"27","author":"Mindlin","year":"2017","journal-title":"Chaos"},{"key":"mlstad2febbib58","doi-asserted-by":"publisher","first-page":"3423","DOI":"10.1175\/JCLI-D-17-0559.1","article-title":"Fast physics and slow physics in the nonlinear Dansgaard\u2013Oeschger relaxation oscillation","volume":"31","author":"Vettoretti","year":"2018","journal-title":"J. Clim."},{"key":"mlstad2febbib59","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/ab33fa","article-title":"Effective bandwidth of non-Markovian packet traffic","author":"Cavallaro","year":"2019","journal-title":"J. Stat. Mech."},{"key":"mlstad2febbib60","doi-asserted-by":"publisher","DOI":"10.1088\/1367-2630\/ac0f18","article-title":"Quantifying entropy production in active fluctuations of the hair-cell bundle from time irreversibility and uncertainty relations","volume":"23","author":"Rold\u00e1n","year":"2021","journal-title":"New J. Phys."},{"key":"mlstad2febbib61","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.102.032209","article-title":"Volterra-series approach to stochastic nonlinear dynamics: linear response of the Van der Pol oscillator driven by white noise","volume":"102","author":"Belousov","year":"2020","journal-title":"Phys. Rev. E"},{"key":"mlstad2febbib62","doi-asserted-by":"publisher","first-page":"595","DOI":"10.1038\/s41567-019-0445-4","article-title":"Stochastic nonlinear dynamics of confined cell migration in two-state systems","volume":"15","author":"Br\u00fcckner","year":"2019","journal-title":"Nat. Phys."},{"key":"mlstad2febbib63","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.127.198101","article-title":"Estimating entropy production from waiting time distributions","volume":"127","author":"Skinner","year":"2021","journal-title":"Phys. Rev. Lett."},{"key":"mlstad2febbib64","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/ncomms14106","article-title":"Prediction and real-time compensation of qubit decoherence via machine learning","volume":"8","author":"Mavadia","year":"2017","journal-title":"Nat. Commun."},{"key":"mlstad2febbib65","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41534-020-0251-y","article-title":"Real-time calibration with spectator qubits","volume":"6","author":"Majumder","year":"2020","journal-title":"npj Quantum Inf."},{"key":"mlstad2febbib66","first-page":"pp 249","article-title":"Understanding the difficulty of training deep feedforward neural networks","author":"Glorot","year":"2010"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,21]],"date-time":"2024-03-21T12:00:31Z","timestamp":1711022431000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad2feb"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,1]]},"references-count":66,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,3,21]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad2feb","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2024,3,1]]},"assertion":[{"value":"The impact of memory on learning sequence-to-sequence tasks","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2023-05-26","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-03-04","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-03-21","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}