{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T01:50:47Z","timestamp":1780969847021,"version":"3.54.1"},"reference-count":33,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T00:00:00Z","timestamp":1626652800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T00:00:00Z","timestamp":1626652800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Despite the tremendous success of deep neural networks in machine learning, the underlying reason for their superior learning capability remains unclear. Here, we present a framework based on statistical physics to study the dynamics of stochastic gradient descent (SGD), which drives learning in neural networks. Using the minibatch gradient ensemble, we construct order parameters to characterize the dynamics of weight updates in SGD. In the case without mislabeled data, we find that the SGD learning dynamics transitions from a fast learning phase to a slow exploration phase, which is associated with large changes in the order parameters that characterize the alignment of SGD gradients and their mean amplitude. In a more complex case, with randomly mislabeled samples, the SGD learning dynamics falls into four distinct phases. First, the system finds solutions for the correctly labeled samples in phase I; it then wanders around these solutions in phase II until it finds a direction that enables it to learn the mislabeled samples during phase III, after which, it finds solutions that satisfy all training samples during phase IV. Correspondingly, the test error decreases during phase I and remains low during phase II; however, it increases during phase III and reaches a high plateau during phase IV. The transitions between different phases can be understood by examining changes in the order parameters that characterize the alignment of the mean gradients for the two datasets (correctly and incorrectly labeled samples) and their (relative) strengths during learning. We find that individual sample losses for the two datasets are separated the most during phase II, leading to a data cleansing process that eliminates mislabeled samples and improves generalization. Overall, we believe that an approach based on statistical physics and stochastic dynamic systems theory provides a promising framework for describing and understanding learning dynamics in neural networks, which may also lead to more efficient learning algorithms.<\/jats:p>","DOI":"10.1088\/2632-2153\/abf5b9","type":"journal-article","created":{"date-parts":[[2021,4,8]],"date-time":"2021-04-08T00:00:56Z","timestamp":1617840056000},"page":"043001","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Phases of learning dynamics in artificial neural networks in the absence or presence of mislabeled data"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3688-0284","authenticated-orcid":false,"given":"Yu","family":"Feng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4589-981X","authenticated-orcid":false,"given":"Yuhai","family":"Tu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2021,7,19]]},"reference":[{"key":"mlstabf5b9bib1","doi-asserted-by":"publisher","first-page":"436 E","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"mlstabf5b9bib2","volume":"vol 1","author":"Goodfellow","year":"2016"},{"key":"mlstabf5b9bib3","first-page":"pp 770","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"mlstabf5b9bib4","article-title":"Google\u2019s neural machine translation system: bridging the gap between human and machine translation","author":"Wu","year":"2016"},{"key":"mlstabf5b9bib5","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"mlstabf5b9bib6","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/d41586-020-03348-4","article-title":"\u2018It will change everything\u2019: Deepmind\u2019s ai makes gigantic leap in solving protein structures","volume":"588","author":"Callaway","year":"2020","journal-title":"Nature"},{"key":"mlstabf5b9bib7","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1214\/aoms\/1177729586","article-title":"A stochastic approximation method","volume":"22","author":"Robbins","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"mlstabf5b9bib8","first-page":"pp 177","article-title":"Large-scale machine learning with stochastic gradient descent","author":"Bottou","year":"2010"},{"key":"mlstabf5b9bib9","author":"Kampen","year":"2010"},{"key":"mlstabf5b9bib10","doi-asserted-by":"crossref","DOI":"10.1109\/ITA.2018.8503224","article-title":"Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks","author":"Chaudhari","year":"2018"},{"key":"mlstabf5b9bib11","author":"Forster","year":"2018"},{"key":"mlstabf5b9bib12","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1073\/pnas.2015617118","article-title":"The inverse variance\u2013flatness relation in stochastic gradient descent is critical for finding flat minima","volume":"118","author":"Feng","year":"2021","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstabf5b9bib13","doi-asserted-by":"publisher","first-page":"3214","DOI":"10.1080\/00268976.2018.1483535","article-title":"Energy\u2013entropy competition and the effectiveness of stochastic gradient descent in machine learning","volume":"116","author":"Zhang","year":"2018","journal-title":"Mol. Phys."},{"key":"mlstabf5b9bib14","article-title":"Unsupervised label noise modeling and loss correction","author":"Arazo","year":"2019"},{"key":"mlstabf5b9bib15","article-title":"Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks","author":"Li","year":"2020"},{"key":"mlstabf5b9bib16","author":"Zhang","year":"2016"},{"key":"mlstabf5b9bib17","author":"Arpit","year":"2017"},{"key":"mlstabf5b9bib18","doi-asserted-by":"publisher","first-page":"15849","DOI":"10.1073\/pnas.1903070116","article-title":"Reconciling modern machine-learning practice and the classical bias\u2013variance trade-off","volume":"116","author":"Belkin","year":"2019","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstabf5b9bib19","author":"Brutzkus","year":"2017"},{"key":"mlstabf5b9bib20","first-page":"8157","article-title":"Learning overparameterized neural networks via stochastic gradient descent on structured data","volume":"31","author":"Li","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"mlstabf5b9bib21","article-title":"The generalization error of random features regression: precise asymptotics and double descent curve","author":"Mei","year":"2019"},{"key":"mlstabf5b9bib22","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/ab633c","article-title":"Scaling description of generalization with number of parameters in deep learning","volume":"2020","author":"Geiger","year":"2020","journal-title":"J. Stat. Mech.: Theory Exp."},{"key":"mlstabf5b9bib23","article-title":"Generalisation error in learning with random features and the hidden manifold model","author":"Gerace","year":"2020"},{"key":"mlstabf5b9bib24","article-title":"Exploring generalization in deep learning","author":"Neyshabur","year":"2017"},{"key":"mlstabf5b9bib25","article-title":"High-dimensional dynamics of generalization error in neural networks","author":"Advani","year":"2017"},{"key":"mlstabf5b9bib26","article-title":"Fantastic generalization measures and where to find them","author":"Jiang","year":"2019"},{"key":"mlstabf5b9bib27","article-title":"Opening the black box of deep neural networks via information","author":"Shwartz-Ziv","year":"2017"},{"key":"mlstabf5b9bib28","doi-asserted-by":"crossref","DOI":"10.1109\/ITW.2015.7133169","article-title":"Deep learning and the information bottleneck principle","author":"Tishby","year":"2015"},{"key":"mlstabf5b9bib29","article-title":"How transferable are features in deep neural networks?","author":"Yosinski","year":"2014"},{"key":"mlstabf5b9bib30","first-page":"pp 41","article-title":"Curriculum learning","author":"Bengio","year":"2009"},{"key":"mlstabf5b9bib31","article-title":"Continual learning in reinforcement environments","author":"Ring","year":"1994"},{"key":"mlstabf5b9bib32","article-title":"Gradient episodic memory for continuum learning","author":"Lopez-Paz","year":"2017"},{"key":"mlstabf5b9bib33","article-title":"Learning to learn without forgetting by maximizing transfer and minimizing interference","author":"Riemer","year":"2018"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T14:57:52Z","timestamp":1639407472000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abf5b9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,19]]},"references-count":33,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,7,19]]},"published-print":{"date-parts":[[2021,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/abf5b9","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,19]]},"assertion":[{"value":"Phases of learning dynamics in artificial neural networks in the absence or presence of mislabeled data","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-12-28","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-04-07","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-07-19","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}