{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T16:24:26Z","timestamp":1768494266077,"version":"3.49.0"},"reference-count":63,"publisher":"MIT Press - Journals","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2020,1]]},"abstract":"<jats:p> We analyze the effect of synchronization on distributed stochastic gradient algorithms. By exploiting an analogy with dynamical models of biological quorum sensing, where synchronization between agents is induced through communication with a common signal, we quantify how synchronization can significantly reduce the magnitude of the noise felt by the individual distributed agents and their spatial mean. This noise reduction is in turn associated with a reduction in the smoothing of the loss function imposed by the stochastic gradient approximation. Through simulations on model nonconvex objectives, we demonstrate that coupling can stabilize higher noise levels and improve convergence. We provide a convergence analysis for strongly convex functions by deriving a bound on the expected deviation of the spatial mean of the agents from the global minimizer for an algorithm based on quorum sensing, the same algorithm with momentum, and the elastic averaging SGD (EASGD) algorithm. We discuss extensions to new algorithms that allow each agent to broadcast its current measure of success and shape the collective computation accordingly. We supplement our theoretical analysis with numerical experiments on convolutional neural networks trained on the CIFAR-10 data set, where we note a surprising regularizing property of EASGD even when applied to the non-distributed case. This observation suggests alternative second-order in time algorithms for nondistributed optimization that are competitive with momentum methods. <\/jats:p>","DOI":"10.1162\/neco_a_01248","type":"journal-article","created":{"date-parts":[[2019,11,9]],"date-time":"2019-11-09T00:19:30Z","timestamp":1573258770000},"page":"36-96","source":"Crossref","is-referenced-by-count":8,"title":["A Continuous-Time Analysis of Distributed Stochastic Gradient"],"prefix":"10.1162","volume":"32","author":[{"given":"Nicholas M.","family":"Boffi","sequence":"first","affiliation":[{"name":"John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, U.S.A."}]},{"given":"Jean-Jacques E.","family":"Slotine","sequence":"additional","affiliation":[{"name":"Nonlinear Systems Laboratory, MIT, Cambridge, MA 02139, U.S.A."}]}],"member":"281","reference":[{"key":"B1","first-page":"773","volume-title":"Advances in neural information processing systems","volume":"26","author":"Bach F.","year":"2013"},{"key":"B2","author":"Banburski A.","year":"2019","journal-title":"CoRR"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1109\/LPT.2005.844008"},{"key":"B4","volume-title":"On-line learning in neural networks","author":"Bottou L.","year":"1998"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"B6","author":"Bouvrie J.","year":"2013","journal-title":"Synchronization and noise: A mechanism for regularization in neural systems"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1561\/2200000016"},{"key":"B8","author":"Chaudhari P.","year":"2017","journal-title":"Parle: Parallelizing stochastic gradient descent"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1007\/s40687-018-0148-y"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1109\/ITA.2018.8503224"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2009.2014125"},{"key":"B12","first-page":"1223","volume-title":"Advances in neural information processing systems","volume":"25","author":"Dean J.","year":"2012"},{"key":"B13","first-page":"1646","volume-title":"Advances in neural information processing systems","volume":"27","author":"Defazio A.","year":"2014"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4243"},{"key":"B15","first-page":"2121","volume":"12","author":"Duchi J.","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.4310\/CMS.2018.v16.n3.a7"},{"key":"B17","volume-title":"Stochastic methods","author":"Gardiner C.","year":"2009","edition":"4"},{"key":"B18","volume-title":"Deep learning","author":"Goodfellow I.","year":"2016"},{"key":"B19","author":"He K.","year":"2015","journal-title":"CoRR"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139020411"},{"key":"B21","author":"Hu W.","year":"2017","journal-title":"On the diffusion approximation of nonconvex stochastic gradient descent"},{"key":"B22","author":"Ioffe S.","year":"2015","journal-title":"CoRR"},{"key":"B23","author":"Jastrz\u0229bski S.","year":"2017","journal-title":"Three factors influencing minima in SGD"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.78.011108"},{"key":"B25","author":"Keskar N. S.","year":"2016","journal-title":"On large-batch training for deep learning: Generalization gap and sharp minima"},{"key":"B26","volume-title":"Nonlinear systems","author":"Khalil H. K.","year":"2002","edition":"3"},{"key":"B27","author":"Kleinberg R.","year":"2018","journal-title":"An alternative view: When does SGD escape local minima?"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-12616-5"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"B30","author":"Li Q.","year":"2018","journal-title":"Stochastic modified equations and dynamics of stochastic gradient algorithms I: Mathematical foundations"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1016\/S0005-1098(98)00019-3"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2015.0203"},{"key":"B33","author":"Mandt S.","year":"2015","journal-title":"NIPS Workshop on Optimization for Machine Learning"},{"key":"B34","first-page":"354","author":"Mandt S.","year":"2016","journal-title":"Proceedings of the 33rd International Conference on Machine Learning"},{"issue":"1","key":"B35","first-page":"4873","volume":"18","author":"Mandt S.","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.micro.55.1.165"},{"key":"B37","author":"Neelakantan A.","year":"2015","journal-title":"Adding gradient noise improves learning for very deep networks"},{"key":"B38","first-page":"367","volume":"26","author":"Nesterov Y.","year":"1983","journal-title":"Soviet Mathematics Doklady"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-8853-9"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2008.2009619"},{"key":"B41","author":"Poggio T.","year":"2017","journal-title":"Theory of deep learning III: Explaining the non-overfitting puzzle"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1137\/0330046"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1007\/s12532-013-0053-8"},{"key":"B44","first-page":"693","volume-title":"Advances in neural information processing systems","volume":"24","author":"Recht B.","year":"2011"},{"key":"B45","first-page":"233","volume-title":"Optimizing methods in statistics","author":"Robbins H.","year":"1971"},{"key":"B46","first-page":"2663","volume-title":"Advances in neural information processing systems","volume":"25","author":"Roux N. L.","year":"2012"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.82.041919"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1016\/J.ACTPSY.2012.07.016"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-016-1030-6"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1002\/acs.754"},{"key":"B51","first-page":"2510","volume-title":"Advances in neural information processing systems","volume":"27","author":"Su W.","year":"2014"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000637"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008811814446"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1007\/s00422-004-0527-x"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2005.04.012"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.cellbio.21.012704.131001"},{"key":"B57","author":"Wibisono A.","year":"2015","journal-title":"On accelerated methods in optimization"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1614734113"},{"key":"B60","author":"Zhang C.","year":"2017","journal-title":"Understanding deep learning requires rethinking generalization"},{"key":"B61","first-page":"3904","volume-title":"Advances in neural information processing systems, 31","author":"Zhang J.","year":"2018"},{"key":"B62","first-page":"685","volume":"28","author":"Zhang S.","year":"2015","journal-title":"Advances in neural information processing systems"},{"key":"B63","doi-asserted-by":"publisher","DOI":"10.1080\/00268976.2018.1483535"},{"key":"B64","author":"Zhu Z.","year":"2018","journal-title":"The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01248","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:43:27Z","timestamp":1615585407000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/32\/1\/36-96\/95571"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1]]},"references-count":63,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,1]]}},"alternative-id":["10.1162\/neco_a_01248"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01248","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1]]}}}