{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T15:15:52Z","timestamp":1783610152921,"version":"3.55.0"},"reference-count":134,"publisher":"Emerald","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,6,30]]},"abstract":"<jats:p>Scalable and efficient distributed learning is one of the main driving forces behind the recent rapid advancement of machine learning and artificial intelligence. One prominent feature of this topic is that recent progress has been made by researchers in two communities: (1) the system community such as database, data management, and distributed systems, and (2) the machine learning and mathematical optimization community. The interaction and knowledge sharing between these two communities has led to the rapid development of new distributed learning systems and theory.<\/jats:p>\n                  <jats:p>In this monograph, we hope to provide a brief introduction of some distributed learning techniques that have recently been developed, namely lossy communication compression (e.g., quantization and sparsification), asynchronous communication, and decentralized communication. One special focus in this monograph is on making sure that it can be easily understood by researchers in both communities \u2014 on the system side, we rely on a simplified system model hiding many system details that are not necessary for the intuition behind the system speedups; while, on the theory side, we rely on minimal assumptions and significantly simplify the proof of some recent work to achieve comparable results.<\/jats:p>","DOI":"10.1561\/1900000062","type":"journal-article","created":{"date-parts":[[2020,6,11]],"date-time":"2020-06-11T08:56:51Z","timestamp":1591865811000},"page":"1-100","source":"Crossref","is-referenced-by-count":21,"title":["Distributed Learning Systems with First-Order Methods"],"prefix":"10.1108","volume":"9","author":[{"given":"Ji","family":"Liu","sequence":"first","affiliation":[{"name":"University of Rochester and Kuaishou Inc. ,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ce","family":"Zhang","sequence":"additional","affiliation":[{"name":"ETH Zurich ,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2020,6,30]]},"reference":[{"key":"2025121112213878800_ref001","article-title":"Distributed learning with sublinear communication","volume-title":"International Conference on Machine Learning","author":"Acharya","year":"2019"},{"key":"2025121112213878800_ref002","doi-asserted-by":"crossref","DOI":"10.1109\/CDC.2012.6426626","article-title":"Distributed delayed stochastic optimization","volume-title":"IEEE Conference on Decision and Control","author":"Agarwal","year":"2012"},{"key":"2025121112213878800_ref003","first-page":"7575","article-title":"cpSGD: Communication-efficient and differentially-private distributed SGD","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Agarwal","year":"2018"},{"key":"2025121112213878800_ref004","first-page":"4618","article-title":"Byzantine stochastic gradient descent","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Alistarh","year":"2018"},{"key":"2025121112213878800_ref005","first-page":"1707","article-title":"QSGD: Communication-efficient SGD via gradient quantization and encoding","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Alistarh","year":"2017"},{"key":"2025121112213878800_ref006","first-page":"5977","article-title":"The convergence of sparsified gradient methods","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Alistarh","year":"2018"},{"key":"2025121112213878800_ref007","article-title":"Variance reduction for faster non-convex optimization","volume-title":"International Conference on Machine Learning","author":"Allen-Zhu","year":"2016"},{"key":"2025121112213878800_ref008","article-title":"Improved SVRG for non-strongly-convex or sum-of-non-convex objectives","volume-title":"International Conference on Machine Learning","author":"Allen-Zhu","year":"2016"},{"key":"2025121112213878800_ref009","article-title":"Limitations on variance-reduction and acceleration schemes for finite sums optimization","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Arjevani","year":"2017"},{"key":"2025121112213878800_ref010","article-title":"Stochastic gradient push for distributed deep learning","volume-title":"International Conference on Machine Learning","author":"Assran","year":"2019"},{"key":"2025121112213878800_ref011","article-title":"An asynchronous distributed proximal gradient method for composite convex optimization","volume-title":"International Conference on Machine Learning","author":"Aybat","year":"2015"},{"key":"2025121112213878800_ref012","first-page":"5151","article-title":"Scalable methods for 8-bit training of neural networks","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Banner","year":"2018"},{"key":"2025121112213878800_ref013","article-title":"Demystifying parallel and distributed deep learning: An in-depth concurrency analysis","volume-title":"arXiv: 1802.09941","author":"Ben-Nun","year":"2018"},{"key":"2025121112213878800_ref014","article-title":"signSGD: Compressed optimisation for non-convex problems","volume-title":"International Conference on Machine Learning","author":"Bernstein","year":"2018"},{"key":"2025121112213878800_ref015","article-title":"On biased compression for distributed learning","volume-title":"arXiv:2002.12410","author":"Beznosikov","year":"2020"},{"key":"2025121112213878800_ref016","first-page":"1622","article-title":"Stochastic optimization with variance reduction for infinite datasets with finite sum structure","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Bietti","year":"2017"},{"key":"2025121112213878800_ref017","first-page":"118","article-title":"Machine learning with adversaries: Byzantine tolerant gradient descent","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Blanchard","year":"2017"},{"key":"2025121112213878800_ref018","article-title":"Optimization methods for large-scale machine learning","volume-title":"arXiv:1606.04838","author":"Bottou","year":"2016"},{"key":"2025121112213878800_ref019","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511804441","volume-title":"Convex Optimization","author":"Boyd","year":"2004"},{"key":"2025121112213878800_ref020","first-page":"1531","article-title":"Asynchronous stochastic convex optimization: The noise is in the noise and SGD don\u2019t care","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Chaturapruek","year":"2015"},{"key":"2025121112213878800_ref021","first-page":"2945","article-title":"Stochastic gradient MCMC with stale gradients","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Chen","year":"2016"},{"key":"2025121112213878800_ref022","article-title":"DRACO: Byzantine-resilient distributed training via redundant gradients","volume-title":"International Conference on Machine Learning","author":"Chen","year":"2018"},{"key":"2025121112213878800_ref023","article-title":"Toward efficient and accurate covariance matrix estimation on compressed data","volume-title":"International Conference on Machine Learning","author":"Chen","year":"2017"},{"key":"2025121112213878800_ref024","doi-asserted-by":"crossref","DOI":"10.1147\/JRD.2019.2947013","article-title":"BlueConnect: Decomposing AllReduce for deep learning on heterogeneous network hierarchy","volume-title":"Conference on Systems and Machine Learning","author":"Cho","year":"2019"},{"key":"2025121112213878800_ref025","article-title":"On acceleration with noise-corrupted gradients","volume-title":"International Conference on Machine Learning","author":"Cohen","year":"2018"},{"key":"2025121112213878800_ref026","article-title":"Gossip dual averaging for decentralized optimization of pairwise functions","volume-title":"International Conference on Machine Learning","author":"Colin","year":"2016"},{"key":"2025121112213878800_ref027","first-page":"1914","article-title":"Distributed stochastic optimization via adaptive SGD","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Cutkosky","year":"2018"},{"key":"2025121112213878800_ref028","article-title":"Asynchronous Byzantine machine learning (the case of SGD)","volume-title":"International Conference on Machine Learning","author":"Damaskinos","year":"2018"},{"key":"2025121112213878800_ref029","article-title":"Taming the wild: A unified analysis of Hogwild-style algorithms","volume-title":"Annual Conference on Neural Information Processing Systems","author":"De Sa","year":"2015"},{"key":"2025121112213878800_ref030","first-page":"250","article-title":"Snap ML: A hierarchical framework for machine learning","volume-title":"Annual Conference on Neural Information Processing Systems","author":"D\u00fcnner","year":"2018"},{"issue":"12","key":"2025121112213878800_ref031","doi-asserted-by":"crossref","first-page":"960","DOI":"10.14778\/2994509.2994515","article-title":"Compressed linear algebra for large-scale machine learning","volume":"9","author":"Elgohary","year":"2016","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2025121112213878800_ref032","article-title":"Communication-efficient algorithms for distributed stochastic principal component analysis","volume-title":"International Conference on Machine Learning","author":"Garber","year":"2017"},{"key":"2025121112213878800_ref033","article-title":"Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization","volume-title":"arXiv:1308.6594","author":"Ghadimi","year":"2013"},{"key":"2025121112213878800_ref034","article-title":"Adaptive sampling for SGD by exploiting side information","volume-title":"International Conference on Machine Learning","author":"Gopal","year":"2016"},{"key":"2025121112213878800_ref035","article-title":"Accurate, large minibatch SGD: Training ImageNet in 1 hour","volume-title":"arXiv:1706.02677","author":"Goyal","year":"2017"},{"key":"2025121112213878800_ref036","article-title":"Faster derivative-free stochastic algorithm for shared memory machines","volume-title":"International Conference on Machine Learning","author":"Gu","year":"2018"},{"key":"2025121112213878800_ref037","first-page":"1737","article-title":"Deep learning with limited numerical precision","volume-title":"International Conference on Machine Learning","author":"Gupta","year":"2015"},{"key":"2025121112213878800_ref038","first-page":"2086","article-title":"SEGA: Variance reduction via gradient sketching","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Hanzely","year":"2018"},{"key":"2025121112213878800_ref039","first-page":"2624","article-title":"Random shuffling beats SGD after finite epochs","volume-title":"International Conference on Machine Learning","author":"Haochen","year":"2019"},{"key":"2025121112213878800_ref040","article-title":"TicTac: Accelerating distributed deep learning with communication scheduling","volume-title":"Conference on Systems and Machine Learning","author":"Hashemi","year":"2019"},{"key":"2025121112213878800_ref041","article-title":"Variance-reduced and projection-free stochastic optimization","volume-title":"International Conference on Machine Learning","author":"Hazan","year":"2016"},{"key":"2025121112213878800_ref042","article-title":"Central server free federated learning over single-sided trust social networks","volume-title":"arXiv:1910.04956","author":"He","year":"2019"},{"key":"2025121112213878800_ref043","first-page":"4541","article-title":"COLA: Decentralized linear learning","volume-title":"Annual Conference on Neural Information Processing Systems","author":"He","year":"2018"},{"key":"2025121112213878800_ref044","first-page":"2305","article-title":"Variance reduced stochastic gradient descent with neighbors","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Hofmann","year":"2015"},{"key":"2025121112213878800_ref045","article-title":"Nonconvex variance reduced optimization with arbitrary sampling","volume-title":"International Conference on Machine Learning","author":"Horv\u00e1th","year":"2019"},{"key":"2025121112213878800_ref046","article-title":"PASSCoDe: Parallel ASynchronous stochastic dual co-ordinate descent","volume-title":"International Conference on Machine Learning","author":"Hsieh","year":"2015"},{"key":"2025121112213878800_ref047","article-title":"Gaia: Geo-distributed machine learning approaching LAN speeds","volume-title":"Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI \u201917)","author":"Hsieh","year":"2017"},{"key":"2025121112213878800_ref048","article-title":"Priority-based parameter propagation for distributed DNN training","volume-title":"Conference on Systems and Machine Learning","author":"Jayarajan","year":"2019"},{"key":"2025121112213878800_ref049","article-title":"Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization","volume-title":"International Conference on Machine Learning","author":"Ji","year":"2019"},{"key":"2025121112213878800_ref050","article-title":"Beyond data and model parallelism for deep neural networks","volume-title":"Conference on Systems and Machine Learning","author":"Jia","year":"2019"},{"key":"2025121112213878800_ref051","doi-asserted-by":"crossref","DOI":"10.1145\/3035918.3035933","article-title":"Heterogeneity-aware distributed parameter servers","volume-title":"International Conference on Management of Data","author":"Jiang","year":"2017"},{"key":"2025121112213878800_ref052","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1145\/3183713.3196894","article-title":"SketchML: Accelerating distributed machine learning with data sketches","volume-title":"Proceedings of the 2018 International Conference on Management of Data","author":"Jiang","year":"2018"},{"key":"2025121112213878800_ref053","first-page":"2530","article-title":"A linear speedup analysis of distributed deep learning with sparse and quantized communication","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Jiang","year":"2018"},{"key":"2025121112213878800_ref054","first-page":"7276","article-title":"Training deep models faster with robust, approximate importance sampling","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Johnson","year":"2018"},{"key":"2025121112213878800_ref055","first-page":"9928","article-title":"Variance-reduced stochastic gradient descent on streaming data","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Jothimurugesan","year":"2018"},{"key":"2025121112213878800_ref056","first-page":"6480","article-title":"Effective parallelisation for machine learning","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Kamp","year":"2017"},{"key":"2025121112213878800_ref057","doi-asserted-by":"crossref","DOI":"10.1145\/3035918.3064042","article-title":"A cost-based optimizer for gradient descent optimization","volume-title":"International Conference on Management of Data","author":"Kaoudi","year":"2017"},{"issue":"4","key":"2025121112213878800_ref058","doi-asserted-by":"crossref","first-page":"348","DOI":"10.14778\/3297753.3297756","article-title":"ColumnML: Column-store machine learning with on-the-fly data transformation","volume":"12","author":"Kara","year":"2018","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2025121112213878800_ref059","first-page":"5440","article-title":"Straggler mitigation in distributed optimization through data encoding","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Karakus","year":"2017"},{"key":"2025121112213878800_ref060","article-title":"Not all samples are created equal: Deep learning with importance sampling","volume-title":"International Conference on Machine Learning","author":"Katharopoulos","year":"2018"},{"key":"2025121112213878800_ref061","article-title":"Decentralized stochastic optimization and Gossip algorithms with compressed communication","volume-title":"International Conference on Machine Learning","author":"Koloskova","year":"2019"},{"key":"2025121112213878800_ref062","article-title":"Stochastic variance reduced optimization for nonconvex sparse learning","volume-title":"International Conference on Machine Learning","author":"Li","year":"2016"},{"key":"2025121112213878800_ref063","article-title":"On linear convergence of two decentralized algorithms","volume-title":"arXiv:1906.07225","author":"Li","year":"2019"},{"key":"2025121112213878800_ref064","first-page":"8056","article-title":"Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Li","year":"2018"},{"issue":"17","key":"2025121112213878800_ref065","doi-asserted-by":"crossref","first-page":"4494","DOI":"10.1109\/TSP.2019.2926022","article-title":"A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates","volume":"67","author":"Li","year":"2019","journal-title":"IEEE Transactions on Signal Processing"},{"key":"2025121112213878800_ref066","article-title":"A primal-dual algorithm with optimal stepsizes and its application in decentralized consensus optimization","volume-title":"arXiv:1711.06785","author":"Li","year":"2017"},{"key":"2025121112213878800_ref067","first-page":"2737","article-title":"Asynchronous parallel stochastic gradient for nonconvex optimization","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Lian","year":"2015"},{"key":"2025121112213878800_ref068","first-page":"5336","article-title":"Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Lian","year":"2017"},{"key":"2025121112213878800_ref069","first-page":"3062","article-title":"A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Lian","year":"2016"},{"key":"2025121112213878800_ref070","article-title":"Asynchronous decentralized parallel stochastic gradient descent","volume-title":"International Conference on Machine Learning","author":"Lian","year":"2019"},{"key":"2025121112213878800_ref071","article-title":"3LC: Lightweight and effective traffic compression for distributed machine learning","volume-title":"Conference on Systems and Machine Learning","author":"Lim","year":"2019"},{"issue":"5","key":"2025121112213878800_ref072","doi-asserted-by":"crossref","first-page":"1185","DOI":"10.1109\/TSP.2013.2295055","article-title":"Decentralized dynamic optimization through the alternating direction method of multipliers","volume":"62","author":"Ling","year":"2013","journal-title":"IEEE Transactions on Signal Processing"},{"key":"2025121112213878800_ref073","doi-asserted-by":"crossref","DOI":"10.1137\/140961134","article-title":"Asynchronous stochastic coordinate descent: Parallelism and convergence properties","volume-title":"SIAM on Optimization","author":"Liu","year":"2015"},{"key":"2025121112213878800_ref074","article-title":"An asynchronous parallel stochastic coordinate descent algorithm","volume-title":"International Conference on Machine Learning","author":"Liu","year":"2014"},{"key":"2025121112213878800_ref075","first-page":"3731","article-title":"Zeroth-order stochastic variance reduction for nonconvex optimization","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Liu","year":"2018"},{"key":"2025121112213878800_ref076","article-title":"A double residual compression algorithm for efficient distributed learning","volume-title":"arXiv:1910.07561","author":"Liu","year":"2019"},{"key":"2025121112213878800_ref077","article-title":"Adding vs. averaging in distributed primal-dual optimization","volume-title":"International Conference on Machine Learning","author":"Ma","year":"2015"},{"issue":"11","key":"2025121112213878800_ref078","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.14778\/3236187.3236188","article-title":"In-RDBMS hardware acceleration of advanced analytics","volume":"11","author":"Mahajan","year":"2018","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2025121112213878800_ref079","article-title":"SGD without replacement: Sharper rates for general smooth convex functions","volume-title":"International Conference on Machine Learning","author":"Nagaraj","year":"2019"},{"key":"2025121112213878800_ref080","article-title":"Adaptive sampling probabilities for non-smooth optimization","volume-title":"International Conference on Machine Learning","author":"Namkoong","year":"2017"},{"issue":"1","key":"2025121112213878800_ref081","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/TAC.2008.2009515","article-title":"Distributed subgradient methods for multi-agent optimization","volume":"54","author":"Nedi\u0107","year":"2009","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2025121112213878800_ref082","article-title":"SGD and Hogwild! Convergence without the bounded gradients assumption","volume-title":"International Conference on Machine Learning","author":"Nguyen","year":"2018"},{"key":"2025121112213878800_ref083","first-page":"1416","article-title":"Stochastic variance reduction methods for saddle-point problems","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Palaniappan","year":"2016"},{"key":"2025121112213878800_ref084","first-page":"2576","article-title":"Cyclades: Conflict-free asynchronous machine learning","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Pan","year":"2016"},{"key":"2025121112213878800_ref085","article-title":"Asynchronous distributed variational Gaussian process for regression","volume-title":"International Conference on Machine Learning","author":"Peng","year":"2017"},{"key":"2025121112213878800_ref086","first-page":"865","article-title":"Quartz: Randomized dual coordinate ascent with arbitrary sampling","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Qu","year":"2015"},{"key":"2025121112213878800_ref087","article-title":"Gradient coding from cyclic MDS codes and expander graphs","volume-title":"International Conference on Machine Learning","author":"Raviv","year":"2018"},{"key":"2025121112213878800_ref088","article-title":"Stochastic variance reduction for nonconvex optimization","volume-title":"International Conference on Machine Learning","author":"Reddi","year":"2016"},{"key":"2025121112213878800_ref089","first-page":"46","article-title":"Without-replacement sampling for stochastic gradient methods","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Shamir","year":"2016"},{"issue":"22","key":"2025121112213878800_ref090","doi-asserted-by":"crossref","first-page":"6013","DOI":"10.1109\/TSP.2015.2461520","article-title":"A proximal gradient algorithm for decentralized composite optimization","volume":"63","author":"Shi","year":"2015","journal-title":"IEEE Transactions on Signal Processing"},{"issue":"2","key":"2025121112213878800_ref091","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1137\/14096668X","article-title":"Extra: An exact firstorder algorithm for decentralized consensus optimization","volume":"25","author":"Shi","year":"2015","journal-title":"SIAM Journal on Optimization"},{"issue":"11","key":"2025121112213878800_ref092","doi-asserted-by":"crossref","first-page":"5724","DOI":"10.1109\/TAC.2017.2694611","article-title":"Decentralized prediction-correction methods for networked timevarying convex optimization","volume":"62","author":"Simonetto","year":"2017","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2025121112213878800_ref093","article-title":"Asynchronous stochastic quasi-Newton MCMC for non-convex optimization","volume-title":"International Conference on Machine Learning","author":"Simsekli","year":"2018"},{"key":"2025121112213878800_ref094","first-page":"2895","article-title":"An approximate efficient solver for LP rounding","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Sridhar","year":"2013"},{"key":"2025121112213878800_ref095","first-page":"4452","article-title":"Sparsified SGD with memory","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Stich","year":"2018"},{"key":"2025121112213878800_ref096","article-title":"Asynchronous coordinate descent under more realistic assumptions","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Sun","year":"2017"},{"key":"2025121112213878800_ref097","article-title":"Gradient coding: Avoiding stragglers in distributed learning","volume-title":"International Conference on Machine Learning","author":"Tandon","year":"2017"},{"key":"2025121112213878800_ref098","first-page":"7663","article-title":"Communication compression for decentralized training","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Tang","year":"2018"},{"key":"2025121112213878800_ref099","article-title":"DeepSqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression","volume-title":"arXiv:1907.07346","author":"Tang","year":"2019"},{"key":"2025121112213878800_ref100","article-title":"D2: Decentralized training over decentralized data","volume-title":"International Conference on Machine Learning","author":"Tang","year":"2018"},{"key":"2025121112213878800_ref101","article-title":"Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression","volume-title":"International Conference on Machine Learning","author":"Tang","year":"2019"},{"key":"2025121112213878800_ref102","first-page":"9872","article-title":"ATOMO: Communication-efficient learning via atomic sparsification","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wang","year":"2018"},{"key":"2025121112213878800_ref103","first-page":"9850","article-title":"Atomo: Communication-efficient learning via atomic sparsification","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wang","year":"2018"},{"key":"2025121112213878800_ref104","article-title":"Efficient distributed learning with sparsity","volume-title":"International Conference on Machine Learning","author":"Wang","year":"2017"},{"key":"2025121112213878800_ref105","article-title":"Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD","volume-title":"Conference on Systems and Machine Learning","author":"Wang","year":"2019"},{"key":"2025121112213878800_ref106","first-page":"7686","article-title":"Training deep neural networks with 8-bit floating point numbers","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wang","year":"2018"},{"key":"2025121112213878800_ref107","first-page":"4243","article-title":"BML: A high-performance, low-cost gradient synchronization algorithm for DML training","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wang","year":"2018"},{"issue":"7","key":"2025121112213878800_ref108","doi-asserted-by":"crossref","first-page":"807","DOI":"10.14778\/3317315.3317322","article-title":"Accelerating generalized linear models with MLWeaving: A one-size-fits-all system for any-precision learning","volume":"12","author":"Wang","year":"2019","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2025121112213878800_ref109","first-page":"1306","article-title":"Gradient sparsification for communication-efficient distributed optimization","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wangni","year":"2018"},{"key":"2025121112213878800_ref110","first-page":"1508","article-title":"TernGrad: Ternary gradients to reduce communication in distributed deep learning","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Wen","year":"2017"},{"key":"2025121112213878800_ref111","article-title":"Error compensated quantized SGD and its applications to large-scale distributed optimization","volume-title":"International Conference on Machine Learning","author":"Wu","year":"2018"},{"key":"2025121112213878800_ref112","article-title":"Zeno: Distributed stochastic gradient descent with suspicion-based fault-tolerance","volume-title":"International Conference on Machine Learning","author":"Xie","year":"2019"},{"issue":"4","key":"2025121112213878800_ref113","doi-asserted-by":"crossref","first-page":"446","DOI":"10.14778\/3297753.3297763","article-title":"HELIX: Holistic optimization for accelerating iterative machine learning","volume":"12","author":"Xin","year":"2018","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2025121112213878800_ref114","article-title":"Communication-computation efficient gradient coding","volume-title":"International Conference on Machine Learning","author":"Ye","year":"2018"},{"key":"2025121112213878800_ref115","article-title":"Byzantine-robust distributed learning: Towards optimal statistical rates","volume-title":"International Conference on Machine Learning","author":"Yin","year":"2018"},{"key":"2025121112213878800_ref116","article-title":"Reducing BERT pre-training time from 3 days to 76 minutes","volume-title":"arXiv:1904.00962","author":"You","year":"2020"},{"key":"2025121112213878800_ref117","first-page":"4689","article-title":"Asynchronous parallel greedy coordinate descent","volume-title":"Annual Conference on Neural Information Processing Systems","author":"You","year":"2016"},{"key":"2025121112213878800_ref118","article-title":"Distributed learning over unreliable networks","volume-title":"International Conference on Machine Learning","author":"Yu","year":"2019"},{"key":"2025121112213878800_ref119","article-title":"GradiVeQ: Vector quantization for bandwidth-efficient gradient aggregation in distributed CNN training","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Yu","year":"2018"},{"issue":"3","key":"2025121112213878800_ref120","doi-asserted-by":"crossref","first-page":"1835","DOI":"10.1137\/130943170","article-title":"On the convergence of decentralized gradient descent","volume":"26","author":"Yuan","year":"2016","journal-title":"SIAM Journal on Optimization"},{"key":"2025121112213878800_ref121","article-title":"Deep learning in mobile and wireless networking: A survey","volume-title":"arXiv:1803.04311","author":"Zhang","year":"2018"},{"key":"2025121112213878800_ref122","article-title":"ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning","volume-title":"International Conference on Machine Learning","author":"Zhang","year":"2017"},{"key":"2025121112213878800_ref123","article-title":"Online stochastic linear optimization under one-bit feedback","volume-title":"International Conference on Machine Learning","author":"Zhang","year":"2016"},{"key":"2025121112213878800_ref124","first-page":"685","article-title":"Deep learning with elastic averaging SGD","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Zhang","year":"2015"},{"key":"2025121112213878800_ref125","article-title":"Stochastic optimization with importance sampling for regularized loss minimization","volume-title":"International Conference on Machine Learning","author":"Zhao","year":"2015"},{"key":"2025121112213878800_ref126","article-title":"Asynchronous stochastic gradient descent with delay compensation","volume-title":"International Conference on Machine Learning","author":"Zheng","year":"2017"},{"key":"2025121112213878800_ref127","article-title":"Stochastic nested variance reduced gradient descent for nonconvex optimization","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Zhou","year":"2018"},{"key":"2025121112213878800_ref128","article-title":"Stochastic variance-reduced cubic regularized Newton methods","volume-title":"International Conference on Machine Learning","author":"Zhou","year":"2018"},{"key":"2025121112213878800_ref129","article-title":"A simple stochastic variance reduced algorithm with fast convergence rates","volume-title":"International Conference on Machine Learning","author":"Zhou","year":"2018"},{"key":"2025121112213878800_ref130","article-title":"Distributed asynchronous optimization with unbounded delays: How slow can you go?","volume-title":"International Conference on Machine Learning","author":"Zhou","year":"2018"},{"key":"2025121112213878800_ref131","first-page":"406","article-title":"Gradient-based sampling: An adaptive importance sampling for least-squares","volume-title":"Annual Conference on Neural Information Processing Systems","author":"Zhu","year":"2016"},{"key":"2025121112213878800_ref132","article-title":"Towards a lower sample complexity for robust one-bit compressed sensing","volume-title":"International Conference on Machine Learning","author":"Zhu","year":"2015"},{"key":"2025121112213878800_ref133","article-title":"Distributed nonparametric regression under communication constraints","volume-title":"International Conference on Machine Learning","author":"Zhu","year":"2018"},{"key":"2025121112213878800_ref134","article-title":"Stochastic variance-reduced Hamilton Monte Carlo methods","volume-title":"International Conference on Machine Learning","author":"Zou","year":"2018"}],"container-title":["Foundations and Trends in Databases"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/ftdbs\/article-pdf\/9\/1\/1\/11024231\/1900000062en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ftdbs\/article-pdf\/9\/1\/1\/11024231\/1900000062en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T14:11:44Z","timestamp":1777471904000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ftdbs\/article\/9\/1\/1\/1326471\/Distributed-Learning-Systems-with-First-Order"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,30]]},"references-count":134,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,6,30]]}},"URL":"https:\/\/doi.org\/10.1561\/1900000062","relation":{},"ISSN":["1931-7883","1931-7891"],"issn-type":[{"value":"1931-7883","type":"print"},{"value":"1931-7891","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,30]]}}}