{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T02:57:17Z","timestamp":1792205837548,"version":"4.4.0"},"publisher-location":"New York, NY, USA","reference-count":72,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,9]],"date-time":"2021-08-09T00:00:00Z","timestamp":1628467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004052","name":"King Abdullah University of Science and Technology","doi-asserted-by":"publisher","award":["OSR-CRG2020-4382"],"award-info":[{"award-number":["OSR-CRG2020-4382"]}],"id":[{"id":"10.13039\/501100004052","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,9]]},"DOI":"10.1145\/3452296.3472904","type":"proceedings-article","created":{"date-parts":[[2021,8,9]],"date-time":"2021-08-09T18:13:15Z","timestamp":1628532795000},"page":"676-691","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":121,"title":["Efficient sparse collective communication and its application to accelerate distributed deep learning"],"prefix":"10.1145","author":[{"given":"Jiawei","family":"Fei","sequence":"first","affiliation":[{"name":"NUDT and KAUST"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chen-Yu","family":"Ho","sequence":"additional","affiliation":[{"name":"KAUST"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atal N.","family":"Sahu","sequence":"additional","affiliation":[{"name":"KAUST"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Canini","sequence":"additional","affiliation":[{"name":"KAUST"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amedeo","family":"Sapio","sequence":"additional","affiliation":[{"name":"Intel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,8,9]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Alham Fikri Aji and Kenneth Heafield. 2017. Sparse Communication for Distributed Gradient Descent. In EMNLP-IJCNLP.  Alham Fikri Aji and Kenneth Heafield. 2017. Sparse Communication for Distributed Gradient Descent. In EMNLP-IJCNLP ."},{"key":"e_1_3_2_2_2_1","volume-title":"QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In NeurIPS.","author":"Alistarh Dan","year":"2017","unstructured":"Dan Alistarh , Demjan Grubic , Jerry Li , Ryota Tomioka , and Milan Vojnovic . 2017 . QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In NeurIPS. Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In NeurIPS."},{"key":"e_1_3_2_2_3_1","unstructured":"Dan Alistarh Torsten Hoefler Mikael Johansson Nikola Konstantinov Sarit Khirirat and C\u00e9dric Renggli. 2018. The Convergence of Sparsified Gradient Methods. In NeurIPS.  Dan Alistarh Torsten Hoefler Mikael Johansson Nikola Konstantinov Sarit Khirirat and C\u00e9dric Renggli. 2018. The Convergence of Sparsified Gradient Methods. In NeurIPS ."},{"key":"e_1_3_2_2_4_1","volume-title":"Can Karakus, and Suhas Diggavi.","author":"Basu Debraj","year":"2019","unstructured":"Debraj Basu , Deepesh Data , Can Karakus, and Suhas Diggavi. 2019 . Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations. In NeurIPS. Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. 2019. Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations. In NeurIPS."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2656877.2656890"},{"key":"e_1_3_2_2_6_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. [arxiv]2005.14165 [cs.CL]  Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. [arxiv]2005.14165 [cs.CL]"},{"key":"e_1_3_2_2_7_1","unstructured":"bytedance\/byteps. 2019. A High Performance and Generic Framework for Distributed DNN Training. https:\/\/github.com\/bytedance\/byteps.  bytedance\/byteps. 2019. A High Performance and Generic Framework for Distributed DNN Training. https:\/\/github.com\/bytedance\/byteps."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"crossref","unstructured":"Ciprian Chelba Tomas Mikolov Mike Schuster Qi Ge Thorsten Brants Phillipp Koehn and Tony Robinson. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. [arxiv]1312.3005 [cs.CL]  Ciprian Chelba Tomas Mikolov Mike Schuster Qi Ge Thorsten Brants Phillipp Koehn and Tony Robinson. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. [arxiv]1312.3005 [cs.CL]","DOI":"10.21437\/Interspeech.2014-564"},{"key":"e_1_3_2_2_9_1","volume-title":"Project Adam: Building an Efficient and Scalable Deep Learning Training System. In OSDI.","author":"Chilimbi Trishul","year":"2014","unstructured":"Trishul Chilimbi , Yutaka Suzue , Johnson Apacible , and Karthik Kalyanaraman . 2014 . Project Adam: Building an Efficient and Scalable Deep Learning Training System. In OSDI. Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In OSDI."},{"key":"e_1_3_2_2_10_1","volume-title":"Hunter","author":"Cho Minsik","year":"2019","unstructured":"Minsik Cho , Ulrich Finkler , David S. Kung , and Hillery C . Hunter . 2019 . BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy. In MLSys . Minsik Cho, Ulrich Finkler, David S. Kung, and Hillery C. Hunter. 2019. BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy. In MLSys."},{"key":"e_1_3_2_2_11_1","unstructured":"Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Marc'aurelio Ranzato Andrew Senior Paul Tucker Ke Yang Quoc Le and Andrew Ng. 2012. Large Scale Distributed Deep Networks. In NeurIPS.  Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Marc'aurelio Ranzato Andrew Senior Paul Tucker Ke Yang Quoc Le and Andrew Ng. 2012. Large Scale Distributed Deep Networks. In NeurIPS ."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"crossref","unstructured":"Wei Deng Junwei Pan Tian Zhou Deguang Kong Aaron Flores and Guang Lin. 2020. DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving. [arxiv]2002.06987 [cs.LG]  Wei Deng Junwei Pan Tian Zhou Deguang Kong Aaron Flores and Guang Lin. 2020. DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving. [arxiv]2002.06987 [cs.LG]","DOI":"10.1145\/3437963.3441727"},{"key":"e_1_3_2_2_13_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. [arxiv]1810.04805 [cs.CL]","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. [arxiv]1810.04805 [cs.CL] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. [arxiv]1810.04805 [cs.CL]"},{"key":"e_1_3_2_2_14_1","volume-title":"Tim Moon, and Brian Van Essen.","author":"Dryden Nikoli","year":"2016","unstructured":"Nikoli Dryden , Sam Ade Jacobs , Tim Moon, and Brian Van Essen. 2016 . Communication Quantization for Data-Parallel Training of Deep Neural Networks. In MLHPC. Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016. Communication Quantization for Data-Parallel Training of Deep Neural Networks. In MLHPC."},{"key":"e_1_3_2_2_15_1","volume-title":"Ahmed M. Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis.","author":"Dutta Aritra","year":"2020","unstructured":"Aritra Dutta , El Houcine Bergou , Ahmed M. Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis. 2020 . On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning. In AAAI. Aritra Dutta, El Houcine Bergou, Ahmed M. Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis. 2020. On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning. In AAAI."},{"key":"e_1_3_2_2_16_1","unstructured":"Facebook. 2021. Gloo. https:\/\/github.com\/facebookincubator\/gloo.  Facebook. 2021. Gloo. https:\/\/github.com\/facebookincubator\/gloo."},{"key":"e_1_3_2_2_17_1","volume-title":"MPI: A Message-Passing Interface Standard. Technical Report","author":"Interface Forum Message Passing","year":"1994","unstructured":"Message Passing Interface Forum . 1994 . MPI: A Message-Passing Interface Standard. Technical Report . University of Tennessee . Message Passing Interface Forum. 1994. MPI: A Message-Passing Interface Standard. Technical Report. University of Tennessee."},{"key":"e_1_3_2_2_18_1","unstructured":"Nadeen Gebara Paolo Costa and Manya Ghobadi. 2021. In-network Aggregation for Shared Machine Learning Clusters. In MLSys.  Nadeen Gebara Paolo Costa and Manya Ghobadi. 2021. In-network Aggregation for Shared Machine Learning Clusters. In MLSys ."},{"key":"e_1_3_2_2_19_1","first-page":"4","article-title":"The MovieLens Datasets","volume":"5","author":"Maxwell Harper F","year":"2015","unstructured":"F Maxwell Harper and Joseph A Konstan . 2015 . The MovieLens Datasets : History and Context. ACM Trans. Interact. Intell. Syst. 5 , 4 (Dec. 2015). F Maxwell Harper and Joseph A Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4 (Dec. 2015).","journal-title":"History and Context. ACM Trans. Interact. Intell. Syst."},{"key":"e_1_3_2_2_20_1","volume-title":"Sangeetha Abdu Jyothi, and Roy Campbell","author":"Hashemi Sayed Hadi","year":"2019","unstructured":"Sayed Hadi Hashemi , Sangeetha Abdu Jyothi, and Roy Campbell . 2019 . TicTac: Accelerating Distributed Deep Learning with Communication Scheduling. In MLSys . Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, and Roy Campbell. 2019. TicTac: Accelerating Distributed Deep Learning with Communication Scheduling. In MLSys."},{"key":"e_1_3_2_2_21_1","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR.  Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR ."},{"key":"e_1_3_2_2_22_1","unstructured":"Xiangnan He Lizi Liao Hanwang Zhang Liqiang Nie Xia Hu and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW.  Xiangnan He Lizi Liao Hanwang Zhang Liqiang Nie Xia Hu and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW ."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"crossref","unstructured":"Michael Hofmann and Gudula R\u00fcnger. 2008. MPI Reduction Operations for Sparse Floating-point Data. In EuroPVM\/MPI.  Michael Hofmann and Gudula R\u00fcnger. 2008. MPI Reduction Operations for Sparse Floating-point Data. In EuroPVM\/MPI .","DOI":"10.1007\/978-3-540-87475-1_17"},{"key":"e_1_3_2_2_24_1","unstructured":"Yanping Huang Youlong Cheng Ankur Bapna Orhan Firat Dehao Chen Mia Chen HyoukJoong Lee Jiquan Ngiam Quoc V Le Yonghui Wu and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS.  Yanping Huang Youlong Cheng Ankur Bapna Orhan Firat Dehao Chen Mia Chen HyoukJoong Lee Jiquan Ngiam Quoc V Le Yonghui Wu and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS ."},{"key":"e_1_3_2_2_25_1","unstructured":"Anand Jayarajan Jinliang Wei Garth Gibson Alexandra Fedorova and Gennady Pekhimenko. 2019. Priority-based Parameter Propagation for Distributed DNN Training. In MLSys.  Anand Jayarajan Jinliang Wei Garth Gibson Alexandra Fedorova and Gennady Pekhimenko. 2019. Priority-based Parameter Propagation for Distributed DNN Training. In MLSys ."},{"key":"e_1_3_2_2_26_1","unstructured":"Zhihao Jia Sina Lin Charles R. Qi and Alex Aiken. 2018. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks. In ICML.  Zhihao Jia Sina Lin Charles R. Qi and Alex Aiken. 2018. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks. In ICML ."},{"key":"e_1_3_2_2_27_1","unstructured":"Zhihao Jia Matei Zaharia and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In MLSys.  Zhihao Jia Matei Zaharia and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In MLSys ."},{"key":"e_1_3_2_2_28_1","unstructured":"Yimin Jiang Yibo Zhu Chang Lan Bairen Yi Yong Cui and Chuanxiong Guo. 2020. A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU\/CPU Clusters. In OSDI.  Yimin Jiang Yibo Zhu Chang Lan Bairen Yi Yong Cui and Chuanxiong Guo. 2020. A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU\/CPU Clusters. In OSDI ."},{"key":"e_1_3_2_2_29_1","unstructured":"Rafal Jozefowicz Oriol Vinyals Mike Schuster Noam Shazeer and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. [arxiv]1602.02410 [cs.CL]  Rafal Jozefowicz Oriol Vinyals Mike Schuster Noam Shazeer and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. [arxiv]1602.02410 [cs.CL]"},{"key":"e_1_3_2_2_30_1","unstructured":"Sai Praneeth Karimireddy Quentin Rebjock Sebastian Stich and Martin Jaggi. 2019. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In ICML.  Sai Praneeth Karimireddy Quentin Rebjock Sebastian Stich and Martin Jaggi. 2019. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In ICML ."},{"key":"e_1_3_2_2_31_1","unstructured":"Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy and Ping Tak Peter Tang. 2017. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR.  Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy and Ping Tak Peter Tang. 2017. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR ."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"crossref","unstructured":"Janis Keuper and Franz-Josef Pfreundt. 2016. Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability. In MLHPC.  Janis Keuper and Franz-Josef Pfreundt. 2016. Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability. In MLHPC .","DOI":"10.1109\/MLHPC.2016.006"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729392"},{"key":"e_1_3_2_2_34_1","volume-title":"Joo Seong Jeong, and Byung-Gon Chun","author":"Kim Soojeong","year":"2019","unstructured":"Soojeong Kim , Gyeong-In Yu , Hojin Park , Sungwoo Cho , Eunji Jeong , Hyeonmin Ha , Sanha Lee , Joo Seong Jeong, and Byung-Gon Chun . 2019 . Parallax : Sparsity-aware Data Parallel Training of Deep Neural Networks. In EuroSys . Soojeong Kim, Gyeong-In Yu, Hojin Park, Sungwoo Cho, Eunji Jeong, Hyeonmin Ha, Sanha Lee, Joo Seong Jeong, and Byung-Gon Chun. 2019. Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks. In EuroSys."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Benjamin Klenk Nan Jiang Greg Thorson and Larry Dennison. 2020. An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives. In ISCA.  Benjamin Klenk Nan Jiang Greg Thorson and Larry Dennison. 2020. An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives. In ISCA .","DOI":"10.1109\/ISCA45697.2020.00085"},{"key":"e_1_3_2_2_36_1","unstructured":"Anastasia Koloskova Sebastian U Stich and Martin Jaggi. 2019. Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication. In ICML.  Anastasia Koloskova Sebastian U Stich and Martin Jaggi. 2019. Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication. In ICML ."},{"key":"e_1_3_2_2_37_1","unstructured":"Kelly Kostopoulou Hang Xu Aritra Dutta Xin Li Alexandros Ntoulas and Panos Kalnis. 2021. DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning. [arxiv]2102.03112 [cs.LG]  Kelly Kostopoulou Hang Xu Aritra Dutta Xin Li Alexandros Ntoulas and Panos Kalnis. 2021. DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning. [arxiv]2102.03112 [cs.LG]"},{"key":"e_1_3_2_2_38_1","volume-title":"ATP: In-network Aggregation for Multi-tenant Learning. In NSDI.","author":"Lao ChonLam","year":"2021","unstructured":"ChonLam Lao , Yanfang Le , Kshiteej Mahajan , Yixi Chen , Wenfei Wu , Aditya Akella , and Michael Swift . 2021 . ATP: In-network Aggregation for Multi-tenant Learning. In NSDI. ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. 2021. ATP: In-network Aggregation for Multi-tenant Learning. In NSDI."},{"key":"e_1_3_2_2_39_1","unstructured":"Youjie Li Iou-Jen Liu Yifan Yuan Deming Chen Alexander Schwing and Jian Huang. 2019. Accelerating Distributed Reinforcement Learning with In-Switch Computing. In ISCA.  Youjie Li Iou-Jen Liu Yifan Yuan Deming Chen Alexander Schwing and Jian Huang. 2019. Accelerating Distributed Reinforcement Learning with In-Switch Computing. In ISCA ."},{"key":"e_1_3_2_2_40_1","unstructured":"Youjie Li Jongse Park Mohammad Alian Yifan Yuan Zheng Qu Peitian Pan Ren Wang Alexander Schwing Hadi Esmaeilzadeh and Nam Sung Kim. 2018. A Network-Centric Hardware\/Algorithm Co-Design to Accelerate Distributed Training of Deep Neural Networks. In Micro.  Youjie Li Jongse Park Mohammad Alian Yifan Yuan Zheng Qu Peitian Pan Ren Wang Alexander Schwing Hadi Esmaeilzadeh and Nam Sung Kim. 2018. A Network-Centric Hardware\/Algorithm Co-Design to Accelerate Distributed Training of Deep Neural Networks. In Micro ."},{"key":"e_1_3_2_2_41_1","volume-title":"Kumar Kshitij Patel, and Martin Jaggi","author":"Lin Tao","year":"2019","unstructured":"Tao Lin , Sebastian U Stich , Kumar Kshitij Patel, and Martin Jaggi . 2019 . Don't Use Large Mini-batches, Use Local SGD. In ICLR. Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. 2019. Don't Use Large Mini-batches, Use Local SGD. In ICLR."},{"key":"e_1_3_2_2_42_1","unstructured":"Yujun Lin Song Han Huizi Mao Yu Wang and William Dally. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. In ICLR.  Yujun Lin Song Han Huizi Mao Yu Wang and William Dally. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. In ICLR ."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Liang Luo Jacob Nelson Luis Ceze Amar Phanishayee and Arvind Krishnamurthy. 2018. PHub: Rack-Scale Parameter Server for Distributed Deep Neural Network Training. In SoCC.  Liang Luo Jacob Nelson Luis Ceze Amar Phanishayee and Arvind Krishnamurthy. 2018. PHub: Rack-Scale Parameter Server for Distributed Deep Neural Network Training. In SoCC .","DOI":"10.1145\/3267809.3267840"},{"key":"e_1_3_2_2_44_1","unstructured":"Mellanox. 2021. Scalable Hierarchical Aggregation and Reduction Protocol (SHARP). https:\/\/www.mellanox.com\/products\/sharp.  Mellanox. 2021. Scalable Hierarchical Aggregation and Reduction Protocol (SHARP). https:\/\/www.mellanox.com\/products\/sharp."},{"key":"e_1_3_2_2_45_1","unstructured":"Microsoft. 2015. Criteo's 1TB Click Prediction Dataset. https:\/\/docs.microsoft.com\/en-us\/archive\/blogs\/machinelearning\/now-available-on-azure-ml-criteos-1tb-click-prediction-dataset.  Microsoft. 2015. Criteo's 1TB Click Prediction Dataset. https:\/\/docs.microsoft.com\/en-us\/archive\/blogs\/machinelearning\/now-available-on-azure-ml-criteos-1tb-click-prediction-dataset."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"crossref","unstructured":"Deepak Narayanan Aaron Harlap Amar Phanishayee Vivek Seshadri Nikhil R Devanur Gregory R Ganger Phillip B Gibbons and Matei Zaharia. 2019. PipeDream: Generalized Pipeline Parallelism for DNN Training. In SOSP.  Deepak Narayanan Aaron Harlap Amar Phanishayee Vivek Seshadri Nikhil R Devanur Gregory R Ganger Phillip B Gibbons and Matei Zaharia. 2019. PipeDream: Generalized Pipeline Parallelism for DNN Training. In SOSP .","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_3_2_2_47_1","unstructured":"NVIDIA. 2021a. Ampere Architecture In-Depth. https:\/\/devblogs.nvidia.com\/nvidia-ampere-architecture-in-depth\/.  NVIDIA. 2021a. Ampere Architecture In-Depth. https:\/\/devblogs.nvidia.com\/nvidia-ampere-architecture-in-depth\/."},{"key":"e_1_3_2_2_48_1","unstructured":"NVIDIA. 2021b. Collective Communication Library (NCCL). https:\/\/developer.nvidia.com\/nccl.  NVIDIA. 2021b. Collective Communication Library (NCCL). https:\/\/developer.nvidia.com\/nccl."},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2008.09.002"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"crossref","unstructured":"Yanghua Peng Yibo Zhu Yangrui Chen Yixin Bao Bairen Yi Chang Lan Chuan Wu and Chuanxiong Guo. 2019. A Generic Communication Scheduler for Distributed DNN Training Acceleration. In SOSP.  Yanghua Peng Yibo Zhu Yangrui Chen Yixin Bao Bairen Yi Chang Lan Chuan Wu and Chuanxiong Guo. 2019. A Generic Communication Scheduler for Distributed DNN Training Acceleration. In SOSP .","DOI":"10.1145\/3341301.3359642"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"crossref","unstructured":"Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. [arxiv]1802.05365 [cs.CL]  Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. [arxiv]1802.05365 [cs.CL]","DOI":"10.18653\/v1\/N18-1202"},{"key":"e_1_3_2_2_52_1","unstructured":"pytorch\/pytorch. 2019. Support sparse gradients in DistributedDataParallel. https:\/\/github.com\/pytorch\/pytorch\/pull\/22037.  pytorch\/pytorch. 2019. Support sparse gradients in DistributedDataParallel. https:\/\/github.com\/pytorch\/pytorch\/pull\/22037."},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Pranav Rajpurkar Robin Jia and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. [arxiv]1806.03822 [cs.CL]  Pranav Rajpurkar Robin Jia and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. [arxiv]1806.03822 [cs.CL]","DOI":"10.18653\/v1\/P18-2124"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"crossref","unstructured":"Cedric Renggli. 2019. SparCML. https:\/\/gitlab.com\/rengglic\/SparCML.  Cedric Renggli. 2019. SparCML. https:\/\/gitlab.com\/rengglic\/SparCML.","DOI":"10.1145\/3295500.3356222"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"crossref","unstructured":"Cedric Renggli Saleh Ashkboos Mehdi Aghagolzadeh Dan Alistarh and Torsten Hoefler. 2019. SparCML: High-Performance Sparse Communication for Machine Learning. In SC.  Cedric Renggli Saleh Ashkboos Mehdi Aghagolzadeh Dan Alistarh and Torsten Hoefler. 2019. SparCML: High-Performance Sparse Communication for Machine Learning. In SC .","DOI":"10.1145\/3295500.3356222"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"crossref","unstructured":"Amedeo Sapio Ibrahim Abdelaziz Abdulla Aldilaijan Marco Canini and Panos Kalnis. 2017. In-Network Computation is a Dumb Idea Whose Time Has Come. In HotNets.  Amedeo Sapio Ibrahim Abdelaziz Abdulla Aldilaijan Marco Canini and Panos Kalnis. 2017. In-Network Computation is a Dumb Idea Whose Time Has Come. In HotNets .","DOI":"10.1145\/3152434.3152461"},{"key":"e_1_3_2_2_58_1","unstructured":"Amedeo Sapio Marco Canini Chen-Yu Ho Jacob Nelson Panos Kalnis Changhoon Kim Arvind Krishnamurthy Masoud Moshref Dan R. K. Ports and Peter Richt\u00e1rik. 2021. Scaling Distributed Machine Learning with In-Network Aggregation. In NSDI.  Amedeo Sapio Marco Canini Chen-Yu Ho Jacob Nelson Panos Kalnis Changhoon Kim Arvind Krishnamurthy Masoud Moshref Dan R. K. Ports and Peter Richt\u00e1rik. 2021. Scaling Distributed Machine Learning with In-Network Aggregation. In NSDI ."},{"key":"e_1_3_2_2_59_1","volume-title":"Saleh Ashkboos, Shigang Li, and Torsten Hoefler.","author":"Sensi Daniele De","year":"2021","unstructured":"Daniele De Sensi , Salvatore Di Girolamo , Saleh Ashkboos, Shigang Li, and Torsten Hoefler. 2021 . Flare : Flexible In-Network Allreduce . [arxiv]2106.15565 [cs.DC] Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos, Shigang Li, and Torsten Hoefler. 2021. Flare: Flexible In-Network Allreduce. [arxiv]2106.15565 [cs.DC]"},{"key":"e_1_3_2_2_60_1","volume-title":"Johnson","author":"Shan Hongzhang","year":"2018","unstructured":"Hongzhang Shan , Samuel Williams , and Calvin W . Johnson . 2018 . Improving MPI Reduction Performance for Manycore Architectures with OpenMP and Data Compression. In PMBS. Hongzhang Shan, Samuel Williams, and Calvin W. Johnson. 2018. Improving MPI Reduction Performance for Manycore Architectures with OpenMP and Data Compression. In PMBS."},{"key":"e_1_3_2_2_61_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR.  Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR ."},{"key":"e_1_3_2_2_62_1","unstructured":"Sebastian U. Stich Jean-Baptiste Cordonnier and Martin Jaggi. 2018. Sparsified SGD with Memory. In NeurIPS.  Sebastian U. Stich Jean-Baptiste Cordonnier and Martin Jaggi. 2018. Sparsified SGD with Memory. In NeurIPS ."},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"crossref","unstructured":"Nikko Strom. 2015. Scalable Distributed DNN Training Using Commodity GPU Cloud Computing. In ISCA.  Nikko Strom. 2015. Scalable Distributed DNN Training Using Commodity GPU Cloud Computing. In ISCA .","DOI":"10.21437\/Interspeech.2015-354"},{"key":"e_1_3_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342005051521"},{"key":"e_1_3_2_2_65_1","unstructured":"Jesper Larsson Tr\u00e4ff. 2010. Transparent Neutral Element Elimination in MPI Reduction Operations. In EuroMPI.  Jesper Larsson Tr\u00e4ff. 2010. Transparent Neutral Element Elimination in MPI Reduction Operations. In EuroMPI ."},{"key":"e_1_3_2_2_66_1","volume-title":"Blink: Fast and Generic Collectives for Distributed ML. In MLSys.","author":"Wang Guanhua","year":"2020","unstructured":"Guanhua Wang , Shivaram Venkataraman , Amar Phanishayee , Jorgen Thelin , Nikhil Devanur , and Ion Stoica . 2020 . Blink: Fast and Generic Collectives for Distributed ML. In MLSys. Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee, Jorgen Thelin, Nikhil Devanur, and Ion Stoica. 2020. Blink: Fast and Generic Collectives for Distributed ML. In MLSys."},{"key":"e_1_3_2_2_67_1","unstructured":"Wei Wen Cong Xu Feng Yan Chunpeng Wu Yandan Wang Yiran Chen and Hai Li. 2017. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. In NeurIPS.  Wei Wen Cong Xu Feng Yan Chunpeng Wu Yandan Wang Yiran Chen and Hai Li. 2017. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. In NeurIPS ."},{"key":"e_1_3_2_2_68_1","volume-title":"Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis.","author":"Xu Hang","year":"2021","unstructured":"Hang Xu , Chen-Yu Ho , Ahmed M. Abdelmoniem , Aritra Dutta , El Houcine Bergou , Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. 2021 . GRACE : A Compressed Communication Framework for Distributed Machine Learning. In ICDCS. Hang Xu, Chen-Yu Ho, Ahmed M. Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. 2021. GRACE: A Compressed Communication Framework for Distributed Machine Learning. In ICDCS."},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"crossref","unstructured":"Zhen Zhang Chaokun Chang Haibin Lin Yida Wang Raman Arora and Xin Jin. 2020. Is Network the Bottleneck of Distributed Training?. In NetAI.  Zhen Zhang Chaokun Chang Haibin Lin Yida Wang Raman Arora and Xin Jin. 2020. Is Network the Bottleneck of Distributed Training?. In NetAI .","DOI":"10.1145\/3405671.3405810"},{"key":"e_1_3_2_2_70_1","volume-title":"Kylix: A Sparse Allreduce for Commodity Clusters. In ICPP.","author":"Zhao Huasha","year":"2014","unstructured":"Huasha Zhao and John Canny . 2014 . Kylix: A Sparse Allreduce for Commodity Clusters. In ICPP. Huasha Zhao and John Canny. 2014. Kylix: A Sparse Allreduce for Commodity Clusters. In ICPP."},{"key":"e_1_3_2_2_71_1","unstructured":"Shuai Zheng Ziyue Huang and James Kwok. 2019. Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback. In NeurIPS.  Shuai Zheng Ziyue Huang and James Kwok. 2019. Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback. In NeurIPS ."},{"key":"e_1_3_2_2_72_1","unstructured":"Shuxin Zheng Qi Meng Taifeng Wang Wei Chen Nenghai Yu Zhi-Ming Ma and Tie-Yan Liu. 2017. Asynchronous Stochastic Gradient Descent with Delay Compensation. In ICML.  Shuxin Zheng Qi Meng Taifeng Wang Wei Chen Nenghai Yu Zhi-Ming Ma and Tie-Yan Liu. 2017. Asynchronous Stochastic Gradient Descent with Delay Compensation. In ICML ."}],"event":{"name":"SIGCOMM '21: ACM SIGCOMM 2021 Conference","location":"Virtual Event USA","acronym":"SIGCOMM '21","sponsor":["SIGCOMM ACM Special Interest Group on Data Communication"]},"container-title":["Proceedings of the 2021 ACM SIGCOMM 2021 Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3452296.3472904","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3452296.3472904","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:13Z","timestamp":1750197673000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3452296.3472904"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,9]]},"references-count":72,"alternative-id":["10.1145\/3452296.3472904","10.1145\/3452296"],"URL":"https:\/\/doi.org\/10.1145\/3452296.3472904","relation":{},"subject":[],"published":{"date-parts":[[2021,8,9]]},"assertion":[{"value":"2021-08-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}