{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T18:05:34Z","timestamp":1789581934349,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":35,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T00:00:00Z","timestamp":1790467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"funder":[{"name":"Sichuan Science and Technology Program","award":["2025ZDZX0007"],"award-info":[{"award-number":["2025ZDZX0007"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,9,28]]},"DOI":"10.1145\/3832810.3832826","type":"proceedings-article","created":{"date-parts":[[2026,9,13]],"date-time":"2026-09-13T00:19:06Z","timestamp":1789258746000},"page":"1041-1050","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Low-bit and Sparsified Gradient Communication for Accelerating Distributed Deep Learning with Convergence Guarantees"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5481-3901","authenticated-orcid":false,"given":"Jiaqi","family":"Li","sequence":"first","affiliation":[{"name":"Chengdu University of Information Technology, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1418-5160","authenticated-orcid":false,"given":"Shaohuai","family":"Shi","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7221-1216","authenticated-orcid":false,"given":"Jing","family":"Peng","sequence":"additional","affiliation":[{"name":"Chengdu University of Information Technology, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4472-2534","authenticated-orcid":false,"given":"Tiejun","family":"Wang","sequence":"additional","affiliation":[{"name":"Chengdu University of Information Technology, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8602-3996","authenticated-orcid":false,"given":"Jiqing","family":"Gu","sequence":"additional","affiliation":[{"name":"Chengdu University of Information Technology, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7659-1631","authenticated-orcid":false,"given":"Xi","family":"Wu","sequence":"additional","affiliation":[{"name":"Chengdu University of Information Technology, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,9,27]]},"reference":[{"key":"e_1_3_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1045"},{"key":"e_1_3_3_1_3_2","unstructured":"Dan Alistarh Demjan Grubic Jerry Li Ryota Tomioka and Milan Vojnovic. 2017. QSGD: Communication-efficient SGD via gradient quantization and encoding. Advances in neural information processing systems 30."},{"key":"e_1_3_3_1_4_2","unstructured":"Baidu. 2017. Baidu AllReduce Library. https:\/\/github.com\/baidu-research\/baidu-allreduce."},{"key":"e_1_3_3_1_5_2","first-page":"560","volume-title":"International conference on machine learning","author":"Bernstein Jeremy","year":"2018","unstructured":"Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. 2018. signSGD: Compressed optimisation for non-convex problems. In International conference on machine learning. PMLR, 560\u2013569."},{"key":"e_1_3_3_1_6_2","unstructured":"Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Marc\u2019aurelio Ranzato Andrew Senior Paul Tucker Ke Yang et\u00a0al. 2012. Large scale distributed deep networks. Advances in neural information processing systems 25."},{"key":"e_1_3_3_1_7_2","unstructured":"Tim Dettmers Mike Lewis Sam Shleifer and Luke Zettlemoyer. 2021. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2110.02861 (2021)."},{"key":"e_1_3_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_3_1_9_2","unstructured":"Priya Goyal Piotr Doll\u00e1r Ross Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia and Kaiming He. 2017. Accurate large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1706.02677 (2017)."},{"key":"e_1_3_3_1_10_2","first-page":"3252","volume-title":"International conference on machine learning","author":"Karimireddy Sai\u00a0Praneeth","year":"2019","unstructured":"Sai\u00a0Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. 2019. Error feedback fixes signsgd and other gradient compression schemes. In International conference on machine learning. PMLR, 3252\u20133261."},{"key":"e_1_3_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503221.3508399"},{"key":"e_1_3_3_1_12_2","unstructured":"Shen Li Yanli Zhao Rohan Varma Omkar Salpekar Pieter Noordhuis Teng Li Adam Paszke Jeff Smith Brian Vaughan Pritam Damania et\u00a0al. 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2006.15704 (2020)."},{"key":"e_1_3_3_1_13_2","unstructured":"Yujun Lin Song Han Huizi Mao Yu Wang and William\u00a0J Dally. 2017. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1712.01887."},{"key":"e_1_3_3_1_14_2","unstructured":"Stephen Merity Caiming Xiong James Bradbury and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1609.07843 (2016)."},{"key":"e_1_3_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3673038.3673140"},{"key":"e_1_3_3_1_16_2","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9."},{"key":"e_1_3_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3295500.3356222"},{"key":"e_1_3_3_1_18_2","first-page":"1058","volume-title":"Interspeech","author":"Seide Frank","year":"2014","unstructured":"Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs.. In Interspeech , Vol.\u00a02014. Singapore, 1058\u20131062."},{"key":"e_1_3_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2019.8737367"},{"key":"e_1_3_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2019.00220"},{"key":"e_1_3_3_1_21_2","unstructured":"Shaohuai Shi Xianhao Zhou Shutao Song Xingyao Wang Zilin Zhu Xue Huang Xinan Jiang Feihu Zhou Zhenyu Guo Liqiang Xie et\u00a0al. 2021. Towards scalable distributed training of deep learning on public cloud clusters. Proceedings of Machine Learning and Systems 3 401\u2013412."},{"key":"e_1_3_3_1_22_2","unstructured":"Mohammad Shoeybi Mostofa Patwary Raul Puri Patrick LeGresley Jared Casper and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1909.08053 (2019)."},{"key":"e_1_3_3_1_23_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1409.1556 (2014)."},{"key":"e_1_3_3_1_24_2","unstructured":"Sebastian\u00a0U Stich Jean-Baptiste Cordonnier and Martin Jaggi. 2018. Sparsified SGD with memory. Advances in neural information processing systems 31 (2018)."},{"key":"e_1_3_3_1_25_2","first-page":"10118","volume-title":"International Conference on Machine Learning","author":"Tang Hanlin","year":"2021","unstructured":"Hanlin Tang, Shaoduo Gan, Ammar\u00a0Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 2021. 1-bit adam: Communication efficient large-scale training with adam\u2019s convergence speed. In International Conference on Machine Learning. PMLR, 10118\u201310129."},{"key":"e_1_3_3_1_26_2","unstructured":"Thijs Vogels Sai\u00a0Praneeth Karimireddy and Martin Jaggi. 2019. PowerSGD: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems 32."},{"key":"e_1_3_3_1_27_2","unstructured":"Guanhua Wang Heyang Qin Sam\u00a0Ade Jacobs Connor Holmes Samyam Rajbhandari Olatunji Ruwase Feng Yan Lei Yang and Yuxiong He. 2023. Zero++: Extremely efficient collective communication for giant model training. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2306.10209."},{"key":"e_1_3_3_1_28_2","unstructured":"Wei Wen Cong Xu Feng Yan Chunpeng Wu Yandan Wang Yiran Chen and Hai Li. 2017. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems 30."},{"key":"e_1_3_3_1_29_2","unstructured":"Mengzhou Xia Tianyu Gao Zhiyuan Zeng and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. 2024 (2024) 5385\u20135409."},{"key":"e_1_3_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.00552"},{"key":"e_1_3_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Guangfeng Yan Tan Li Shao-Lun Huang Tian Lan and Linqi Song. 2022. AC-SGD: Adaptively compressed SGD for communication-efficient distributed learning. IEEE Journal on Selected Areas in Communications 40 9 (2022) 2678\u20132693.","DOI":"10.1109\/JSAC.2022.3192050"},{"key":"e_1_3_3_1_32_2","first-page":"169","volume-title":"European Conference on Parallel Processing","author":"Yang Xinrui","year":"2025","unstructured":"Xinrui Yang and Shaohuai Shi. 2025. SQ-DeAR: Sparsified and Quantized Gradient Compression for Distributed Training. In European Conference on Parallel Processing. Springer, 169\u2013182."},{"key":"e_1_3_3_1_33_2","first-page":"181","volume-title":"2017 USENIX Annual Technical Conference (USENIX ATC 17)","author":"Zhang Hao","year":"2017","unstructured":"Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, and Eric\u00a0P Xing. 2017. Poseidon: An efficient communication architecture for distributed deep learning on { GPU} clusters. In 2017 USENIX Annual Technical Conference (USENIX ATC 17). 181\u2013193."},{"key":"e_1_3_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS57875.2023.00054"},{"key":"e_1_3_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS57875.2023.00031"},{"key":"e_1_3_3_1_36_2","unstructured":"Yanli Zhao Andrew Gu Rohan Varma Liang Luo Chien-Chin Huang Min Xu Less Wright Hamid Shojanazeri Myle Ott Sam Shleifer et\u00a0al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2304.11277 (2023)."}],"event":{"name":"ICPP '26: Proceedings of the 55th International Conference on Parallel Processing","location":"Singapore , Singapore","acronym":"ICPP '26"},"container-title":["Proceedings of the 55th International Conference on Parallel Processing"],"original-title":[],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T17:58:36Z","timestamp":1789581516000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3832810.3832826"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,27]]},"references-count":35,"alternative-id":["10.1145\/3832810.3832826","10.1145\/3832810"],"URL":"https:\/\/doi.org\/10.1145\/3832810.3832826","relation":{},"subject":[],"published":{"date-parts":[[2026,9,27]]},"assertion":[{"value":"2026-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}