{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T16:00:17Z","timestamp":1786118417175,"version":"build-2736575974"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T00:00:00Z","timestamp":1785974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62272185"],"award-info":[{"award-number":["62272185"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"Key Laboratory of Information Storage System Ministry of Education of China","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"Shenzhen Science and Technology Program","doi-asserted-by":"publisher","award":["JCYJ20220530161006015"],"award-info":[{"award-number":["JCYJ20220530161006015"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>\n                    Distributed deep neural network (DNN) training across multiple workers is widely adopted to accelerate the training of large models and datasets. However, their performance is often limited by the synchronization of inter-worker data (i.e., gradient tensors). Top-\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    sparsification serves as the de facto compression approach to alleviate the communication bottleneck by retaining only the\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    largest-magnitude gradient elements before transmission to other nodes. However, traditional top-\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    sparsification has two key limitations. First, gradients at each DNN layer are represented as multi-dimensional tensors, with the selected\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    elements typically concentrated in a subset of dimensions. This leads to many dimensions being missed during training, and hence poor convergence. Second, traditional top-\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    sparsification relies on the global sorting of all gradient elements per layer, leading to underutilized GPU core parallelism and low training throughput. To this end, we propose\n                    <jats:sans-serif>ADTopk<\/jats:sans-serif>\n                    , an all-dimension top-\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    sparsification scheme that selects the\n                    <jats:italic toggle=\"yes\">k<\/jats:italic>\n                    largest elements across\n                    <jats:italic toggle=\"yes\">all<\/jats:italic>\n                    dimensions of the per-layer gradient tensor, ensuring that every dimension contributes at least one element and eliminating dimension missing.\n                    <jats:sans-serif>ADTopk<\/jats:sans-serif>\n                    also enables independent local sorting within each dimension, allowing parallel execution across dimensions to enhance GPU utilization. Furthermore, we enhance\n                    <jats:sans-serif>ADTopk<\/jats:sans-serif>\n                    with system-level optimizations: (i) interleaved sparsification to accelerate convergence, (ii) partial sparsification to reduce sparsification overhead, and (iii) hybrid collective communication to improve sparse communication efficiency. Experiments on local and cloud clusters show that\n                    <jats:sans-serif>ADTopk<\/jats:sans-serif>\n                    improves convergence accuracy by 2.75%-22.31% and achieves 20.8%-268.0% higher training throughput than state-of-the-art sparsification approaches.\n                  <\/jats:p>","DOI":"10.1145\/3832779","type":"journal-article","created":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T11:35:33Z","timestamp":1784633733000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Enabling Efficient All-Dimension Top-k Sparsification for High-Performance Distributed DNN Training Systems"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1616-8054","authenticated-orcid":false,"given":"Zhangqiang","family":"Ming","sequence":"first","affiliation":[{"name":"School of computer science and technology, Huazhong University of Science and Technology","place":["Wuhan, China"]},{"name":"The Chinese University of Hong Kong","place":["Wuhan, China"]},{"name":"Innovation Research Institute of Cethik Group Co., Ltd.","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1265-7141","authenticated-orcid":false,"given":"Yuchong","family":"Hu","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4501-4364","authenticated-orcid":false,"given":"Patrick P. C.","family":"Lee","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong","place":["Hong Kong, Hong Kong"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4643-2574","authenticated-orcid":false,"given":"Yuanhao","family":"Shu","sequence":"additional","affiliation":[{"name":"Innovation Research Institute of Cethik Group Co., Ltd.","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6391-7798","authenticated-orcid":false,"given":"Wenxiang","family":"Zhou","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7321-9264","authenticated-orcid":false,"given":"Xinjue","family":"Zheng","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4674-6006","authenticated-orcid":false,"given":"Dan","family":"Feng","sequence":"additional","affiliation":[{"name":"Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,6]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2025. BytePS. Retrieved January 1 2025 from https:\/\/github.com\/bytedance\/byteps"},{"key":"e_1_3_1_3_2","unstructured":"2025. Horovod. Retrieved January 1 2025 from https:\/\/github.com\/horovod\/horovod"},{"key":"e_1_3_1_4_2","first-page":"55","volume-title":"Proceedings of the 4th Machine Learning and Systems","author":"Agarwal Saurabh","year":"2021","unstructured":"Saurabh Agarwal, Hongyi Wang, Kangwook Lee, Shivaram Venkataraman, and Dimitris Papailiopoulos. 2021. Adaptive gradient communication via critical learning regime identification. In Proceedings of the 4th Machine Learning and Systems. 55\u201380."},{"key":"e_1_3_1_5_2","unstructured":"Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for distributed gradient descent. arXiv:1704.05021. Retrieved from https:\/\/arxiv.org\/abs\/1704.05021"},{"key":"e_1_3_1_6_2","first-page":"8024","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems","author":"Alistarh Dan","year":"2017","unstructured":"Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Proceedings of the 31st Conference on Neural Information Processing Systems. 8024\u20138035."},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483553"},{"key":"e_1_3_1_8_2","article-title":"Bloom: A 176b-parameter open-access multilingual language model","year":"2022","unstructured":"BigScience, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili\u0107, Daniel Hesslow, Roman Castagn\u00e9, Alexandra Sasha Luccioni, Fran\u00e7ois Yvon, et\u00a0al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. Retrieved from https:\/\/arxiv.org\/abs\/2211.05100","journal-title":"arXiv preprint"},{"key":"e_1_3_1_9_2","first-page":"1877","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems. 1877\u20131901."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS57875.2023.00015"},{"key":"e_1_3_1_11_2","first-page":"13551","volume-title":"Proceedings of the 34th Conference on Neural Information Processing Systems","author":"Chen Chia-Yu","year":"2020","unstructured":"Chia-Yu Chen, Jiamin Ni, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Xiao Sun, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Wei Zhang, et\u00a0al. 2020. ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training. In Proceedings of the 34th Conference on Neural Information Processing Systems. 13551\u201313563."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_1_13_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Dettmers Tim","year":"2022","unstructured":"Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_1_14_2","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805","journal-title":"arXiv preprint"},{"key":"e_1_3_1_15_2","article-title":"An image is worth 16x16 words: Transformers for image recognition at scale","author":"Dosovitskiy Alexey","year":"2020","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et\u00a0al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Retrieved from https:\/\/arxiv.org\/abs\/2010.11929","journal-title":"arXiv preprint"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2019.05.016"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3452296.3472904"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.5555\/3488766.3488792"},{"key":"e_1_3_1_20_2","article-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky, Geoffrey Hinton, et\u00a0al. 2009. Learning multiple layers of features from tiny images. Master\u2019s thesis, University of Tront (2009).","journal-title":"Master\u2019s thesis, University of Tront"},{"key":"e_1_3_1_21_2","first-page":"1191","volume-title":"Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation","author":"Li Minghao","year":"2024","unstructured":"Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. 2024. THC: Accelerating distributed deep learning using tensor homomorphic compression. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation. 1191\u20131211."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503221.3508399"},{"key":"e_1_3_1_23_2","article-title":"Pytorch distributed: Experiences on accelerating data parallel training","author":"Li Shen","year":"2020","unstructured":"Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et\u00a0al. 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704. Retrieved from https:\/\/arxiv.org\/abs\/2006.15704","journal-title":"arXiv preprint"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650200.3656596"},{"key":"e_1_3_1_25_2","article-title":"Deep gradient compression: Reducing the communication bandwidth for distributed training","author":"Lin Yujun","year":"2017","unstructured":"Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J. Dally. 2017. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887. Retrieved from https:\/\/arxiv.org\/abs\/1712.01887","journal-title":"arXiv preprint"},{"key":"e_1_3_1_26_2","first-page":"14542","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Luo Xu","year":"2022","unstructured":"Xu Luo, Jing Xu, and Zenglin Xu. 2022. Channel importance matters in few-shot image classification. In Proceedings of the International Conference on Machine Learning. 14542\u201314559."},{"key":"e_1_3_1_27_2","first-page":"297","volume-title":"Proceedings of the Machine Learning and Systems","author":"Abdelmoniem Ahmed M.","year":"2021","unstructured":"Ahmed M. Abdelmoniem, Ahmed Elzanaty, Mohamed-Slim Alouini, and Marco Canini. 2021. An efficient statistical-based gradient compression technique for distributed training systems. In Proceedings of the Machine Learning and Systems. 297\u2013322."},{"key":"e_1_3_1_28_2","article-title":"Regularizing and optimizing LSTM language models","author":"Merity Stephen","year":"2017","unstructured":"Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182. Retrieved from https:\/\/arxiv.org\/abs\/1708.02182","journal-title":"arXiv preprint"},{"key":"e_1_3_1_29_2","article-title":"Pointer sentinel mixture models","author":"Merity Stephen","year":"2016","unstructured":"Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Retrieved from https:\/\/arxiv.org\/abs\/1609.07843","journal-title":"arXiv preprint"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3731545.3731581"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3625549.3658678"},{"key":"e_1_3_1_32_2","first-page":"1752","volume-title":"Proceedings of the 2026 IEEE 42nd International Conference on Data Engineering","author":"Ming Zhangqiang","year":"2026","unstructured":"Zhangqiang Ming, Rui Wang, Yuchong Hu, Yuanhao Shu, Xinjue Zheng, Wenxiang Zhou, and Dan Feng. 2026. SSFusion: Tensor fusion with selective sparsification for efficient distributed DNN training. In Proceedings of the 2026 IEEE 42nd International Conference on Data Engineering. 1752\u20131764."},{"key":"e_1_3_1_33_2","unstructured":"NVIDIA. 2025. The Developer Tools Documentation of NVIDIA Nsight Systems. Retrieved January 1 2025 from https:\/\/docs.nvidia.com\/nsight-systems"},{"key":"e_1_3_1_34_2","first-page":"8026","volume-title":"Proceedings of the 33rd Conference on Neural Information Processing Systems","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et\u00a0al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd Conference on Neural Information Processing Systems. 8026\u20138037."},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3673038.3673140"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24685-5_1"},{"issue":"8","key":"e_1_3_1_37_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.","journal-title":"OpenAI blog"},{"key":"e_1_3_1_38_2","first-page":"1","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","author":"Rajbhandari Samyam","year":"2020","unstructured":"Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1\u201316."},{"key":"e_1_3_1_39_2","article-title":"Know what you don\u2019t know: Unanswerable questions for SQuAD","author":"Rajpurkar Pranav","year":"2018","unstructured":"Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don\u2019t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822. Retrieved from https:\/\/arxiv.org\/abs\/1806.03822","journal-title":"arXiv preprint"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2024.3381373"},{"key":"e_1_3_1_41_2","first-page":"8133","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Sahu Atal","year":"2021","unstructured":"Atal Sahu, Aritra Dutta, Ahmed M Abdelmoniem, Trambak Banerjee, Marco Canini, and Panos Kalnis. 2021. Rethinking gradient sparsification as total error minimization. In Proceedings of the Advances in Neural Information Processing Systems. 8133\u20138146."},{"key":"e_1_3_1_42_2","article-title":"Horovod: Fast and easy distributed deep learning in TensorFlow","author":"Sergeev Alexander","year":"2018","unstructured":"Alexander Sergeev and Mike Del Balso. 2018. Horovod: Fast and easy distributed deep learning in TensorFlow. (2018).arXiv preprint arXiv:1802.05799. Retrieved from https:\/\/arxiv.org\/abs\/1802.05799","journal-title":"arXiv preprint"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3183735"},{"key":"e_1_3_1_44_2","article-title":"ASHL: An adaptive multi-stage distributed deep learning training scheme for heterogeneous environments","author":"Shen Zhaoyan","year":"2023","unstructured":"Zhaoyan Shen, Qingxiang Tang, Tianren Zhou, Yuhao Zhang, Zhiping Jia, Dongxiao Yu, Zhiyong Zhang, and Bingzhe Li. 2023. ASHL: An adaptive multi-stage distributed deep learning training scheme for heterogeneous environments. IEEE Transactions on Computers 73, 1 (2023), 30\u201343.","journal-title":"IEEE Transactions on Computers"},{"key":"e_1_3_1_45_2","article-title":"Understanding top-k sparsification in distributed deep learning","author":"Shi Shaohuai","year":"2019","unstructured":"Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. 2019. Understanding top-k sparsification in distributed deep learning. arXiv preprint arXiv:1911.08772. Retrieved from https:\/\/arxiv.org\/abs\/1911.08772","journal-title":"arXiv preprint"},{"key":"e_1_3_1_46_2","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556","journal-title":"arXiv preprint"},{"key":"e_1_3_1_47_2","first-page":"4452","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Stich Sebastian U.","year":"2018","unstructured":"Sebastian U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. 2018. Sparsified SGD with memory. In Proceedings of the Advances in Neural Information Processing Systems. 4452\u20134463."},{"key":"e_1_3_1_48_2","first-page":"6000","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems. 6000\u20136010."},{"key":"e_1_3_1_49_2","first-page":"14236","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Vogels Thijs","year":"2019","unstructured":"Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. 2019. PowerSGD: Practical low-rank gradient compression for distributed optimization. In Proceedings of the Advances in Neural Information Processing Systems. 14236\u201314245."},{"key":"e_1_3_1_50_2","first-page":"9872","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Wang Hongyi","year":"2018","unstructured":"Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. 2018. Atomo: Communication-efficient learning via atomic sparsification. In Proceedings of the Advances in Neural Information Processing Systems. 9872\u20139883."},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3552326.3567505"},{"key":"e_1_3_1_52_2","first-page":"373","volume-title":"Proceedings of the Machine Learning and Systems","author":"Wang Zhuang","year":"2023","unstructured":"Zhuang Wang, Xinyu Wu, Zhaozhuo Xu, and T. S. Eugene Ng. 2023. Cupcake: A compression scheduler for scalable communication-efficient distributed training. In Proceedings of the Machine Learning and Systems. 373\u2013386."},{"key":"e_1_3_1_53_2","first-page":"1306","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Wangni Jianqiao","year":"2018","unstructured":"Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. 2018. Gradient sparsification for communication-efficient distributed optimization. In Proceedings of the Advances in Neural Information Processing Systems. 1306\u20131316."},{"key":"e_1_3_1_54_2","first-page":"1508","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Wen Wei","year":"2017","unstructured":"Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2017. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Proceedings of the Advances in Neural Information Processing Systems. 1508\u20131518."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS51616.2021.00060"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE60146.2024.00142"},{"key":"e_1_3_1_57_2","article-title":"Pytorch fsdp: Experiences on scaling fully sharded data parallel","author":"Zhao Yanli","year":"2023","unstructured":"Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et\u00a0al. 2023. Pytorch fsdp: Experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277. Retrieved from https:\/\/arxiv.org\/abs\/2304.11277","journal-title":"arXiv preprint"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3832779","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T15:43:55Z","timestamp":1786031035000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3832779"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,6]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3832779"],"URL":"https:\/\/doi.org\/10.1145\/3832779","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,6]]},"assertion":[{"value":"2025-11-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-13","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}