{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T02:38:42Z","timestamp":1785206322643,"version":"3.55.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"CoNEXT2","funder":[{"name":"Institute of Information & Communications Technology Planning & Evaluation","award":["IITP-2026-RS-2024-00418784"],"award-info":[{"award-number":["IITP-2026-RS-2024-00418784"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Netw."],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"<jats:p>Stragglers remain an open challenge in large model training, forcing other GPUs to wait at synchronization barriers. While severe, persistent stragglers have been actively studied, their transient counterparts are often overlooked due to their short-lived nature. We reveal that the interplay between the transient straggler effects and network contention drastically amplifies the relative delays across workers, significantly slowing down training iterations. To address this, an intuitive approach is to prioritize bandwidth for the straggler, allowing it to catch up to the front-runners. However, while the transport layer is well-positioned to accommodate such responsive bandwidth control, conventional designs lack visibility into process-level progress. To bridge this gap, we propose PRC (Process-centric Rate Control), a new sending-rate control designed to mitigate the transient straggler effects. PRC adjusts NIC sending rates by inferring local GPU process-level information, enabling transient straggler processes to utilize more bandwidth on time. Extensive experiments on both a real-world cluster and large-scale simulations confirm that PRC effectively accelerates transient stragglers, achieving a training speedup of up to 28% compared to using state-of-the-art datacenter congestion control schemes.<\/jats:p>","DOI":"10.1145\/3808669","type":"journal-article","created":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:45:46Z","timestamp":1780346746000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-7649-5808","authenticated-orcid":false,"given":"Taekyoung","family":"Han","sequence":"first","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1571-6120","authenticated-orcid":false,"given":"Daewook","family":"Hwang","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7375-8587","authenticated-orcid":false,"given":"Shinik","family":"Park","sequence":"additional","affiliation":[{"name":"Samsung Electronics, Suwon, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3602-0720","authenticated-orcid":false,"given":"Donggyu","family":"Yang","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3659-2615","authenticated-orcid":false,"given":"Gibum","family":"Park","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8667-5521","authenticated-orcid":false,"given":"Sanghyun","family":"Han","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2829-8212","authenticated-orcid":false,"given":"Jaeyoon","family":"Hwang","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6811-2766","authenticated-orcid":false,"given":"Byunggu","family":"Kang","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5983-5430","authenticated-orcid":false,"given":"Sangtae","family":"Ha","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Colorado, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8647-1476","authenticated-orcid":false,"given":"Kyunghan","family":"Lee","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,1]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2005. Pluggable congestion avoidance modules. http:\/\/lwn.net\/Articles\/ 128681\/ (2005)."},{"key":"e_1_2_1_2_1","volume-title":"NVIDIA ConnectX-6 Dx datasheet. https:\/\/www.nvidia.com\/content\/dam\/en-zz\/ Solutions\/networking\/ ethernetadapters\/ connectX-6-dx-datasheet.pdf","year":"2021","unstructured":"2021. NVIDIA ConnectX-6 Dx datasheet. https:\/\/www.nvidia.com\/content\/dam\/en-zz\/ Solutions\/networking\/ ethernetadapters\/ connectX-6-dx-datasheet.pdf (2021)."},{"key":"e_1_2_1_3_1","volume-title":"NVIDIA Nsight Systems. https:\/\/developer.nvidia.com\/nsight-systems","year":"2021","unstructured":"2021. NVIDIA Nsight Systems. https:\/\/developer.nvidia.com\/nsight-systems (2021)."},{"key":"e_1_2_1_4_1","unstructured":"2021. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B the world's largest and most powerful generative language model. https:\/\/developer.nvidia.com\/blog\/ using-deepspeed-and-megatron-to-train-megatronturing- nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model\/ (2021)."},{"key":"e_1_2_1_5_1","volume-title":"Getting started with DeviceMesh. https:\/\/docs.pytorch.org\/ tutorials\/ recipes\/ distributed_device_mesh.html","year":"2024","unstructured":"2024. Getting started with DeviceMesh. https:\/\/docs.pytorch.org\/ tutorials\/ recipes\/ distributed_device_mesh.html (2024)."},{"key":"e_1_2_1_6_1","volume-title":"Mlxreg utility. https:\/\/docs.nvidia.com\/networking\/ display\/ mftv422\/mlxregutility","year":"2024","unstructured":"2024. Mlxreg utility. https:\/\/docs.nvidia.com\/networking\/ display\/ mftv422\/mlxregutility (2024)."},{"key":"e_1_2_1_7_1","volume-title":"NVIDIA DOCA PCC application guide. https:\/\/docs.nvidia.com\/doca\/ sdk\/ nvidiadocapccapplicationguide\/ index.html","year":"2024","unstructured":"2024. NVIDIA DOCA PCC application guide. https:\/\/docs.nvidia.com\/doca\/ sdk\/ nvidiadocapccapplicationguide\/ index.html (2024)."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 185-198","author":"Ananthanarayanan Ganesh","year":"2013","unstructured":"Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, and Ion Stoica. 2013. Effective straggler mitigation: Attack of the clones. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 185-198."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/1924943.1924962"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 93-109","author":"Arashloo Mina Tahmasbi","year":"2020","unstructured":"Mina Tahmasbi Arashloo, Alexey Lavrov, Manya Ghobadi, Jennifer Rexford, David Walker, and David Wentzlaff. 2020. Enabling programmable transport protocols in high-speed NICs. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 93-109."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 219-236","author":"Arslan Serhat","year":"2023","unstructured":"Serhat Arslan, Yuliang Li, Gautam Kumar, and Nandita Dukkipati. 2023. Bolt: Sub-RTT congestion control for ultra-low latency. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 219-236."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 662-680","author":"Basat Ran Ben","year":"2020","unstructured":"Ran Ben Basat, Sivaramakrishnan Ramanathan, Yuliang Li, Gianni Antichi, Minian Yu, and Michael Mitzenmacher. 2020. PINT: Probabilistic in-band network telemetry. In Proceedings of the ACM Conference on SIGCOMM. 662-680."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/324133.324234"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2-11","author":"Chan Ernie","year":"2006","unstructured":"Ernie Chan, Robert Van De Geijn, William Gropp, and Rajeev Thakur. 2006. Collective communication on architectures that support simultaneous communication over multiple links. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2-11."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 29th ACMInternational Conference on Architectural Support for Programming Languages and Operating Systems. 178-191","author":"Chen Chang","year":"2024","unstructured":"Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. 2024. Centauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning. In Proceedings of the 29th ACMInternational Conference on Architectural Support for Programming Languages and Operating Systems. 178-191."},{"key":"e_1_2_1_16_1","volume-title":"Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN systems 17, 1","author":"Chiu Dah-Ming","year":"1989","unstructured":"Dah-Ming Chiu and Raj Jain. 1989. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN systems 17, 1 (1989), 1-14."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the ACM Annual International Symposium on Computer Architecture. 1703-1716","author":"Chu Weiwei","year":"2025","unstructured":"Weiwei Chu, Xinfeng Xie, Jiecao Yu, Jie Wang, Amar Phanishayee, Chunqiang Tang, Yuchen Hao, Jianyu Huang, Mustafa Ozdal, Jun Wang, et al. 2025. Scaling Llama 3 training with efficient parallelism strategies. In Proceedings of the ACM Annual International Symposium on Computer Architecture. 1703-1716."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the USENIX Workshop on Hot Topics in Operating Systems.","author":"Cipar James","year":"2013","unstructured":"James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Gregory R. Ganger, Garth Gibson, Kimberly Keeton, and Eric Xing. 2013. Solving the straggler problem with bounded staleness. In Proceedings of the USENIX Workshop on Hot Topics in Operating Systems."},{"key":"e_1_2_1_19_1","volume-title":"Xputimer: Anomaly diagnostics for divergent LLM training in GPU clusters of thousand-plus scale. arXiv preprint arXiv:2502.05413","author":"Cui Weihao","year":"2025","unstructured":"Weihao Cui, Ji Zhang, Han Zhao, Chao Liu, Wenhao Zhang, Jian Sha, Quan Chen, Bingsheng He, and Minyi Guo. 2025. Xputimer: Anomaly diagnostics for divergent LLM training in GPU clusters of thousand-plus scale. arXiv preprint arXiv:2502.05413 (2025)."},{"key":"e_1_2_1_20_1","volume-title":"Mycroft: Tracing dependencies in collective communication towards reliable LLM training. arXiv preprint arXiv:2509.03018","author":"Deng Yangtao","year":"2025","unstructured":"Yangtao Deng, Lei Zhang, Qinlong Wang, Xiaoyun Zhi, Xinlei Zhang, Zhuo Jiang, Haohan Xu, Lei Wang, Zuquan Song, Gaohong Liu, et al. 2025. Mycroft: Tracing dependencies in collective communication towards reliable LLM training. arXiv preprint arXiv:2509.03018 (2025)."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the ACM International Conference on High Performance Computing Networking, Storage and Analysis. 1-11","author":"Dinan James","year":"2009","unstructured":"James Dinan, D Brian Larkins, Ponnuswamy Sadayappan, Sriram Krishnamoorthy, and Jarek Nieplocha. 2009. Scalable work stealing. In Proceedings of the ACM International Conference on High Performance Computing Networking, Storage and Analysis. 1-11."},{"key":"e_1_2_1_22_1","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov DirkWeissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)."},{"key":"e_1_2_1_23_1","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1111322.1111336"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 57-70","author":"Gangidi Adithya","year":"2024","unstructured":"Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. 2024. RDMA over ethernet for distributed training at meta scale. In Proceedings of the ACM Conference on SIGCOMM. 57-70."},{"key":"e_1_2_1_26_1","unstructured":"Google. 2025. Taming the stragglers: Maximize AI training performance with automated straggler detection. https: \/\/ cloud.google.com\/blog\/ products\/compute\/stragglers-in-ai-a-guide-to-automated-straggler-detection?hl=en (2025)."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 202-215","author":"Guo Chuanxiong","year":"2016","unstructured":"Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over commodity ethernet at scale. In Proceedings of the ACM Conference on SIGCOMM. 202-215."},{"key":"e_1_2_1_28_1","first-page":"450","volume-title":"Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems","volume":"2","author":"Guo Cong","year":"2024","unstructured":"Cong Guo, Rui Zhang, Jiale Xu, Jingwen Leng, Zihan Liu, Ziyu Huang, Minyi Guo, Hao Wu, Shouren Zhao, Junping Zhao, et al. 2024. GMLake: Efficient and transparent GPU memory defragmentation for large-scale DNN training with virtual memory stitching. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 450-466."},{"key":"e_1_2_1_29_1","volume-title":"GPU-initiated networking for NCCL. arXiv preprint arXiv:2511.15076","author":"Hamidouche Khaled","year":"2025","unstructured":"Khaled Hamidouche, John Bachan, Pak Markthub, Peter-Jan Gootzen, Elena Agostini, Sylvain Jeaugey, Aamir Shafi, Georgios Theodorakis, and Manjunath Gorentla Venkata. 2025. GPU-initiated networking for NCCL. arXiv preprint arXiv:2511.15076 (2025)."},{"key":"e_1_2_1_30_1","volume-title":"Unicron: Economizing self-healing LLM training at scale. arXiv preprint arXiv:2401.00134","author":"He Tao","year":"2023","unstructured":"Tao He, Xue Li, Zhibin Wang, Kun Qian, Jingbo Xu, Wenyuan Yu, and Jingren Zhou. 2023. Unicron: Economizing self-healing LLM training at scale. arXiv preprint arXiv:2401.00134 (2023)."},{"key":"e_1_2_1_31_1","volume-title":"Benchmarking Methodology for AI Inference Serving Network Fabrics. https:\/\/datatracker.ietf .org\/ doc\/ html\/ draft-calabria-bmwg-ai-fabric-inference-bench-00","year":"2026","unstructured":"Huawei. 2026. Benchmarking Methodology for AI Inference Serving Network Fabrics. https:\/\/datatracker.ietf .org\/ doc\/ html\/ draft-calabria-bmwg-ai-fabric-inference-bench-00 (2026)."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 745-760","author":"Jiang Ziheng","year":"2024","unstructured":"Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. 2024. MegaScale: Scaling large language model training to more than 10,000 GPUs. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 745-760."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 514-528","author":"Kumar Gautam","year":"2020","unstructured":"Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, et al. 2020. Swift: Delay is simple and effective for congestion control in the datacenter. In Proceedings of the ACM Conference on SIGCOMM. 514-528."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_2_1_35_1","volume-title":"STrack: A reliable multipath transport for AI\/ML clusters. arXiv preprint arXiv:2407.15266","author":"Le Yanfang","year":"2024","unstructured":"Yanfang Le, Rong Pan, Peter Newman, Jeremias Blendin, Abdul Kabbani, Vipin Jain, Raghava Sivaramu, and Francis Matus. 2024. STrack: A reliable multipath transport for AI\/ML clusters. arXiv preprint arXiv:2407.15266 (2024)."},{"key":"e_1_2_1_36_1","volume-title":"Malleus: Straggler-resilient hybrid parallel training of large-scale models via malleable data and model parallelization. arXiv preprint arXiv:2410.13333","author":"Li Haoyang","year":"2024","unstructured":"Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Yujie Wang, Hailin Zhang, Xiaonan Nie, and Bin Cui. 2024. Malleus: Straggler-resilient hybrid parallel training of large-scale models via malleable data and model parallelization. arXiv preprint arXiv:2410.13333 (2024)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNET.2024.3441039"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 44-58","author":"Li Yuliang","year":"2019","unstructured":"Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: High precision congestion control. In Proceedings of the ACM Conference on SIGCOMM. 44-58."},{"key":"e_1_2_1_39_1","unstructured":"Wanchao Liang Tianyu Liu Less Wright Will Constable Andrew Gu Chien-Chin Huang Iris Zhang Wei Feng Howard Huang Junjie Wang et al. 2024. TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training. arXiv preprint arXiv:2410.06511 (2024)."},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the USENIX Conference on Operating Systems Design and Implementation. 483-498","author":"Lin Jinkun","year":"2025","unstructured":"Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi,Wei Jia, Zherui Liu, ShuguangWang, Haibin Lin, Xin Liu, Aurojit Panda, and Jinyang Li. 2025. Understanding stragglers in large model training using what-if analysis. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation. 483-498."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 809-824","author":"Mahajan Kshiteej","year":"2023","unstructured":"Kshiteej Mahajan, Ching-Hsiang Chu, Srinivas Sridharan, and Aditya Akella. 2023. Better together: Jointly optimizing ML collective scheduling and execution planning using SYNDICATE. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 809-824."},{"key":"e_1_2_1_42_1","volume-title":"DeepSpeed: Extreme-scale model training for everyone. https:\/\/www.microsoft.com\/en-us\/ research\/ blog\/ deepspeed-extreme-scale-model-training-for-everyone\/","year":"2020","unstructured":"Microsoft. 2020. DeepSpeed: Extreme-scale model training for everyone. https:\/\/www.microsoft.com\/en-us\/ research\/ blog\/ deepspeed-extreme-scale-model-training-for-everyone\/ (2020)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1145\/2829988.2787510","article-title":"TIMELY: RTT-based congestion control for the datacenter","volume":"45","author":"Mittal Radhika","year":"2015","unstructured":"Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, HassanWassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. 2015. TIMELY: RTT-based congestion control for the datacenter. ACM Computer Communication Review 45, 4 (2015), 537-550.","journal-title":"ACM Computer Communication Review"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the ACM\/IEEE Annual International Symposium on Computer Architecture. 993-1011","author":"Mudigere Dheevatsa","year":"2022","unstructured":"Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, et al. 2022. Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the ACM\/IEEE Annual International Symposium on Computer Architecture. 993-1011."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the ACM Symposium on Operating Systems Principles. 439-455","author":"Murray Derek G.","year":"2013","unstructured":"Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Mart\u00edn Abadi. 2013. Naiad: A timely dataflow system. In Proceedings of the ACM Symposium on Operating Systems Principles. 439-455."},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Deepak Narayanan Mohammad Shoeybi Jared Casper Patrick LeGresley Mostofa Patwary Vijay Korthikanti Dmitri Vainbrand Prethvi Kashinkunti Julie Bernauer and Bryan others Catanzaro. 2021. Efficient large-scale language model training on GPU clusters using Megatron-LM. (2021) 1-15.","DOI":"10.1145\/3458817.3476209"},{"key":"e_1_2_1_47_1","first-page":"693","article-title":"HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent","author":"Niu Feng","year":"2011","unstructured":"Feng Niu, Benjamin Recht, Christopher Re, and Stephen J. Wright. 2011. HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems. 693-701.","journal-title":"Advances in Neural Information Processing Systems."},{"key":"e_1_2_1_48_1","unstructured":"NVIDIA. [n.d.]. Communication Overlap. https:\/\/docs.nvidia.com\/nemo-framework\/ user-guide\/24.09\/nemotoolkit\/ features\/ optimizations\/communication_overlap.html ([n.d.])."},{"key":"e_1_2_1_49_1","volume-title":"CUDA Driver API. https:\/\/docs.nvidia.com\/cuda\/ cuda-driver-api\/ group__CUDA__VA.html","author":"NVIDIA.","year":"2025","unstructured":"NVIDIA. 2025. CUDA Driver API. https:\/\/docs.nvidia.com\/cuda\/ cuda-driver-api\/ group__CUDA__VA.html (2025)."},{"key":"e_1_2_1_50_1","volume-title":"Resiliency features. https:\/\/docs.nvidia.com\/nemo-framework\/ user-guide\/ latest\/ resiliency.html","author":"NVIDIA.","year":"2025","unstructured":"NVIDIA. 2025. Resiliency features. https:\/\/docs.nvidia.com\/nemo-framework\/ user-guide\/ latest\/ resiliency.html (2025)."},{"key":"e_1_2_1_51_1","volume-title":"Scale and Serve Generative AI | NVIDIA Dynamo. https:\/\/www.nvidia.com\/en-us\/ ai\/dynamo\/","author":"NVIDIA.","year":"2026","unstructured":"NVIDIA. 2026. Scale and Serve Generative AI | NVIDIA Dynamo. https:\/\/www.nvidia.com\/en-us\/ ai\/dynamo\/ (2026)."},{"key":"e_1_2_1_52_1","unstructured":"NVIDIA Developer Blog. 2022. Doubling all2all Performance with NVIDIA Collective Communication Library 2.12. https:\/\/developer.nvidia.com\/blog\/doubling-all2all-performance-with-nvidia-collective-communication-library- 2-12."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the ACM European Conference on Computer Systems. 541-557","author":"Pan Lichen","year":"2025","unstructured":"Lichen Pan, Juncheng Liu, Yongquan Fu, Jinhui Yuan, Rongkai Zhang, Pengze Li, and Zhen Xiao. 2025. Comprehensive deadlock prevention for GPU collective communication. In Proceedings of the ACM European Conference on Computer Systems. 541-557."},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the ACM\/IEEE Conference on Supercomputing. 55","author":"Petrini Fabrizio","year":"2003","unstructured":"Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The case of the missing supercomputer performance: Achieving optimal performance on the 8,192 processors of ASCI Q. In Proceedings of the ACM\/IEEE Conference on Supercomputing. 55."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 691-706","author":"Qian Kun","year":"2024","unstructured":"Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai. 2024. Alibaba HPN: A data center network for large language model training. In Proceedings of the ACM Conference on SIGCOMM. 691-706."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the ACM Annual International Symposium on Computer Architecture. 498-513","author":"Qin Le","year":"2025","unstructured":"Le Qin, Junwei Cui, Weilin Cai, and Jiayi Huang. 2025. Chimera: Communication fusion for hybrid parallelism in large language models. In Proceedings of the ACM Annual International Symposium on Computer Architecture. 498-513."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3563766.3564115"},{"key":"e_1_2_1_58_1","volume-title":"MLTCP: Congestion control for DNN training. arXiv preprint arXiv:2402.09589","author":"Rajasekaran Sudarsanan","year":"2024","unstructured":"Sudarsanan Rajasekaran, Sanjoli Narang, Anton A Zabreyko, and Manya Ghobadi. 2024. MLTCP: Congestion control for DNN training. arXiv preprint arXiv:2402.09589 (2024)."},{"key":"e_1_2_1_59_1","doi-asserted-by":"crossref","unstructured":"K Ramakrishnan Sally Floyd and David Black. 2001. Rfc3168: The addition of explicit congestion notification (ecn) to ip.","DOI":"10.17487\/rfc3168"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of IEEE International Symposium on Performance Analysis of Systems and Software. 81-92","author":"Rashidi Saeed","year":"2020","unstructured":"Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2020. ASTRA-SIM: Enabling SW\/HWcodesign exploration for distributed DL training platforms. In Proceedings of IEEE International Symposium on Performance Analysis of Systems and Software. 81-92."},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the ACM International Symposium on Computer Architecture. 34-48","author":"Rashidi Saeed","year":"2025","unstructured":"Saeed Rashidi, William Won, Sudarshan Srinivasan, Puneet Gupta, and Tushar Krishna. 2025. FRED: A wafer-scale fabric for 3D parallel DNN training. In Proceedings of the ACM International Symposium on Computer Architecture. 34-48."},{"key":"e_1_2_1_62_1","first-page":"1","article-title":"Exploiting simultaneous communications to accelerate data parallel distributed deep learning","author":"Shi Shaohuai","year":"2021","unstructured":"Shaohuai Shi, Xiaowen Chu, and Bo Li. 2021. Exploiting simultaneous communications to accelerate data parallel distributed deep learning. In Proceedings of IEEE INFOCOM. 1-10.","journal-title":"Proceedings of IEEE INFOCOM."},{"key":"e_1_2_1_63_1","volume-title":"Megatron- LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053","author":"Shoeybi Mohammad","year":"2019","unstructured":"Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron- LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)."},{"key":"e_1_2_1_64_1","first-page":"296","article-title":"Programmable congestion control communication scheme","volume":"11","author":"Shpigelman Yuval","year":"2022","unstructured":"Yuval Shpigelman, Idan Burstein, Noam Bloch, Reut Zuck, and Roee Moyal. 2022. Programmable congestion control communication scheme. US Patent 11,296,988.","journal-title":"US Patent"},{"key":"e_1_2_1_65_1","volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512798.3512815"},{"key":"e_1_2_1_67_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_2_1_68_1","volume-title":"Proceedings of theUSENIX Symposium on Networked Systems Design and Implementation. 1421-1443","author":"Tian Han","year":"2024","unstructured":"HaoWang, Han Tian, Jingrong Chen, XinchenWan, Jiacheng Xia, Gaoxiong Zeng,Wei Bai, Junchen Jiang, YongWang, and Kai Chen. 2024. Towards domain-specific network transport for distributed DNN training. In Proceedings of theUSENIX Symposium on Networked Systems Design and Implementation. 1421-1443."},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 685-703","author":"Warraich Ertza","year":"2025","unstructured":"Ertza Warraich, Omer Shabtai, Khalid Manaa, Shay Vargaftik, Yonatan Piasetzky, Matty Kadosh, Lalith Suresh, and Muhammad Shahbaz. 2025. OptiReduce: Resilient and tail-optimal AllReduce for distributed deep learning in the cloud. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 685-703."},{"key":"e_1_2_1_70_1","volume-title":"Adaptra: Straggler-resilient hybrid-parallel training with pipeline adaptation. arXiv preprint arXiv:2504.19232","author":"Wu Tianyuan","year":"2025","unstructured":"Tianyuan Wu, Lunxi Cao, Hanfeng Lu, Xiaoxiao Jiang, Yinghao Yu, Siran Yang, Guodong Yang, Jiamang Wang, Lin Qu, Liping Zhang, and Wei Wang. 2025. Adaptra: Straggler-resilient hybrid-parallel training with pipeline adaptation. arXiv preprint arXiv:2504.19232 (2025)."},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the USENIX Annual Technical Conference. 731-747","author":"Wu Tianyuan","year":"2025","unstructured":"Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wenchao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. 2025. GREYHOUND: Hunting fail-slows in hybrid-parallel training at scale. In Proceedings of the USENIX Annual Technical Conference. 731-747."},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the USENIX Conference on Operating Systems Design and Implementation. 595-610","author":"Xiao Wencong","year":"2018","unstructured":"Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, et al. 2018. Gandiva: Introspective cluster scheduling for deep learning. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation. 595-610."},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the USENIX Annual Technical Conference. 835-850","author":"Xiong Yifan","year":"2024","unstructured":"Yifan Xiong, Yuting Jiang, Ziyue Yang, Lei Qu, Guoshuai Zhao, Shuguang Liu, Dong Zhong, Boris Pinzur, Jie Zhang, Yang Wang, et al. 2024. SuperBench: Improving cloud AI infrastructure reliability with proactive validation. In Proceedings of the USENIX Annual Technical Conference. 835-850."},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the ACM Conference on SIGCOMM. 384-397","author":"Yan Siyu","year":"2021","unstructured":"Siyu Yan, Xiaoliang Wang, Xiaolong Zheng, Yinben Xia, Derui Liu, and Weishan Deng. 2021. ACC: Automatic ECN tuning for high-speed datacenter networks. In Proceedings of the ACM Conference on SIGCOMM. 384-397."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855741.1855744"},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the ACM International Conference on Emerging Networking Experiments and Technologies. 233-245","author":"Zhang Junxue","year":"2019","unstructured":"Junxue Zhang,Wei Bai, and Kai Chen. 2019. Enabling ECN for datacenter networks with RTT variations. In Proceedings of the ACM International Conference on Emerging Networking Experiments and Technologies. 233-245."},{"key":"e_1_2_1_77_1","volume-title":"Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 427-449","author":"Zhao Chenxingyu","year":"2025","unstructured":"Chenxingyu Zhao, Jaehong Min, Ming Liu, and Arvind Krishnamurthy. 2025. White-boxing RDMA with packetgranular software control. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 427-449."},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611569"},{"key":"e_1_2_1_79_1","volume-title":"Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557-62583."},{"key":"e_1_2_1_80_1","volume-title":"Jiaqi Gao, Yilong Zhao, Yongji Wu, Kaichao You, et al.","author":"Zhou Yang","year":"2025","unstructured":"Yang Zhou, Zhongjie Chen, Ziming Mao, ChonLam Lao, Shuo Yang, Pravein Govindan Kannan, Jiaqi Gao, Yilong Zhao, Yongji Wu, Kaichao You, et al. 2025. An extensible software transport layer for GPU networking. arXiv preprint arXiv:2504.17307 (2025)."},{"key":"e_1_2_1_81_1","unstructured":"Yibo Zhu. 2016. NS-3 simulator for RDMA over converged ethernet v2 (RoCEv2). (2016). https:\/\/github.com\/bobzhuyb\/ ns3-rdma."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2829988.2787484"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the ACM International Conference on Emerging Networking Experiments and Technologies. 313-327","author":"Zhu Yibo","year":"2016","unstructured":"Yibo Zhu, Monia Ghobadi, Vishal Misra, and Jitendra Padhye. 2016. ECN or delay: Lessons learnt from analysis of DCQCN and TIMELY. In Proceedings of the ACM International Conference on Emerging Networking Experiments and Technologies. 313-327."}],"container-title":["Proceedings of the ACM on Networking"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808669","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:47:34Z","timestamp":1780346854000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808669"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,1]]},"references-count":83,"journal-issue":{"issue":"CoNEXT2","published-print":{"date-parts":[[2026,6,1]]}},"alternative-id":["10.1145\/3808669"],"URL":"https:\/\/doi.org\/10.1145\/3808669","relation":{},"ISSN":["2834-5509"],"issn-type":[{"value":"2834-5509","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,1]]},"assertion":[{"value":"2026-06-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}