{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:40:10Z","timestamp":1755870010731,"version":"3.44.0"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,8]]},"DOI":"10.1145\/3721145.3725772","type":"proceedings-article","created":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T12:57:17Z","timestamp":1755867437000},"page":"355-367","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["CTCCL: Cost-Efficient Joint Device-Network Load Balancing for LLM Training in RoCE-based Intelligent Computing Network"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5182-0453","authenticated-orcid":false,"given":"Zhuotong","family":"Li","sequence":"first","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7558-8787","authenticated-orcid":false,"given":"Liang","family":"Xu","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1186-6225","authenticated-orcid":false,"given":"Ziqi","family":"Huang","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5941-0153","authenticated-orcid":false,"given":"Shuyun","family":"Qian","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9118-8883","authenticated-orcid":false,"given":"Hongwei","family":"Bu","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2756-8264","authenticated-orcid":false,"given":"Ming","family":"Yang","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4078-1696","authenticated-orcid":false,"given":"Mengyun","family":"Luan","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1172-944X","authenticated-orcid":false,"given":"Weiguo","family":"Chen","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8087-7088","authenticated-orcid":false,"given":"Xu","family":"Wen","sequence":"additional","affiliation":[{"name":"State Cloud, China Telecom, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,8,22]]},"reference":[{"key":"e_1_3_3_1_2_2","unstructured":"Vamsi Addanki Prateesh Goyal and Ilias Marinos. 2024. Challenging the Need for Packet Spraying in Large-Scale Distributed Training. arxiv:https:\/\/arXiv.org\/abs\/2407.00550\u00a0[cs.NI] https:\/\/arxiv.org\/abs\/2407.00550"},{"key":"e_1_3_3_1_3_2","doi-asserted-by":"crossref","unstructured":"Mohammad Alizadeh Tom Edsall Sarang Dharmapurikar Ramanan Vaidyanathan Kevin Chu Andy Fingerhut Vinh\u00a0The Lam Francis Matus Rong Pan Navindra Yadav and George Varghese. 2014. CONGA: distributed congestion-aware load balancing for datacenters. 44 4 (Aug. 2014) 503\u2013514. https:\/\/doi.org\/10.1145\/2740070.2626316","DOI":"10.1145\/2740070.2626316"},{"key":"e_1_3_3_1_4_2","doi-asserted-by":"crossref","unstructured":"Guo Chen Yuanwei Lu Bojie Li Kun Tan Yongqiang Xiong Peng Cheng Jiansong Zhang and Thomas Moscibroda. 2019. Mp-rdma: enabling rdma with multi-path transport in datacenters. IEEE\/ACM Transactions on Networking 27 6 (2019) 2308\u20132323.","DOI":"10.1109\/TNET.2019.2948917"},{"key":"e_1_3_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFCOM.2013.6567015"},{"key":"e_1_3_3_1_6_2","doi-asserted-by":"crossref","unstructured":"Jianbo Dong Shaochuang Wang Fei Feng Zheng Cao Heng Pan Lingbo Tang Pengcheng Li Hao Li Qianyuan Ran Yiqun Guo Shanyuan Gao Xin Long Jie Zhang Yong Li Zhisheng Xia Liuyihan Song Yingya Zhang Pan Pan Guohui Wang and Xiaowei Jiang. 2021. ACCL: Architecting Highly Scalable Distributed Training Systems With Highly Efficient Collective Communication Library. IEEE Micro 41 5 (2021) 85\u201392. https:\/\/doi.org\/10.1109\/MM.2021.3091475","DOI":"10.1109\/MM.2021.3091475"},{"key":"e_1_3_3_1_7_2","unstructured":"Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu Qinghao Hu Guoteng Wang Qizhen Weng Hang Yan Xingcheng Zhang Xipeng Qiu Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang and Peng Sun. 2024. Efficient Training of Large Language Models on Distributed Infrastructures: A Survey. arxiv:https:\/\/arXiv.org\/abs\/2407.20018\u00a0[cs.DC] https:\/\/arxiv.org\/abs\/2407.20018"},{"key":"e_1_3_3_1_8_2","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et\u00a0al. 2024. The Llama 3 Herd of Models. arxiv:https:\/\/arXiv.org\/abs\/2407.21783\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"crossref","unstructured":"Adithya Gangidi Rui Miao Shengbao Zheng Sai\u00a0Jayesh Bondu Guilherme Goes Hany Morsy Rohit Puri Mohammad Riftadi Ashmitha\u00a0Jeevaraj Shetty Jingyi Yang Shuqiang Zhang Mikel\u00a0Jimenez Fernandez Shashidhar Gandham and Hongyi Zeng. 2024. RDMA over Ethernet for Distributed Training at Meta Scale(ACM SIGCOMM \u201924). Association for Computing Machinery New York NY USA 57\u201370. https:\/\/doi.org\/10.1145\/3651890.3672233","DOI":"10.1145\/3651890.3672233"},{"key":"e_1_3_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3098822.3098839"},{"key":"e_1_3_3_1_11_2","doi-asserted-by":"crossref","unstructured":"Albert Greenberg James Hamilton David\u00a0A. Maltz and Parveen Patel. 2009. The cost of a cloud: research problems in data center networks. SIGCOMM Comput. Commun. Rev. 39 1 (Dec. 2009) 68\u201373. https:\/\/doi.org\/10.1145\/1496091.1496103","DOI":"10.1145\/1496091.1496103"},{"key":"e_1_3_3_1_12_2","first-page":"280","volume-title":"Smoky Mountains Computational Sciences and Engineering Conference","author":"Groves Taylor","year":"2021","unstructured":"Taylor Groves, Damian Hazen, Glenn Lockwood, and Nicholas\u00a0J Wright. 2021. Use It or Lose It: Cheap Compute Everywhere. In Smoky Mountains Computational Sciences and Engineering Conference. Springer, 280\u2013298."},{"key":"e_1_3_3_1_13_2","unstructured":"Jinbin Hu Houqiang Shen Xuchong Liu and Jin Wang. 2024. RDMA Transports in Datacenter Networks: Survey. IEEE Network (2024)."},{"key":"e_1_3_3_1_14_2","first-page":"745","volume-title":"21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Jiang Ziheng","year":"2024","unstructured":"Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et\u00a0al. 2024. { MegaScale} : Scaling large language model training to more than 10,000 { GPUs}. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 745\u2013760."},{"key":"e_1_3_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2890955.2890968"},{"key":"e_1_3_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3672198.3673799"},{"key":"e_1_3_3_1_17_2","first-page":"357","volume-title":"15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18)","author":"Lu Yuanwei","year":"2018","unstructured":"Yuanwei Lu, Guo Chen, Bojie Li, Kun Tan, Yongqiang Xiong, Peng Cheng, Jiansong Zhang, Enhong Chen, and Thomas Moscibroda. 2018. Multi-Path Transport for RDMA in Datacenters. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). USENIX Association, Renton, WA, 357\u2013371. https:\/\/www.usenix.org\/conference\/nsdi18\/presentation\/lu"},{"key":"e_1_3_3_1_18_2","unstructured":"Huimin Luo Jiao Zhang Mingxuan Yu Yongchen Pan Tian Pan and Tao Huang. 2024. SeqBalance: Congestion-Aware Load Balancing with no Reordering for RoCE. arxiv:https:\/\/arXiv.org\/abs\/2407.09808\u00a0[cs.NI] https:\/\/arxiv.org\/abs\/2407.09808"},{"key":"e_1_3_3_1_19_2","unstructured":"Meta. [n. d.]. Gloo: Collective communications library with various primitives for multi-machine training. https:\/\/github.com\/facebookincubator\/gloo?tab=readme-ov-file"},{"key":"e_1_3_3_1_20_2","volume-title":"Llama 2","year":"2024","unstructured":"Meta. 2024. Llama 2. Retrieved January 15, 2025 from https:\/\/huggingface.co\/meta-llama\/Llama-2-7b \""},{"key":"e_1_3_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230543.3230557"},{"key":"e_1_3_3_1_22_2","volume-title":"NVIDIA Collective Communications Library (NCCL)","year":"2016","unstructured":"Nvidia. 2016. NVIDIA Collective Communications Library (NCCL). Retrieved January 15, 2025 from https:\/\/developer.nvidia.com\/nccl"},{"key":"e_1_3_3_1_23_2","volume-title":"NCCL-Tests","year":"2017","unstructured":"Nvidia. 2017. NCCL-Tests. Retrieved January 15, 2025 from https:\/\/github.com\/NVIDIA\/nccl-tests\/blob\/master\/doc\/PERFORMANCE.md"},{"key":"e_1_3_3_1_24_2","volume-title":"NVIDIA Spectrum-X Networking Platform","year":"2023","unstructured":"Nvidia. 2023. NVIDIA Spectrum-X Networking Platform. Retrieved January 15, 2025 from https:\/\/www.nvidia.com\/en-us\/networking\/spectrumx\/"},{"key":"e_1_3_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672265"},{"key":"e_1_3_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544216.3544226"},{"key":"e_1_3_3_1_27_2","first-page":"18332","volume-title":"International conference on machine learning","author":"Rajbhandari Samyam","year":"2022","unstructured":"Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza\u00a0Yazdani Aminabadi, Ammar\u00a0Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning. PMLR, 18332\u201318346."},{"key":"e_1_3_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3406703"},{"key":"e_1_3_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Katharine Sanderson. 2023. GPT-4 is here: what scientists think. Nature 615 7954 (2023) 773.","DOI":"10.1038\/d41586-023-00816-5"},{"key":"e_1_3_3_1_30_2","unstructured":"Mohammad Shoeybi Mostofa Patwary Raul Puri Patrick LeGresley Jared Casper and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arxiv:https:\/\/arXiv.org\/abs\/1909.08053\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/1909.08053"},{"key":"e_1_3_3_1_31_2","unstructured":"Gemini Team Rohan Anil Sebastian Borgeaud Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew\u00a0M Dai Anja Hauth Katie Millican et\u00a0al. 2024. Gemini: A Family of Highly Capable Multimodal Models. arxiv:https:\/\/arXiv.org\/abs\/2312.11805\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"e_1_3_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3482898.3483363"},{"key":"e_1_3_3_1_33_2","volume-title":"Large Language Model Statistics And Numbers (2024)","author":"Uspenskyi Serhii","year":"2024","unstructured":"Serhii Uspenskyi. 2024. Large Language Model Statistics And Numbers (2024). https:\/\/springsapps.com\/knowledge\/large-language-model-statistics-and-numbers-2024#"},{"key":"e_1_3_3_1_34_2","first-page":"407","volume-title":"14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)","author":"Vanini Erico","year":"2017","unstructured":"Erico Vanini, Rong Pan, Mohammad Alizadeh, Parvin Taheri, and Tom Edsall. 2017. Let it flow: Resilient asymmetric load balancing with flowlet switching. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 407\u2013420."},{"key":"e_1_3_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-401673-6.00011-8"},{"key":"e_1_3_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1117\/12.3038265"},{"key":"e_1_3_3_1_37_2","first-page":"805","volume-title":"2022 USENIX Annual Technical Conference (USENIX ATC 22)","author":"Xu Yunhong","year":"2022","unstructured":"Yunhong Xu, Keqiang He, Rui Wang, Minlan Yu, Nick Duffield, Hassan Wassel, Shidong Zhang, Leon Poutievski, Junlan Zhou, and Amin Vahdat. 2022. Hashing Design in Modern Networks: Challenges and Mitigation Techniques. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 805\u2013818."},{"key":"e_1_3_3_1_38_2","first-page":"805","volume-title":"2022 USENIX Annual Technical Conference (USENIX ATC 22)","author":"Xu Yunhong","year":"2022","unstructured":"Yunhong Xu, Keqiang He, Rui Wang, Minlan Yu, Nick Duffield, Hassan Wassel, Shidong Zhang, Leon Poutievski, Junlan Zhou, and Amin Vahdat. 2022. Hashing Design in Modern Networks: Challenges and Mitigation Techniques. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association, Carlsbad, CA, 805\u2013818. https:\/\/www.usenix.org\/conference\/atc22\/presentation\/xu"},{"key":"e_1_3_3_1_39_2","doi-asserted-by":"crossref","unstructured":"Ling Zhang Xuefei Yang Zhenlong Wan Hang Liu Wei Gu Pingjing Liu Qilin Dai Shanwei Ye and Yingcheng Lin. 2024. A High-Performance RDMA NIC With Ultra-Highly Scalable Connections. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2024) 1\u20131. https:\/\/doi.org\/10.1109\/TCAD.2024.3514782","DOI":"10.1109\/TCAD.2024.3514782"},{"key":"e_1_3_3_1_40_2","first-page":"855","volume-title":"2021 USENIX Annual Technical Conference (USENIX ATC 21)","author":"Zhang Zhehui","year":"2021","unstructured":"Zhehui Zhang, Haiyang Zheng, Jiayao Hu, Xiangning Yu, Chenchen Qi, Xuemei Shi, and Guohui Wang. 2021. Hashing linearity enables relative path control in data centers. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 855\u2013862."},{"key":"e_1_3_3_1_41_2","first-page":"559","volume-title":"16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Zheng Lianmin","year":"2022","unstructured":"Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric\u00a0P. Xing, Joseph\u00a0E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 559\u2013578. https:\/\/www.usenix.org\/conference\/osdi22\/presentation\/zheng-lianmin"}],"event":{"name":"ICS '25: 2025 International Conference on Supercomputing","location":"Salt Lake City USA","acronym":"ICS '25","sponsor":["SIGARCH ACM Special Interest Group on Computer Architecture"]},"container-title":["Proceedings of the 39th ACM International Conference on Supercomputing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3721145.3725772","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:02:30Z","timestamp":1755867750000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3721145.3725772"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,8]]},"references-count":40,"alternative-id":["10.1145\/3721145.3725772","10.1145\/3721145"],"URL":"https:\/\/doi.org\/10.1145\/3721145.3725772","relation":{},"subject":[],"published":{"date-parts":[[2025,6,8]]},"assertion":[{"value":"2025-08-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}