{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T16:36:27Z","timestamp":1777307787665,"version":"3.51.4"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"CoNEXT4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Netw."],"published-print":{"date-parts":[[2025,11,24]]},"abstract":"<jats:p>Distributed machine learning (DML), such as large language model (LLM) training, has become one of the most critical services in multi-tenant cloud computing. However, communication contention among concurrent DML jobs significantly degrades overall GPU utilization, leading to inefficient training cluster performance. Existing approaches either achieve high performance at the cost of long scheduling runtime or reduce scheduling time at the expense of poor performance. We present Harmonics, a novel two-tier scheduling framework that strikes a balance between scheduling latency and performance. It coordinates decisions between Local Schedulers and a lightweight Global Coordinator to enable scalable and adaptive scheduling. By combining rack-level epoch-based optimization with global coordination, Harmonics alleviates communication contention and improves resource efficiency. We implement and evaluate Harmonics on real distributed ML workloads running on a GPU testbed. Compared to state-of-the-art methods such as fair sharing, optimal scheduling, Crux, and Cassini, Harmonics reduces training time by up to 33% and communication time by up to 48%. Large-scale simulations show that it reduces scheduling time by up to 91\u00d7 while improving training time by 26% in large-cluster settings.<\/jats:p>","DOI":"10.1145\/3768985","type":"journal-article","created":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T17:09:56Z","timestamp":1764090596000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Harmonics: Scalable Collective Scheduling in Multi-Tenant GPU Clusters"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0633-404X","authenticated-orcid":false,"given":"Hossein","family":"Shafieirad","sequence":"first","affiliation":[{"name":"Huawei Technologies Canada, Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7833-8652","authenticated-orcid":false,"given":"Amir","family":"Shani","sequence":"additional","affiliation":[{"name":"Huawei Technologies Canada, Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3688-1594","authenticated-orcid":false,"given":"Manaf","family":"Bin-Yahya","sequence":"additional","affiliation":[{"name":"Huawei Technologies Canada, Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8058-4127","authenticated-orcid":false,"given":"Seyed Hossein","family":"Mortazavi","sequence":"additional","affiliation":[{"name":"Huawei Technologies Canada, Toronto, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5467-548X","authenticated-orcid":false,"given":"Geng","family":"Li","sequence":"additional","affiliation":[{"name":"China Telecom Cloud Computing Research Institute, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8918-9580","authenticated-orcid":false,"given":"Xinle","family":"Du","sequence":"additional","affiliation":[{"name":"Huawei Technologies China, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3919-3734","authenticated-orcid":false,"given":"Tao","family":"Su","sequence":"additional","affiliation":[{"name":"Huawei Technologies China, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1231-5539","authenticated-orcid":false,"given":"Wei","family":"Wang","sequence":"additional","affiliation":[{"name":"Huawei Technologies China, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4781-1876","authenticated-orcid":false,"given":"Jingbin","family":"Zhou","sequence":"additional","affiliation":[{"name":"Huawei Technologies China, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3783-4346","authenticated-orcid":false,"given":"Majid","family":"Ghaderi","sequence":"additional","affiliation":[{"name":"University of Calgary, Calgary, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,11,25]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proc. 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 593-612","author":"Aashaka","unstructured":"Aashaka Shah et al. 2023. TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches. In Proc. 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 593-612."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the ACM SIGCOMM 2024 Conference, 57-70","author":"Adithya","unstructured":"Adithya Gangidi et al. 2024. Rdma over ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference, 57-70."},{"key":"e_1_2_1_3_1","volume-title":"Pollux: Co-Adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)","author":"Aurick","unstructured":"Aurick Qiao et al. 2021. Pollux: Co-Adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)."},{"key":"e_1_2_1_4_1","volume-title":"Proc. 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 741-761","unstructured":"ChonLam Lao et al. 2021. ATP: In-Network Aggregation for Multi-tenant Learning. In Proc. 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 741-761."},{"key":"e_1_2_1_5_1","volume-title":"(NIPS '22)","author":"Dacheng","unstructured":"Dacheng Li et al. 2022. Amp: automatically finding model parallel strategies with heterogeneity awareness. In (NIPS '22) Article 480. Curran Associates Inc., New Orleans, LA, USA, 10 pages. isbn: 9781713871088."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740070.2626322"},{"key":"e_1_2_1_7_1","volume-title":"Proc. 3rd Machine Learning and Systems Conference (MLSys), 172-186","author":"G. Wang","year":"2020","unstructured":"G. Wang et al. 2020. Blink: Fast and Generic Collectives for Distributed ML. In Proc. 3rd Machine Learning and Systems Conference (MLSys), 172-186."},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Hossein Shafieirad et al. 2024. Bounded low latency via inverse reinforcement learning. IEEE Access.","DOI":"10.1109\/ACCESS.2024.3411073"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the ACM SIGCOMM 2024 Conference, 1-15","author":"Jiamin","unstructured":"Jiamin Cao et al. 2024. Crux: gpu-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, 1-15."},{"key":"e_1_2_1_10_1","volume-title":"GLOBECOM 2023-2023 IEEE Global Communications Conference. IEEE, 4674-4679","author":"Mahmoud","unstructured":"Mahmoud Bahnasy et al. 2023. Host-assisted transport layer in data centers using network-aware rate adjustment. In GLOBECOM 2023-2023 IEEE Global Communications Conference. IEEE, 4674-4679."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 2014 ACM conference on SIGCOMM, 443-454","author":"Mosharaf","unstructured":"Mosharaf Chowdhury et al. 2014. Efficient Coflow Scheduling with Varys. In Proceedings of the 2014 ACM conference on SIGCOMM, 443-454."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.12.017"},{"key":"e_1_2_1_13_1","first-page":"945","volume-title":"MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)","author":"Qizhen","unstructured":"Qizhen Weng et al. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), 945-960."},{"key":"e_1_2_1_14_1","volume-title":"Proc. 49th Annual International Symposium on Computer Architecture. ACM, 581-596","author":"Saeed","unstructured":"Saeed Rashidi et al. 2022. Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models. In Proc. 49th Annual International Symposium on Computer Architecture. ACM, 581-596."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 14th IEEE\/ACM International Conference on Utility and Cloud Computing, 1-10","author":"Hossein Seyed","unstructured":"Seyed Hossein Mortazavi et al. 2021. Accord: application-driven networking in the datacenter. In Proceedings of the 14th IEEE\/ACM International Conference on Utility and Cloud Computing, 1-10."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the ACM SIGCOMM Workshop on Network-Application Integration, 40-45","author":"Hossein Seyed","unstructured":"Seyed Hossein Mortazavi et al. 2022. Earlybird: automating application signalling for network application integration in datacenters. In Proceedings of the ACM SIGCOMM Workshop on Network-Application Integration, 40-45."},{"key":"e_1_2_1_17_1","volume-title":"International Conference on Machine Learning, 2397-2430","author":"Stella","unstructured":"Stella Biderman et al. 2023. Pythia: A Suite for Analyzing Large Language Models across Training and Scaling. In International Conference on Machine Learning, 2397-2430."},{"key":"e_1_2_1_18_1","first-page":"1403","volume-title":"CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Sudarsanan","unstructured":"Sudarsanan Rajasekaran et al. 2024. CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 1403-1420."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejco.2022.100031"},{"key":"e_1_2_1_20_1","first-page":"739","volume-title":"TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Weiyang","unstructured":"Weiyang Wang et al. 2023. TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 739-767."},{"key":"e_1_2_1_21_1","volume-title":"2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 283-294","author":"William","unstructured":"William Won et al. 2023. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 283-294."},{"key":"e_1_2_1_22_1","volume-title":"2024 57th IEEE\/ACM International Symposium on Microarchitecture (MICRO). IEEE, 856-870","author":"William","unstructured":"William Won et al. 2024. Tacos: topology-aware collective algorithm synthesizer for distributed machine learning. In 2024 57th IEEE\/ACM International Symposium on Microarchitecture (MICRO). IEEE, 856-870."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2025.111285"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672249"},{"key":"e_1_2_1_25_1","volume-title":"2015 IEEE Conference on Computer Communications (INFOCOM). IEEE, 424-432","author":"Yangming","unstructured":"Yangming Zhao et al. 2015. Rapier: integrating routing and scheduling for coflow-aware data center networks. In 2015 IEEE Conference on Computer Communications (INFOCOM). IEEE, 424-432."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Yanli Zhao et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277.","DOI":"10.14778\/3611540.3611569"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2829988.2787484"},{"key":"e_1_2_1_28_1","volume-title":"ACM SIGCOMM 2024 Conference (SIGCOMM).","author":"Yongji","unstructured":"Yongji Wu et al. 2024. MCCS: A Service-based Approach to Collective Communication for Multi-Tenant Cloud. ACM SIGCOMM 2024 Conference (SIGCOMM)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_2_1_30_1","volume-title":"Proc. 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). ACM, 62-75","author":"Z. Cai","year":"2021","unstructured":"Z. Cai et al. 2021. Synthesizing Optimal Collective Algorithms. In Proc. 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). ACM, 62-75."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the Eighteenth European Conference on Computer Systems, 639-653","author":"Zhaoyi","unstructured":"Zhaoyi Li et al. 2023. A2tp: Aggregator-Aware In-Network Aggregation for Multi-Tenant Learning. In Proceedings of the Eighteenth European Conference on Computer Systems, 639-653."},{"key":"e_1_2_1_32_1","unstructured":"GitHub Inc. 2024. GitHub Copilot. Accessed: 2025-10-03. GitHub Inc. https:\/\/github.com\/features\/copilot."},{"key":"e_1_2_1_33_1","volume-title":"Gurobi Optimizer Reference Manual. Version 11.0","author":"Optimization Gurobi","year":"2025","unstructured":"Gurobi Optimization, LLC. 2025. Gurobi Optimizer Reference Manual. Version 11.0; accessed 2025-02-21. Gurobi Optimization, LLC. https:\/\/www.gurobi.com."},{"key":"e_1_2_1_34_1","unstructured":"Microsoft. 2024. Microsoft 365. Accessed: 2025-10-03. Microsoft. https:\/\/www.microsoft.com\/microsoft-365."},{"key":"e_1_2_1_35_1","volume-title":"Accessed","author":"Team PyTorch","year":"2024","unstructured":"PyTorch Team. 2024. Distributed data parallel implementation. https:\/\/github.com\/pytorch\/pytorch\/blob\/main\/torch\/nn\/parallel\/data_parallel.py. Accessed: January 14, 2025. Meta (2024)."},{"key":"e_1_2_1_36_1","volume-title":"Accessed","author":"Team PyTorch","year":"2024","unstructured":"PyTorch Team. 2024. Fully sharded data parallel implementation. https:\/\/github.com\/pytorch\/pytorch\/blob\/main\/torch\/distributed\/fsdp\/fully_sharded_data_parallel.py. Accessed: January 14, 2025. Meta (2024)."},{"key":"e_1_2_1_37_1","volume-title":"Version 1.0. Tech. rep. Accessed 2025-09-29","author":"Ultra Ethernet Consortium","year":"2025","unstructured":"Ultra Ethernet Consortium. 2025. Ultra Ethernet Specification, Version 1.0. Tech. rep. Accessed 2025-09-29. Ultra Ethernet Consortium (June 2025). https:\/\/ultraethernet.org\/wp-content\/uploads\/sites\/20\/2025\/06\/UE-Specification-6.11.25.pdf."}],"container-title":["Proceedings of the ACM on Networking"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3768985","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T15:45:45Z","timestamp":1777304745000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3768985"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,24]]},"references-count":37,"journal-issue":{"issue":"CoNEXT4","published-print":{"date-parts":[[2025,11,24]]}},"alternative-id":["10.1145\/3768985"],"URL":"https:\/\/doi.org\/10.1145\/3768985","relation":{},"ISSN":["2834-5509"],"issn-type":[{"value":"2834-5509","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,24]]},"assertion":[{"value":"2025-11-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}