{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T16:53:37Z","timestamp":1783097617517,"version":"3.54.6"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2025,3,21]],"date-time":"2025-03-21T00:00:00Z","timestamp":1742515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62225205, 92055213, 62302160"],"award-info":[{"award-number":["62225205, 92055213, 62302160"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Science and Technology Program of Changsha","award":["kh2301011"],"award-info":[{"award-number":["kh2301011"]}]},{"DOI":"10.13039\/501100017607","name":"Shenzhen Basic Research Project","doi-asserted-by":"crossref","award":["JCYJ20210324140002006"],"award-info":[{"award-number":["JCYJ20210324140002006"]}],"id":[{"id":"10.13039\/501100017607","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004735","name":"Natural Science Foundation of Hunan Province","doi-asserted-by":"crossref","award":["2024JJ6154"],"award-info":[{"award-number":["2024JJ6154"]}],"id":[{"id":"10.13039\/501100004735","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>Ring all-reduce is currently the most commonly used collective communication technique in the fields of data parallel and distributed computing. It consists of three phases: communication establishment, data transmission, and data processing at each step. However, this method may suffer from increased communication latency as the number of computation nodes increases, excessive communication steps and data processing procedures can lead to insufficient bandwidth utilization.<\/jats:p>\n          <jats:p>To address this issue, this article proposes an Interleaved Bidirectional Ring (IBing) all-reduce method, which uses specially crafted communication operations to improve communication efficiency by reducing the effects of both communication establishment and data processing time. IBing reduces the number of communication steps by half compared to the Ring all-reduce. The results of extensive experiments indicate that the proposed IBing design can reduce total communication consumption by an average of 8.49% and up to 49.73%.<\/jats:p>","DOI":"10.1145\/3711818","type":"journal-article","created":{"date-parts":[[2025,1,9]],"date-time":"2025-01-09T03:44:37Z","timestamp":1736394277000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["IBing: An Efficient Interleaved Bidirectional Ring All-Reduce Algorithm for Gradient Synchronization"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-0530-0561","authenticated-orcid":false,"given":"Ruixing","family":"Zong","sequence":"first","affiliation":[{"name":"College of Computer Science and Electronic Engineering, Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0364-3568","authenticated-orcid":false,"given":"Jiapeng","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Electronic Engineering, Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9081-8153","authenticated-orcid":false,"given":"Zhuo","family":"Tang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Electronic Engineering, Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2635-7716","authenticated-orcid":false,"given":"Kenli","family":"Li","sequence":"additional","affiliation":[{"name":"Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,3,21]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Baidu Research. 2017. baidu-allreduce. Retrieved January 15 2025 from https:\/\/github.com\/baidu-research\/baidu-allreduce\/"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM41043.2020.9155446"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/LATINCOM.2016.7811594"},{"key":"e_1_3_1_5_2","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D. Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 1 (2020) 1877\u20131901."},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385414"},{"key":"e_1_3_1_7_2","unstructured":"H. Mikami H. Suganuma Y. Tanaka et\u00a0al. Massively distributed SGD: ImageNet\/ResNet-50 training in a flash. arXiv:1811.05233. Retrieved from https:\/\/arxiv.org\/abs\/1811.05233"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3572848.3577391"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30218-6_19"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_11_2","unstructured":"Xianyan Jia Shutao Song Wei He Yangzihao Wang Haidong Rong Feihu Zhou Liqiang Xie Zhenyu Guo Yuanzhou Yang Liwei Yu Tiegang Chen Guangxiao Hu Shaohuai Shi and Xiaowen Chu. 2018. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv:1807.11205. Retrieved from https:\/\/arxiv.org\/abs\/1807.11205"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2020.3003307"},{"issue":"11","key":"e_1_3_1_13_2","first-page":"2867","article-title":"Energy-efficient stochastic task scheduling on heterogeneous computing systems","volume":"25","author":"Li Kenli","year":"2013","unstructured":"Kenli Li, Xiaoyong Tang, and Keqin Li. 2013. Energy-efficient stochastic task scheduling on heterogeneous computing systems. IEEE Transactions on Parallel and Distributed Systems 25, 11 (2013), 2867\u20132876.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503221.3508399"},{"key":"e_1_3_1_15_2","unstructured":"Yujun Lin Song Han Huizi Mao Yu Wang and William J. Dally. 2018. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv:1712.01887. Retrieved from https:\/\/arxiv.org\/abs\/1712.01887"},{"issue":"1","key":"e_1_3_1_16_2","first-page":"14","article-title":"A new service mechanism for profit optimizations of a cloud provider and its users","volume":"9","author":"Liu Chubo","year":"2017","unstructured":"Chubo Liu, Kenli Li, Keqin Li, and Rajkumar Buyya. 2017. A new service mechanism for profit optimizations of a cloud provider and its users. IEEE Transactions on Cloud Computing 9, 1 (2017), 14\u201326.","journal-title":"IEEE Transactions on Cloud Computing"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672249"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s42514-022-00095-y"},{"key":"e_1_3_1_19_2","unstructured":"NVIDIA. 2017. Retrieved January 15 2025 from https:\/\/github.com\/nvidia\/nccl\/"},{"key":"e_1_3_1_20_2","first-page":"307","volume-title":"Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC 20)","author":"Park Jay H.","year":"2020","unstructured":"Jay H. Park, Gyeongchan Yun, M. Yi Chang, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, and Young-ri Choi. 2020. \\(\\lbrace\\) HetPipe \\(\\rbrace\\) : Enabling large \\(\\lbrace\\) DNN \\(\\rbrace\\) training on (whimpy) heterogeneous \\(\\lbrace\\) GPU \\(\\rbrace\\) clusters through integration of pipelined model parallelism and data parallelism. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC 20). 307\u2013321."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/505202.505215"},{"key":"e_1_3_1_22_2","first-page":"593","volume-title":"Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023, Boston, MA, April 17-19, 2023","author":"Shah Aashaka","year":"2023","unstructured":"Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. 2023. TACCL: Guiding collective algorithm synthesis using communication sketches. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023, Boston, MA, April 17-19, 2023. Mahesh Balakrishnan and Manya Ghobadi (Eds.), USENIX Association, 593\u2013612. Retrieved from https:\/\/www.usenix.org\/conference\/nsdi23\/presentation\/shah"},{"key":"e_1_3_1_23_2","first-page":"593","volume-title":"Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Shah Aashaka","year":"2023","unstructured":"Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. \\(\\lbrace\\) TACCL \\(\\rbrace\\) : Guiding collective algorithm synthesis using communication sketches. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 593\u2013612."},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2019.00057"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-020-03590-7"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-022-00202-7"},{"issue":"2","key":"e_1_3_1_27_2","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1109\/TPDS.2022.3221821","article-title":"Distributed approaches to butterfly analysis on large dynamic bipartite graphs","volume":"34","author":"Weng Tongfeng","year":"2022","unstructured":"Tongfeng Weng, Xu Zhou, Kenli Li, Kian-Lee Tan, and Keqin Li. 2022. Distributed approaches to butterfly analysis on large dynamic bipartite graphs. IEEE Transactions on Parallel and Distributed Systems 34, 2 (2022), 431\u2013445.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2019.2907537"},{"key":"e_1_3_1_29_2","unstructured":"Chris Ying Sameer Kumar Dehao Chen Tao Wang and Youlong Cheng. 2018. Image classification at supercomputer scale. arXiv:1811.06992. Retrieved from https:\/\/arxiv.org\/abs\/1811.06992"},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","unstructured":"Chen Zhang Yu Xie Hang Bai Bin Yu Weihong Li and Yuan Gao. 2021. A survey on federated learning. Knowledge Based Systems 216 1 (2021) 106775.","DOI":"10.1016\/j.knosys.2021.106775"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM42981.2021.9488678"},{"issue":"11","key":"e_1_3_1_32_2","first-page":"2979","article-title":"An efficient parallel reinforcement learning approach to cross-layer defense mechanism in industrial control systems","volume":"33","author":"Zhong Kai","year":"2021","unstructured":"Kai Zhong, Zhibang Yang, Guoqing Xiao, Xingpei Li, Wangdong Yang, and Kenli Li. 2021. An efficient parallel reinforcement learning approach to cross-layer defense mechanism in industrial control systems. IEEE Transactions on Parallel and Distributed Systems 33, 11 (2021), 2979\u20132990.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711818","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711818","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:19:15Z","timestamp":1750295955000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711818"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,21]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3711818"],"URL":"https:\/\/doi.org\/10.1145\/3711818","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,21]]},"assertion":[{"value":"2024-07-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-13","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}