{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T16:33:34Z","timestamp":1781109214767,"version":"3.54.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,8,7]],"date-time":"2021-08-07T00:00:00Z","timestamp":1628294400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,8,7]],"date-time":"2021-08-07T00:00:00Z","timestamp":1628294400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100014440","name":"Ministerio de Ciencia, Innovaci\u00f3n y Universidades","doi-asserted-by":"publisher","award":["TIN-2017-82972-R"],"award-info":[{"award-number":["TIN-2017-82972-R"]}],"id":[{"id":"10.13039\/100014440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100016028","name":"Agencia Valenciana de la Innovaci\u00f3n","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100016028","id-type":"DOI","asserted-by":"crossref"}]},{"name":"PRACE preparatory access","award":["2010PA5531"],"award-info":[{"award-number":["2010PA5531"]}]},{"DOI":"10.13039\/100014440","name":"Ministerio de Ciencia, Innovaci\u00f3n y Universidades","doi-asserted-by":"publisher","award":["FJC2019-039222-I"],"award-info":[{"award-number":["FJC2019-039222-I"]}],"id":[{"id":"10.13039\/100014440","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cluster Comput"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>TensorFlow (TF) is usually combined with the Horovod (HVD) workload distribution package to obtain a parallel tool to train deep neural network on clusters of computers. HVD in turn utilizes a blocking Allreduce primitive to share information among processes, combined with a communication thread to overlap communication with computation. In this work, we perform a thorough experimental analysis to expose (1) the importance of selecting the best algorithm in MPI libraries to realize the Allreduce operation; and (2) the performance acceleration that can be attained when replacing a blocking Allreduce with its non-blocking counterpart (while maintaining the blocking behaviour via the appropriate synchronization mechanism). Furthermore, (3) we explore the benefits of applying pipelining to the communication exchange, demonstrating that these improvements carry over to distributed training via TF+HVD. Finally, (4) we show that pipelining can also boost performance for applications that make heavy use of other collectives, such as Broadcast and Reduce-Scatter.<\/jats:p>","DOI":"10.1007\/s10586-021-03370-9","type":"journal-article","created":{"date-parts":[[2021,8,7]],"date-time":"2021-08-07T20:02:45Z","timestamp":1628366565000},"page":"3797-3813","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Accelerating distributed deep neural network training with pipelined MPI allreduce"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8576-8451","authenticated-orcid":false,"given":"Adri\u00e1n","family":"Castell\u00f3","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enrique S.","family":"Quintana-Ort\u00ed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9","family":"Duato","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,8,7]]},"reference":[{"key":"3370_CR1","unstructured":"Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et\u00a0al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265\u2013283 (2016)"},{"issue":"3","key":"3370_CR2","doi-asserted-by":"publisher","first-page":"1493","DOI":"10.1007\/s10586-017-1561-5","volume":"21","author":"I Alsmadi","year":"2018","unstructured":"Alsmadi, I., Khreishah, A., Dianxiang, X.: Network slicing to improve multicasting in hpc clusters. Clust. Comput. 21(3), 1493\u20131506 (2018)","journal-title":"Clust. Comput."},{"key":"3370_CR3","doi-asserted-by":"crossref","unstructured":"Awan, A.A., Bedorf, J., Chu, C.-H., Subramoni, H., Panda, D.K.: Scalable distributed DNN training using TensorFlow and CUDA-aware MPI: characterization, designs, and performance evaluation (2018). arXiv:1810.11112","DOI":"10.1109\/CCGRID.2019.00064"},{"key":"3370_CR4","doi-asserted-by":"crossref","unstructured":"Awan, A.A., Chu, C.-H., Subramoni, H., Panda, D.K.: Optimized broadcast for deep learning workloads on dense-GPU InfiniBand clusters: MPI or NCCL? In: Proceedings of the 25th European MPI Users\u2019 Group Meeting, pp. 1\u20139 (2018)","DOI":"10.1145\/3236367.3236381"},{"issue":"4","key":"3370_CR5","doi-asserted-by":"publisher","first-page":"65:1","DOI":"10.1145\/3320060","volume":"52","author":"T Ben-Nun","year":"2019","unstructured":"Ben-Nun, T., Hoefler, T.: Demystifying parallel and distributed deep learning: an in-depth concurrency analysis. ACM Comput. Surv. 52(4), 65:1\u201365:43 (2019)","journal-title":"ACM Comput. Surv."},{"key":"3370_CR6","doi-asserted-by":"crossref","unstructured":"Castell\u00f3, A., Catal\u00e1n, M., Dolz, M.F., Mestre, J.I., Quintana-Ort\u00ed, E.S., Duato, J.: Evaluation of MPI Allreduce for distributed training of convolutional neural networks. In: 29th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP) (2021)","DOI":"10.1109\/PDP52278.2021.00025"},{"key":"3370_CR7","doi-asserted-by":"crossref","unstructured":"Castell\u00f3, A., Dolz, M.F., Quintana-Ort\u00ed, E.S., Duato, J.: Analysis of model parallelism for distributed neural networks. In: Proceedings of the 26th European MPI Users\u2019 Group Meeting, EuroMPI \u201919, New York, NY, USA (2019). Association for Computing Machinery","DOI":"10.1145\/3343211.3343218"},{"key":"3370_CR8","doi-asserted-by":"crossref","unstructured":"Castell\u00f3, A., Dolz, M.F., Quintana-Ort\u00ed, E.S., Duato, J.: Theoretical scalability analysis of distributed deep convolutional neural networks. In: 2019 19th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 534\u2013541 (2019)","DOI":"10.1109\/CCGRID.2019.00068"},{"issue":"13","key":"3370_CR9","doi-asserted-by":"publisher","first-page":"1749","DOI":"10.1002\/cpe.1206","volume":"19","author":"E Chan","year":"2007","unstructured":"Chan, E., Heimlich, M., Purkayastha, A., van de Geijn, R.: Collective communication: theory, practice, and experience. Concurr. Comput. 19(13), 1749\u20131783 (2007)","journal-title":"Concurr. Comput."},{"key":"3370_CR10","doi-asserted-by":"crossref","unstructured":"Clarke, L., Glendinning, I., Hempel, R.: The MPI message passing interface standard. In: Programming Environments for Massively Parallel Distributed Systems, pp. 213\u2013218. Springer, Berlin (1994)","DOI":"10.1007\/978-3-0348-8534-8_21"},{"key":"3370_CR11","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248\u2013255 (2009)","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"2","key":"3370_CR12","doi-asserted-by":"publisher","first-page":"713","DOI":"10.1007\/s11227-016-1779-7","volume":"73","author":"K Hasanov","year":"2017","unstructured":"Hasanov, K., Lastovetsky, A.: Hierarchical redesign of classic MPI reduction algorithms. J. Supercomput. 73(2), 713\u2013725 (2017)","journal-title":"J. Supercomput."},{"key":"3370_CR13","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"3370_CR14","unstructured":"Google Inc. Tensorflow benchmarks. https:\/\/github.com\/tensorflow\/benchmarks"},{"key":"3370_CR15","unstructured":"Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., Hoefler, T.: Data movement is all you need: a case study on optimizing transformers (2020). arXiv:2007.00072"},{"key":"3370_CR16","doi-asserted-by":"crossref","unstructured":"Kang, Q., Tr\u00e4ff, J.L., Al-Bahrani, R., Agrawal, A., Choudhary, A., Liao, W.-k.: Full-duplex inter-group all-to-all broadcast algorithms with optimal bandwidth. In: Proceedings of the 25th European MPI Users\u2019 Group Meeting, pp. 1\u201310 (2018)","DOI":"10.1145\/3236367.3236374"},{"key":"3370_CR17","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1016\/j.parco.2019.04.015","volume":"85","author":"Q Kang","year":"2019","unstructured":"Kang, Q., Tr\u00e4ff, J.L., Al-Bahrani, R., Agrawal, A., Choudhary, A., Liao, W-k: Scalable algorithms for MPI intergroup Allgather and Allgatherv. Parallel Comput. 85, 220\u2013230 (2019)","journal-title":"Parallel Comput."},{"issue":"3","key":"3370_CR18","doi-asserted-by":"publisher","first-page":"2287","DOI":"10.1007\/s10586-020-03144-9","volume":"23","author":"Y Kim","year":"2020","unstructured":"Kim, Y., Choi, H., Lee, J., Kim, J.-S., Jei, H., Roh, H.: Towards an optimized distributed deep learning framework for a heterogeneous multi-gpu cluster. Clust. Comput. 23(3), 2287\u20132300 (2020)","journal-title":"Clust. Comput."},{"key":"3370_CR19","unstructured":"Krizhevsky, A., Hinton, G., et\u00a0al.: Learning multiple layers of features from tiny images. Technical report, Department of Computer Sciences, University of Toronto (2009)"},{"key":"3370_CR20","unstructured":"Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS\u201912, pp. 1097\u20131105, Curran Associates Inc. (2012)"},{"key":"3370_CR21","doi-asserted-by":"crossref","unstructured":"Kurnosov, M., Tokmasheva, E.: Shared memory based mpi broadcast algorithms for numa systems. In: Russian Supercomputing Days, pp. 473\u2013485. Springer, Berlin (2020)","DOI":"10.1007\/978-3-030-64616-5_41"},{"issue":"4","key":"3370_CR22","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.1007\/s10586-014-0361-4","volume":"17","author":"S Li","year":"2014","unstructured":"Li, S., Hoefler, T., Chungjin, H., Snir, M.: Improved MPI collectives for MPI processes in shared address spaces. Clust. Comput. 17(4), 1139\u20131155 (2014)","journal-title":"Clust. Comput."},{"key":"3370_CR23","unstructured":"Nguyen, T.T., Wahib, M., Takano, R.: Hierarchical distributed-memory multi-leader MPI\\_Allreduce for deep learning workloads. In: 2018 Sixth International Symposium on Computing and Networking Workshops (CANDARW), pp. 216\u2013222. IEEE (2018)"},{"key":"3370_CR24","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et\u00a0al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8026\u20138037 (2019)"},{"key":"3370_CR25","unstructured":"Sergeev, A., Del Balso, M.: Horovod: fast and easy distributed deep learning in TensorFlow (2018). arXiv:1802.05799"},{"key":"3370_CR26","doi-asserted-by":"crossref","unstructured":"Shalf, J.: HPC interconnects at the end of Moore\u2019s Law. In: 2019 Optical Fiber Communications Conference and Exhibition (OFC), pp. 1\u20133 (2019)","DOI":"10.1364\/OFC.2019.Th3A.1"},{"key":"3370_CR27","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556"},{"key":"3370_CR28","volume-title":"MPI: The Complete Reference","author":"M Snir","year":"1996","unstructured":"Snir, M., Otto, S.W., Huss-Lederman, S., Walker, D.W., Dongarra, J.: MPI: The Complete Reference. The MIT Press, New York (1996)"},{"issue":"12","key":"3370_CR29","doi-asserted-by":"publisher","first-page":"2295","DOI":"10.1109\/JPROC.2017.2761740","volume":"105","author":"V Sze","year":"2017","unstructured":"Sze, V., Chen, Y.-H., Yang, T.-J., Emer, J.S.: Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295\u20132329 (2017)","journal-title":"Proc. IEEE"},{"issue":"1","key":"3370_CR30","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1177\/1094342005051521","volume":"19","author":"R Thakur","year":"2005","unstructured":"Thakur, R., Rabenseifner, R., Gropp, W.: Optimization of collective communication operations in MPICH. Int. J. High Perform. Comput. Appl. 19(1), 49\u201366 (2005)","journal-title":"Int. J. High Perform. Comput. Appl."},{"key":"3370_CR31","unstructured":"Worringen, J.: Pipelining and overlapping for MPI collective operations. In: 28th Annual IEEE International Conference on Local Computer Networks, 2003, pp. 548\u2013557. IEEE (2003)"},{"key":"3370_CR32","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Wang, L., Wu, W., Bosilca, G., Vuduc, R., Ye, J., Tang, W., Xu, Z.: Efficient communications in training large scale neural networks. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 110\u2013116 (2017)","DOI":"10.1145\/3126686.3126749"},{"key":"3370_CR33","doi-asserted-by":"crossref","unstructured":"Zhong, D., Cao, Q., Bosilca, G., Dongarra, J.: Using advanced vector extensions avx-512 for mpi reductions. In: 27th European MPI Users\u2019 Group Meeting, pp. 1\u201310 (2020)","DOI":"10.1145\/3416315.3416316"}],"container-title":["Cluster Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03370-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10586-021-03370-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03370-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T15:14:49Z","timestamp":1637334889000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10586-021-03370-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,7]]},"references-count":33,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["3370"],"URL":"https:\/\/doi.org\/10.1007\/s10586-021-03370-9","relation":{},"ISSN":["1386-7857","1573-7543"],"issn-type":[{"value":"1386-7857","type":"print"},{"value":"1573-7543","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,7]]},"assertion":[{"value":"25 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 June 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2021","order":5,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":6,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The original online version of this article was revised: Funding note has been added.","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The pipelined codes may be found at .","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}}]}}