{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T07:14:17Z","timestamp":1782544457381,"version":"3.54.5"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T00:00:00Z","timestamp":1643932800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T00:00:00Z","timestamp":1643932800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001656","name":"Helmholtz-Gemeinschaft","doi-asserted-by":"publisher","award":["ZT-I-0003"],"award-info":[{"award-number":["ZT-I-0003"]}],"id":[{"id":"10.13039\/501100001656","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001656","name":"Helmholtz-Gemeinschaft","doi-asserted-by":"publisher","award":["Helmholtz AI platform"],"award-info":[{"award-number":["Helmholtz AI platform"]}],"id":[{"id":"10.13039\/501100001656","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001656","name":"Helmholtz-Gemeinschaft","doi-asserted-by":"publisher","award":["HAICORE@KIT"],"award-info":[{"award-number":["HAICORE@KIT"]}],"id":[{"id":"10.13039\/501100001656","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>With increasing data and model complexities, the time required to train neural networks has become prohibitively large. To address the exponential rise in training time, users are turning to data parallel neural networks (DPNN) and large-scale distributed resources on computer clusters. Current DPNN approaches implement the network parameter updates by synchronizing and averaging gradients across all processes with blocking communication operations after each forward-backward pass. This synchronization is the central algorithmic bottleneck. We introduce the distributed asynchronous and selective optimization (DASO) method, which leverages multi-GPU compute node architectures to accelerate network training while maintaining accuracy. DASO uses a hierarchical and asynchronous communication scheme comprised of node-local and global networks while adjusting the global synchronization rate during the learning process. We show that DASO yields a reduction in training time of up to 34% on classical and state-of-the-art networks, as compared to current optimized data parallel training methods.<\/jats:p>","DOI":"10.1186\/s40537-021-00556-1","type":"journal-article","created":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T06:13:54Z","timestamp":1643955234000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Accelerating neural network training with distributed asynchronous and selective optimization (DASO)"],"prefix":"10.1186","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8552-5153","authenticated-orcid":false,"given":"Daniel","family":"Coquelin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotte","family":"Debus","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"G\u00f6tz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabrice","family":"von der Lehr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Kahn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Siggel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Achim","family":"Streit","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,2,4]]},"reference":[{"key":"556_CR1","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770\u2013778 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"556_CR2","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention Is All You Need. [accessed on 2021-08-06] (2017). arXiv:1706.03762."},{"issue":"4","key":"556_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3320060","volume":"52","author":"T Ben-Nun","year":"2019","unstructured":"Ben-Nun T, Hoefler T. Demystifying parallel and distributed deep learning: an in-depth concurrency analysis. ACM Comput Survey. 2019;52(4):1\u201343. https:\/\/doi.org\/10.1145\/3320060.","journal-title":"ACM Comput Survey"},{"key":"556_CR4","unstructured":"Lee S, Purushwalkam S, Cogswell M, et al. Why M heads are better than one: Training a diverse ensemble of deep networks. CoRR (2015). arXiv:1511.06314."},{"key":"556_CR5","unstructured":"Krizhevsky A. One weird trick for parallelizing convolutional neural networks. CoRR abs\/1404.5997 (2014). arXiv:1404.5997."},{"key":"556_CR6","doi-asserted-by":"publisher","unstructured":"Geng J, Li D, Wang S. Elasticpipe: An efficient and dynamic model-parallel solution to dnn training. In: Proceedings of the 10th Workshop on Scientific Cloud Computing. ScienceCloud \u201919, pp. 5\u20139. Association for Computing Machinery, New York, NY, USA (2019). https:\/\/doi.org\/10.1145\/3322795.3331463","DOI":"10.1145\/3322795.3331463"},{"key":"556_CR7","doi-asserted-by":"publisher","unstructured":"Byungik\u00a0Ahn J. Neuron machine: Parallel and pipelined digital neurocomputing architecture. In: 2012 IEEE International Conference on Computational Intelligence and Cybernetics (CyberneticsCom), pp. 143\u2013147 (2012). https:\/\/doi.org\/10.1109\/CyberneticsCom.2012.6381635.","DOI":"10.1109\/CyberneticsCom.2012.6381635"},{"key":"556_CR8","doi-asserted-by":"publisher","unstructured":"Zickenheiner S, Wendt M, Klauer B, Waldschmidt K. Pipelining and parallel training of neural networks on distributed-memory multiprocessors. In: Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN\u201994). 1994;4:2052\u201320574. https:\/\/doi.org\/10.1109\/ICNN.1994.374529.","DOI":"10.1109\/ICNN.1994.374529"},{"key":"556_CR9","unstructured":"Huang Y, Cheng Y, Bapna A, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In: Wallach, H., Larochelle, H., Beygelzimer, A., et al. (eds.) Advances in Neural Information Processing Systems, vol. 32 (2019). https:\/\/proceedings.neurips.cc\/paper\/2019\/file\/093f65e080a295f8076b1c5722a46aa2-Paper.pdf."},{"key":"556_CR10","doi-asserted-by":"publisher","DOI":"10.1155\/2015\/297672","author":"Y Liu","year":"2015","unstructured":"Liu Y, Yang J, Huang Y, Xu L, Li S, Qi M. Mapreduce based parallel neural networks in enabling large scale machine learning. Intell Neurosc. 2015. https:\/\/doi.org\/10.1155\/2015\/297672.","journal-title":"Intell Neurosc"},{"key":"556_CR11","doi-asserted-by":"publisher","unstructured":"Keuper J, Preundt FJ. Distributed training of deep neural networks: Theoretical and practical limits of parallel scalability. In: 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), pp. 19\u201326 (2016). https:\/\/doi.org\/10.1109\/MLHPC.2016.006.","DOI":"10.1109\/MLHPC.2016.006"},{"key":"556_CR12","doi-asserted-by":"publisher","unstructured":"Dryden N, Moon T, Jacobs SA, Van\u00a0Essen B. Communication quantization for data-parallel training of deep neural networks. In: 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), pp. 1\u20138 (2016). https:\/\/doi.org\/10.1109\/MLHPC.2016.004.","DOI":"10.1109\/MLHPC.2016.004"},{"key":"556_CR13","unstructured":"Yamazaki M, Kasagi A, Tabuchi A, Honda T, et al. Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds. 2019. arXiv:1903.12650. Accessed 06 Aug 2021."},{"key":"556_CR14","unstructured":"Tao A, Sapra K, Catanzaro B. Hierarchical Multi-Scale Attention for Semantic Segmentation. 2020. arXiv:2005.10821. Accessed 06 Aug 2021."},{"key":"556_CR15","unstructured":"Sergeev A, Balso MD. Horovod: fast and easy distributed deep learning in TensorFlow. 2018. arXiv:1802.05799. Accessed 06 Aug 2021."},{"key":"556_CR16","unstructured":"Bottou L, Curtis FE, Nocedal J. Optimization Methods for Large-Scale Machine Learning. ArXiv; 2018. Accessed 06 Aug 2021."},{"key":"556_CR17","doi-asserted-by":"publisher","unstructured":"De Sa C, Feldman M, R\u00e9 C, Olukotun K. Understanding and optimizing asynchronous low-precision stochastic gradient descent. In: Proceedings of the 44th Annual International Symposium on Computer Architecture. ISCA \u201917, pp. 561\u2013574. Association for Computing Machinery, New York, NY, USA (2017). https:\/\/doi.org\/10.1145\/3079856.3080248.","DOI":"10.1145\/3079856.3080248"},{"key":"556_CR18","unstructured":"Lian X, Zhang W, Zhang C, Liu J. Asynchronous Decentralized Parallel Stochastic Gradient Descent. In: Proceedings of the 35th International Conference on Machine Learning (ICML); 2018, pp. 3043\u20133052."},{"key":"556_CR19","doi-asserted-by":"publisher","unstructured":"Zhang S, Zhang C, You Z, et al. Asynchronous Stochastic Gradient Descent for DNN Training. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 6660\u20136663. https:\/\/doi.org\/10.1109\/ICASSP.2013.6638950","DOI":"10.1109\/ICASSP.2013.6638950"},{"key":"556_CR20","unstructured":"Dutta S, Wang J, Joshi G. Slow and Stale Gradients Can Win the Race. 2020. arXiv:2003.10579. Accessed 06 Aug 2021."},{"key":"556_CR21","unstructured":"Zhang W, Gupta S, Lian X, Liu J. Staleness-aware Async-SGD for Distributed Deep Learning. 2016. arXiv:1511.05950. Accessed 06 Aug 2021"},{"key":"556_CR22","doi-asserted-by":"crossref","unstructured":"Bogoychev N, Junczys-Dowmunt M, Heafield K, Aji AF. Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation. 2018. arXiv:1808.08859. . Accessed 06 Aug 2021.","DOI":"10.18653\/v1\/D18-1332"},{"key":"556_CR23","unstructured":"Lin T, Stich SU, Jaggi M. Don\u2019t use large mini-batches, use local sgd. arXiv:1808.07217 (2020)"},{"key":"556_CR24","unstructured":"Paszke A, Gross S, Massa F, Lerer A, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Advances in Neural Information Processing Systems 32, pp. 8024\u20138035 (2019). http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"556_CR25","unstructured":"Abadi M, Agarwal A, Barham P, et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org, 2015. http:\/\/tensorflow.org\/. . Accessed 04 Aug 2021."},{"key":"556_CR26","unstructured":"Message Passing Interface Forum: MPI: A Message-Passing Interface Standard, Version 3.1, 2015. https:\/\/fs.hlrs.de\/projects\/par\/mpi\/\/mpi31\/"},{"key":"556_CR27","doi-asserted-by":"publisher","unstructured":"Ueno Y, Yokota R. Exhaustive Study of Hierarchical AllReduce Patterns for Large Messages Between GPUs. In: 2019 19th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 430\u2013439 (2019). https:\/\/doi.org\/10.1109\/CCGRID.2019.00057.","DOI":"10.1109\/CCGRID.2019.00057"},{"key":"556_CR28","unstructured":"Mikami H, Suganuma H, U Chupala P, et al. Massively Distributed SGD: ImageNet\/ResNet-50 Training in a Flash. 2018. arXiv:1811.05233. Accessed 06 Aug 2021."},{"key":"556_CR29","unstructured":"Clauset A. A Brief Primer on Probability Distributions. 2011. http:\/\/tuvalu.santafe.edu\/aaronc\/courses\/7000\/csci7000-001_2011_L0.pdf. Accessed 06 Aug 2021."},{"issue":"1","key":"556_CR30","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1109\/tpds.2019.2928289","volume":"31","author":"A Li","year":"2020","unstructured":"Li A, Song SL, Chen J, et al. Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. IEEE Trans Parallel Distrib Syst. 2020;31(1):94\u2013110. https:\/\/doi.org\/10.1109\/tpds.2019.2928289.","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"556_CR31","unstructured":"Alistarh D, Grubic D, Li J, Tomioka R, Vojnovic M. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding; 2017. arXiv:1610.02132. Accessed 06 Aug 2021."},{"key":"556_CR32","doi-asserted-by":"publisher","unstructured":"G\u00f6tz M, Debus C, Coquelin D, Krajsek K, Comito C, Knechtges P, Hagemeier B, Tarnawa M, Hanselmann S, Siggel M, Basermann A, Streit A. HeAT - a Distributed and GPU-accelerated Tensor Framework for Data Analytics. In: 2020 IEEE International Conference on Big Data (Big Data), pp. 276\u2013287 (2020). https:\/\/doi.org\/10.1109\/BigData50022.2020.9378050","DOI":"10.1109\/BigData50022.2020.9378050"},{"key":"556_CR33","unstructured":"Li S, Zhao Y, Varma R, et al. PyTorch Distributed: Experiences on Accelerating Data Parallel Training; 2020. arXiv:2006.15704. Accessed 06 Aug 2021."},{"key":"556_CR34","doi-asserted-by":"publisher","first-page":"135","DOI":"10.17815\/jlsrf-5-171","volume":"5","author":"D Krause","year":"2019","unstructured":"Krause D. JUWELS: Modular Tier-0\/1 Supercomputer at the J\u00fclich Supercomputing Centre. J Large-scale Res Facil. 2019;5:135. https:\/\/doi.org\/10.17815\/jlsrf-5-171.","journal-title":"J Large-scale Res Facil"},{"key":"556_CR35","unstructured":"NVIDIA: NVIDIA A100 TENSOR CORE GPU. (2021). NVIDIA. https:\/\/www.nvidia.com\/en-us\/data-center\/a100\/."},{"key":"556_CR36","doi-asserted-by":"publisher","unstructured":"Deng J, Dong W, Socher R, Li LJ, et al. ImageNet: A Large-scale Hierarchical Image Database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248\u2013255 (2009). https:\/\/doi.org\/10.1109\/CVPR.2009.5206848.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"556_CR37","unstructured":"NVIDIA Corporation: NVIDIA Data Loading Library (DALI); 2021. https:\/\/developer.nvidia.com\/DALI. Accessed 05 Aug 2021."},{"key":"556_CR38","unstructured":"Goyal P, Doll\u00e1r P, Girshick R, Noordhuis P, et al. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour; 2018. arXiv:1706.02677. Accessed 06 Aug 2021."},{"key":"556_CR39","unstructured":"Wu Y, Kirillov A, Massa F, et al. Detectron2. https:\/\/github.com\/facebookresearch\/detectron2; 2019."},{"key":"556_CR40","doi-asserted-by":"publisher","unstructured":"Cordts M, Omran M, Ramos S, et al. The Cityscapes Dataset for Semantic Urban Scene Understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213\u20133223 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.350.","DOI":"10.1109\/CVPR.2016.350"},{"key":"556_CR41","doi-asserted-by":"crossref","unstructured":"Rezatofighi H, Tsoi N, Gwak J, et al. Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression; 2019. arXiv:1902.09630. Accessed 06 Aug 2021.","DOI":"10.1109\/CVPR.2019.00075"},{"key":"556_CR42","unstructured":"Zhao S, Wang Y, Yang Z, Cai D. Region Mutual Information Loss for Semantic Segmentation; 2019. arXiv:1910.12037. Accessed 06 Aug 2021."},{"key":"556_CR43","doi-asserted-by":"publisher","unstructured":"Yoo AB, Jette MA, Grondona M. SLURM: Simple Linux Utility for Resource Management. In: Workshop on Job Scheduling Strategies for Parallel Processing, pp. 44\u201360 (2003). https:\/\/doi.org\/10.1007\/10968987_3.","DOI":"10.1007\/10968987_3"}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-021-00556-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-021-00556-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-021-00556-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,16]],"date-time":"2022-05-16T02:06:50Z","timestamp":1652666810000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-021-00556-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,4]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["556"],"URL":"https:\/\/doi.org\/10.1186\/s40537-021-00556-1","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-832355\/v1","asserted-by":"object"}]},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,4]]},"assertion":[{"value":"21 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 December 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 May 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The original publication was missing a funding note declaring funding enabled by Projekt DEAL.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"14"}}