{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,11]],"date-time":"2026-01-11T02:46:10Z","timestamp":1768099570929,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":27,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,1,15]],"date-time":"2020-01-15T00:00:00Z","timestamp":1579046400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,1,15]]},"DOI":"10.1145\/3368474.3368498","type":"proceedings-article","created":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T13:31:42Z","timestamp":1576589502000},"page":"52-60","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Rethinking the Value of Asynchronous Solvers for Distributed Deep Learning"],"prefix":"10.1145","author":[{"given":"Arissa","family":"Wongpanich","sequence":"first","affiliation":[{"name":"Computer Science Division, UC Berkeley, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"You","sequence":"additional","affiliation":[{"name":"Computer Science Division, UC Berkeley, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Demmel","sequence":"additional","affiliation":[{"name":"Computer Science Division, UC Berkeley, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,1,15]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325","author":"Akiba T.","year":"2017","unstructured":"T. Akiba , S. Suzuki , and K. Fukuda . Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325 , 2017 . T. Akiba, S. Suzuki, and K. Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017."},{"key":"e_1_3_2_1_2_1","volume-title":"Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595","author":"Amodei D.","year":"2015","unstructured":"D. Amodei , R. Anubhai , E. Battenberg , C. Case , J. Casper , B. Catanzaro , J. Chen , M. Chrzanowski , A. Coates , G. Diamos , Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595 , 2015 . D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015."},{"key":"e_1_3_2_1_3_1","volume-title":"Feb","author":"Czarnecki W. M.","year":"2019","unstructured":"W. M. Czarnecki , R. Pascanu , S. Osindero , S. M. Jayakumar , G. Swirszcz , and M. Jaderberg . Distilling Policy Distillation. arXiv e-prints, page arXiv:1902.02186 , Feb 2019 . W. M. Czarnecki, R. Pascanu, S. Osindero, S. M. Jayakumar, G. Swirszcz, and M.Jaderberg. Distilling Policy Distillation. arXiv e-prints, page arXiv:1902.02186, Feb 2019."},{"key":"e_1_3_2_1_4_1","first-page":"1223","volume-title":"Advances in neural information processing systems","author":"Dean J.","year":"2012","unstructured":"J. Dean , G. Corrado , R. Monga , K. Chen , M. Devin , M. Mao , A. Senior , P. Tucker , K. Yang , Q. V. Le , Large scale distributed deep networks . In Advances in neural information processing systems , pages 1223 -- 1231 , 2012 . J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pages 1223--1231, 2012."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_1_6_1","volume-title":"Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677","author":"Goyal P.","year":"2017","unstructured":"P. Goyal , P. Doll\u00e1r , R. Girshick , P. Noordhuis , L. Wesolowski , A. Kyrola , A. Tulloch , Y. Jia , and K. He . Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017 . P. Goyal, P. Doll\u00e1r, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_1_8_1","volume-title":"Densely connected convolutional networks. arXiv preprint arXiv:1608.06993","author":"Huang G.","year":"2016","unstructured":"G. Huang , Z. Liu , K. Q. Weinberger , and L. van der Maaten . Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 , 2016 . G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.284"},{"key":"e_1_3_2_1_10_1","volume-title":"On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836","author":"Keskar N. S.","year":"2016","unstructured":"N. S. Keskar , D. Mudigere , J. Nocedal , M. Smelyanskiy , and P. T. P. Tang . On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016 . N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016."},{"key":"e_1_3_2_1_11_1","volume-title":"Learning multiple layers of features from tiny images","author":"Krizhevsky A.","year":"2009","unstructured":"A. Krizhevsky . Learning multiple layers of features from tiny images . 2009 . A. Krizhevsky. Learning multiple layers of features from tiny images. 2009."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/2984093.2984354"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_1_14_1","volume-title":"Imagenet\/resnet-50 training in 224 seconds. arXiv preprint arXiv:1811.05233","author":"Mikami H.","year":"2018","unstructured":"H. Mikami , H. Suganuma , Y. Tanaka , Y. Kageyama , Imagenet\/resnet-50 training in 224 seconds. arXiv preprint arXiv:1811.05233 , 2018 . H. Mikami, H. Suganuma, Y. Tanaka, Y. Kageyama, et al. Imagenet\/resnet-50 training in 224 seconds. arXiv preprint arXiv:1811.05233, 2018."},{"key":"e_1_3_2_1_15_1","volume-title":"Feb","author":"Mnih V.","year":"2016","unstructured":"V. Mnih , A. Puigdom\u00e8nech Badia , M. Mirza , A. Graves , T. P. Lillicrap , T. Harley , D. Silver , and K. Kavukcuoglu . Asynchronous Methods for Deep Reinforcement Learning. arXiv e-prints, page arXiv:1602.01783 , Feb 2016 . V. Mnih, A. Puigdom\u00e8nech Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. arXiv e-prints, page arXiv:1602.01783, Feb 2016."},{"key":"e_1_3_2_1_16_1","volume-title":"Jul","author":"Nair A.","year":"2015","unstructured":"A. Nair , P. Srinivasan , S. Blackwell , C. Alcicek , R. Fearon , A. De Maria , V. Panneershelvam , M. Suleyman , C. Beattie , S. Petersen , S. Legg , V. Mnih , K. Kavukcuoglu , and D. Silver . Massively Parallel Methods for Deep Reinforcement Learning. arXiv e-prints, page arXiv:1507.04296 , Jul 2015 . A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver. Massively Parallel Methods for Deep Reinforcement Learning. arXiv e-prints, page arXiv:1507.04296, Jul 2015."},{"key":"e_1_3_2_1_17_1","volume-title":"Hogwild!: A lock-free approach to parallelizing stochastic gradient descent","author":"Niu F.","year":"2011","unstructured":"F. Niu , B. Recht , C. R\u00e9 , and S. J. Wright . Hogwild!: A lock-free approach to parallelizing stochastic gradient descent . 2011 . F. Niu, B. Recht, C. R\u00e9, and S. J. Wright. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. 2011."},{"key":"e_1_3_2_1_18_1","volume-title":"Language models are unsupervised multitask learners","author":"Radford A.","year":"2019","unstructured":"A. Radford , J. Wu , R. Child , D. Luan , D. Amodei , and I. Sutskever . Language models are unsupervised multitask learners . 2019 . A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019."},{"key":"e_1_3_2_1_19_1","first-page":"693","volume-title":"Advances in Neural Information Processing Systems","author":"Recht B.","year":"2011","unstructured":"B. Recht , C. Re , S. Wright , and F. Niu . Hogwild: A lock-free approach to parallelizing stochastic gradient descent . In Advances in Neural Information Processing Systems , pages 693 -- 701 , 2011 . B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 693--701, 2011."},{"key":"e_1_3_2_1_20_1","volume-title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815","author":"Silver D.","year":"2017","unstructured":"D. Silver , T. Hubert , J. Schrittwieser , I. Antonoglou , M. Lai , A. Guez , M. Lanctot , L. Sifre , D. Kumaran , T. Graepel , Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017 . D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_2_1_22_1","volume-title":"Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds. arXiv preprint arXiv:1903.12650","author":"Yamazaki M.","year":"2019","unstructured":"M. Yamazaki , A. Kasagi , A. Tabuchi , T. Honda , M. Miwa , N. Fukumoto , T. Tabaru , A. Ike , and K. Nakashima . Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds. arXiv preprint arXiv:1903.12650 , 2019 . M. Yamazaki, A. Kasagi, A. Tabuchi, T. Honda, M. Miwa, N. Fukumoto, T. Tabaru, A. Ike, and K. Nakashima. Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds. arXiv preprint arXiv:1903.12650, 2019."},{"key":"e_1_3_2_1_23_1","first-page":"9","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","author":"You Y.","unstructured":"Y. You , A. Bulu\u00e7 , and J. Demmel . Scaling deep learning on gpu and knights landing clusters . In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , page 9 . ACM, 2017. Y. You, A. Bulu\u00e7, and J. Demmel. Scaling deep learning on gpu and knights landing clusters. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, page 9. ACM, 2017."},{"key":"e_1_3_2_1_24_1","volume-title":"Aug","author":"You Y.","year":"2017","unstructured":"Y. You , I. Gitman , and B. Ginsburg . Large Batch Training of Convolutional Networks. arXiv e-prints, page arXiv:1708.03888 , Aug 2017 . Y. You, I. Gitman, and B. Ginsburg. Large Batch Training of Convolutional Networks. arXiv e-prints, page arXiv:1708.03888, Aug 2017."},{"key":"e_1_3_2_1_25_1","volume-title":"Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888","author":"You Y.","year":"2017","unstructured":"Y. You , I. Gitman , and B. Ginsburg . Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888 , 2017 . Y. You, I. Gitman, and B. Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017."},{"key":"e_1_3_2_1_26_1","volume-title":"Imagenet training in minutes. CoRR, abs\/1709.05011","author":"You Y.","year":"2017","unstructured":"Y. You , Z. Zhang , C. Hsieh , J. Demmel , and K. Keutzer . Imagenet training in minutes. CoRR, abs\/1709.05011 , 2017 . Y. You, Z. Zhang, C. Hsieh, J. Demmel, and K. Keutzer. Imagenet training in minutes. CoRR, abs\/1709.05011, 2017."},{"key":"e_1_3_2_1_27_1","first-page":"685","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang S.","year":"2015","unstructured":"S. Zhang , A. E. Choromanska , and Y. LeCun . Deep learning with elastic averaging sgd . In Advances in Neural Information Processing Systems , pages 685 -- 693 , 2015 . S. Zhang, A. E. Choromanska, and Y. LeCun. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, pages 685--693, 2015."}],"event":{"name":"HPCAsia2020: International Conference on High Performance Computing in Asia-Pacific Region","location":"Fukuoka Japan","acronym":"HPCAsia2020","sponsor":["IPSJ","SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing"]},"container-title":["Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368474.3368498","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368474.3368498","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:26Z","timestamp":1750197686000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368474.3368498"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,15]]},"references-count":27,"alternative-id":["10.1145\/3368474.3368498","10.1145\/3368474"],"URL":"https:\/\/doi.org\/10.1145\/3368474.3368498","relation":{},"subject":[],"published":{"date-parts":[[2020,1,15]]},"assertion":[{"value":"2020-01-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}