{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,24]],"date-time":"2025-08-24T01:31:14Z","timestamp":1755999074116,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":33,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,11,12]],"date-time":"2017-11-12T00:00:00Z","timestamp":1510444800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,11,12]]},"DOI":"10.1145\/3146347.3146353","type":"proceedings-article","created":{"date-parts":[[2017,10,31]],"date-time":"2017-10-31T12:31:37Z","timestamp":1509453097000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Towards Scalable Parallel Training of Deep Neural Networks"],"prefix":"10.1145","author":[{"given":"Sam Ad\u00e9","family":"Jacobs","sequence":"first","affiliation":[{"name":"Lawrence Livermore National Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nikoli","family":"Dryden","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, Lawrence Livermore National Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roger","family":"Pearce","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brian","family":"Van Essen","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,11,12]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http:\/\/tensorflow.org\/ Software available from tensorflow.org.  Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http:\/\/tensorflow.org\/ Software available from tensorflow.org."},{"key":"e_1_3_2_1_2_1","unstructured":"Ian Goodfellow Yoshua Bengio and Aaron Courville. 2016. Deep Learning. (2016). http:\/\/www.deeplearningbook.org MIT Press.  Ian Goodfellow Yoshua Bengio and Aaron Courville. 2016. Deep Learning. (2016). http:\/\/www.deeplearningbook.org MIT Press."},{"key":"e_1_3_2_1_3_1","first-page":"19","article-title":"Deep learning of representations for unsupervised and transfer learning","volume":"7","author":"Bengio Yoshua","year":"2012","unstructured":"Yoshua Bengio . 2012 . Deep learning of representations for unsupervised and transfer learning . Unsupervised and Transfer Learning Challenges in Machine Learning 7 (2012), 19 . Yoshua Bengio. 2012. Deep learning of representations for unsupervised and transfer learning. Unsupervised and Transfer Learning Challenges in Machine Learning 7 (2012), 19.","journal-title":"Unsupervised and Transfer Learning Challenges in Machine Learning"},{"key":"e_1_3_2_1_4_1","volume-title":"Fully Automated Learning Rate and Size Adjustment. In The Learning Workshop. Online. Extended Abstract.","author":"Breuel Thomas","year":"2010","unstructured":"Thomas Breuel and Faisal Shafait . 2010 . AutoMLP: Simple, Effective , Fully Automated Learning Rate and Size Adjustment. In The Learning Workshop. Online. Extended Abstract. Thomas Breuel and Faisal Shafait. 2010. AutoMLP: Simple, Effective, Fully Automated Learning Rate and Size Adjustment. In The Learning Workshop. Online. Extended Abstract."},{"key":"e_1_3_2_1_5_1","volume-title":"Revisiting Distributed Synchronous SGD. CoRR abs\/1604.00981","author":"Chen Jianmin","year":"2016","unstructured":"Jianmin Chen , Rajat Monga , Samy Bengio , and Rafal J\u00f3zefowicz . 2016. Revisiting Distributed Synchronous SGD. CoRR abs\/1604.00981 ( 2016 ). http:\/\/arxiv.org\/abs\/1604.00981 Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal J\u00f3zefowicz. 2016. Revisiting Distributed Synchronous SGD. CoRR abs\/1604.00981 (2016). http:\/\/arxiv.org\/abs\/1604.00981"},{"key":"e_1_3_2_1_6_1","unstructured":"M. Cho U. Finkler S. Kumar D. Kung V. Saxena and D. Sreedhar. 2017. PowerAI DDL. ArXiv e-prints (Aug. 2017). arXiv:cs.DC\/1708.02188  M. Cho U. Finkler S. Kumar D. Kung V. Saxena and D. Sreedhar. 2017. PowerAI DDL. ArXiv e-prints (Aug. 2017). arXiv:cs.DC\/1708.02188"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40763-5_51"},{"key":"e_1_3_2_1_8_1","volume-title":"International conference on artificial intelligence and statistics. 215--223","author":"Coates Adam","year":"2011","unstructured":"Adam Coates , Andrew Y Ng , and Honglak Lee . 2011 . An analysis of single-layer networks in unsupervised feature learning . In International conference on artificial intelligence and statistics. 215--223 . Adam Coates, Andrew Y Ng, and Honglak Lee. 2011. An analysis of single-layer networks in unsupervised feature learning. In International conference on artificial intelligence and statistics. 215--223."},{"key":"e_1_3_2_1_9_1","unstructured":"Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Andrew Senior Paul Tucker Ke Yang Quoc V Le etal 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems. 1223--1231.  Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Andrew Senior Paul Tucker Ke Yang Quoc V Le et al. 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems. 1223--1231."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/MLHPC.2016.004"},{"key":"e_1_3_2_1_11_1","volume-title":"Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.","author":"Goyal Priya","year":"2017","unstructured":"Priya Goyal , Piotr Dollar , Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017 . Accurate, Large Minibatch SGD : Training ImageNet in 1 Hour . arXiv preprint arXiv:1706.02677 (2017). Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677 (2017)."},{"key":"e_1_3_2_1_12_1","volume-title":"Ng","author":"Hannun Awni Y.","year":"2014","unstructured":"Awni Y. Hannun , Carl Case , Jared Casper , Bryan C. Catanzaro , Greg Diamos , Erich Elsen , Ryan Prenger , Sanjeev Satheesh , Shubho Sengupta , Adam Coates , and Andrew Y . Ng . 2014 . Deep Speech : Scaling up end-to-end speech recognition. CoRR abs\/1412.5567 (2014). http:\/\/arxiv.org\/abs\/1412.5567 Awni Y. Hannun, Carl Case, Jared Casper, Bryan C. Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014. Deep Speech: Scaling up end-to-end speech recognition. CoRR abs\/1412.5567 (2014). http:\/\/arxiv.org\/abs\/1412.5567"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_3_2_1_14_1","volume-title":"FireCaffe: Near-linear acceleration of deep neural network training on compute clusters. arXiv preprint arXiv:1511.00175","author":"Iandola Forrest N","year":"2015","unstructured":"Forrest N Iandola , Khalid Ashraf , Mattthew W Moskewicz , and Kurt Keutzer . 2015. FireCaffe: Near-linear acceleration of deep neural network training on compute clusters. arXiv preprint arXiv:1511.00175 ( 2015 ). Forrest N Iandola, Khalid Ashraf, Mattthew W Moskewicz, and Kurt Keutzer. 2015. FireCaffe: Near-linear acceleration of deep neural network training on compute clusters. arXiv preprint arXiv:1511.00175 (2015)."},{"key":"e_1_3_2_1_15_1","volume-title":"Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093","author":"Jia Yangqing","year":"2014","unstructured":"Yangqing Jia , Evan Shelhamer , Jeff Donahue , Sergey Karayev , Jonathan Long , Ross Girshick , Sergio Guadarrama , and Trevor Darrell . 2014 . Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014). Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014)."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.223"},{"key":"e_1_3_2_1_17_1","volume-title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv preprint arXiv:1609.04836","author":"Keskar Nitish Shirish","year":"2016","unstructured":"Nitish Shirish Keskar , Dheevatsa Mudigere , Jorge Nocedal , Mikhail Smelyanskiy , and Ping Tak Peter Tang . 2016. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv preprint arXiv:1609.04836 ( 2016 ). Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv preprint arXiv:1609.04836 (2016)."},{"volume-title":"d.]. CIFAR-10","author":"Krizhevsky Alex","key":"e_1_3_2_1_18_1","unstructured":"Alex Krizhevsky , Vinod Nair , and Geoffrey Hinton . [n. d.]. CIFAR-10 ( Canadian Institute for Advanced Research) . ([n. d.]). http:\/\/www.cs.toronto.edu\/~kriz\/cifar.html Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. [n. d.]. CIFAR-10 (Canadian Institute for Advanced Research). ([n. d.]). http:\/\/www.cs.toronto.edu\/~kriz\/cifar.html"},{"key":"e_1_3_2_1_19_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoff Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. 1097--1105.  Alex Krizhevsky Ilya Sutskever and Geoff Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. 1097--1105."},{"key":"e_1_3_2_1_20_1","unstructured":"Lawrence Livermore National Laboratory. 2016. Sierra. https:\/\/asc.llnl.gov\/coral-info. (2016).  Lawrence Livermore National Laboratory. 2016. Sierra. https:\/\/asc.llnl.gov\/coral-info. (2016)."},{"key":"e_1_3_2_1_21_1","unstructured":"Lawrence Livermore National Laboratory. 2017. Livermore Computing. https:\/\/hpc.llnl.gov\/hardware\/platforms. (2017).  Lawrence Livermore National Laboratory. 2017. Livermore Computing. https:\/\/hpc.llnl.gov\/hardware\/platforms. (2017)."},{"key":"e_1_3_2_1_22_1","volume-title":"In International Conference on Machine Learning","author":"Le Quoc V.","year":"2012","unstructured":"Quoc V. Le , Rajat Monga , Matthieu Devin , Kai Chen , Greg S. Corrado , Jeff Dean , and Andrew Y. Ng . 2012. Building high-level features using large scale unsupervised learning . In In International Conference on Machine Learning , 2012 . 103. Quoc V. Le, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, and Andrew Y. Ng. 2012. Building high-level features using large scale unsupervised learning. In In International Conference on Machine Learning, 2012. 103."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553453"},{"key":"e_1_3_2_1_24_1","volume-title":"Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv","author":"Lee Stefan","year":"2015","unstructured":"Stefan Lee , Senthil Purushwalkam , Michael Cogswell , David J. Crandall , and Dhruv Batra . 2015. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv ( 2015 ). http:\/\/arxiv.org\/abs\/1511.06314 Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. 2015. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv (2015). http:\/\/arxiv.org\/abs\/1511.06314"},{"key":"e_1_3_2_1_25_1","volume-title":"Damian Borth, Barry Chen, and Eric Wang.","author":"Ni Karl","year":"2015","unstructured":"Karl Ni , Roger Pearce , Kofi Boakye , Brian Van Essen , Damian Borth, Barry Chen, and Eric Wang. 2015 . Large-Scale Deep Learning on the YFCC100M Dataset . arXiv preprint arXiv:1502.03409 (2015). Karl Ni, Roger Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, Barry Chen, and Eric Wang. 2015. Large-Scale Deep Learning on the YFCC100M Dataset. arXiv preprint arXiv:1502.03409 (2015)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553486"},{"key":"e_1_3_2_1_27_1","volume-title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems. 693--701.","author":"Recht Benjamin","year":"2011","unstructured":"Benjamin Recht , Christopher Re , Stephen Wright , and Feng Niu . 2011 . Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems. 693--701. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems. 693--701."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Frank Seide Hao Fu Jasha Droppo Gang Li and Dong Yu. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs.. In INTERSPEECH. 1058--1062.  Frank Seide Hao Fu Jasha Droppo Gang Li and Dong Yu. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs.. In INTERSPEECH. 1058--1062.","DOI":"10.21437\/Interspeech.2014-274"},{"key":"e_1_3_2_1_29_1","first-page":"10","article-title":"Scalable distributed DNN training using commodity GPU cloud computing","volume":"7","author":"Strom Nikko","year":"2015","unstructured":"Nikko Strom . 2015 . Scalable distributed DNN training using commodity GPU cloud computing . In INTERSPEECH , Vol. 7. 10 . Nikko Strom. 2015. Scalable distributed DNN training using commodity GPU cloud computing. In INTERSPEECH, Vol. 7. 10.","journal-title":"INTERSPEECH"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Christian Szegedy Wei Liu Yangqing Jia Pierre Sermanet Scott Reed Dragomir Anguelov Dumitru Erhan Vincent Vanhoucke and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR). http:\/\/arxiv.org\/abs\/1409.4842  Christian Szegedy Wei Liu Yangqing Jia Pierre Sermanet Scott Reed Dragomir Anguelov Dumitru Erhan Vincent Vanhoucke and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR). http:\/\/arxiv.org\/abs\/1409.4842","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2834892.2834897"},{"volume-title":"Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments (MLHPC '15)","author":"Young Steven R.","key":"e_1_3_2_1_32_1","unstructured":"Steven R. Young , Derek C. Rose , Thomas P. Karnowski , Seung-Hwan Lim , and Robert M. Patton . 2015. Optimizing Deep Learning Hyper-parameters Through an Evolutionary Algorithm . In Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments (MLHPC '15) . ACM, New York, NY, USA, Article 4, 5 pages. https:\/\/doi.org\/10.1145\/2834892.2834896 10.1145\/2834892.2834896 Steven R. Young, Derek C. Rose, Thomas P. Karnowski, Seung-Hwan Lim, and Robert M. Patton. 2015. Optimizing Deep Learning Hyper-parameters Through an Evolutionary Algorithm. In Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments (MLHPC '15). ACM, New York, NY, USA, Article 4, 5 pages. https:\/\/doi.org\/10.1145\/2834892.2834896"},{"key":"e_1_3_2_1_33_1","volume-title":"Deep learning with Elastic Averaging SGD. CoRR abs\/1412.6651","author":"Zhang Sixin","year":"2014","unstructured":"Sixin Zhang , Anna Choromanska , and Yann LeCun . 2014. Deep learning with Elastic Averaging SGD. CoRR abs\/1412.6651 ( 2014 ). http:\/\/arxiv.org\/abs\/1412.6651 Sixin Zhang, Anna Choromanska, and Yann LeCun. 2014. Deep learning with Elastic Averaging SGD. CoRR abs\/1412.6651 (2014). http:\/\/arxiv.org\/abs\/1412.6651"}],"event":{"name":"SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis","sponsor":["SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing","IEEE CS"],"location":"Denver CO USA","acronym":"SC '17"},"container-title":["Proceedings of the Machine Learning on HPC Environments"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3146347.3146353","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3146347.3146353","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:33Z","timestamp":1750212813000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3146347.3146353"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,11,12]]},"references-count":33,"alternative-id":["10.1145\/3146347.3146353","10.1145\/3146347"],"URL":"https:\/\/doi.org\/10.1145\/3146347.3146353","relation":{},"subject":[],"published":{"date-parts":[[2017,11,12]]},"assertion":[{"value":"2017-11-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}