{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T05:49:11Z","timestamp":1715320151535},"reference-count":37,"publisher":"Institute of Electronics, Information and Communications Engineers (IEICE)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IEICE Trans. Inf. & Syst."],"published-print":{"date-parts":[[2020,12,1]]},"DOI":"10.1587\/transinf.2020pap0007","type":"journal-article","created":{"date-parts":[[2020,11,30]],"date-time":"2020-11-30T22:20:04Z","timestamp":1606774804000},"page":"2444-2456","source":"Crossref","is-referenced-by-count":2,"title":["An Efficient Method for Training Deep Learning Networks Distributed"],"prefix":"10.1587","volume":"E103.D","author":[{"given":"Chenxu","family":"WANG","sequence":"first","affiliation":[{"name":"School of Computer Science, National University of Defense Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yutong","family":"LU","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou"},{"name":"School of Data and Computer Science, Sun Yat-sen University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiguang","family":"CHEN","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou"},{"name":"School of Data and Computer Science, Sun Yat-sen University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junnan","family":"LI","sequence":"additional","affiliation":[{"name":"School of Computer Science, National University of Defense Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"532","reference":[{"key":"1","doi-asserted-by":"crossref","unstructured":"[1] Z. Chen and X. Huang, \u201cEnd-to-end learning for lane keeping of self-driving cars,\u201d 2017 IEEE Intelligent Vehicles Symposium (IV), pp.1856-1860, 2017. 10.1109\/ivs.2017.7995975","DOI":"10.1109\/IVS.2017.7995975"},{"key":"2","doi-asserted-by":"crossref","unstructured":"[2] Y. Goldberg, \u201cA primer on neural network models for natural language processing,\u201d J. Artif. Int. Res., vol.57, no.1, pp.345-420, Sept. 2016. 10.1613\/jair.4992","DOI":"10.1613\/jair.4992"},{"key":"3","doi-asserted-by":"publisher","unstructured":"[3] Y. Song, E.-L. Tan, X. Jiang, J.-Z. Cheng, D. Ni, S. Chen, B. Lei, and T. Wang, \u201cAccurate cervical cell segmentation from overlapping clumps in Pap smear images,\u201d IEEE Trans. Med. Imag., vol.36, no.1, pp.288-300, Jan. 2017. 10.1109\/tmi.2016.2606380","DOI":"10.1109\/TMI.2016.2606380"},{"key":"4","unstructured":"[4] A. Krizhevsky, I. Sutskever, and G.E. Hinton, \u201cImagenet classification with deep convolutional neural networks,\u201d Advances in Neural Information Processing Systems 25, pp.1097-1105, Curran Associates, 2012."},{"key":"5","unstructured":"[5] K. Simonyan and A. Zisserman, \u201cVery deep convolutional networks for large-scale image recognition,\u201d arXiv 1409.1556, Sept. 2014."},{"key":"6","doi-asserted-by":"publisher","unstructured":"[6] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.C. Berg, and L. Fei-Fei, \u201cImageNet large scale visual recognition challenge,\u201d International Journal of Computer Vision (IJCV), vol.115, no.3, pp.211-252, 2015. 10.1007\/s11263-015-0816-y","DOI":"10.1007\/s11263-015-0816-y"},{"key":"7","unstructured":"[7] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Q.V. Le, and A.Y. Ng, \u201cLarge scale distributed deep networks,\u201d Advances in Neural Information Processing Systems 25, pp.1223-1231, Curran Associates, 2012."},{"key":"8","unstructured":"[8] F.N. Iandola, K. Ashraf, M.W. Moskewicz, and K. Keutzer, \u201cFirecaffe: Near-linear acceleration of deep neural network training on compute clusters,\u201d CoRR, vol.abs\/1511.00175, 2015."},{"key":"9","doi-asserted-by":"crossref","unstructured":"[9] Y. You, A. Bulu\u00e7, and J. Demmel, \u201cScaling deep learning on GPU and knights landing clusters,\u201d Proc. International Conference for High Performance Computing, Networking, Storage and Analysis, SC'17, New York, NY, USA, pp.9:1-9:12, ACM, 2017. 10.1145\/3126908.3126912","DOI":"10.1145\/3126908.3126912"},{"key":"10","unstructured":"[10] T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, \u201cProject adam: Building an efficient and scalable deep learning training system,\u201d 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pp.571-582, Jan. 2014."},{"key":"11","doi-asserted-by":"publisher","unstructured":"[11] E.P. Xing, Q. Ho, W. Dai, J.K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu, \u201cPetuum: A new platform for distributed machine learning on big data,\u201d IEEE Trans. Big Data, vol.1, no.2, pp.49-67, June 2015. 10.1109\/tbdata.2015.2472014","DOI":"10.1109\/TBDATA.2015.2472014"},{"key":"12","unstructured":"[12] A. Krizhevsky, \u201cOne weird trick for parallelizing convolutional neural networks,\u201d CoRR, abs\/1404.5997, April 2014."},{"key":"13","unstructured":"[13] Y. You, Z. Zhang, C. Hsieh, and J. Demmel, \u201c100-epoch imagenet training with alexnet in 24 minutes,\u201d CoRR, abs\/1709.05011, 2017."},{"key":"14","unstructured":"[14] T. Kurth, J. Zhang, N. Satish, E. Racah, I. Mitliagkas, M.M.A. Patwary, T. Malas, N. Sundaram, W. Bhimji, M. Smorkalov, J. Deslippe, M. Shiryaev, S. Sridharan, Prabhat, and P. Dubey, \u201cDeep learning at 15PF: Supervised and semi-supervised classification for scientific data,\u201d Proc. International Conference for High Performance Computing, Networking, Storage and Analysis, SC'17, New York, NY, USA, pp.7:1-7:11, ACM, 2017. 10.1145\/3126908.3126916"},{"key":"15","unstructured":"[15] Y. You, I. Gitman, and B. Ginsburg, \u201cScaling SGD batch size to 32k for imagenet training,\u201d CoRR, vol.abs\/1708.03888, 2017."},{"key":"16","unstructured":"[16] P. Goyal, P. Doll\u00e1r, R.B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, \u201cAccurate, large minibatch SGD: Training imagenet in 1 hour,\u201d CoRR, vol.abs\/1706.02677, 2017."},{"key":"17","doi-asserted-by":"crossref","unstructured":"[17] E. Georganas, S. Avancha, K. Banerjee, D. Kalamkar, G. Henry, H. Pabst, and A. Heinecke, \u201cAnatomy of high-performance deep learning convolutions on SIMD architectures,\u201d SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.830-841, 2018. 10.1109\/sc.2018.00069","DOI":"10.1109\/SC.2018.00069"},{"key":"18","doi-asserted-by":"crossref","unstructured":"[18] J. Fang, H. Fu, W. Zhao, B. Chen, W. Zheng, and G. Yang, \u201cswDNN: A library for accelerating deep learning applications on Sunway TaihuLight,\u201d 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp.615-624, May 2017. 10.1109\/ipdps.2017.20","DOI":"10.1109\/IPDPS.2017.20"},{"key":"19","doi-asserted-by":"crossref","unstructured":"[19] X. Chen, D.Z. Chen, Y. Han, and X.S. Hu, \u201cmoDNN: Memory optimal deep neural network training on graphics processing units,\u201d IEEE Trans. Parallel Distrib. Syst., vol.30, no.3, pp.646-661, March 2019. 10.1109\/tpds.2018.2866582","DOI":"10.1109\/TPDS.2018.2866582"},{"key":"20","doi-asserted-by":"crossref","unstructured":"[20] L. Li, J. Fang, H. Fu, J. Jiang, W. Zhao, C. He, X. You, and G. Yang, \u201cswCaffe: A parallel framework for accelerating deep learning applications on Sunway TaihuLight,\u201d 2018 IEEE International Conference on Cluster Computing (CLUSTER), pp.413-422, Sept. 2018. 10.1109\/cluster.2018.00087","DOI":"10.1109\/CLUSTER.2018.00087"},{"key":"21","unstructured":"[21] D. Das, S. Avancha, D. Mudigere, K. Vaidyanathan, S. Sridharan, D.D. Kalamkar, B. Kaul, and P. Dubey, \u201cDistributed deep learning using synchronous stochastic gradient descent,\u201d ArXiv, vol.abs\/1602.06709, 2016."},{"key":"22","unstructured":"[22] P.H. Jin, Q. Yuan, F.N. Iandola, and K. Keutzer, \u201cHow to scale distributed deep learning?,\u201d CoRR, vol.abs\/1611.04581, 2016."},{"key":"23","doi-asserted-by":"publisher","unstructured":"[23] W. Zhao, H. Fu, J. Fang, W. Zheng, L. Gan, and G. Yang, \u201cOptimizing convolutional neural networks on the Sunway TaihuLight supercomputer,\u201d ACM Trans. Archit. Code Optim., vol.15, no.1, pp.13:1-13:26, March 2018. 10.1145\/3177885","DOI":"10.1145\/3177885"},{"key":"24","doi-asserted-by":"crossref","unstructured":"[24] F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, \u201c1-bit stochastic gradient descent and application to data-parallel distributed training of speech DNNs,\u201d Interspeech 2014, Sept. 2014.","DOI":"10.21437\/Interspeech.2014-274"},{"key":"25","unstructured":"[25] C. Chen, J. Choi, D. Brand, A. Agrawal, W. Zhang, and K. Gopalakrishnan, \u201cAdacomp: Adaptive residual gradient compression for data-parallel distributed training,\u201d CoRR, vol.abs\/1712. 02679, 2017."},{"key":"26","unstructured":"[26] S. Han, H. Mao, and W.J. Dally, \u201cDeep compression: Compressing deep neural network with pruning, trained quantization and Huffman coding,\u201d CoRR, vol.abs\/1510.00149, 2015."},{"key":"27","unstructured":"[27] Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally, \u201cDeep gradient compression: Reducing the communication bandwidth for distributed training,\u201d 6th International Conference on Learning Representations, ICLR 2018, 2018."},{"key":"28","doi-asserted-by":"crossref","unstructured":"[28] S. Pumma, M. Si, W.-C. Feng, and P. Balaji, \u201cTowards scalable deep learning via I\/O analysis and optimization,\u201d 2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS), pp.223-230, Dec. 2017. 10.1109\/hpcc-smartcity-dss.2017.29","DOI":"10.1109\/HPCC-SmartCity-DSS.2017.29"},{"key":"29","doi-asserted-by":"crossref","unstructured":"[29] A.A. Awan, K. Hamidouche, J.M. Hashmi, and D.K. Panda, \u201cS-caffe: Co-designing MPI runtimes and caffe for scalable deep learning on modern GPU clusters,\u201d Proc. 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP'17, New York, NY, USA, pp.193-205, ACM, 2017. 10.1145\/3018743.3018769","DOI":"10.1145\/3018743.3018769"},{"key":"30","unstructured":"[30] J. Chen, R. Monga, S. Bengio, and R. J\u00f3zefowicz, \u201cRevisiting distributed synchronous SGD,\u201d CoRR, vol.abs\/1604.00981, 2016."},{"key":"31","unstructured":"[31] \u201cDeepbench,\u201d https:\/\/github.com\/baidu-research\/DeepBench, 2018 [online; accessed 11-April-2018]."},{"key":"32","doi-asserted-by":"publisher","unstructured":"[32] W. Gropp, E. Lusk, N. Doss, and A. Skjellum, \u201cA high-performance, portable implementation of the MPI message passing interface standard,\u201d Parallel Computing, vol.22, no.6, pp.789-828, 1996. 10.1016\/0167-8191(96)00024-5","DOI":"10.1016\/0167-8191(96)00024-5"},{"key":"33","doi-asserted-by":"crossref","unstructured":"[33] A. Geist, W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, W. Saphir, T. Skjellum, and M. Snir, \u201cMPI-2: Extending the message-passing interface,\u201d Euro-Par'96 Parallel Processing, ed. L. Boug\u00e9, P. Fraigniaud, A. Mignotte, and Y. Robert, Lecture Notes in Computer Science, vol.1123, pp.128-135, Springer, Berlin, Heidelberg, 1996. 10.1007\/3-540-61626-8_16","DOI":"10.1007\/3-540-61626-8_16"},{"key":"34","doi-asserted-by":"publisher","unstructured":"[34] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, \u201cGradient-based learning applied to document recognition,\u201d Proc. IEEE, vol.86, no.11, pp.2278-2324, Nov. 1998. 10.1109\/5.726791","DOI":"10.1109\/5.726791"},{"key":"35","unstructured":"[35] A. Krizhevsky, \u201cLearning multiple layers of features from tiny images,\u201d University of Toronto, May 2012."},{"key":"36","unstructured":"[36] K. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep residual learning for image recognition,\u201d CoRR, vol.abs\/1512.03385, 2015."},{"key":"37","doi-asserted-by":"crossref","unstructured":"[37] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, \u201cGoing deeper with convolutions,\u201d Computer Vision and Pattern Recognition (CVPR), 2015. 10.1109\/cvpr.2015.7298594","DOI":"10.1109\/CVPR.2015.7298594"}],"container-title":["IEICE Transactions on Information and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E103.D\/12\/E103.D_2020PAP0007\/_pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,1]],"date-time":"2022-12-01T12:58:49Z","timestamp":1669899529000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E103.D\/12\/E103.D_2020PAP0007\/_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,1]]},"references-count":37,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2020]]}},"URL":"https:\/\/doi.org\/10.1587\/transinf.2020pap0007","relation":{},"ISSN":["0916-8532","1745-1361"],"issn-type":[{"value":"0916-8532","type":"print"},{"value":"1745-1361","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,1]]}}}