{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:17:13Z","timestamp":1750220233130,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":16,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T00:00:00Z","timestamp":1655942400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,23]]},"DOI":"10.1145\/3546000.3546007","type":"proceedings-article","created":{"date-parts":[[2022,8,19]],"date-time":"2022-08-19T16:08:37Z","timestamp":1660925317000},"page":"44-53","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Modern Distributed Data-Parallel Large-Scale Pre-training Strategies For NLP models"],"prefix":"10.1145","author":[{"given":"Hao","family":"Bai","sequence":"first","affiliation":[{"name":"University of Illinois Urbana-Champaign, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,8,19]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467(2016).","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Ashish Agarwal , Paul Barham , Eugene Brevdo , Zhifeng Chen , Craig Citro , Greg\u00a0 S Corrado , Andy Davis , Jeffrey Dean , Matthieu Devin , 2016 . Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467(2016). Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg\u00a0S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467(2016)."},{"key":"e_1_3_2_2_2_1","volume-title":"12th USENIX symposium on operating systems design and implementation (OSDI 16)","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . {TensorFlow}: A System for {Large-Scale} Machine Learning . In 12th USENIX symposium on operating systems design and implementation (OSDI 16) . 265\u2013283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, 2016. {TensorFlow}: A System for {Large-Scale} Machine Learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16). 265\u2013283."},{"key":"e_1_3_2_2_3_1","volume-title":"Language models are few-shot learners. Advances in neural information processing systems 33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared\u00a0 D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , 2020. Language models are few-shot learners. Advances in neural information processing systems 33 ( 2020 ), 1877\u20131901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared\u00a0D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877\u20131901."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2019.10.004"},{"key":"e_1_3_2_2_5_1","volume-title":"Keynote GPU Technical Conference, Vol.\u00a03.","author":"Dean Jeff","year":"2015","unstructured":"Jeff Dean . 2015 . Large scale deep learning . In Keynote GPU Technical Conference, Vol.\u00a03. 2015. Jeff Dean. 2015. Large scale deep learning. In Keynote GPU Technical Conference, Vol.\u00a03. 2015."},{"key":"e_1_3_2_2_6_1","volume-title":"Large scale distributed deep networks. Advances in neural information processing systems 25","author":"Dean Jeffrey","year":"2012","unstructured":"Jeffrey Dean , Greg Corrado , Rajat Monga , Kai Chen , Matthieu Devin , Mark Mao , Marc\u2019aurelio Ranzato , Andrew Senior , Paul Tucker , Ke Yang , 2012. Large scale distributed deep networks. Advances in neural information processing systems 25 ( 2012 ). Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc\u2019aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, 2012. Large scale distributed deep networks. Advances in neural information processing systems 25 (2012)."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/7902.7903"},{"key":"e_1_3_2_2_8_1","volume-title":"Hal: Computer system for scalable deep learning. In Practice and Experience in Advanced Research Computing. 41\u201348.","author":"Kindratenko Volodymyr","year":"2020","unstructured":"Volodymyr Kindratenko , Dawei Mu , Yan Zhan , John Maloney , Sayed\u00a0Hadi Hashemi , Benjamin Rabe , Ke Xu , Roy Campbell , Jian Peng , and William Gropp . 2020 . Hal: Computer system for scalable deep learning. In Practice and Experience in Advanced Research Computing. 41\u201348. Volodymyr Kindratenko, Dawei Mu, Yan Zhan, John Maloney, Sayed\u00a0Hadi Hashemi, Benjamin Rabe, Ke Xu, Roy Campbell, Jian Peng, and William Gropp. 2020. Hal: Computer system for scalable deep learning. In Practice and Experience in Advanced Research Computing. 41\u201348."},{"key":"e_1_3_2_2_9_1","unstructured":"Mu Li Li Zhou Zichao Yang Aaron Li Fei Xia David\u00a0G Andersen and Alexander Smola. 2013. Parameter server for distributed machine learning. In Big learning NIPS workshop Vol.\u00a06. 2.  Mu Li Li Zhou Zichao Yang Aaron Li Fei Xia David\u00a0G Andersen and Alexander Smola. 2013. Parameter server for distributed machine learning. In Big learning NIPS workshop Vol.\u00a06. 2."},{"key":"e_1_3_2_2_10_1","unstructured":"Shen Li Yanli Zhao Rohan Varma Omkar Salpekar Pieter Noordhuis Teng Li Adam Paszke Jeff Smith Brian Vaughan Pritam Damania 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704(2020).  Shen Li Yanli Zhao Rohan Varma Omkar Salpekar Pieter Noordhuis Teng Li Adam Paszke Jeff Smith Brian Vaughan Pritam Damania 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704(2020)."},{"key":"e_1_3_2_2_11_1","volume-title":"Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , 2019 . Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2008.09.002"},{"key":"e_1_3_2_2_13_1","volume-title":"Language models are unsupervised multitask learners. OpenAI blog 1, 8","author":"Radford Alec","year":"2019","unstructured":"Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9."},{"key":"e_1_3_2_2_14_1","unstructured":"Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter\u00a0J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683(2019).  Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter\u00a0J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683(2019)."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Adam Roberts Colin Raffel and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model?arXiv preprint arXiv:2002.08910(2020).  Adam Roberts Colin Raffel and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model?arXiv preprint arXiv:2002.08910(2020).","DOI":"10.18653\/v1\/2020.emnlp-main.437"},{"key":"e_1_3_2_2_16_1","unstructured":"Alexander Sergeev and Mike Del\u00a0Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799(2018).  Alexander Sergeev and Mike Del\u00a0Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799(2018)."}],"event":{"name":"HP3C'22: 2022 6th International Conference on High Performance Compilation, Computing and Communications","acronym":"HP3C'22","location":"Jilin China"},"container-title":["Proceedings of the 6th International Conference on High Performance Compilation, Computing and Communications"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546000.3546007","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3546000.3546007","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:18Z","timestamp":1750188618000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546000.3546007"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,23]]},"references-count":16,"alternative-id":["10.1145\/3546000.3546007","10.1145\/3546000"],"URL":"https:\/\/doi.org\/10.1145\/3546000.3546007","relation":{},"subject":[],"published":{"date-parts":[[2022,6,23]]},"assertion":[{"value":"2022-08-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}