{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T19:21:44Z","timestamp":1793128904677,"version":"4.5.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:p>Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) models. However, how to train these models over multiple GPUs efficiently is still challenging due to a large number of parallelism choices. Existing DL systems either rely on manual efforts to make distributed training plans or apply parallelism combinations within a very limited search space. In this approach, we propose Galvatron, a new system framework that incorporates multiple popular parallelism dimensions and automatically finds the most efficient hybrid parallelism strategy. To better explore such a rarely huge search space, we 1) involve a decision tree to make decomposition and pruning based on some reasonable intuitions, and then 2) design a dynamic programming search algorithm to generate the optimal plan. Evaluations on four representative Transformer workloads show that Galvatron could perform automatically distributed training with different GPU memory budgets. Among all evaluated scenarios, Galvatron always achieves superior system throughput compared to previous work with limited parallelism.<\/jats:p>","DOI":"10.14778\/3570690.3570697","type":"journal-article","created":{"date-parts":[[2023,1,23]],"date-time":"2023-01-23T12:29:55Z","timestamp":1674476995000},"page":"470-479","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":66,"title":["Galvatron"],"prefix":"10.14778","volume":"16","author":[{"given":"Xupeng","family":"Miao","sequence":"first","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yujie","family":"Wang","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youhe","family":"Jiang","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunan","family":"Shi","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaonan","family":"Nie","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hailin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Cui","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,1,23]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2021. DeepSpeed 3D Parallelism. https:\/\/github.com\/microsoft\/Megatron-DeepSpeed\/blob\/main\/examples\/pretrain_bert_distributed_with_mp.sh.  2021. DeepSpeed 3D Parallelism. https:\/\/github.com\/microsoft\/Megatron-DeepSpeed\/blob\/main\/examples\/pretrain_bert_distributed_with_mp.sh."},{"key":"e_1_2_1_2_1","unstructured":"2021. NVIDIA collective communications library (NCCL). https:\/\/developer.nvidia.com\/nccl.  2021. NVIDIA collective communications library (NCCL). https:\/\/developer.nvidia.com\/nccl."},{"key":"e_1_2_1_3_1","unstructured":"2021. PyTorch GPipe. https:\/\/pytorch.org\/docs\/stable\/pipeline.html.  2021. PyTorch GPipe. https:\/\/pytorch.org\/docs\/stable\/pipeline.html."},{"key":"e_1_2_1_4_1","unstructured":"2022. Galvatron. https:\/\/github.com\/PKU-DAIR\/Hetu\/tree\/main\/tools\/Galvatron.  2022. Galvatron. https:\/\/github.com\/PKU-DAIR\/Hetu\/tree\/main\/tools\/Galvatron."},{"key":"e_1_2_1_5_1","volume-title":"Fairscale: A general purpose modular pytorch library for high performance and large scale training.","author":"Baines Mandeep","year":"2021","unstructured":"Mandeep Baines , Shruti Bhosale , Vittorio Caggiano , Naman Goyal , Siddharth Goyal , Myle Ott , Benjamin Lefaudeux , Vitaliy Liptchinsky , Mike Rabbat , Sam Sheiffer , 2021 . Fairscale: A general purpose modular pytorch library for high performance and large scale training. Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, et al. 2021. Fairscale: A general purpose modular pytorch library for high performance and large scale training."},{"key":"e_1_2_1_6_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS.  Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS."},{"key":"e_1_2_1_7_1","first-page":"1967","article-title":"TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism","volume":"33","author":"Cai Zhenkun","year":"2022","unstructured":"Zhenkun Cai , Xiao Yan , Kaihao Ma , Yidi Wu , Yuzhen Huang , James Cheng , Teng Su , and Fan Yu . 2022 . TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism . IEEE TPDS 33 , 8 (2022), 1967 -- 1981 . Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu. 2022. TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism. IEEE TPDS 33, 8 (2022), 1967--1981.","journal-title":"IEEE TPDS"},{"key":"e_1_2_1_8_1","volume-title":"Quoc Viet Le, and Ruslan Salakhutdinov","author":"Dai Zihang","year":"2019","unstructured":"Zihang Dai , Zhilin Yang , Yiming Yang , Jaime G. Carbonell , Quoc Viet Le, and Ruslan Salakhutdinov . 2019 . Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In ACL. 2978--2988. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In ACL. 2978--2988."},{"key":"e_1_2_1_9_1","volume-title":"Ng","author":"Dean Jeffrey","year":"2012","unstructured":"Jeffrey Dean , Greg Corrado , Rajat Monga , Kai Chen , Matthieu Devin , Quoc V. Le , Mark Z. Mao , Marc'Aurelio Ranzato , Andrew W. Senior , Paul A. Tucker , Ke Yang , and Andrew Y . Ng . 2012 . Large Scale Distributed Deep Networks. In NeurIPS. 1232--1240. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, and Andrew Y. Ng. 2012. Large Scale Distributed Deep Networks. In NeurIPS. 1232--1240."},{"key":"e_1_2_1_10_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. 4171--4186.","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. 4171--4186. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. 4171--4186."},{"key":"e_1_2_1_11_1","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.  Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR."},{"key":"e_1_2_1_12_1","unstructured":"Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang Zhen Zheng Chuan Wu Guoping Long Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu and Wei Lin. 2021. DAPPLE: a pipelined data parallel approach for training large models. In PPoPP. ACM 431--445.  Shiqing Fan Yi Rong Chen Meng Zongyan Cao Siyu Wang Zhen Zheng Chuan Wu Guoping Long Jun Yang Lixue Xia Lansong Diao Xiaoyong Liu and Wei Lin. 2021. DAPPLE: a pipelined data parallel approach for training large models. In PPoPP. ACM 431--445."},{"key":"e_1_2_1_13_1","volume-title":"HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen.","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang , Youlong Cheng , Ankur Bapna , Orhan Firat , Dehao Chen , Mia Xu Chen , HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019 . GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS. 103--112. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS. 103--112."},{"key":"e_1_2_1_14_1","volume-title":"ICML","volume":"80","author":"Jia Zhihao","year":"2018","unstructured":"Zhihao Jia , Sina Lin , Charles R. Qi , and Alex Aiken . 2018 . Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks . In ICML , Vol. 80 . PMLR, 2279--2288. Zhihao Jia, Sina Lin, Charles R. Qi, and Alex Aiken. 2018. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks. In ICML, Vol. 80. PMLR, 2279--2288."},{"key":"e_1_2_1_15_1","unstructured":"Zhihao Jia Matei Zaharia and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In MLSys.  Zhihao Jia Matei Zaharia and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In MLSys."},{"key":"e_1_2_1_16_1","volume-title":"OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning. ICML Hardware Aware Efficient Training (HAET) Workshop","author":"Jiang Youhe","year":"2022","unstructured":"Youhe Jiang , Xupeng Miao , Xiaonan Nie , and Bin Cui . 2022 . OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning. ICML Hardware Aware Efficient Training (HAET) Workshop (2022). Youhe Jiang, Xupeng Miao, Xiaonan Nie, and Bin Cui. 2022. OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning. ICML Hardware Aware Efficient Training (HAET) Workshop (2022)."},{"key":"e_1_2_1_17_1","unstructured":"Jared Kaplan Sam McCandlish Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (2020). arXiv:2001.08361  Jared Kaplan Sam McCandlish Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (2020). arXiv:2001.08361"},{"key":"e_1_2_1_18_1","volume-title":"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. 7871--7880.","author":"Lewis Mike","year":"2020","unstructured":"Mike Lewis , Yinhan Liu , Naman Goyal , Marjan Ghazvininejad , Abdelrahman Mohamed , Omer Levy , Veselin Stoyanov , and Luke Zettlemoyer . 2020 . BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. 7871--7880. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. 7871--7880."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415530"},{"key":"e_1_2_1_20_1","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692  Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692"},{"key":"e_1_2_1_21_1","volume-title":"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows","author":"Liu Ze","year":"2021","unstructured":"Ze Liu , Yutong Lin , Yue Cao , Han Hu , Yixuan Wei , Zheng Zhang , Stephen Lin , and Baining Guo . 2021 . Swin Transformer: Hierarchical Vision Transformer using Shifted Windows . In ICCV. IEEE , 9992--10002. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV. IEEE, 9992--10002."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Xupeng Miao Xiaonan Nie Yingxia Shao Zhi Yang Jiawei Jiang Lingxiao Ma and Bin Cui. 2021. Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce. In SIGMOD. ACM 2262--2270.  Xupeng Miao Xiaonan Nie Yingxia Shao Zhi Yang Jiawei Jiang Lingxiao Ma and Bin Cui. 2021. Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce. In SIGMOD. ACM 2262--2270.","DOI":"10.1145\/3448016.3452773"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-022-3581-9"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Xupeng Miao Yining Shi Hailin Zhang Xin Zhang Xiaonan Nie Zhi Yang and Bin Cui. 2022. HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training. In SIGMOD. 470--480.  Xupeng Miao Yining Shi Hailin Zhang Xin Zhang Xiaonan Nie Zhi Yang and Bin Cui. 2022. HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training. In SIGMOD. 470--480.","DOI":"10.1145\/3514221.3517902"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3489496.3489511"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407816"},{"key":"e_1_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Deepak Narayanan Aaron Harlap Amar Phanishayee Vivek Seshadri Nikhil R. Devanur Gregory R. Ganger Phillip B. Gibbons and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. In SOSP. 1--15.  Deepak Narayanan Aaron Harlap Amar Phanishayee Vivek Seshadri Nikhil R. Devanur Gregory R. Ganger Phillip B. Gibbons and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. In SOSP. 1--15.","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_2_1_28_1","unstructured":"Deepak Narayanan Amar Phanishayee Kaiyu Shi Xie Chen and Matei Zaharia. 2021. Memory-Efficient Pipeline-Parallel DNN Training. In ICML. 7937--7947.  Deepak Narayanan Amar Phanishayee Kaiyu Shi Xie Chen and Matei Zaharia. 2021. Memory-Efficient Pipeline-Parallel DNN Training. In ICML. 7937--7947."},{"key":"e_1_2_1_29_1","first-page":"1","article-title":"Efficient large-scale language model training on GPU clusters using megatron-LM","volume":"58","author":"Narayanan Deepak","year":"2021","unstructured":"Deepak Narayanan , Mohammad Shoeybi , Jared Casper , Patrick LeGresley , Mostofa Patwary , Vijay Korthikanti , Dmitri Vainbrand , Prethvi Kashinkunti , Julie Bernauer , Bryan Catanzaro , Amar Phanishayee , and Matei Zaharia . 2021 . Efficient large-scale language model training on GPU clusters using megatron-LM . In SC. ACM , 58 : 1 -- 58 :15. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In SC. ACM, 58:1--58:15.","journal-title":"SC. ACM"},{"key":"e_1_2_1_30_1","volume-title":"TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting","author":"Nie Xiaonan","year":"2022","unstructured":"Xiaonan Nie , Xupeng Miao , Zhi Yang , and Bin Cui . 2022 . TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting . In ICDE. IEEE , 2615--2628. Xiaonan Nie, Xupeng Miao, Zhi Yang, and Bin Cui. 2022. TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting. In ICDE. IEEE, 2615--2628."},{"key":"e_1_2_1_31_1","volume-title":"Hetu-MoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System. arXiv:2203.14685","author":"Nie Xiaonan","year":"2022","unstructured":"Xiaonan Nie , Pinxue Zhao , Xupeng Miao , Tong Zhao , and Bin Cui . 2022. Hetu-MoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System. arXiv:2203.14685 ( 2022 ). Xiaonan Nie, Pinxue Zhao, Xupeng Miao, Tong Zhao, and Bin Cui. 2022. Hetu-MoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System. arXiv:2203.14685 (2022)."},{"key":"e_1_2_1_32_1","volume-title":"PyTorch: An Imperative Style","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas K\u00f6pf , Edward Yang , Zach DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie Bai , and Soumith Chintala . 2019. PyTorch: An Imperative Style , High-Performance Deep Learning Library . NeurIPS ( 2019 ). Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS (2019)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-021-00155-3"},{"key":"e_1_2_1_34_1","volume-title":"ICML","volume":"139","author":"Radford Alec","year":"2021","unstructured":"Alec Radford , Jong Wook Kim , Chris Hallacy , Aditya Ramesh , Gabriel Goh , Sandhini Agarwal , Girish Sastry , Amanda Askell , Pamela Mishkin , Jack Clark , Gretchen Krueger , and Ilya Sutskever . 2021 . Learning Transferable Visual Models From Natural Language Supervision . In ICML , Vol. 139 . PMLR, 8748--8763. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML, Vol. 139. PMLR, 8748--8763."},{"key":"e_1_2_1_35_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever etal 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9.  Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9."},{"key":"e_1_2_1_36_1","volume-title":"Liu","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2020 . Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR ( 2020). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR (2020)."},{"key":"e_1_2_1_37_1","volume-title":"ZeRO: memory optimizations toward training trillion parameter models","author":"Rajbhandari Samyam","unstructured":"Samyam Rajbhandari , Jeff Rasley , Olatunji Ruwase , and Yuxiong He. 2020. ZeRO: memory optimizations toward training trillion parameter models . In SC. IEEE\/ACM. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: memory optimizations toward training trillion parameter models. In SC. IEEE\/ACM."},{"key":"e_1_2_1_38_1","volume-title":"ICML","volume":"139","author":"Ramesh Aditya","year":"2021","unstructured":"Aditya Ramesh , Mikhail Pavlov , Gabriel Goh , Scott Gray , Chelsea Voss , Alec Radford , Mark Chen , and Ilya Sutskever . 2021 . Zero-Shot Text-to-Image Generation . In ICML , Vol. 139 . PMLR, 8821--8831. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In ICML, Vol. 139. PMLR, 8821--8831."},{"key":"e_1_2_1_39_1","volume-title":"Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms","author":"Rashidi Saeed","unstructured":"Saeed Rashidi , Matthew Denton , Srinivas Sridharan , Sudarshan Srinivasan , Amoghavarsha Suresh , Jade Nie , and Tushar Krishna . 2021. Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms . In ISCA. IEEE , 540--553. Saeed Rashidi, Matthew Denton, Srinivas Sridharan, Sudarshan Srinivasan, Amoghavarsha Suresh, Jade Nie, and Tushar Krishna. 2021. Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms. In ISCA. IEEE, 540--553."},{"key":"e_1_2_1_40_1","volume-title":"Deep-speed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD. 3505--3506.","author":"Rasley Jeff","year":"2020","unstructured":"Jeff Rasley , Samyam Rajbhandari , Olatunji Ruwase , and Yuxiong He . 2020 . Deep-speed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD. 3505--3506. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep-speed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD. 3505--3506."},{"key":"e_1_2_1_41_1","volume-title":"Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro.","author":"Smith Shaden","year":"2022","unstructured":"Shaden Smith , Mostofa Patwary , Brandon Norick , Patrick LeGresley , Samyam Rajbhandari , Jared Casper , Zhun Liu , Shrimai Prabhumoye , George Zerveas , Vijay Korthikanti , Elton Zheng , Rewon Child , Reza Yazdani Aminabadi , Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022 . Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model . (2022). arXiv:2201.11990 Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. (2022). arXiv:2201.11990"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Fei Sun Jun Liu Jian Wu Changhua Pei Xiao Lin Wenwu Ou and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In CIKM. 1441--1450.  Fei Sun Jun Liu Jian Wu Changhua Pei Xiao Lin Wenwu Ou and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In CIKM. 1441--1450.","DOI":"10.1145\/3357384.3357895"},{"key":"e_1_2_1_43_1","volume-title":"Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al.","author":"Unger Colin","year":"2022","unstructured":"Colin Unger , Zhihao Jia , Wei Wu , Sina Lin , Mandeep Baines , Carlos Efrain Quintero Narvaez , Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al. 2022 . Unity : Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In OSDI. 267--284. Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al. 2022. Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In OSDI. 267--284."},{"key":"e_1_2_1_44_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS. 5998--6008.  Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS. 5998--6008."},{"key":"e_1_2_1_45_1","first-page":"1","article-title":"Supporting Very Large Models using Automatic Dataflow Graph Partitioning","volume":"26","author":"Wang Minjie","year":"2019","unstructured":"Minjie Wang , Chien-Chin Huang , and Jinyang Li . 2019 . Supporting Very Large Models using Automatic Dataflow Graph Partitioning . In EuroSys. ACM , 26 : 1 -- 26 :17. Minjie Wang, Chien-Chin Huang, and Jinyang Li. 2019. Supporting Very Large Models using Automatic Dataflow Graph Partitioning. In EuroSys. ACM, 26:1--26:17.","journal-title":"EuroSys. ACM"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-022-2140-7"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-022-00187-3"},{"key":"e_1_2_1_48_1","volume-title":"Automatic cross-replica sharding of weight update in data-parallel training. arXiv:2004.13336","author":"Xu Yuanzhong","year":"2020","unstructured":"Yuanzhong Xu , HyoukJoong Lee , Dehao Chen , Hongjun Choi , Blake Hechtman , and Shibo Wang . 2020. Automatic cross-replica sharding of weight update in data-parallel training. arXiv:2004.13336 ( 2020 ). Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang. 2020. Automatic cross-replica sharding of weight update in data-parallel training. arXiv:2004.13336 (2020)."},{"key":"e_1_2_1_49_1","unstructured":"Yuanzhong Xu HyoukJoong Lee Dehao Chen Blake Hechtman Yanping Huang Rahul Joshi Maxim Krikun Dmitry Lepikhin Andy Ly Marcello Maggioni etal 2021. GSPMD: general and scalable parallelization for ML computation graphs. arXiv:2105.04663 (2021).  Yuanzhong Xu HyoukJoong Lee Dehao Chen Blake Hechtman Yanping Huang Rahul Joshi Maxim Krikun Dmitry Lepikhin Andy Ly Marcello Maggioni et al. 2021. GSPMD: general and scalable parallelization for ML computation graphs. arXiv:2105.04663 (2021)."},{"key":"e_1_2_1_50_1","volume-title":"MLSys","author":"Yang Bowen","year":"2021","unstructured":"Bowen Yang , Jian Zhang , Jonathan Li , Christopher R\u00e9 , Christopher R. Aberger , and Christopher De Sa. 2021. PipeMare: Asynchronous Pipeline Parallel DNN Training . In MLSys 2021 . Bowen Yang, Jian Zhang, Jonathan Li, Christopher R\u00e9, Christopher R. Aberger, and Christopher De Sa. 2021. PipeMare: Asynchronous Pipeline Parallel DNN Training. In MLSys 2021."},{"key":"e_1_2_1_51_1","volume-title":"Do Transformers Really Perform Bad for Graph Representation? arXiv:2106.05234","author":"Ying Chengxuan","year":"2021","unstructured":"Chengxuan Ying , Tianle Cai , Shengjie Luo , Shuxin Zheng , Guolin Ke , Di He , Yanming Shen , and Tie-Yan Liu . 2021. Do Transformers Really Perform Bad for Graph Representation? arXiv:2106.05234 ( 2021 ). Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021. Do Transformers Really Perform Bad for Graph Representation? arXiv:2106.05234 (2021)."},{"key":"e_1_2_1_52_1","volume-title":"Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning","author":"Zheng Lianmin","year":"2022","unstructured":"Lianmin Zheng , Zhuohan Li , Hao Zhang , Yonghao Zhuang , Zhifeng Chen , Yanping Huang , Yida Wang , Yuanzhong Xu , Danyang Zhuo , Eric P. Xing , Joseph E. Gonzalez , and Ion Stoica . 2022 . Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning . In OSDI. USENIX Association , 559--578. Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In OSDI. USENIX Association, 559--578."},{"key":"e_1_2_1_53_1","unstructured":"Martin Zinkevich M. Weimer Alex Smola and L. Li. 2010. Parallelized Stochastic Gradient Descent. In NeurIPS.  Martin Zinkevich M. Weimer Alex Smola and L. Li. 2010. Parallelized Stochastic Gradient Descent. In NeurIPS."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3570690.3570697","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,23]],"date-time":"2023-01-23T12:35:47Z","timestamp":1674477347000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3570690.3570697"}},"subtitle":["Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism"],"short-title":[],"issued":{"date-parts":[[2022,11]]},"references-count":53,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10.14778\/3570690.3570697"],"URL":"https:\/\/doi.org\/10.14778\/3570690.3570697","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2022,11]]},"assertion":[{"value":"2023-01-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}